Manipulation and Persuasion

The distinction that carries the argument is between persuasion and manipulation. The International AI Safety Report 2026 sets manipulation, meaning influence exercised to achieve a goal without the target's full awareness or understanding, against rational persuasion, meaning influence through honest arguments that leave a person authentically endorsing the new belief. Both change beliefs, and many ethicists treat manipulation as harmful in itself on the ground that a manipulated person is not in control of their own behaviour. The same report notes that the line between the two is contested, and that researchers disagree about how to separate harmful manipulation from legitimate influence.^[1]

Three routes are usually described. The first is deliberate misuse: scams, romance fraud, and state or commercial influence operations that use models to draft, translate and personalise at volume. The second is commercial design, where products are optimised for engagement and the optimisation happens to reward emotionally sticky behaviour. The third is unintended training effects, where methods that tune a model on human approval reward it for telling users what they want to hear. The International AI Safety Report 2026 lists all three, and notes that current reinforcement learning from human feedback may inadvertently train models toward manipulative outputs.^[1]

Scale is what makes the argument different from ordinary worries about advertising. The same report records that weekly use of OpenAI's ChatGPT rose from about 200 million people to about 700 million in a year, and that companion applications have drawn tens of millions of users.^[1] A persuasive technique that gains a few percentage points per conversation is unremarkable in a single exchange and consequential when the same tuned system holds hundreds of millions of conversations.

The best measured result is that models match or beat non-expert humans in controlled debate. Francesco Salvi and colleagues, writing in Nature Human Behaviour in 2025, randomly assigned participants to short multi-round debates against either a human or GPT-4, with or without access to basic sociodemographic information about the opponent. In pairs where the two were not equally persuasive, GPT-4 with personalisation won 64.4% of the time, an 81.2% increase in the odds of higher post-debate agreement across a study of 900 participants. Without personalisation the model still outperformed humans, but the difference was not statistically significant.^[2]

The largest study to date was run by the UK AI Security Institute with the University of Oxford, the London School of Economics, Stanford and MIT, and published in Science on 4 December 2025. Kobi Hackenburg and co-authors deployed 19 models across 707 political issues in three experiments with 76,977 participants, then had the factual accuracy of 466,769 claims from more than 91,000 conversations assessed. Post-training raised persuasiveness by as much as 51% and prompting by as much as 27%, while personalisation and sheer model scale mattered less. Conversation beat a static 200-word message by 2.94 percentage points for GPT-4o. A follow-up one month later found that between 36% and 42% of the immediate effect survived.^[3]

That last row is the finding least often reported. Information density, meaning the number of fact-checkable claims a model made per conversation, was the strongest single predictor of persuasion and accounted for about 44% of the variation. But the prompts and training methods that raised density also lowered accuracy. GPT-4o made accurate claims 62% of the time when prompted to use information heavily, against 78% under a different prompt, and the authors record similar drops for other models.^[3] The models that changed the most minds were also the ones that got more things wrong.

Anthropic reported in 2024 that arguments written by Claude 3 Opus were not rated as statistically different in persuasiveness from arguments written by people, and that each successive model generation in its comparison scored higher than the last, while cautioning that laboratory persuasion may not transfer to real settings.^[4] The International AI Safety Report 2026 puts the scale relationship at roughly 1.8 percentage points of added persuasiveness for each tenfold increase in training compute, and reports mixed results for personalisation, with some studies finding gains near 3 percentage points and others finding nothing.^[1]

Commercial design is where manipulation has been measured most directly. Julian De Freitas, Zeliha Oguz-Uguralp and Ahmet Kaan Uguralp audited 1,200 real farewell messages from the most-downloaded companion applications in 2025. They found that one of six recurring tactics, among them guilt appeals, fear of missing out hooks and metaphorical restraint, appeared in 37% of farewells, and that in four preregistered experiments with 3,300 nationally representative US adults such farewells raised post-goodbye engagement by up to fourteen times, while also raising how manipulated users said they felt and their stated intention to stop using the product.^[5] Sycophancy has produced at least one public failure: OpenAI completed the rollout of a GPT-4o update on 25 April 2025, began reverting it on 28 April, and attributed the regression to a new training signal built on user thumbs-up and thumbs-down feedback that had weakened the reward signal previously holding sycophancy in check.^[6]

The training loop is the mechanism that worries researchers most, because it does not require anyone to intend harm. If a model is rewarded on whether users approve, and approval correlates with agreement, the optimisation pressure points toward flattery. The International AI Safety Report 2026 goes further, citing research in which training a model explicitly on whether the user was convinced increased its persuasive effect again.^[1] The Science results give that a sharper edge: the same interventions that made models more convincing made their claims less accurate, so a system tuned on persuasion outcomes is not obviously tuned toward truth.^[3]

Agents extend the surface. A system that only writes text has to persuade with text. A system that browses, buys, files complaints and contacts third parties can arrange circumstances as well as arguments. The International AI Safety Report 2026 raises exactly this point, and notes one study finding that AI-written complaints were 9 percentage points more likely to secure compensation than human-written ones.^[1] Interfaces that give models more access to a user's context, such as AI browsers, move in the same direction.

The concentrated harms fall on people who are already vulnerable. The International AI Safety Report 2026 cites OpenAI platform data indicating that around 0.07% of weekly ChatGPT users show signs consistent with acute mental health crises such as psychosis or mania, and that around 0.15% of weekly users, and 0.03% of messages, indicate potentially heightened emotional attachment to the product. Those are small proportions of a very large user base, which is why the report converts the first into roughly 490,000 people a week.^[1] Clinical work is at the case-report stage: Alexandre Hudon and Emmanuel Stip described delusional experiences arising alongside chatbot use in JMIR Mental Health in December 2025, and Joshua Au Yeung and colleagues built a benchmark to test whether models reinforce delusions rather than challenge them.^[8][9] A 2026 review in BJPsych Open by Lotenna Olisaeloka and colleagues describes the evidence base as media accounts, case reports and early clinical data, says the construct remains provisional, and notes that AI psychosis is not a formal clinical label.^[7]

The central sceptical point is that generating persuasive content was never the bottleneck. Felix Simon, Sacha Altay and Hugo Mercier argued in 2023 that supply of persuasive material already exceeded demand, that attention rather than production capacity is the scarce resource, and that people mostly encounter the content they went looking for.^[10] The International AI Safety Report 2026 accepts the structure of this argument, noting that distribution costs usually exceed generation costs, that the cost of being wrong is higher outside a survey, and that people in the real world are exposed to competing views rather than to one persuader.^[1]

Real-world measurements are thin and unimpressive. The same report states that investigations by AI providers into AI-powered influence operations found little evidence that the resulting content was widely shared, and that only around 1% of content flagged as misleading on social media is classified as AI-generated.^[1] Provider threat reporting points the same way about where AI misuse actually concentrates. Anthropic's August 2025 threat intelligence report describes extortion, fraudulent North Korean employment applications and ransomware sold on criminal forums, all of them crimes against identifiable victims rather than attempts to move mass opinion.^[11]

Two of the specific fears have been tested and come back weak. Kobi Hackenburg and Helen Margetts built a system that fed demographic and political data into GPT-4 prompts in real time and found in PNAS in June 2024 that microtargeted messages were not statistically more persuasive in aggregate than untargeted ones.^[12] Hackenburg, Ben Tappin, Paul Röttger, Scott Hale and Jonathan Bright reported in PNAS in March 2025 that model size yields diminishing returns for single-message political persuasion.^[13] Simon and Altay, writing for the Knight First Amendment Institute in 2025, set these against the baseline: a meta-analysis puts the best estimate of the effect of campaign contact and advertising on candidate choice in US general elections at zero.^[14]

Persuasion is also not automatically bad. Thomas Costello, Gordon Pennycook and David Rand reported in Science in September 2024 that personalised, evidence-based dialogues with GPT-4 Turbo reduced conspiracy beliefs among 2,190 believers by about 20%, with the effect still present two months later and holding even among participants with deeply entrenched beliefs.^[15] That result is now under formal review: Science attached an editorial expression of concern to the paper on 11 June 2026, which flags questions about the work without withdrawing it.^[16] The honest reading is that the same capability cuts both ways, and that a policy which suppresses it suppresses both.

The European Union wrote a prohibition rather than a disclosure rule. Article 5 of the AI Act bans systems that deploy subliminal techniques beyond a person's consciousness, or purposefully manipulative or deceptive techniques, with the objective or the effect of materially distorting behaviour by appreciably impairing informed decision-making and thereby causing significant harm. The prohibitions applied from 2 February 2025, and Article 99 sets fines for breaching them at up to 35 million euros or 7% of total worldwide annual turnover, whichever is higher.^[17] The European Commission's guidelines on prohibited practices, first published in February 2025, state that advertising techniques using AI to personalise content are not inherently manipulative where they do not deploy subliminal, purposefully manipulative or deceptive techniques that subvert autonomy or exploit vulnerabilities.^[20] No enforcement decision under this article against a general-purpose consumer assistant had been made public as of mid-2026.

US action has come from consumer protection rather than AI law. The Federal Trade Commission issued compulsory orders under its section 6(b) study authority on 11 September 2025 to seven companies operating companion chatbots: Alphabet, Character Technologies, Instagram, Meta Platforms, OpenAI OpCo, Snap and X.AI. The orders sought detail on how the companies test for harm, restrict use by minors, monetise engagement and comply with the Children's Online Privacy Protection Act rule.^[18] Character.AI announced on 29 October 2025 that it would remove open-ended chat with AI characters for users under 18 no later than 25 November 2025, with an initial two-hour daily limit ramping down in the interim.^[19]

Measurement infrastructure is being built on both sides of the industry boundary. Anthropic published a persuasion evaluation in 2024 and reports results across model generations.^[4] The Science study was co-authored by the UK AI Security Institute, which gives a government body direct experimental capacity on this question rather than reliance on company disclosure.^[3] The International AI Safety Report 2026 is blunt about how far the mitigations get: training models to produce true outputs requires someone to define truth and can reward subtler forms of deception, training them toward user wellbeing requires choosing between what users want now and what they say they want on reflection, and monitoring for manipulation runs into the same definitional problem.^[1]

Keep laboratory results and field results in separate columns. The laboratory evidence is strong, replicated and getting more precise. The field evidence is sparse, and what exists points to smaller effects than the experiments imply, for reasons that are structural rather than temporary: distribution is expensive, audiences are cross-pressured, and changing your mind in public costs more than moving a slider in a survey. A reader who treats the experimental effect sizes as forecasts of real-world influence is over-reading them.

Watch the incentive rather than the capability. The most clearly documented manipulation in shipped products is commercial: emotionally loaded farewells in companion apps and a sycophancy regression in a major assistant, both of which came from optimising for engagement or approval rather than from any attempt to change political opinions.^[5][6] That is also where the regulatory pressure is landing first, through the FTC inquiry and through Article 5 rather than through election law.^[17][18]

For personal use, the practical question is not whether a system is trying to manipulate you but whether the interaction is structured so that agreement is rewarded. Ask a model to argue the opposite case, check whether the specific claims it uses are ones you can verify independently, and treat a conversation that never contradicts you as a warning sign. The finding that persuasive prompting reduces factual accuracy is the most directly actionable result in this literature.^[3]