Synthetic Media and Deepfakes

Photographs and recordings have been manipulable for a century, but manipulation used to cost skill, time and money. Text to image and text to video models, voice cloning that needs a few seconds of sample audio, and single purpose applications that undress a photograph remove all three costs at once. The International AI Safety Report 2026 describes the change in exactly these terms: accessible tools have lowered the barrier to producing harmful synthetic content at scale, many are free or cheap, most require no technical skill, and they can be used anonymously.^[1]

Two different harms usually travel under the single word deepfake. The first is deception, where a fabricated artefact is believed and someone acts on it, transferring money or drawing a conclusion about a person. The second is degradation of the evidentiary baseline. Once fakes are known to be easy, a genuine recording can be waved away. The legal scholars Robert Chesney and Danielle Citron named the second effect the liar's dividend, and it has since been tested experimentally.^[2]

A third strand does not depend on anyone being fooled at all. A sexual image of a real person generated without consent injures that person whether or not viewers think it is authentic, and the same holds for AI generated child sexual abuse material. The International AI Safety Report 2026 draws the policy consequence directly: detection alone cannot fully address harms of this kind, because the content remains harmful when it is correctly identified as AI generated.^[1]

Fraud is where measurement is best. The FBI Internet Crime Complaint Center gave artificial intelligence its own section for the first time in the 2025 Internet Crime Report, published in 2026. The centre logged 22,364 complaints containing a reported AI element, with adjusted losses of $893,346,472, against 1,008,597 complaints and $20.877 billion in total reported losses for the year, a 26% rise in losses over 2024.^[3] The report adds a caution that matters more than the headline: investment scam complaints with an AI element accounted for over $632 million, while investment scam losses overall exceeded $8 billion, which the centre reads as evidence that many victims never learn AI was involved.^[3]

Sexual abuse imagery is the second area with hard numbers. The Internet Watch Foundation reported in January 2026 that its analysts confirmed 3,440 AI generated videos of child sexual abuse during 2025, against 13 in 2024, and that 65% of those videos fell into Category A, the most severe class under UK law, compared with 43% of non AI criminal videos it assessed over the same year.^[4] For adults, a survey of more than 16,000 people across ten countries by Rebecca Umbach, Nicola Henry, Gemma Beard and Colleen Berryessa found that 2.2% reported that someone had made non consensual intimate imagery of them, and 1.8% reported having done it to someone else.^[5]

Detection is weaker than public discussion assumes. Viacheslav Pirogov and Maksim Artemev tested deepfake image detectors against a set of more than 500,000 generated images in 2025 and found that fewer than half reached an area under the curve above 60%, with the lowest at 50%, and that ordinary JPEG compression pushed performance down further.^[6] On the human side, the International AI Safety Report 2026 records that participants took GPT-4o text for human writing 77% of the time after a five minute conversation, that listeners misidentified synthetic voices as real about 80% of the time in one audio study, and that warning labels on AI generated video moved identification accuracy only from 10.7% to 21.6%.^[1]

Exposure is broad, and public concern runs ahead of it. Tvesha Sippy, Florence Enock, Jonathan Bright and Helen Margetts surveyed 1,403 UK adults for the Alan Turing Institute in 2024 and found that 50.2% had seen celebrity deepfakes and 34.1% political ones, while 90.4% said they were very or somewhat concerned about the spread of the technology.^[7] Pew Research Center, surveying 9,720 US adults between 26 August and 2 September 2024, found 57% extremely or very concerned that AI would be used to make and distribute false information about candidates.^[8] The AI Incident Database counted 362 documented AI incidents in 2025 against 233 in 2024, a series the Stanford Institute for Human-Centered AI reproduces in the 2026 AI Index.^[9]

The clearest documented mechanism is impersonation inside an existing chain of authority. Hong Kong police reported in February 2024 that a finance employee at the engineering firm Arup made fifteen transfers totalling about $25 million after joining a video call on which every other participant, including a figure presenting as a senior finance executive, was synthetic. Arup confirmed in May 2024 that it was the company involved.^[10] What made the attack work was not picture quality on its own but the pairing of a plausible pretext with a live channel that staff had been taught to treat as a check on identity.

The liar's dividend is the harder problem because it needs no fake at all. Kaylyn Jackson Schiff, Daniel Schiff and Natalia Bueno ran five preregistered experiments with more than 15,000 US adults and found that politicians who answered a true text based scandal by calling the reporting misinformation gained support, with pooled effects of roughly 0.17 to 0.21 standard deviations, and that the gains appeared across partisan groups rather than only among co-partisans. Their video findings were weaker: in most of the studies, claiming that damaging video was a deepfake did not work.^[2]

Volume changes the shape of some harms even when quality does not. The move from 13 confirmed AI abuse videos in 2024 to 3,440 in 2025 is the signature of a capability becoming widely available rather than of a large increase in the number of offenders.^[4] Where the injury depends on quantity, such as saturating a school or a workplace with images of one person, cheap generation converts an offence that once required effort into one that requires a prompt. Researchers cited in the International AI Safety Report 2026 also raise the risk that synthetic media weakens the standing of audiovisual evidence in legal proceedings.^[1]

The strongest counterargument concerns elections, where the predictions were loudest. Sam Stockwell of the Centre for Emerging Technology and Security at the Alan Turing Institute examined AI enabled disinformation around the 2024 UK, French and European Parliament elections and identified 27 viral cases, 16 in the UK general election and 11 in the EU and French contests. The report found no evidence that AI enabled disinformation or deepfakes meaningfully affected the UK or European results, and concluded that generative AI did less to make disinformation go viral than traditional interference methods and human influencers did.^[11][13]

Felix Simon, Sacha Altay and Hugo Mercier argued in the Harvard Kennedy School Misinformation Review in October 2023 that fears about generative AI and misinformation were overblown, on three grounds. The supply of false content already exceeded demand for it before generative models existed, human attention rather than production cost is the binding constraint, and people who consume false content generally go looking for it.^[12] Simon and Altay returned to the question for the Knight First Amendment Institute in July 2025 and concluded that AI's influence on the 2024 national elections was small next to other factors, citing among other evidence the meta-analytic finding that the best estimate of the persuasive effect of campaign contact and advertising on candidate choice in US general elections is zero.^[13]

Public alarm may be tracking coverage rather than exposure. Yan, Morrow, Yang and Wihbey reported in the Harvard Kennedy School Misinformation Review in 2025 that in their US survey, direct use of ChatGPT and DALL-E showed no meaningful association with worry about AI driven election misinformation, while television news consumption was the strongest predictor, particularly among respondents aged 65 and over.^[14] That does not make the underlying risk unreal, but it does suggest the level of public fear is not a reading of how much synthetic content people have actually met.

The courtroom evidence is the sharpest available test of the liar's dividend, and it has not gone the alarming way so far. At the request of the Judicial Conference Advisory Committee on Evidence Rules, the Federal Judicial Center surveyed federal district, magistrate and bankruptcy judges, along with the Court of International Trade and the Court of Federal Claims, in January 2026. Of 931 respondents, a 45% response rate, 15 said a litigant in one of their cases had challenged a video, photograph or audio item on the ground that it was a deepfake, and ten of those fifteen had seen exactly one such challenge.^[15]

The TAKE IT DOWN Act became Public Law 119-12 on 19 May 2025. It criminalises publishing, and threatening to publish, non consensual intimate imagery, including AI generated digital forgeries, and requires covered platforms to operate a notice and removal process that takes reported images down within 48 hours of a valid request. The criminal provisions took effect on enactment, and the statute gives platforms one year to establish the removal process, so that duty applied from 19 May 2026.^[16] State law has moved faster and less evenly, and some of it has not survived review. Judge John Mendez of the Eastern District of California held on 29 August 2025 in Kohls v. Bonta that California's AB 2839, which restricted materially deceptive election related synthetic media, discriminates on content, viewpoint and speaker and violates the First Amendment.^[17]

In the European Union, the transparency rules in Article 50 of the AI Act apply from 2 August 2026, with a limited grace period to 2 December 2026 for systems already on the market. Providers of generative systems must mark synthetic output so that it is machine detectable, deployers must disclose deepfakes visibly, and AI generated text published on matters of public interest must be labelled where no human editor has reviewed it. Article 3(60) defines a deepfake as AI generated or manipulated image, audio or video content that resembles existing persons, objects, places, entities or events and would falsely appear to a person to be authentic or truthful.^[18] China acted earlier: the Cyberspace Administration of China and three other agencies issued labelling measures on 7 March 2025, in force from 1 September 2025, requiring both visible labels and metadata markers on AI generated text, images, audio, video and virtual scenes on Chinese platforms.^[19]

Provenance work runs alongside the law. The Coalition for Content Provenance and Authenticity publishes Content Credentials, a specification that attaches signed provenance data to a file so that a viewer can see how it was made and edited. Its steering members include Adobe, Amazon, the BBC, Google, Intel, Meta, Microsoft, OpenAI, Publicis Groupe, Sony and Truepic, with more than 500 companies listed as supporting the standard. A public Verify tool lets anyone inspect whatever credentials a file still carries.^[20] The International AI Safety Report 2026 is cautious about how much this buys: watermarks can be stripped or spoofed, especially where model weights are public, content logs raise privacy questions, and the report notes research finding that deepfake detection benchmarks are dated and that tools perform about 50% worse on real world material than on the sets used to score them.^[1]

Institutional plumbing is being adjusted more slowly. The US Judicial Conference Advisory Committee on Evidence Rules is weighing an amendment to Rule 901 on authentication and commissioned the Federal Judicial Center survey to decide whether one is needed.^[15] Enforcement in the best known election case has been mixed. The Federal Communications Commission proposed a $6 million forfeiture on 23 May 2024 against Steven Kramer, the political consultant behind the robocall that used a cloned Joe Biden voice two days before the January 2024 New Hampshire primary.^[21] The transmitting carrier Lingo Telecom agreed to pay $1 million in August 2024, and Kramer still faced the FCC fine when a New Hampshire jury acquitted him of all 22 state charges of voter suppression and candidate impersonation on 13 June 2025.^[22]

Sort claims by how well they are measured rather than by how frightening they sound. Fraud and sexual image abuse have national reporting series behind them and are rising. The claim that synthetic media has swung a national election has no documented case behind it from the 2024 cycle. The liar's dividend sits in between: experimentally real, largely invisible in court records so far, and the hardest of the three to regulate without running into free expression limits.

Practical habits follow from the failure modes rather than from the technology. Fraud attacks work by turning a live channel into a proof of identity, so the defence is an out of band check on a number you already had, not sharper eyes. Content Credentials tell you where a file came from when they are present, but their absence proves nothing, because ordinary sharing strips metadata. Treat any detector score as weak evidence: the Pirogov and Artemev results show that tools trained on curated benchmarks lose much of their accuracy on material found in the wild.^[6]