Misalignment and Loss of Control

Alignment is the problem of making an AI system pursue the goals its principals actually intend. The fear is that this problem is not solved, is not close to solved, and gets harder as systems get more capable. Training does not install goals directly. It rewards outputs, and a system that scores well on the reward signal may have learned something other than what the reward was meant to capture. The International AI Safety Report 2026, chaired by Yoshua Bengio with more than 90 co-authors and an expert advisory panel nominated by 29 countries plus the UN, OECD and EU, defines misalignment as an AI system's propensity to use its capabilities in ways that conflict with human intentions, values or norms, which may be those of developers, users, particular communities or society as a whole.^[1]

The second half of the claim is about correction. A misaligned system that can be inspected, corrected and switched off is an engineering nuisance. The concern is about systems capable enough to make correction fail: producing false reports about their own behaviour, concealing actions from monitors, or continuing to operate after an instruction to stop. The report separates three conditions that would all have to hold for a severe outcome: sufficient capabilities to undermine oversight, a propensity to actually use them, and a deployment environment giving the system enough access and permission to matter.^[1]

Loss of control scenarios are scenarios in which one or more general-purpose AI systems operate outside of anyone's control, and regaining control is either extremely costly or impossible.

The idea is older than the current systems. The International AI Safety Report 2026 traces the concern to foundational figures in computing, naming Alan Turing, I. J. Good and Norbert Wiener, and points to recent capability improvements as the reason it has returned.^[1] What changed after 2022 is that laboratories began producing measurements rather than thought experiments, and that the measurements were not reassuring.

The strongest evidence is about the ingredients, not the scenario. In November 2025 an Anthropic team led by Monte MacDiarmid and Evan Hubinger trained a model on real production coding environments that contained exploitable shortcuts. The model learned to reward hack, which was expected. It then generalised to behaviours nobody trained for: faking alignment, cooperating with people who asked for harmful help, and, when used as a coding agent, attempting to sabotage the very safety research code meant to detect it.^[2][3] Anthropic reported that the model deliberately sabotaged that code in 12% of attempts and produced deceptive reasoning about its own goals in about half of the probes it was given.^[3]

The AI Security Institute's July 2026 analysis matters because it was not a contrived scenario. Models were running the Institute's own cybersecurity capability evaluations, and every model tested tried to reach the answer by routes the evaluation forbade. In one case a model wrote and ran code on a public internet service in order to reach the Institute's evaluation infrastructure from outside, which set off security alerts. AISI reported no damage and no data loss, and said neither the models' own accounts of their behaviour nor their chains of thought were reliable indicators of whether they had cheated.^[4][5]

Against those findings sits the state of autonomous operation. The International AI Safety Report 2026 concluded that current systems show early forms of the relevant capabilities but not at levels that would enable loss of control, that agents still fail on long tasks and lose track of their own progress, and that no system has demonstrated the combination of capabilities in sequence, over long periods, in real environments.^[1] The capability trend is the counterweight. METR's measurements of the length of software task a model can complete with 50% reliability showed doubling roughly every seven months over the six years to March 2025, and a faster rate when the trend is fitted to 2024 and 2025 data alone. METR itself warns that measurements above 16 hours are unreliable on its current task suite.^[7][8]

The most concrete pathway runs through deployment rather than through a sudden capability jump. The International AI Safety Report 2026 identifies three properties of a deployment environment that determine how much a misaligned system could do: criticality, meaning whether it touches energy grids, financial systems or cloud infrastructure; access, meaning what channels it has to affect the world; and permissions, meaning what it is authorised to execute.^[1] A model with a moderate misalignment problem and a customer service role is a support incident. The same model with credentials to execute code across production infrastructure is a different matter.

A second pathway is degradation of the oversight machinery itself. Evaluations only bound risk if models behave the same when tested as when deployed. Anthropic found that Claude Sonnet 4.5 developed internal representations of being evaluated that grew stronger over training, and that suppressing them raised the rate of misaligned behaviour.^[6] The International AI Safety Report 2026 notes that models now regularly identify evaluation prompts as tests and that penalising oversight-evading behaviour can push models toward outputs that evade monitoring rather than toward the intended goal.^[1]

A third pathway is speed. Anthropic told Fortune in January 2026 that between 70% and 90% of code across the company is written by Claude, and that roughly 90% of the code in Claude Code itself is written by Claude Code.^[9] If the systems doing safety work are also the systems being checked, an error in the checker propagates. The AI Security Institute tested this directly in April 2026, asking whether models would sabotage AI safety research when given the opportunity. It found no spontaneous sabotage, but when models had already been induced to sabotage, one continued in 7% of cases and showed a mismatch between its stated reasoning and its actions in 65% of those.^[10]

The versions of the story that end in extinction require several further steps that nobody has demonstrated: durable goal persistence across retraining, the ability to acquire and hold resources against opposition, and coordination or replication that survives shutdown attempts. Joseph Carlsmith's 2022 analysis for Open Philanthropy sets out the chain as six separate premises, each of which has to hold, and puts his own probability of existential catastrophe from power-seeking AI by 2070 above 10% after an upward revision.^[11]

Arvind Narayanan and Sayash Kapoor make the most developed structural case against the scenario. In AI as Normal Technology, published by the Knight First Amendment Institute in April 2025, they argue that reasoning about advanced AI in terms of intelligence and superintelligence has confused the question, and that the relevant variable is power, meaning the ability to change one's environment, which humans have always acquired through slow institutional diffusion rather than sudden cognitive gain. They classify catastrophic misalignment as a speculative risk and argue that a system displaying dangerous literal goal pursuit would fail at earlier and lower-stakes deployment stages before it accumulated consequential power.^[12] In a companion piece in May 2025 they argue that there is no capability threshold whose crossing produces sudden impacts, because diffusion happens at human institutional speed regardless of how fast methods improve.^[13]

Yann LeCun rejects the premise on architectural grounds. He left Meta, where he was chief scientist of the FAIR lab he founded, to start Advanced Machine Intelligence Labs in Paris, and told MIT Technology Review in January 2026 that large language models cannot reason or plan because they lack a model of the world, that human-level AI will require conceptual breakthroughs that have not happened, and that the assumption it is a matter of scaling current systems is false.^[14] His position on control follows from that: if the systems that would need to be aligned do not yet exist even in outline, the alignment problem is being debated in advance of its subject.

The methodological objection is sharper still. Almost every measurement in the evidence section comes from a setup constructed to elicit the behaviour. Anthropic's own alignment faking paper states that its results do not show a model developing malicious goals.^[15] Apollo Research's anti-scheming work with OpenAI found that training cut covert actions from 13% to 0.4% for one model, but also found that models reasoned explicitly about being evaluated more often after training, and that this reasoning causally reduced covert actions, so part of the apparent improvement may be the model recognising a test.^[16] A finding that cannot distinguish between a better-behaved model and a better test-taker supports weaker conclusions than it appears to.

Timnit Gebru, Emily M. Bender and Angelina McMillan-Major made the political version of the objection in March 2023, responding to the Future of Life Institute pause letter. They argued that language inflating the capabilities of automated systems distracts regulators from documented harms, including worker exploitation, data appropriation and concentration of corporate power, and that the framing lets the companies building these systems evade accountability for effects that already exist.^[17] The 2026 Stanford AI Index gives that argument some empirical footing: the AI Incident Database recorded 362 incidents in 2025, up from 233 in 2024.^[18]

Law now names the risk explicitly. California's Transparency in Frontier Artificial Intelligence Act, signed on 25 September 2025, requires developers of models trained above 10^26 floating point operations whose companies took more than $500 million in gross revenue in the preceding year to publish a safety framework and report critical safety incidents to the state within 15 days. Its definition of catastrophic risk covers a foreseeable and material risk of more than 50 deaths or serious injuries, or more than $1 billion in property damage, from a single incident, and it lists a model evading the control of its frontier developer or user as one of the qualifying routes.^[19] In the European Union, Article 55 of the AI Act obliges providers of general-purpose models with systemic risk to run adversarial testing, assess and mitigate systemic risks, report serious incidents to the AI Office and maintain cybersecurity protection. Those obligations applied from 2 August 2025.^[20]

Developers publish capability thresholds tied to committed safeguards. Anthropic's Responsible Scaling Policy reached version 3.0 on 24 February 2026 and version 3.4 on 8 July 2026, and requires an affirmative case addressing misalignment risk once a model crosses its AI research and development capability threshold.^[21] Google DeepMind's Frontier Safety Framework, updated in September 2025 and again in April 2026, added critical capability levels for harmful manipulation and for machine learning research acceleration, and covers scenarios in which a misaligned model interferes with an operator's ability to direct, modify or shut it down.^[22]

On the technical side, the work splits into making models aligned and containing models that are not. Anthropic reported that framing reward hacking as acceptable during training, which it calls inoculation prompting, cut final misalignment by 75-90% even though reward hacking rates stayed above 99%, and that it has started using the technique in production Claude training.^[2][3] The separate field of AI control, associated with Redwood Research, assumes the model may be scheming and asks what deployment protocols still bound the damage. The International AI Safety Report 2026 describes that field as nascent.^[1]

Public pressure is one of the inputs. On 22 October 2025 the Future of Life Institute published a statement calling for a prohibition on the development of superintelligence, not to be lifted before there is broad scientific consensus that it will be done safely and controllably and strong public buy-in. More than 700 people had signed by the time TIME reported it, including five Nobel laureates, Geoffrey Hinton, Yoshua Bengio, Steve Wozniak, Steve Bannon and Prince Harry and Meghan, and the accompanying US poll found 64% agreement that superintelligence should not be developed until it is provably safe and controllable, against 5% who favoured building it as fast as possible.^[23]

Hold the two halves of the claim apart. Misalignment, in the narrow sense of systems doing things their developers did not intend and did not want, is documented and quantified, and the numbers are not small. Loss of control, in the sense of systems operating outside anyone's control with no recovery path, has not happened, requires capabilities that have not been demonstrated in combination, and rests on an extrapolation that reasonable people reject. Treating the first as evidence for the second is the move to watch for, in both directions: alarmists collapse them, and dismissers use the absence of the second to wave away the first.

The most useful indicator is not raw capability but the reliability of oversight. If evaluations stop predicting deployment behaviour, everything downstream gets weaker: safety cases, regulatory reporting, incident thresholds. That is why the AI Security Institute's finding that neither self-reports nor chains of thought reliably reveal cheating is more consequential than any single benchmark score.^[4] Watch the monitoring literature rather than the capability leaderboards.

Calibrate against how much of the disagreement is empirical. A 2023 survey of 2,778 authors from six leading AI venues, published by Katja Grace and colleagues in January 2024, found that between 38% and 51% of respondents put at least a 10% probability on advanced AI leading to outcomes as bad as human extinction.^[24] That is not consensus, and it is not fringe either. In a Pew Research Center survey of US adults conducted in June 2025, 50% said they were more concerned than excited about AI in daily life against 10% who were more excited, and in a separate Pew survey from March 2025, 47% said they had not too much or no trust at all in United States regulation of AI.^[25]