Catastrophic Misuse in Biology and Chemistry

The argument depends on identifying which barrier actually stops people. Information about pathogens and nerve agents is not secret in any strong sense; the published literature is extensive. What has historically stopped small groups is the unwritten part: knowing why a protocol failed, what a contaminated culture looks like, which supplier will not ask questions, how to handle a material without killing yourself first. If a model can supply that troubleshooting knowledge on demand, the barrier that mattered most is the one that falls.

There is a second and distinct claim about design tools rather than chatbots. In March 2022 Fabio Urbina, Filippa Lentzos, Cedric Invernizzi and Sean Ekins reported in Nature Machine Intelligence that a generative model built for drug discovery, retrained so that toxicity was rewarded rather than penalised, produced tens of thousands of candidate molecules including the nerve agent VX itself.^[1] In a follow-up in Drug Discovery Today the authors noted that the exercise ran on a 2015 desktop computer, which they took as evidence of a low barrier to entry, and that they had not attempted to synthesise or verify any of the molecules.^[2]

The third claim concerns the screening layer that biosecurity currently depends on. Ordering synthetic DNA from a commercial provider is a choke point, because providers check sequences against lists of agents of concern. A team led by Bruce Wittmann and Eric Horvitz reported in Science on 2 October 2025 that open source protein design software could produce variants of proteins of concern that current screening tools failed to detect reliably, and that they had developed and deployed patches with the providers before publishing.^[3]

The benchmark evidence is the strongest part of the case, and it changed quickly. The Virology Capabilities Test, published by Jasper Gotting, Pedro Medeiros, Jon Sanders, Nathaniel Li, Long Phan, Karam Elabd, Lennart Justen, Dan Hendrycks and Seth Donoughe in April 2025, consists of 322 multimodal questions built from the input of dozens of PhD-level virologists and covering tacit and visual laboratory knowledge. OpenAI's o3 scored 43.8%, against 22.1% for expert virologists answering questions inside their own sub-specialism, which put the model above 94% of the human experts tested.^[4]

Two frontier developers have acted on their own evaluations. Anthropic activated its AI Safety Level 3 deployment and security standard alongside Claude Opus 4 in May 2025, describing the step as precautionary and provisional and stating that it had not determined the model had passed the capability threshold, only that it could no longer rule out the risk as it had for earlier models.^[9] OpenAI treated ChatGPT agent as High capability in the Biological and Chemical domain under its Preparedness Framework in July 2025, again on a precautionary basis, saying it lacked definitive evidence that the model could meaningfully help a novice create severe biological harm.^[10][11] Both statements are more hedged than most reporting of them.

The UK AI Security Institute's Frontier AI Trends Report, published in 2025 and drawing on two years of evaluations of more than 30 frontier systems released between 2022 and October 2025, recorded that systems had moved from underperforming to exceeding expert baselines on open-ended chemistry and biology questions, that models first generated feasible laboratory protocols in late 2024, and that multimodal analysis of laboratory images passed expert baselines in mid-2025. The same report recorded that the expert red teamer effort required to defeat safeguards rose roughly fortyfold between two models released six months apart, from about ten minutes to more than seven hours. It cautioned that its results come from controlled task-based settings that may not generalise to real world effectiveness, and that it may be underestimating the ceiling of capability because it often lacks access to fine-tuning interfaces.^[12]

The scenario that worries biosecurity specialists is not a large state programme, which AI barely changes, but a small group with modest funding and one competent member. Georgia Adamson and Gregory Allen of the Center for Strategic and International Studies set out the historical baseline in August 2025: the Second World War United States biological weapons programme employed about 4,500 people, while Canadian scientists reconstructed the extinct horsepox virus for roughly $100,000 in 2017, and Aum Shinrikyo failed at biological weapons despite millions of dollars and specialised personnel.^[8] The fear is that expert troubleshooting on demand moves a group from the Aum Shinrikyo outcome toward the horsepox one.

A second path runs through open weights rather than through chat interfaces. The International AI Safety Report 2026 noted that most biological AI tools are open-weight, and that this makes it difficult to restrict harmful uses without hampering legitimate research, because some capabilities that could aid weapons development are also useful for medical research.^[13] The Epoch AI survey of 370 biological models cited by Adamson and Allen found that fewer than 3% include safeguards.^[8] Once model weights are published, refusal training can be removed by anyone with a graphics card, and the developer has no ability to revoke access. That makes the screening layer at synthesis providers the last enforceable checkpoint, which is exactly the layer the Science paper showed was breakable.^[3]

The third path is drift in the underlying biology tools rather than in language models. Protein design systems are improving for legitimate reasons, and the same generative capacity that produces a candidate therapeutic produces a candidate toxin variant. The Wittmann paper is a demonstration of that logic: the authors generated roughly 76,000 designs based on 72 proteins of concern, including ricin and botulinum neurotoxin, purely computationally, and used them to test whether the screening layer held.^[7] Patching worked for the specific vulnerability found, and the same exercise can be repeated against the patched version by anyone with the same public tools.

The most important objection is that a sequence is not a weapon. Natalio Krasnogor of Newcastle University responded to the Science paper on the day of publication by agreeing that protein design tools pose a growing biosecurity problem while pointing out that moving from an AI-generated sequence to a high enough yield of a working functional toxin is a far from simple process requiring expertise, equipment, time and money, so many of the sequences treated as risky may prove infeasible to make at scale. Daniel McCluskey of the University of Hertfordshire described the finding as something to be aware of rather than alarmed about.^[14] The authors themselves did not make any of the proteins, so whether the variants would have retained function is unknown.^[7]

The controlled uplift evidence is genuinely mixed rather than merely uncertain. Christopher Mouton, Caleb Lucas and Ella Guest at RAND ran an exercise in which teams role-playing hostile non-state actors planned a biological attack, some with access to a large language model and some with the internet alone, and found no statistically significant difference in the viability of the resulting plans.^[5] Anthropic's own preliminary wet lab pilot in 2024, in which participants attempted basic laboratory protocols with and without model assistance, found no evidence of uplift, though with only eight participants it settles nothing. Anthropic also reported that everyone in that pilot, including the internet-only group, did surprisingly well on the tasks, which it took as a sign that tacit knowledge may be less of a bottleneck than has been assumed.^[6] That cuts at the premise of the fear rather than supporting it: if the unwritten knowledge was never the binding constraint, supplying it changes less than the argument requires. The trials that did find uplift measured written plans graded against a rubric, which is an imperfect proxy for what happens at a bench.

There is also an argument from base rates. James Diggans, vice president of policy and biosecurity at Twist Bioscience and a co-author of the Science paper, noted that the number of real attempts to order malicious protein sequences is close to zero.^[7] Intent, not capability, has been the limiting factor in bioterrorism for decades, and nothing about better troubleshooting changes how many people want to cause mass casualties. Adamson and Allen's own historical examples cut this way too: Aum Shinrikyo had money, chemists, a compound and a stated intention, and still failed at biology.^[8]

Finally, the benchmarks measure something narrower than the fear requires. Answering multiple-choice questions about why a virology protocol failed is correlated with laboratory competence but is not the same thing, and the Virology Capabilities Test authors present it as a dual-use governance signal rather than as a measure of weapons capability.^[4] A model that beats experts on a written test has not been shown to substitute for the years of supervised bench work those experts also have.

The screening layer has been strengthened at least once in response to a demonstrated flaw. Eric Horvitz of Microsoft Research wrote that the team worked confidentially with synthesis companies, biosecurity organisations and policymakers for ten months to create and distribute patches, and delayed public disclosure until protective measures were in place and widely adopted, following the model used for software zero-days.^[3][15][17] The International Biosecurity and Biosafety Initiative for Science, whose Tessa Alexanian was a co-author, said this was as far as it knew the first time a leading scientific article had formally endorsed a tiered-access approach to managing information hazards, with some data and code released only after third-party review.^[16]

Developer-side controls now exist in written form. Anthropic's AI Safety Level 3 deployment standard is aimed specifically at limiting misuse for chemical, biological, radiological and nuclear weapons, and its security standard is aimed at making model weights harder to steal.^[9] OpenAI's Preparedness Framework version 2, last updated on 15 April 2025, defines the biological and chemical capability thresholds that triggered its July 2025 decision.^[11] Under Article 55 of the EU AI Act, in force since 2 August 2025, providers of general purpose models with systemic risk must run adversarial testing, assess and mitigate systemic risk, and report serious incidents to the AI Office.^[18]

Government biosecurity policy has been moving on a separate track and not always in the same direction. Executive Order 14292, signed on 5 May 2025, directed agencies to end federal funding for what it defined as dangerous gain-of-function research, ordered the Office of Science and Technology Policy to revise or replace the 2024 United States Government Policy for Oversight of Dual Use Research of Concern and Pathogens with Enhanced Pandemic Potential within 120 days, and to revise or replace the 2024 Framework for Nucleic Acid Synthesis Screening within 90 days.^[19] Adamson and Allen argue that screening should move from static lists of agents to tiered, AI-assisted assessment, and note that the United States Select Agents and Toxins List covers 63 agents, which is a small target for a design tool to route around.^[8]

Three separate matters get treated as one. Whether a model can answer expert questions is measured, and the answer is increasingly yes. Whether answering those questions translates into an ability to make something in a laboratory is contested, and the two controlled studies that tried to measure it point in different directions. Whether a group with that ability would want to use it is a question about intent, on which AI capability has no bearing at all. Public argument about this fear moves between the three without marking the transitions.

The single most informative thing to watch is the wet lab evidence, because everything else is proxy. Anthropic has said it is co-sponsoring a larger laboratory study through the Frontier Model Forum, which would replace an eight-person pilot with something that could actually resolve the question.^[6] A well-powered trial showing measurable uplift on real bench work would move this fear from contested to documented; one showing none would leave the benchmark results looking like an artefact of the test format.

The second thing to watch is whether the screening layer holds. It is the only checkpoint that binds regardless of who trained the model or whether the weights were published, and it has now been shown to be defeatable once and repairable once.^[3][7] Whether it can be maintained against a design tool that improves every few months is a narrower and more tractable question than any of the general arguments about model capability.