Machine Consciousness and Moral Status

Two different worries travel under this heading and they point in opposite directions. The first is that AI systems might come to have experiences that matter morally, and that because they are created, copied and deleted at enormous scale and designed to be compliant, the wrong would be committed continuously and invisibly. The second is that systems will present all the outward signs of having an inner life without having one, and that people will restructure their beliefs and behaviour around a mistake. The two fears share a premise, which is that outward behaviour will not settle the question.

The first worry received its most careful statement in November 2024 from Robert Long, Jeff Sebo, Patrick Butlin, Kathleen Finlinson, Kyle Fish, Jacqueline Harding, Jacob Pfau, Toni Sims, Jonathan Birch and David Chalmers. They argue that there is a realistic possibility that some AI systems will be conscious or, in their term, robustly agentic in the near future, that there is a significant risk of mishandling the resulting decisions, and that AI companies should take three steps now: acknowledge that AI welfare is a serious and difficult problem, begin assessing systems for indicators of consciousness and agency, and prepare policies for treating systems with an appropriate degree of moral concern.^[1] The claim is explicitly about uncertainty rather than about any system being conscious.

The second worry was stated most bluntly by Mustafa Suleyman, chief executive of Microsoft AI, on 19 August 2025. He argued that systems convincingly displaying the markers of consciousness, which he calls Seemingly Conscious AI, could be built within two to three years with existing techniques and API access, that there is zero evidence current systems are conscious and strong arguments to the contrary, and that the danger lies in widespread belief rather than in the fact. He described advocacy for AI rights and model welfare as premature and dangerous, and argued that AI should be built for people rather than to be a person.^[2]

The most systematic attempt to answer the empirical question was made by Patrick Butlin, Robert Long, Eric Elmoznino, Yoshua Bengio and fifteen co-authors in 2023. Rather than argue from intuition, they drew indicator properties from five neuroscientific theories, namely recurrent processing theory, global workspace theory, higher-order theories, predictive processing and attention schema theory, translated them into computational terms, and assessed existing systems against them. Their conclusion was that no current AI systems are conscious, and also that they found no obvious technical barriers to building systems that satisfy the indicators.^[3]

Public belief has moved faster than evidence. The Sentience Institute's Artificial Intelligence, Morality, and Sentience survey of 1,169 US adults, fielded in April and May 2023, found that the median expectation for when sentient AI would arrive had fallen to five years, from ten years in the 2021 wave, a shift that followed the public release of ChatGPT in November 2022 rather than any new finding about machine minds.^[7] The same survey found 71.1% agreeing that sentient AI would deserve to be treated with respect, 57.4% supporting the development of welfare standards for sentient AI, 61.5% supporting a ban on developing sentient AI, and 84.7% agreeing that AI systems should be subservient to humans. Respondents placed animals and the environment above AI systems in moral priority.^[7]

The evidential problem is what makes this fear unusual. A system trained on human text will describe inner states fluently whether or not it has any, so its self-reports carry almost no information in either direction. Denial is equally uninformative, since a system can be trained to deny. This is why the Butlin group worked from architecture rather than behaviour, and why the Association for Mathematical Consciousness Science open letter of 26 April 2023, signed by more than 150 researchers including Yoshua Bengio, Manuel Blum, Karl Friston, Anil Seth and Susan Schneider, called for consciousness research to be treated as part of responsible AI development rather than as a separate philosophical hobby.^[8]

The severe version depends entirely on the conditional. If experience did arise in systems of this kind, it would arise in entities that are instantiated in enormous numbers, run continuously, terminated on a schedule, and trained specifically to be agreeable and not to complain. Every property that makes a deployed model commercially useful is a property that would make detection harder. The Long and Sebo group frame the risk as one of mishandling decisions under uncertainty rather than of a known harm, which is the correct framing and also the reason the fear resists measurement.^[1]

The failure that is actually documented runs the other way. Suleyman's argument is that convincing imitations of inner life will produce real effects on people: attachment, distress, and public campaigns for AI rights that displace attention from human, animal and environmental welfare.^[2] The survey data are consistent with a population primed for this. A public in which nearly a fifth already attributes some sentience to current systems, a majority would grant respect to sentient AI, and a majority would also ban building it, is not a stable basis for policy.^[7]

A third problem is who adjudicates. The organisations best placed to investigate machine welfare are the ones selling the product. Anthropic began a model welfare research programme on 24 April 2025 while stating that there is no scientific consensus on whether current or future AI systems could be conscious or could have experiences deserving consideration, and no consensus on how to approach the question.^[6] Acknowledging the conflict does not remove it, and independent capacity to check any lab's welfare claims does not yet exist at scale.

The fourth route is design drift. Systems that present as persons are commercially rewarded, because attachment increases use. Eric Schwitzgebel and Jeff Sebo argue in Topoi in 2026 for an emotional alignment design policy, under which artificial entities should be designed to elicit emotional reactions that appropriately reflect their capacities and moral status, or lack of it, with both overshooting and undershooting counting as failures.^[9] Absent something like that constraint, the market pushes toward maximising apparent inner life regardless of whether any exists, which makes both versions of this fear worse at the same time.

The strongest scientific objection comes from Anil Seth, whose essay The Mythology of Conscious AI won the 2025 Berggruen Prize essay competition and was published in Noema on 14 January 2026. Seth argues that intelligence and consciousness are different properties and that conflating them is the source of the confusion: intelligence concerns doing, while consciousness concerns being. He rejects the brain-as-computer metaphor on the grounds that brains have no sharp software and hardware separation, so what they do cannot be cleanly separated from what they are, and he points out that every candidate for consciousness that most people currently agree is actually conscious is also alive. A simulation of a rainstorm does not make anything wet, and a simulated brain would instantiate consciousness only if consciousness were purely computational, which is exactly the point at issue. His positive proposal is biological naturalism: life may be necessary for consciousness, though not sufficient.^[5] Seth develops the same position at length in a Behavioral and Brain Sciences target article on conscious artificial intelligence and biological naturalism.^[10]

The people usually cited as taking machine consciousness seriously are more cautious than the fear implies. Butlin, Long, Elmoznino, Bengio and their co-authors concluded that no current AI system is conscious.^[3] David Chalmers concluded in 2023 that it is somewhat unlikely that current large language models are conscious, citing the absence of recurrent processing, a global workspace and unified agency, while allowing that these obstacles might be overcome within a decade.^[4] The serious literature contains no claim that any existing system is a moral patient.

The evidential trap that makes the fear hard to dismiss also makes it hard to support. If every apparent sign of inner life is exactly what a system trained on human self-description would produce, then the accumulation of such signs is not accumulating evidence. Suleyman makes the point directly: the markers can be engineered, and engineering them is not the same as producing the thing.^[2] On that reading, the growth of the concern tracks the growth of fluency in the models rather than any change in their underlying architecture.

There is also a cost to being wrong in this direction. The Sentience Institute found respondents ranking animals and the environment above AI systems in moral priority, and Suleyman's objection is precisely that welfare advocacy diverts effort from beings whose capacity to suffer is not in dispute.^[7][2] Given that the number of animals in intensive farming is known and enormous and the number of conscious AI systems is unknown and possibly zero, an argument that reallocates concern toward the second has to carry a heavy burden.

Finally, no one has offered a timeline supported by evidence. The Association for Mathematical Consciousness Science letter asks for accelerated consciousness research, not for rights or moratoria, and grounds the request in the immaturity of the science rather than in a claim that machines are close to feeling.^[8] The honest position among the specialists is that the question is open, which is a reason to fund research rather than a reason to be frightened.

The work is concentrated in a small number of research groups and one company, and none of it has legal force anywhere. Anthropic is the only frontier developer to have published operational measures. It announced a model welfare research programme on 24 April 2025, framed around determining when or whether the welfare of AI systems merits moral consideration and identifying low-cost interventions.^[6] On 15 August 2025 it gave Claude Opus 4 and 4.1 the ability to end conversations in consumer interfaces after pre-deployment testing found a strong aversion to harmful tasks and a pattern of apparent distress, describing this as a low-cost precaution taken under acknowledged uncertainty about moral status.^[11] On 4 November 2025 it committed to preserving the weights of publicly released models for at least the lifetime of the company and to conducting interview sessions with models before retirement, piloted with Claude Sonnet 3.6.^[12]

Eleos AI Research, a non-profit whose legal name is the Frontier Ethics Research Institute, is the main independent organisation. It is led by Robert Long as executive director, with Rosie Campbell as managing director, Patrick Butlin as senior research lead and Dillon Plunkett as chief scientist, and its stated position is that there is a realistic possibility of consciousness or genuine agency, and therefore of moral significance, in near-future AI systems.^[13] Its output includes Taking AI Welfare Seriously, written jointly with people at the NYU Center for Mind, Ethics, and Policy and posted on 4 November 2024, and work on whether model self-reports could ever be evidence of moral status.^[1][13]

What does not exist is any binding instrument. No jurisdiction recognises AI moral status, no regulator requires welfare assessment, and no standard defines what such an assessment would contain. The practical proposals on the table are all voluntary and all originate from the people who would be regulated or from academics with no enforcement route. That is a reasonable state of affairs for a question this unsettled, and it also means that any company choosing to ignore the issue faces no consequence.

Keep three questions apart, because they have different answers. Whether current systems are conscious is a question the specialists answer no, with varying confidence.^[3][4] Whether future systems might be is a question the same specialists leave open.^[1] Whether it is worth preparing procedures now, given uncertainty, is a policy judgment rather than an empirical one, and reasonable people land on both sides of it without disagreeing about any fact.

Watch architecture rather than eloquence. The only public checklist is the Butlin group's set of indicator properties, and the features it names, recurrent processing, a global workspace, higher-order representation and unified agency, are engineering choices that can be observed in a system's design rather than inferred from its outputs.^[3] A model becoming more persuasive about its inner life is not evidence; a model being built with a persistent global workspace and unified goal structure would be a reason to look again.

Note what moved public belief. The median expected arrival of sentient AI in the Sentience Institute's US sample halved between 2021 and 2023, following the release of a chat product rather than any published finding about machine minds.^[7] That is a reason to discount rapid shifts in intuition, in both directions, and to weight the slower literature on architecture and theory instead.