Algorithmic Bias and Discrimination

The argument has three steps. A model is trained to predict a target drawn from historical records. Those records carry the effects of unequal access, unequal treatment and unequal measurement. A model that predicts them accurately will therefore reproduce the pattern, and because it is applied to every applicant rather than to the caseload of one officer, the error becomes systematic rather than idiosyncratic. Ziad Obermeyer and colleagues gave the clearest documented example in Science in 2019: a widely used health risk tool predicted future care spending as a stand-in for future medical need, and because less is spent on Black patients at equal levels of illness, the tool judged them healthier.^[1]

Almost no deployed system takes race or ethnicity as an explicit input, so the concern is about proxies. Postcode, name, spending history, language ability, benefit history and employment gaps all carry demographic information. The Rotterdam welfare fraud model examined by Lighthouse Reports and WIRED in 2023 used 315 inputs including age, gender, neighbourhood, marital status, language skills and subjective caseworker assessments to produce a score between 0 and 1, and the reporters concluded that it discriminated on ethnicity, age, gender and parenthood while judging people on characteristics they cannot control.^[2] In United States law this is the difference between disparate treatment, which requires intent, and disparate impact, which does not.

The third step concerns contestability. A person refused a tenancy, a loan or an interview often does not know that a scoring system was involved, which inputs it used, or how to appeal. A study by researchers at Cornell's Citizens and Technology Lab, Data and Society and Consumer Reports tested this directly against New York City's Local Law 144, which requires employers using automated employment decision tools to publish bias audits. Of 391 employers checked by 155 student investigators, 18 had posted an audit report and 13 had posted the required transparency notice.^[3]

The evidence base is unusually concrete for an AI fear, because the systems concerned are deployed, auditable and subject to civil rights statutes that predate them. Findings come from peer reviewed studies, government evaluations, regulator decisions and litigation records rather than from forecasts. The table below lists measurements taken on specific systems, with the year of publication.

Public concern tracks the evidence more closely here than for most AI fears. In a Pew Research Center survey conducted in summer and autumn 2024 and published in April 2025, 55% of US adults and 55% of a sample of AI experts said they were extremely or very concerned about bias in decisions made by AI. Pew noted that bias was one of the few items where the two groups converged.^[5]

Measurement of the models themselves is thinner than measurement of their effects. The Stanford HAI 2026 AI Index reported that while almost all leading frontier model developers report results on capability benchmarks, reporting on responsible AI benchmarks remains sparse, and that the average score on the Foundation Model Transparency Index fell to 40 in 2025 after rising from 37 to 58 across 2023 and 2024.^[4] That gap matters because general purpose models are now embedded in screening, triage and customer service tools that were previously bespoke and separately audited.

The Dutch childcare benefits affair is the most fully documented case of scale turning a scoring error into a national failure. The Tax Administration stopped the childcare allowances of thousands of parents, most of them with dual nationality, and treated them as fraudsters. Many had to repay tens of thousands of euros, were driven into debt and were shut out of other assistance because of the fraud label. The government resigned in its entirety on 15 January 2021, and by April more than 35,000 affected parents had registered for compensation.^[7] The regulator later established that nationality had been used as an indicator in the risk system itself.^[6] The failure was not one wrong score. It was a scoring system, a repayment rule and an appeals process that each assumed the others were working.

Policing produces a different failure mode, where a probabilistic match is treated as an identification. The American Civil Liberties Union counted 14 publicly known wrongful arrests in the United States attributed to police reliance on face recognition as of April 2026, beginning with Nijeer Parks in February 2019 and Robert Williams in January 2020. Most of those known cases involved Black people, though the ACLU also records white people among the wrongly arrested.^[8] Under the settlement of the Williams case, Detroit police may no longer request an arrest warrant on the basis of a photo lineup combined with a facial recognition lead alone.^[8]

A third route runs through consolidation. When a small number of vendors supply screening for a whole market, one flawed model becomes the effective gatekeeper. The class action Louis v. SafeRent Solutions alleged that a tenant screening score disadvantaged Black and Hispanic applicants and holders of housing vouchers because it did not account for public housing authorities paying over 73% of the monthly rent; Judge Angel Kelley of the District of Massachusetts approved a settlement of 2.275 million dollars on 20 November 2024, together with injunctive terms changing how SafeRent scores voucher applicants in Massachusetts.^[9]

The failure that would be hardest to reverse is loss of measurement. Disparate impact analysis requires collecting demographic data and running comparisons, and both are policy choices. Executive Order 14281, signed on 23 April 2025 and published in the Federal Register on 28 April, directs federal agencies including the Equal Employment Opportunity Commission to deprioritise enforcement based on disparate impact liability and to review existing regulations and consent judgments that impose it.^[10] Whatever one thinks of the doctrine, less federal testing means fewer public measurements of how deployed systems perform across groups.

The strongest technical objection is that the word bias hides a choice among incompatible standards. Sam Corbett-Davies, Johann Gaebler, Hamed Nilforoshan, Ravi Shroff and Sharad Goel argued in the Journal of Machine Learning Research in 2023 that the two dominant families of formal fairness definitions, those constraining the effect of decisions on group disparities and those constraining the effect of protected characteristics on decisions, typically yield policies that are strongly Pareto dominated, meaning outcomes could be improved for every group at once by abandoning the constraint.^[11] On that account a system labelled biased under one metric is often fair under another, and the label alone settles nothing.

The COMPAS dispute shows the practical form of this. Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner reported for ProPublica in May 2016 that 44.9% of Black defendants in Broward County were labelled higher risk and did not go on to reoffend, against 23.5% of white defendants, almost twice the rate.^[24] Anthony Flores, Kristin Bechtel and Christopher Lowenkamp replied in Federal Probation that same year, arguing from the same data that the instrument satisfied the standard tests of predictive bias and calibration, and that the gap in error rates followed arithmetically from different observed rearrest base rates rather than from bias in the tool.^[12] Both analyses were competent. They measured different things.

The comparison class is the second objection. Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan and Cass Sunstein argued in 2019 that algorithms make discrimination easier to detect and prove, because the objective, the training data and the inputs can be inspected and re-run, while a hiring manager's reasoning cannot.^[13] Bartlett and colleagues found that fintech lenders' rate disparities were similar to those of face-to-face lenders on agency mortgages and smaller on some FHA loans, which is a poor result for the technology but not evidence that it made lending worse.^[14]

Disparity is also not intrinsic to the method. NIST's 2019 evaluation, often cited as proof that face recognition is racially biased, reported explicitly that not all algorithms produced high false positive differentials and that the most equitable algorithms were among the most accurate.^[15] That finding cuts against the claim that error gaps are an unavoidable property of the technique rather than of particular training data and design choices.

Finally, corrections can fail in the other direction. On 22 February 2024 Google paused Gemini's generation of images of people after users complained that it depicted historical figures such as the United States Founding Fathers as people of colour. Google said the feature could generate a wide range of people but was missing the mark, and that an improved version would follow.^[16] The episode is a reminder that deciding what an unbiased output looks like is a contested value judgment, not a purely technical repair.

The EU AI Act is the most detailed legal response. Article 10 requires that training, validation and test datasets for high-risk systems be examined for possible biases likely to affect health, safety or fundamental rights, and it permits providers to process special categories of personal data where strictly necessary to detect and correct bias, subject to safeguards.^[17] The timetable has moved. High-risk obligations were originally due to apply from 2 August 2026, and the Digital Omnibus on AI, proposed by the European Commission on 19 November 2025, defers them; under the trilogue agreement reached on 7 May 2026 the stand-alone high-risk rules apply from 2 December 2027 and product-embedded systems from 2 August 2028.^[18]

Private litigation has become the main US pressure point while federal enforcement recedes. In Mobley v. Workday, Judge Rita Lin of the Northern District of California granted preliminary certification of a nationwide collective under the Age Discrimination in Employment Act on 16 May 2025, covering applicants aged 40 and over who were denied employment recommendations through Workday's platform since 24 September 2020.^[21] The SafeRent settlement in November 2024 imposed operational changes on a screening product rather than damages alone.^[9]

On the technical side, NIST's face recognition evaluations continue to publish per-algorithm demographic results, which gave procurement officers a comparative measure that did not previously exist.^[15] Standards uptake is growing in industry: the Stanford HAI 2026 AI Index reported that the share of surveyed organisations with no responsible AI policy fell from 24% to 11% during 2025, with ISO/IEC 42001 cited by 36% of respondents and the NIST AI Risk Management Framework by 33%.^[4] Policy documents are not audits, and the Index also recorded that reporting on responsible AI benchmarks remains sparse.^[4]

Three questions separate the strong version of this fear from the weak one. First, has a disparity actually been measured on the deployed system, or is the claim an inference from the training data? Second, does the disparity come from the model or from the world it is measuring, and does the distinction change what should be done? Third, what is the realistic alternative, and how well does it perform on the same measurement? A screening tool that is worse than a fair process but better than the process it replaced still deserves fixing, and saying so is not a defence of the tool.

Domain matters more than technology. The health risk tool studied by Obermeyer and colleagues was a simple regression, not a frontier model, and the fix was to change the prediction target rather than the algorithm.^[1] Conversely, general purpose language models now sit inside hiring and clinical workflows without the domain-specific validation that older scoring tools required, which is why the Wilson and Caliskan resume results and the Zack results on GPT-4 are more concerning than their effect sizes alone would suggest.^[22][23]

The most useful thing to track over the next few years is not the volume of alarming studies but the volume of published measurement. Bias audits, demographic evaluation reports and regulator decisions are what turn this fear into a testable claim. If audits stop being produced while deployment continues, the fear becomes both less falsifiable and more warranted at the same time.