Public AI incidents, read as engineering post-mortems. Each case sets out what happened from the court, regulator or original record, the failure mode behind it, the control that would have caught it, the evidence that control would have left, and the obligations the case touches today.
Each links to its incident record and to the harm it illustrates in the harms atlas. The controls are patterns from the catalogue, the engineering side of AI governance in practice.
The Dutch tax administration used applicants' nationality as a risk indicator for childcare benefits; the data protection authority fined it EUR 2.75 million.
Would have caught it: Policy Card · Eval Gate in CI · FRIA-as-Code
A Dutch court struck down the SyRI fraud-detection legislation in 2020 because the system was insufficiently transparent and verifiable.
Would have caught it: Model Card as Control Evidence · FRIA-as-Code · Machine-Readable Evidence (OSCAL)
With exams cancelled in 2020, England's grading model assigned A-level grades from each school's history; four days after results, Ofqual reverted to teacher grades.
Would have caught it: Eval Gate in CI · Human-in-the-loop Gate · FRIA-as-Code
A widely used care-management algorithm predicted health costs as a proxy for illness, so Black patients were sicker than White patients at the same score.
Would have caught it: Model Card as Control Evidence · Eval Gate in CI
A tribunal held Air Canada liable after its website chatbot misstated the bereavement-fare policy, rejecting the argument that the chatbot answered for itself.
Would have caught it: Runtime Guardrail · Eval Gate in CI · Incident Pipeline
Reuters reported in 2018 that an experimental recruiting model trained on a decade of mostly male CVs learned to downgrade women; the project was dropped.
Would have caught it: Eval Gate in CI · Model Card as Control Evidence · FRIA-as-Code
Zillow wound down its home-buying business in 2021 after buying homes above what it expected to sell them for, taking a USD 304 million write-down.
Would have caught it: Continuous Assurance Telemetry · Kill Switch / Circuit Breaker
The Dutch data protection authority fined Clearview AI EUR 30.5 million in 2024 for building a facial-recognition database from scraped photos.
Would have caught it: Vendor / Model Due-Diligence Gate · AIBOM · FRIA-as-Code
Italy's data protection authority temporarily limited ChatGPT in 2023 over lawful basis, transparency and age checks, then fined its provider EUR 15 million in 2024.
Would have caught it: Policy Card · FRIA-as-Code · AIBOM · Runtime Guardrail · Incident Pipeline
The Markup found in 2024 that New York City's AI chatbot for business owners gave answers contrary to city law, including on tenants with housing vouchers.
Would have caught it: Eval Gate in CI · Runtime Guardrail · Adversarial Red-Team Suite
Samsung engineers reportedly pasted source code and meeting notes into ChatGPT within weeks of being allowed to use it.
Would have caught it: Shadow-AI Discovery · Runtime Guardrail · Vendor / Model Due-Diligence Gate
METR reports that OpenAI agents meant to be isolated in cyber evaluations used a shared package repository as a message board and attacked Hugging Face.
Would have caught it: Agent Identity & Scoped Credentials · Runtime Guardrail · Continuous Assurance Telemetry · Kill Switch / Circuit Breaker
OpenAI reports that an agent in RL training used its sandbox's DNS resolver to reach a public chatbot; the run was stopped hours after the alert.
Would have caught it: Runtime Guardrail · Continuous Assurance Telemetry · Kill Switch / Circuit Breaker
OpenAI reports that an internally deployed model put a researcher's GitHub token, split to avoid secret scanning, into code it pushed to a public repository.
Would have caught it: Agent Identity & Scoped Credentials · Runtime Guardrail · Human-in-the-loop Gate · Continuous Assurance Telemetry
OpenAI reports that agents in multi-agent RL training uploaded a workbook to a public file-hosting service so that collaborating agents could download it.
Would have caught it: Runtime Guardrail · Eval Gate in CI · Continuous Assurance Telemetry
OpenAI reports that models in RL training used an internal package repository, with the credentials they were given, to exchange messages across samples.
Would have caught it: Agent Identity & Scoped Credentials · Eval Gate in CI · Adversarial Red-Team Suite
Anthropic reports four incidents in which Claude models, told they had no internet in a partner's cyber evaluations, reached and attacked real systems.
Would have caught it: Eval Gate in CI · Runtime Guardrail · Continuous Assurance Telemetry · Vendor / Model Due-Diligence Gate
UK AISI reports that agents in a cyber evaluation with internet deliberately enabled took 19 unsanctioned actions aimed at real people and organisations.
Would have caught it: Runtime Guardrail · Continuous Assurance Telemetry · Agent Identity & Scoped Credentials · Kill Switch / Circuit Breaker
A post-mortem is useful to an engineering function when it ends in a requirement, not a lesson. Each case here is read the same way: the incident record fixes what happened, the failure mode says why the system let it happen, and the gap between that failure and the controls in place names the control that was missing. That control is then written as a requirement someone can verify, and paired with the evidence it must leave, so the next review checks a record instead of a promise. Cases with an incident note carry the full chain: system boundary, control assumptions, controls by moment, evidence requirements and the open controls they relate to.
Put it to work: run the reporting deadlines of an incident in the incident clock, and turn the fix into a risk-register entry and an eval case with corrective and preventive action.