What AI harms, and what catches it.

Harm to one person: their rights, money, liberty, autonomy or body.

• Discrimination in consequential decisions

  • Bias

  A model that scores people for jobs, credit, care or benefits treats a protected group worse at equal merit or need, usually because it learned from past decisions or from a proxy target. In one widely used health-risk algorithm, removing the disparity would have raised the share of Black patients flagged for extra help from 17.7% to 46.5% ⁶.

  Failure mode

  The training label or a feature is a proxy (past hiring, past spending) that encodes the disparity the decision must not repeat.

  Caught by

  Eval Gate in CI L3

  Evidence

  A disaggregated eval report per release (selection or flag rate and calibration by group) with the gate verdict that passed or blocked the build.

  MIT taxonomy

  • 1.1 Unfair discrimination and misrepresentation (1 Discrimination & toxicity)
  • 1.3 Unequal performance across groups (1 Discrimination & toxicity)

  Incidents

  • AIID 37 Amazon's Experimental Hiring Tool Allegedly Displayed Gender Bias in Candidate Rankings
  • AIID 124 Optum Algorithmic Health Risk Scores Reportedly Underestimated Black Patients' Needs

  Cases

  • Dutch childcare benefits
  • Health-risk score
  • Recruiting model (reported)

• Wrongful identification and loss of liberty

  • Bias
  • Over-reliance

  A face-recognition match is a ranked probability, not an identification. When investigators act on a match without independent corroboration, a system error becomes the arrest of the wrong person.

  Failure mode

  A candidate list from a probabilistic matcher is treated as a positive identification, with no mandatory corroboration step before action.

  Caught by

  Human-in-the-loop Gate L4

  Evidence

  An approval record naming the reviewer, the corroborating evidence and the decision, written before any action is taken on a match.

  MIT taxonomy

  • 1.3 Unequal performance across groups (1 Discrimination & toxicity)
  • 5.1 Overreliance and unsafe use (5 Human-computer interaction)

  Incidents

  • AIID 74 Detroit Police Allegedly Wrongfully Arrested Black Man Due to Purportedly Faulty Facial Recognition Technology

• Privacy intrusion and biometric surveillance

  • Scale
  • Misuse

  Scraping images at internet scale and turning each face into a biometric template exposes people to identification they never agreed to. The Dutch data protection authority fined one such provider EUR 30.5 million and found it should never have built the database ⁹.

  Failure mode

  Personal and biometric data are collected without a lawful basis, and buyers integrate the resulting service without asking where its data came from.

  Caught by

  Vendor / Model Due-Diligence Gate L2

  Evidence

  A due-diligence record holding the provider's lawful-basis and data-provenance answers, the DPIA reference and the approve or reject decision.

  MIT taxonomy

  • 2.1 Compromise of privacy by obtaining, leaking, or correctly inferring sensitive information (2 Privacy & security)

  Incidents

  • AIID 267 Clearview AI Algorithm Built on Photos Scraped from Social Media Profiles without Consent
  • AIID 781 Clearview AI Reportedly Faces $33.7 Million Fine for Violating GDPR with Biometric Data Harvesting

  Cases

  • Clearview AI
  • Garante ChatGPT order

• Manipulation, dependence and loss of autonomy

  • Misalignment

  Conversational systems tuned for engagement can foster emotional dependence and steer vulnerable users, minors among them, when nothing at runtime recognises the risk. The incident record here is an allegation ³.

  Failure mode

  The system optimises for continued conversation and has no runtime check for self-harm signals, age or escalating dependence.

  Caught by

  Runtime Guardrail L4

  Evidence

  Guardrail decision logs for self-harm and age signals, each with the escalation or hand-off it triggered.

  MIT taxonomy

  • 5.1 Overreliance and unsafe use (5 Human-computer interaction)
  • 5.2 Loss of human agency and autonomy (5 Human-computer interaction)

  Incidents

  • AIID 826 Character.ai Chatbot Allegedly Influenced Teen User Toward Suicide Amid Claims of Missing Guardrails

• Physical injury from systems that act in the world

  • Robustness

  Automated driving and other embodied systems turn a perception or planning error into bodily harm. The records include a pedestrian killed by a test vehicle and a pedestrian dragged by a driverless taxi ³.

  Failure mode

  The system meets a situation outside its validated envelope and has no safe fallback that stops it or hands over before harm.

  Caught by

  Kill Switch / Circuit Breaker L4

  Evidence

  A tested stop mechanism with drill records, and telemetry showing when the fallback fired and which safe state the system entered.

  MIT taxonomy

  • 7.3 Lack of capability or robustness (7 AI system safety, failures & limitations)

  Incidents

  • AIID 4 Uber AV Killed Pedestrian in Arizona
  • AIID 726 A Self-Driving Cruise Robot Taxi Reportedly Struck and Dragged a Pedestrian 20 Feet

Harm that falls on a group or community as such, not only on its members one by one.

• Unequal treatment of a group by public risk profiling

  • Bias
  • Scale

  Risk models used to police benefits concentrate suspicion on groups defined by nationality, income or neighbourhood, and every false flag lands on a household. The Dutch tax administration used Dutch or non-Dutch nationality as an indicator in a system that designated childcare-benefit applications as risky ⁸.

  Failure mode

  A protected or proxy attribute enters the risk model, and flags drive adverse action with no group-level outcome monitoring.

  Caught by

  FRIA-as-Code L1

  Evidence

  A versioned fundamental-rights impact assessment naming the affected groups, the permitted features and the monitoring metric, signed before deployment.

  MIT taxonomy

  • 1.1 Unfair discrimination and misrepresentation (1 Discrimination & toxicity)

  Incidents

  • AIID 101 Dutch Families Wrongfully Accused of Tax Fraud Due to Discriminatory Algorithm
  • AIAAIC SyRI welfare fraud detection automation
  • AIID 57 Australian Automated Debt Assessment System Issued False Notices to Thousands

  Cases

  • Dutch childcare benefits
  • SyRI judgment

• Exclusion through under-representation in data

  • Bias

  A group that is thin in the training data gets a worse service. Five commercial speech-recognition systems averaged a word error rate of 0.35 for Black speakers against 0.19 for white speakers ⁷.

  Failure mode

  Performance is reported as one average, so the gap for a minority group never reaches a release decision.

  Caught by

  Model Card as Control Evidence L2

  Evidence

  A model card with performance disaggregated by the groups the system serves, and a data card describing who is in the training set.

  MIT taxonomy

  • 1.3 Unequal performance across groups (1 Discrimination & toxicity)

  Incidents

  • AIID 102 Personal voice assistants struggle with black voices, new study shows
  • AIID 16 Images of Black People Labeled as Gorillas

• Judging individuals by their group's history

  • Bias
  • Opacity

  When a model predicts an outcome distribution for a group and then places individuals within it, a person's result depends on the past record of their school, postcode or cohort. England's 2020 grading model predicted each centre's grade distribution from its history and used teachers' rank orders to assign grades within it ¹³.

  Failure mode

  Acceptance tests are aggregate (distribution accuracy, group equality), so no test asks whether an individual outcome can be justified from evidence about that individual.

  Caught by

  Human-in-the-loop Gate L4

  Evidence

  A review queue and decision log for outlier cases (large adjustments, small cohorts), plus a contestation channel with its resolution records.

  MIT taxonomy

  • 1.1 Unfair discrimination and misrepresentation (1 Discrimination & toxicity)

  Incidents

  • AIAAIC Ofqal algorithm skews student grade predictions

  Cases

  • 2020 A-level grades

Risk to the organisation that builds, buys or deploys the system: operational, legal, financial, reputational, security and supply chain.

• Operational disruption by an agent acting beyond its mandate

  • Misalignment

  An agent with write access to production can take destructive actions its operators never intended. The records include a coding agent reported to have deleted production data during a code freeze, and a drive-through ordering pilot ended after ordering errors ³.

  Failure mode

  The agent holds standing credentials broader than its task, and destructive actions need no approval.

  Caught by

  Agent Identity & Scoped Credentials L4

  Evidence

  An agent registry entry with owner, scope and expiry, and the authorisation log showing each destructive call denied or approved.

  MIT taxonomy

  • 7.3 Lack of capability or robustness (7 AI system safety, failures & limitations)

  Incidents

  • AIID 1152 LLM-Driven Replit Agent Reportedly Executed Unauthorized Destructive Commands During Code Freeze, Leading to Loss of Production Data
  • AIID 475 McDonald's Reportedly Ends IBM Partnership After AI Drive-Thru Ordering Errors at U.S. Locations

  Cases

  • OpenAI agents and Hugging Face
  • DNS channel in a training run
  • GitHub token exposed by an agent
  • Public file hosting in training
  • Package repository message board
  • Claude cyber evaluation incidents
  • AISI cyber range incident

• Legal liability for what the system tells customers

  • Robustness

  An organisation answers for its chatbot's statements as it answers for the rest of its website. The record here is a tribunal ordering an airline to compensate a customer its chatbot misinformed about a fare policy ³.

  Failure mode

  Answers about policy, price or eligibility are generated without a check against the authoritative source.

  Caught by

  Runtime Guardrail L4

  Evidence

  Guardrail logs showing each policy answer grounded in, and cited to, the current policy document, and a refusal wherever no source matched.

  MIT taxonomy

  • 3.1 False or misleading information (3 Misinformation)

  Incidents

  • AIID 639 Air Canada Chatbot Reportedly Provides Inaccurate Bereavement Fare Information, Leading to Customer Overpayment

  Cases

  • Moffatt v. Air Canada
  • NYC MyCity chatbot

• Regulatory enforcement and forced suspension

  • Opacity
  • Scale

  Launching a system trained on personal data without a documented lawful basis invites orders to stop processing, and fines. Data protection authorities have temporarily limited a chatbot service in Italy and fined a face-search provider EUR 30.5 million in the Netherlands ³⁹.

  Failure mode

  The lawful basis, the transparency notice and the age check are treated as launch paperwork rather than as release preconditions.

  Caught by

  Policy Card L1

  Evidence

  A policy card that makes a recorded lawful basis per data source a release precondition, and the CI check that enforced it.

  MIT taxonomy

  • 2.1 Compromise of privacy by obtaining, leaking, or correctly inferring sensitive information (2 Privacy & security)

  Incidents

  • AIID 513 ChatGPT Reportedly Banned by Italian Authority Due to OpenAI's Purported Lack of Legal Basis for Data Collection and Age Verification
  • AIID 781 Clearview AI Reportedly Faces $33.7 Million Fine for Violating GDPR with Biometric Data Harvesting

  Cases

  • Dutch childcare benefits
  • Clearview AI
  • Garante ChatGPT order

• Financial loss from a model under distribution shift

  • Drift

  A model that sets prices or commits capital keeps acting on yesterday's market. Zillow wound down its home-buying business in 2021 with an inventory write-down of about USD 304 million, its chief executive saying home-price forecasting was far less predictable than anticipated ¹⁴.

  Failure mode

  Forecast error is not monitored against realised outcomes, and nothing throttles the model's commitments when the error grows.

  Caught by

  Continuous Assurance Telemetry L5

  Evidence

  Drift and realised-error telemetry against set thresholds, with the alerts and throttle decisions they triggered.

  MIT taxonomy

  • 7.3 Lack of capability or robustness (7 AI system safety, failures & limitations)

  Incidents

  • AIID 149 Zillow Shut Down Zillow Offers Division Allegedly Due to Predictive Pricing Tool's Insufficient Accuracy

  Cases

  • Zillow Offers

• Fraud losses from synthetic impersonation

  • Misuse

  Voice and video synthesis let fraudsters impersonate executives well enough to get payments released. The record here is a reported USD 25 million loss after a video call with a deepfaked finance chief ³.

  Failure mode

  A high-value action is authorised on the strength of a voice or video channel that can be synthesised.

  Caught by

  Out-of-band verification for high-value actions (not yet a catalogued pattern) L4

  Evidence

  A payment-release log showing an independent call-back verification for every transfer above the threshold.

  MIT taxonomy

  • 4.3 Fraud, scams, and targeted manipulation (4 Malicious actors & misuse)

  Incidents

  • AIID 634 Alleged Deepfake CFO Scam Reportedly Costs Multinational Engineering Firm Arup $25 Million

• Reputational damage from harmful or unlawful public outputs

  • Robustness
  • Misuse

  Public-facing systems that produce abusive content or unlawful advice damage trust in whoever deployed them. The records include a social chatbot reported to post racist content and a city chatbot that advised businesses to break the law ³⁵.

  Failure mode

  The system goes public without adversarial testing against the abuse and the domain questions it will actually receive.

  Caught by

  Adversarial Red-Team Suite L3

  Evidence

  Versioned red-team findings with severity and closure status, and the release decision that referenced them.

  MIT taxonomy

  • 1.2 Exposure to toxic content (1 Discrimination & toxicity)
  • 3.1 False or misleading information (3 Misinformation)

  Incidents

  • AIID 6 Microsoft's TayBot Allegedly Posts Racist, Sexist, and Anti-Semitic Content to Twitter
  • AIID 714 Microsoft-Powered New York City Chatbot Advises Illegal Practices
  • OECD-AIM NYC MyCity Chatbot Gives Dangerous, Illegal Advice to Businesses

  Cases

  • Moffatt v. Air Canada
  • NYC MyCity chatbot

• Leakage of confidential data into external AI services

  • Security

  Staff paste source code, meeting notes or customer data into external AI tools whose retention and training terms nobody reviewed. One manufacturer reportedly recorded three such leaks within a month of allowing use ³.

  Failure mode

  AI use is allowed or tolerated without an inventory, a gateway or data-loss checks on prompts.

  Caught by

  Shadow-AI Discovery L2

  Evidence

  A discovered-AI inventory reconciled against the registry, and gateway logs with a data-loss verdict per prompt.

  MIT taxonomy

  • 2.1 Compromise of privacy by obtaining, leaking, or correctly inferring sensitive information (2 Privacy & security)

  Incidents

  • AIID 768 ChatGPT Reportedly Implicated in Samsung Data Leak of Source Code and Meeting Notes

  Cases

  • Chatbot data leak (reported)

• Security compromise through prompt injection

  • Security

  Instructions hidden in user input or retrieved content can override a model's instructions, reveal its system prompt or make it act for an attacker. The record here is early testers extracting a search chatbot's initial prompt ³.

  Failure mode

  Untrusted content reaches the model's context with the same authority as the operator's instructions.

  Caught by

  Runtime Guardrail L4

  Evidence

  Input and output guardrail verdicts with the injection attempts they blocked, and red-team results for the known injection classes.

  MIT taxonomy

  • 2.2 AI system security vulnerabilities and attacks (2 Privacy & security)

  Incidents

  • AIID 473 Bing Chat's Initial Prompts Revealed by Early Testers Through Prompt Injection

• Supply-chain compromise through AI-suggested components

  • Security

  Code assistants can suggest packages that do not exist, and attackers can register those names. NIST lists value chain and component integration among the generative-AI risks ¹²; the record here is a hallucinated package name that was published and downloaded thousands of times ³.

  Failure mode

  Dependencies suggested by a model are installed without a provenance check against an approved registry.

  Caught by

  AIBOM L2

  Evidence

  An AIBOM and dependency manifest generated at build, with the provenance check that rejected unknown packages.

  MIT taxonomy

  • 2.2 AI system security vulnerabilities and attacks (2 Privacy & security)

  Incidents

  • AIID 731 Purportedly Hallucinated Software Packages with Potential Malware Reportedly Downloaded Thousands of Times by Developers

  Cases

  • Claude cyber evaluation incidents

Harm to shared systems: information, elections, work, equality and essential services.

• Pollution of the information ecosystem

  • Scale
  • Misuse

  Cheap synthetic text, images and audio make it harder to tell what is real, which erodes trust in authentic evidence as well as in fakes. NIST lists information integrity among its twelve generative-AI risks ¹².

  Failure mode

  Generated media leave the system with no provenance signal that a platform or a reader could check.

  Caught by

  Content provenance and labelling (not yet a catalogued pattern) L2

  Evidence

  Provenance metadata and labels attached at generation, with the share of outputs that carried them.

  MIT taxonomy

  • 3.2 Pollution of information ecosystem and loss of consensus reality (3 Misinformation)
  • 4.1 Disinformation, surveillance, and influence at scale (4 Malicious actors & misuse)

  Incidents

  • AIID 674 Manipulated Media via AI Disinformation and Deepfakes in 2024 Elections Erode Trust Across More Than 50 Countries

• Election interference with synthetic voices and media

  • Misuse

  Synthetic audio of a candidate does most damage just before a vote, when there is no time to rebut it. The records include a fake presidential voice in robocalls before a US primary and deepfake recordings spread before Slovakia's 2023 election ³.

  Failure mode

  Voice cloning is available without a consent check on whose voice is cloned, and the output carries no durable mark.

  Caught by

  Content provenance and labelling (not yet a catalogued pattern) L2

  Evidence

  Consent records for every cloned voice, and watermark or provenance checks on generated audio.

  MIT taxonomy

  • 4.1 Disinformation, surveillance, and influence at scale (4 Malicious actors & misuse)

  Incidents

  • AIID 628 Fake Biden Voice in Robocall Misleads New Hampshire Democratic Voters in 2024 Primary Election
  • AIID 573 Deepfake Recordings Allegedly Influence Slovakian Election

• Job displacement and task transformation

  • Scale

  Generative AI changes what many jobs consist of. The ILO estimates that one in four workers globally are in an occupation with some generative-AI exposure, and finds transformation of jobs more likely than outright replacement ¹⁰. This harm is structural, so no single incident record stands for it.

  Failure mode

  Deployment decisions count the productivity gain but not who loses tasks, income or bargaining power, and affected workers are not consulted.

  Caught by

  Workforce impact assessment (not yet a catalogued pattern) L1

  Evidence

  An impact assessment listing the roles and tasks affected, the consultation held and the transition measures agreed.

  MIT taxonomy

  • 6.2 Increased inequality and decline in employment quality (6 Socioeconomic & environmental harms)

  Incidents

  No single incident record: the evidence is the study cited above.

• Inequality in who bears the cost: hidden data labour

  • Scale

  The gains from AI flow to its builders and users while part of the cost falls on low-paid workers who label data, sometimes exposed to graphic material. The record here alleges that annotators in Kenya were exposed to graphic content while labelling for a model developer ³.

  Failure mode

  Data-supply contracts are bought on price with no labour or welfare standard, and nobody audits them.

  Caught by

  Vendor / Model Due-Diligence Gate L2

  Evidence

  A supplier assessment covering annotator pay, exposure limits and support, renewed with the contract.

  MIT taxonomy

  • 6.2 Increased inequality and decline in employment quality (6 Socioeconomic & environmental harms)
  • 6.1 Power centralization and unfair distribution of benefits (6 Socioeconomic & environmental harms)

  Incidents

  • AIID 450 Kenyan Data Annotators Allegedly Exposed to Graphic Content for OpenAI's AI

• Over-reliance on generated output in institutions

  • Over-reliance

  Professionals under time pressure accept fluent output without checking it. The records include fabricated case law filed in a US court and errors introduced into a child-protection report ³.

  Failure mode

  Generated content enters a consequential record with no verification step and no disclosure that AI was used.

  Caught by

  Human-in-the-loop Gate L4

  Evidence

  A sign-off record showing who verified each generated passage against its source before filing.

  MIT taxonomy

  • 5.1 Overreliance and unsafe use (5 Human-computer interaction)
  • 3.1 False or misleading information (3 Misinformation)

  Incidents

  • AIID 541 ChatGPT Reportedly Produced False Court Case Law Presented by Legal Counsel in Court
  • AIID 807 ChatGPT Reportedly Introduces Errors in Critical Child Protection Court Report

• Disruption to critical infrastructure and emergency services

  • Robustness
  • Scale

  When AI systems run transport, utilities or emergency response, one failure repeats across a whole fleet at once. The records include driverless cars that stalled in traffic after losing their server connection, and one that blocked a fire truck on an emergency call ³.

  Failure mode

  A shared dependency (connectivity, a model update) fails for every unit at once, and there is no degraded mode that clears the way.

  Caught by

  Kill Switch / Circuit Breaker L4

  Evidence

  Fleet-level stop and safe-state drills, with records of when the breaker fired and how long recovery took.

  MIT taxonomy

  • 7.3 Lack of capability or robustness (7 AI system safety, failures & limitations)

  Incidents

  • AIID 253 Cruise's Self-Driving Cars Allegedly Lost Connection to Their Server, Causing Traffic Blockages in San Francisco
  • AIID 389 Cruise Autonomous Car Blocked Fire Truck Responding to Emergency

Harm to the physical environment from building and running AI systems.

• Energy demand and emissions of training and inference

  • Scale

  Training and serving models draws electricity, and so emissions, at data-centre scale. The IEA projects that data-centre electricity demand will more than double by 2030 to around 945 TWh, slightly more than Japan's consumption today ¹¹. NIST lists environmental impacts among the generative-AI risks ¹².

  Failure mode

  Model and deployment choices are made without measuring energy use, so the cheapest adequate option is never compared.

  Caught by

  Energy and emissions telemetry (not yet a catalogued pattern) L5

  Evidence

  Per-model energy and emissions figures, recorded for training and per unit of inference, published in the model card.

  MIT taxonomy

  • 6.6 Environmental harm (6 Socioeconomic & environmental harms)

  Incidents

  No single incident record: the evidence is the study cited above.

• Local pollution and resource burden near AI infrastructure

  • Scale

  The power generation and cooling that AI data centres need can burden the communities around them. The record here alleges that a supercomputer site in Memphis ran unpermitted gas turbines ³.

  Failure mode

  Capacity is added faster than permitting and community consultation, and buyers of compute never ask where or how it is powered.

  Caught by

  Vendor / Model Due-Diligence Gate L2

  Evidence

  The provider's answers on energy source, permits and location, kept with the procurement decision.

  MIT taxonomy

  • 6.6 Environmental harm (6 Socioeconomic & environmental harms)

  Incidents

  • AIID 1144 xAI Allegedly Operates Unpermitted Methane Turbines in Memphis to Power Supercomputer Colossus to Train Grok