A recruiting model that learned the past (reported)

What happened

Reuters reported on 10 Oct 2018 that Amazon had built an experimental tool, from 2014, to score job applicants; that it was trained on about ten years of CVs drawn largely from men; that it learned to penalise CVs containing the word "women's" and graduates of certain all-women colleges; and that the project was abandoned ¹². These are reported facts: no primary company document is public.

The company is reported to have said that recruiters never relied solely on the tool's rankings ².

Failure mode

Historical label bias. The model learned what past hiring looked like, past hiring skewed male, and the skew became a scoring rule. Removing explicit gendered terms does not remove proxies for them, which is why reports say the fixes did not guarantee fairness ².

Which control would have caught it

An eval gate that computes selection rates by sex (and other protected characteristics) on a held-out set, and fails the build when an impact ratio falls below a set threshold, catches this in the first release candidate rather than after years of development. A data card that states the composition of the training population makes the risk visible before training starts. The fundamental-rights impact assessment is where the decision to automate screening at all is argued.

Patterns: Eval Gate in CI · Model Card as Control Evidence · FRIA-as-Code

The evidence that would have existed

What an auditor could have read, and the stack layer that produces it.

Obligations it touches today

As of 2026-09-24. Mappings are illustrative, not a claim of conformity.