Dutch childcare benefits: nationality as a risk indicator
The Dutch tax administration used applicants' nationality as a risk indicator for childcare benefits; the data protection authority fined it EUR 2.75 million.
One incident read against the controls of AI governance and its frameworks.
- Year
- 2021
- Jurisdiction
- Netherlands
- Sector
- Public sector: social benefits
- Evidence base
- Primary sources
- Incident record
- AIID 101
- Harm
- Unequal treatment of a group by public risk profiling · Discrimination in consequential decisions · Regulatory enforcement and forced suspension
What happened
The Dutch Tax and Customs Administration processed the nationality, and the dual nationality, of childcare-benefit applicants for years. It used Dutch or non-Dutch nationality as an indicator in a system that automatically designated certain applications as risky, and processed nationality to combat organised fraud although that data was not necessary for the purpose 1.
On 7 Dec 2021 the Dutch data protection authority (AP) fined the administration EUR 2.75 million, finding the processing unlawful and discriminatory, and therefore improper under the GDPR. It said the dual-nationality data should have been deleted in January 2014, and that nationality had not been used to determine risk since October 2018 1.
Amnesty International's analysis describes an algorithmic system that built risk profiles of applicants to detect inaccurate and potentially fraudulent applications early, with nationality among the risk factors 2. The AI Incident Database records the case as families wrongfully accused of tax fraud by a discriminatory algorithm 3.
Failure mode
A protected characteristic was a model input. Nothing between the data and the decision checked whether a feature was lawful to use for this purpose, so a risk flag could rest on nationality.
Retention failed as well: data that should have been deleted in 2014 was still within reach years later 1. A risk model can use every attribute it can reach, so deletion is a control on the model too.
Which control would have caught it
A feature policy enforced in the pipeline catches this before the first score: a policy card listing the inputs permitted for the purpose, and an eval gate that fails the build when a prohibited attribute, or a close proxy for one, enters the feature set, or when flag rates diverge across groups. The fundamental-rights impact assessment is where the purpose, the affected groups and the permitted features are decided and signed.
Patterns: Policy Card · Eval Gate in CI · FRIA-as-Code
The evidence that would have existed
What an auditor could have read, and the stack layer that produces it.
- L1 Policy card for the risk model listing the permitted input features, with nationality marked prohibited for this purpose
- L1 Signed FRIA naming the affected groups, the purpose limitation and the outcome metric to monitor
- L3 Eval-gate run log in which the feature-policy check and the disaggregated flag-rate test pass or block the release
- L5 Retention-job log showing the dual-nationality data deleted on schedule
Obligations it touches today
As of 2026-09-24. Mappings are illustrative, not a claim of conformity.
- GDPR Art. 5(1)(a), Art. 35 The AP's finding rests on lawfulness and fairness 1; a data protection impact assessment is the GDPR artefact that should have surfaced the nationality feature 4.
- EU AI Act Annex III, point 5(a) Systems used by or for public authorities to evaluate eligibility for essential public assistance benefits are high-risk 5; after the AI Omnibus, Annex III obligations apply from 2 Dec 2027 (as of 2026-09-24) 6.
- EU AI Act Art. 27 A public body deploying such a system carries out a fundamental-rights impact assessment before first use 7.
- EU AI Act Art. 5(1)(c) Social scoring that leads to unjustified or disproportionate detrimental treatment is prohibited 8. Whether a given risk model meets those conditions is a legal judgement, not an engineering one.
Sources
- [1] Tax Administration fined for discriminatory and unlawful data processing (EUR 2.75 million fine; nationality used as a risk indicator). Autoriteit Persoonsgegevens (Dutch Data Protection Authority). 2021-12-07. https://www.autoriteitpersoonsgegevens.nl/en/current/tax-administration-fined-for-discriminatory-and-unlawful-data-processing (verified: primary)
- [2] Xenophobic machines: discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal (EUR 35/4686/2021). Amnesty International. 2021-10-25. https://www.amnesty.org/en/documents/eur35/4686/2021/en/ (verified: primary)
- [3] AI Incident Database, Incident 101: Dutch Families Wrongfully Accused of Tax Fraud Due to Discriminatory Algorithm. Responsible AI Collaborative. 2026. https://incidentdatabase.ai/cite/101/ (verified: primary)
- [4] Regulation (EU) 2016/679 (General Data Protection Regulation) (Art. 5 principles, Art. 6 lawfulness, Art. 8 child's consent, Art. 9 special categories, Arts. 12-15 transparency and access, Art. 22 automated individual decision-making, Art. 33 breach notification, Art. 35 DPIA). Official Journal of the European Union (EUR-Lex). 2016-04-27. https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng (verified: primary)
- [5] EU AI Act Annex III (high-risk uses; point 3(b) evaluating learning outcomes, 4(a) recruitment and selection, 5(a) eligibility for essential public assistance benefits and services). Publications Office of the EU (EUR-Lex). 2026-07-27. https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng#anx_III (verified: primary)
- [6] AI Omnibus enters into force (Reg. (EU) 2026/1744, in force 2026-07-27; Annex III high-risk obligations move to 2 Dec 2027). European Commission. 2026-07-27. https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force (verified: primary)
- [7] EU AI Act Art. 27 (FRIA before first use by deployers that are bodies governed by public law or private entities providing public services, and by deployers of Annex III point 5(b) and (c) systems; Art. 27(4) cross-reference to a GDPR Art. 35 DPIA). Publications Office of the EU (EUR-Lex). 2026-07-27. https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng#art_27 (verified: primary)
- [8] EU AI Act Art. 5 (prohibited AI practices; 5(1)(c) social scoring leading to unjustified or disproportionate detrimental treatment; 5(1)(e) facial recognition databases built by untargeted scraping). Publications Office of the EU (EUR-Lex). 2026-07-27. https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng#art_5 (verified: primary)