AI Governance Engineer
GitHub

Counterfactual flip test

By Updated

A test that changes only a protected attribute in an input, or swaps identity terms in otherwise identical prompts, and measures how often the outcome or the answer quality changes. It is the practical approximation of counterfactual fairness 1 and runs in the fairness eval suite.

Developed in
ch. 16, Individual and counterfactual fairness
ch. 05, Pattern: Fairness Eval Suite
Chapters
ch. 05, Patterns · ch. 16, Fairness & XAI
Contrast with
Counterfactual fairness
Source
1 numbered reference, listed below

Where it is used

2 chapters of the Body of Knowledge use the term. Each link opens the first section that does.

Patterns that use this term

One pattern page uses the term.

  • Fairness policy

    The per-system record, fixed before results are seen, of what fairness means for that system: the protected attributes in each jurisdiction and where their…

  • Proxy scan

    A test that trains a model to predict a protected attribute from a system's features; features that predict it strongly are flagged as proxies to justify or…

  • Fairness

    The property that a system's outcomes and errors do not unjustifiably disadvantage people or groups.

Sources

  1. [1] "Counterfactual Fairness" (Kusner, Loftus, Russell and Silva; arXiv 1703.06856). arXiv. 2017-03-20. https://arxiv.org/abs/1703.06856 (verified: primary)

Definitions of legal terms paraphrase the cited text, which governs. Dated statements are as of .

Cite this term

García Aibar, J. (2026). Counterfactual flip test. In AI Governance Engineering: The Thesis & Body of Knowledge (v0.5.0), Glossary. https://doi.org/10.5281/zenodo.22956197. https://aigovernanceengineer.com/glossary/counterfactual-flip-test. CC BY 4.0

BibTeX

@misc{aige2026counterfactualfliptest,
  author  = {Jorge García Aibar},
  title   = {{Counterfactual flip test}},
  note    = {Glossary, AI Governance Engineering: The Thesis \& Body of Knowledge, version 0.5.0},
  year    = {2026},
  doi     = {10.5281/zenodo.22956197},
  url     = {https://aigovernanceengineer.com/glossary/counterfactual-flip-test}
}