---
title: "Counterfactual flip test"
description: "A test that changes only a protected attribute in an input, or swaps identity terms in otherwise identical prompts, and measures how often the outcome or the answer quality changes."
canonical: https://aigovernanceengineer.com/glossary/counterfactual-flip-test
author: "Jorge García Aibar"
license: "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)"
doi: https://doi.org/10.5281/zenodo.22956197
version: "0.5.0"
updated: 2026-09-25
---

# Counterfactual flip test

A test that changes only a protected attribute in an input, or swaps identity terms in otherwise identical prompts, and measures how often the outcome or the answer quality changes. It is the practical approximation of counterfactual fairness [1] and runs in the fairness eval suite.

- Developed in: [ch. 16, Individual and counterfactual fairness](https://aigovernanceengineer.com/bok/fairness-and-explainability#individual-and-counterfactual-fairness); [ch. 05, Pattern: Fairness Eval Suite](https://aigovernanceengineer.com/patterns/fairness-eval-suite)
- Chapters: [ch. 05, Patterns](https://aigovernanceengineer.com/bok/patterns) · [ch. 16, Fairness & XAI](https://aigovernanceengineer.com/bok/fairness-and-explainability)
- Contrast with: [Counterfactual fairness](https://aigovernanceengineer.com/glossary/counterfactual-fairness)
- In the glossary chapter: https://aigovernanceengineer.com/bok/glossary#t-counterfactual-flip-test

## Sources

[1] "Counterfactual Fairness" (Kusner, Loftus, Russell and Silva; arXiv 1703.06856). arXiv. 2017-03-20. https://arxiv.org/abs/1703.06856 (verified: primary)