Fairness Eval Suite

Fairness Eval Suite A workflow diagram generated by Archify. 01 / Layer 01 Govern-as-Code 02 / Layer 03 Evals & Red Teaming as Evidence 03 / Evidence and monitoring EX / Build fails Policy Run the suite Gate + file Fairness policy · metric, floor, min cell · Layer 01 Govern-as-Code › Policy Fairness policy metric, floor, min cell Group metrics · incl. intersections · Layer 03 Evals & Red Teaming as Evidence › Run the suite · with intervals Group metrics incl. intersections with intervals Proxy scan · features predict group? · Layer 03 Evals & Red Teaming as Evidence › Run the suite Proxy scan features predict group? Counterfactual flip · change only the group · Layer 03 Evals & Red Teaming as Evidence › Run the suite Counterfactual flip change only the group Gate on the interval · small cells listed · Layer 03 Evals & Red Teaming as Evidence › Gate + file · pass / fail Gate on the interval small cells listed pass / fail Live monitoring · same metrics, rolling · Evidence and monitoring › Gate + file Live monitoring same metrics, rolling Eval results · per metric and slice · Evidence and monitoring › Gate + file Eval results per metric and slice Release blocked · unless justified · Build fails › Gate + file Release blocked unless justified scores below floor after release file then configure then Legend Agent logic Policy Context / trace External system

Choose, then measure

  • • Common fairness criteria conflict, so the metric is a decision with an owner
  • • The policy names the metric, the threshold, the minimum cell and the approver
  • • Commit it before looking at the next run

Intervals, not points

  • • Every rate and ratio carries a confidence interval
  • • Cells below the minimum are reported as insufficient data, never as passes
  • • Search for the worst slice, not only the listed groups

Release and after

  • • Results feed the model card's disaggregated metrics
  • • The same metrics run on live decisions so drift raises an alert