Eval Gate in CI

Eval Gate in CI A workflow diagram generated by Archify. 01 / Development 02 / CI / CD Pipeline 03 / Eval Suite (Layer 03) 04 / Registry & Evidence EX / Blocked Release Change Run evals Gate + record Eval run in CI Below threshold Model / Prompt Change · retrained · new tool · Development › Change Model / Prompt Change retrained · new tool CI Pipeline · already runs tests · CI / CD Pipeline › Change CI Pipeline already runs tests Ship Release · deploy to prod · CI / CD Pipeline › Gate + record Ship Release deploy to prod Versioned Eval Suite · versioned with model · Eval Suite (Layer 03) › Eval run in CI › Run evals Versioned Eval Suite versioned with model Capability Eval · coverage floor · Eval Suite (Layer 03) › Eval run in CI › Run evals · e.g. promptfoo Capability Eval coverage floor e.g. promptfoo Adversarial Eval · injection-resistance · Eval Suite (Layer 03) › Eval run in CI › Gate + record · e.g. Garak Adversarial Eval injection-resistance e.g. Garak Threshold Gate · traces to failure mode · Eval Suite (Layer 03) › Gate + record · pass / fail Threshold Gate traces to failure mode pass / fail Structured Result · score · verdict · time · Registry & Evidence › Gate + record Structured Result score · verdict · time Registry Entry · evidence accrues · Registry & Evidence › Gate + record Registry Entry evidence accrues Pipeline Blocked · does not ship · Blocked Release › Below threshold › Gate + record Pipeline Blocked does not ship score score triggers CI file against entry load suite adversarial eval capability eval below threshold emit result above threshold Legend Agent logic Policy Context / trace External system

Give every control teeth

  • • A failing eval blocks the build instead of filing a finding
  • • The threshold traces to a named failure mode or obligation
  • • The gate is built at the earliest point it can block

Evidence accrues automatically

  • • Structured result: suite id, model version, score, threshold, verdict, timestamp
  • • Filed against the registry entry for the model
  • • Regressions are caught before production

Trade-offs

  • • Eval maintenance and run-time cost in CI
  • • Thresholds tuned to avoid flaky gates