Pattern: Eval Gate in CI

Layer 03 · Evals & Red Teaming as Evidence In the chapter 05 catalogue

One working control on the engineering side of AI governance.

Summary: Wire an evaluation suite into the CI/CD pipeline so that a model or agent must pass a defined test, above a documented threshold, before it can ship. The eval run is the control and its result is the evidence; a failing eval blocks the build.

Objectives

Make “give every control teeth” concrete: give a testable property a consequence, so failure stops a release instead of filing a finding.

Target users

AI governance engineer, ML engineer, platform team.

Impacted stakeholders

Model owners, users exposed to the system, auditors.

Relevant principles

Give every control teeth; build the control at the earliest point it can block.

Context

A model or agent that changes (retrained, re-prompted, given a new tool) and a pipeline that already runs tests for functional correctness.

Problem

Evaluations run once before launch and pasted into a slide prove nothing after the next change. A review board that can only rate findings cannot stop a scheduled launch. Without a gate, evaluation is research, not control.

Solution

Version an eval suite alongside the model. Run at least one capability eval and one adversarial eval in CI (for example with Inspect, promptfoo, Garak or Giskard; illustrative). Set a threshold that traces to a named failure mode or obligation. Fail the pipeline below the threshold. Emit a structured result (suite id, model version, score, threshold, pass/fail, timestamp) filed against the registry entry.

Illustrative schema for the result:

{
  "suite_id": "injection-resistance.v4",
  "model_version": "csa-01@2026-09-18",
  "score": 0.982,
  "threshold": 0.95,
  "result": "pass",
  "timestamp": "2026-09-18T14:22:03Z"
}

Singapore’s framework for agentic AI sets the same expectation before release: “Organisations should test agents for safety and security before deployment. This provides confidence that the agents work as expected and controls are effective” (IMDA, section 2.3.2, p. 38)³. It does not say that a failed evaluation blocks the release.

Consequences

Regressions are caught before production and evidence accrues automatically. The trade-off is eval maintenance, run-time cost in CI, and the need to tune thresholds to avoid flaky gates.

Related patterns

Policy Card; Adversarial Red-Team Suite; Continuous Assurance Telemetry; Machine-Readable Evidence (OSCAL); Model Card as Control Evidence.

Maps to: EU AI Act Art. 15, Art. 55 · ISO/IEC 42001 · NIST AI RMF (Measure) · OWASP Agentic ASI01/ASI02 · Layer 03 Evals & Red Teaming as Evidence.

Threat IDs follow the OWASP Top 10 for Agentic Applications 2026 ¹ and function labels the NIST AI RMF². Mappings are illustrative, not a claim of conformity.

Sources

1. [1] Top 10 for Agentic Applications 2026 (ASI IDs). OWASP GenAI Security Project. 2025-12-09. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ (verified: primary)
2. [2] AI Risk Management Framework (AI RMF 1.0; Govern, Map, Measure, Manage). NIST. 2023-01-26. https://www.nist.gov/itl/ai-risk-management-framework (verified: primary)
3. [3] Model AI Governance Framework for Agentic AI, v1.5 (section 2.3.2, p. 38: testing agents before deployment; published 2026-05-20, updated 2026-06-05). IMDA. 2026-06-05. https://www.imda.gov.sg/-/media/imda/files/about/emerging-tech-and-research/artificial-intelligence/mgf-for-agentic-ai.pdf (verified: primary)