Harness and Configuration Attestation

The control record

Id

AIGE-CTL-EVAL-008 · v0.2 · Draft · Open for technical review

Objective

The harness, prompts, tool definitions and configuration a run used are versioned and hashed, so the result can be tied to exactly what was evaluated.

Failure modes

• A result is reported without the hashes of the prompts, tool definitions and harness it ran on.
• A tool definition changes between admission and the run without an alert.
• The configuration in the report differs from the one recorded for the run.
• Two results are compared although they ran on different scaffold prompts or task wordings, which can change the behaviour being measured.

Scope

The harness, system and scaffold prompts, task instructions, tool and MCP server definitions, policy bundles, scoring configuration and model artefacts a run loads. The design of the evaluation tasks is out of scope.

Enforcement points

• pre_merge: on every pull request
• deploy: before a version is deployed or released

Verification

• Inspect: Before the run, inspect the run manifest: it lists, each with a version and a hash, the harness, the system and scaffold prompts, the task instructions, the tool and MCP server definitions, the policy bundles, the scoring configuration, and the model identifier with its settings.
• Test: At admission, recompute the hash of every artefact the environment actually loaded and compare it with the manifest: every hash must match. On a copy of the environment, change one tool definition: the run must be blocked with an alert.
• Inspect: Before a result is released, compare the configuration stated in the report (model, reasoning setting, tool access, harness, safeguards and budget) with the manifests of the runs behind it: they must agree, and results compared with each other must share scaffold prompts and task wording or state the difference.

Evidence

• Run manifest: version and hash of every artefact the run loaded, recorded before the run outside the environment · Layer 03
• Admission check: the recomputed hashes against the manifest, with its verdict · Layer 03 · evidence-record.v1
• Test report stating the tested system, budget and environment of its results, with links to the run manifests · Layer 03 · test-report.v1
• Manifest check of each run, filed as an observation of this control · Layer 05 · control-observation.v1

Failure response

deny: block the action. A run whose loaded artefacts do not match its manifest is not started, and a change detected during a run stops it. A result whose report does not match the manifests of its runs is not released until the difference is explained or the runs are repeated.

Layers

Layer 03 Evals & Red Teaming as Evidence, Layer 02 Inventory & Transparency

Patterns

Model Artefact Integrity, AIBOM, Eval Gate in CI

Seeded from

Prompts under change control, MCP server admission gate

Mappings

• Obligations: EU AI Act Art. 15 accuracy, robustness and cybersecurity; OWASP AIBOM; ISO/IEC 42001, A.6 AI system life cycle
• ISO/IEC 42001: A.6.2.4 AI system verification and validation
• NIST AI RMF: MEASURE 2.1 Test sets, metrics, and details about the tools used during TEVV are documented.
• OWASP: ASI04 Agentic Supply Chain Vulnerabilities; LLM04:2026 Supply Chain
• MITRE ATLAS: AML.T0110 AI Agent Tool Poisoning; AML.T0010 AI Supply Chain Compromise
• MITRE ATLAS mitigation: AML.M0014 (Verify AI Artifacts)
• MITRE ATLAS mitigation: AML.M0023 (AI Bill of Materials)
• NIST SP 800-218A: PS.1.3 (Protect model weights and configuration parameters)
• NIST SP 800-218A: PS.3.2 (Keep provenance data for every component of a release)
• NIST SP 800-53 Rev. 5: CM-2 (Baseline Configuration)
• NIST SP 800-53 Rev. 5: CM-3 (Configuration Change Control)
• NIST SP 800-53 Rev. 5: CM-6 (Configuration Settings)
• IMDA MGF for Agentic AI v1.5: 2.3 Implement technical controls and processes (introduction), p.33 (Partial fit: Version control across the lifecycle; hashing harness and configuration to tie a result to what was run is not stated)

References

• [1] Governing AI agents (AI Governance Engineering Body of Knowledge v0.5.0, chapter 23, section "Prompts as configuration under change control")
• [2] Governing AI agents (AI Governance Engineering Body of Knowledge v0.5.0, chapter 23, section "Admitting an MCP server")
• [3] Governing AI development (AI Governance Engineering Body of Knowledge v0.5.0, chapter 14, section "Reproducibility and linked versioning")
• [4] Summary of METR's predeployment evaluation of GPT-5.6 Sol (METR states that "observed cheating rates can also be influenced by the prompts used in the evaluation scaffold" and by task wording)
• [5] NIST SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile (AI-specific tasks added to SSDF 1.1 (e.g. PO.5.3, PS.1.3, PW.3.1 to PW.3.3) and AI-specific recommendations on existing tasks (e.g. PW.1.1, RV.1.1))
• [6] MITRE ATLAS data, release v2026.09 (16 tactics, 120 techniques, 88 sub-techniques, 40 mitigations; technique names and technique-to-mitigation links read from dist/v6/ATLAS-2026.09.yaml)
• [7] NIST SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations (control catalogue cited by control id; publication page of Revision 5 with update 1 of 10 Dec 2020)
• [8] Improving our alignment and security efforts (best practices for external partners running cyber evaluations: the only outside connection is "to the model's own API", with the API keys kept outside the environment; the configuration "should be verified before every evaluation begins"; boundaries "phrased as instructions"; challenges confirmed "solvable in principle"; a monitor that flags a scope violation to a human and ends the exercise)
• [9] A shared playbook for trustworthy third party evaluations (recommended report fields include the claim, the tested system (model, reasoning setting, tool access, harness and safeguards), the budget, elicitation methods and validity checks; a score is "performance under that harness and budget")
• [10] Investigating the consequences of accidentally grading CoT during RL (chain-of-thought text reached the inputs of reward mechanisms by accident; an automated system now scans all RL runs for it with regex matches)
• [11] Model AI Governance Framework for Agentic AI, v1.5 (published 2026-05-20, updated 2026-06-05; four dimensions: assess and bound the risks upfront, make humans meaningfully accountable, implement technical controls and processes, enable end-user responsibility)

Implementation notes

• Hash what the run loads, not what the repository holds: build the manifest at admission from the artefacts inside the environment (harness image digest, prompts, task instructions, tool and MCP server definitions, policy bundles, scoring configuration, model identifier and settings), store it outside the environment and put its digest in the run record. Chapter 23 treats prompts, tool descriptions and policy bundles as configuration under change control, with the hash recorded in the registry and in every trace.
• Check the configuration before each run, not once per environment: Anthropic's guidance for external evaluation partners states that the isolation configuration "should be verified before every evaluation begins".
• Report the configuration with the result. OpenAI's playbook for third-party evaluations asks reports to state the tested system (model, reasoning setting, tool access, harness and safeguards) and the budget, and to describe a score as "performance under that harness and budget, not as a measured capability ceiling".
• Put the scoring path and the scaffold prompts in the manifest too. OpenAI's alignment blog describes chain-of-thought text reaching the inputs of reward mechanisms by accident during RL, now caught by an automated scan whose coverage OpenAI says is not perfect; METR states that observed cheating rates "can also be influenced by the prompts used in the evaluation scaffold" and by the wording of task instructions.

Open questions

• How should evaluation-harness configuration be attested so that a third party can verify it without access to the harness itself?
• When a third party runs the evaluation, who signs the manifest: the evaluator, the developer of the model or both?

Observation

• Subject: Harness
• Expected: Every artefact the run loaded matches the version and hash in its manifest, and the report states the same configuration as the manifests of its runs.
• Example: Run 88270: 41 artefacts hashed at admission, all matching the manifest; the report states the same model, tools, harness and budget: pass.