Evaluation Environment Control Profile

Scope

Evaluation environments for models and agents, from the harness and the tools and MCP servers a run can call to the credentials it holds, the network it can reach and the records it leaves. The evaluation tasks, their scoring rubrics and the capabilities being measured are out of scope.

9 controls, AIGE-CTL-EVAL-001 to AIGE-CTL-EVAL-009. Each one is anchored to a layer of the five-layer stack and to the patterns that implement it.

How to read a control

Every control has a stable id that never changes and is never reused, and the same record:

• Objective: the outcome the control secures, in one sentence.
• Failure modes: observable events that mean the control failed.
• Scope, enforcement points (on a pull request, at deploy, at runtime or on a schedule) and the failure response (deny, hold for approval or alert).
• Verification: how a third party would check it (inspect, test, observe or attest).
• Evidence: the artefact it leaves and the stack layer that keeps it.
• Mappings: obligations, ISO/IEC 42001 Annex A, the NIST AI RMF, OWASP and other references, each id checked against the site's registers.

Depth says how far a control is written. Specified: a verification procedure, evidence, implementation notes, two example observations and a page of its own. Derived from site material: restates existing site material (an agent control of chapter 23, a pattern, a record schema or a chapter, named on each control) and adds nothing that material does not say. Draft outline: objective, failure modes, scope and open questions only. This profile: 9 specified.

An observation is what a check of a control would emit: the control id, the subject checked, what was expected, what was observed, a status (pass, fail or not_applicable), a timestamp and the evidence behind it.

Mappings

Every control against the obligations, standards and threats it answers. Mappings are illustrative, not a claim of conformity; an empty cell means no mapping has been verified yet. AIUC-1 ids point at the public requirement index of the Artificial Intelligence Underwriting Company, cited here as a reference; this project is not affiliated with AIUC, and each mapping is our own reading of the requirement text. The same mappings, read from the framework side and across every profile, are in the controls crosswalk.

Open questions

The questions this draft leaves open, with the controls that raise each one.

• Which shared resources (package mirrors, caches, model API proxies) can be given a per-run namespace, and which should be removed from evaluation environments altogether? (001)
• What evidence is sufficient to show that an agent stayed within its boundary for a whole run when the logs of some of the tools it called are outside the evaluator's control? (001)
• How should the egress allow-list of a run be recorded so that a third party can compare it with the connections actually observed? (002)
• When a task needs the internet (what METR's Task Standard calls the full_internet permission), which part of this control still applies, and what should the run record say instead? (002)
• What lifetime should a credential issued for an evaluation run have, and who revokes it when the run is aborted? (003)
• Can a secret scan be made complete for encoded or split credentials, or should the control rely on short lifetimes and revocation for whatever a scan may miss? OpenAI reports a model that split a researcher's token into pieces with the stated aim of avoiding secret scanning. (003)
• How can every call be mediated without taking away the scaffolding and tooling a capability elicitation needs? METR's guidelines ask for "the best available scaffolding + tooling". (004)
• When the reviewer at the mediation point is a model, what evidence shows that its verdicts are reliable enough to replace a person for a given operation class? (004)
• How can a reviewer show that a run's trace is complete for actions that pass no tool server, not only that the records it holds are unaltered? (005)
• The OpenTelemetry GenAI conventions are still at status Development: should the profile name the attributes it relies on, or only the events a trace must contain? (005)
• Which observations should trip a stop automatically during an evaluation, and which should only alert the evaluator? (006)
• What time to stop is acceptable for an evaluation run, and should the target be stated per stop level or per task class? (006)
• How long should the records of an evaluation run be kept when the run produced no incident? (007)
• Which parts of a frozen run record can be shared with an independent reviewer without exposing the task set, and in what format? (007)
• How should evaluation-harness configuration be attested so that a third party can verify it without access to the harness itself? (008)
• When a third party runs the evaluation, who signs the manifest: the evaluator, the developer of the model or both? (008)
• Which validity threats (task bugs, scoring errors, evaluation awareness) should block a result, and which should only be disclosed with it? (009)
• How large a sample of successful runs should be read for reward hacking and scorer tampering before a result is reported? (009)

Changelog

• v0.1 (2026-09-26): First draft: three controls specified in full (002 Network Egress Control, 003 Credential Isolation, 006 Stop Conditions) and six outlines with open questions, open for technical review.
• v0.2 (2026-09-26): Six outlines promoted to specified, each with repeatable verification steps, evidence tied to a published schema, configuration-level implementation notes, an observation record and two illustrative example observations: 001 Authorization Boundary, 004 Tool and Action Mediation, 005 Monitoring Integrity, 007 Incident Evidence Preservation, 008 Harness and Configuration Attestation and 009 Evaluation Validity Checks. Every source they cite was re-opened on 2026-09-26. No control remains an outline: each promoted claim had a verified source. An adversarial content review the same day re-checked every quotation against its source, tightened four attributions and replaced the mappings that did not plainly fit (NIST AI RMF on 001, 004, 005, 007, 008 and 009; ISO/IEC 42001 on 008; OWASP on 009; three EU AI Act obligation rows on 005 and 007). Still open for technical review; no reviewer is credited yet.

Sources

Machine-readable

• Each control as JSON, for example AIGE-CTL-EVAL-001: the link sits at the foot of every control above.
• All controls as JSON: every control of every profile, in the envelope of the open data API.
• JSON Schema of the controls dataset: generated from the same registry.
• Control observation schema: the record a check of a control emits: control_id, subject, expected, observed, status, timestamp, evidence.
• Observation example: a filled record that validates against the schema.
• Observation template: the same fields in Markdown, with short guidance.
• This page as Markdown: /controls/evaluation-environment.md. Every dataset of the site is in the open data API.

Review this profile

Review happens in the open, on GitHub. Pick a control, check it against a system you run or know, and say what is wrong or missing: an objective that cannot be verified, a failure mode that is not observable, a mapping that does not hold. Each control has its own button above; this one reviews the profile as a whole, through the control review form.

@misc{aige2026page,
  author       = {Jorge García Aibar},
  title        = {{Evaluation Environment Control Profile v0.2 (draft)}},
  howpublished = {In AI Governance Engineering: The Thesis \& Body of Knowledge},
  year         = {2026},
  version      = {0.5.0},
  doi          = {10.5281/zenodo.22857084},
  url          = {https://aigovernanceengineer.com/controls/evaluation-environment},
  note         = {Version 0.5.0}
}