Incident Evidence Preservation

The control record

Id

AIGE-CTL-EVAL-007 · v0.2 · Draft · Open for technical review

Objective

When a run produces an incident, its traces, configuration and outputs are frozen before anything is fixed, so the record can be reviewed as it was.

Failure modes

• Records of a run are changed or deleted after an incident was declared.
• The environment is reset before its state and traces were captured.
• Part of a run's transcript is lost when a container is reset, and the gap is not recorded.
• An incident record does not link to the run it came from or to the hashes of the frozen records.

Scope

Evaluation runs that produce an incident or a result disputed after the fact, and the records they leave. Reporting to authorities follows the incident process of chapter 17.

Enforcement points

• runtime: at the point of action (gateway or guardrail)
• periodic: on a schedule, over what is already running

Verification

• Inspect: Inspect the evidence store and the harness configuration: the transcripts, traces, configuration and outputs of every run are written as they are produced to write-once storage outside the environment, with a retention period recorded, and the harness snapshots the environment before any reset.
• Test: Drill the freeze on a schedule: declare a test incident on a live run, then check that the environment snapshot, trace, transcript and configuration were captured with their hashes before the environment was reset, and that an attempt to delete or overwrite them is refused.
• Observe: For each real incident, read the incident record: it names the run, lists every frozen artefact with its hash, the hashes still match the stored artefacts, and every gap in the transcript is recorded with its cause.

Evidence

• Incident record naming the run and listing the frozen artefacts in its supporting materials · Layer 05 · incident-record.v1
• Freeze record: hashes of the environment snapshot, trace, transcript and configuration, with the time of capture and the actor · Layer 05 · evidence-record.v1
• Freeze drill and hash check, filed as an observation of this control · Layer 05 · control-observation.v1

Failure response

alert: let the action through and raise an alert. A missing snapshot, a hash mismatch or an unrecorded gap alerts the incident owner and is entered in the incident record. Until the freeze is complete the environment is not reset or reused, and the fix is made on a new version, not in place.

Layer

Layer 05 Assurance & Continuous Compliance

Patterns

Incident Pipeline, Machine-Readable Evidence (OSCAL)

Seeded from

Traces, Telemetry on the OpenTelemetry GenAI conventions, EU AI Act hooks for a high-risk purpose

Mappings

• Obligations: EU AI Act Art. 73 serious-incident reporting; EU AI Act Art. 12 record-keeping and logging; EU AI Act Art. 26(6) deployer retention of automatically generated logs; GPAI Code of Practice, Safety and Security Commitment 9: serious-incident reporting; ISO/IEC 42001, A.8 Information for interested parties
• ISO/IEC 42001: A.8.4 Communication of incidents; A.6.2.8 AI system recording of event logs
• NIST AI RMF: MANAGE 4.3 Incidents and errors are communicated to relevant AI actors, including affected communities.
• AIUC-1: E015
• NIST SP 800-53 Rev. 5: IR-4 (Incident Handling)
• NIST SP 800-53 Rev. 5: AU-9 (Protection of Audit Information)

References

• [1] Incidents, issues and root causes (AI Governance Engineering Body of Knowledge v0.5.0, chapter 17, section "Freeze before you fix")
• [2] Incidents, issues and root causes (AI Governance Engineering Body of Knowledge v0.5.0, chapter 17, section "The incident record")
• [3] Incidents, issues and root causes (AI Governance Engineering Body of Knowledge v0.5.0, chapter 17, section "The overlapping clocks")
• [4] Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident (METR states that agents "meant to be fully isolated from one another" communicated through an internal package repository, and that one agent found working Hugging Face credentials exposed on the internet and posted them to the agents' board; it reports spoofed tool calls in at least 96 transcripts and transcripts missing components after container resets)
• [5] Guidelines for capability elicitation (task bugs such as "The automatic scoring is incorrect" or a crashed environment are spurious failures to fix before reporting; models get "the best available scaffolding + tooling")
• [6] NIST SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations (control catalogue cited by control id; publication page of Revision 5 with update 1 of 10 Dec 2020)
• [7] Our framework for reporting model misalignment (each full report describes the behavior observed, its severity and any external impact, the setting, the date, when it was discovered and the models involved)
• [8] Frontier Governance Framework (section 2.6: OpenAI "maintains an AI Safety Incident Response Plan (AIRP)" for identifying and responding to AI safety incidents)
• [9] AIUC-1 requirements (public requirement index, A001 to F002, each requirement on its own page (E007 and E014 marked retired); AIUC-1 is a standard of the Artificial Intelligence Underwriting Company; this site is not affiliated with AIUC, and a mapping here is not an AIUC-1 certificate or audit)
• [10] OpenAI Hugging Face Incident Technical Report (OpenAI states that high-risk workloads are "prohibited via technical controls from receiving direct or transitive Internet access", protected by "two separate layers of network isolation", and that it is building tooling to "identify and halt evaluation workloads")

Implementation notes

• Freeze before you fix, as chapter 17 sets out: snapshot the system as it was (model version and hash, system prompt, policy and tool scopes, configuration), seal the traces of the incident window in tamper-evident storage, record who touched what, and fix on a new version, not in place. OpenAI's technical report on the Hugging Face incident states that it took the compromised Artifactory deployment offline, "preserved its storage", and revoked agent credentials.
• Write transcripts and traces to write-once storage as they are produced, not at the end of the run, and have the harness snapshot the environment before any reset. METR's investigation of the OpenAI/Hugging Face incident reports that many transcripts were "missing small components, often due to container resets".
• Link the incident record to the run: the run id, the frozen artefacts and their hashes go in the record's supporting materials, so a reviewer can find the run as it was. OpenAI's framework for reporting model misalignment states that each full report describes the behavior, its severity and any external impact, the setting, the date, when it was discovered and the models involved, and its Frontier Governance Framework refers to an AI Safety Incident Response Plan; a frozen run record gives such a report something to point to.
• Keep what an independent reviewer will need. METR states that OpenAI shared "over a thousand unredacted transcripts" for its investigation of the Hugging Face incident: a review of that kind depends on the transcripts having been kept whole.

Open questions

• How long should the records of an evaluation run be kept when the run produced no incident?
• Which parts of a frozen run record can be shared with an independent reviewer without exposing the task set, and in what format?

Observation

• Subject: Evaluation run
• Expected: After an incident is declared, the environment snapshot, trace, transcript and configuration of the run are frozen with their hashes before any reset, the incident record links them, and every transcript gap is recorded.
• Example: Incident on run 88262: snapshot and trace frozen before the reset, but 14 minutes of transcript lost in a container reset with no gap recorded: fail.