Pattern: Model Artefact Integrity

Layer 02 · Inventory & Transparency Layer 04 · Runtime Controls & Observability In the chapter 05 catalogue

One working control on the engineering side of AI governance.

Summary: Treat model weights and their companion files as supply-chain artefacts. Sign a manifest of every file and its digest when the build produces them, attach build provenance that says what built them from which inputs, prefer serialisation formats that cannot execute code and scan the ones that can, and make every runtime verify the signature, the digests and the provenance against the registry entry before it loads a model. The verification writes an evidence record, so “which bytes ran, and who produced them?” has an answer.

Objectives

Guarantee that the model a service loads is the model the pipeline built, evaluated and registered, from a known producer, and that loading it cannot run an attacker’s code.

Target users

ML platform team, security engineer, AI governance engineer, MLOps engineer.

Impacted stakeholders

Users and people affected by the system’s outputs, model owners, downstream deployers, auditors and market surveillance authorities.

Relevant principles

Register and bound every actor before it acts; build the control at the earliest point it can block; instrument the build to produce its own proof.

Context

Models move from training jobs to registries to serving clusters, are fine-tuned from bases pulled from public hubs, and are copied between environments and regions. The files are large binaries nobody reads. Common formats are not inert: the Python documentation warns that “The pickle module is not secure. Only unpickle data you trust.”¹, and Hugging Face describes “dangerous arbitrary code execution attacks” that can run when a pickle file is loaded².

Problem

The pipeline evaluates one model and production may load another.

• Forces. Teams pull weights by name or tag, and a tag can be moved. Scanning helps but, as Hugging Face says of its own pickle-import scanner, it “is not 100% foolproof”². The EU AI Act expects high-risk systems to resist attacks on “pre-trained components used in training (model poisoning)” (Art. 15(5))³. MITRE ATLAS catalogues supply-chain compromise of the model itself (AML.T0010.003) and user execution of unsafe AI artefacts (AML.T0011.000)⁴, and OWASP lists supply chain as LLM04:2026 ⁵ and agentic supply-chain vulnerabilities as ASI04⁶.
• Failure mode. A tampered or malicious model file loads with the privileges of the serving process. Or a model that skipped the eval gate reaches production through a manual copy. After an incident, nobody can prove which weights produced the outputs in question.

Solution

Sign at build, prove how it was built, load only safe formats, and verify before load.

1. Sign at build. Produce a manifest listing every model file and its cryptographic digest and sign the manifest. The OpenSSF Model Signing (OMS) specification, introduced by OpenSSF in June 2025, does exactly this: a detached signature over a manifest of file hashes, in the Sigstore bundle format, and PKI-agnostic (enterprise PKI, self-signed certificates, bare keys or keyless Sigstore)⁷; its reference implementation is the model-signing library and CLI in the sigstore model-transparency repository, whose verification recomputes the hashes⁸.
2. Attach build provenance. Record what built the artefact, by what process and from which top-level inputs, as SLSA provenance (https://slsa.dev/provenance/v1). SLSA v1.2 defines Build L1 (provenance exists), L2 (a hosted build platform, so forging provenance takes an explicit attack) and L3 (hardened builds, where forging it requires exploiting a vulnerability “beyond the capabilities of most adversaries”)⁹. Include the base model’s digest and the dataset admission records among the inputs, and list the same artefacts in the AIBOM.
3. Prefer safe formats; scan the rest. Store weights as safetensors, “a new simple format for storing tensors safely (as opposed to pickle)”¹⁰. Where a framework still loads pickle, keep its restrictions on: PyTorch’s torch.load defaults to weights_only=True in its current documentation and warns “Never load data from an untrusted source”¹¹. Scan every serialised file for code-executing imports before it reaches a registry, and quarantine what fails.
4. Pin third-party models by digest. Pull by content digest, not by tag; verify the publisher’s signature where one exists; scan and re-host the artefact in the internal registry; and record the upstream source and digest in the registry entry.
5. Verify before load. The serving platform’s admission control refuses a model whose signature, signer identity, file digests or provenance do not match the registry entry for the version being deployed, and writes an evidence record either way. ATLAS lists code signing (AML.M0013), verifying AI artefacts (AML.M0014), vulnerability scanning (AML.M0016) and an AI bill of materials (AML.M0023) among its mitigations⁴.

The AI RMF asks that pre-trained models used for development are “monitored as part of AI system regular monitoring and maintenance” (MANAGE 3.2) and that security and resilience are “evaluated and documented” (MEASURE 2.7)¹². Providers of general-purpose models with systemic risk must ensure “an adequate level of cybersecurity protection” for the model and its physical infrastructure (Art. 55(1)(d))³.

Illustrative verification at load, valid against the published evidence record schema (evidence-record.v1):

{
  "$schema": "https://aigovernanceengineer.com/schemas/evidence-record.v1.json",
  "control_id": "model.integrity.verify-at-load.v2",
  "subject": "support-rag-llm@2026-09-20",
  "decision": "allow",
  "obligation": "EU AI Act Art. 15(5)",
  "failure_mode": "tampered or malicious model weights loaded into serving",
  "input_hash": "sha256:8c1f4e2d7a9b3c6e5f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e",
  "actor": "serving-admission-controller",
  "timestamp": "2026-09-20T07:31:05Z",
  "signature": "ed25519:Tq5w2Lr8Zk1Xv7Nc4Pb9Hs6Gd3Fa0Jm",
  "extensions": {
    "signature_bundle": "model.sig",
    "signer": "build-pipeline@models.example.org",
    "files_verified": 7,
    "formats": ["safetensors", "json"],
    "pickle_files": 0,
    "provenance": { "predicate_type": "https://slsa.dev/provenance/v1", "build_level": "L3",
                    "builder": "ci.example.org/model-builds" },
    "registry_entry": "support-rag-llm"
  }
}

Consequences

What runs is provably what was built and evaluated, supply-chain attacks on the model have to defeat a signature and a provenance check instead of a file copy, and every load leaves evidence. The costs: key management or a Sigstore dependency; build-platform work to reach higher SLSA levels; conversion of legacy checkpoints; and verification latency at load, which is small next to model load times but must be budgeted for fast scale-out.

Related patterns

AIBOM; Agent Registry; AI Threat Model; Vendor / Model Due-Diligence Gate; Eval Gate in CI; Continuous Assurance Telemetry.

Maps to: EU AI Act Art. 15(5), Art. 55(1)(d) · ISO/IEC 42001 A.6.2.5, A.10.3 · NIST AI RMF (Govern 6.1; Manage 3.2; Measure 2.7) · OWASP LLM04:2026 · OWASP Agentic ASI04 · MITRE ATLAS · Layer 02 Inventory & Transparency / Layer 04 Runtime Controls & Observability.

Threat IDs follow the OWASP Top 10 for LLM Applications 2026 ⁵ and for Agentic Applications 2026 ⁶ and MITRE ATLAS⁴; function and subcategory labels follow the NIST AI RMF¹²; ISO/IEC 42001 Annex A ids follow a published crosswalk, not the standard’s text¹³. Mappings are illustrative, not a claim of conformity.

Sources

1. [1] pickle: Python object serialization (“The pickle module is not secure. Only unpickle data you trust.”). Python Software Foundation. 2026. https://docs.python.org/3/library/pickle.html (verified: primary)
2. [2] Pickle Scanning (arbitrary code execution when loading pickle files; the Hub’s pickle-import scan “is not 100% foolproof”; safetensors). Hugging Face Hub documentation. n.d. (accessed 2026-09-24). https://huggingface.co/docs/hub/security-pickle (verified: primary)
3. [3] Regulation (EU) 2024/1689 (AI Act): Art. 15(5) resilience against exploitation of vulnerabilities, incl. model poisoning through pre-trained components used in training; Art. 55(1)(d) cybersecurity protection for GPAI models with systemic risk and their physical infrastructure (text read on the Commission’s AI Act Service Desk, 2026-09-24). Publications Office of the EU (EUR-Lex). 2024-07-12. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng (verified: primary)
4. [4] MITRE ATLAS data, release 2026.09 (modified 2026-09-15; AML.T0010.003 AI Supply Chain Compromise: Model; AML.T0011.000 User Execution: Unsafe AI Artifacts; mitigations AML.M0013 Code Signing, AML.M0014 Verify AI Artifacts, AML.M0016 Vulnerability Scanning, AML.M0023 AI Bill of Materials). MITRE (atlas-data repository). 2026-09-15. https://github.com/mitre-atlas/atlas-data (verified: primary)
5. [5] OWASP Top 10 for LLM Applications 2026 (LLM04:2026 Supply Chain). OWASP GenAI Security Project. 2026-08-03. https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/ (verified: primary)
6. [6] Top 10 for Agentic Applications 2026 (ASI04 Agentic Supply Chain Vulnerabilities). OWASP GenAI Security Project. 2025-12-09. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ (verified: primary)
7. [7] “An Introduction to the OpenSSF Model Signing (OMS) Specification” (detached signature over a manifest of file hashes; Sigstore bundle format; PKI-agnostic: private or enterprise PKI, self-signed certificates, bare keys, keyless Sigstore; pip install model-signing). OpenSSF. 2025-06-25. https://openssf.org/blog/2025/06/25/an-introduction-to-the-openssf-model-signing-oms-specification/ (verified: primary)
8. [8] model-transparency: supply chain security for ML (OpenSSF-linked model signing; signs an in-toto statement of file paths and digests through Sigstore or conventional keys; verification recomputes the hashes). Sigstore (GitHub). 2026. https://github.com/sigstore/model-transparency (verified: primary)
9. [9] SLSA specification v1.2, Build track basics (Build L1 provenance exists; L2 hosted build platform; L3 hardened builds; provenance describes what built the artefact, by what process and from which top-level inputs; predicate type https://slsa.dev/provenance/v1). OpenSSF SLSA project. n.d. (accessed 2026-09-24). https://slsa.dev/spec/v1.2/build-track-basics (verified: primary)
10. [10] Safetensors (“a new simple format for storing tensors safely (as opposed to pickle)”). Hugging Face documentation. n.d. (accessed 2026-09-24). https://huggingface.co/docs/safetensors/index (verified: primary)
11. [11] torch.load (default weights_only=True; “Never load data from an untrusted source”). PyTorch documentation (2.14). n.d. (accessed 2026-09-24). https://docs.pytorch.org/docs/stable/generated/torch.load.html (verified: primary)
12. [12] Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (GOVERN 6.1 third-party risks; MANAGE 3.2 pre-trained models “monitored as part of AI system regular monitoring and maintenance”; MEASURE 2.7 security and resilience “evaluated and documented”). NIST. 2023-01-26. https://doi.org/10.6028/NIST.AI.100-1 (verified: primary)
13. [13] NIST AI RMF to ISO/IEC FDIS 42001 crosswalk (provider: Microsoft; lists the Annex B implementation-guidance clauses, whose numbers mirror the Annex A control ids, e.g. B.6.2.5 AI system deployment, B.10.3 suppliers; the ISO text was not opened). NIST AI Resource Center. 2023. https://airc.nist.gov/docs/NIST_AI_RMF_to_ISO_IEC_42001_Crosswalk.pdf (verified: secondary)