AI Security ยท SEC-02
AI threat modelling
Each use of AI has its AI-specific threats assessed before it goes live and again when it changes, and the results shape its design and controls.
Why it matters
AI introduces ways of attacking a service that conventional threat assessment does not look for, such as manipulated inputs, poisoned training data and misuse of connected tools. Assessing these per use, in proportion to risk, finds problems while they are still cheap to fix.
Levels
| 0 Absent | Confirmed absent: the evidence shows nothing is in place. Not known is recorded as insufficient evidence, not 0. |
|---|---|
| 1 Initial | AI-specific threats are considered only when an individual raises them. |
| 2 Developing | Threat modelling covers AI for some uses or teams, or uses conventional methods that miss AI-specific threats. |
| 3 Defined | A defined approach covering AI-specific threats, including synthetic media impersonation, is required for every use of AI, with findings recorded and acted on. |
| 4 Managed | Threat models are reviewed when uses change or new threats emerge, completion is measured, and untreated findings are escalated. |
| 5 Optimising | Threat modelling is refined from incidents, testing results and external threat intelligence, and common threats are handled once in shared patterns. |
Level 3 criteria
Level 3 is met when every applicable criterion is met. Criteria marked with an overlay apply only to organisations with that characteristic (overlays).
- SEC-02.1 A defined threat modelling approach covers AI-specific threats.
- SEC-02.2 Threat modelling is required for every use of AI, proportionate to risk.
- SEC-02.3 Threats assessed include impersonation of the organisation or its people through synthetic media.
- SEC-02.4 Threat modelling findings are recorded for each use of AI.
- SEC-02.5 Threat modelling findings are acted on.
- SEC-02.6 Threat modelling for agents covers instructions hidden in content they read, misuse of their tools, and chains of actions that go beyond their purpose. AGT
Assessment questions
- Is threat modelling required for every use of AI, and does it cover threats specific to AI?
- Who carries it out, and do they have the AI knowledge to do it well?
- Show how the findings of a recent threat model changed the design or controls.
- Agent threat models covering hidden instructions and tool misuse
- The threat modelling method or guidance, showing AI-specific threats
- Completed threat models for a sample of uses, with findings and actions
- Synthetic media impersonation covered in threat models
How to reach Level 3
- Adopt a threat modelling approach that covers AI-specific threats, such as manipulated inputs, poisoned training data and impersonation through synthetic media.
- Require a threat model for every use of AI, scaled to its risk (RSK-02), before go-live and when it changes.
- Record findings and track the resulting actions to completion.
Common pitfalls
- Using a conventional method that misses threats unique to AI.
- Producing a threat model that nobody acts on.
Starter artefacts
- AI threat modelling guide
- Threat model template
- Findings and actions log
Cross-references
| AI Cyber Security Code of Practice | 2.1 direct; 2.2 partial; 2.6 partial; 3.1 direct; 6.2 partial; 9.1 partial; 9.4 partial; 11.2 partial |
|---|---|
| Algorithmic Transparency Recording Standard | Tier 2 Risks, mitigations and impact assessments partial |
| EU AI Act | Art 15 partial |
| GDS AI Playbook | P3 direct |
| JSP 936 Part 1 | para 36 partial; para 79 direct; para 80 direct; para 81 partial; para 195 partial; para 196 direct; para 197 direct; para 209 partial |
| MITRE AI Maturity Model | P4.2 partial; P5.2 partial |
| NIST AI 600-1 | R1 partial; R3 partial; R8 partial; R9 direct |
| NIST AI RMF 1.0 | MEASURE 2.7 partial |
| UK Data and AI Ethics Framework | Safety 5 partial; SAT 1.4 partial; SAT 2.1 partial |
| UNESCO AI Maturity Framework | 5.3 partial |
direct: this control at Level 3 meets the item. partial: it covers part of it.