AI Security ยท SEC-06
Security testing of AI before release
Before release, each AI system is security tested against the threats in its threat model by suitably skilled testers, including checks that it does not disclose personal or sensitive information.
Why it matters
An AI system that passes its accuracy tests can still be manipulated, talked out of its safeguards or made to reveal information it holds. Testing against the threats found for the use, by people who know how AI is attacked, finds these weaknesses before an attacker does.
Levels
| 0 Absent | Confirmed absent: the evidence shows nothing is in place. Not known is recorded as insufficient evidence, not 0. |
|---|---|
| 1 Initial | Systems are released without security testing, or with conventional checks that miss AI-specific threats. |
| 2 Developing | Some security testing is done, but it is not driven by the threat model, testers lack AI skills, or safeguard bypass and information disclosure are not covered. |
| 3 Defined | Every release is security tested against its threat model by suitably skilled testers, independent for higher-risk uses, including safeguard bypass for generative AI and checks for disclosure of personal or sensitive information. |
| 4 Managed | Security test coverage and findings are reviewed across uses, and releases with unresolved findings are blocked or formally accepted. |
| 5 Optimising | Security test methods are refined from incidents and live performance, and security testing is repeated for every change. |
Level 3 criteria
Level 3 is met when every applicable criterion is met. Criteria marked with an overlay apply only to organisations with that characteristic (overlays).
- SEC-06.1 Security testing before release covers the threats in the use's threat model.
- SEC-06.2 Security testing is carried out by testers with the right skills.
- SEC-06.3 For higher-risk uses, security testers are independent of the build team.
- SEC-06.4 Testing of generative AI includes attempts to bypass its safeguards. GEN
- SEC-06.5 Testing checks that personal or sensitive information is not disclosed through the system.
- SEC-06.6 Before release, testing tries to make each agent act outside its approved permissions, including through instructions hidden in content it reads. AGT
Assessment questions
- Does security testing before release cover the threats in the use's threat model?
- Who carries out the security testing, what AI skills do they have, and are they independent of the build team for higher-risk uses?
- How do you test that safeguards cannot be bypassed and that personal or sensitive information is not disclosed?
- Test records of attempts to make agents act outside their permissions
- Security test plans and results for a sample of releases, traced to the threat model
- Tester skills and independence records for higher-risk uses
- Safeguard bypass test results for generative AI
- Information disclosure test results
How to reach Level 3
- Plan security testing for each release from the use's threat model (SEC-02), so every identified threat is tested.
- Use testers with the right skills for AI, and for higher-risk uses testers who are independent of the build team.
- For generative AI, include attempts to bypass safeguards, and for every use check that personal or sensitive information is not disclosed through the system.
- Record the results alongside the other test results so they inform the release sign-off (TEC-06).
Common pitfalls
- Running conventional security tests that miss threats unique to AI.
- Letting the build team mark its own security testing on a higher-risk use.
Starter artefacts
- Security test plan traced to the threat model
- Security test results record
Cross-references
| AI Cyber Security Code of Practice | 2.2 partial; 4.4 partial; 9.1 direct; 9.2 direct; 9.4 partial; 11.2 direct |
|---|---|
| EU AI Act | Art 15 partial |
| JSP 936 Part 1 | para 209 partial |
| MITRE AI Maturity Model | P4.2 partial |
| NIST AI 600-1 | R3 partial; R9 partial |
| NIST AI RMF 1.0 | GOVERN 4.1 partial; MEASURE 2.7 direct |
| UK Data and AI Ethics Framework | Safety 7 partial |
direct: this control at Level 3 meets the item. partial: it covers part of it.