Risk, Ethics & Compliance ยท RSK-07
Human oversight proportionate to risk
Each use of AI has human oversight designed to match its risk, stating who reviews, at which point, and with what authority to override or stop it.
Why it matters
Putting a person in the loop does not help if they lack the time, information or authority to disagree with the system. Oversight has to be designed for each use so that human control is real where it is needed and not imposed as a formality where it adds nothing.
Levels
| 0 Absent | Confirmed absent: the evidence shows nothing is in place. Not known is recorded as insufficient evidence, not 0. |
|---|---|
| 1 Initial | Oversight is not designed; whether anyone checks AI outputs depends on the individual. |
| 2 Developing | Some uses have a human review step, but its purpose, timing and authority are not defined, or reviewers lack the information to challenge outputs. |
| 3 Defined | Every use of AI has a documented, risk-based oversight model with trained reviewers, tested before go-live to show they can detect problems and intervene. |
| 4 Managed | Oversight is checked in practice, for example override rates and review times, and signs of rubber-stamping or overload are acted on. |
| 5 Optimising | Oversight designs are refined from evidence of where human review adds value and where it does not. |
Level 3 criteria
Level 3 is met when every applicable criterion is met. Criteria marked with an overlay apply only to organisations with that characteristic (overlays).
- RSK-07.1 For every use of AI, a documented oversight model, set according to risk, states who reviews, when, with what information, and their authority to override or stop the use.
- RSK-07.2 Reviewers are trained for their oversight role.
- RSK-07.3 The oversight design is tested before go-live to show that reviewers can detect problems and intervene in time.
- RSK-07.4 Agent actions that are consequential or hard to reverse need a person's approval before they take effect, and the oversight model lists which actions these are. AGT
Assessment questions
- For a given use of AI, who reviews its outputs, at what point, and can they override it?
- How was the level of oversight chosen, and how does it relate to the risk?
- How do you know reviewers are genuinely checking rather than approving by default?
- List of agent actions that need a person's approval, with approval records
- Documented oversight models for in-scope uses of AI
- Training records for people in oversight roles
- Override and review data from live uses
How to reach Level 3
- Define a small set of oversight models (for example a person decides, a person reviews before action, a person monitors) and link each to risk tiers.
- For every use, document the chosen model, who reviews, when, with what information, and their authority to override or stop.
- Train reviewers for their role, including the limits of the AI and the risk of over-trusting it (see PPL-05).
- Test the oversight design before go-live to show reviewers can detect problems and intervene in time.
Common pitfalls
- Naming a reviewer who lacks the time, information or authority to intervene.
- Assuming oversight works without testing whether reviewers actually catch errors.
Starter artefacts
- Oversight model catalogue linked to risk tiers
- Per-use oversight design record
- Oversight test scenario pack
Ownership and sequence
- Accountable
- business owner
- Contributors
- delivery lead, risk lead, people lead
- Depends on
- RSK-02
- Roadmap step
- 4
- Lifecycle
- design, deploy, operate
Cross-references
| AI Cyber Security Code of Practice | 4.1 direct; 4.2 partial; 4.3 partial |
|---|---|
| Algorithmic Transparency Recording Standard | Tier 2 Deployment context direct |
| EU AI Act | Art 26(2) direct; Art 26(5) partial; Art 27 partial; Art 14 partial |
| GDS AI Playbook | P4 direct |
| ISO/IEC 42001 | 6.1.3 partial; 8.3 partial |
| JSP 936 Part 1 | para 35 direct; para 49b direct; para 49c partial; para 75 partial; para 91 partial; para 125 partial; para 180 partial; para 185 partial |
| MITRE AI Maturity Model | P1.1 partial; P1.2 partial; P4.1 partial; P6.4 partial |
| NIST AI 600-1 | R2 partial; R7 direct |
| NIST AI RMF 1.0 | GOVERN 3.2 direct; MAP 2.2 partial; MAP 3.4 partial; MAP 3.5 direct; MANAGE 2.4 direct |
| UK Data and AI Ethics Framework | Accountability 8 direct; Societal impact 5 direct; Safety 6 direct; SAT 3.4 direct |
| UNESCO AI Maturity Framework | 5.5 partial |
direct: this control at Level 3 meets the item. partial: it covers part of it.