Risk, Ethics & Compliance ยท RSK-08
Explanation and challenge for affected people
People affected by decisions that AI supports can find out that AI was involved, get an explanation they can understand, and challenge the outcome through a route that can change it.
Why it matters
People are entitled to understand decisions made about them, and some have legal rights to do so. A working route to explanation and challenge also corrects individual errors and shows the organisation where a system is going wrong.
Levels
| 0 Absent | Confirmed absent: the evidence shows nothing is in place. Not known is recorded as insufficient evidence, not 0. |
|---|---|
| 1 Initial | Affected people are not told AI was involved and have no specific route to question the outcome. |
| 2 Developing | Explanations or challenge are available for some uses, or on request only, and the route may not be able to change outcomes. |
| 3 Defined | People know when AI is involved in decisions about them or in their interactions, can get a meaningful explanation, and can challenge outcomes through human review. |
| 4 Managed | Requests for explanation and challenges are logged, outcomes and response times are measured, and patterns are fed back to the owners of the use. |
| 5 Optimising | Explanation and challenge are designed in from the start, and the route is improved from the experience of the people who use it. |
Level 3 criteria
Level 3 is met when every applicable criterion is met. Criteria marked with an overlay apply only to organisations with that characteristic (overlays).
- RSK-08.1 For every use of AI that supports decisions about people, those people are told that AI was involved.
- RSK-08.2 People affected by such a decision can obtain a meaningful explanation of it.
- RSK-08.3 People affected by such a decision can challenge the outcome through a defined route with human review and authority to change it.
- RSK-08.4 People are told when they are interacting with AI, at first contact, clearly and accessibly, unless it is obvious.
- RSK-08.5 People are told when they are exposed to emotion recognition or biometric categorisation, at first contact, clearly and accessibly, unless it is obvious. HIR
Assessment questions
- How do people know when AI has contributed to a decision about them?
- What explanation can they get, and is it one a non-specialist can understand?
- How can they challenge an outcome, and how often do challenges succeed?
- Notices or communications telling people AI is involved
- The documented explanation and challenge route
- A log of challenges and their outcomes
- Notices shown to people at first contact with AI services
How to reach Level 3
- Identify every use of AI that supports decisions about people.
- For each, write a clear notice telling people AI was involved, and set up a way for them to obtain a meaningful explanation.
- Set up a challenge route with human review and authority to change the outcome.
- Tell people clearly and accessibly at first contact when they are interacting with AI or are exposed to emotion recognition or biometric categorisation, unless it is obvious.
Common pitfalls
- Offering an explanation that describes the technology but not why this person got this outcome.
- Routing challenges to staff who cannot change the decision.
Starter artefacts
- AI involvement notice template
- Explanation and challenge procedure
Cross-references
| AI Cyber Security Code of Practice | 4.2 partial |
|---|---|
| Algorithmic Transparency Recording Standard | Tier 2 Deployment context direct; Scope tools in scope partial |
| EU AI Act | Art 26(11) direct; Art 27 partial; Art 50(1) direct; Art 50(3) direct; Art 86 direct; Art 13 partial |
| ISO/IEC 42001 | A.6.2.7 partial; A.8.2 partial; A.8.3 partial |
| JSP 936 Part 1 | para 39 partial; para 49c partial; para 51 partial; para 61 partial; para 64 partial |
| MITRE AI Maturity Model | P1.1 direct; P1.2 direct; P6.4 partial |
| NIST AI 600-1 | R8 partial |
| NIST AI RMF 1.0 | MEASURE 2.8 partial; MEASURE 2.9 partial; MEASURE 3.3 partial; MANAGE 4.1 partial |
| UK Data and AI Ethics Framework | Transparency 5 partial; Accountability 7 direct; SAT 2.5 partial; SAT 4.5 direct; SAT 4.6 direct |
| UNESCO AI Maturity Framework | 5.5 direct |
direct: this control at Level 3 meets the item. partial: it covers part of it.