Measure & Improve ยท MON-01
Performance measures and baselines
Each use of AI has agreed performance measures and a recorded baseline of current performance before it goes live.
Why it matters
Without a baseline taken before go-live, nobody can show later whether AI made things better or worse. Agreeing measures in advance also stops success being defined after the event to fit whatever happened.
Levels
| 0 Absent | Confirmed absent: the evidence shows nothing is in place. Not known is recorded as insufficient evidence, not 0. |
|---|---|
| 1 Initial | Measures are not set; success is judged by impression or anecdote. |
| 2 Developing | Some uses of AI have measures, but they are set after go-live, lack a baseline or cover only technical performance. |
| 3 Defined | Every use of AI has owned measures of accuracy, outcome, effect on people and reliance, with baselines set before go-live and proxies for risks not yet measurable. |
| 4 Managed | Measures and baselines are checked at stage gates, and uses without them are stopped or held until they are in place. |
| 5 Optimising | Measures are refined from experience of which ones actually predicted success or problems, and common measures allow comparison across uses. |
Level 3 criteria
Level 3 is met when every applicable criterion is met. Criteria marked with an overlay apply only to organisations with that characteristic (overlays).
- MON-01.1 Every use of AI has agreed measures covering accuracy, service outcome and effect on people, with a named owner.
- MON-01.2 A baseline for each measure is recorded before go-live.
- MON-01.3 Measures include reliance, meaning how often outputs are accepted, checked or overridden, compared with measured accuracy so that overreliance and underreliance show up.
- MON-01.4 Risks that cannot yet be measured are recorded, with the indicators or proxies used to track them.
- MON-01.5 Where the environmental footprint of a use is material, measures include it, such as energy, compute and water, with a baseline.
Assessment questions
- For a recent use of AI, what measures were agreed and when?
- What was the baseline before go-live, and how was it measured?
- Do the measures cover effects on people as well as technical performance?
- Measure definitions and baseline data for a sample of uses of AI
- Stage gate records showing measures were checked
- Recorded risks that cannot yet be measured, with their proxy indicators
- Reliance measures compared with accuracy for live uses
How to reach Level 3
- For each use of AI, agree measures with a named owner covering accuracy, service outcome and effect on people, plus environmental footprint where it is material.
- Include a reliance measure, such as how often outputs are accepted, checked or overridden, and plan to compare it with measured accuracy.
- Record a baseline for every measure before go-live, using current performance of the process the AI will support.
- List any risks that cannot yet be measured and set the indicators or proxies you will use to track them.
Common pitfalls
- Measuring only technical accuracy and missing the effect on people and services.
- Setting measures after go-live, when no baseline can be taken.
- Ignoring reliance, so people accepting wrong outputs without checking goes unseen.
Starter artefacts
- Measures and baseline template for each use
- Unmeasurable risks and proxies list
Ownership and sequence
- Accountable
- business owner
- Contributors
- delivery lead, data owner
- Depends on
- POR-05
- Roadmap step
- 6
- Lifecycle
- design, acquire_build, deploy
Cross-references
| Algorithmic Transparency Recording Standard | Tier 2 Description and rationale partial |
|---|---|
| EU AI Act | Art 14 partial |
| ISO/IEC 42001 | 6.2 partial; 9.1 direct |
| JSP 936 Part 1 | para 6 partial; para 49a partial; para 76 direct |
| MITRE AI Maturity Model | P4.2 partial; P6.3 partial; P6.4 direct |
| NIST AI 600-1 | R5 partial; R7 direct |
| NIST AI RMF 1.0 | MEASURE 1.1 partial; MEASURE 2.3 partial; MEASURE 2.12 direct; MEASURE 3.2 direct; MEASURE 4.1 partial |
| UK Data and AI Ethics Framework | Environmental sustainability 3 direct; SAT 2.12 partial; SAT 3.7 partial; SAT 3.9 partial; SAT 4.1 partial; SAT 4.12 partial |
| UNESCO AI Maturity Framework | 1.4 direct; 4.2 partial |
direct: this control at Level 3 meets the item. partial: it covers part of it.