Measure & Improve
Covers what happens once AI is in use: setting measures before go-live, monitoring live behaviour, tracking whether benefits arrive, handling non-security incidents and notifying serious ones, independent assurance, listening to users, re-running this assessment and reporting to leadership. Without measurement the organisation cannot tell whether AI is working, causing harm or worth its cost.
Performance measures and baselines
Each use of AI has agreed performance measures and a recorded baseline of current performance before it goes live.
Monitoring live AI
Live AI is monitored against its measures for accuracy, drift, misuse and unexpected behaviour, with thresholds that trigger action.
Benefits tracking
Whether the expected benefits of AI arrive is tracked and reported, for each use and across the portfolio, and shortfalls are acted on.
AI incidents and near misses
AI incidents and near misses that are not security incidents, such as wrong outputs, harm to people or misuse, are reported, investigated and learned from.
Independent assurance reviews
Uses of AI are reviewed by people independent of their delivery, with the depth and frequency of review proportionate to risk.
User feedback
Feedback from people who use AI and people affected by it is gathered on a regular basis, analysed, and acted on.
Re-assessing AI maturity
The organisation re-runs this framework's assessment on a set cycle, compares results with earlier ones, and acts on the gaps it finds.
Reporting to leadership
Leadership and the board receive consolidated, evidence-based reporting on AI use, performance, benefits, incidents and risk, on a set cycle.
Serious AI incidents and external notification
The organisation defines serious AI incidents, can suspend a use of AI when needed, and tells suppliers, regulators and affected people through set routes and within set deadlines.