Data & Information ยท DAT-03
Data quality fit for purpose
The quality of data used by each AI system is assessed against what that use needs, and shortfalls are fixed or their effects managed before and during live use.
Why it matters
AI reproduces the gaps, errors and skews in its data, often at scale and with confidence. Quality is relative to the use: data adequate for monthly reporting may be unfit for an automated decision about an individual.
Levels
| 0 Absent | Confirmed absent: the evidence shows nothing is in place. Not known is recorded as insufficient evidence, not 0. |
|---|---|
| 1 Initial | Data quality is assumed; problems are found when outputs look wrong. |
| 2 Developing | Quality is checked for some uses or some data sets, without agreed measures or a link to what the AI use needs. |
| 3 Defined | Quality requirements are set for each use of AI, data is assessed against them before and during use, and shortfalls are owned and remediated or mitigated. |
| 4 Managed | Quality measures are tracked over time for data used by AI, falls below threshold trigger action, and the effect of quality on AI outputs is understood. |
| 5 Optimising | Quality issues are fixed at source, and requirements and checks are refined from evidence of which quality problems most affected AI outcomes. |
Level 3 criteria
Level 3 is met when every applicable criterion is met. Criteria marked with an overlay apply only to organisations with that characteristic (overlays).
- DAT-03.1 Quality requirements, including relevance and representativeness for the intended purpose, are set for each use of AI.
- DAT-03.2 Data is assessed against those requirements before use.
- DAT-03.3 Data is reassessed against those requirements at intervals during use.
- DAT-03.4 Each quality shortfall has an owner and a remediation or mitigation plan.
Assessment questions
- What quality does this use of AI need from its data, and who decided that?
- When was the data last assessed against those requirements, and what did it find?
- What happens when data quality drops while the system is live?
- Data quality requirements for a sample of uses of AI
- Data quality assessment results and the remediation actions that followed
How to reach Level 3
- For each use of AI, set quality requirements for its data, including relevance and representativeness for the intended purpose.
- Assess the data against those requirements before use and record the result.
- Reassess at set intervals while the AI is live.
- Give every shortfall an owner and a plan to fix it or reduce its effect.
Common pitfalls
- Checking completeness and accuracy but not whether the data represents the people or cases the AI will meet.
- Assessing quality once before launch and never again.
Starter artefacts
- Data quality requirements per use
- Data quality assessment record
- Shortfall and remediation log
Ownership and sequence
- Accountable
- data owner
- Contributors
- delivery lead, business owner
- Depends on
- DAT-02
- Roadmap step
- 2
- Lifecycle
- design, acquire_build, deploy, operate
Cross-references
| AI Cyber Security Code of Practice | 5.4 partial |
|---|---|
| Algorithmic Transparency Recording Standard | Tier 2 Development data specification direct |
| EU AI Act | Art 26(4) direct; Art 10 partial |
| ISO/IEC 42001 | A.7.2 partial; A.7.4 direct; A.7.6 partial |
| JSP 936 Part 1 | para 69 partial; para 144 partial; para 145 partial |
| MITRE AI Maturity Model | P1.3 partial; P5.3 direct |
| NIST AI 600-1 | R6 partial |
| NIST AI RMF 1.0 | MAP 2.3 partial |
| UK Data and AI Ethics Framework | Safety 5 direct; SAT 2.2 direct; SAT 2.8 direct; SAT 3.1 direct |
| UNESCO AI Maturity Framework | 6.1 partial; 6.2 direct; 6.4 partial |
direct: this control at Level 3 meets the item. partial: it covers part of it.