Portfolio & Use Cases ยท POR-07
Pilot to production decisions
Every pilot ends in a recorded decision to scale, extend or stop, made against criteria and evidence agreed before the pilot started.
Why it matters
Pilots that are never formally ended either drift into live use without proper checks or linger and absorb effort. Agreeing success criteria in advance stops results being judged by enthusiasm after the event.
Levels
| 0 Absent | Confirmed absent: the evidence shows nothing is in place. Not known is recorded as insufficient evidence, not 0. |
|---|---|
| 1 Initial | Pilots continue or end by default; decisions are not recorded. |
| 2 Developing | Some pilots have success criteria, but they are set late, loosely defined, or not used to decide. |
| 3 Defined | Every pilot has success and stop criteria agreed at the start and ends in a recorded, evidence-based decision to scale, extend or stop. |
| 4 Managed | Pilot outcomes are tracked across the portfolio, extensions are time-limited and reviewed, and pilots running past their end date are flagged and resolved. |
| 5 Optimising | Decision criteria and pilot design are improved from how scaled uses perform in live service. |
Level 3 criteria
Level 3 is met when every applicable criterion is met. Criteria marked with an overlay apply only to organisations with that characteristic (overlays).
- POR-07.1 Every pilot has success and stop criteria agreed at the start.
- POR-07.2 Every pilot ends in a recorded decision to scale, extend or stop.
- POR-07.3 Each pilot decision is based on the evidence gathered in the pilot.
Assessment questions
- What criteria were agreed before your most recent pilot began?
- How was the decision to scale, extend or stop it made, and by whom?
- Are any pilots running past their planned end without a decision?
- Pilot plans showing success and stop criteria
- Recorded scale, extend or stop decisions with supporting evidence
How to reach Level 3
- Before each pilot starts, agree success and stop criteria and the evidence that will be gathered (see MON-01).
- Gather the evidence during the pilot as planned.
- End every pilot with a recorded scale, extend or stop decision based on the evidence, logged in the pipeline register (POR-03).
Common pitfalls
- Setting criteria after the pilot has run, so they fit the results.
- Extending pilots repeatedly to avoid making a decision.
Starter artefacts
- Pilot charter with success and stop criteria
- Pilot decision record
Cross-references
| ISO/IEC 42001 | A.6.2.5 partial |
|---|---|
| JSP 936 Part 1 | para 206 partial |
| NIST AI RMF 1.0 | MANAGE 1.1 direct |
| UK Data and AI Ethics Framework | Safety 8 partial; SAT 3.9 partial |
| UNESCO AI Maturity Framework | 1.5 direct |
direct: this control at Level 3 meets the item. partial: it covers part of it.