Pi Mapper

Insights / Assurance

Go-live is a threshold, not a date

What it takes to give a risk committee an evidence-based answer to the only question that matters: how do you know it is safe?

Malika Bhatia, June 2026. 5 min read.

The go-live date for an AI agent is usually set months in advance by a programme plan. The go-live decision is usually taken in a meeting where someone says the testing "looked fine". Regulated firms have spent decades learning not to run other changes this way. Agents deserve the same discipline.

What "fine" needs to mean

Before the first evaluation run, four things need to be agreed in writing with the process owner and second line:

  1. The decisions. What does the agent actually decide? Classify a document, judge a pack complete, draft a letter, route a case. Each is evaluated separately.
  2. The correct answer. For each decision, what would a competent senior operator have done? This comes from real historical cases, labelled by the people who know.
  3. The threshold. How often must the agent match that answer before it is trusted? Some thresholds are 96%. Some, like recognising a vulnerable customer, are 100% with no exceptions.
  4. The evidence set. Enough real cases, stratified across the variants and exceptions discovery found, that the result means something. Not the ten cases the pilot happened to use.

Set these after the run and the numbers will be argued into acceptance. Set them before, and the run produces a decision.

Conformance is a separate test

Accuracy asks whether the agent reached the right answer. Conformance asks whether it followed the designed process on the way: did it stop where it must stop, update what it must update, escalate what it must escalate? An agent can be accurate and non-conformant, and in a regulated process the second failure is often the more serious one. We test both.

Failing well

A first run that passes every gate is a sign the gates were too loose. Expect failures, and expect most of them to be fixable at the context layer: a missing alias, an unstated rule, a precedence order that was wrong. The evaluation set tells you exactly which cases failed and why, so the fix is targeted. A failure that genuinely requires a different model is rare and worth knowing about early.

Release is a gradient

Crossing the threshold does not mean switching the agent on. It means starting a controlled release: shadow mode with outputs compared daily, then assisted mode with a person approving every output and every override reviewed, then live with sampling that steps down as evidence accumulates. Each stage has its own exit criteria. The kill switch is a button operations can press without calling engineering.

The question a good CRO asks is not "is it safe?" but "how will you know when it stops being safe?" The answer is monitoring designed alongside the gates: classification confidence, stop rate, override rate, drift against the evaluation set. Any of them moving triggers a review of the context pack first and the model second.

Treated this way, go-live stops being a leap of faith on a date and becomes a threshold the agent earns its way across. That is a decision a risk committee can sign, and one that survives the first hard week in production.