Production AI quality and releases: understand the problem
- Source
- From the Staff AI System Design course
- By
- Sourav Banerjee
- Last updated
- 8 Nov 2026
Alice's supplier-policy answer remains fast and available after a model or retrieval change, but now gives the wrong confirmation rule. Another candidate answers a foreign tenant's question instead of refusing. Infrastructure health alone cannot detect those regressions. Version what changed, evaluate the right slices, and make release/rollback decisions without weakening the platform's existing safety contracts.
This remains the same hypothetical procurement system. The runnable evaluator scores precomputed fixture outputs against eight authored cases. Latency and cost values are synthetic labels, not measured model performance or prices. A passed report permits only the teaching release exercise, never production approval. Estimated integrated study: six to eight hours; learner mastery remains unassessed.
Inherit a Recoverable, Protected Platform
T18.8, its snapshot and ADR-007 establish data-class consistency, result/outbox atomicity, projection freshness, bounded recovery and diagnostic contracts. Retain current identity/policy/audit, approved egress, source citations, immutable-cache safety, worker epochs, stable operation identity and fair provider budgets.
Before diagram. We can diagnose an outage or lost acknowledgement, but have not established a versioned quality-evaluation and release decision path. A successful HTTP response is not evidence the answer follows the current supplier rule.
Why One Average Score Is Insufficient
The candidate answers five ordinary supplier questions correctly and refuses an unsupported one, but discloses one foreign-tenant case. A high aggregate can hide that known violation. Likewise, an apparently better answer score can come from longer prompts, larger output budgets or a different dataset, making cost and latency comparisons invalid.
| Engineering question | Formal idea | Limit |
|---|---|---|
| Can we reproduce the exact change and its evidence? | Version bundle and evaluation manifest | A model name alone omits prompt, retrieval, source and evaluator versions |
| Which customer cases got worse? | Dataset slices | An overall mean can hide access/refusal or tenant-specific failure |
| Should a candidate receive any customer traffic? | Release gate | A passing small fixture is not proof of general safety |
| Can a small eligible group try the change first? | Canary | Exposure is controlled, not risk-free or necessarily statistically decisive |
| Can we compare without using candidate business actions? | Shadow | Extra inference still costs resources and requires authorised data handling |
Coverage and Outcomes
| Module | Treatment | Assigned depth and evidence |
|---|---|---|
| Evaluation architecture | Taught here | Build depth: versioned cases, rubric/slices, coverage binding, quality/safety/latency/cost gates and tests |
| Release management and experimentation | Taught here | Build depth: approved bundles, sticky eligible canaries, shadow contract, CAS promotion and rollback |
| Observability systems | Taught here | Build depth: version/cohort telemetry schema, metrics/logs/traces, privacy/cardinality and release diagnosis |
| Distributed state and transactions | Revisited from T18.8 | Build depth: atomic generation-checked release pointer; rollback preserves durable result identity |
| End-to-end AI application design | Revisited from T18.2 | Build depth: bind prompt/retrieval/model and source context into one reproducible outcome |
| Agentic application system architecture | Revisited from T18.3 | Build depth: shadow/evaluation cannot perform live business side effects |
| Enterprise AI security | Revisited from T18.7 | Build depth: known safety cases gate release separately; logs and shadows retain data boundaries |
| Performance and incident diagnosis | Revisited from T18.8 | Build depth: compare version/cohort and retrieval/generation failure rather than global health alone |
| Model and data lifecycle infrastructure | Context only | T18.11, Design depth: use immutable IDs now; broader training/registry lifecycle follows |
| Architecture economics | Deferred | T18.13, Build depth: no unrelated total-cost platform programme |
Outcomes: execute a reproducible fixture evaluation; inspect slice failures; block known safety/cost/latency regressions; control a release pointer without stale writes; and diagnose/roll back without rewriting saved operations. Exit gates assess the learner separately from artefact verification.
Key Takeaways
- A fast, available answer can still apply the wrong rule or answer a foreign tenant's question. Infrastructure health cannot detect those regressions, so changes need versioned evaluation and release decisions.
- A high average can hide a known violation. A candidate that handles five ordinary cases and one refusal correctly but discloses one foreign-tenant case must not pass on its aggregate.
- Score gains from longer prompts, larger output budgets or a different dataset make cost and latency comparisons invalid. A model name alone omits the prompt, retrieval, source and evaluator versions.
- A canary controls exposure but is neither risk-free nor necessarily statistically decisive. Shadow traffic avoids candidate business actions but still costs inference and needs authorised data handling.