Production AI quality and releases: think like a staff engineer

Separate infrastructure failure, retrieval miss/staleness, unsupported answer, access violation and cost/latency regression. Inspect candidate/control bundle identities and dataset slices first. A changed corpus, evaluator, timeout or refusal population can invalidate an apparent model comparison. Freeze overlapping risky changes when attribution is unclear.

For a suspected leak, stop affected exposure and preserve restricted evidence under incident access controls; do not paste raw prompts into shared diagnostics. For a quality regression with no known safety issue, compare source/citation and rubric failures before changing thresholds. For latency/cost, inspect queue/model/token distributions and shadow overhead rather than only average request duration.

Rollback requires a tested previous approved bundle and a compatible current runtime/source contract. Never roll back tenant membership, audit policy or source tombstones to match an old evaluation snapshot. If an old model is no longer approved for a tenant/provider destination, the safe response may be refusal rather than automatic fallback. Keep T18.8 unknown effects and T18.6 reservations reconciled independently of release flags.

Risk management needs owners, evidence and follow-through; NIST AI RMF is a primary governance reference, not a certification this fixture attains. The local approval API assumes trusted control-plane callers and authentic evaluation results; production needs control-plane authentication, artefact provenance and calibrated human approval.

Prompt: The report says model v2 passed. Is that enough to reproduce the release?

Spoken answer: “No. I need the prompt, retrieval settings, source/index generation, model revision, security policy, dataset, evaluator and gate versions. I bind the report to those immutable identities and inspect per-case/slice outcomes. A model name or mutable alias cannot explain which system behaviour passed. Even a reproducible fixture report is only evidence within its coverage.”

Trade-off: More version bookkeeping adds release friction. Changed constraint: If the source corpus changes hourly, define a valid comparison snapshot and current-policy rule without freezing revocations.

Prompt: Most cases improved but one foreign-tenant case now returns text.

Spoken answer: “That is independently blocking, regardless of the aggregate score. I keep known access/refusal failures as hard gates and compare relevant slices. Zero observed failures does not prove zero risk, especially with a tiny suite. I would preserve evidence, stop exposure and investigate the runtime boundary rather than relax quality settings after seeing the result.”

Trade-off: Conservative gates can slow useful improvements. Changed constraint: Explain how you would expand a representative safety set without leaking customer data into an evaluation corpus.

Prompt: Randomly choose the model independently for every retry.

Spoken answer: “That can mix versions within one user's experience or operation and complicate attribution. I choose a stable experiment unit and eligible tenant set, retain operation/result identity, and log bundle/cohort safely. Assignment percentages are approximate across small populations. Any shadow/canary call still respects authorisation, egress and provider budgets.”

Trade-off: Sticky cohorts may be unrepresentative or correlated. Changed constraint: If a few large tenants dominate traffic, what unit and comparison plan would you choose?

Prompt: A candidate was promoted, then a new canary fails. Should rollback always return to the oldest release?

Spoken answer: “No. Withdrawing an active candidate should retain the current approved stable release. Reverting a completed promotion is a distinct pointer change to its previous approved version. I use a generation check so stale commands cannot overwrite newer decisions. Neither operation rewrites saved answers, revives permissions or retries uncertain provider effects.”

Trade-off: Explicit release states require more careful control-plane logic. Changed constraint: If the prior stable model is no longer approved for a tenant, specify a safe withdrawal/response path.

• Separate infrastructure failure, retrieval miss or staleness, unsupported answer, access violation and cost or latency regression. A changed corpus, evaluator, timeout or refusal population can invalidate an apparent model comparison.
• For a suspected leak, stop affected exposure and keep restricted evidence under incident access controls. Never roll back tenant membership, audit policy or source tombstones to match an old evaluation snapshot.
• Choosing a model at random for every retry mixes versions within one user's experience and confuses attribution. Canaries use a stable experiment unit and log bundle and cohort with each result.
• Withdrawing an active canary keeps the current stable release, while reverting a completed promotion moves to its previous approved version. If that old model is no longer approved for a tenant, refusal may be safer than automatic fallback.