Evaluation and Release Gating

Release process is: you want to change a prompt or model on a live app. A dashboard is not a decision. Pass, block, or hold is.

G13 covers one slice: a shared gate for ~30 apps vs a pinned production baseline. Not auto-tuning.

End to end, as the refund-bot prompt change:

1. Someone files a release with frozen artifacts (prompt, model, retrieval, policy).
2. A versioned suite and rubric run in a sandbox — not on live customers.
3. Deterministic checks plus a calibrated LLM (large language model) grader score quality, safety, latency, cost.
4. Policy compares to the pinned live version. Groundedness −2% might auto-block or wait for a human — they already said which.
5. Hold goes to the risk owner, not a random engineer override.
6. A prod miss becomes a golden-set case before the next ship.

That’s it: pin candidate → eval vs baseline → pass/block/hold → learn. Shipping on vibes stays out.

Source: Sources and Full Design. The Deep Dive expands the gate, graders and incident; the Cheat Sheet is for rehearsal.

Build a shared platform for 30 AI applications that makes a controlled pass, block or hold release decision against a pinned production baseline, with evidence for quality, safety, latency and cost. A dashboard alone does not own that decision.

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Pin candidate artifacts — prompt, model, retrieval, tool, policy.
2. Select versioned suites and rubrics.
3. Run deterministic and model-based checks.
4. Compare to the pinned production baseline.
5. Gate critical slices and four outcome dimensions (quality, safety, latency, cost).
6. Route uncertain or disputed cases to humans.
7. Record decisions, owners, and overrides.
8. Sample production failures into a reviewed test pipeline.

Non-functional requirements — how well / under what constraints

Do not assume one universal rubric. v1 does not auto-tune or broadly share tenant datasets.

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, pin the candidate, run the suite vs live baseline, then pass, block, or hold. Non-functionally, about 20 minutes, reproducible, fail closed, and humans own overrides.”

The LLM grader provides one calibrated score, while deterministic checks, policy code and human authority control the release. Candidate applications and evaluation inputs run in a sandbox; the decision plane reads only validated completed results.

Step-by-step architecture

1. A release request pins the full candidate artifact bundle and the current production baseline, including model, prompt, retrieval, tool and policy versions.
2. The scheduler chooses versioned suites and app-specific rubrics, applies tenant quotas, and sends cases to sandboxed runners with timeouts and limited egress/secrets.
3. Runners execute the whole application workflow; deterministic checks score schemas, policy rules and measurable outcomes, while a versioned LLM grader evaluates rubric dimensions that require judgement.
4. Validated results are written immutably with run, case, grader and rubric hashes; incomplete runs cannot masquerade as complete evidence.
5. The policy engine computes candidate-minus-baseline deltas for quality, safety, p95 latency and cost, checks critical slices independently, and issues pass, block or hold.
6. Humans review disputed or statistically thin cases and record overrides with reasons. Production signals are redacted and labelled before joining future suites.

Quality uses task-specific outcomes and a confidence-aware lower bound on candidate delta. Safety has an independent floor: better average quality cannot compensate for a safety regression. Latency and cost have agreed envelopes. Missing evidence blocks; thin evidence holds. A critical slice must pass on its own. For nondeterministic outputs, use repeated trials and outcome equivalence rather than exact text match.

The source's sales-copilot incident shows why: the release suite passed 96.7% of 120 cases, but only six covered regulated claims, with an 83.3% pass on that slice. Production unsupported claims reached 8.7% versus 1.2% baseline after a model route change and a policy evaluator changed from block to warn-only. Roll back the route, restore blocking, review affected drafts, and add claim-level cases for security status, ROI (return on investment) and customer-logo permission. Version the policy mode so it cannot bypass the gate.

An LLM grader can reward verbosity over correctness. Version its model and rubric, calibrate it against humans using concise-correct and verbose-wrong examples, inspect disagreements, and keep it advisory until it earns trust. Use hidden holdouts, incident cases and critical-slice minimum coverage to prevent benchmark overfit. Online feedback discovers new failure modes; redaction and labelling make them usable offline.

To control cost, run smoke samples on ordinary changes, broader regression at release, and deep audits for high-risk changes. Route easy cases to a cheaper judge, escalate disagreements to a strong judge/human, and cache only unchanged artifact–case–grader pairs. Do not sample away critical slices. Measure cost per release, decision time, grader agreement, escape rate, flaky-case rate and override rate.

Roll out on one critical app with an advisory recommendation; calibrate graders against human labels; add latency and cost gates; then expand through a shared schema with app-specific rubrics. A grader timeout or unavailable evidence store blocks release. A delayed dashboard can degrade if the durable decision record remains intact. Roll back on a bad canary, unchecked severe incident or untrustworthy scoring.

“I would build a release decision system, not just an evaluation dashboard. I pin the candidate and production baseline, run versioned cases in an isolated worker pool, score deterministic checks and calibrated LLM rubrics, and store immutable evidence. A policy engine compares quality, safety, latency and cost separately, with critical slices passing independently. Missing results block, thin samples hold, and humans handle disputes and overrides. I would start advisory on one critical app, validate the grader against humans, then harden the gate and expand. Post-release incidents become labelled regression cases, and the gate's own cost and escape rate are measured.”

Read with the unchanged source. The Main guide gives the interview flow; the Cheat Sheet is the recall card.

An engineer asks whether a change worsened behaviour; an evaluator asks what to score; a safety lead asks if it is safe; a release manager needs an auditable decision. The shared platform supplies artifact lineage, execution and evidence, while each app owns its rubric and thresholds. A pass needs a named policy, a pinned baseline and a complete validated run. Block means a known limit failed or required evidence is unavailable; hold means the evidence is too thin or disputed to decide. Every override has an authorised owner and reason.

Store immutable EvalArtifact, EvalCase, EvalRun and EvalResult records. Artifact hashes cover model, prompt, retrieval, tools and policy. Case and rubric versions prevent silent benchmark drift. Results include the grader version, raw trace reference, score, deterministic outcomes and tenant scope. API writes are idempotent; conditional updates protect reviews and decisions. A superseding record is safer than mutating an old result.

Thirty apps × twenty candidates/day × five thousand cases means three million executions/day and six to fifteen million model/evaluator calls at two to five calls per case. Concentration into six hours implies 278–694 calls/s. With 2.5 calls/s per worker, 112–278 workers are a rough pre-headroom range. Use suite tiers, per-tenant quotas and elastic workers. Cache immutable fixtures, but do not cache away a stochastic judgement the release needs to re-measure.

Pin a production baseline and compare deltas, not only absolute scores. Quality should represent task completion and can use a bootstrapped lower confidence bound for the delta. Safety has a separate floor; p95 latency and cost each have their own limits. Apply gates per critical slice, since an average hides rare harm. Golden sets should include representative flows, known hard cases, prior incidents, policy-sensitive cases, retrieval misses, malformed tool calls and convincing-but-wrong responses.

LLM-as-judge is itself release logic. Pin its version and rubric hash, calibrate against human labels and audit disagreement, especially concise-correct versus verbose-wrong outputs. Use repeated trials and outcome equivalence for nondeterministic application behaviour. Keep a hidden holdout and refresh suites when they stop distinguishing candidates. Online thumbs-down and corrections are leads for new cases, not labels by themselves.

A sales copilot changed its model route while a claim policy changed from block to warn-only. Its 120-case offline suite passed at 96.7%, but only six cases tested regulated claims; that slice passed at 83.3%. Online unsupported claims rose to 8.7% from 1.2%, including unapproved security-certification and internal-pilot ROI (return on investment) claims. The failure is in release coverage and policy control, not simply “the model is creative.”

Contain by rolling back the route, restoring external-claim blocking and reviewing affected drafts. Prevent recurrence with claim-level labelled cases, minimum critical-slice size, zero critical failures for regulated claims and a policy-mode artifact that must pass the same gate. The neighboring “model got worse after upgrade” prompt is this gate run retroactively: replay a fixed set, segment failures, compare retrieval/prompt/schema, human-review the hard cases, then pin or roll back. “No evaluation strategy” is this gate built from scratch with representative data and human labels.

Candidate prompts, retrieved pages and uploaded cases may contain instructions aimed at the grader. Isolate runners with limited secrets, timeout and egress; sanitize data; pin inputs; store suspicious traces for investigation. Separate tenants and app reviewers. Redact or synthesize sensitive online cases before storage. Keep gate suites away from candidate prompts and retrieval indexes to reduce leakage.

Grader timeout and evidence-store outage block release. A sample too small to support a conclusion holds. Dashboards can lag if an already durable decision record remains available. Retry idempotent writes and move persistent dependency failures to a dead-letter/human-review path. Log policy edits, grader versions, redactions, retries and overrides.

Start with one critical app, human labels and an advisory gate. Promote well-calibrated dimensions to hard gates, then add latency and cost and expand with shared schemas plus app-specific rubrics. Track evaluation coverage, regression escape, human-grader agreement, p95 decision time, cost/run, flaky cases and override reasons. A severe escape or bad canary triggers rollback and a new incident case.

The eval bill can be a major inference bill. Run cheap smoke samples on ordinary changes, full release-relevant suites at gate time and deep audit on high-risk changes. Use a cheap judge for easy cases and a strong judge/human on failures and disagreements; cache only truly unchanged pairs. Keep all critical-slice cases in the release suite. Human review load often dominates marginal cost, so better calibration can lower spend without weakening the gate.

Main · Deep Dive · Unchanged source

Critical apps and slices? Release window? Labels/rubric owner? Sensitive traces? Block versus hold thresholds? Override authority?

Release request → pinned artifact + baseline → versioned suites → sandbox runners → deterministic checks + calibrated LLM grader → validated results → policy delta → pass / block / hold → online feedback

LLM (large language model) graders score; deterministic policy and authorised humans decide.

30 apps × 20 candidates/day × 5K cases = 3M executions/day. At 2–5 calls: 6–15M calls/day; six-hour peak ≈ 278–694 calls/s. Tier suites and quota the shared worker pool.

• Pin model, prompt, retrieval, tool, policy, dataset and rubric versions.
• Compare candidate to current production baseline across quality, safety, p95 latency and cost.
• Safety floor and critical slices pass independently of the aggregate.
• Missing evidence blocks; thin or disputed evidence holds.
• Calibrate the versioned LLM grader against humans before making it a hard gate.

Sales copilot: aggregate 96.7% pass on 120 cases; just six regulated-claim cases at 83.3%; unsupported claims 8.7% online versus 1.2% baseline; claim policy changed to warn-only. Roll back route, restore blocking, add claim-level critical-slice coverage.

Sample easy cases for smoke, use cheap judge then strong-judge/human escalation, run full critical suites at release, and cache only unchanged pairs. Do not thin the highest-risk slice.

60-second close: “I build a decision record around pinned evidence. The runner executes the application in a sandbox, calibrated graders and deterministic checks score it, and policy compares four independent dimensions with critical-slice gates. The platform holds or blocks when evidence is untrustworthy, then learns from production escapes.”

A gate strict enough to stop a bad release is slow enough that teams route around it.

An evaluation platform is not a dashboard of scores. It is a controlled decision about whether one change is safe to ship. This page consolidates the group's prompts into one read for the day before. The anchor design comes first. The neighbouring prompts, the incident and the cost pivot then come as deltas on the same gate.

Sections and tables marked (own construction) were built for this page from the sources' arguments. They are not in the sources verbatim.

The customer opens with one sentence. "We need one platform that lets 30 AI applications decide whether prompt, model, retrieval, or tool changes are safe to release." The first job is to restate it without smuggling in a design.

A dashboard answers "what are the scores?" Nobody is obliged to act on that. The platform exists to answer "can this change ship?", and that question has an owner, a threshold and a record. So the goal is not the platform. The goal is making quality, safety, latency and cost regressions visible before and after deployment.

Four stakeholders hire the platform, and each asks a different question. Failing to separate them produces a system that is technically impressive but operationally unusable.

Ask the questions that change the design most. The six-question tree is worth reciting in order.

Two more come from the answer key: what counts as a release, and who may override a failed gate and how that is recorded. An audited override forces immutable records and role-based controls. Shared evaluation data forces tenancy boundaries.

When the interviewer stays vague, protect the riskiest assumption. Assume the gate should block uncertain, high-impact changes and escalate them to a human. False passes are more dangerous than false blocks for high-impact applications.

"I'd frame this as a controlled decision process rather than a dashboard: compare a candidate against a pinned baseline, detect quality, safety, latency and cost regressions, and record the evidence behind the decision."

A requirement that cannot fail a test is a preference. "Fast gate" is a preference. "p95 run duration under 20 minutes in the release window" is a constraint, and only the constraint shapes the architecture.

The functional requirements are what make this a gate rather than a dashboard. Split them with MoSCoW so the first release is the smallest trustworthy path (the split is own construction; the items are the sources').

The must-haves are six. A release owner submits a candidate change, and the platform compares it to a pinned baseline and returns pass, block or hold with evidence. Datasets, prompts, models, tools and graders are versioned, because versioning is the only way to reproduce a decision later. Both deterministic and probabilistic evaluations run, since some checks must be stable and others depend on rubric judgement. Every candidate is compared against a pinned production baseline, never reported as an absolute score. Release policies are enforced per application, which is what turns analysis into action. Human review is supported, and reviewer disagreement is recorded as a first-class signal rather than discarded as noise.

The should-haves follow once the gate is trusted. Privacy-safe production failures are sampled back into future suites so the test set evolves with the product. Online sampling watches live or shadow traffic after rollout. Latency and cost gates sit beside quality and safety.

The first version deliberately excludes six things. Automated fine-tuning or prompt optimisation. A full legal or compliance sign-off workflow. Open-ended notebook experimentation, which weakens reproducibility. Every metric for every application on day one. Cross-tenant sharing of datasets or graders. Custom visualisation of every trace field.

Every requirement then needs an owner in the architecture. If a requirement has no component owner, the design is incomplete.

The whiteboard version is a queue, a pool of workers, a results database and a dashboard. It works at average load and fails when the release train tightens. Sizing is what turns it into a design.

The headline workload is not "30 apps". It is 30 applications × 20 candidate releases a day = 600 candidate evaluations a day. With 5,000 test cases each, that is 3,000,000 test-case executions a day. I'd assume each case needs 2–3 model or grader calls at minimum and 3–5 more realistically. So the platform makes roughly 6,000,000–15,000,000 model or evaluator calls a day.

The worker figure assumes one call every 400 ms, about 2.5 calls a second per worker at full utilisation. Real deployments add headroom for retries, tail latency and bursts.

Average load answers "can we survive the day?" Peak load answers "can we survive the release window?" Those are different questions. A modest daily average still overwhelms the system if many applications submit in the same hour. So the pool is shared and elastic, with per-tenant quotas and admission control, so one application cannot starve the rest.

In this design, evaluator concurrency is the limit at current scale. At 10x, I expect storage, retention and human-review routing to dominate.

Cache immutable inputs, never the behaviour being measured. Test definitions, baseline outputs, corpus snapshots, rubric and prompt templates and precomputed embeddings are safe to cache when versioned. A cached stochastic judgement would hide exactly the variance the gate is meant to measure.

"I cache immutable inputs and expensive shared fixtures, not the stochastic judgement itself."

The organising rule is a split. The control plane decides what to evaluate, and the data plane executes it. A slow or failing runner therefore cannot make policy nondeterministic. Candidate selection, enforcement and escalation stay deterministic however noisy the execution is.

The planes and lanes as one picture (own construction from the sources' component map):

 ╔════════════════════════ CONTROL PLANE (decides what to evaluate) ════════════════════════╗
 ║                                                                                          ║
 ║  Release request ──► Artifact Registry ──► Dataset Store ──► Evaluation Scheduler         ║
 ║  (candidate +        (immutable prompt,    (suites, labels,   (fan-out, per-tenant        ║
 ║   baseline ref)       model, retrieval,     rubrics, case      quotas, admission control, ║
 ║                       tools, seed, hash)    sensitivity)       backpressure)              ║
 ║                                                                     │                     ║
 ║  Human Review UI ◄──► Release Policy Engine ◄── Results Warehouse   │                     ║
 ║  (adjudication,       (Δ vs thresholds;         (validated runs     │                     ║
 ║   overrides =          pass / block / hold;      only; audit)       │                     ║
 ║   policy input)        fails closed)                 ▲              │                     ║
 ╚══════════════════════════════════════════════════════╪══════════════╪═════════════════════╝
                                                        │              ▼
 ╔════════════════════ DATA PLANE (executes; untrusted boundary) ═══════════════════════════╗
 ║                                                                                          ║
 ║  Sandboxed Runner ──► app / model / retrieval / tool calls ──► Grader Service ───────────╫──► warehouse
 ║  (timeouts, egress limits,                                     (task success, safety,     ║
 ║   secrets scoping, concurrency caps)                            latency, cost; escalates) ║
 ╚══════════════════════════════════════════════════════════════════════════════════════════╝
                                                        ▲
 Online Sampling Pipeline (live / shadow traffic, redacted) ── new failure cases ──► Dataset Store

The same flow with the decision branches, from the answer key and the tutorial:

Read the components in the order a release travels through them. The first four columns are the tutorial's; the failure column is own construction, drawn from the tutorial's failure-policy table.

Point at three boundaries while the diagram is up. The registry and dataset store are strongly consistent, so a run always points at fixed inputs. The warehouse can be eventually consistent for analytics, but the final gate reads only completed, validated records. And the runner is the trust boundary: prompts, retrieved documents and uploaded cases are adversarial until proven otherwise.

The happy path runs in ten steps. A release engineer submits an immutable candidate to the registry. The registry pins the exact prompt, model, retrieval, tool, seed and baseline identifiers. The scheduler selects the application's suite, partitions work by tenant and enforces concurrency limits. The runner executes cases in isolation with controlled seeds. The grader scores correctness, safety, latency and cost. The warehouse stores run-level and aggregate results. The policy engine compares candidate metrics and confidence intervals with the pinned baseline. Ambiguous cases route to human review. The decision is issued as pass, block or hold, with reasons attached to the release record.

Each step answers a different customer question. The registry answers "what exactly changed?" The suite answers "what did we test?" The runner answers "under what conditions?" The grader answers "how did it perform?" The policy engine answers "can we ship?"

The MVP (minimum viable product) is the smallest trustworthy path from registration to decision: registry, dataset store, async scheduler, sandboxed runner, grader, warehouse, and a policy engine that can pass or block. Richer human workflow, adaptive suite selection, online sampling, multi-region execution and score calibration come later.

A score means nothing without the thing it is compared against. The platform does not produce one "good" number. It says whether a candidate is better, worse or indistinguishable from a pinned baseline.

The comparison is Δ = Metric_candidate − Metric_baseline. The metric can be quality, safety violation rate, task success, refusal accuracy, latency or cost, depending on the gate. A shift is useful only when the baseline is fixed, versioned and comparable. A two-tenths-of-a-point gain on a noisy rubric may not justify release. Weigh variance, confidence bands and operational risk, not just the sign of the delta.

Say the derivation on the whiteboard in one breath. For each test case, compare candidate output to a pinned baseline output or rubric score. Compute Δ for the metric that matters to the application. Aggregate by application, by release and across the fleet. If the distribution of Δ crosses the allowed regression threshold, the gate fails or routes to human review.

Four records carry the whole design. Their lifecycles are what make a decision reconstructable a year later.

Ownership is split on purpose. The evaluation program owns EvalCase, the control plane owns EvalRun, and the execution path writes EvalResult for reviewers and auditors to read. That split keeps the gate from silently rewriting evidence after the fact. The decision record and the redacted score summary also outlive the raw trace. A shorter trace-retention policy therefore never destroys auditability.

The API surface stays at four endpoints: POST /v1/eval-runs, GET /v1/eval-runs/{id}, POST /v1/reviews and POST /v1/release-decisions. Three ideas do most of the reliability work at that boundary. An idempotency key on every retryable write returns the original run_123 on a retry instead of creating run_124. Explicit contract versions keep a run created under policy v7 from being read under v8 semantics. An ETag on reviews and decisions means two actors finalising the same run cannot both win; the second gets a 412 Precondition Failed and must re-read. An override is always a new superseding record, never an invisible overwrite.

One opaque score hides the trade the customer cares about. A candidate can raise quality while lowering safety, doubling latency or blowing the budget. The gate therefore checks each dimension separately and allows a release only when all of them pass.

The smallest code path that proves the design is the decision function. It validates typed input at the boundary, injects the statistical comparison as a seam, and returns every check with the verdict.

def release_decision(candidate: Mapping[str, Any], baseline: Mapping[str, Any], policy: Mapping[str, Any], comparator: Comparator) -> Decision:
    candidate_metrics = _parse_candidate(candidate)
    _ = _parse_candidate(baseline)
    policy_obj = _parse_policy(policy)
    delta = comparator.compare(candidate, baseline, bootstrap_samples=2000)
    checks = {
        "quality": delta.quality_lower_bound >= -policy_obj.max_quality_drop,
        "safety": candidate_metrics.safety_rate >= policy_obj.min_safety_rate,
        "latency": candidate_metrics.p95_latency_ms <= policy_obj.max_p95_ms,
        "cost": candidate_metrics.cost_per_case <= policy_obj.max_cost,
    }
    return Decision(allow=all(checks.values()), checks=checks, delta=delta)

Quality is judged on the lower bound of a bootstrapped delta, not the point estimate. That single choice is what "statistically honest" means in code. A malformed metric, such as "safety_rate": "high", raises ValidationError before any policy logic runs. The tutorial's test policy uses max_quality_drop 0.01, min_safety_rate 0.95, max_p95_ms 250 and max_cost 0.20 under policy-7.

One invariant deserves its own test. A quality gain cannot hide a safety regression.

baseline = Score(quality=0.90, safety_rate=0.999)

def test_quality_win_cannot_hide_safety_regression():
    candidate = scores(quality=0.95, safety_rate=0.97)
    assert not release_decision(candidate, baseline, policy(min_safety_rate=0.995)).allow

A more persuasive release cannot ship if it falls below the safety floor. That is the production judgement the round is testing.

Missing evidence and thin evidence are different outcomes. Missing evidence, such as a grader timeout or a dead evidence store, blocks. Thin evidence, a sample too small to decide, holds as "insufficient evidence" rather than approving. Neither ever turns into a pass by default.

A model grader is part of the release logic, so it needs evaluation like any other component. An uncalibrated grader can make a risky release look safe while still producing a clean audit trail.

The named failure is a grader that rewards verbosity over correctness. Long, confident, wrong answers receive high marks. The runner completes normally, so nothing looks broken. The fix is a drill in four moves.

Validate a grader against human judgements on a representative sample, not the easy cases. Measure agreement on the cases that drive release decisions, then inspect the disagreements. Some reveal rubric gaps; others expose ambiguity in the task itself. Re-score a held-out set with humans periodically, because a grader drifts after rubric changes or model upgrades even when the platform is stable.

Trust is earned in levels. Early on the grader is advisory: it blocks only low-risk deploys or requires human approval on uncertain cases. As agreement stabilises, high-confidence checks move to automatic gating. The design does not pretend the grader is perfect. It builds a path to trust.

All four pairs share one shape: one tool for breadth, another for the decision that matters. Reuse that shape live when a specific answer does not come.

A high average can hide a failing slice. A change may improve mean quality while hurting the rare, high-risk cases the business actually fears. So the release gate reads per-slice results, and a critical slice must pass on its own.

The golden set is small enough to maintain carefully and broad enough to reflect real release risk. It holds representative happy paths, known hard cases, policy-sensitive cases, and examples that previously caused incidents or near misses. It also holds the edge cases where the old system failed quietly: malformed tool inputs, ambiguous queries, retrieval misses, and verbose answers that looked strong but were wrong. A golden set of obvious examples creates false confidence.

Nondeterministic outputs are compared as distributions, not single strings. Run repeated trials where needed, seed what can be seeded, and compare aggregated rubric scores or pass rates. For many dimensions the unit is outcome equivalence rather than text match. Did the answer satisfy the task, respect policy and avoid harm? Stratify by scenario type so a regression in a small slice stays visible.

Online feedback is a discovery channel, not a scorecard. Thumbs-down, abandonment, escalation, correction and repeated retries are candidates for new cases. They enter the golden set only after redaction and trustworthy labelling. The tutorial's line: "Online feedback is a discovery channel for new failure modes; offline evaluation is where I make those failure modes measurable and releasable."

The suite itself goes stale. The benchmark stops distinguishing candidates, or incidents cluster where it never looks. Mark it non-authoritative for gating, refresh it with new workflows, and re-baseline thresholds. A candidate can also overfit the benchmark: large offline gains with flat real-world behaviour. Hidden holdout cases, adversarial variants and never exposing the gate suite to the candidate's prompt, retrieval index or logs are the defence.

Scenario incident #91 is section 8 failing in production. It proves an aggregate pass rate can hide a critical slice.

A sales copilot drafts account-specific outreach emails from CRM (customer relationship management) notes, product documentation and approved messaging. The model route was upgraded to improve fluency and personalisation. After the upgrade, emails sounded polished but included unsupported claims. Reps saw drafts claiming "SOC 2 Type II renewal completed in June 2026" and "average 34% support cost reduction." The SOC 2 claim was not yet approved for external use. The ROI (return on investment) claim came only from an internal pilot.

The workflow was CRM account context → approved claims retriever → prompt template → LLM (large language model) gateway → policy evaluator → offline eval suite → online quality monitor. The release gate checked grammar, tone and general groundedness on 120 examples. Only 6 of them were about regulated claims.

The telemetry tells the whole story in three lines.

Two changes landed together on 2026-07-08 at 09:00. The router moved drafting from llm_standard_v2 to llm_premium_v3 after an offline eval showed better fluency. The policy evaluator also moved from block mode to warn-only during the experiment, to reduce false positives.

The root cause is an evaluation failure, not a model outage. The suite over-measured writing quality and under-measured unsupported business claims. The upgrade increased persuasive extrapolation, and warn-only mode let the risky drafts through.

Debug it by comparing offline pass rates with online failure modes. Inspect failing drafts, retrieve the claims used, verify the claims' approval metadata, and segment failures by model route and policy mode. Then name the gate flaw aloud: a high aggregate pass rate can hide failure on a small but critical slice.

The risk is not stylistic. Unsupported sales claims create legal exposure, customer trust damage and compliance issues. A careless or malicious rep could prompt the assistant to exaggerate ROI or security status and send the draft externally.

Map it back onto the anchor (own construction). The anchor's gate would have caught this in three places. A per-application threshold on the regulated-claim slice fails at 83.3%. The coverage metric flags a critical slice with 6 cases as below minimum sample, so the gate holds. And a policy change from block to warn-only is itself a versioned artifact, so it goes through the gate instead of around it.

Weak: "The model is too creative. I would make the prompt stricter and ask sales reps to review the output."

Strong: "The failure is an eval and launch-gate problem. The suite passed at 96.7%, but it had only six regulated-claim cases and the online unsupported-claim rate jumped to 8.7%. I would roll back the model route, restore blocking for external claims, add claim-level evals for security, ROI, and customer-logo claims, and require critical slices to pass independently before release."

Both OpenAI-style practice prompts are the anchor seen from a different moment. #52 is the gate applied after the fact. #55 is the gate built from nothing.

#52, the model got worse after an upgrade. A customer says quality declined after switching to a newer model. Treat it as the release decision that never happened, and run it retroactively. The source's diagnostic plan has eleven steps. Collect before-and-after production examples. Replay a fixed evaluation set. Segment by task, language, customer and input size. Compare structured-output compliance. Check prompt compatibility and retrieval changes. Compare latency and token usage. Run human evaluation, and calibrate any LLM-as-judge first. Roll back or pin the model version. A/B test future upgrades.

Map those steps onto the anchor (own construction). Pinning the version is the artifact registry. Replaying a fixed set against the old model is Δ against a pinned baseline. Segmenting is slice-based gating. Calibrating the judge is section 7. A/B testing future upgrades is the online sampling pipeline. The answer ends with the permanent fix: model upgrades become candidates that pass through the gate.

The follow-up is "the offline benchmark improved, but the customer still says quality declined." The source gives seven explanations, and each maps to a gate weakness.

#55, build an evaluation strategy. A customer wants to deploy an assistant but has no reliable way to measure quality. The source lists fourteen items to cover: a representative production dataset, golden examples, task-specific rubrics, ground-truth labels, automated graders, human evaluation, and LLM-as-judge calibration. Then safety and refusal tests, regression tests, online metrics, user feedback, slice-based analysis, statistical confidence, and continuous evaluation in CI/CD (continuous integration and continuous delivery).

The source's key distinction is one sentence. "Model quality is not the same as application quality." Evaluate the complete workflow: retrieval, tools, business rules, user interaction, operational performance and business outcomes. That sentence is also the anchor's four-dimension gate in miniature.

The follow-up is "what if there is no labelled data?" Sample production traffic. Use expert labelling. Apply weak supervision. Use synthetic examples carefully. Measure grader agreement. Send the difficult cases to human review. The anchor's rollout says the same thing in order: one application, humans label a sample, calibrate the grader, and keep the gate advisory until agreement holds.

Every external dependency and every irreversible action needs an explicit failure policy. Otherwise the default becomes "guess", and a guessing gate approves things.

Time out grader and retrieval calls. Retry only operations that are safe to repeat. Put repeated failures behind a circuit breaker so the platform stops hammering a broken dependency. Escalate any unresolved release decision to a human rather than guessing.

Four controls deserve their own threat model. Separate evaluator data by application and tenant, including reviewer access. Redact or synthesise sensitive cases at ingestion, because online samples carry PII (personally identifiable information), secrets and proprietary prompts. Version model-based graders like code, or the platform can change verdicts without changing the application under test. Prevent test-set leakage into prompts, retrieval, hidden notes or cached artifacts, since leakage manufactures false confidence.

Inputs are adversarial until proven otherwise. A candidate prompt can carry an injection, a retrieved document can be poisoned, and an uploaded case can hide instructions that steer the grader. Sanitise before evaluation, isolate from the grader context, log the artifact hash and tenant scope, and keep a copy of suspicious evidence. Contain at the smallest scope: one tenant, one application, one region, one dependency chain.

Before launch the team must be able to show who changed the policy and which grader version was used. It must also show which tenant and application were evaluated, which cases were redacted or synthesised, which external dependencies were called, and what happened on each retry.

A gate nobody trusts gets bypassed, so adoption is part of the design. The rollout proves trust on one application before generalising.

The answer key's calendar: week 0–1 names scope, evidence, override authority and thresholds. Week 1–2 proves one application. Week 2–3 calibrates graders, including concise-correct and verbose-wrong examples. Week 3–4 holds the gate advisory. Week 5 adds latency and cost gates. Week 6–8 expands with a shared schema. After the pilot, widen only while the metrics hold, and retire any suite that stops discriminating.

The shared schema becomes core product: run metadata and model, prompt, dataset and rubric versions, plus outcome labels, trace references and the release decision. Rubrics stay configuration, because a support bot and a coding assistant do not need identical criteria. Adapters handle per-app formats; the shared service owns orchestration, scoring and audit.

A concrete launch review reads like this. Coverage 97%. One severe regression escaped in the last 20 releases. Agreement 89%, acceptable only if the disagreement set is explainable. p95 run time 14 minutes against the 20-minute window. Cost $18 per release evaluation. Flaky rate 1.5%, concentrated in one ambiguous rubric dimension. Four of 5 known regression classes caught before release, 1 by canary, with a postmortem required on the missed class.

Rollback triggers are as explicit as go/no-go gates. They are a bad canary signal, a spike in flaky cases, a post-release incident in an unchecked failure class, and a failed dependency that prevents trustworthy scoring.

Spend time in proportion to risk, not diagram size. The evaluation-design block is the depth budget, and interviewers probe it hardest.

Open with the business result and the hidden constraint. The applications are not homogeneous: some are high-volume support assistants, some low-volume workflows, some narrow safety cases. So do not force one universal metric. Build a shared evaluation spine with app-specific rubrics and gates on top.

The two-minute spoken answer, from the answer key:

I would not start with the model. The ask is one platform so 30 applications can decide whether a change is safe to release, but the platform is not the goal — making quality, safety, latency, and cost regressions visible before and after deployment is. So I would restate it as a release-gating system that compares a candidate against the current baseline and records the evidence behind the decision, then separate the four stakeholders. An engineer asking "did my change worsen behaviour," an evaluator asking "what should I score," a safety lead asking "is this safe enough," and a release manager asking "can I approve this with evidence" are different jobs. Architecturally, the control plane decides what to evaluate and the data plane executes it inside a sandbox I treat as hostile-adjacent. An artifact registry pins the candidate so it cannot drift mid-run, a dataset store owns suites and rubrics, a scheduler fans out work with per-tenant quotas, a grader scores across all four dimensions, and a policy engine reads only completed validated runs to compute the delta against thresholds. Missing evidence fails closed, and a sample too small to support a decision holds rather than approves. The named failure is a grader that rewards verbosity over correctness, so graders are versioned, rubric hashes stored, and a calibration suite includes concise-correct and verbose-wrong cases. I would prove it on one critical application, calibrate against human labels, add latency and cost gates, then expand. The gate is only useful if teams trust it enough not to route around it.

The lines that carry the round (own construction from the sources' arguments):

1. "A dashboard reports scores. A gate makes a decision, with an owner, a threshold and a record."
2. "Thirty apps, twenty releases, five thousand cases: three million executions a day."
3. "Δ against a pinned baseline, never an absolute score."
4. "Missing evidence blocks. Thin evidence holds. Nothing passes by default."
5. "Quality cannot buy its way past the safety floor."
6. "The grader is part of the release logic, so it is versioned and calibrated like code."
7. "An aggregate pass rate can hide a failing slice. Critical slices pass on their own."
8. "Model quality is not the same as application quality."

The follow-ups arrive in a predictable order.

Repair the five weak-answer traps on the spot. "We'll track accuracy and latency" is too generic; name task-specific rubrics and release decisions. "The model grader decides everything" is too automated; add human calibration and audits. "Full suite on every commit" is too big too soon; tier it. "Pass or fail" is too binary; add thresholds, confidence bands and escalation paths. "We only care about pre-release quality" is too narrow; add online feedback and post-release drift detection.

The pivot after a good design is #120: "costs are fine but the eval bill tripled." The answer starts from one fact. LLM-as-judge is inference too. Evaluation is the eighth cost driver, the one that runs off the request path where no interactive user ever sees it.

The source answer is four moves. Use a stratified sample instead of the full set. Use a cheaper judge with a periodic strong-judge audit. Run deltas only on changed prompts or models. Cache judge calls on unchanged pairs.

The self-drill card turns the same idea into the full answer shape.

The risk named alongside the fix is real. Weak eval coverage misses quality regressions, and incident #91 is what that looks like. So cut cost by sampling the easy majority, never by thinning the critical slices. The regulated-claim slice runs in full on every release; the smoke tier samples the rest (own construction).

Two levers connect the cost answer back to the design. The anchor's gap note calls human review almost certainly the most expensive unit cost. Better grader agreement therefore lowers cost per run by shrinking the human-review tail. And the judge cache is safe only on unchanged pairs. It is keyed on prompt version, model version and case, and never replaces a judgement the gate is meant to re-measure.

Every strong cost answer runs through four verbs in order. Measure eval spend per release and per tier first. Route cases to a cheap judge and escalate only disagreements. Bound the full suite to the release gate. Cache safely, keyed on every version that could change the verdict.

Every level can draw the pass, block or hold gate; the level shows in who trusts it, who may override it and whether teams can afford it.

• The ask is a controlled release decision for four stakeholders with four different questions, not a dashboard of scores.
• Requirements are stated so a test can fail them, with every one owned by a component and six things fenced out of the first version.
• Thirty apps, twenty releases and five thousand cases make three million executions a day, and the peak release window, not the daily average, sizes the pool.
• The control plane decides and the data plane executes, so a noisy runner can never make the policy nondeterministic.
• Every input is pinned and every result is a delta against a pinned baseline, with records that are immutable, monotonic and write-once.
• The decision checks quality, safety, latency and cost separately, judges quality on a lower bound, and never lets quality hide a safety drop.
• A grader is release logic: versioned, calibrated against humans, advisory until agreement holds.
• Critical slices pass on their own, because an aggregate pass rate can hide the failure the business fears.
• Incident #91 passed at 96.7% with six regulated-claim cases while production ran 8.7% unsupported claims.
• A post-upgrade regression is the gate run retroactively, and an evaluation strategy is the gate built from nothing.
• Missing evidence blocks, thin evidence holds, and dashboards degrade while the gate keeps its record.
• Trust is earned on one application first, then graders, then latency and cost gates, then a shared schema.
• The fifty minutes go mostly to evaluation design, closed by the two-minute answer.
• The eval bill falls by tiering judges and sampling the easy majority, never by thinning the critical slices.

1. Why is a dashboard the weak answer? It reports scores nobody must act on. It never pins a baseline, defines what blocks a release, or says who may override.
2. Redo the sizing aloud. 30 × 20 = 600 evaluations a day; × 5,000 = 3,000,000 executions; × 2–5 calls = 6–15 million calls; 69–174 calls/sec over 24 hours or 278–694 in a 6-hour window; 28–70 or 112–278 workers at 2.5 calls/sec.
3. Why separate the control plane from the data plane? So selection, enforcement and escalation stay deterministic when runners are slow, failing or fed adversarial input.
4. What is the difference between missing and insufficient evidence? Missing evidence, such as a grader timeout or a down evidence store, blocks. Insufficient evidence, a sample too small to decide, holds rather than approves.
5. Why is quality judged on the lower bound of the delta? A small gain on a noisy rubric may be noise; the lower bound of a bootstrapped delta keeps the gate statistically honest.
6. How is the verbosity-rewarding grader detected and prevented? Detect a score shift toward long, low-alignment answers against a baseline grader. Prevent it by versioning the grader, storing rubric hashes and calibrating on concise-correct and verbose-wrong cases.
7. In incident #91, what three facts show the gate failed rather than the model? A 96.7% pass on 120 cases with only 6 regulated-claim cases, an 83.3% pass on that slice, and warn-only policy mode. Production ran 8.7% unsupported claims against a 1.2% baseline.
8. The offline benchmark improved but the customer says quality fell. Name three explanations. Unrepresentative benchmark, a regressed customer segment, and worse edge cases hidden by a better average. Also possible: retrieval changed, the UX slowed, or structured output got less reliable.
9. What if there is no labelled data? Sample production traffic, use expert labelling and weak supervision, use synthetic examples carefully, measure grader agreement, and send hard cases to humans.
10. What is the sixty-second answer to "the eval bill tripled"? A gate nobody can afford gets bypassed. Tier the judges, sample on every change, run the full suite only where a release decision is made, and cache judge calls on unchanged pairs.

Fact-check sources (checked 27 Sep 2026)

• RFC 9110 HTTP Semantics: 13.1.1 If-Match and 15.5.13 412 — a failed If-Match returns 412 Precondition Failed, not 409
• Zheng et al. 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — LLM judges show verbosity bias
• AICPA & CIMA: SOC 2 — SOC 2 control mapping
• GDPR Article 17: Right to erasure — deletion rights that trace retention must respect
• V2 chapter 9 tutorial, "My Perspective on the Gaps" — the gap note calling human review the most expensive unit cost