Observability and Production Diagnosis

Diagnosis process is: a user says “it’s slow and sometimes wrong.” You need the hop that hurt — without turning traces into a second copy of their tickets.

G14 covers one slice: one correlation id through the AI path, redacted spans, tenant-scoped debug. Not a new product feature.

End to end, as Acme’s German refund flow:

1. Request gets a correlation id at auth.
2. Each hop records time and outcome — retrieval, model, tool — not the raw prompt by default.
3. A classifier redacts before storage. “Retrieval 1.2s, 3 docs” yes; full SSN (social security number) no.
4. Support sees Acme only. L1 cannot open Globex tickets to “help debug.”
5. You see first token vs last sentence, empty retrieval vs made-up answer vs tool 500.
6. Errors and outliers stay; most happy traffic is sampled.

That’s it: trace the path → hide the secrets → find the hop. Telemetry must not stall the user.

Full source: Sources and Full Design, especially §§1–9 for the anchor architecture and §§10–15 for incidents and variants. Use the Deep Dive for diagnostic detail and the Cheat Sheet for rehearsal.

The anchor is a customer-facing AI application that is “slow and sometimes wrong.” The goal is to connect a complaint to a cause quickly without making telemetry a second copy of customer data. The same traces answer several related prompts.

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Propagate one correlation ID through auth, tenant, retrieval, model, tools, post-process, and queues.
2. Collect structured per-hop timings and outcome tags.
3. Separate global health from tenant-scoped support views.
4. Let support find a request family without raw content.
5. Keep an access audit independent of sampled debug traces.

Non-functional requirements — how well / under what constraints

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, one ID through every hop, timings and outcomes, tenant-scoped debug. Non-functionally, don’t block the user, don’t store the prompt, and still find why it was slow or wrong.”

The AI request path may use retrieval, an LLM (large language model) and tools; the observability path describes those steps but cannot authorise, alter or stall them. A bounded asynchronous export separates the two. Nothing raw crosses the classifier boundary into the trace store.

Step-by-step architecture

• Give each authorised request a correlation ID and tenant scope; propagate both across services, queues and integrations.
• Trace retrieval, model/LLM, tool and post-processing timings with structured version, error and outcome metadata while the request continues to the user.
• Export spans asynchronously through a bounded buffer. If export fails, serve the user and alert on the diagnostic gap.
• Classify and redact sensitive fields before storage; use an HMAC (hash-based message authentication code) fingerprint with a secret key where correlation is needed, and cap attribute cardinality.
• Preferentially keep error and slow traces, with exemplars and incident overrides so rare failures remain diagnosable.
• Store only bounded, retention-limited traces; send platform-wide aggregates to the global dashboard and enforce tenant/role access for support views.
• Write support access events to a separate, durable audit path so trace sampling cannot erase who viewed evidence.

Start with the user outcome: p95/p99, first-token time, failure/partial rate and affected tenant. Then split request time into retrieval, rerank, model queue, provider, tools and delivery. For a wrong answer, compare retrieved evidence with the answer: empty/stale/off-topic evidence points upstream; good evidence contradicted by the answer points to synthesis or prompt; one-tenant failures point to config, permission or integration state. Track completion, abandonment and support contacts alongside technical quality.

If sampling dropped the only bad trace: compare complaints with trace coverage, temporarily raise fidelity for the affected slice, reconstruct from gateway/dependency evidence, mark the trace gap, then add error-class overrides and always-on exemplars. Do not claim the absence of a trace means no failure.

Latency incident #87: p95 reached 22.18s against a 7s target. Retrieval 4.2s, reranker 3.1s, gateway queue 2.8s and provider 9.36s stacked after a larger reranker and model-route change. Roll back the route, cap candidates, enable a light-reranker fallback and keep EU traffic near an EU endpoint; load-test peak queue depth before repeating the experiment.

Cost incident #88: requests rose only 7%, while input tokens rose 351%, cache hit rate fell from 67% to 4%, and a larger prompt crossed an expensive model-routing threshold. Roll back the template, restore compression and context caps, invalidate the flawed cache entries, and version cost gates. The cache key also lost role_scope, making this a permission incident as well as a budget incident.

Telemetry export may degrade; missing auth, cross-tenant support lookup, raw-content redaction failure and required audit evidence fail closed. Retriever/model outages can use bounded retries or safe reduced answers; repeated failures trip a circuit breaker. Queue overflow needs a restricted dead-letter path and alert. Limit blast radius by tenant, region, workflow and dependency.

Roll out in four phases: agree on user-facing SLOs; instrument one critical path; give support a trace-linked workflow; then add quality and cost signals with sampling. Gate wider release on adequate trace coverage and tolerable overhead. Measure mean time to diagnose, privacy invariant, trace coverage, telemetry overhead, latency, retrieval/tool success, quality-event rate and token cost by workflow.

“I would first define what slow and wrong mean for this customer and write the no-capture list. Every request gets a correlation ID and structured spans through retrieval, model and tools, but full prompts and raw documents stay out of telemetry by default. Spans export asynchronously through redaction and bounded sampling into a retention-limited store. Global dashboards show aggregate health; authorised support gets tenant-scoped evidence, with a separate audit trail. I diagnose from the complaint down: user outcome, then the slow or failing span, then retrieval evidence versus final answer. I retain error and slow traces preferentially so the rare incident is still visible, and I prove the instrumentation itself does not hurt latency or privacy before expanding.”

Read with the unchanged source. The Main guide is the interview path; the Cheat Sheet is the recall card.

Use a fixed schema: correlation ID, tenant, span name, timing, model and prompt-template versions, retrieval references, tool outcome, error class and user-visible outcome. Exclude full prompts, raw documents, secrets and arbitrary labels by default. A narrow break-glass path can capture deeper evidence only with approval, strict access, a short timer and an audit record. If raw content is never collected, it cannot be leaked through the trace store.

The supplementary example is 500 requests/s × six spans ≈ 3,000 raw spans/s. Keep every error and latency outlier, approximately 2–5% of traffic, plus a 5% head sample of normal traffic. At roughly 200 bytes per redacted span for 30 days, storage is about 110–155 GB (roughly 270–390 GB at a more typical ~500 bytes per span). Exact volume depends on actual span size, overlap between sampling classes and traffic shape. Classifier CPU and redaction correctness can become the bottleneck even when storage is affordable.

Request handling never waits for telemetry export. A bounded asynchronous buffer emits spans; export failure reduces visibility and alerts but does not stall the user. Correlation and tenant scope must be serialised across queue boundaries or traces become orphan spans. Trace ingestion is idempotent so retries do not double-count. Classify and hash before storage. An HMAC-SHA256 fingerprint with a secret key can correlate repeated queries while resisting dictionary reversal, as long as the key stays secret, so it lives in a vault and gets rotated; a missing key is an error, not a reason to fall back to a plain unkeyed hash. Reject or collapse unknown high-cardinality attributes.

Debug traces and access audit have separate paths. The trace store is sampled and short-lived. The audit store records who viewed which trace, when and why, and has stricter durability. Global dashboards aggregate across tenants; support views are tenant-scoped by role. Check the access gateway itself for leakage and audit every break-glass use.

Head sampling is cheap but cannot know a request's final outcome. Tail-biased retention costs more buffering but preserves slow and failed traces. Combine normal low-rate sampling with error-class overrides, always-on exemplars and temporary tenant/route incident escalation. Monitor trace coverage against complaints and request errors. If a bad trace is lost, raise sampling for the affected slice, reconstruct from adjacent gateway/dependency evidence and record uncertainty. A missing trace must never be interpreted as proof of health.

For latency, distinguish first-token, per-token and total time. Split client/network, auth, middleware, retrieval, prompt assembly, model prefill, decode, tool calls, post-processing and stream delivery. Queue wait versus service time matters at peak. Long input prompts increase prefill and time to first token; output length increases decode time. A rising p99 with flatter p50 often suggests queueing. A synchronous evaluator or logger accidentally placed on the hot path can erase the value of streaming.

Incident #87 shows a compounded budget failure: eu-central-1 p95 22,180ms and p99 34,890ms; timeout rate 12.8% versus 1.1%. Retrieval p95 was 4,200ms, large reranker 3,100ms with 80 candidates and GPU queue depth 29, LLM (large language model) gateway queue 2,800ms, provider 9,360ms through cross-region routing, and input context 18,500 tokens. The canary ran off-peak. Immediate fixes were route rollback, 30-candidate cap, light reranker when depth >10, context cap and regional endpoint; permanent fixes were per-stage budgets and peak load gates.

For quality, compare retrieval evidence and output. Use recall@k/precision@k or ranking metrics for retrieval; citation correctness/context sufficiency for assembly; faithfulness, relevance and abstention for generation; task success for the application. Fine-tuning mainly addresses behaviour; it is a poor fix for missing knowledge, stale indexing or unauthorised retrieval.

Incident #88 shows cost attribution: daily budget $240, actual $912 by 14:20, projected $1,680; traffic +7%, input tokens +351%. exec_summary_v6 expanded context from 4,200 to 19,000 tokens, disabled compression and crossed a premium-route threshold. Cache hit rate fell to 4% from 67%. The changed key omitted date range, snapshot ID and role scope. Fix both spend and permissions: restore the key, invalidate unsafe entries, cap context and route routine work to the appropriate model.

Missing auth and out-of-tenant support lookups fail closed. Telemetry export degrades with a bounded buffer; an audit sink failure blocks an audited action or requires a human. Retrieval may retry once then return a clearly reduced answer. Model limits may queue briefly or return a retry instruction. Breakers stop repeated downstream timeouts. A buffer overflow enters restricted quarantine with an incident marker. Raw-prompt export is a privacy incident: contain sink access, rotate exposed credentials if necessary, delete/redact by contract and fix the exporter guardrail.

Build dashboards top-down: user outcome, probable cause, support mechanics. Alert on sustained SLO (service level objective) breaches, empty retrieval spikes, fallback growth, critical tool failures and wrong-answer feedback by tenant. Roll out SLO agreement, one end-to-end path, support workflow, then quality/cost signals. Go/no-go depends on trace coverage and overhead. Keep trace schemas compatible during migration and a quick rollback switch available.

Measure by tenant, workflow, model and prompt version. Route simple tasks to smaller models and segregate batch/eval spend. Bound output, context, top-k, retries, agent steps and budgets. Cache only with tenant, permission scope and version in the key. For peak 429s, separate provider from gateway limits and shed non-interactive work. For abandoned streams, cancel generation and tools; that saves spend where the provider stops generating on cancel. For a sudden cache miss, inspect normalisation, prompt prefix bytes, TTL (time to live) and version before adding capacity.

Main · Deep Dive · Unchanged source

What is slow? What is wrong? Which tenants? What content is forbidden to collect? Who may diagnose? What overhead and retention are acceptable?

Request: auth/tenant → retrieval → LLM → tools → response
Spans: async collector → classify/redact/HMAC → error-and-slow biased sampling → bounded store → global aggregate + tenant-scoped support views
Access audit is separate. Telemetry export never blocks the user.

• No full prompts, raw documents, secrets or unbounded labels by default.
• Redact before storage; a missing HMAC (hash-based message authentication code) key is an error.
• Keep rare failures with error overrides, exemplars and incident sampling.
• Missing auth, cross-tenant lookup and required audit evidence fail closed.

Illustrative: 500 requests/s × six spans ≈ 3,000 raw spans/s; retain errors/slow traces plus about 5% of normal traffic. #87: p95 22.18s from stacked retrieval, reranker, queue and provider delays. #88: traffic +7%, input tokens +351%, cache 67% → 4%; missing role_scope also risks leakage.

Start at user outcome, then stage spans. Empty/stale retrieval → upstream. Good evidence contradicted by answer → model/prompt. One tenant → configuration, access or integration. First-token versus total time separates prompt/prefill from long decode.

60-second close: “Correlate every request, collect structured evidence without raw content, retain anomalies preferentially, and let support trace a complaint to a cause within its tenant. Prove both privacy and low overhead before widening rollout.”

Capture enough to explain every failure without turning the telemetry into a second copy of the customer's data.

"The AI is slow and sometimes wrong, and nobody knows why" is three problems wearing one sentence. This page consolidates the group's prompts into one read for the day before. The anchor design comes first. Two incident walk-throughs follow, then the neighbouring prompts, then a card for every cost and latency pivot the group contains.

Sections and tables marked (own construction) were built for this page from the sources' arguments and are not in the sources verbatim. The sizing numbers in section 4 come from the V2 tutorial's "My Perspective on the Gaps" note, which is supplementary and not chapter content.

The customer is not asking for logging. The customer is asking to connect a user-visible failure to a technical cause, fast, safely, and without drowning operators in noise. A vague complaint cannot be instrumented, so the first job is to make it measurable.

Good discovery separates three problems the complaint compresses. Latency asks whether the request is slow, and where in the pipeline. Correctness asks whether the answer is wrong, and whether the fault is upstream in retrieval or downstream in the model or a tool. Diagnosability asks whether support and engineering can explain a failure without exposing sensitive content.

Each stakeholder wants a different signal. Support wants to triage a complaint without waiting on engineering. Engineering wants to know whether the fault is retrieval, the model or a tool integration. Security wants assurance that telemetry does not become a new exposure surface. The executive sponsor wants proof that quality improves release over release, not just green dashboards.

Open the round with one sentence: connect user-visible failures to technical causes without violating privacy or drowning operators in telemetry. Then ask the first two questions, state assumptions aloud, and invite redirection.

The unusual first design decision is the no-capture list. Full capture maximises forensic power, but it raises privacy exposure, access-control burden and storage cost at once. Data never collected cannot leak, cannot explode a metrics backend, and costs nothing to store.

The balanced verdict is not "never capture." Capture enough structure to diagnose most incidents. Redact by default and record hashes or stable references. Keep a narrow, audited break-glass path for approved deep debugging, with time-bounded retention.

The interviewer usually injects this as a hidden constraint mid-round: the customer will not allow raw prompts to be broadly retained. A design that started from "capture everything" has to restart. A design that started from the no-capture list only has to name its break-glass path.

A requirement that cannot fail a test is a preference. "Observable" is a preference. "Support moves from a complaint to the request family without seeing raw content" is a constraint, and it shapes the architecture.

The functional requirements are six. Support the core workflow: a support agent moves from a user complaint to the exact request family and explains the failure without seeing raw sensitive content. Attach a stable correlation ID to every request and propagate trace context across services, integrations and queue boundaries. Capture per-hop timing across authentication, tenant resolution, retrieval, model call, tool use, post-processing and delivery. Record retrieval evidence, model version, prompt template version, generation settings and tool outcomes as structured attributes. Keep an aggregate global dashboard for platform health separate from tenant-scoped views for customer debugging. Tag every request with a user-visible outcome, so complaints map to telemetry.

Every requirement then needs an owner in the architecture (own construction).

The chapter frames estimation around collection policy, not raw QPS (queries per second). Telemetry volume, storage and query load all scale with sampling rate, cardinality bounds and retention window. Observability data is not free, and instrumentation that is not budgeted becomes the outage.

The chapter itself gives no numbers. The V2 gap note supplies a worked version, and it is worth saying aloud because it lands on a surprising conclusion.

The note's conclusion, which I'd defend at this scale, is that storage is not the constraint. Classifier CPU and redaction-policy correctness are. Every span passes through the classifier before storage, so the classifier sits on the observability path's critical line.

Sampling is a control, not a compromise. Head sampling decides at the start of a request, which is simple and cheap but blind to how the request ends. Tail sampling decides after the request completes, so it can keep slow and failed traces, but it needs more infrastructure and careful policy. The interview-safe position is preferential retention for latency-heavy, error-heavy and anomalous traces.

The design is tracing-first: structured spans, selective redaction, and a stable correlation ID on every request from the edge inward. Two paths run side by side. The request path serves the user. The observability path explains the request path, and it must never be able to break it.

The same system drawn with its trust boundaries (own construction):

 REQUEST PATH (serves the user; never waits on telemetry)
 ┌──────────┐  ┌──────────┐  ┌───────────┐  ┌───────┐  ┌──────┐  ┌──────────┐  ┌──────────┐
 │ AuthN +  │─▶│ Tenant   │─▶│ Retrieval │─▶│ Model │─▶│ Tool │─▶│ Post-    │─▶│ Response │
 │ corr. ID │  │ resolve  │  │           │  │ call  │  │ use  │  │ process  │  │ + outcome│
 └────┬─────┘  └────┬─────┘  └─────┬─────┘  └───┬───┘  └──┬───┘  └────┬─────┘  └────┬─────┘
      │ span        │ span         │ span       │ span    │ span      │ span        │ outcome tag
      ▼             ▼              ▼            ▼         ▼           ▼             ▼
 ═════════════════ async, bounded buffer; export failure degrades, never blocks ═══════════════
      │
 OBSERVABILITY PATH
 ┌────────────────────┐   ┌──────────────────────────┐   ┌────────────────────┐
 │ Trace collector    │──▶│ Sensitivity classifier   │──▶│ Sampling policy    │
 │ correlation ID,    │   │ classify, keyed HMAC,    │   │ tail-biased, error │
 │ structured spans   │   │ cap cardinality          │   │ overrides, exemplar│
 └────────────────────┘   └──────────────────────────┘   └─────────┬──────────┘
          ── TRUST BOUNDARY: nothing raw crosses this line ──       │
                                                        ┌──────────▼──────────┐
                                                        │ Trace store         │
                                                        │ retention-bounded   │
                                                        └───┬─────────────┬───┘
                                   ┌────────────────────────┘             └──────────────┐
                        ┌──────────▼──────────┐                              ┌───────────▼───────────┐
                        │ Global dashboard    │                              │ Support + access      │
                        │ aggregates only,    │                              │ gateway → tenant-     │
                        │ cross-tenant        │                              │ scoped views          │
                        └─────────────────────┘                              └───────────────────────┘
 AUDIT PATH (separate write path, durable, fail closed): who accessed which trace, when, why

The source's diagram, kept verbatim:

The source's component table, with a column added for how each part fails (the last column is own construction).

Four questions organise the walk-through. How does each request get a correlation ID? How does trace context flow across services and customer integrations? How does a support operator jump from an alert to the request family without seeing raw content? How are global and tenant visibility separated? Answer each one against the diagram.

A classifier that runs after storage protects nothing, because the raw value has already been written. So classification happens before a value can reach a trace. A keyed hash (HMAC (hash-based message authentication code) with a secret key) then preserves correlation without retaining content.

The data model is small. Request carries tenant ID, prompt and an optional user ID. ExportedTrace carries only a bounded, structured attribute dictionary: correlation ID, tenant ID, prompt classification, query hash, spans and result status. It never carries raw prompt text by default.

The hash is HMAC-SHA256 with a secret key, so the same prompt always maps to the same fingerprint. That fingerprint deduplicates and correlates requests across the store. Without the secret key, nobody can brute-force common prompts back out of the hash. If the key leaks, they can, so it lives in a vault and gets rotated. A missing key raises an error rather than silently hashing without one.

Three tests turn the design into executable contracts:

The last test protects the chapter's central failure drill directly. Sampling and redaction must never make a failure invisible.

Telemetry attributes are schema-validated at ingestion. Unknown high-cardinality fields are rejected or collapsed into a bounded "other" bucket. Queue serialisation contracts carry trace context, idempotency key and tenant scope explicitly, or traces break silently at async handoffs.

The chapter's signature phrase is "sampling drops the only bad trace." Uniform sampling keeps a fixed fraction of everything. When failures are rare, a fixed fraction of a rare thing is often zero. Expect the interviewer to inject this mid-round and have the answer ready cold.

Walk it as a four-step drill.

The first move under this injection is to contain impact, preserve evidence and make the failure legible. Defending the sampler or arguing probability is the weak move.

Every dependency needs an explicit answer to one question: what happens when it is gone? Five terms carry that answer consistently. Fail open continues with reduced visibility. Fail closed refuses the action when integrity, privacy or authorisation would be compromised. Degrade serves a reduced but safe experience. Queue buffers with explicit bounds and retention. Human intervention escalates when risk or ambiguity is too high for automation.

The reason each row differs is the useful thing to say aloud. A user-facing answer can sometimes degrade; an audit trail usually cannot. A retriever timeout can be retried; a permission check cannot be guessed.

Name the blast radius along four axes: tenant, region, workflow and dependency. A raw-prompt leak is usually tenant-scoped but workflow-wide. A queue-boundary trace break can hit one integration path across many tenants. A model outage may be regional. A misconfigured support tool can span every tenant.

Four more named failures get the same drill.

The abuse case deserves one sentence of its own. A developer adds a raw prompt field to a debug span to speed triage. It works once, then becomes the easiest way to move PII into every downstream system. The fix is a redaction policy, an exporter guardrail, and a test that fails whenever exported telemetry contains raw sensitive strings.

Inject sampler loss, queue-boundary breakage, redaction regressions and dependency timeouts in staging. Chaos practice proves the failure policy works before production forces the discovery.

A dashboard built from component metrics shows rates without telling anyone whether the user's experience improved. So build it from one incident story, top down. The complaint is "the AI is slow and sometimes wrong."

The top panel shows the user outcome: latency percentile, failure rate, incident volume. The middle panel shows likely causes: retrieval success, tool success, model error class, token cost. The bottom panel shows support mechanics: trace coverage, redaction rate, queue age for incidents, mean time to diagnose.

The correctness question has a precise answer. Compare the retrieved evidence with the final answer. If retrieval is empty, stale or off-topic, the fault is upstream. If retrieval looked good but the answer contradicts it, suspect the model or prompt construction. If the bad answer appears for only one tenant, suspect tenant configuration, access control or integration state before blaming the shared model.

The diagnostic signals that make this possible are the ones section 2 kept. Request and tenant correlation IDs. Retrieval query fingerprints and document identifiers, not raw text. Model version, prompt template version and generation settings. Tool call status, latency and error class. Outcome tags.

Alert on customer harm or likely impending harm, not on telemetry volume. Good triggers are sustained latency above the SLO (service level objective) and elevated error rates. So are repeated empty retrievals, a spike in fallback responses, and critical tool failures. A sudden rise in "wrong answer" feedback for one tenant or cohort belongs on the list too.

A slow pipeline is rarely one slow component. Incident #87 is the proof: no single span explains it, and every small change looked safe alone.

Setting. A B2B (business-to-business) SaaS support copilot runs inside a live support console. Agents expect answers in under 5 seconds because the customer is waiting in chat. The path is Support Console → RAG (retrieval-augmented generation) API → query rewrite → hybrid retriever → cross-encoder reranker → context compressor → LLM (large language model) gateway → model provider → response evaluator → UI stream. The target is a 7-second p95 with a 12-second hard timeout.

Symptom. On 2026-07-08, during peak European hours, agents saw the spinner for 18–35 seconds. Some answers arrived after the human agent had already replied. Handle time rose 21%. It was not an outage: success rate stayed acceptable, but p95 and p99 broke the SLO.

Telemetry.

What changed. Two changes landed within 24 hours. The reranker moved from cross_encoder_small_v2 to cross_encoder_large_v4. The router moved standard answers from a faster mid-tier model to a larger one after a quality experiment. The canary ran only in low-traffic hours and never tested peak GPU queue depth.

Root cause. Compounded latency. A larger candidate set, a slower reranker, a bigger context, gateway queueing and cross-region routing stacked up. The pipeline broke its budget because several "small" changes landed together.

Debugging path. Decompose the trace into spans and compare each stage's p95 with its baseline. Check candidate count, context tokens, queue time, provider latency and regional routing. Compare canary with control and peak with off-peak. Check whether timeouts land before or after the first token. A 14-second first token means streaming cannot rescue the experience.

Fix. Now: roll back the model route, cap reranker candidates at 30, and enable the light reranker when GPU queue depth exceeds 10. Also reduce max context tokens and route EU traffic to an EU-capable endpoint. Later: add a latency budget per stage and load-test gates for reranker queue depth. Enforce context caps per workflow and add circuit breakers for provider degradation. Quality experiments must pass latency and cost SLOs, not only answer quality.

Abuse angle. Latency becomes a safety risk when users abandon the assistant or bypass review. Broad queries that trigger large candidate sets and expensive reranking are a cheap denial-of-wallet or denial-of-service vector.

"I would start with span-level latency decomposition. The trace shows retrieval at 4.2s, reranker at 3.1s, LLM queue at 2.8s, and provider latency at 9.3s, with 18.5k input tokens. That means this is a pipeline-budget failure, not just one slow API. I would roll back the model route, cap reranker candidates, enforce context compression, add queue-depth fallback, and make future quality experiments pass p95 latency and timeout-rate gates."

The weak answer is "increase the timeout and maybe use a faster model." It never reads a span.

Production GenAI systems fail economically as well as technically. Incident #88 produced no quality complaint at all. The only symptom was a budget alert.

Setting. An executive dashboard copilot summarises revenue, pipeline, customer-risk notes and operational metrics. The path is UI → metrics API → CRM-note retrieval → prompt assembler → semantic cache → LLM (large language model) gateway → cost attribution. The cache key should include tenant, dashboard type, date range, normalised query and data snapshot ID. The router picks a mid-tier or larger model by context size and sensitivity.

Symptom. On 2026-07-08 the daily budget alert fired at 14:20. The CFO (chief financial officer) asked why the assistant had consumed almost four days of budget before lunch.

Telemetry.

What changed. The exec_summary_v6 upgrade added "more supporting context" after executives asked for richer explanations. The same release changed cache-key normalisation and dropped the date-range and snapshot fields. It also disabled compression while debugging a formatting issue.

Root cause. Three effects multiplied. The template expanded context. The larger context crossed the routing threshold into the expensive model. The cache-key regression collapsed the hit rate, so the system paid full price for every large prompt.

Debugging path. Start with cost per workflow, not global provider spend. Compare request count, input and output tokens, model route, cache hit rate and prompt version before and after the deploy. Sample traces to see what was assembled. Check whether the added context improved answers enough to justify its cost.

The hidden correctness bug. The key also lost role_scope. One user's executive summary could be served to another role whenever the query text matched. A cost incident turned out to be a permission incident too.

Fix. Now: roll back exec_summary_v6, restore compression, cap CRM (customer relationship management) notes and table rows, and force the mid-tier model for routine summaries. Invalidate every cache entry written under cache_key_version=v5. Later: add cost regression tests to CI and enforce per-workflow token budgets. Require cost and latency approval for template upgrades, track hit rate by version, and route with budget awareness and graceful degradation.

"The telemetry shows usage grew only 7%, but input tokens grew 351%, cache hit rate dropped to 4%, and requests started routing to a GPT-4-class model. I would roll back the prompt template, restore compression, cap context, fix the cache key, and add CI gates for token and cost regression. I would also audit cache safety because missing role_scope can create data exposure."

The weak answer is "ask users to make shorter queries or switch to a cheaper model." It blames usage that grew only 7%.

Three prompts in the group look different and use the same instrument. Each one is answered by reading the anchor's traces at a different depth.

#53, latency regression. "An enterprise customer reports that responses have become slow." First clarify which latency: time to first token, time between tokens, total completion time, or user-perceived latency. Then decompose across ten stages: client and network, authentication, middleware, retrieval, prompt construction, model prefill, token generation, tool calls, post-processing and streaming. Check input and output token counts and retrieval p95 and p99. Then check retry rates, rate-limit throttling and traffic patterns. Finish with model version, prompt changes, cache hit rates and regional routing. Incident #87 is this prompt with real numbers.

The follow-up is "it only happens at peak traffic." Peak-only latency points at queueing, not a slower model. Separate queue wait from service time. Queueing delay climbs steeply as utilisation nears 100%; for a simple queue the wait is about 4× service time at 80% and 9× at 90%. A rising p99 with a flat p50 usually means queueing. Add admission control and shed non-interactive work to batch (the utilisation point comes from the additions file, section D; the multiples are simple single-queue arithmetic).

#54, RAG (retrieval-augmented generation) with poor answer quality. Separate seven failure classes: retrieval, context assembly, reasoning or synthesis, prompt or instruction, output format, evaluation, and user experience. Measure each separately. Use recall@k, precision@k and MRR (mean reciprocal rank) or nDCG for retrieval. Use citation correctness and context sufficiency for assembly. Use answer faithfulness, answer relevance and abstention quality for generation. Use end-to-end task success for the whole. The section 9 flowchart is the first cut; these metrics are the second.

The follow-up is "would you fine-tune the model?" Classify the problem first. A knowledge problem is usually retrieval or data access. A behaviour problem is prompting, fine-tuning or structured output. A freshness problem is indexing and the data pipeline. A permission problem is access-control-aware retrieval. Fine-tuning mainly helps the second; it is a poor fix for the other three.

#74, diagnose high latency in an LLM inference pipeline. This is the layer below the application. Walk the full stack: tokenization, network, batch size, KV (key–value) cache, post-processing. Two facts carry the answer (from the additions file, section B). Prefill processes the whole prompt in parallel and sets time to first token, so it scales with input tokens. Decode generates one token at a time and sets tokens per second, so it scales with output tokens. A frozen UI points at prompt size, retrieval and queueing. An answer that starts fast then drags points at output length.

Batch size trades throughput for per-request latency. With continuous batching nothing waits to fill; each decode step just gets slower as more sequences share it. The KV cache holds attention state for tokens already processed. Its memory caps how many sequences fit on a GPU at once, and a full cache forces queueing or preemption. Post-processing is usually small, but a synchronous evaluator or logger on the hot path is not. The same stack walk anchors G20 on inference serving, so prepare it once.

Observability fails socially before it fails technically. If no one owns a signal, no one owns the decision. So the rollout has names, gates and exit criteria, not only components.

The answer key gives a weekly version. SLOs come in weeks 0-1 and measurable "slow" and "wrong" in weeks 1-2. One path follows in weeks 2-3 and redaction scanning in weeks 3-4. The support workflow lands in week 5, then quality and cost signals in weeks 6-8.

The go/no-go gate is explicit. If trace coverage falls below the minimum or overhead moves latency, the release pauses. Rollback triggers are concrete: loss of trace propagation, a spike in quality-event rate, or an unsustainable cost slope.

Decide what is product, adapter, service or configuration. Per-customer sampling rates, retention windows and alert thresholds are configuration. Ticketing and alerting integrations are adapters. Collection, redaction, correlation propagation and the access gateway are a shared service. The request lifecycle, outcome tags and the minimal schema are core product. That split keeps customer edge cases out of the core.

Launch readiness needs a canary on one tenant or slice and a fast rollback switch. Keep old and new trace schemas interoperable during migration. Train support on what a healthy trace looks like, and publish an ops guide with metric definitions, access rules and rollback steps. Have three artifacts ready before launch: an evidence policy, a response runbook and an access review.

Spend time in proportion to risk, not diagram size. The pacing below is the chapter's own.

Four trade-offs must be defended. Full capture against privacy and cost: capture structure, keep a break-glass path. Head against tail sampling: prefer tail retention for slow and failed traces. Raw prompts against hashed metadata: store template ID, field presence, sanitized entity markers and fingerprints. Global dashboard against tenant views: build both, with a permission boundary.

Repair the common weak answers on the spot. "Log everything and sort it out later" becomes a tiered capture strategy with named privacy and cost consequences. "Use tracing" becomes what is traced, how context propagates, and which layer each span isolates. "Sample 10 percent" becomes a question about the failure distribution. "Put it on one dashboard" becomes global health plus tenant-scoped debugging. "If the answer is wrong, the model is wrong" becomes a walk through auth, retrieval, prompt construction, tools and integrations.

"My design goal is to connect user-visible failures to technical causes without collecting more sensitive data than we need. I would instrument the full request path with correlation IDs, structured spans, and bounded metadata across authentication, retrieval, model calls, and tools. I would store raw prompts and customer content only through a narrow, audited escalation path, because the default should be privacy-preserving and cost-aware. For debugging, I would rely on template IDs, document references, timing, error classes, and outcome labels so support can separate auth, retrieval, model, and integration problems. I'd use a global operational view plus tenant-scoped views for authorised support, and I'd prefer selective retention for slow or anomalous traces over blind random sampling so we do not lose the one bad trace. The riskiest trade-off is observability depth versus privacy and cost, and my first production rollout gate would be proving that the system can consistently identify the root cause of real customer incidents without exposing unnecessary content or overwhelming the team."

Two gap answers are worth having ready (V2 gap note, supplementary). On build versus buy: buy the span plumbing, build the redaction and classification boundary, because "what counts as sensitive for this tenant" is business logic no vendor can own. On regulation: hashed prompts make a deletion request smaller, not trivial. A keyed hash tied to a user is still personal data under GDPR (General Data Protection Regulation), so delete or unlink those fingerprints too, along with quarantine and break-glass records. The retention window should follow the customer's data-processing agreement.

Ten members of this group are the same system asked a cost or latency question. Each gets a card: the dominant driver, the cheapest lever, the metric that proves it, the move to refuse, and the line to say. The "Do not" field for the §15 scenarios is (own construction); the playbook gives no weak answer for them. Every card shares the playbook's customer-communication sentence: "Acknowledge impact, show measured cause, provide immediate mitigation, then explain long-term prevention and success metric."

15.1 LLM costs exploded after launch

15.2 LLM cost increased 5× after rollout

15.3 Latency jumped from 2 seconds to 20 seconds

15.4 Executive demo is too slow

15.5 Customer complains the AI is too expensive

15.6 Prompt update increased token usage

15.7 Model provider latency became unstable

15.8 Cache hit rate dropped suddenly

Two facts sharpen this card (additions file, section A1). Prompt caching is a byte-exact prefix match, so one changed byte invalidates everything after it. A timestamp in the system prompt, unsorted JSON keys, a reordered tool list or a model swap each cause a silent miss. Incident #88 adds the other half: cache changes need a security review too, because a key missing role_scope serves one role's answer to another.

15.9 Rate limits are throttling us at peak

15.10 Streaming works but users abandon mid-answer

Every strong answer on this page runs through four verbs in order. Measure by attributing cost and latency per stage, tenant and prompt version first. Route by matching model and path to risk. Bound steps, tokens, top-k, timeouts and budgets. Cache safely, with tenant, permission scope and version in the key.

Every level traces the request; the level shows in what the answer refuses to collect and who may read what is left.

• "Slow and sometimes wrong" is three problems, latency, correctness and diagnosability, and each is made measurable before any design.
• The first design decision is the no-capture list: full prompts, raw documents, secrets and unbounded labels stay out by default.
• Requirements are stated so a test can fail them, with the support workflow as the functional core and overhead as a budget.
• Telemetry cost is set by collection policy; the worked sizing shows storage is small and the classifier is the real constraint.
• The architecture is tracing-first, with the observability path unable to block the request path and nothing raw crossing the trust boundary.
• Classification and keyed hashing happen before storage, and three tests pin idempotency, privacy and failure visibility.
• Sampling is tail-biased with error overrides and exemplars, because uniform sampling drops the only bad trace.
• Every layer has a named failure policy, and audit evidence fails closed where user answers can degrade.
• The dashboard is built top down from the complaint, and wrong answers are triaged by comparing evidence with output.
• Incident #87 was compounded latency from several small changes, found only by comparing each span with its baseline.
• Incident #88 was prompt growth, a routing threshold and a broken cache key multiplying, and it hid a permission leak.
• Prompts #53, #54 and #74 are the same traces read at different depths: the stage decomposition, the seven RAG (retrieval-augmented generation) failure classes, and the inference stack.
• Rollout runs in four phases with named owners, an eight-metric scorecard, explicit go/no-go gates and a risk register.
• The fifty minutes go to the no-capture list, the dropped-trace drill and the support test.
• Every cost and latency pivot is answered with a card built from measure, route, bound and cache safely.

1. What are the three problems inside "slow and sometimes wrong"? Latency (where in the pipeline), correctness (retrieval or generation), and diagnosability (can support explain it without exposing content or drowning in telemetry).
2. Name the four things never captured by default. Full prompts, raw customer documents, secrets, and high-cardinality free-form labels.
3. Why must the sensitivity classifier run before storage? Once a raw value is written, every system that reads the store has it; classification after storage protects nothing.
4. Why a keyed hash (HMAC (hash-based message authentication code)) rather than a plain hash? It keeps correlation and deduplication while resisting brute-force reversal of common prompts, as long as the secret key stays secret; a missing key must raise an error.
5. Walk the "sampling drops the only bad trace" drill. Detect by comparing complaints with sampled trace volume; contain by raising fidelity for the affected slice; recover from neighbouring evidence and mark the gap; prevent with error-class overrides, exemplars and escalation hooks.
6. When does a failure fail closed rather than degrade? When integrity, privacy or authorisation would be compromised: a missing auth token, an unavailable audit sink, a lookup outside tenant scope.
7. How do you tell a retrieval fault from a model fault? Compare retrieved evidence with the answer. Empty, stale or off-topic evidence points upstream; good evidence contradicted by the answer points at the model or prompt; a one-tenant failure points at configuration, access or integration state.
8. In #87, which spans stacked up, and what was the process failure? Retrieval 4200 ms, reranker 3100 ms, gateway queue 2800 ms and provider 9360 ms; the canary never tested peak GPU queue depth.
9. In #88, why was the cost spike also a security incident? The v5 cache key dropped role_scope, so one role's summary could be served to another.
10. For #74, why do input and output tokens hurt in different places? Prefill processes the prompt in parallel and sets time to first token; decode generates one token at a time and sets total time.
11. What does the go/no-go gate check? Trace coverage above the minimum for the top failure path, and telemetry overhead not moving latency.
12. Give the four verbs and the line for #44. Measure, route, bound, cache safely. Contain, attribute, then diagnose, keeping low-cost safe paths online; budget guardrails must exist before launch.

Fact-check sources (checked 27 Sep 2026)

• Grafana Tempo docs: Manage trace ingestion — typical span size (~500 bytes) behind the storage sizing
• RFC 2104: HMAC, section 3 Keys — HMAC security rests on a secret key, not a salt
• OpenAI API docs: Deprecations — GPT-4-class list prices used to check the $2.74 figure
• Ovadia et al. 2023, Fine-Tuning or Retrieval? (arXiv 2312.05934) — fine-tuning is a weak way to add knowledge compared with retrieval
• Kwon et al. 2023, Efficient Memory Management for LLM Serving with PagedAttention — continuous batching and KV-cache limits
• vLLM docs: Optimization and Tuning — batch size, preemption and chunked prefill trade-offs
• GDPR (Regulation 2016/679), Recital 26 — pseudonymised data is still personal data
• EDPB Guidelines 01/2025 on Pseudonymisation — keyed hashes as pseudonymisation, not anonymisation
• GDPR Article 17: Right to erasure — deletion requests reach linked fingerprints
• Claude docs: Prompt caching — exact prefix match; timestamps, unsorted JSON, tool changes and model swaps break the cache
• OpenAI API docs: Prompt caching — exact prefix match and minimum cacheable length
• OpenRouter help: Which providers stop billing when a stream is cancelled — cancelling a stream stops billing only on some providers
• vLLM docs: OpenAI-compatible server — self-hosted engines abort generation on client disconnect