Reliability and consistency: think like a staff engineer

Classify the symptom first: missing saved answer, stale source, denied identity, rising queue wait, provider timeout or audit failure. Protect customer data and freeze the suspected risky change. Assign incident coordination, diagnosis and communication roles; retain request/job/reservation/event IDs without copying raw prompts into chat/logs.

For a missing saved answer: inspect primary commit and the same scoped key; compare acknowledged minimum, projection watermark and missing sequences; fall back only where the data-class contract permits. Never regenerate simply because a lagging replica returned empty. For a stuck projection, inspect outbox age, delivery attempts and consumer failures; replay idempotently before considering cursor repair.

For high latency: separate ingress/pool wait, policy, retrieval, tenant queue wait, provider service and commit/audit. Check offered/admitted/completed rates and concurrency/token budgets. A timeout spike with normal provider duration may be queue or connection setup time. A trace sample is not the full population; use counters/histograms and representative tail samples together.

For dependency outage: preserve remaining deadlines, open a bounded breaker, shed excess work and reconcile unknown remote calls. Before primary promotion, fence old writers, establish the authoritative replacement and assess possible lost commits. Verify scoped history, keys, worker epochs, quota reservations and current policy after recovery. A restart is not a complete runbook.

The runbook ends with evidence: restored authority/current version, drained/reconciled outbox, no unexpected duplicate effects, error/latency recovery, and a documented customer-impact window. A post-incident review should identify causal conditions and assign measurable fixes, not merely blame a slow model. Real paging, dashboards, restore drills and failover tests are not implemented by this fixture.

Prompt: Read replicas improve availability and latency. Should every read use them?

Spoken answer: “No. Current membership, permissions, worker ownership and quota reservations need their authoritative contract. A saved-history read can use a replica only when it covers the acknowledged minimum version, otherwise I use the primary within budget. Derived search can lag behind a current-version gate. An immutable body cache can reuse exact bytes after live permission checks.”

Trade-off: Stronger current reads concentrate dependency load. Changed constraint: If primary reachability fails, identify which customer-visible operations must stop rather than silently use old authority.

Prompt: The database result commits, but notification publication fails.

Spoken answer: “I commit result and notification intent together locally. A delivery process can later publish the retained intent, and the consumer records event identity with its projection change. Publication may repeat; the outbox does not grant exactly-once external effects. A missing projection should trigger replay/reconciliation, not blind regeneration.”

Trade-off: Async delivery introduces lag and retention operations. Changed constraint: If messages arrive out of order, explain why maximum observed sequence is not a contiguous applied watermark.

Prompt: A provider times out after starting. Immediately retry through another provider?

Spoken answer: “The first provider may still be computing or have charged. Another provider is a new external attempt, possibly with different approved data-handling and answer behaviour. I preserve the operation's unknown outcome and budget reservation until the provider/reconciliation contract permits another attempt. Failover is safe only under explicit semantics, not because a second endpoint responds.”

Trade-off: Conservative ambiguity handling reduces availability. Changed constraint: A provider exposes reliable terminal status and durable idempotency; specify what additional evidence permits retry.

Prompt: End-to-end latency rises, app CPU remains low, and provider execution time is unchanged.

Spoken answer: “I inspect waiting time: admission/tenant queues, DB pool acquisition, identity/audit calls and connection setup. I correlate request traces with offered/admitted/completed counts and dependency histograms. Adding app replicas can worsen pool pressure and cache misses if the bottleneck is shared. I test one hypothesis against the critical path and compare the affected population before changing capacity.”

Trade-off: More diagnostics cost telemetry/storage and operator time. Changed constraint: If only 1% of traces are sampled, explain what aggregate signals prevent a false diagnosis of the tail.

• Classify the symptom first. For a missing saved answer, inspect the primary commit under the same scoped key and compare the acknowledged minimum with the projection watermark, and never regenerate because a lagging replica returned empty.
• Before promoting a primary, fence old writers and assess possible lost commits. A restart is not a runbook, which ends with evidence such as restored authority, a reconciled outbox, no unexpected duplicate effects and a documented customer-impact window.
• Read replicas suit saved history only past the acknowledged minimum version. Membership, permissions, worker ownership and quota reservations keep their authoritative contract.
• Retrying a timed-out call through another provider is a new external attempt that may duplicate work or charges. When latency rises while provider time is unchanged, look first at waits such as tenant queues, pool acquisition and connection setup.