Build the MVP: think like a staff engineer

11. Operations and Failure Diagnosis

Observable signal	Likely boundary	Immediate action	Persistent fix or next check
`401`, no request row, no provider attempt	Identity verification	Check token validity and trusted issuer/audience	Repair identity configuration; never bypass scope checks
Provider timeouts with normal database latency	Dependency or deadline	Stop automatic retries; inspect provider outcome/latency	Revisit timeout budget and provider integration
`503`, provider call happened, row processing	Final storage operation	Treat outcome as unknown; inspect storage and request ID	Restore storage, reconcile row, design later recovery
Old processing rows after restart	Process died between commits	Check age and request traces before retry	Explicit recovery policy; durable execution in a later stage
Fast server timing, slow Client	DNS/connect/TLS/network	Compare phase and end-to-end timings	Connection reuse and correct timeout configuration
History contains another user’s request	Authorisation regression	Block affected access and investigate scope	Fix trusted predicates and run negative tests before reopening

Record request ID, route, outcome/status, elapsed time, provider duration, database duration, input/output usage where available, and an appropriately protected tenant identifier. Never record bearer tokens or raw prompts/answers in ordinary logs. Useful counters include request outcomes, provider attempts, completed commits, and storage failures; compare attempts and commits to investigate gaps. A metric exporter and an alerting backend are proposed operations work, not shipped components of this lab.

Use a proposed pilot alert when timeout/error proportion breaches its agreed window, and when a processing row is older than the client budget plus investigation margin. Set thresholds after measuring the pilot; this lesson does not invent incident data. During a bad application release, stop new generation, retain the database, and roll back compatible code. A rollback cannot recover an answer never saved. Before opening a real pilot, test backups, restore, secret handling, identity, retention deletion, request size controls, and real-provider deadlines.

12. Staff Interview Practice

Why Start With One Application?

Prompt: A reviewer wants separate gateway, identity, model, and history microservices on day one. Defend your design.

Spoken answer: “The pilot needs one saved drafting flow for one active user. I keep identity policy and the model adapter as clear code boundaries inside one application. That makes the acknowledgement contract and failure trace easy to inspect. Separate services add deployments, credentials, and network failure points before we have independent scaling or ownership needs. I would split a role when its workload, team ownership, or isolation requirement justifies the operational cost.”

Trade-off: Simpler operation shares a failure domain. Follow-up: If two teams need independent release schedules, identify the first boundary to extract and its migration contract.

What Does Success Guarantee?

Prompt: The model answered but the final database write failed. Can you return 200 and retry the save later?

Spoken answer: “The user asked for a saved draft, so a successful response must follow the answer commit. I return a storage error if that commit fails. The earlier processing row lets us see an attempt existed, but it cannot prove what the provider did. I will not automatically call the provider again, because it may already have charged or generated. I need reconciliation and a stronger execution design before promising recovery.”

Trade-off: Durable acknowledgement may withhold generated text during a storage incident. Follow-up: If partial text is acceptable, specify a separate provisional-content state instead of silently weakening completed success.

Is the Pilot Small Enough to Ignore Capacity?

Prompt: Two hundred daily requests sounds tiny. Which limit could still fail first?

Spoken answer: “Daily mean hides the brief 0.5 request-per-second peak. At 1,500 expected tokens per request, that peak is 45,000 tokens per minute. Against our hypothetical 50,000-token quota it has little headroom even though request quota is only half used. Average in-flight work is about 1.31 requests at 2.62 seconds, but that is not a safe concurrency cap. I would measure burst and token distributions, then enforce limits when the workload requires them.”

Trade-off: Planning estimates guide limits but do not replace load measurements. Follow-up: Double input length and recompute the token bottleneck without changing RPS.

Why Not Stream Every Answer?

Prompt: A product manager wants immediate visible tokens. Does SSE solve the five-second target?

Spoken answer: “SSE can improve time to first visible text, but the total answer still needs generation and persistence. If tokens arrive before the answer commit, I must label them provisional and emit a distinct saved-completion event. A disconnect needs a history lookup, not an assumption that the operation failed. For this short pilot path I choose complete JSON first. I would adopt streaming after measuring the user benefit and defining the final commit contract.”

Trade-off: Simpler final responses offer less progressive feedback. Follow-up: Describe what the Client displays when a stream disconnects after the last token but before the saved event.

How Do You Prevent Cross-Tenant History Reads?

Prompt: Alice puts tenant_id: B in the body and guesses Bob’s request ID. What enforces isolation?

Spoken answer: “The trusted identity adapter determines tenant and user. The body cannot override them. Every read uses both scope fields, and an out-of-scope ID returns the same not-found response as a missing row. The model has no authorisation role. I test list and item reads with two identities. This is the baseline application control; enterprise identity lifecycle and stronger isolation options need later work and real integration tests.”

Trade-off: Application predicates are simple but demand consistent enforcement. Follow-up: If database row-level policy is introduced later, explain how it complements trusted application scope.

Key Takeaways

Each signal points at a boundary. A 401 with no row means identity, a 503 after a provider call with a processing row means final storage, and fast server timing with a slow client points at DNS, connection or TLS.
Record request ID, outcome, elapsed time, provider and database durations and usage, but never bearer tokens or raw prompts and answers. Compare provider attempts with completed commits to find gaps.
Split a role into its own service only when its workload, team ownership or isolation requirement justifies the operational cost. One application keeps the acknowledgement contract easy to inspect, at the price of a shared failure domain.
Streaming improves time to first visible text but not total generation and persistence. Tokens sent before the answer commit must be labelled provisional, with a distinct saved-completion event and a history lookup after a disconnect.