Scale the application: think like a staff engineer

Measure offered/admitted/completed rates separately; app CPU, in-flight requests, DB pool waits/query time, cache hit/miss/eviction, source/policy lookup time, provider latency/limits, and startup/drain time. A high hit rate is not a safety metric: also count rejected stale/forbidden cache lookups. Protect per-tenant identifiers in telemetry and never log cached source text or tokens by default.

If p95 rises, locate waiting time rather than immediately scaling. DB pool wait with low CPU suggests connection/contention issues. Provider wait with low CPU suggests dependency capacity. A cache-miss spike immediately after scaling may be a warmup effect. Removing the live authority gate to improve a latency graph would violate the inherited security contract.

Keep cache misses from creating unlimited duplicate fills in production: bounded single-flight, jittered expiry, and a controlled stale-safe body strategy may help, but may never serve forbidden/current-version-mismatched content. The fixture has a bounded LRU and bypass flag; it does not implement distributed single-flight or benchmark a stampede solution. If authoritative storage fails, return the established error/refusal path rather than promote stale cache contents into authority.

Scale-in must stop new routing and drain requests with a deadline. Durable job workers use their existing lease/epoch protocol rather than app-instance memory. Database migration, ingress configuration, session identity, real provider deadlines, load generation, and autoscaler stability are deployment checks still to run. Advanced cluster design remains outside this stage’s assigned modules.

Prompt: A reviewer says the app cannot be stateless because users have saved requests.

Spoken answer: “Stateless here means a replica does not own the only copy of session, operation, or history state. Requests and jobs remain durable in shared authority. A new ready instance can reconstruct the request from trusted identity and stored state. It may keep an expendable cache, but losing that cache changes cost or latency rather than correctness.”

Trade-off: Shared authority becomes a dependency. Changed constraint: If a regional network partition separates app and database, what response preserves the saved-answer contract?

Prompt: Most users ask similar supplier questions. Why cache solely by query?

Spoken answer: “The same words can refer to different tenants, permissions, source versions, or model configurations. I start with immutable passage bytes under tenant/source/version/chunk identity, and recheck live authority before returning them. Final-answer reuse would need a much richer dependency and scope contract. A fast hit that bypasses current access is a regression, not a scaling success.”

Trade-off: Narrow caching saves less total computation. Changed constraint: If an answer is explicitly public and tied to a frozen corpus/model, describe a safe answer-key scheme.

Prompt: Four replicas provide 112 planned RPS. Can you commit to the 100 RPS target?

Spoken answer: “Only as an initial model under fixed CPU and 70% planning target. I must check database work, provider and token budgets, burst distribution, warmup, and latency/error SLOs. In the no-cache case the database budget permits about 83.33 RPS, so replicas alone are insufficient. After caching, provider capacity can still become the minimum.”

Trade-off: Headroom costs resources without proving tail latency. Changed constraint: If passage sizes triple, which hit-rate and query-cost assumptions become unsafe?

Prompt: Can each new replica start with a copy of the SQLite file?

Spoken answer: “That creates independent authorities for history, keys, and worker leases. I preserve the schema and invariants in a shared relational design, test transaction/claim semantics, and use a controlled writer cutover with reconciliation. The local two-object test establishes the desired semantics on one file; it does not prove a production PostgreSQL migration or a network-filesystem deployment is safe.”

Trade-off: Controlled cutover may need a write pause. Changed constraint: If downtime must be near zero, specify the additional replication/change-capture checks rather than casually adding dual writes.

• Stateless means no replica owns the only copy of session, operation or history state. Losing its expendable cache changes cost or latency, never correctness.
• When p95 rises, find the waiting time before scaling. DB pool waits with low CPU suggest connection contention, provider waits with low CPU suggest dependency capacity, and a miss spike just after scaling may be warmup.
• A high hit rate is not a safety metric, so also count rejected stale or forbidden cache lookups. If authoritative storage fails, return the established error path rather than promote stale cache contents into authority.
• Starting each new replica with its own copy of the SQLite file creates independent authorities for history, keys and worker leases. Migrate with a controlled writer cutover and reconciliation, not casual dual writes.