Handle overload and fair scheduling: think like a staff engineer

Track offered/admitted/rejected/expired/completed counts by tenant and workload class, oldest waiting age, per-tenant service share, queue bytes, active/unknown slots, token reservations versus observed use, provider errors, and retry amplification. A global healthy throughput graph can hide B or C receiving zero timely service. Record protected IDs and reservation reasons, not raw source/prompts or credentials.

Interactive grounded queries, long document preparation, and periodic ingestion may need separate classes with different deadlines and minimum shares. Reserve enough interactive headroom without giving strict priority an unlimited starvation path. A batch class can use idle capacity while respecting its queue/storage bounds. When a tenant is idle, borrowing unused capacity improves utilisation, but the policy must still honour entitlements as other queues become backlogged.

When overload appears, inspect the binding resource before changing weights. If token limits bind, reducing approved context/output limits or changing the request mix may help. More app replicas cannot alter the fixed provider quota. A cache hit may save DB reads while still requiring a full generation budget. Never drop tenant checks or use stale permission authority as a degradation strategy.

A production dispatcher must coordinate fair selection, durable claims, budget grants, policy revalidation, deadlines, and settlement/reconciliation. The slice tests these boundaries together for one happy worker path and separately under guard races; it does not implement an atomic distributed dispatch transaction. A crash between claim/grant/start/commit can strand capacity or leave uncertainty. Defaulting to conservative failure is safer than repeating a charged operation, but needs an operational recovery policy.

Prompt: Every tenant is allowed ten requests/minute. Why are long queries still causing outages?

Spoken answer: “Request counts hide token cost and time occupying a provider slot. Ten small queries and ten huge generations can consume very different resources. I separately bound request rate, reserved tokens, simultaneous calls, queued bytes/jobs, and the deadline. A global dependency budget still applies across all tenants and app replicas. The limits need shared authority rather than multiplying per replica.”

Trade-off: Conservative reservations can waste capacity. Changed constraint: If actual output is much shorter than reserved, design a safe usage-reconciliation rule tied to the provider’s quota model.

Prompt: B and C time out behind A, but overall provider throughput looks good.

Spoken answer: “A global FIFO can allocate the useful waiting window to the earlier large burst. I would bound each tenant’s queue and choose explicit shares, using RR for equal request entitlements or WRR for known weights. Strict priority is appropriate only for justified classes with starvation controls. These policies allocate the same provider capacity; they cannot repair sustained total demand beyond it.”

Trade-off: Fair shares may reduce the largest tenant’s immediate throughput. Changed constraint: With variable token costs, compare dispatch-count fairness against deficit/cost-based credits.

Prompt: Increase the queue 100 times so fewer callers receive rejection.

Spoken answer: “It can absorb a bounded burst if the backlog drains within useful deadlines. Under sustained arrivals above service, it stores more waiting and delays visible failure. I would track oldest age and admitted completion, not just rejection count. A finite queue and clear deadline policy protect memory and useful work; callers need honest backpressure.”

Trade-off: Early rejection sacrifices some potentially recoverable burst work. Changed constraint: Show how you would size a buffer for a measured ten-second burst that is followed by a long idle period.

Prompt: A worker disappears after starting generation. Its lease has expired, and all concurrency slots are occupied.

Spoken answer: “Lease expiry fences our local completion authority but does not establish that the remote model stopped. Releasing every slot immediately can oversubscribe the provider and duplicate work. I preserve the operation and unknown reservation, then reconcile using provider evidence or an explicit policy with a documented uncertainty bound. The conservative local fixture requires that reconciliation; it does not invent an automatic cancellation guarantee.”

Trade-off: Safety can temporarily reduce availability. Changed constraint: If the provider exposes a reliable terminal-status API, define how job and reservation states converge after a lost response.

• Track offered, admitted, rejected, expired and completed counts per tenant and workload class. A healthy global throughput graph can hide B or C receiving zero timely service.
• Request counts hide token cost and slot time, so bound request rate, reserved tokens, simultaneous calls, queued work and the deadline separately. Limits need shared authority, or per-replica counters multiply them.
• Lease expiry fences local completion authority but does not prove the remote model stopped. Keep the unknown reservation and reconcile with provider evidence, because releasing slots early can oversubscribe the provider and duplicate work.
• Inspect the binding resource before changing weights. More replicas cannot alter a fixed provider quota, and stale permission authority is never a degradation strategy.