Handle overload and fair scheduling: validate and carry forward
- Source
- From the Staff AI System Design course
- By
- Sourav Banerjee
- Last updated
- 8 Nov 2026
Self-Test and Memory Card
- What changes in WRR? The allocation of existing eligible service, not physical provider throughput.
- Why are per-tenant queues bounded? One caller cannot occupy all waiting storage; deadlines still limit useful backlog.
- What makes a token estimate safe? It must cover the actual allowed input/output envelope under the real provider’s counting rules; the fixture integer is not a tokeniser.
- What does an unknown reservation mean? External termination is not established, so release/retry requires reconciliation.
- Why is a finite burst not the same as an unstable steady queue? A burst may stop and drain; persistent arrivals above departures keep growing backlog absent rejection.
The 60-second card: authenticate first; deduplicate operations; budget requests/tokens/slots separately; bound each queue; schedule entitlements; expire old work; backpressure without blind retries; reconcile unknown effects. Review at about one, seven, and 21 days.
- Explain rate, concurrency, queueing, and scheduling without notes in two minutes.
- Draw inherited retrieval/history plus the new admission/scheduler/guard path.
- Defend FIFO, priority, RR, and WRR with workload and cost assumptions.
- Diagnose tenant starvation, token saturation, retry storms, and unknown slots with concrete signals.
- Run/modify the quota, scheduler, and durable-worker integration tests; learner completion remains unassessed.
Decision and Stopping Point
ADR-005 accepts bounded fair dispatch under fixed dependency budgets. Snapshot preserves checked artefacts and limits.
Stop after T18.6. A future T18.7 expands enterprise identity, isolation, audit, private connectivity, and retrieval governance. It inherits the current tenant/policy gates and all execution/overload uncertainty boundaries; it has not been authored here.
Primary References
- RFC 6585: the HTTP
429response used by the proposed admission facade. - RabbitMQ consumer prefetch: example of limiting outstanding deliveries, separate from global provider slots.
- Google SRE: handling overload: resource limits, degradation, and rejection as operating choices.
- AWS: timeouts, retries, and backoff with jitter: retry load and side-effect boundaries.
Sources checked 2 November 2026. Workloads and model outputs are illustrative; executed checks and untested production integrations are in validation.
Key Takeaways
- Weighted round-robin changes how existing eligible service is allocated, not physical provider throughput.
- A finite burst can stop and drain. Persistent arrivals above departures keep growing the backlog unless work is rejected.
- A token estimate is safe only if it covers the allowed input and output envelope under the real provider's counting rules.
- An unknown reservation means external termination is not established, so releasing or retrying it requires reconciliation.