Skip to content
Forward Deployed

Forward Deployed
Lessons and contents
Learn

Handle overload and fair scheduling: validate and carry forward

Source
From the Staff AI System Design course
By
Sourav Banerjee
Last updated
8 Nov 2026

Report an issue

Self-Test and Memory Card

  1. What changes in WRR? The allocation of existing eligible service, not physical provider throughput.
  2. Why are per-tenant queues bounded? One caller cannot occupy all waiting storage; deadlines still limit useful backlog.
  3. What makes a token estimate safe? It must cover the actual allowed input/output envelope under the real provider’s counting rules; the fixture integer is not a tokeniser.
  4. What does an unknown reservation mean? External termination is not established, so release/retry requires reconciliation.
  5. Why is a finite burst not the same as an unstable steady queue? A burst may stop and drain; persistent arrivals above departures keep growing backlog absent rejection.

The 60-second card: authenticate first; deduplicate operations; budget requests/tokens/slots separately; bound each queue; schedule entitlements; expire old work; backpressure without blind retries; reconcile unknown effects. Review at about one, seven, and 21 days.

  • Explain rate, concurrency, queueing, and scheduling without notes in two minutes.
  • Draw inherited retrieval/history plus the new admission/scheduler/guard path.
  • Defend FIFO, priority, RR, and WRR with workload and cost assumptions.
  • Diagnose tenant starvation, token saturation, retry storms, and unknown slots with concrete signals.
  • Run/modify the quota, scheduler, and durable-worker integration tests; learner completion remains unassessed.

Decision and Stopping Point

ADR-005 accepts bounded fair dispatch under fixed dependency budgets. Snapshot preserves checked artefacts and limits.

Stop after T18.6. A future T18.7 expands enterprise identity, isolation, audit, private connectivity, and retrieval governance. It inherits the current tenant/policy gates and all execution/overload uncertainty boundaries; it has not been authored here.

Primary References

Sources checked 2 November 2026. Workloads and model outputs are illustrative; executed checks and untested production integrations are in validation.

Key Takeaways

  • Weighted round-robin changes how existing eligible service is allocated, not physical provider throughput.
  • A finite burst can stop and drain. Persistent arrivals above departures keep growing the backlog unless work is rejected.
  • A token estimate is safe only if it covers the allowed input and output envelope under the real provider's counting rules.
  • An unknown reservation means external termination is not established, so releasing or retrying it requires reconciliation.

Rate your confidence and keep notes on the module page.