Skip to content
Forward Deployed

Forward Deployed
Lessons and contents
Architecture decision recordsLesson 5 of 13
13 lessons
Learn

ADR-005: Bounded fair dispatch under fixed provider budgets

Source
From the Staff AI System Design course
By
Sourav Banerjee
Last updated
8 Nov 2026

Report an issue

Status: accepted educational architecture. Module: T18.6. Date: 2026-11-02. Supersedes: none; retain ADR-001–004 contracts.

Situation and Alternatives

A hypothetical Tenant A burst crowds out B/C while the shared model budget remains fixed. Bigger app replicas and larger global queues do not add provider request/token/concurrency capacity. Compare FIFO, strict priority, RR, and WRR under explicit equal-cost assumptions. Choose bounded per-tenant queues and work-conserving WRR with hypothetical weights 2:1:1; retain FIFO within each tenant.

Decision

Validate trusted scope, operation identity, size, token estimate, and deadline. Bound admission/waiting storage per tenant and globally. Before provider dispatch, atomically reserve request credits, token credits, and a global concurrency slot under consistent authority/configuration. Only a fresh accepted reservation permits a new call. Duplicates inspect/replay existing state; conflicting payloads reject.

Expire unstarted work at its deadline. Settle slots only on known completion; preserve unknown external outcomes and stable operation identity for reconciliation. Backpressure must reach clients and ingestion workers without unsafe new-key or cross-layer retry amplification. Preserve authorised retrieval, cache live gates, worker epochs, shared history, and commit-before-success.

Evidence and Boundaries

The runnable SQLite guard tests shared local reservations under contention and independently binding budgets. The deterministic finite-burst scheduler tests queue cap/deadline and four scheduling policies with equal one-second jobs. A test combines a grant with the actual T18.3 durable worker store. This is not an atomic distributed dispatcher, a real provider quota implementation, a tokenizer, a network broker, or a GPU scheduler.

Claim/grant/start/result/settlement remain separate durable boundaries. A crash may strand a slot conservatively; automatic reconciliation and distributed ownership are not implemented. Local lease expiry does not terminate a remote provider call. Capacity and fairness guarantees apply only to the model assumptions; variable token/compute work needs cost-aware policy and measurement.

Operations and Revisit

Observe offered/admitted/rejected/expired/completed work, per-tenant share and oldest age, reserved/observed tokens, active/unknown slots, retry amplification, and provider outcomes. Fix the binding budget or admission policy before scaling app replicas. T18.7 expands enterprise identity/governance without bypassing these limits; higher-depth gateway/inference architecture remains assigned to T18.10.

Key Takeaways

  • A tenant burst is handled with bounded per-tenant queues and work-conserving weighted round robin at hypothetical weights 2:1:1, FIFO within each tenant. A provider call needs a fresh atomic reservation of request credits, token credits and a global concurrency slot.
  • Bigger replicas or a larger global queue are rejected because neither adds provider request, token or concurrency capacity. FIFO, strict priority and plain round robin were compared under the same equal-cost assumptions.
  • Slots settle only on known completion, so a crash may strand one conservatively and unknown outcomes wait for reconciliation. Variable token or compute cost needs a cost-aware policy, and the binding budget is fixed before app replicas scale.

Rate your confidence and keep notes on the module page.