LLM Inference Serving

Source: Sources and Full Design, especially §§1–11 for the anchor and §§12–15 for variants. Use the Deep Dive for scheduling, KV (key–value) and failover mechanics and the Cheat Sheet for rehearsal. The source states that its serving design and all capacity figures are constructed examples, not measurements of an actual engine.

The anchor asks for end-to-end batching of LLM (large language model) requests, including which GPU has capacity and what happens when one fails. A GPU is a stateful server with a KV-memory budget. Prefill drives time to first token; decode drives per-token speed and completion time.

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Validate and tokenize requests.
2. Apply per-tenant token budgets.
3. Route to a healthy GPU with free KV capacity.
4. Continuously batch variable-length generation.
5. Demultiplex by request ID; stream and cancel.
6. Handle replica / zone failure; reject overload early.
7. Canary/drain model versions and split queue vs service metrics.

Non-functional requirements — how well / under what constraints

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, admit, place on a replica with KV room, batch, stream, cancel, fail loud. Non-functionally, TTFT and TPOT SLOs, no lost streams, and tenant fairness — the model only does prefill and decode.”

The LLM performs prefill and token-by-token decode. Code owns request identity, quotas, admission, placement, scheduling policy, health, retry budget and response routing. Replica-local memory checks have final authority over whether a sequence fits.

Step-by-step architecture

• The gateway authenticates, assigns a request ID, validates limits and tokenizes on CPU so estimated token work is known before GPU use.
• Admission charges tenant tokens, places accepted requests in a bounded priority queue and returns a clear retry/defer signal when the SLO cannot be met.
• The router samples healthy replicas using free KV blocks, queue length and shared-prefix warmth; the replica performs the final capacity check to avoid stale-registry herding.
• A replica scheduler continuously adds waiting sequences and removes finished ones at decode steps; chunked prefill protects ongoing token streams from a long incoming prompt.
• The GPU model runs prefill and decode while paged KV blocks are allocated on demand. Tokens are matched to callers by request ID, detokenized and streamed.
• Health probes evict sick replicas. Unstreamed requests may retry under a budget; after partial output, the product either resumes by re-prefilling with caveats or returns an explicit partial error.
• A client disconnect or stop request cancels model/tool work and frees KV. Rollouts drain replicas and canary new versions rather than dropping active requests.

Continuous versus static batching: static generative batches wait for the longest output; short requests waste slots. Iteration-level batching lets each sequence leave as soon as it finishes. A size-or-time flush suits one-GPU fixed-shape synchronous work; variable-length generation still benefits from continuous batching.

KV memory is capacity: the source's illustrative 32-layer, eight-KV-head, 128-dimension fp16 model uses about 128KiB of KV per token. A 1,250-token sequence uses ~160MB; an 8,000-token sequence uses ~1GB. Paged allocation reduces reservation waste and fragmentation. The router must use free KV, not round-robin or only running-request count. Prefix affinity saves prefill only while the preferred replica has headroom.

Failover: liveness alone misses a hung GPU. Add readiness and a one-token deep probe. A failed replica loses its KV. Before any token streamed, retry elsewhere. After streaming, rebuilding state from prompt plus emitted text very likely matches under greedy decoding, but batch size and GPU kernels can still shift the floating-point results; sampling can diverge, so an explicit partial error may be the honest product choice. Cap retries and retain spare zone capacity to avoid a retry storm while replacement GPUs warm.

Cost: measure goodput per GPU-hour, batch occupancy, KV use, queue wait and cost per million output tokens. Cap output and cancel abandoned streams before buying more GPUs. Move non-interactive work to a batch tier and use prefix caching only on identical prefixes.

For #75, a single GPU batcher flushes at 100 requests or a short wait; each synchronous caller has a request-ID future and timeout, and the queue is bounded. For #77, price the workload before saying 100K rps: the source's assumed chat shape implies ~21,400 replicas at 70%, while short completions imply ~2,570. Use independent cells and global backpressure so one hot tenant or deploy does not damage the fleet. For #79, lead a review with the three highest risks and their fixes.

Track p50/p95 TTFT, TPOT, queue wait, service time, prefill/decode throughput, prefix hit rate, KV occupancy/preemptions, cancellation delay, explicit failures and goodput (requests finished within SLO). Load-test actual prompt/output distributions, zone failure, stale capacity heartbeats, long-prefill interference and overload before claiming the illustrative replica count works.

“I would ask for input and output token distributions and the first-token and per-token targets before sizing. I tokenize at the gateway, admit by per-tenant token budget and bounded queue, then route by free KV capacity rather than round-robin. Each GPU schedules continuously, admitting and finishing sequences at decode steps, with chunked prefill and paged KV to protect latency and memory. Tokens are demultiplexed by request ID and streamed; disconnect cancels the sequence. A capacity registry plus readiness and one-token probes evicts sick GPUs. Unstreamed work retries under a budget; streamed work has an explicit continuation or partial-error policy. The source’s 214-replica example is arithmetic on assumed throughput, so I would validate it with goodput, TTFT, TPOT and KV use on the actual model.”

Read with the unchanged source. The Main guide gives the interview route; the Cheat Sheet is the recall card. Serving-engine mechanisms and all numeric rates in the source are its own illustrative construction.

Prefill processes input tokens and drives time to first token (TTFT); decode adds output tokens one at a time and drives time per output token (TPOT) and total duration. Size with replica-seconds: under assumed H100-class rates, 1,000 input/20,000 prefill + 250 output/2,500 aggregate decode = 0.15 replica-seconds. At 1,000 rps and 70% utilisation, ~214 replicas. Little's Law gives ~7,000 concurrent sequences for a seven-second stream; 64 slots/replica suggests ~110 replicas by slots, so the larger compute estimate binds. Both calculations depend on actual model, hardware, batch occupancy and token distributions.

CPU tokenization at the gateway permits token-weighted admission and estimated KV (key–value) placement, and rejects huge prompts before the GPU queue. Incremental detokenization buffers incomplete characters and checks stop sequences on assembled text. Measure queue wait separately from service time or a peak overload will masquerade as a slow model.

Static batching waits for the longest generative output and wastes slots; dynamic size-or-time batches suit fixed-shape classifiers. Continuous batching schedules each decode iteration, dropping completed sequences and admitting new ones. Chunk long prefills to limit interference with existing streams, trading some new-request TTFT for better TPOT. Tune batch/concurrency limits against goodput, not raw token throughput alone.

For the source's illustrative 32 layers, eight KV heads, dimension 128, fp16 cache: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128KiB per token. At 1,250 tokens that is ~160MB/sequence; 64 sequences ~10GB. At 8,000 tokens, ~1GB/sequence; context length can halve or worse the concurrency that fits. Paged KV blocks allocate as generated rather than reserving maximum length and can share identical prefix blocks. When full, preempt lower-priority sequences and recompute their KV later (vLLM V1's default; swap-to-CPU was the V0 option) rather than crashing the replica.

Each replica heartbeats free KV blocks, waiting/running sequences and health. The router samples two candidates and chooses the better to reduce herding on stale registry data; prefix affinity is a preference, not permission to queue on a hot replica. The replica makes a final fit check. Autoscale from queue depth and KV pressure, with reserved headroom because provisioning a replica, pulling the image and loading weights often takes minutes.

Health has process liveness, model-loaded readiness and a one-token deep probe for hung devices. A failed replica loses its KV state. Before streaming, retry from scratch elsewhere. After streaming, re-prefill the original prompt plus emitted text if continuation semantics allow; greedy decoding very likely reproduces the continuation, though batch size and GPU kernels can still change the floating-point results, while sampling may diverge. Otherwise return an explicit partial error. Budget retries, use jitter and request-ID dedupe, and retain N+1 zone headroom so a failure does not trigger a retry storm.

Admission uses estimated tokens per tenant, a bounded queue and 429/Retry-After or batch deferral. A queue deeper than the SLO (service level objective) can tolerate only converts overload into late failures. Backpressure must propagate from replica saturation to router and gateway. Interactive work outranks offline jobs under load. On disconnect, make sure every hop passes it on, so the engine cancels the sequence and frees KV blocks at the next scheduler step.

One-GPU #75: assign a request ID and future, flush a batch when 100 inputs arrive or oldest wait reaches a timeout, run GPU, then resolve each future by ID. The source assumes a full batch of 100 in 200ms, giving a theoretical 500 rps; 400 rps is 80% and may queue badly in bursts. Bound the queue, e.g. two batches, and time out synchronous callers. Variable-length output still favors continuous batching.

For #77 at 100K rps with the source's assumed rates, 1,000/250-token chat needs ~21,400 replicas, 200/20 short completion ~2,570, and 500/1 scoring ~3,630 at 70%. The huge spread is the point: price request shape first. Partition the fleet into independent cells with local routers/registries, a global balancer and per-cell backpressure; reserved headroom absorbs bursts and failures.

For #79, review a junior design by ranking the top risks: static batches for variable output, unbounded global FIFO (first in, first out) and round-robin placement. Then inspect size-only flush, weak health checks, unlimited retries, missing request IDs, no cancel and GPU-side tokenization. Explain each failure and its concrete correction rather than reading a long checklist.

TTFT regression: inspect client/network, CPU tokenization, queue, long-prompt prefill and KV preemption. TPOT regression: inspect batch size, decode saturation and prefill interference. Slow after last token: detokenization, safety/schema post-process or a buffering proxy. Peak-only high p99 with flat p50 points toward queueing. Goodput is requests finished within SLO; raw throughput can rise while useful throughput falls.

For a high GPU bill, measure goodput/GPU-hour, occupancy, KV utilisation and tokens generated after disconnect. Continuous batching, output caps, identical-prefix reuse, appropriate quantization and off-peak batch work are levers. Changing the model or buying GPUs before measuring occupancy may leave the bottleneck untouched.

Main · Deep Dive · Unchanged source

Input/output tokens p50/p95? Interactive or batch? TTFT (time to first token) or TPOT (time per output token) target? Variable output? Peak, tenants, priorities? Mid-stream failover contract?

Gateway tokenize + ID → token-weighted bounded admission → router by free KV → replica continuous batch → LLM prefill/decode + paged KV → demux by ID → stream
Health evicts sick GPUs; disconnect cancels and frees KV (key–value).

1000/20000 + 250/2500 = 0.15 replica-seconds. 1000 rps × 0.15 / 0.70 ≈ 214 replicas. ~7,000 in-flight streams ÷ 64 slots ≈ 110 slot-limited replicas; compute binds. Source rates are illustrative (H100-class GPU), not benchmarks.

• Prefill → TTFT; decode → TPOT and total time.
• Static generative batches wait for the longest output; continuous batches release finished sequences.
• KV memory, not request count, decides GPU placement; source example ≈128KiB KV/token.
• Unstreamed failed requests retry under budget; streamed ones need explicit continuation/partial-error semantics.
• Refuse early with bounded queues; optimise goodput, not raw throughput.

#75: one GPU, 100-or-timeout batch, request-ID futures. #77: 100K rps needs request-shape pricing and independent cells. #79: critique static batches, unbounded FIFO (first in, first out) and round-robin first. #74: split queue/prefill from decode/post-process latency.

60-second close: “Tokenize and admit before GPU work, route by free KV, schedule continuously, stream by request ID, cancel abandoned work and handle GPU failure with a bounded, honest retry policy.”

A bigger batch buys throughput, and every request inside it pays for that throughput in latency.

These prompts look like AI questions and are graded as distributed-systems questions: queues, batches, load balancing, health and backpressure. Treat the GPU as an expensive, stateful server with a memory budget, and the rest follows. This page consolidates the group's prompts into one read for the day before.

Read this first. No worked design exists for this group, and the sources have no notes on continuous batching, paged attention or any serving engine. The design on this page is therefore own construction. It is built from general serving practice. It is grounded where the sources do speak: the air-gapped chapter's replica memory and sizing formula, and the cost playbook additions on prefill, decode, Little's Law and goodput. Every number below is a stated assumption, never a benchmark. In the room, say "I would replace these with a load test of the actual model."

The request shape sets the design, and "100,000 requests per second" means nothing until the tokens per request are known. A 20-token completion and a 2,000-token essay differ in decode work by two orders of magnitude.

Ask five questions first, and state an assumption for each one the interviewer declines (own construction).

The weak answer draws a load balancer, a queue and a pool of GPUs, then stops. It never says what a batch is or why a GPU fills up. The strong answer names the two phases of inference in the first two minutes (from the additions file, section B1). Prefill processes the whole prompt in parallel and sets time to first token. Decode generates one token at a time and sets tokens per second. Every later decision is a way of scheduling those two phases on scarce memory.

"A GPU is a stateful server with a memory budget. Prefill sets first-token time, decode sets streaming speed, and the KV (key–value) cache decides how many requests fit at once. I'll design the scheduler around those three facts."

A requirement that cannot fail a test is a preference. "Fast" is a preference. "First token under 500 ms at p95 at 1,000 rps" is a constraint. The split and the numbers are own construction; the nouns come from the prompts.

The must-haves come straight from #76's wording. Accept generation requests with a prompt, a token limit, sampling parameters and a stream flag. Batch them onto GPUs. Route each batch or sequence to a GPU with capacity. Deliver the response to the right caller, streamed or whole. Detect a failed GPU and move its work elsewhere. Reject or queue excess load rather than collapse under it.

The should-haves make it operable. Per-tenant quotas and priorities. Cancellation when a caller disconnects. Prefix reuse for repeated system prompts. Rolling model upgrades without dropping traffic. Metrics that separate queue wait from service time.

Every must-have then needs an owner in the architecture (own construction).

GPU count is the answer, never the method. The method converts requests into token work, then token work into replicas. The air-gapped chapter gives the formula verbatim:

Replicas = ⌈ (QPS × Tokens_request) / (TokensPerSecond_replica × UtilizationTarget) ⌉

That chapter also warns that a candidate who talks GPU count without throughput misses the bottleneck. One who talks memory without throughput misses the placement constraint. Both checks follow.

Step one: token work per request. Prefill and decode run at very different rates, so price each request in replica-seconds. Every rate below is an assumption for an illustrative 8B-class model on one 80 GB H100-class GPU; an A100's peak compute could not reach this prefill rate (own construction).

One request costs 1,000 ÷ 20,000 + 250 ÷ 2,500 = 0.05 + 0.10 = 0.15 replica-seconds. At 1,000 rps that is 150 replicas of pure work. Divided by 0.70, provision about 214 replicas. Decode is two-thirds of the bill, so output tokens are the lever that matters most.

Step two: concurrency by Little's Law. Work in flight equals arrival rate times time in system (additions file, section B3). Each sequence streams at about 2,500 ÷ 64 ≈ 39 tokens/s, or roughly 26 ms a token. A request therefore lives about 0.5 s to first token plus 250 × 26 ms ≈ 6.4 s, so ~7 s in total. At 1,000 rps that is ~7,000 sequences in flight. At 64 slots a replica, that needs ~110 replicas' worth of slots.

Step three: take the larger number. Compute says 214 and slots say 110, so compute binds. Provision about 214, then add N+1 per zone for failover. Say both checks aloud. A reviewer trusts the number that survived two independent methods.

Step four: check memory. The KV cache stores each sequence's attention keys and values. It grows with every token (section 7 does the arithmetic). The air-gapped chapter's replica budget, kept verbatim, shows the shape. Quantized weights 12 GB, runtime overhead 2 GB, KV cache and activations 6 GB, batch buffers and fragmentation slack 2 GB. That totals roughly 22 GB, too tight for a 24 GB card. The same chapter's rule applies here: if the replica count exceeds the pool, challenge the request profile rather than ask for more hardware.

The organising rule is admit, then route, then batch per GPU. Admission and routing are global and stateless. Batching is local, because only the replica knows its own memory.

The whole system (own construction), with the control plane above and the request path below:

 ╔══════════════════════ CONTROL PLANE (changes are deployments) ═══════════════════════╗
 ║  model registry + versions · autoscaler (queue depth, KV use) · tenant quotas         ║
 ║  routing policy · health policy · SLO targets · rollout controller (drain, canary)    ║
 ╚═══════════════════════════════════════╤═══════════════════════════════════════════════╝
                                         │ configures every box below
 ╔══════════════════════ DATA PLANE (per request) ═══════════════════════════════════════╗
 ║                                                                                       ║
 ║ client ─> GATEWAY ─> ADMISSION ────────────> ROUTER ────────────────> REPLICA k      ║
 ║           authN      token bucket per         reads capacity           ┌───────────┐ ║
 ║           validate   tenant, weighted by      registry: free KV,       │ SCHEDULER │ ║
 ║           TOKENIZE   estimated tokens;        queue, running seqs;     │ continuous│ ║
 ║           (CPU)      bounded queue;           power-of-two choice      │ batching  │ ║
 ║                      429 + Retry-After        + prefix affinity        │ ┌───────┐ │ ║
 ║                        │ reject                    │                   │ │  GPU  │ │ ║
 ║                        v                           │                   │ │prefill│ │ ║
 ║                   shed or defer                    │                   │ │decode │ │ ║
 ║                   to batch tier                    │                   │ │KV pool│ │ ║
 ║                                                    │                   │ └───────┘ │ ║
 ║                                                    │                   └─────┬─────┘ ║
 ║  CAPACITY REGISTRY <── heartbeat every ~1 s ───────┼─────────────────────────┤       ║
 ║  (free KV blocks, waiting, running, healthy)       │                         │ tokens║
 ║                                                    │                         v       ║
 ║  HEALTH CHECKER ── liveness · readiness · 1-token probe ──> evict / drain    │       ║
 ║                                                                              │       ║
 ║ client <── STREAM (SSE) <── DETOKENIZE + post-process <── DEMUX by request_id┘       ║
 ║            cancel on disconnect ──────────────────────────> free KV slot now          ║
 ║                                                                                       ║
 ║  OBSERVABILITY: TTFT · TPOT · queue wait vs service time · goodput · KV use · errors  ║
 ╚═══════════════════════════════════════════════════════════════════════════════════════╝

The same flow for viewers that render Mermaid (own construction):

Read the components in request order (own construction).

Point at three boundaries while the diagram is up. Admission is the only place load is refused, so no queue downstream grows without limit. The router decides where but never how, because only the replica sees its memory. And the request ID is the thread through everything, so a response can never reach the wrong caller.

Tokenization is CPU work, and CPU work on the GPU's request thread steals time from the most expensive machine in the building. Move it to the gateway.

Tokenizing early buys three things. The admission controller can charge a tenant by real tokens, not by request count. The router can estimate KV memory before it picks a replica. And a malformed or oversized prompt fails before it reaches a queue. Cap input length at the gateway and return a clear error, because one 100,000-token prompt can monopolise a replica's prefill.

Detokenization runs at the other end, incrementally. Some tokens are partial characters, so the streaming layer must buffer until a complete character exists. Stop sequences are checked on text, not tokens, for the same reason.

A GPU is efficient only when it multiplies many rows at once, so batching is the whole point. The question is when a batch forms and when it ends. Three answers exist (own construction; general serving practice).

Static batching fails generative traffic for one reason. A batch finishes when its longest member finishes. A 10-token answer batched with a 1,000-token answer waits for the long one, and its slot sits idle for 990 steps. Continuous batching fixes this by scheduling at the level of a single decode step. Every iteration, the scheduler removes finished sequences and admits waiting ones into the freed slots.

Prefill complicates the picture. A long prompt's prefill can stall every decoding sequence in the batch for tens of milliseconds, which shows up as a per-token latency spike. Split long prefills into chunks and interleave them with decode steps. This trades a little first-token time for the new request against smooth streaming for everyone else. Say that trade aloud; it is the same throughput-versus-latency tension the tagline names.

Batch size is a knob, not a constant. Larger batches raise throughput and raise per-token latency. Tune it against the TPOT (time per output token) SLO (service level objective), and let the scheduler cap concurrency when per-token latency drifts up.

The KV (key–value) cache, not compute, usually caps how many sequences fit on a GPU. Every generated token adds keys and values for every layer, and they stay resident until the sequence ends.

The arithmetic, for the illustrative model (own construction; the architecture numbers are assumptions):

Per token: 2 (keys and values) × 32 × 8 × 128 × 2 bytes = 128 KiB. The anchor sequence of 1,250 tokens needs about 160 MB. Sixty-four of them need about 10 GB, which fits beside 16 GB of fp16 weights on an 80 GB card. Now raise the context to 8,000 tokens. One sequence needs about 1 GB, and a ~50 GB KV pool holds only ~50 sequences. Context length divides concurrency. This is why the router must know free KV, not just request counts.

Reserving each sequence's maximum length up front wastes most of that memory, because most sequences stop early. Paged attention allocates KV in small fixed-size blocks, such as 16 tokens, on demand. Blocks need not be contiguous, so fragmentation mostly disappears. Blocks can also be shared. Requests with an identical system prompt point at the same prefix blocks, which is prefix caching at the serving layer.

When the pool fills anyway, the scheduler must choose a victim. Preempt the lowest-priority or youngest sequence. Recompute its KV later, which is vLLM V1's default, or, in engines that support it, swap it to CPU memory. Never let an allocation failure crash the replica, because that turns one long prompt into an outage for 64 callers.

Round-robin assumes every request costs the same, and here none do. The follow-up "how do you determine which GPU has capacity?" is answered by a capacity registry (own construction).

Each replica publishes its state on a heartbeat of about one second, and piggybacks it on every response. The state is free KV blocks, sequences running, sequences waiting, and health. The router scores a replica by the free KV left after reserving the new request's estimated tokens. Queue length breaks ties.

Registry data is always slightly stale. If every router picks the single best replica, they all pick the same one and herd onto it. Sample two replicas at random and send to the better of the two. This "power of two choices" spreads load almost as well as perfect knowledge and tolerates stale data. The replica makes the final check. If it cannot fit the request, it rejects quickly and the router tries the next choice.

Prefix affinity layers on top. Hash the prompt's shared prefix, such as the system prompt, to a preferred replica, so its cached prefix blocks get reused. Override affinity when the preferred replica is hot. A cache hit is not worth a queue.

A GPU fails in ways a process check cannot see. The process stays alive while the device throws memory errors or hangs mid-kernel. Health checks therefore need three layers (own construction).

A failed replica takes its KV cache with it, so in-flight sequences cannot continue elsewhere. The recovery depends on what the caller has already seen. If no token has streamed, retry the request from scratch on another replica. The caller sees only extra latency. If tokens have streamed, re-prefill the prompt plus the emitted tokens on a new replica and continue. Greedy decoding makes that continuation very likely to match, but not guaranteed, because batch size and GPU kernels can change the floating-point results. With sampling, the text may diverge. The honest alternative is an explicit error with the partial output, and the product decides which.

Retries need a budget. When a zone degrades, naive retries multiply load exactly when capacity drops, which is a retry storm. Cap retries at a fixed share of traffic, such as 10%, with jittered backoff. Key them by request_id so a duplicate is detected, not re-run. Keep N+1 headroom per zone, and fail over between zones at the router. A new GPU replica often takes minutes to provision, pull the image and load weights, so headroom, not autoscaling, absorbs a sudden loss (the cold-start driver, CORE_8 drivers file).

An unbounded queue converts overload into latency for everyone, then into timeouts for everyone. Refuse work early and explicitly instead.

Admission control sits at the gateway. Each tenant holds a token bucket charged by estimated tokens, not by request count, so one tenant's long prompts cannot starve another's short ones. The queue behind it is bounded, and a full queue returns 429 with a Retry-After header. The bound follows from Little's Law: a queue deeper than the SLO allows will only ever serve requests that have already failed their SLO.

Backpressure flows upstream. Replicas signal saturation through the registry. The router stops sending, admission tightens, and the gateway rejects. Shed by priority. Interactive traffic stays and offline traffic moves to a batch tier, which the provider-limits scenario in G14 answers the same way (additions file, section E). The same rule appears in G10 at batch scale: admission control belongs before the quota breach, not after.

Streaming changes perceived latency, not total latency. A user reading the first sentence after 500 ms experiences a fast system, even if the answer takes seven seconds to finish.

Stream over server-sent events or a websocket, and flush per token. Watch for proxies that buffer responses; one buffering hop turns streaming back into a seven-second wait. Report TTFT (time to first token) and TPOT as separate SLOs, because they break for different reasons (additions file, section D).

Cancellation is a cost control, not only a courtesy. When a caller disconnects, the sequence can keep decoding if any hop fails to pass the disconnect on, burning GPU time on tokens nobody reads (additions file, section E). Propagate the disconnect to the scheduler, which evicts the sequence and frees its KV blocks at the next step.

#75 is the anchor with the distribution removed: one GPU, up to 100 inputs per batch, callers who block and wait. The whole design becomes intake, a batcher, the GPU and a way back to the right caller.

The mechanism (own construction). The HTTP handler assigns a request_id, parks a future for it, and enqueues the request. A batcher thread flushes when it holds 100 requests or when the oldest has waited, say, 20 ms. The GPU worker runs the batch. The demultiplexer resolves each future by request_id, and the handler returns. A timeout on the future turns a stuck batch into an error rather than a hung connection.

Say the arithmetic. Assume a full batch of 100 takes 200 ms. Maximum throughput is then 500 requests per second. At 400 rps the GPU runs at 80% and queues start to form in bursts. Latency is wait time plus queue time plus 200 ms. Synchronous callers each hold a connection, so by Little's Law 400 rps at ~0.3 s holds about 120 open connections. That is trivial until overload, when it grows without limit. Bound the queue at two batches, 200 requests, and return 503 with Retry-After beyond that.

One follow-up changes the answer. If the model generates variable-length output, a static batch of 100 finishes at its longest member. Move to continuous batching from section 6, even on one GPU.

#77 is the anchor at a scale where the request shape decides everything. Price it before drawing anything, using section 3's replica-seconds (own construction; same assumed rates).

The spread is almost an order of magnitude. The air-gapped chapter's pointer applies directly: expect numbers that produce an absurd replica count, and challenge the request mix instead of defending the count.

The architecture adds four things at this scale.

Cells. Split the fleet into independent cells of a few hundred replicas, each with its own router and registry. A global load balancer spreads traffic across cells and regions. A bad deployment or a hot tenant then damages one cell, not the service.

Admission by tokens. At 100,000 rps, request counts lie. Charge every tenant's bucket by estimated tokens, and shed the lowest priority first.

Backpressure across layers. Cells report saturation upward. The global balancer shifts traffic before a cell's queue deepens.

Headroom over autoscaling. New GPU replicas often take minutes to provision and load, so autoscaling follows the daily curve while reserved headroom absorbs bursts. Prefix caching and short output caps are the cheapest capacity at this size, because they cut replica-seconds per request directly.

#79 is not a design question. It hands over a design and asks what to change, so the anchor becomes the rubric. Open with what works. Clarify the requirements the design assumed. Then name the top three risks by severity, not every flaw found.

The checklist (own construction):

Deliver it as a conversation, not a verdict. A review that ranks three fixes and explains the failure each prevents shows seniority. Eleven findings read aloud shows only that the list was memorised.

#74 asks for the full stack: tokenization, network, batch size, KV (key–value) cache, post-processing. G14 section 12 answers it from the application's traces. This section answers it from inside the serving layer. First ask which latency is slow: first token, per token, or total (from question 9 of the decomposition questions file).

The rule from the additions file carries the diagnosis. A frozen screen points at the prompt, the queue or prefill. An answer that starts fast then drags points at output length. The peak-only follow-up gets the answer from G14. Separate queue wait from service time, then add admission control and shed non-interactive traffic.

The hour belongs to batching, routing and failover, because those are the words in the prompt. Spend it in this order (own construction).

The two-minute summary to rehearse:

"I'd treat each GPU as a stateful server with a memory budget. Requests are tokenized at the gateway and admitted against per-tenant token buckets, with a bounded queue that returns 429 early. A router picks a replica from a capacity registry of free KV blocks. It uses power-of-two choices so stale data doesn't cause herding, and prefix affinity so shared prompts reuse cache. Each replica runs continuous batching on a paged KV pool, with chunked prefill so long prompts don't stall streams. A three-layer health check evicts bad GPUs. Un-streamed requests retry under a budget, and streamed ones re-prefill or fail explicitly. At a thousand requests a second with this shape, that's about 214 replicas plus N+1 per zone. The metric that would prove me wrong is goodput, not throughput."

The pivot after a good design is "the GPU bill is too high." There is no playbook drill for this group, so the card is own construction. It is built from the output-token and cold-start drivers and the additions file.

Decode was two-thirds of the anchor's replica-seconds in section 3, so output length is the first number to cut. Cancellation on disconnect is the second, because it recovers tokens nobody reads. Every strong cost answer follows four verbs in order. Measure, with occupancy and goodput. Route, sending offline work to the batch tier. Bound, with output caps and admission. Cache safely, sharing prefix blocks only for identical prefixes.

Every level can draw a queue in front of GPUs; the level shows in whether each GPU is treated as a stateful server with a memory budget, and how far the plan reaches past one cell.

• Request shape comes first, because prefill and decode price input and output tokens differently.
• Requirements name TTFT, TPOT, goodput and a utilisation ceiling, each with a number a test can fail.
• Sizing converts requests to replica-seconds, then cross-checks concurrency with Little's Law, and the larger answer wins.
• The architecture admits globally, routes by capacity, and batches locally where memory is visible.
• Tokenization runs on CPU at the gateway so admission and routing can use real token counts.
• Continuous batching lets each sequence leave when it finishes, and chunked prefill protects streaming.
• KV memory divides concurrency by context length, and paged blocks remove fragmentation and share prefixes.
• The router picks by free KV with power-of-two choices, and the replica makes the final check.
• Failover needs a deep health probe, a retry budget, and a rule for requests that have already streamed.
• Admission refuses work early by tokens per tenant, and backpressure flows upward to the gateway.
• Streaming fixes perceived latency, and cancellation on disconnect saves real GPU time.
• The single-GPU variant is size-or-timeout batching, futures keyed by request ID, and a bounded queue.
• At 100,000 rps the request mix swings the fleet by an order of magnitude, so price it before drawing it.
• A design review ranks three fixes by severity, using the anchor as the rubric.
• Latency diagnosis first asks which latency, then walks tokenization, network, queue, batch, prefill, KV, decode and post-processing.
• The forty-five minutes go to batching, routing and failover, closed by the two-minute summary.
• The cost pivot is answered with occupancy, output caps, prefix reuse and a batch tier, proven by goodput per GPU-hour.

1. Why does input length hurt first-token time while output length hurts total time? Prefill processes the whole prompt in parallel and sets time to first token; decode generates one token at a time and sets streaming speed and total time.
2. Size the anchor aloud. 1,000 ÷ 20,000 + 250 ÷ 2,500 = 0.15 replica-seconds; × 1,000 rps = 150; ÷ 0.7 ≈ 214 replicas. Little's Law gives ~7,000 in flight, ~110 replicas of slots, so compute binds.
3. Why does static batching fail generative traffic? A batch ends when its longest sequence ends, so short answers wait and their slots sit idle.
4. How does context length change concurrency? At 128 KiB a token, an 8,000-token sequence needs about 1 GB of KV, so a ~50 GB pool holds about 50 sequences.
5. How is "which GPU has capacity?" answered? Replicas heartbeat free KV, running and waiting counts to a registry. The router samples two, picks the better one, and the replica rejects quickly if the request does not fit.
6. A GPU dies mid-generation. What happens to its requests? Un-streamed ones retry from scratch elsewhere. Streamed ones re-prefill prompt plus emitted tokens, likely but not guaranteed to match under greedy decoding, or fail explicitly with the partial output.
7. Why charge admission by tokens rather than requests? One tenant's long prompts cost many times another's short ones, so request counts let a heavy tenant starve light ones.
8. In #75, why bound the queue at two batches? A deeper queue only serves requests that have already missed the SLO, and synchronous callers hold a connection each while they wait.
9. What is the first question for #77? What each request is. A chat answer and a one-token score differ by almost an order of magnitude in fleet size.
10. Name three findings to lead a #79 review with. Static batching for generative output, an unbounded global FIFO (first in, first out), and round-robin routing that ignores memory.
11. Latency is fine at p50 and bad at p99 only at peak. Where to look? Queueing: separate queue wait from service time, then tighten admission and shed non-interactive work.
12. What is the cost card's 60-second line? A GPU is paid for by the hour, so idle slots are the bill. Fill batches continuously, cap output, reuse prefixes, quantize, push non-interactive work to a cheaper tier, and prove it with goodput per GPU-hour.

Fact-check sources (checked 27 Sep 2026)

• NVIDIA H100 datasheet — the assumed 20,000 tok/s prefill for an 8B model needs H100-class compute
• Hugging Face tokenizers: Decoders — streaming decode must buffer partial characters
• Agrawal et al. 2024, Sarathi-Serve — a long prefill stalls co-batched decodes; chunked prefill fixes it
• vLLM docs: Optimization and Tuning — V1 preempts by recompute by default; chunked prefill
• Kwon et al. 2023, PagedAttention — KV memory caps concurrency; max-length reservation wastes memory; paged, non-contiguous blocks
• vLLM API: CacheConfig — default KV block size of 16 tokens
• Mitzenmacher 2001, The Power of Two Choices in Randomized Load Balancing — two random choices spread load almost as well as full information
• Mitzenmacher 2000, How Useful Is Old Information? — two choices tolerate stale load data
• Thinking Machines 2025, Defeating Nondeterminism in LLM Inference — greedy output can differ across batch sizes and kernels
• NVIDIA blog: Reducing cold-start latency with Run:ai Model Streamer — weight loading alone can take seconds; provisioning is what takes minutes
• Azure SDK blog: Load models up to 6x faster with Run:ai Model Streamer — faster weight streaming for cold starts
• vLLM docs: OpenAI-compatible server — the engine aborts a request when the client disconnects
• vLLM issue #10087 — abort breaks when middleware fails to pass the disconnect on