Consumer-Scale Chat Service

Consumer chat is ChatGPT-shaped: many people, many turns, stream tokens, remember the thread, don’t go bankrupt on GPUs. The model is stateless; the product is not.

G18 covers one slice: sessions, streaming, admission, moderation, cost. Not training the foundation model.

End to end, as free-tier user Sam:

1. Sam hits send. Admission and quotas — paid may skip the line.
2. We load the conversation (and deletion rules). Not the whole history dumped raw if it won’t fit.
3. Moderation on the way in.
4. A GPU pool streams tokens. Batching and fair queues keep neighbors alive.
5. Moderation on the way out. Tools (files, search, code) are sandboxed and step-capped if they exist.
6. Turn is stored. If inference dies, failover — don’t pretend the chat never happened.

That’s it: admit → load state → infer stream → moderate → persist. “We’ll train a new model in this round” stays out unless they ask.

Source: Sources and Full Design, especially §§1–12. Use the Deep Dive for sizing and trade-offs and the Cheat Sheet for rehearsal. The source builds its design from one-line “Design ChatGPT/Claude” prompts; all scale figures are explicit whiteboard assumptions, not measured service facts.

“Design ChatGPT” asks for a stateful product around a stateless model: conversations, streaming, admission, moderation and token cost at consumer scale. The “Design the Claude chat service” variant uses the same architecture, with more emphasis on batching, GPU capacity, queues and failover. There is no separate worked design for either prompt.

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Send and stream messages.
2. Persist, list, rename, delete conversations.
3. Carry within-conversation context.
4. Enforce user / tier limits.
5. Moderate input and output.
6. Stop on user cancel or disconnect.

Files, search, sandboxed code, inspectable memory, and sharing are follow-on features.

Non-functional requirements — how well / under what constraints

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, stream a turn, store the thread, moderate both ways, cancel cleanly. Non-functionally, first token fast, no cross-user leak, fair queues, and the model stays stateless — the product holds state.”

The LLM (large language model) generates responses; code owns identity, quotas, context assembly, routing, moderation, tool scope, persistence and cancellation. The model is stateless between turns. The product stores history but sends only selected context to the model.

Step-by-step architecture

• Route the client to a healthy region; the gateway authenticates, applies per-user and per-tier token budgets, and checks input moderation before GPU admission.
• The conversation service reads user-scoped durable history and persists the new message; it also saves partial output as tokens arrive for reconnects.
• The context builder sends a versioned stable prompt prefix, rolling summary, recent turns, relevant inspectable memory and user-filtered file chunks. Full history remains in storage, not in every prompt.
• A router selects a model allowed by tier and suitable for the task; a bounded fair queue protects tiers and prefers a warm replica only while it has capacity.
• The LLM pool generates with continuous batching and prefix/KV reuse. Optional search, file or code tools run with step limits; untrusted files never gain instruction authority.
• Output moderation checks streamed chunks before delivery; the gateway sends SSE (server-sent events) tokens to the client and records the partial answer durably.
• Stop or disconnect propagates cancellation to model and tools, freeing GPU/KV capacity. Regional failover is possible but may be cold and slower.

Store everything, send little. A rolling summary prevents context growth from making every turn slower and costlier, but can lose a detail; recent turns and user-visible memory mitigate that. Keep a stable, byte-exact prompt prefix and deterministic tool order for cache reuse. A model cascade may start a cold cache on each model, so compare total cost per successful conversation before routing across models.

Admission versus fairness: a user token bucket decides whether a request enters; a weighted fair queue decides when an admitted request runs. Count tokens, not only messages. Under saturation, reject early with a retry signal rather than growing an unbounded queue. Separate your own quota 429 from provider/pool capacity events.

Trust boundary: user ID comes from the authenticated session, never a model tool argument. Files are scanned and parsed asynchronously, retrieval is user-filtered, and code tools use a credential-free sandbox. Moderate both input and streamed output; calibrate false positives and false negatives. Delete history, memory, files and caches consistently when requested.

Regional failure: each region has independent gateway/inference capacity and a home region for conversation history. Failover may lose warm KV (key–value) state and some asynchronously replicated history, so the first recovered turn can be slower. Residency constraints may restrict destination regions. Use circuit breakers and spare capacity rather than assuming instant GPU startup.

Measure p50/p95 time to first token by tier/region, queue wait, cache-read token share, regenerate and edit-resend rate, abandonment, output tokens after abandon, moderation errors, cost per daily active user and cross-user-read tests. Treat model and system-prompt changes as releases: canary, evaluate quality/safety/latency/cost, and watch cache hit rate. For the Anthropic wording, be ready to zoom into replica capacity, batching and failover from G20.

“I would clarify the product and request shape first, then say the model is stateless but the product is not. A regional gateway authenticates, rate-limits and moderates before GPU admission. A conversation service stores every turn and partial answer, while a context builder sends only a stable prompt prefix, summary, recent turns and user-scoped memory. A tier/task router feeds a bounded fair queue and continuously batched LLM pools. Output is moderated as it streams; disconnect cancels generation and tools. I would size from messages per second to open streams and GPU capacity, but label those figures as assumptions. The hard trade-offs are history versus token cost, cache warmth versus model routing, and fairness versus latency under overload. I would prove the design with first-token latency, abandon rate, isolation tests and cost per daily active user.”

Read with the unchanged source. The Main guide is the interview route; the Cheat Sheet is the recall card. The source's design and sizing are constructed from one-line prompts; validate them against a real workload.

The source assumes 100M daily users, ten messages/user/day, 3× peak, 2,000 input and 400 output tokens per turn. That gives 1B messages/day, ~11,600/s average, ~35,000/s peak, 70M input and 14M output tokens/s at peak. At 50 output tokens/s a 400-token answer takes eight seconds; ~35,000/s × eight seconds ≈ 280,000 concurrent streams. If one replica supports 100 such streams, ~2,800 replicas is only a first sizing pass; actual prefill/decode throughput and KV (key–value) memory may require more. About 2KB/message yields ~2TB of conversation text per day before indexes, replication and retention.

Prefill processes input and drives first-token latency. Decode produces one output token at a time and drives streaming duration. Long history primarily hurts first-token time; verbose answers primarily hurt total time and concurrency. Report both, not only request latency.

Shard conversations by authenticated user_id; persist sent messages and partial answers during streaming. A reconnect resumes a known message ID rather than silently generating another answer. Context assembly uses frozen system prompt, rolling summary, recent turns, user-visible/erasable memory and user-scoped retrieved file chunks. A roughly 15-turn fold is the source's illustrative memory-guide choice, not a universal threshold. Summarisation costs another call and may drop a critical detail; inspectable memory and recent turns help.

Cross-chat semantic memory and episodic memory have different update rules. Stable facts overwrite by stable key; episodic records append and get pruned. Server-side auth determines user scope on every read. Deletion propagates to conversation store, memory, uploaded files and caches. Neither the model nor a tool parameter is allowed to choose a different user ID.

SSE (server-sent events) fits one-way text streaming; a two-way voice path may need a different transport. Persist partial text, detect closed connections, cancel the inference sequence and in-flight tools, and release KV blocks. Streaming helps perceived latency but does not reduce total compute. Track tokens generated after abandon.

Prefix caching is byte-exact. Keep system text, tool order and serialised keys stable; put timestamps and volatile content after cache breakpoints. Prompt-version changes and model changes invalidate prefix reuse. Soft conversation-to-replica affinity helps reuse warm KV but yields to capacity and health. A cheap-to-medium-to-premium cascade may sacrifice enough cached input to erase the nominal model-price saving; compare cost per successful conversation and regeneration rate. A prompt change is a release and a cache-cost event.

Per-user token buckets cap messages and tokens by tier; weighted fair queues keep admitted paid and free work from starving each other. Bound queues; reject or defer before latency exceeds the SLO (service level objective). Distinguish gateway-policy 429 from provider/pool saturation. Tools have per-turn step and time caps. Uploads are scanned, parsed and embedded per user. Fetched pages/PDFs are untrusted data. A code sandbox has no network or credentials and is destroyed afterward.

Input moderation prevents severe requests before GPU use. Output moderation catches responses that expose earlier sensitive content and can halt a stream. Trained refusals, moderation classifiers and application policy are different controls. Evaluate false positives and negatives on labelled samples.

Independent regional serving pools and a home region for history contain failures. Asynchronous replication may leave slightly stale history on failover, and a new region has a cold KV cache. Reserved capacity and a circuit breaker bridge failures while new GPUs warm. Residency rules can limit which region is eligible; do not let failover bypass them.

Observe time to first token by tier/region, queue versus service time, cache-read token share, output length, abandonment, regeneration, moderation outcomes, isolation failures and cost/DAU. Keep raw message text out of ordinary telemetry. Canary model, route and prompt changes and gate on quality, safety, latency and cost. The Claude-service variant asks for the same answer with a deeper dive on batching, GPU capacity registry and cold failover.

Main · Deep Dive · Unchanged source

Consumer or API? DAU (daily active users) and turns? Tiers? History/memory? Files/tools? Serve or train models?

Client → regional gateway/auth/quota/input moderation → conversation store → bounded context → tier/task router → fair queue → LLM pool → output moderation → SSE stream + partial persistence

Disconnect cancels model and tools. User scope comes from auth at every store and retrieval boundary.

100M DAU × ten turns = 1B/day ≈ 11.6K/s average, 35K/s at 3× peak. Eight-second answers imply ~280K open streams; 100 streams/replica implies ~2,800 replicas before token-throughput/KV checks.

• The model is stateless; the product is not.
• Store every turn, send only the summary/recent turns/relevant memory.
• Rate limit decides whether; fair queue decides when.
• Stable byte-exact prefix and warm replica save input prefill; a model cascade may lose that cache.
• Streaming improves perceived latency; cancellation saves actual GPU cost.

60-second close: “Admit safely, assemble bounded user-scoped context, stream from continuously batched pools, persist partial answers and cancel abandoned work. Measure first-token latency, fairness, isolation and cost per user.”

Every user expects the first token instantly and the whole history remembered, while every remembered token is paid for again on the next turn.

"Design ChatGPT" is not a model question. It is a distributed-systems question about conversation state, token streaming, admission control and cost at hundreds of millions of users. This page consolidates the group's prompts into one read for the day before.

No worked design exists for this prompt. Both sources are one-line prompts that candidates report. Everything below the prompt is own construction, assembled from the designs above and this site's material on memory, caching, tenancy, observability and cost. Sizing numbers are stated assumptions, not measurements. Treat each figure as a placeholder to be replaced by measured throughput.

Ask what "ChatGPT" means in this room before drawing anything. The question bank's first failure mode is jumping to architecture before clarifying scope. An OpenAI FDE (Forward Deployed Engineer) candidate reported the interviewer stopping them to ask: "What questions would you ask the customer before designing anything?"

The prompt, from the reported-question bank: "Design ChatGPT." Nothing else is given. The framework on the same page says to clarify the mission first: "Are we optimising for response time, cost, accuracy, or equity of coverage? Who's the primary user?" State assumptions explicitly if the interviewer does not confirm them.

The weak answer draws a load balancer in front of a model and stops. It forgets that the model is stateless and the product is not. The strong answer names the three hard problems in the first two minutes: conversation state that grows every turn, streaming at huge concurrency, and cost that scales with tokens rather than requests.

"The model is stateless and the product is not. The design is conversation state, streaming, admission control and cost, with the model as one expensive dependency behind them."

A requirement that cannot fail a test is a preference. "Fast" is a preference. "Time to first token under one second at p95" is a constraint, and only the second shapes the architecture. The split below is own construction.

The must-haves are six. Send a message and receive a streamed answer. Keep and list conversation history per user, with rename and delete. Carry context across turns within a conversation. Enforce per-user and per-tier usage limits. Moderate input and output. Stop generation on request or when the client leaves.

The should-haves round out a competitive product: file uploads, web search, a code sandbox, cross-conversation memory the user can inspect and erase, regenerate and edit-and-resend, and sharing a conversation by link. Voice and image generation come later.

Every must-have then needs an owner in the architecture (own construction).

Say the arithmetic aloud, because it decides whether multi-region and caching are optional. They are not. Every number below is an assumption for the whiteboard (own construction). The method is what gets scored.

Little's Law is concurrency = throughput × latency, from the cost study guide. It turns latency work into capacity work: halving the answer length halves the concurrent streams to provision. The input line explains why prefix caching matters more here than anywhere. Seventy million input tokens a second are mostly history the model already processed last turn.

Split the two latency phases before going further. Prefill processes the whole prompt in parallel and sets time to first token, so it scales with input tokens. Decode generates one token at a time and sets total time, so it scales with output tokens. A bloated history is a first-token problem. A verbose answer is a completion-time problem.

The organising rule is admit, assemble, stream, persist. Admission control decides whether a request may use a GPU at all. The context builder decides what the turn costs. The stream decides what the user feels. Persistence decides what survives.

The whole system, drawn with its planes (own construction):

 ╔══════════════════════ CONTROL PLANE (changes are releases) ═══════════════════════╗
 ║ model catalogue + routing policy per tier · quotas and rate limits per tier       ║
 ║ system prompts (versioned, prefix-stable) · moderation thresholds · region map    ║
 ║ feature flags · eval gates (G13) · budgets and alerts                             ║
 ╚═══════════════════════════════════════╤═══════════════════════════════════════════╝
                                         │ configures every box below
 ╔══════════════════════ DATA PLANE, per region (calls are requests) ════════════════╗
 ║                                                                                   ║
 ║ client (web, mobile) ──SSE──> edge + GeoDNS ──> STREAMING GATEWAY                 ║
 ║                                                 authN · rate limit · quota        ║
 ║                                                 input moderation · cancel-on-close║
 ║                                                        │ admitted                 ║
 ║                                                        v                          ║
 ║                   CONVERSATION SERVICE ──> CONTEXT BUILDER                        ║
 ║                   (history, titles,        system prompt + summary + recent turns ║
 ║                    partial answers)        + user memory + retrieved file chunks  ║
 ║                        │                        │                                 ║
 ║            conversation store         MODEL ROUTER (tier + task)                  ║
 ║            (sharded by user_id)             │                                     ║
 ║            memory store                     v                                     ║
 ║            (namespaced by user_id)   SCHEDULER + FAIR QUEUE per tier              ║
 ║                                      session affinity -> warm KV cache            ║
 ║                                             │                                     ║
 ║                                             v                                     ║
 ║                          INFERENCE POOLS (small · default · premium)              ║
 ║                          continuous batching · prefix/KV cache                    ║
 ║                                             │ tokens                              ║
 ║                                             v                                     ║
 ║                          OUTPUT MODERATION on the stream ──> gateway ──> client   ║
 ║                                                                                   ║
 ║ TOOLS (bounded steps): web search · code sandbox · file retrieval                 ║
 ║ FILES: upload -> object store -> scan -> parse -> chunk + embed (per user)        ║
 ║                                                                                   ║
 ║ OBSERVABILITY: stage timings, tokens, cache reads, abandon, moderation — no text  ║
 ╚═══════════════════════════════════════════════════════════════════════════════════╝
      replicated across regions: conversation store (home region per user),
      control-plane config; inference pools are regional and independent

The same flow for viewers that render Mermaid (own construction):

Read the components in request order. The failure column is own construction.

Point at three boundaries while the diagram is up (own construction). The admission boundary sits at the gateway, so nothing reaches a GPU without passing auth, quota and input moderation. The isolation boundary sits at every store, because each one is namespaced by a user_id that comes from auth. The cost boundary sits at the context builder, because it decides how many tokens each turn re-sends.

Store everything and send little. The memory study guide draws the line: trimming bounds what reaches the model, not what is stored. A model cannot be billed for history the context builder never sent.

The conversation service writes every message durably, keyed by (user_id, conversation_id). It also persists the partial answer as tokens stream. A disconnected user then reconnects to the text generated so far rather than to nothing. The store is sharded by user_id, so one user's history lives on one shard and a list-conversations call touches one partition.

The context builder assembles each turn in a fixed order: system prompt, rolling summary, recent turns, memory facts, retrieved file chunks, then the new message. A sliding window alone loses anything that scrolls out. Summarisation keeps a condensed record once the buffer crosses a threshold, at the cost of one extra model call per compression. The memory guide's worked answer folds the buffer at about 15 turns, an illustrative threshold rather than a rule, so cost stays flat on long chats. The failure to volunteer is losing a detail the summariser judged unimportant.

Cross-conversation memory is a different store with a different key. A checkpoint key answers "which conversation"; a user namespace answers "which person". Semantic memory holds stable facts under a stable key that overwrites in place. Episodic memory appends one record per completed task and is pruned. The isolation rule from the memory guide is non-negotiable: read user_id from the server-side session on every request, never from a tool argument the model could set. Make memory visible and erasable in the product, because a user who cannot see a remembered fact cannot correct it.

Stream every answer, and treat the closed connection as a cost signal. Streaming buys perceived latency, not real latency. The production study guide makes the point: total generation time is unchanged, but the first token arrives in well under a second instead of after the whole answer.

Use Server-Sent Events for the chat stream. The traffic is one-way from server to client after the request, SSE runs over plain HTTP, so it passes most proxies that don't buffer, and the browser's EventSource reconnects on its own. Keep WebSockets for the paths that genuinely need both directions at once, such as voice. Each stream carries a message ID, so a reconnect resumes from the persisted partial answer instead of regenerating it (own construction).

A user who leaves mid-answer is still costing GPU time. The cost additions name this scenario: "cancel the stream + any in-flight tool calls on abandon — otherwise you pay for tokens nobody read." The gateway detects the closed connection and sends a cancel to the inference request. That frees the batch slot and its KV (key–value) cache and stops any tool call in flight. The Stop button uses the same path. Measure abandonment against first-token time and answer length, because a rising abandon rate is often a first-token problem in disguise. G14's card 15.10 carries the same lesson.

Route on two axes: the user's tier and the turn's difficulty. The tier sets which models a user may reach and at what quota. The task sets which of those models a turn actually needs. A greeting does not need the premium model; a long proof might.

The cost additions quantify the lever. On Claude's current price list, output costs five times input on every tier; other vendors' ratios differ. A router that moves 80% of traffic from Opus 5.5 ($4/$20 per million input/output tokens) to Haiku 4.5 ($1/$5) is a ~4× unit-cost cut on that slice (list prices as of Sep 2026). That single sentence is what "model routing" is worth, and it belongs in the answer.

Then name the trade-off most candidates miss. Prompt caches are scoped to a model, so a cheap-to-medium-to-premium cascade means separate cold prefixes. On a chat product whose input is mostly repeated history, a cascade can forfeit more in cache than it saves in price. The additions give the rule: measure one strong model at lower effort before building the cascade. Route at the start of a conversation where possible, and keep a conversation on one model unless the user changes it (own construction).

A rate limit and a fair queue answer different questions, and a consumer service needs both. G07 section 7 draws the line. A rate limit decides whether this request may proceed at all. A fair queue decides in what order admitted requests are served under load.

Put a token bucket per user at the gateway, sized by tier: messages per window and tokens per day. Count tokens, not only requests, because one long request can cost as much as, say, fifty short ones. Behind admission, feed a weighted fair queue per tier into the shared inference pools. A free-tier surge then waits longer, and paid users keep their latency (own construction, from G07's weighted-fair-queuing argument).

Separate the two kinds of 429 when load spikes. The additions say it plainly: "Separate provider 429s from your own gateway limits." A self-inflicted limit is a policy decision. A provider or pool limit is a capacity event. Short term, retry with jitter onto a fallback model or region and shed non-interactive traffic. Long term, buy capacity and queue non-urgent work rather than retrying into the same wall. Abuse is a rate-limit problem too: per-account and per-IP limits, anomaly detection on bursts, and stricter quotas on new or unverified accounts (own construction).

Moderate input before it reaches a GPU and output before it reaches a screen. The production guide's rule settles why both: the model can repeat something sensitive the user said two turns ago, so an input-only check never catches it. Input moderation buys half the cost; input plus output buys the multi-turn catch. A consumer chat product is multi-turn by definition.

Input moderation runs as a fast classifier at the gateway, before admission, so a blocked message never spends GPU time. Output moderation runs on the stream in chunks and can halt it mid-answer with an explanation. Moderation APIs score a fixed set of categories, and at default thresholds milder or app-specific abuse can still pass. The guide names two independent layers, the model's trained refusals and the moderation classifier, and neither substitutes for an app-specific policy. Track false positives as closely as false negatives, because over-blocking is a product failure users notice first (own construction).

Files and tools carry indirect prompt injection, which the question bank lists as a Tier 2 probe. An uploaded PDF or a fetched web page is data, and data must not gain the authority of instructions. Store uploads in object storage, scan them, parse asynchronously, then chunk and embed per user. Retrieval over them is filtered by user_id at query time, the same two-layer idea as G01 section 6. Tool steps are capped per turn. The code sandbox runs with no network and no credentials, and is destroyed after the session (own construction).

Chat is the best case for prefix caching, because each turn is the previous turn plus a little. The cost additions give the mechanism: cache matching is byte-exact on the prefix, rendered in the order tools, then system, then messages. One changed byte invalidates everything after it.

Build the prompt so the stable parts come first and never move. Freeze the system prompt per version. Order the tool list deterministically. Put timestamps, request IDs and the user's new message after the last cache breakpoint. The additions list the silent invalidators behind a sudden cache-hit drop: a datetime.now() in the system prompt, unsorted JSON keys, a tool list whose order varies, a prompt-version bump, a model swap. Prove it with the cache-read token counter, not by guessing.

A cache is only warm on the machine that holds it. Route every turn of a conversation to the replica that served the previous one, so its KV cache still holds the history (own construction). Session affinity trades some load-balancing freedom for a large cut in prefill work. When the replica is gone, the turn still succeeds, only slower, because the new replica must prefill the whole history again. Summarisation interacts with caching too: rewriting the summary changes the prefix, so fold history in large steps rather than every turn.

Serve each user from the nearest healthy region, and let a regional failure cost latency, not availability. GeoDNS or anycast sends the client to a regional edge. Each region runs its own gateway, services and inference pools, so one region's failure does not queue behind another's (own construction).

Give each user a home region for the conversation store and replicate it asynchronously to a second region. A failover then reads slightly stale history rather than none. Keep the control plane central for policy and regional for enforcement, the balance G07 section 8 argues for. Residency rules can pin some users' data to a region, which the region map in the control plane records.

Failover is cold by nature. The new region has no KV cache for the conversation and may have less capacity. Accept the slower first turn, shed the free tier before the paid tier, and use a circuit breaker on a sick pool so requests stop paying to rediscover a failure. The production guide gives the breaker's three states: closed, open after repeated failures, half-open for one probe. A model outage degrades to another model in the same tier; it never degrades to an error page if any healthy model remains.

Measure time to first token, not only total latency, because first-token time is what a chat user feels. G14's triage section decomposes a slow request by stage: queue wait, prefill, decode, moderation, tools. That trace is the first tool when latency moves. Record stage timings, token counts, cache reads, abandonment and moderation outcomes, and keep message text out of telemetry by default, per G14 section 2.

The question bank's second failure mode is hand-waving evaluation. Tie every design choice to a number (own construction).

Ship model and prompt changes through a release gate with a canary slice, as G13 designs it. A system prompt change is a release, because it also changes the cache prefix and the cost line.

"Design the Claude chat service" is the same prompt as "Design ChatGPT", and the design above answers it unchanged. The sheet records no content delta. The emphasis differs, and saying so shows the candidate read the room (own construction).

The question bank frames the Anthropic questions directly: "AI/ML framing, but they fundamentally test distributed-systems fundamentals: batching, queuing, load balancing." The neighbouring Anthropic prompts are inference batching, GPU routing and a 100,000-requests-per-second token service. Those are group G20 in the sheet. For #78, then, spend more of the hour on sections 3, 8, 10 and 11: the Little's Law sizing, the fair queue, continuous batching with session affinity, and failover. Expect the follow-ups "how do you determine which GPU has capacity?" and "how do you handle failover?" from the batching prompt in the same bank.

The Tier 2 safety probe applies with extra weight here: circuit breakers, rate limiting and graceful degradation when the model is unavailable. Have section 11's degradation ladder ready as a spoken list.

Spend the hour on state, streaming, admission and cost, because those are the parts a generic answer skips. The model choice is a sentence.

The two-minute spoken answer (own construction):

I would start by saying the model is stateless and the product is not. At 100 million daily users and ten messages each, peak is around 35,000 messages a second, and with eight-second answers that is about 280,000 open streams — so admission, streaming and cost dominate. A streaming gateway holds each SSE (server-sent events) connection and does auth, per-user token-bucket limits, tier quotas and input moderation before anything reaches a GPU. The conversation service stores every message durably, sharded by user, and persists partial answers so a reconnect resumes. A context builder sends only what the turn needs: a frozen system prompt, a rolling summary, recent turns and user memory, in a prefix-stable order so prompt caching works. A router picks the model by tier and task, and a fair queue per tier feeds inference pools that batch continuously and keep each conversation on a warm replica. Output moderation runs on the stream and can halt it. When the client leaves, the gateway cancels generation and tool calls, because tokens nobody reads still cost money. Each region is independent, with a home region for history and cold failover. I would prove it with time to first token, regenerate and abandon rates, moderation error rates and cost per daily active user.

The lines that carry the round (own construction):

1. "The model is stateless and the product is not."
2. "Store everything, send little."
3. "Streaming buys perceived latency, not real latency."
4. "Cancel on abandon. Tokens nobody reads still cost money."
5. "A rate limit decides whether; a fair queue decides in what order."
6. "Prefix caching is a byte-exact prefix match. Stable first, volatile last."
7. "A cascade forfeits the cache. Measure one model at lower effort first."
8. "Failover is cold, not closed."

The follow-ups come from the question bank's Tier 2 probes and the neighbouring prompts.

The interviewer's pivot is "the bill is growing faster than users." This group has no playbook drill, so the card below is a self-drill (own construction), grounded in the cost study guide's drivers.

Two drivers from the cost study guide explain almost every surprise here. Input tokens grow with the conversation: teams count the user's question and ignore the system prompt and hidden history, so the bill rises with flat traffic. Output tokens grow with verbosity: teams measure requests, not generated tokens. Streaming improves perceived latency but does not reduce total compute cost. Quote cost per successful conversation rather than per request, as the additions insist: a cheaper model that needs a regenerate is not cheaper.

Every strong cost answer follows four verbs in order. Measure tokens and stage timings first. Route model to tier and task. Bound history, answer length and tool steps. Cache safely, with a stable prefix and the user's scope in every key.

Every level can put a gateway in front of a model pool; the level shows in how the answer handles conversation state, cost and failure at this size.

• Clarify the product and state the assumptions before drawing, because "Design ChatGPT" gives nothing else.
• Requirements are testable: first token under a stated p95, zero cross-user reads, moderation error rates measured.
• The sizing walk from daily users to open streams to replicas shows why admission, streaming and caching are mandatory.
• One diagram shows admit, assemble, stream, persist, with boundaries at the gateway, the stores and the context builder.
• Store every message durably, and send the model only a summary, recent turns and user-scoped memory.
• Stream with SSE, persist partial answers, and cancel generation and tools when the client leaves.
• Route by tier and task, and count the prompt cache a cascade forfeits.
• A per-user token bucket decides admission; a weighted fair queue per tier decides order.
• Moderate input and output, and treat every file and fetched page as untrusted data.
• Keep the prefix byte-stable and each conversation on a warm replica, so history is a cache read.
• Serve per region with a home region for history; failover is cold, not closed.
• Measure time to first token, regenerate and abandon rates, and cost per daily active user, and gate every prompt change.
• The Anthropic wording is the same design, with more time on batching, queuing and failover.
• The hour goes to state, streaming, admission and cost.
• The cost pivot is a stable prefix, stepped summaries, short answers, routing down and cancel on abandon.

1. Why is "the model is stateless and the product is not" the opening line? It names where the design effort goes: conversation state, streaming and admission, with the model as one dependency.
2. Walk the sizing from 100M daily users to open streams. 1B messages a day, about 11,600/s average, about 35,000/s at a 3× peak; at 8 s per answer, Little's Law gives about 280,000 open streams.
3. Why does time to first token depend on history length? Prefill processes the whole prompt before the first token, and history is most of the prompt.
4. What does the context builder send on turn 40 of a conversation? The frozen system prompt, a rolling summary, the recent turns, relevant user memory and file chunks, then the new message, in that order.
5. Why SSE rather than WebSockets for chat? The stream is one-way after the request, SSE runs over plain HTTP, so it passes most proxies that don't buffer, and the browser's EventSource reconnects on its own.
6. What happens when a user closes the tab mid-answer? The gateway cancels the inference request and any tool call, freeing the batch slot and KV cache; the partial answer is already persisted.
7. Why can a model cascade raise cost on a chat product? Prompt caches are scoped to a model, so each model in the cascade starts cold on a prompt that is mostly repeated history.
8. How do a rate limit and a fair queue differ? The limit decides whether a request may proceed; the queue decides the order admitted requests are served under load.
9. Why moderate output as well as input? The model can repeat something sensitive from an earlier turn, which an input-only check never sees.
10. Name three silent prefix-cache invalidators. A timestamp in the system prompt, unsorted JSON keys, a tool list whose order varies (also a prompt-version bump or a model swap).
11. What changes when the prompt is "Design the Claude chat service"? Nothing in the design; more time goes to batching, queuing, GPU capacity and failover.
12. What is the sixty-second cost answer? Chat cost is history times turns plus answers nobody finished: stable prefix, stepped summaries, short answers, route down, cancel on abandon.

Fact-check sources (checked 27 Sep 2026)

• Artificial Analysis LLM leaderboard (TTFT column) — real first-token times are hundreds of ms to over a second, not milliseconds
• WHATWG HTML Standard: Server-sent events — SSE over plain HTTP, EventSource auto-reconnect, proxies that drop idle connections
• vLLM docs: OpenAI-compatible server — disconnect aborts the request and frees its slot; KV reuse is compute, not a per-token price, when self-hosted
• vLLM issue #10087 — abort on disconnect can break behind middleware, so cancellation must be propagated
• Claude docs: Models overview — Opus 5.5 is the current Opus; Opus 5 is legacy
• Claude docs: Pricing — Opus 5.5 $4/$20 and Haiku 4.5 $1/$5 per million tokens (4×); output is 5× input on every Claude tier
• Claude docs: Prompt caching — caches are per model, exact-prefix matched in tools → system → messages order, and broken by volatile content
• OpenAI API docs: Prompt caching — exact-prefix matching and per-model caches on another vendor
• OpenAI API docs: Moderation — moderation returns category scores; thresholds and policy are the app's job