Scale a Prototype from 100 to 100,000 Users

The prototype proved the answer is useful to a hundred forgiving users, and nothing about how it behaves for a hundred thousand who did not ask for it.

Scaling a prototype a thousandfold is not a bigger deployment of the same system. It is a different system that happens to share a prompt. This page is the day-before pack for this case, a standalone case with no group.

The source has talking points for this prompt, not a worked design. Question 17 lists what to cover and one follow-up. Everything that turns that list into an architecture, a sizing walk and a rollout is own construction, built from the sources above. Every number is a stated assumption for the whiteboard, not a measurement.

The prompt, verbatim from the question bank:

"A prototype works for 100 users. The customer now wants to deploy it to 100,000 users."

"Works" is the word to attack first. A prototype is usually judged on a demo and a few enthusiastic users. That evidence says the idea is valuable. It says nothing about cost, tail latency, abuse, or the users who never volunteered. The bank's own list of biggest mistakes includes "Treating a prototype as a production system."

Ask before sizing anything. The assumed answers are own construction, stated aloud so the interviewer can correct them.

The weak answer draws autoscaling and a load balancer. It treats the jump as a compute problem. The strong answer says three things in the first two minutes. The evidence of success does not transfer. The bill scales with request shape, not with users. And adoption is a deliverable, not a side effect.

"A hundred volunteers proved the idea is useful. A hundred thousand employees will test everything the volunteers forgave: cost, the slow tail, permissions, abuse and whether anyone changes how they work."

A prototype hides problems by construction, not by accident. The cost cram sheet names four things demos hide. They hide traffic shape: one user, a warm cache, clean data. They hide tail latency: p95 and p99, not the average. They hide usage expansion: longer questions and bigger uploads, so tokens grow faster than requests. And they hide governance cost: security review, audit logging, evaluation, human review and support.

The same sheet contrasts prototype and production on six axes, verbatim in the table's middle column.

Add three gaps the table does not name (own construction). Prompts drift, because the builder still edits the live prompt by hand. Provider quotas are sized for a pilot account. And the support model is a chat message to the one engineer who built it. At a thousand times the users, that engineer becomes the outage.

The bank has a neighbouring prompt with the same shape: "A team has a successful notebook-based AI prototype. How do you productionise it?" Its discussion list is verbatim: reproducibility, CI/CD (continuous integration and continuous delivery), evaluation gates, model and prompt versioning, secrets management, monitoring, cost controls, security, SLAs (service level agreements), rollback, ownership and support. Answer it with this page. The notebook variant adds one first step: turn the notebook into versioned, tested code before any of the rest applies.

A requirement that cannot fail a test is a wish. "Scalable" is a wish. "Sustains 20 requests a second at p95 first-token under 2 s" is a constraint. The split below is own construction.

The functional core does not change from the prototype. Answer questions from company documents with citations. Call the two read-only tools. Keep a short conversation history. What changes is everything around that core: identity, limits, fairness, failure handling, evaluation, rollout and support.

Every new constraint needs an owner in the architecture (own construction).

Say the arithmetic aloud, because it turns "scale" into three numbers the customer must approve: requests per second, tokens per minute and dollars per month. Every figure below is an assumption (own construction). The method is what gets scored.

Little's Law is concurrency = throughput × latency, from the cost additions. It turns latency work into capacity work. Halving average latency halves the in-flight slots to provision.

Now price it. The cost additions list current tiers: Claude Sonnet 5 at $2.00 input and $10.00 output per million tokens, and Claude Haiku 4.5 at $1.00 and $5.00 (checked Sep 2026 on Anthropic's pricing page). Prices change constantly, so state the mechanism and say "I'd check current pricing" for the digits.

Two points carry the table. First, the bill went up about 300 times, not 1,000 times, because only 30% of licensed users are active. Adoption is a cost variable. Second, the three cheapest levers cut the naive bill by nearly half without touching quality on the hard 30%. The cost additions' rule of thumb is that a real agent request can become 5–20 sub-requests. If the tools grow into an agent, multiply the call count before trusting any of this.

Then check the provider. 7.8M tokens a minute at peak must fit the account's tokens-per-minute and requests-per-minute limits, split by model after routing. Plan for twice the peak, because retries during an incident and the Monday spike stack on each other (own construction). That headroom is a procurement conversation, and it starts weeks before launch.

Scale in tenfold steps, because each step breaks something different. A plan that jumps from 100 to 100,000 finds all of them on the same morning. The table is own construction, built from the cost incidents and the production guide.

Read the table left to right in the interview. I'd expect the prototype's weaknesses to show up in roughly this order: identity first, capacity second, fairness and organisation last. The fix for each step is cheap before that step and expensive during it.

The organising rule is admit, assemble, answer, learn. The gateway decides who may use a model and how much. Retrieval decides what the answer may see. The model path decides what it costs and how it fails. The control plane decides what changes reach users, and on what evidence.

The whole system, drawn with its planes (own construction):

 ╔═════════════════════ CONTROL PLANE (changes are releases) ═══════════════════════╗
 ║ prompt registry + live pointer · routing policy · quotas per department          ║
 ║ eval gate in CI (G13) · budgets + anomaly alerts · feature flags per cohort      ║
 ╚══════════════════════════════════════╤═══════════════════════════════════════════╝
                                        │ configures every box below
 ╔═════════════════════ DATA PLANE (calls are requests) ════════════════════════════╗
 ║                                                                                   ║
 ║ web / chat client ──SSE──> GATEWAY: SSO token check · per-user token bucket       ║
 ║                            budget meter · input guard · fair queue per department ║
 ║                                        │ admitted                                 ║
 ║                                        v                                          ║
 ║                   ORCHESTRATOR ──> RETRIEVAL (permission filter BEFORE search)    ║
 ║                        │           read-only TOOLS (step cap, timeouts)           ║
 ║                        v                                                          ║
 ║                   CACHE (key = prompt ver + model + permission scope)             ║
 ║                        │ miss                                                     ║
 ║                        v                                                          ║
 ║                   MODEL ROUTER ──> primary provider (small · mid)                 ║
 ║                        │           retry + backoff inside a circuit breaker       ║
 ║                        └─────────> warm fallback provider ──> degraded answer     ║
 ║                                        │ tokens                                   ║
 ║                                        v                                          ║
 ║                   OUTPUT CHECK (PII, citations present) ──> stream to client      ║
 ║                                                                                   ║
 ║ ASYNC LANE: queue ──> workers (reports, summaries, re-indexing) ──> batch API     ║
 ║ INGESTION: sources ──> parse ──> chunk + embed ──> index with ACL metadata        ║
 ║ OBSERVABILITY: trace per request · tokens · cost · stage timings · outcome        ║
 ╚═══════════════════════════════════════════════════════════════════════════════════╝

The same flow for viewers that render Mermaid (own construction):

Read the components in request order. The failure column is own construction.

Point at three boundaries while the diagram is up (own construction). The admission boundary is the gateway: nothing reaches a model without identity, a quota and a budget check. The isolation boundary is retrieval and the cache key: both carry the user's permission scope. The change boundary is the control plane: a prompt edit is a release with a gate, never a hot edit.

Put every model call behind one gateway, because the prototype's direct calls leave no single place to enforce identity, limits, fairness or cost. Standalone #100 designs that gateway in depth, in its sections 5 to 10. This section names only what the scale-up adds.

Identity first. Handbook Module 06 doc 1 draws the line: authorisation logic assumes identity is already established. Validate the customer's own SSO token, map their groups to internal roles, and derive the user's permission scope once per request. The prototype's shared key cannot tell 100,000 people apart, so it cannot enforce a single rule about them.

Limits and fairness are two mechanisms. The same Handbook doc says it in one line: a rate limit answers may this request proceed at all; a fair queue answers in what order do requests get served under load. Give each user a token bucket sized in tokens, not only requests. Feed admitted work into a weighted fair queue per department, so the finance quarter-end burst waits for its share rather than starving everyone. G07 section 7 has the full argument.

Meter before the call. The production guide's rule: "The cap has to fire before the API call, not after — checking cost post-hoc only tells you what you already spent." Use a real tokeniser for the estimate, since the guide shows a word-count heuristic letting expensive requests through.

Route by intent. The production guide calls routing "The single biggest cost lever in a production LLM (large language model) system." Pick the routing threshold from a labelled sample of real traffic, not intuition. The cost additions add a warning: prompt caches are scoped to a model, so a three-model cascade means three cold prefixes. Measure one strong model at lower effort before building a cascade.

Cache what is safe. The cram sheet's rule: caching is acceptable only when the key includes tenant and permission boundaries and invalidation respects document and access changes. For #61, the cheapest win is the provider's prefix cache on the frozen system prompt and tool list. The answer cache comes second and carries the user's permission scope in its key.

Stream, and move slow work off the critical path. Streaming buys perceived latency, not real latency. Report generation and document summaries do not need a waiting user. Send them through a queue to the batch API, which the cost additions put at 50% of standard cost. Workers scale by queue depth, as the cram sheet's infrastructure section says, not by CPU.

At 100 users, a provider outage is an anecdote. At 100,000, it is an incident with a headcount. The production guide gives the chain in order: retry with backoff, a circuit breaker around the retried call, a fallback provider, and graceful degradation as the floor. Its one-liner: "Retries handle one bad call; a circuit breaker handles a bad dependency — you want both, with the breaker wrapping the retried call."

Set the reliability target honestly. The guide's FDE (Forward Deployed Engineer) track answers the "99.9% uptime" request directly: "Separate your infra's uptime from the upstream provider's — you can't promise more than your dependency provides, and fallback/circuit-breaker design is how you buy margin, not a guarantee." Say that sentence to the customer before the contract says something else.

Keep the fallback warm and validated (own construction). A fallback provider that has never served real traffic fails at the worst moment. The guide's agent track names the trap: tool schemas and structured-output contracts are not portable across models, so the fallback path needs its own prompt and its own eval run. Route a small, steady share of traffic to it so its latency and quality stay measured. Tag every response with the model that answered, because cost and quality shift silently when traffic moves.

Degrade in front of people and fail loudly in front of machines. That is the guide's own trade-off line. The assistant shows a labelled degraded answer. The async lane stops and alerts rather than writing a half-finished report.

The prototype's eval set, if it exists, is the demo questions. It measures what already worked on the questions the builder thought to ask. That is survivorship, not evaluation. G13 designs the full release gate; this section names what the scale-up changes (own construction, from G13's argument).

Build the new set from the first cohorts' real traffic. Sample by intent and by department, because 100,000 users ask questions the 100 never did. Label with the customer's subject-matter experts, not the builder. Add adversarial cases for permissions and prompt injection, and hold the security slice at zero failures. Handbook Module 08 doc 1 is explicit: the safety gate is "A hard block, not a review comment."

Gate on slices, not on the aggregate. G13 section 8 makes the point: an average can rise while one department's intent collapses. Report each slice against a named baseline, which is the version currently serving.

Keep evaluating after launch. Run a sampled online evaluation on live traffic, plus thumbs, regenerates and escalations. The cost additions warn about the new bill: LLM-as-judge is inference too. Stratify the sample, use a cheaper judge with a periodic strong-judge audit, and cache judge calls on unchanged pairs. G13 section 14 has the full cost card.

Never launch to 100,000 people on one day. Roll out by cohort, so each tenfold step in section 5 happens on purpose, to a group that can be supported. Handbook Module 08 doc 1 gives shadow, canary and promote, and the risk each puts on a real user. I add cohort waves between canary and promote.

Split cohorts per user, never per request. The Handbook's reason: one person's experience must not flicker between versions mid-conversation. Hold the safety gate identical across arms.

Make rollback a pointer flip. The Handbook's line settles it: "If rollback requires a build, it is not a rollback; it is a hot-fix under pressure." The version that was live five minutes ago stays deployed, ready to take traffic. Test the rollback before the first wave, because Handbook Module 10 doc 4 makes rollback_tested a gate that blocks limited production.

At 100 users, the builder is the owner, the on-call and the support desk. At 100,000, that arrangement is the single point of failure (own construction). Name the owners before the first cohort, not after the first incident.

Write runbooks for the incidents that will happen (own construction). The cost cram sheet lists fifteen, and four matter from week one. Cost 5× after rollout: "The increase is likely from request shape, not just user count." Latency 2 s to 20 s after a deploy: roll back the prompt or reranker, activate the fallback provider. One tenant over-using: "distinguish healthy adoption from runaway automation." And cache hit rate dropping: look for a silent prefix invalidator. Add a fifth for a permissions leak, owned by security.

Run a security review against evidence, not a checkbox. Handbook Module 10 doc 4 names the failure: a gate can be structurally real and still accept free text as evidence. Require adversarial testing and permission-boundary probes as listed artefacts before the review passes.

A system can be technically ready and still fail, because nobody uses it. Adoption is the last item on the bank's list, and it is where the follow-up goes.

Measure adoption with numbers that cannot be faked by a launch email. Handbook Module 10 doc 4 gives two. The human-approval override rate: "Falling over time = trust being earned; flat = the agent is not ready for less supervision." And week-4 retention: "Whether the thing deployed is still the thing being used a month later." Add active users as a share of licensed users, task success by intent, and cost per successful task (own construction). The cost additions insist on the last one: a cheaper model that needs a retry or a human correction is not cheaper.

The bank's follow-up, verbatim: "The system is technically ready but users are not adopting it. What do you investigate?" Its answer list, verbatim: whether the problem is important enough, workflow fit, user trust, response quality, latency, training and onboarding, incentives, integration friction, and whether the product solves the user's actual pain point.

Order that list into an investigation (own construction). Start with data, not opinions. Segment usage by department and intent, and find where users try once and stop. Then read those sessions. A drop after the first answer points to quality or trust. A drop before any answer points to latency or onboarding. Low first use points to workflow fit and integration friction: the assistant lives in a separate tab while the work lives in another tool. Only then talk to users, and ask about their pain rather than the product. The cost cram sheet's executive table says it plainly: adoption depends on "Perceived latency, workflow fit, reliability, explainability."

"Adoption is a deliverable. I would measure it per cohort, and treat a flat week-4 retention as a bug with a root cause, not a marketing problem."

Question 17 lists seventeen things to cover. Check the answer against them, in the bank's order (mapping is own construction).

Failure modes worth volunteering, with the signal that catches each (own construction):

Spend the time on the gap between prototype and production. The model choice is a sentence.

The two-minute spoken answer (own construction):

First I would ask what "works" meant, because a hundred volunteers prove the idea is useful and nothing about cost, the slow tail, permissions or adoption. Assuming an internal document assistant going to 100,000 employees, I would size it: about 30% daily active at eight requests each is 240,000 requests a day, around 20 a second at peak and roughly 8 million tokens a minute. On one mid-tier model that is about $120,000 a month; routing, prefix caching and an output cap cut it by nearly half. Then I would walk the tenfold steps. At a thousand users I need SSO (single sign-on), versioned prompts and traces. At ten thousand I need a gateway with per-user limits, retry inside a circuit breaker, a fallback provider and permission-filtered retrieval. At a hundred thousand I need fair queuing per department, a capacity commitment, sampled online evaluation and on-call. I would rebuild the eval set from real traffic, gate every release on slices with security at zero, and roll out by department cohort with shadow, canary and a pointer-flip rollback. Finally I would measure adoption, week-4 retention and override rate, because a system nobody uses has failed however well it scales.

The lines that carry the round (own construction unless quoted):

1. "A hundred volunteers proved the idea, not the system."
2. "The bill scales with request shape, not with users."
3. "Each tenfold step breaks something different."
4. "A rate limit decides whether; a fair queue decides in what order."
5. "You can't promise more than your dependency provides." (production guide)
6. "If rollback requires a build, it is not a rollback." (Handbook Module 08 doc 1)
7. "Adoption is a deliverable."

Follow-ups beyond the bank's adoption question (own construction):

The interviewer's pivot: "The pilot cost $400 a month. Finance just saw the forecast." There is no drill for this prompt, so the card below is a self-drill (own construction), grounded in the cost cram sheets.

The cram sheet's incident line applies unchanged: "The increase is likely from request shape, not just user count. I would isolate token growth, retries, agent steps, and batch jobs before changing the model." Its roadmap sentence is the right shape for finance: first instrument the bottlenecks, then apply safe quick wins, and finally redesign heavy workflows with routing, caching and async processing.

Every strong cost answer follows four verbs in order. Measure tokens and cost per request path first. Route by intent. Bound output, context and tool steps. Cache safely, with the permission scope in every key.

The levels split on whether the thousandfold jump is treated as a capacity problem, an operating problem, or a change programme for a whole organisation.

• Ask what "works" meant, because a hundred volunteers prove usefulness, not cost, tail latency, permissions or adoption.
• Name what demos hide: traffic shape, tail latency, usage expansion and governance cost, plus drifting prompts and one-person support.
• State constraints as tests: requests a second, p95 first token, zero forbidden citations, fairness between departments, budget.
• Size from licensed users to requests, tokens per minute and dollars; adoption is a cost variable.
• Scale in tenfold steps, because I'd expect identity to break first, capacity second, fairness and organisation last.
• Draw admit, assemble, answer, learn, with boundaries at the gateway, retrieval and the control plane.
• Put one gateway in front of every call for identity, per-user limits, a fair queue, metering, routing and safe caching.
• Chain retry, breaker, a warm fallback and degradation, and never promise more uptime than the provider gives.
• Rebuild the eval set from production traffic, gate on slices, and hold security at zero.
• Roll out by cohort through shadow and canary, with a rollback that is a pointer flip.
• Name owners, runbooks and a support ladder before the first cohort.
• Measure adoption with week-4 retention and override rate, and investigate a stall from data to sessions to users.
• Check the answer against the bank's seventeen items and volunteer the failure modes.
• Spend the forty-five minutes on the gap between prototype and production.
• Answer the cost pivot with routing, prefix caching, output caps, batching and budgets before the call.

1. Why is "the prototype works" weak evidence for scaling? It was judged by a hundred forgiving volunteers on demo questions; it says nothing about cost, tail latency, permissions, abuse or adoption.
2. Walk the sizing from 100,000 licensed users to tokens per minute. 30% daily active × 8 requests = 240,000 a day; 80% in 8 hours with a 3× peak ≈ 20/s; × 6,500 tokens ≈ 130,000 tokens/s ≈ 7.8M tokens a minute.
3. Why did the bill rise about 300 times rather than 1,000? Only 30% of licensed users are active each day, so adoption sets cost as much as headcount does.
4. What breaks first between 100 and 1,000 users? Identity and process: a shared key, a demo eval set, live prompt edits and one-person support.
5. How do a rate limit and a fair queue differ? The limit decides whether a request may proceed; the queue decides the order admitted requests are served under load.
6. Why must the budget check fire before the call? Checking after the call only reports what was already spent.
7. What do you tell a customer who asks for 99.9% uptime? The service cannot promise more than its provider gives; fallback and breakers buy margin, not a guarantee.
8. Why rebuild the eval set? The demo set measures what already worked; production traffic from real cohorts contains the questions that will fail.
9. What makes a rollback a real rollback? It repoints to a version that is still deployed, in minutes, with no build.
10. Why split cohorts per user rather than per request? So one person's experience does not flicker between versions mid-conversation.
11. The system is ready but nobody uses it. Where do you start? Segment usage by department and intent, read the sessions where users stop, then check quality, trust, latency, onboarding and workflow fit.
12. What is the sixty-second cost answer? Route easy intents down, cache the stable prefix, cap output, batch non-interactive work, budget per department before the call, and prove it with cost per successful task.

Fact-check sources (checked 27 Sep 2026)

• Anthropic pricing — Claude Sonnet 5 at $2/$10 and Claude Haiku 4.5 at $1/$5 per million tokens; cache reads at 0.1× input; Batch API at 50% off
• Anthropic prompt caching — minimum cacheable prompt length: 1,024 tokens for Sonnet 5, 4,096 for Haiku 4.5