High-Volume Batch Inference Pipeline

Overnight batch is: a huge file must be classified before morning. Miss the window and the site publishes nothing — or publishes a partial, if they allow it.

G10 covers one slice: freeze the job, chew through 100 million records, retry the flaky ones, quarantine the junk, publish by policy. Not a chat agent picking the next task.

End to end, as Tuesday’s 6 a.m. catalog job:

1. Ops creates the job. Input and model versions freeze. No silent prompt tweaks mid-run.
2. We partition and queue work. Workers pull with leases.
3. The model labels a record. Typed validation accepts or rejects the shape.
4. A 429 retries. Garbage JSON goes to quarantine, not an infinite loop.
5. Each success writes once with an idempotency key, then checkpoints.
6. At 4 a.m. we are behind. Named owner publishes 90% or holds all — they already said which.

That’s it: snapshot → partition → infer → validate → checkpoint → publish or hold. Interactive Q&A is out.

Source: Sources and Full Design. This guide is the interview route through the unchanged source; use the Deep Dive for mechanics and the Cheat Sheet for rehearsal.

Classify 100 million records overnight, finish within a six-hour window, recover predictably from failures, and know what can be published if the deadline cannot be met. The first design choice is the customer's answer to partial results or all-or-nothing?

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Create, inspect, pause, and replay jobs.
2. Freeze input and model configuration.
3. Partition and process records.
4. Validate typed outputs.
5. Write once logically.
6. Retry transient failures; quarantine permanent ones.
7. Reconcile and publish by the agreed policy.

Non-functional requirements — how well / under what constraints

Replace token assumptions with the customer’s distribution and measured provider throughput.

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, freeze the job, process partitions, validate, write once, retry or quarantine, then publish by policy. Non-functionally, hit the six-hour window, isolate tenants, and know at 4 a.m. whether partial output is allowed.”

The model or LLM (large language model) performs classification or extraction. Deterministic orchestration owns partitioning, deadlines, retries and publication. This workload does not need an autonomous agent to decide what work to do next.

Step-by-step architecture

1. The scheduler accepts an idempotent job request and records its deadline, publication policy, model, prompt and schema versions.
2. The planner freezes an immutable input snapshot and creates a partition manifest; the ETA forecaster watches remaining work.
3. Durable leases assign opaque record IDs to autoscaled workers; workers fetch scoped input and batch where the provider allows it.
4. A shared rate coordinator respects provider and per-tenant quotas before workers call the classifier model or LLM.
5. Workers validate typed output, then upsert using (job_id, record_id, model_version) so replay cannot duplicate a result.
6. Only after a successful sink write does the worker advance the durable checkpoint and acknowledge the lease. Transient errors retry; permanent errors enter quarantine.
7. Reconciliation compares results, failures and the manifest. Publication follows the customer's pre-agreed complete or partial-output rule.

At 4 a.m., only 45% is done: use the forecast before this point to alert. Diagnose compute saturation, hot partitions, quota pressure and sink latency. Increase concurrency only if quota and sink headroom exist; split deterministic remaining work; use an approved fallback model or partial publication only when policy allows. If the deadline is impossible, stop new leases, preserve checkpoints and page the owner. Never silently publish a partial dataset.

Exactly-once effect: workers may run more than once. Leases plus an idempotent sink and checkpoint-after-write provide one logical result per key. Avoid claiming physical exactly-once execution.

Scale, latency and cost: shard the immutable manifest and autoscale workers to measured records/s; batch calls while respecting output size and tail latency; use global rate control so local worker increases do not overload a provider. Track tokens per record, retry spend, cost per million records, queue age and deadline forecast. Hash and skip unchanged documents for nightly PDF work; incremental indexing avoids unnecessary re-embedding. These are workload-specific cost pivots, not a reason to weaken correctness.

Document-platform variant: for bank documents, add file validation, classification, OCR (optical character recognition) or multimodal extraction, typed schema checks, confidence/business rules, human review, lineage and reprocessing by model version. Document-format drift becomes a first-class risk.

Measure completion before deadline, predicted versus actual ETA, throughput, partition skew, duplicate-key conflicts, retry and quarantine rates, typed-output validity, provider quota errors and cost per million records. Test representative token sizes and deliberately crash a worker after sink write but before ack. Ramp from representative data to 1% and 10%, then exercise quota and sink failures. Assign a contingency owner and write the missed-deadline decision before launch.

“I would first establish whether the six-hour deadline requires a complete dataset or permits an explicitly labelled partial one. I freeze the input and model configuration, partition 100 million records, and process them from durable leases with autoscaled workers behind global and tenant rate limits. Each worker validates the model output and idempotently upserts it before checkpointing or acknowledging. That makes retries safe. At roughly 4,630 records per second baseline, I plan for about 5,320 with headroom and forecast ETA continuously. If we fall behind, I diagnose quota, skew, compute and sink capacity, then apply only a pre-approved fallback. Reconciliation controls publication, and the operations dashboard shows deadline risk, failures and cost.”

Read with the unchanged source. The Main guide is the interview path; the Cheat Sheet is the quick recall card.

The question is recovery under a deadline. Get the output contract first: complete-only or partial publish; what to do with permanently bad records; acceptable fallback models; who approves changes. A queue cannot answer those product decisions. Store the policy with the job so an operator cannot improvise under pressure.

Job creation should be idempotent. Expose create, status, pause and replay-failures operations, guarded by authorisation. Use ETags or equivalent conditional updates for operator actions. The job record pins snapshot ID, partition manifest, model/prompt/schema versions, deadline and publication rule. Preserve enough lineage to reproduce an individual result and the whole run.

100 million / six hours is about 4,630 records/s. Fifteen percent capacity margin brings the target to about 5,320. At 10× volume, the same window needs about 53,200/s including margin. If one record averages 270 tokens, the base run is 27 billion tokens; include retries, long-tail records and output variance in the estimate. Check it as 100 million × 270 = 27 billion, not the tokens/s rate times one hour. A provider's tokens/s quota may be the real ceiling even if CPU workers look idle.

Plan with measured distributions: p50/p95 record tokens, model latency, batch efficiency, provider throttling, sink throughput and hot partitions. ETA (estimated time of arrival) is remaining work divided by observed effective throughput, adjusted for the straggler tail. An average-rate graph alone hides deadline failure.

The immutable snapshot prevents input drift during a long run. Partition IDs are deterministic and bounded; queue messages contain identifiers, while workers fetch scoped records. A lease expiration makes crashed work visible again. The result sink's unique key (job_id, record_id, model_version) turns reprocessing into an upsert rather than a duplicate. Commit output before checkpoint, checkpoint before queue ack. A crash at any point can replay work safely; it may still pay for another model call, so track replay cost.

Transient errors include throttling, network failure and temporary sink outage. Use backoff, jitter and capped retries. Validation failures or invalid source records should be quarantined with reason and lineage. Reconciliation compares manifest count, successful unique results and quarantined records. Publish only when those counts and the stored policy agree.

Keep input and output encrypted, isolate tenants, scope snapshots and credentials, and avoid full customer payloads in queue messages or logs. Record operator actions and model version changes. A per-tenant rate coordinator prevents one customer from starving others; global controls protect provider quota. Circuit breaking stops repeated failing calls while retaining durable pending work.

At 4 a.m. with 45% complete, calculate the throughput still required, then identify the limiting stage. Add workers only if provider and sink capacity permit. Splitting a straggler partition must preserve deterministic IDs. A fallback model changes output semantics, so it must be approved and versioned. If success is impossible, preserve the state, stop leasing and communicate the exact incomplete count. Partial output is a business rule, not an accident.

For bank documents, the pipeline expands to file validation, OCR/multimodal parsing, document classification and schema extraction, followed by confidence checks and human review. Lineage must point from output fields to source pages. Reprocessing by model version is necessary when a model improves or formats drift.

For nightly PDF summarisation, deduplicate by content hash and process only changed documents; a cheaper batch model may handle routine summaries, with stronger-model audits. For an embedding-cost spike, incremental indexing and avoiding unchanged content are the first levers. None of these shortcuts should hide permanent failures or erase provenance.

Track throughput, tokens/s, provider throttles, lease retries, oldest queued partition, predicted completion time, unique committed records, quarantine reasons, sink p95, cost per million and replay spend. Begin rollout on a representative token distribution, then 1%, 10% and higher load. Inject worker crash after write, provider quota exhaustion, and sink outage. A successful demo is not enough; demonstrate restart and reconciliation under those failures.

Main · Deep Dive · Unchanged source

“Is 6 a.m. a hard deadline, and can we publish partial results?” Ask about token distribution, provider quotas, permanent errors, fallback approval and replay/version requirements.

Job API → immutable snapshot + partition manifest → durable leases → workers → rate coordinator → classifier model/LLM → typed validation → idempotent sink → checkpoint + ack → reconcile + publish

The model classifies or extracts; deterministic orchestration owns the workflow. Checkpoint after sink success.

100M / 6h ≈ 4,630 records/s; +15% ≈ 5,320/s. At 270 tokens/record, ≈ 27B tokens/run before retries (100M × 270). At 10×, ≈ 53,200/s with margin. Confirm against measured token and latency distributions.

At 4 a.m. and 45% complete: forecast remaining capacity; inspect quota, hot shards, compute and sink; increase concurrency only with headroom; use approved fallback or explicitly labelled partial output only if policy allows. Preserve checkpoints and page the owner if impossible.

• Immutable snapshot and pinned model/prompt/schema make replay meaningful.
• Unique (job_id, record_id, model_version) gives one logical result under retries.
• Transient failures retry; permanent failures quarantine with lineage.
• Watch ETA (estimated time of arrival), oldest partition, throttles, cost/million and reconciliation counts.
• For document variants, add OCR (optical character recognition), typed extraction, human review and versioned reprocessing.

60-second close: “I design for safe replay and deadline recovery. The immutable manifest feeds leased workers behind shared rate limits; outputs are validated and upserted before checkpointing. A live ETA and a pre-agreed publish policy determine the response when the window is at risk.”

Finish a hundred million records before the business day starts, without paying twice for any of them or losing the ability to replay the night.

Throughput is not the hard part of this system. The hard part is the decision at 4 a.m., when the job is behind and someone has to choose what ships. This page consolidates the group's prompts into one read for the day before. The anchor is the nightly classification batch, and every other member is a delta on it.

Sections and tables marked (own construction) were built for this page from the sources' arguments and are not in the sources verbatim.

The prompt sounds simple: classify 100 million records every night before 6:00 a.m. The room splits three ways at once. The business owner says, "We just need classifications ready by morning." The data platform lead says, "Only if it doesn't break the warehouse." The on-call engineer says, "Only if we can rerun it safely when the model or upstream feed fails."

That disagreement is the real problem, not the throughput number. So restate the target before drawing anything.

"Complete a replayable, cost-controlled batch by deadline with clear recovery decisions when behind schedule."

Lead with that sentence verbatim. Interviewers use it as a checkpoint. It tells them whether the candidate is solving the business problem or only the throughput problem.

The technology-neutral opening does the same work in a longer form:

"We need a nightly batch inference system that classifies about 100 million records and finishes before 6:00 a.m. The design has to account for model rate limits, cost constraints, and downstream capacity so the output is usable by morning. I'd first clarify who consumes the results, what happens if we're behind schedule, and what replay and audit requirements exist before choosing the execution architecture."

The weak version names tools: "We should build a batch classifier with a fast inference service and a scheduler." The corrected version names value: "The customer needs reliable nightly classification results delivered before business users need them, with predictable cost and a clear recovery path when the batch falls behind."

Map the stakeholders next. Each one hires the platform for a different job, and naming them settles who owns which decision.

The business does not decide batch retry semantics. The on-call engineer does not decide whether a stale classification is acceptable. The model owner does not own the downstream SLA. That division keeps the design aligned with reality rather than with one enthusiastic stakeholder.

The brief is incomplete on purpose. The interviewer may confirm the volume and the deadline, then stop. That is the test, not a trap.

Protect the one constraint that most changes the architecture. Can the batch be partial, or must it be all-or-nothing? A late but correct job can be rerun or explained. A job that silently drops or duplicates records is much harder to trust. The partial-result answer sets the retry strategy, the checkpoint cadence and how much to spend on the last 10%.

When an answer is withheld, state the assumption and keep moving. The reusable form:

"Given the missing details, I'm going to assume the records are independently partitionable, partial results are only useful if they're explicit and auditable, and the highest-risk constraint is finishing by the deadline without duplicates. That means I'll prioritise shardable work, checkpointed progress, idempotent writes, and rate limiting before I optimise for cost or advanced scheduling. If any of those assumptions are wrong, the design changes in specific ways, so I'd want to validate them early."

Assumptions carry risk too, so name it. If one record depends on another's result, naive parallelism breaks correctness. If a tenant expects strict completion, a degraded mode that skips low-priority work breaks the contract. If the provider's batch API has different semantics from synchronous calls, a naive retry loop duplicates output.

Split what the system must do from how well it must do it. "Batch the model calls" is a feature. "Finish by deadline" is a constraint. "Scale workers up and down" is a mechanism, not an outcome.

Functional requirements.

Non-functional requirements.

Prioritise with a strict ladder when answers are partial. Must: independent partitioning, deadline protection, idempotent writes, restartable execution. Should: autoscaling, batching where it helps, global plus per-tenant limits. Could: ranking low-value records, richer per-customer scheduling, manual override tooling.

Fence the MVP (minimum viable product) explicitly. It excludes cross-job optimisation across unrelated workloads and any per-record interactive feedback loop. It also excludes a training pipeline in the batch path, arbitrary ad hoc queries, and optimal cost in every failure mode. More responsibilities make deadline risk harder to reason about.

Every component then traces to a named requirement:

The first pass is always too optimistic: a queue, a few workers and a model endpoint. That works at average load. It ignores the deadline, the retry rate and the hard last 5%.

Derive the number live rather than reciting it. With 100,000,000 records and a window of 6 × 3,600 = 21,600 seconds, the base rate is about 4,630 records/second. That is the floor, not a design. Adding 15% headroom for retries, stragglers and reconciliation gives a planning rate of about 5,320 records/second.

The 10x row is not tomorrow's expectation. It tests whether the design is fragile or absurdly overbuilt. Keep headroom and growth apart: headroom covers noise inside one batch, while growth covers next quarter's larger batch. State average, peak, growth and headroom rather than one point estimate.

Tokens turn the record rate into a factory. At 250 input plus 20 output tokens, each record costs 270 tokens before retries. I'd let that token figure, not the record count, drive partitioning and component choice.

Say the uncertainty out loud rather than hiding it:

"At 100 million records, I would plan for about 4,630 records per second before retries. If the average prompt is smaller than expected, model throughput becomes easier; if the tail of large records is heavier, we need more headroom or more aggressive partitioning. I would size the system around the tail, not the mean."

The chapter treats cost only qualitatively. The V2 tutorial's author closes the gap with a live dollar formula, which is supplementary perspective and not chapter content. At 270 tokens a record and ~5,320 rps, the fleet sustains roughly 1.43M tokens/second. Across the window that is about 27B tokens for 100M records (100M × 270). Apply the per-million-token price for a cost per run, then divide by 100 for cost per million records. Retries of calls that actually ran re-spend the full 270 tokens each; a call rejected up front, such as a 429, generally costs nothing. At a 5% retry rate that is roughly 5% more spend. The 15% headroom is itself a cost line, and stragglers burn compute past the window. So the cost ceiling belongs inside the rate-limit coordinator's backoff behaviour, not in a separate finance conversation.

The organising rule is that each layer owns a different kind of truth. The planner owns what "this batch" means. The queue owns who is working on what. The sink owns what has been published. Planner, manifest, forecaster and console are control-plane concerns that coordinate the run. Workers and sink are data-plane concerns that move records through the model into durable output. They scale and change at different rates, which is why they are drawn apart.

The system with its planes and trust boundaries (own construction):

 ╔══════════════════════════ CONTROL PLANE (coordinates the run) ══════════════════════════╗
 ║  customer scheduler / API ─> JOB PLANNER ─> PARTITION MANIFEST ─> ETA FORECASTER          ║
 ║  (idempotency key)          freeze snapshot  ranges, attempts,     progress vs deadline   ║
 ║                             model_version,   checkpoints              │                   ║
 ║                             deadline, policy     │                    v                   ║
 ║                                                  │            OPERATIONS CONSOLE          ║
 ║                                                  │            status · pause · replay ·   ║
 ║                                                  │            contingency mode + owner    ║
 ╚══════════════════════════════════════════════════╪════════════════════════════════════════╝
            TRUST BOUNDARY 1: snapshot frozen ───────┤ enqueue leases (opaque ids only)
 ╔══════════════════════════ DATA PLANE (moves records) ═════════╪═════════════════════════╗
 ║                                                               v                         ║
 ║   DURABLE WORK QUEUE ── lease ──> INFERENCE WORKERS (autoscaled) ── read range once     ║
 ║   leases, visibility timeout      │   chunk into batches (e.g. 64)                        ║
 ║   retries, dead letter            │                                                      ║
 ║        ^                          v                                                      ║
 ║        │               RATE-LIMIT COORDINATOR  grant / delay tokens + concurrency        ║
 ║        │               (global + per tenant; backpressure lives here)                    ║
 ║        │                          │                                                      ║
 ║        │     TRUST BOUNDARY 2 ────┼──> EXTERNAL MODEL / PROVIDER (timeouts, 429s, junk)   ║
 ║        │                          v                                                      ║
 ║        │               validate_output()  typed boundary; bad output -> quarantine       ║
 ║        │                          │                                                      ║
 ║        │     TRUST BOUNDARY 3 ────┼──> IDEMPOTENT RESULT SINK                             ║
 ║        │                          │    upsert on job_id + record_id + model_version      ║
 ║        │                          v                                                      ║
 ║        └── ack / release ── CHECKPOINT STORE  advances only after the sink write succeeds ║
 ╚═════════════════════════════════════════════════════════════════════════════════════════╝

The source's control-plane and data-plane diagram:

The components, with the failure behaviour of each (the "Fails how" column is own construction):

Walk the happy path in seven steps. First, snapshot the input and build the manifest, recording dataset version, deadline and run policy. Second, split into partitions balanced by processing cost. Tenant, payload-size band or historical latency beats record count alone. Third, lease work to workers through the durable queue, so the planner never waits on any worker. Fourth, batch records and ask the rate-limit coordinator for capacity before calling the model. Fifth, checkpoint the completed range and write rows to the idempotent sink. Sixth, retry transient errors per policy and quarantine permanent ones with triage context. Seventh, reconcile the manifest and publish the completed dataset.

Three trust boundaries carry the security story. After the planner freezes the snapshot, the run cannot drift if upstream data changes. The model call is external, so the worker assumes timeouts, throttling, malformed responses and partial failures. Publication must be idempotent, so replays cannot create duplicate classifications.

The MVP (minimum viable product) is planner, manifest, queue, worker fleet, checkpointing, idempotent sink and a basic ETA panel. Smarter rebalancing, historical throughput prediction and tenant-aware scheduling come later. The common mistake is treating those later items as the architecture itself. Draw the flow, but speak in obligations: every component protects deadline, cost, replayability or operator confidence.

A crashed worker will redo some work. The design choice is whether that is dangerous or harmless. A sink cannot duplicate a row it upserts by the same key.

Three records carry the durable state.

The input_snapshot is an immutable pointer to the exact dataset version. It is the ownership boundary that makes replay possible. The result is keyed by business identity plus model version, never by worker or attempt. Be ready for "why not just record_id?" Without the model version, a rerun under a new model silently overwrites yesterday's answer and replay loses its meaning.

The API surface stays small.

Idempotency applies at every write boundary, not only the sink. A duplicate create must not create a second job. A duplicate lease renewal must not invalidate the original lease. A duplicate result write overwrites the same logical record. A duplicate pause or replay request is harmless. Versioning matters for the same reason: when the model, schema or validation policy changes, old and new results must stay distinguishable.

The smallest code path that proves the design is one worker loop. It claims a lease and reads the range once. It chunks records into batches of 64 and acquires tokens from the limiter before the call, because admission control belongs before the quota breach. It then validates output at a typed boundary and upserts by record_id. The checkpoint advances only after the sink write succeeds, which prevents false progress. The partition completes only after every batch succeeds. A TransientError retries with backoff. A ValidationError retries once without backoff and then surfaces as a data-quality problem.

Two tests prove it. The contract test repeats POST /v1/batch-jobs with the same key and confirms one job and one id. The failure-injection test crashes the worker after upsert_many() but before complete(). On retry the sink still holds one logical result per record_id. The source's assertion is that distinct record_id count equals row count.

The sketch deliberately omits multi-worker contention, logging, metrics, tracing, cancellation, authentication, throttling and dead-letter routing. That is acceptable only when each omission can be named and placed.

The most important security control is not on the model. It is that every worker processes a versioned snapshot that cannot change beneath it. A retry against a mutable table cannot prove what it retried.

Snapshot immutability does three jobs at once. It removes ambiguity during retries, it makes evidence replayable after an incident, and it limits blast radius. A malformed or malicious tenant dataset touches only that tenant's snapshot, not a global table.

Keep raw payloads out of queue metadata. Messages carry only opaque identifiers, snapshot versions, partition keys and lease information. An exported or replayed queue then discloses nothing. Encrypt result and checkpoint stores and restrict them to least privilege. Checkpoint state reveals business data shape and partial outputs, so treat it as sensitive.

Record model version, code version, prompt or feature schema version and deployment identity for every run. That is how the team proves which artifact produced which result when a customer asks why today's classification differs from yesterday's.

The chapter names no external framework, and the V2 author treats that as a gap. Their supplementary answer: ask which framework applies before claiming compliance. With personal data, purpose limitation and data minimisation apply to snapshot and result retention. In healthcare, HIPAA's audit-control rule applies to the audit trail; in finance, rules such as SEC (US Securities and Exchange Commission) 17a-4 or SOX (Sarbanes-Oxley Act) do. SOC 2 is a voluntary audit report that customers in any industry may ask for. They also propose a per-run "processing basis" field on BatchJob. The controls are necessary but not sufficient without a named framework to test them against.

"We retry" is not a failure policy. A policy answers what to retry, how long, against which snapshot, with what evidence, and who is paged. Every external dependency and every irreversible action needs one.

Memorise one trigger per row. An interviewer who invents a novel failure wants it classified into this table on the spot.

The five drills, each as detection, containment, recovery and prevention:

The pattern across all five: never push harder against a failing dependency. Slow the arrival rate at the coordinator, preserve evidence before repair, and escalate when the deadline is no longer recoverable. The ownership statement to say in the room:

"This design contains blast radius by tenant, region, workflow, and dependency; it retries only within a bounded policy; it preserves evidence before repair; and it escalates when the deadline is no longer recoverable."

The signature follow-up is that the job is 45% complete at 4 a.m. The forecaster should have flagged it already. The console should answer three questions, not just turn red. Can we still finish by 6:00 a.m.? Which lever do we pull first? What is the blast radius if we keep going?

Find the bottleneck before choosing the lever. It is compute, a hot partition, provider throttling or downstream backpressure. Then choose the smallest intervention that restores schedule confidence. That means raising concurrency if there is headroom, splitting slow partitions, or switching remaining work to a fallback model. Alternatively, publish only the subset that meets the correctness and policy bar. If throttling is temporary, the coordinator reduces concurrency and keeps the run stable. If it is persistent, stop taking new leases, checkpoint everything and page the customer with a recovery plan.

The contingency modes are business decisions agreed before launch, each with a named owner. They are: continue at full accuracy until cutoff, switch to a cheaper or smaller model, or reduce low-value record classes. The other two are narrowing scope to the highest-priority segments and pausing nonessential enrichment. The product owner and operations lead agree them before the first run, not after the first miss.

Partial results are publishable only under a partial-acceptance policy the customer defined. Some workloads can publish completed partitions if each unit is independently valid and clearly labelled. Others need all-or-nothing, because downstream cannot handle mixed vintages. State which regime applies. If partial, define the exact publish gate and completeness metadata.

Two more probes arrive in this part of the round. To rebalance a hot partition, detect it early through lag, runtime histograms and queue age. Split it with a deterministic key so work stays replayable. If it is in flight, move only the unprocessed remainder and record the new ownership. To avoid paying twice after a timeout, treat the timeout as ambiguous: it may or may not have completed. Idempotency keys, durable checkpoints and write-side deduplication make the retry cheap and harmless.

State each verdict in one sentence before the justification. Slogans like "always batch bigger" cost more than missing detail.

The riskiest assumption is that the backlog is evenly distributed and model runtime is stable. When the interviewer challenges it, agree rather than defend.

"If skew is worse than expected, I would treat partition imbalance as the first-class risk, because a few hot shards can dominate the batch window. I would size the system so that we can split or retry those shards without restarting the entire job."

"The job finished" is not an evaluation. A batch can finish and still have published duplicates, run twice the budget, or been saved by one operator at 5:50. The scorecard separates throughput, correctness and cost.

Build the dashboard as a story, not a wall of charts. The headline asks one question: will the batch complete by 6:00 a.m.? It shows the forecast, remaining partitions and the contingency mode. Below it sit the levers that explain the forecast: records/sec, tokens/sec, retry rate, straggler age, queue depth and sink lag. Below those sit correctness signals: duplicates, rejected inputs, checkpoint freshness and replayed partitions. A business user sees whether the run is on track. An operator sees which subsystem is slowing it. An engineer traces a retry spike to a quota cut or a bad input cohort.

Keep four layers distinct: technical health, model quality from sampled evaluation, adoption by downstream teams, and business outcome.

Launch is a sequence of controlled reductions in uncertainty. The customer's question is not "does it work?" but "when can this be trusted in production?"

A wrong input distribution makes every later estimate wrong: throughput, queue pressure, quota burn and downstream lag. That is why the benchmark comes first.

Carry a risk register with owner, mitigation and trigger.

The go/no-go gate is blunt. Without stable throughput, rehearsed crash recovery and an available contingency owner, do not expand. That sounds conservative until the batch misses one deadline because nobody wanted to say no.

Complete the launch plan with canarying, rollback, migration, training, support and documentation. A slice of nightly work runs the new path while the old one stays as fallback. Rollback stops new work, preserves checkpoints and resumes on the stable path. Separate reusable value from delivery work too. Thresholds, deadlines, model choice and contingency policy are configuration. Warehouse, queue and identity integrations are adapters. Retry, checkpoint and rate-limit wrappers become a common service. Partition orchestration, replay semantics and the push/degrade/stop policy engine are core product.

Member #49 is an OpenAI-style practice prompt, Q5: "A bank processes millions of financial documents and wants to extract structured data using AI." It is the same spine with a different payload. Records become documents, and a label becomes a schema.

The source lists the decomposition. It runs ingestion, file validation, document classification, OCR (optical character recognition) or multimodal extraction, schema-constrained output, and confidence scoring. It continues with validation against business rules, human review, correction feedback, audit storage, and reprocessing with model versioning. It asks for discussion of batch versus online processing, idempotency, retry behaviour, data lineage, model versioning and document-format drift.

Map each stage onto the anchor rather than drawing a new system (own construction).

Batch versus online is a per-workflow decision. Bulk backfills and nightly statements are batch, which buys throughput and quota smoothing. A document a customer uploads while waiting needs an online path with its own latency budget. The two paths can share the extraction schema and the typed boundary (own construction).

The likely follow-up is "How would you handle a new document format appearing without warning?" The answer (own construction) has four parts. Detect it: classification confidence drops, and schema-validation and business-rule failure rates rise for one cohort. Contain it: fail that cohort to the review queue rather than publishing low-confidence fields. Recover: add the format to the classifier and schema, then replay only the quarantined cohort through replay-failures. Prevent: monitor confidence and rejection rates by source and template, because format drift is a data-quality incident, not a model bug.

Open with the outcome, then ask the judgement question. If everything cannot be perfect, what matters most: deadline, accuracy, cost or safe replay? That one question prevents overbuilding the wrong axis.

"I'll first pin down the batch deadline, correctness tolerance, and failure recovery expectations. Then I'll estimate throughput and identify the bottlenecks that actually threaten the schedule. After that I'll propose a partitioned ingestion and inference pipeline with idempotent writes, monitoring, and a deliberate fallback path if we're behind at 4 a.m."

Running out of time before the trade-offs or the failure drill is how a technically strong answer scores as not interview-strong.

The two-minute spoken answer, from the answer key:

I would not start with the model. The prompt sounds like a throughput problem — classify 100 million records every night before 6 a.m. — but the room splits three ways immediately: the business owner wants classifications by morning, the platform lead wants the warehouse intact, and the on-call engineer wants to rerun safely when something fails. That disagreement is the real problem, so I would restate the target as completing a replayable, cost-controlled batch by deadline with clear recovery decisions when behind schedule. The first question I need answered is whether partial results have value, because that one answer determines retry strategy, checkpointing cadence, and how much money to spend on the last ten percent. On sizing, 100 million records across a six-hour window is about 4,630 records per second, and with fifteen percent headroom for retries and stragglers, roughly 5,320 — and I would size from the tail of large records, not the mean. Architecturally, a planner freezes an immutable snapshot, a manifest splits it into partitions balanced for processing cost rather than row count, a durable queue hands out leases, and a rate-limit coordinator holds backpressure so tightening quota slows leases instead of triggering a retry storm. Checkpoints make recovery bounded and the sink upserts on the record key so replay cannot duplicate. The failure I would rehearse out loud is being 45% complete at 4 a.m., and the answer is that the contingency mode — narrow scope, cheaper model, or partial output with explicit status — was agreed with a named owner before launch.

The ninety-second architecture summary, from the V2 tutorial:

"We have a nightly batch that must classify all records before the business deadline, so I would prioritise replayability, clear ownership, and deadline-aware recovery over exotic optimisation. I'd partition the input into deterministic shards, track each shard in a durable manifest, and run workers that call either a provider API or a self-hosted model depending on the cost, latency, and control constraints. Every write would be idempotent, every shard would checkpoint progress, and every retry would be safe to rerun. I'd monitor shard lag, provider throttling, queue age, and downstream sink health so we can detect whether we're on track by the middle of the window. If we're behind at 4 a.m., I'd first identify the bottleneck, then decide whether to split hot partitions, raise concurrency, or switch the remaining work to a fallback model if the quality policy allows it. The main trade-off is between throughput and flexibility: larger batches improve efficiency, but smaller, deterministic units make recovery safer. I'd start rollout with a small representative slice, verify stable throughput and idempotent replay, and only then expand to the full nightly workload."

The three answer tiers, from the answer key. The weak answer builds a batch classifier with an inference service and a scheduler and runs it nightly. It names tools instead of value and says nothing about snapshots, 4 a.m. or replay. The average answer partitions, runs workers with retries, checkpoints and alerts when late. It treats "alert when late" as a recovery plan and never shapes calls against quota. The strong answer freezes a snapshot and balances partitions by processing cost. It leases durably, holds backpressure in a coordinator, checkpoints for bounded replay, upserts by record key, and agrees contingency modes before the run is ever late.

The lines that carry the round (own construction from the sources' arguments):

1. "Throughput is not the hard part. The decision at 4 a.m. is."
2. "Can the batch be partial? That one answer sets retry, checkpointing and the spend on the last ten percent."
3. "4,630 records a second is the floor. 5,320 with headroom. Size from the tail, not the mean."
4. "Backpressure lives in the rate-limit coordinator. Quota tightening slows leases, it does not trigger a retry storm."
5. "The checkpoint advances only after the sink write succeeds."
6. "Upsert on job, record and model version. Duplicate work becomes harmless instead of dangerous."
7. "Fail closed on the snapshot hash, the model version and the checkpoint. Degrade on everything else."
8. "The contingency mode has a named owner before launch, not after the first miss."

The follow-ups arrive in a predictable order.

The interviewer's pivot after a good design is "the nightly bill is too high." Two members of the group are that pivot, and both have the same root: reprocessing what did not change.

#42, nightly PDF summarisation is too expensive. Thousands of PDFs are processed nightly and cost is too high. Ask first. Are documents new or changed? Is full summarisation required every time? What is the SLA (service level agreement)? Can work be batched off-peak? What quality checks are needed?

The source's strong answer adds incremental processing, queue budgets and sampled evaluation. Its architecture is ingest, hash and dedupe, parse, changed-only queue, batch summarisation, quality sample, publish. The trade-off: batch optimises throughput, not interactive latency, and quality is checked by sampling plus strong-model audits.

#116, §15 scenario 13, the batch embedding job became too expensive. The nightly embedding bill spikes after a document refresh. Ask whether all documents are re-embedded or only changed content. The causes are full re-indexing, duplicate documents, no content hash, or a larger embedding model.

The debugging step is a single comparison: changed document count against embedded document count. If the two differ, the pipeline is the bill. Prevention is an indexing budget with anomaly alerts. A silent embedding-model upgrade is also a re-index, so the model version belongs in the cache key (own construction).

Both cards connect back to the anchor. The content hash is the same idea as the immutable snapshot: know exactly what this run is about, and skip what it is not. The sampled audit is the same idea as the tail-sized evaluation: prove quality on a sample instead of paying the premium model on everything.

Every strong cost answer is generated by four verbs in order. Measure, by comparing changed against processed and tracing cost per document. Route, sending the bulk to a cheaper model and the audit sample to the strong one. Bound, with queue budgets, an indexing budget and anomaly alerts. Cache safely, keyed on content hash and model version.

On this design the levels separate on who owns the 4 a.m. decision and how far replay safety reaches, not on raw throughput.

• The prompt is a throughput problem on the surface and a 4 a.m. recovery decision underneath, so restate it as a replayable, cost-controlled batch by deadline.
• The partial-result question is asked first, because it sets retry, checkpointing and the spend on the last ten percent.
• Requirements are split into features and constraints, prioritised must, should, could, and each traced to a component.
• Sizing starts at 4,630 records per second, plans for 5,320 with headroom, and is driven by the tail and the 270 tokens a record.
• One diagram separates the control plane that coordinates the run from the data plane that moves records, with three trust boundaries.
• Duplicate work is made harmless by keying results on job, record and model version and advancing the checkpoint only after the sink write.
• Workers read immutable snapshots, queues carry opaque identifiers, and every run records its model, code and schema versions.
• The failure policy is decided before launch: fail closed, degrade, queue or human intervention, with five rehearsed drills.
• The 4 a.m. answer finds the bottleneck, pulls the smallest lever, and uses a contingency mode that already had an owner.
• Four trade-offs are defended with a verdict first, and skew is treated as the riskiest assumption.
• The scorecard tracks completion forecast, duplicates, straggler age, retry rate and cost per million records, not "the job finished".
• Rollout climbs from a representative benchmark through 1% and 10% gates to deliberate failure drills.
• The bank document platform is the same spine, with a schema at the typed boundary, a review lane and format drift treated as a data-quality incident.
• The hour opens with the outcome, spends its middle on failure and trade-offs, and closes on the first rollout gate.
• Both cost pivots are answered with change detection: never reprocess or re-embed what did not change.

1. What is the one-sentence restatement of the prompt? Complete a replayable, cost-controlled batch by deadline with clear recovery decisions when behind schedule.
2. Which clarifying question is protected first when time is short, and why? Whether partial results have value; it decides retry strategy, checkpoint cadence and how much to spend on the last 10%.
3. Derive the planning rate. 100,000,000 / 21,600 s ≈ 4,630 rps; ×1.15 headroom ≈ 5,320 rps.
4. Why is headroom kept separate from growth factor? Headroom covers noise inside one batch; growth covers next quarter's larger batch. Mixing them hides both.
5. Where does backpressure live, and what does it do when quota tightens? In the rate-limit coordinator; it slows leases or per-worker concurrency instead of letting retries stampede the provider.
6. Why is InferenceResult keyed on job_id + record_id + model_version, not record_id? So replay is idempotent within a run and a rerun under a new model cannot silently overwrite the old answer.
7. When does the checkpoint advance? Only after the sink write succeeds, which prevents false progress.
8. Name one trigger for each failure policy. Fail closed: broken snapshot hash. Degrade: transient sink throttling. Queue: quota reduction. Human intervention: 45% at 4 a.m. with no path to the deadline.
9. What three questions should the console answer at 4 a.m.? Can we still finish? Which lever first? What is the blast radius if we keep going?
10. What changes when the payload is a bank's financial documents (#49)? A schema at the typed boundary, confidence and business-rule checks, a human-review lane, and lineage down to document hash and page.
11. What is the sixty-second line for the nightly PDF bill (#42)? Never reprocess an unchanged document without a reason. Batch buys throughput not latency, so quality is proven by sampling.
12. What single comparison diagnoses the embedding bill spike (#116)? Changed document count against embedded document count.

Fact-check sources (checked 27 Sep 2026)

• OpenAI API docs: Batch API — batch jobs have their own semantics, so a naive retry loop can duplicate work
• Dean & Ghemawat, MapReduce (OSDI 2004) — stragglers set batch finish time; backup tasks fix it
• OpenAI Help Center: Troubleshooting 429 errors — rate-limited calls are rejected up front
• Claude docs: Rate limits — tokens-per-minute quotas can be the real ceiling; 429 behaviour
• IETF draft-ietf-httpapi-idempotency-key-header-07 (expired), s2.7 — same key with a different payload is an error
• Stripe API reference: Idempotent requests — same key with different parameters is rejected
• RFC 9110 HTTP Semantics, 403 and 404 — a server may return 404 to hide a forbidden resource
• OWASP Authorization Regression Testing Cheat Sheet — distinct 403/404 answers leak existence
• AICPA & CIMA: SOC 2 — SOC 2 is a voluntary attestation, not an industry rule
• 45 CFR 164.312(b) Audit controls — HIPAA's audit-control requirement
• FINRA: Books and Records (SEA 17a-4, FINRA 4511) — finance record-keeping rules
• Amazon Textract pricing — document extraction is priced per page