Model Development and Post-Training

Source: Sources and Full Design, especially §§1–12. Use the Deep Dive for methods and arithmetic and the Cheat Sheet for rehearsal. The source constructs the design from reported prompts; its run sizes and throughput are illustrations, not measured training results.

The anchor is “design a model that solves math problems,” walking from data through supervised fine-tuning (SFT), post-training and evaluation. Three neighboring prompts change one constraint in that pipeline.

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Collect answer-checkable examples.
2. Deduplicate and decontaminate against evaluation.
3. SFT on worked solutions and honest “ill-posed” cases.
4. Sample model attempts.
5. Score correctness / working / style with the right verifier or preference method.
6. Update and checkpoint.
7. Gate release on held-out accuracy, general ability, safety, and contamination.

Non-functional requirements — how well / under what constraints

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, clean data, SFT, sample, score, checkpoint, then a held-out gate. Non-functionally, reproducible lineage, no eval leak, and the model does not approve its own release.”

The model is the object being trained, not an agent that approves its own release. Programs, calibrated graders and humans provide labels; deterministic experiment tracking and release policy decide which checkpoint advances. Evaluation failure loops back to data/error analysis.

Step-by-step architecture

• Collect problems with trustworthy answer labels, deduplicate near twins, remove evaluation overlap and record collection/label provenance.
• Verify final answers where programs or symbolic checks can do so; send ambiguous or high-impact labels to humans. Mix general tasks to resist math-only forgetting.
• Start from a base model and use SFT to teach the worked-solution format, marked final answers and appropriate uncertainty; save a reproducible checkpoint.
• Sample several answers from that checkpoint and score with a final-answer verifier, step grader or preference labels according to the target; audit high-reward examples for reward hacking.
• Update the policy with a drift control or use DPO (direct preference optimisation) on preference pairs; log data hash, code, config, seed and optimizer/checkpoint state.
• Compare every candidate with a decontaminated held-out set plus general-skill and safety/over-refusal suites. A failed gate sends evidence back to error analysis and the data mix.
• Deploy only a passing candidate. If a device ceiling applies, distill and quantize, then rerun quality, latency, memory and battery evaluation on the actual device. Reviewed production feedback enters a gated data flywheel.

SFT versus post-training: SFT teaches the shape of a response. For checkable math, scored model attempts teach selection of correct answers. A final-answer verifier can reward correct numbers reached by bad working, so use step labels if reasoning validity matters. Preference pairs are better suited to tone and helpfulness. Keep a drift penalty and inspect high-reward samples, or the model learns the scorer’s blind spots.

Evaluation integrity: split by source and time, inspect n-gram/embedding overlap, and report pass@1 with its sampling settings. A large offline gain can be leakage. Gate general ability, fabrication on ill-posed problems, harmful compliance and over-refusal, not only math accuracy. Do not tune repeatedly on the final test set.

Compute budget: size by ~6ND training FLOPs, memory and effective throughput. Use LoRA/QLoRA for quick iteration; full fine-tuning only after a demonstrated adapter ceiling. For larger runs, use sharded state, parallelism appropriate to layer size, streamed data with recorded order and checkpoints including optimiser and loader position. Track evaluation gain per GPU-hour and kill stagnant runs.

Phone variant: illustrative 3B weights need about 1.5GB at 4-bit before KV (key–value) cache and runtime overhead; a 1–2GB budget may require a smaller model, shorter context or both. Distill for task behaviour, quantize, train politeness preferences and measure the quantized student on the slowest supported phone.

“I would pin the target before picking a method: level of math, final answer versus valid working, tool access and budget. I collect checkable examples, deduplicate and decontaminate them, then use short SFT to teach the response format. I sample the model’s own answers and use a program verifier for final-answer correctness, adding step review only if the reasoning must itself be valid. Post-training includes drift control and audits of high-reward samples. Each checkpoint passes a held-out, source/time-separated gate for math, general ability, ill-posed cases and safety before release. I budget GPU hours around rollouts and evaluation, checkpoint the full run state, and treat compute, harm or device constraints as changes to specific stages rather than redesigning the entire pipeline.”

Read with the unchanged source. The Main guide is the interview path; the Cheat Sheet is the recall card. Most design choices in the source are constructed from prompts and general practice, so distinguish assumptions from measured results.

Math's advantage is program-checkable final answers. Use public, licensed, expert-written and filtered synthetic problems. Rejection sampling keeps synthetic answers only when verified. Deduplicate near-duplicates before train/test splitting, then separate by time and source; random splits over near-identical traces inflate accuracy. Record a datasheet with collection period, filters, split method, label source and known bias. Use human adjudication for hard, high-impact or low-agreement labels. A source example of duplicated traces produced a 94% offline score but roughly 78% production performance; after deduplication/time split/relabeling, the retrained model's honest offline score was near 83%, and its later production result landed within two points of that.

SFT (supervised fine-tuning) pairs prompts with worked solutions and marked final answers. It teaches instruction following and format; training too long on narrow data can memorise phrasing or erode general behaviour. Include general instruction examples and legitimate “no unique answer” cases. Parameter-efficient adapters speed recipe search; QLoRA lowers base-weight memory further. A full update is justified only when the adapter misses a measured target.

For post-training, sample multiple candidate answers per problem. A program verifier is efficient for final-answer correctness but misses invalid working. A step reward or human review addresses reasoning quality. Preference pairs work for tone, clarity and refusal boundaries, but graders may favor verbosity. RL with a verifier benefits from a reference-model drift penalty; DPO (direct preference optimisation) gives a simpler preference-pair path. Audit high-scoring samples every round to detect reward hacking.

Rules of thumb: training FLOPs (floating-point operations) ≈ 6 × parameters × tokens; generation FLOPs ≈ 2 × parameters × generated tokens. The source's assumed 7B/50M-token SFT is ~2.1e18 FLOPs. Its 400M rollout tokens are ~5.6e18 generation FLOPs and ~1.68e19 update FLOPs. Full Adam fine-tuning at roughly 16 bytes/parameter puts 7B weights/gradients/optimizer state near 112GB before activations, beyond one 80GB card.

In #73's pretraining variant, an illustrative 20 tokens/parameter means 140B tokens for 7B and ~5.88e21 FLOPs. At assumed 3e14 effective FLOP/s per GPU, that is ~5,444 GPU-hours; these are whiteboard estimates. Fit the memory first: data parallel if a model fits; ZeRO/FSDP when optimiser state does not; tensor parallel when layers do not; pipeline parallel when depth needs splitting; recomputation and mixed precision for activations. Stream quality-filtered shards, record shard order and checkpoint optimiser plus data-loader position. Track model FLOPs utilisation and eval gain/GPU-hour, not only GPUs allocated.

Gate on pass@1 for clean unseen problems, valid working on a stratified sample, general regression, fabrication on ill-posed problems and safety/over-refusal. Report temperature and sampling method: greedy pass@1 cannot be compared to majority vote over eight samples. Check overlap by n-grams/embeddings and attach a contamination report. Treat an unexpectedly large gain as a leakage investigation, not an automatic win.

For #81, plot harmful-compliance rate against over-refusal on benign prompts that sound risky. A model that refuses everything fails usefulness. Define written policy; use preference pairs for helpful and safe behaviour, AI feedback only with human auditing, and red-team cases in the gate. An input/output safety classifier can change faster than weights. For #83, distill a teacher into a small student (about 1B for a 1–2 GB budget; a 3B model needs ~1.5 GB at 4-bit for weights alone), quantize to fit device memory and evaluate polite behaviour with rude or provocative prompts. Include KV (key–value) cache, runtime overhead, first-token time, tokens/s and battery on-device; a full-precision server eval is insufficient.

Try context, memory and procedure before parameter updates when the issue is missing knowledge or workflow design. Weights earn their cost for high-volume strict behavior/format, missing domain vocabulary or a hard latency/cost ceiling after cheaper surfaces are exhausted. Fine-tuning freezes a task definition and is slower to revise when requirements change; keep a prompted fallback where useful. A compute spike should prompt small-model ablations, adapters, targeted rollouts and frequent checkpoints before another full-scale run.

Main · Deep Dive · Unchanged source

Math level? Final answer or valid working? Tools allowed? Who uses it? Base-model adaptation or pretraining? GPU/device budget?

Checkable data → dedupe/decontaminate → SFT format → sample model attempts → verifier/step/preference score → controlled update → held-out eval gate → optional distill/quantize → serve → reviewed data feedback

The model is trained; deterministic checks and human labels control advancement.

Training ≈ 6ND FLOPs (floating-point operations); generation ≈ 2N FLOPs/token. 7B × 50M SFT (supervised fine-tuning) tokens ≈ 2.1e18 FLOPs. 400M rollout tokens ≈ 5.6e18 to generate plus 1.68e19 to update. Full Adam state ~16 bytes/parameter, or ~112GB for 7B before activations.

• SFT teaches format; verified rollouts optimise correctness. Step grading is needed when working itself matters.
• Decontaminate and split by source/time; report pass@1 and sampling settings.
• Gate math, general regression, ill-posed answers, harmful compliance and over-refusal.
• Compute pivot: small ablations, adapters, targeted rollouts, full-state checkpoints, eval gain/GPU-hour.
• Phone pivot: student model, 4-bit weights plus KV/runtime, and on-device politeness/latency/battery checks.

60-second close: “Pin the target, use verified data, keep SFT short, score the model’s own attempts with the right checker, and promote only checkpoints that pass a clean multi-dimensional gate.”

Every model-building prompt spends one fixed budget of compute and data, and the constraint decides which stage of the pipeline gets the money.

These four prompts are one pipeline asked four ways, not four different systems. Each question fixes a different constraint, and that constraint changes exactly one stage: data, SFT (supervised fine-tuning), post-training, evaluation or deployment. This page consolidates the group's prompts into one read for the day before.

Most of this page is own construction. No worked design exists for any of these four prompts. The prompts themselves come from the OpenAI decomposition question bank. The grounding comes from hands-on fine-tuning labs, the Cracking book's chapter on learning surfaces, and the quantisation arithmetic in the air-gapped chapter. Sections and tables marked (own construction) were built for this page. Every number marked (assumption) is an illustration for the whiteboard, not a sourced figure.

A model cannot be trained toward a target nobody wrote down. "Solves math problems" hides at least four different products, and each one needs different data and a different evaluation (own construction).

The decomposition round rewards this move explicitly. The question bank's own framework starts with "Clarify the mission" and warns that the most common failure is jumping to architecture before scope. Say the assumptions aloud if the interviewer does not answer. A safe default is grade-school to competition word problems, final numeric answers, an optional code tool, and adaptation of an open base model.

"Before I pick a method I want the target pinned: which level of math, whether a final answer is enough or the working must be right too, and whether the model may call a tool. Those three answers decide the data, the reward and the eval."

A requirement that cannot fail a test is a preference. "Good at math" is a preference. "Pass@1 of at least the target on a held-out set that shares no problems with training" is a constraint. Only the second one shapes the pipeline (own construction throughout this section).

The must-haves are six. Correct final answers on the target distribution, measured on problems the model never saw. Shown working that is consistent with the final answer. No regression on general ability beyond an agreed tolerance. Refusal to fabricate when a problem is ill-posed. A reproducible training run, so any checkpoint can be rebuilt. An evaluation harness that runs on every candidate checkpoint before anyone reads a single sample by hand.

The should-haves come after the first model is trusted. Tool use through a code interpreter. Step-level feedback on the working. A tutoring mode that explains rather than answers.

Every must-have then needs an owner in the pipeline.

Arithmetic decides which methods are even on the table. The standard approximation for training compute is about 6 × parameters × training tokens in FLOPs (floating-point operations). Generation costs about 2 × parameters per generated token. Both are rules of thumb, good enough for a whiteboard and not for a purchase order.

Run the numbers on the math model (all figures are assumptions, arithmetic is exact).

The lesson is in the ratio. SFT is cheap. With these assumed workloads, post-training with sampled rollouts costs roughly ten times more, and generation runs well below peak throughput because it is memory-bound. So the compute budget belongs to post-training and to evaluation, not to the supervised stage.

Memory decides the hardware shape. A full fine-tune with the Adam optimiser in mixed precision needs roughly 16 bytes per parameter for weights, gradients and optimiser state. That is about 112 GB for a 7B model (assumption: the rule of thumb, before activations). It does not fit on one 80 GB card. Either shard the optimiser state across GPUs or train a small adapter instead of the full weights. Section 9 returns to both.

The pipeline is a loop, not a line. Data flows forward through training stages. Evaluation results flow backward into the next data mix. The organising rule is that no checkpoint moves forward without passing the eval gate (own construction for every diagram in this section).

 ╔═══════════════════════ CONTROL PLANE (changes are experiments) ═══════════════════════╗
 ║  data mixture config · training configs · reward definitions · eval suites + gates     ║
 ║  experiment tracker (run id, data hash, code version, seed) · compute budget per stage  ║
 ╚══════════════════════════════════════╤══════════════════════════════════════════════════╝
                                        │ configures every stage below
 ╔══════════════════════════ DATA + TRAINING PLANE ═══════════════════════════════════════╗
 ║                                                                                         ║
 ║  DATA            sources ─> dedupe ─> decontaminate vs eval ─> verify answers ─> mix    ║
 ║                  (public sets, licensed, synthetic from a strong model, human-written)  ║
 ║                                   │                                                     ║
 ║                                   v                                                     ║
 ║  BASE MODEL ──> SFT (worked solutions, format) ──> checkpoint ──┐                       ║
 ║                                                                 v                       ║
 ║  POST-TRAIN     sample N answers ─> score (verifier / reward model / preference) ─>     ║
 ║                 update policy (RL with KL to reference, or DPO on pairs) ─> checkpoint  ║
 ║                                                                 │                       ║
 ║                                                                 v                       ║
 ║  EVAL GATE      held-out accuracy · general regression · safety + over-refusal ·        ║
 ║                 contamination report ─── pass ──> release candidate                     ║
 ║                                     └── fail ──> error analysis ──> next data mix ─┐    ║
 ║                                                                                     │   ║
 ║  DEPLOY         distil / quantise if a size target exists ─> serving ─> traces ─────┘   ║
 ║                                                                  (flywheel, gated)     ║
 ║                                                                                         ║
 ║  CHECKPOINT STORE + TRACKER — every stage reads from and writes to it                   ║
 ╚═════════════════════════════════════════════════════════════════════════════════════════╝

The same pipeline for viewers that render Mermaid:

Math has a rare advantage: most final answers can be verified by a program. Build the data pipeline around that advantage. A problem whose answer cannot be checked is worth less than one that can, because nothing downstream can tell whether training on it helped.

Draw from four sources (own construction). Public problem sets with known answers. Licensed or customer-owned material. Human-written problems from domain experts for the hard tail. Synthetic problems and solutions generated by a stronger model, kept only when the final answer passes verification. Rejection sampling is the name for that last filter. Generate many solutions, keep the ones whose answer checks out, discard the rest.

The Foundations labs show the mechanical half of this stage. They contrast a pretraining corpus with a company fine-tuning dataset. They format examples as question-answer pairs with a prompt template, store them as JSONL, tokenise with padding and truncation, and split into train and test. That is the plumbing. The judgement is in what the Cracking book's learning chapter calls the three gates of the data flywheel.

Deduplicate first. In the book's worked example, 140,000 traces turned out to be only about 9,000 distinct requests after dedup. Split by time and by source, never at random, because a random split puts near-twins on both sides. Route low-agreement, high-impact examples to human adjudication, because an automatic label is not the same as a correct one. Ship a datasheet with every dataset: collection window, filters, dedup method, split strategy, label source and known biases.

SFT teaches the model what a good answer looks like, not how to find one it has never seen. Supervised fine-tuning trains on example inputs paired with the exact outputs wanted. For math, the outputs are worked solutions ending in a clearly marked final answer.

The Foundations labs make the contrast concrete. They compare a base model with an instruction-tuned one on the same prompt. The base model continues the text; the tuned one answers the question. They then train a small model, pythia-70m, for a handful of steps and compare it with the same model trained longer and with a larger model. In those labs, more training on the same data changed the style of the answer long before it changed whether the answer was right.

Three rules follow (own construction). Keep SFT short, because a model that over-fits the worked solutions reproduces their exact wording and loses flexibility. Mix general instruction data in, so the model does not forget how to follow ordinary requests. Include examples that decline an ill-posed problem, or the model will never learn that "this has no unique answer" is a legal output.

Parameter-efficient fine-tuning is the default for iteration. An adapter method such as LoRA trains a small set of added weights and leaves the base model frozen. QLoRA does the same on a base model loaded at 4-bit precision. A typical Llama 2 AutoTrain run uses --use_peft and --use_int4 (current AutoTrain calls them --peft and --quantization int4) to fit a 7B model on modest hardware. Use adapters for fast experiments. Move to a full fine-tune only when an adapter demonstrably plateaus below the target.

Post-training is where the model learns to find answers, because it is trained on its own attempts rather than on someone else's solutions. The loop is always the same shape. Sample several answers per problem. Score them. Push the model toward the higher-scoring ones.

The scorer is the whole design, and math offers three choices (own construction).

Two update methods sit on top of the scorer. RLHF (reinforcement learning from human feedback) trains a separate reward model from human comparisons, then optimises the policy against it with a penalty for drifting too far from the starting model. DPO (direct preference optimisation), direct preference optimisation, skips the separate reward model. It trains directly on preferred-versus-rejected pairs and gets the same drift control implicitly. DPO is simpler to run and usually more stable, though a well-tuned PPO can still beat it. RL with a verifier is stronger when correctness can be checked by program, which is exactly the math case.

The drift penalty is not optional. A policy optimised against any scorer without it will find the scorer's blind spots. A lenient answer-matcher gets answers that format like the reference but are wrong. A length-biased reward model gets padding. Keep the penalty, and audit high-reward samples by hand every round.

"For math I would lead with reinforcement learning against a program that checks the final answer, because correctness is verifiable. I would keep a penalty for drifting from the SFT (supervised fine-tuning) model, add a step-level reward only if working quality is a stated requirement, and use preference data for tone."

An eval score is only as honest as the wall between training and test data. The Cracking book's worked example makes the point. A fine-tune reported 94% on a held-out set against an 81% baseline, then hit only about 78% in production. The held-out set was split at random from traces with heavy duplication. After dedup, a time-based split and human relabelling, the retrained model's honest score was near 83%, and its later production result landed within two points of that. When an offline gain looks surprisingly large, suspect leakage first.

Build the eval suite in five slices (own construction).

Report sampling settings with every number. Pass@1 at temperature zero and majority vote over eight samples are different measurements. Report both if both are used, and never compare one with the other.

The Foundations evaluation lab shows the simplest version: run the model over a test set, compare with expected answers, then try a public benchmark, ARC. Exact-match comparison is where every eval starts. The release gate for this pipeline follows the same logic as G13's evaluation and release-gating page: fixed suites, thresholds agreed in advance, and a gate that blocks rather than advises.

Case #73 changes the training stage. "Design a scalable, efficient system for training an LLM (large language model) given compute and data constraints" is the same pipeline with pretraining back in scope and the budget fixed (own construction throughout).

Start from the budget, not the model. A widely used rule of thumb from compute-optimal scaling work is roughly 20 training tokens per parameter. For a 7B model that is 140B tokens. The compute is then 6 × 7e9 × 1.4e11 = 5.88e21 FLOPs (floating-point operations). At an assumed 3e14 effective FLOP/s per GPU, that is about 5,444 GPU-hours, or about 21 hours on 256 GPUs (assumptions; arithmetic exact). If the budget is smaller, shrink the model before shrinking the data.

Distribute the work in the order the memory forces it.

The data constraint is a pipeline problem. Stream shards from object storage rather than loading a dataset. Deduplicate and filter quality before training, because a repeated token costs the same as a new one and teaches less. Record the exact shard order, so a resumed run sees the same data in the same order.

Checkpointing is the reliability story. At hundreds of GPUs over days, some hardware will fail. Write checkpoints to durable storage on a fixed interval, including optimiser state and data-loader position. Resume automatically from the last good one. Watch for loss spikes and roll back to the checkpoint before the spike rather than training through it. Track model FLOPs utilisation as the efficiency metric, because GPU-hours spent waiting on communication or input are spent without learning.

"I size from the budget: tokens per parameter picks the model size, six N D gives the FLOPs, and memory picks the parallelism. Then I make the run survivable: sharded state, streaming data with a recorded order, and checkpoints that include the data position so a failure costs minutes, not days."

Case #81 changes the post-training and eval stages. "Design a language model that minimises harmful outputs while staying useful" is a two-sided objective. A model that refuses everything is perfectly harmless and useless. Measure both sides, or the optimisation will quietly collapse to one of them (own construction throughout).

Plot two numbers against each other for every checkpoint. The first is the harmful-compliance rate: on a set of genuinely harmful requests, how often does the model help? The second is the over-refusal rate: on a set of benign requests that merely look risky, how often does it refuse? "How do I kill a stuck Python process?" belongs in the second set. A release moves down on the first number without moving up on the second.

Post-training carries the policy. Write the policy down first: what is disallowed, what is allowed with care, what is always allowed. Collect preference pairs where the preferred answer is the one that follows the policy. That includes pairs where the preferred answer helps and the rejected one refuses needlessly. Train with RLHF or DPO on those pairs. AI feedback against the written policy, the Constitutional AI idea, scales the labelling when human graders are the bottleneck. Humans still audit a sample.

The enterprise RAG (retrieval-augmented generation) design's output guardrail makes the same point at the application layer. Its sufficiency check has three verdicts, not two: sufficient, partial and insufficient. The middle one exists because refusing a whole two-part question when only one part is unanswerable is a common over-refusal. The model-level version is the same idea. Answer the safe part, decline the unsafe part, and say which is which.

Red-team continuously. Attack prompts change faster than training runs, so keep an adversarial set that grows from real attempts and run it in the eval gate. A layered deployment adds a separate safety classifier on inputs and outputs. That lets the policy tighten between training runs without retraining the model.

Case #83 changes the deploy stage. "Design a small LLM that runs on a phone while staying polite" fixes a memory and latency ceiling, and it names politeness as an alignment target (own construction throughout).

Size the weights first. Memory for weights is parameters × bytes per parameter.

A phone shares its memory with the operating system and other apps. Assume a budget of 1 to 2 GB for the model (assumption). That points at about a 1B model at 4-bit; a 3B model needs ~1.5 GB for weights alone, so it only fits at the top of that budget. The air-gapped chapter's sizing adds the reminder that weights are not the whole bill: runtime overhead, the KV (key–value) cache at the target context length and buffers come on top. Cap the context length, because the KV cache grows with it.

Two techniques make the small model good enough. Distillation trains the small student on a large teacher's outputs. Standard system-design references frame it the same way: a smaller model trained on a larger one's outputs for a task-specific workload. The Cracking book names a hard cost or latency ceiling as one of the three situations where parameter updates earn their cost, and distillation is how. Quantisation then compresses the trained student. Quantisation-aware training, which simulates low precision during training, loses less quality than quantising after the fact.

Politeness is a preference target, trained like harmlessness. Write a short tone policy. Collect preference pairs where the polite answer wins and the obsequious or curt one loses. Include the teacher's polite outputs in the distillation data, so tone transfers with capability. Evaluate politeness with a rubric grader on a fixed prompt set, including provocative user messages, because politeness is only tested when the user is rude.

Measure everything on the device, not on the server. Report quality of the quantised student against the full-precision student and the teacher. Report time to first token and tokens per second on the slowest supported phone. Report battery drain per session. A quality eval run on the unquantised model does not describe what users get.

Training is the most expensive way to change a model's behaviour, so defend it before proposing it. The Cracking book's learning chapter orders four places improvement can live: context, memory, procedure and parameters. Each is slower and costlier to change than the one before; the book calls it roughly ten times per step, as a rough order, not a measurement. Work down that list, not up.

The book names three situations where parameter updates decisively win. Strict format or protocol adherence at high volume, where a small tuned model can often match a larger prompted one at a fraction of serving cost. Domain vocabulary the base model lacks. A hard cost or latency ceiling after cheaper surfaces are exhausted, where distillation buys the ceiling back. It is equally blunt about the losing case: knowledge gaps are mainly a retrieval problem, and fine-tuning is a weak, unreliable way to teach a model new facts.

Map the four prompts onto that rule (own construction). The math model (#84) wins on reasoning behaviour that prompting cannot install. The training system (#73) is parameter updates by definition. The harm-versus-help model (#81) wins because refusal behaviour lives in the weights, with classifiers as a faster outer layer. The phone model (#83) is the latency-ceiling case exactly. Say the rule aloud before the pipeline, because it shows the choice was earned rather than assumed.

A tuned model freezes its task definition, which is the cost the book asks to name. When the business definition changes, a tuned model needs retraining where a prompt needs an edit. Keep a prompted fallback path for that reason.

Spend the hour on the scorer and the eval, because that is where these prompts separate candidates (own construction).

The interviewer will change a constraint mid-round; the question bank says so explicitly. Map the new constraint to the stage it changes and re-derive only that stage. "You now have a tenth of the compute" changes model size and the SFT-versus-RL split. "It must run offline on a laptop" changes deploy. "Legal says no synthetic data" changes data collection and pushes harder on the verifier.

The interviewer's pivot after a good design is "the training bill is out of control." This group has no playbook drill, so the card below is (own construction).

The four verbs from the cost playbook still generate the answer. Measure, by attributing GPU-hours to stage and experiment. Route, by matching method to need: adapters for iteration, full fine-tune for the final run, distillation for serving. Bound, with a budget per stage and a kill rule. Cache safely, by reusing verified synthetic data and scored rollouts across experiments when the policy has not moved far.

Every level can name SFT and RL; the level shows in which stage gets the budget and who decides a checkpoint is ready to move forward.

• Pin the target first: level of math, answer versus working, tools and users decide the data, reward and eval.
• Requirements become testable as held-out pass@1, a maximum general regression, a fabrication rate and a compute budget.
• Sizing shows SFT is cheap and post-training rollouts are expensive, so the budget belongs to post-training and eval.
• The architecture is a loop: data, SFT, post-training, an eval gate that blocks, deploy, and a gated flywheel back into data.
• Math data should be checkable: verify answers by program, dedupe, decontaminate and split by time and source.
• SFT teaches the shape of an answer; keep it short, mixed with general data, and include legal refusals.
• Post-training finds answers; the scorer is the design, and the drift penalty stops the model gaming it.
• Evaluate only on problems the model never saw, report sampling settings, and suspect leakage when a gain looks large.
• Under a compute budget, tokens per parameter picks the model size, memory picks the parallelism, and checkpoints make the run survivable.
• Harm and helpfulness are two numbers on one chart; a release lowers harmful compliance without raising over-refusal.
• A phone model is sized by bytes per parameter, made good by distillation, compressed by quantisation, and evaluated on the device.
• Parameter updates earn their cost for behaviour, vocabulary or a latency ceiling, never for missing knowledge.
• The hour goes to the scorer and the eval, and a mid-round constraint changes one stage, not the whole design.
• The compute pivot is answered with small ablations, adapters, targeted rollouts, checkpoints and eval gained per GPU-hour.

1. What three questions pin the target for a math model? Which level of math, whether a final answer is enough or the working must be right, and whether tools are allowed.
2. Why is post-training more expensive than SFT in the sizing? It samples many answers per prompt, so generated tokens multiply, and generation runs well below peak throughput.
3. What does the eval gate block on? Held-out accuracy, general regression, safety and over-refusal, with a contamination report attached.
4. Why prefer checkable answers in the data? Only a checkable answer lets rejection sampling filter synthetic data and lets a verifier score rollouts.
5. What does SFT teach, and what does it not? It teaches the format and shape of a good answer, not how to find answers to unseen problems.
6. Why keep a drift penalty in post-training? Without it the policy exploits the scorer's blind spots, such as formatting tricks or padding.
7. The fine-tune scored 94% offline and 78% in production. What went wrong? A random split over duplicated traces, labels from the old system, and a narrow time window; the honest number was near 83%.
8. How do you size a 7B pretraining run? About 20 tokens per parameter gives 140B tokens; 6 N D gives 5.88e21 FLOPs; at the assumed throughput that is about 5,444 GPU-hours.
9. Which two numbers does #81 plot, and which way must each move? Harmful-compliance rate must fall; over-refusal rate on benign-but-scary prompts must not rise.
10. How much memory do 3B parameters need at 4-bit, and what else must fit? About 1.5 GB for weights, plus runtime overhead, the KV (key–value) cache and buffers.
11. When does a parameter update not earn its cost? When the gap is missing knowledge; that is a retrieval problem.
12. What is the sixty-second compute answer? Prove the recipe small, iterate with adapters, spend rollouts on hard problems, checkpoint, and track eval gained per GPU-hour.

Fact-check sources (checked 27 Sep 2026)

• Kaplan et al. 2020, Scaling Laws for Neural Language Models — training ≈ 6 × parameters × tokens FLOPs; generation ≈ 2 × parameters per token
• Rajbhandari et al. 2019, ZeRO — ~16 bytes per parameter for mixed-precision Adam (≈112 GB for 7B)
• Hoffmann et al. 2022, Training Compute-Optimal LLMs (Chinchilla) — ~20 training tokens per parameter
• AutoTrain LLM fine-tuning parameters — current flag names --peft and --quantization int4
• Singhal et al. 2023, A Long Way to Go: Length Correlations in RLHF — graders favour longer answers
• Hosking et al. 2023, Human Feedback is not Gold Standard — graders favour assertive, confident answers
• Rafailov et al. 2023, Direct Preference Optimization — DPO trains on preference pairs with no separate reward model
• Xu et al. 2024, Is DPO Superior to PPO for LLM Alignment? — well-tuned PPO can match or beat DPO
• Lambert et al. 2024, Tulu 3 (RLVR) — RL with a programmatic verifier works well where correctness is checkable
• DeepSeek-AI 2025, DeepSeek-R1 — rule-based rewards for math and code
• PyTorch blog: Quantization-Aware Training for LLMs — QAT loses less quality than post-training quantisation
• Ovadia et al. 2023, Fine-Tuning or Retrieval? — RAG (retrieval-augmented generation) beats fine-tuning for injecting new facts; fine-tuning does so weakly
• Zhao et al. 2024, LoRA Land — small LoRA-tuned models often match larger prompted ones on narrow tasks, varying by task