Fine-tuning and model customisation: think like a staff engineer

Use tenant-scoped storage credentials for datasets, checkpoints, outputs, and tracking metadata. Access to training artefacts is separate from document read access. Record approver, purpose, budget, and lineage without raw training examples in metrics. Freeze evaluator and gate identities alongside each report. Any changed base, recipe, tokeniser, data, or serving configuration invalidates assumed compatibility and requires new evidence.

A deletion request must traverse snapshots, examples, checkpoints, registry versions, serving copies, caches, and retained backups according to approved policy. Deleting a row does not erase learned parameters. Quarantine affected descendants, stop future serving where required, and plan retraining from an approved clean snapshot. Physical erasure, legal retention choices, and machine unlearning are outside this fixture. Do not promise selective forgetting.

Prompt: Alice requests fine-tuning because the current output schema sometimes fails.

Spoken answer: “I would first classify the failures. A stale supplier rule needs current authorised retrieval. Formatting may improve with a clearer prompt, examples, schema constraints, and validation. If stable behaviour still fails, I would run a bounded adapter challenger against the frozen baseline. I would compare behaviour and safety by slice, and include training, serving, and operating cost. Approval would select a versioned candidate without weakening current access.”

Trade-off: more custom control creates a longer lifecycle. Follow-up: if the supplier rule changes daily, keep it in retrieval rather than retraining daily.

Prompt: Two teams report different quality for the same friendly model name.

Spoken answer: “I would resolve the name to immutable base, adapter, tokeniser, dataset, recipe, and serving identities. I would also pin the evaluator and frozen test population. A seed helps but does not prove identical floating-point execution across hardware. I need traceable inputs and comparable evaluation evidence, plus checkpoint compatibility for recovery. The model alias is an operational pointer; it cannot stand in for that record.”

Trade-off: detailed provenance costs storage and review time. Follow-up: a changed serving quantisation requires a new evaluated bundle.

Prompt: A worker disappears halfway through a paid run.

Spoken answer: “I would inspect remote run state and the last verified checkpoint before creating another run. Recovery needs compatible training code, input position, optimiser and scheduler state, and random state. A newer log line is insufficient. I would bound retries and remaining budget, preserve lineage from the parent attempt, and keep training capacity separate from interactive serving. The course fixture checks checkpoint ordering only; real resume needs a trainer integration test.”

Trade-off: frequent checkpoints reduce recomputation but consume I/O and storage. Follow-up: if remote status is unknown, reconcile before retrying.

Prompt: Tenant A revokes permission to use a supplier corpus after training.

Spoken answer: “I would locate every derived model and checkpoint through dataset lineage and apply the approved retention and serving policy. Row deletion cannot remove information already learned in weights. I would block affected candidates, assess deployed versions, and retrain from an approved snapshot where required. Document retrieval would still check current permission separately. I would avoid promising selective forgetting or treating an older model rollback as an authorisation rollback.”

Trade-off: retirement can reduce availability while clean candidates are prepared. Follow-up: if backups must remain, restrict access and document expiry separately.

• Classify failures before tuning. A stale supplier rule needs current authorised retrieval, formatting may yield to prompts and schema constraints, and only stable residual misses justify a bounded adapter challenger against the frozen baseline.
• Reproducibility means resolving a friendly model name to immutable base, adapter, tokeniser, dataset, recipe, serving, evaluator and test identities. A seed does not prove identical floating-point execution across hardware.
• Real training resume needs compatible code, input position, optimiser, scheduler and random state, not a newer log line. Frequent checkpoints cut recomputation but cost I/O and storage.
• Deleting a dataset row does not erase learned parameters. Trace descendants through lineage, quarantine them, retrain from an approved clean snapshot, and never promise selective forgetting.