Fine-tuning and model customisation: observe and experiment

The offline walkthrough traces the approved candidate path and blocked safety/report paths. Each step is a design boundary; the lab implements metadata checks rather than external execution.

Sequence diagram. No training completion implicitly changes the live Model Gateway route. A production release still passes T18.9 and T18.10 gates.

Assume each training example contains 700 input and 100 target tokens. Then 10,000 × 800 = 8 million tokens per epoch; three epochs process 24 million tokens. At an assumed effective 2,000 training tokens/second, that is 12,000 seconds, or 3.33 hours, excluding setup, checkpoint, evaluation, retries, and failed runs. This assumed rate is not measured inference throughput. At an invented planning rate of $4/device-hour, one-device training time costs $13.33 before those extras. The rate is a scenario input, not a provider quote.

Serving an adapter adds model/base compatibility, load time, memory, and tenant routing work. A smaller adapter file does not show end-to-end savings. If custom behaviour saves 100 input tokens per request across 50,000 requests/day, it saves 5 million input tokens/day under that assumption; compare actual token pricing, quality, serving overhead, and staff time before approving a cost case.

Training memory includes activations, gradients, and optimiser state as well as weights. T18.10's inference KV worksheet cannot size a trainer. Reserve training separately from latency-sensitive GPU Pool capacity, enforce tenant budgets, and checkpoint before planned eviction where supported. Pin image/device requirements and verify readiness and artefact integrity before scheduling. Full GPU scheduling and hardware isolation remain proposed.

Open the offline lab. Change schema passes (0–200 cases), safety failures (0–5 cases), and whether the report matches the candidate identity. The deterministic gate accepts at least 196 schema passes, exactly zero known safety failures, and matching identity. It does not simulate training improvement, statistical confidence, or a real safety evaluator.

Run this stage's Python lab in your terminal to see each idea in the stage hold. It needs Python 3.10 or later and nothing else: the standard library and SQLite only.

From the root of your local clone of the repository, run:

python3 10_AI_System_Design/stages/10-fine-tuning/labs/test_lifecycle.py -v

-v prints one line per test, and each test name states one behaviour. Read a test, change one value in the lab code, predict which test fails, and run it again.

The executable slice freezes example metadata, rejects supplier-group split leakage, scopes job keys, records monotonically increasing checkpoints, binds trusted reports, and checks role and release generation. It uses in-memory records and supplied counts. It does not call the T18.9 release controller or T18.10 gateway; those contracts are retained as production integration requirements, not tested composition.

Inject a safety failure, then attempt approval. Expect a false gate result, a quality gate blocked error, and an unchanged live pointer. Inject a report for another artefact and expect report identity mismatch before approval. A trainer role cannot approve. A stale generation cannot overwrite a newer pointer. In production, alert on blocked promotion count, orphan checkpoint age, spend against budget, invalid lineage, and job queue age; separate configuration errors from genuine model regressions.

A worker reports step 200 locally but only step 100 has a verified checkpoint. What may recovery claim?

• At 800 tokens per example, 10,000 examples are 8 million tokens per epoch and 24 million over three epochs. At an assumed 2,000 tokens per second that is about 3.33 hours, or $13.33 at an invented $4 per device-hour, before setup and retries.
• Training memory includes activations, gradients and optimiser state as well as weights, so the inference KV worksheet cannot size a trainer. Reserve training capacity separately from latency-sensitive serving.
• The gate accepts at least 196 schema passes, exactly zero known safety failures and a report bound to the candidate's identity. A safety failure gives quality gate blocked, a report for another artefact gives report identity mismatch, and the live pointer stays unchanged.
• If a lost worker logged step 200 but only step 100 has a verified checkpoint, recovery restarts from step 100 and may repeat work. Inspect a remote run of unknown status before submitting another billed job.