Instruction tuning

What you'll learn

• Explain what instruction tuning teaches a base model, and why the lesson is a pattern rather than a set of facts.
• Trace an Alpaca row through a prompt template into one line of training data.
• Diagnose template drift, where the serving prompt no longer matches the training text.
• Apply three rules that keep training and inference in step.

Lesson 2 of T14.1 stored each example as a prompt field and an answer field. What sits between and around those fields, the headers, blank lines and end marker, is the part this lesson is about. A Larkspur engineer who fine-tunes on rows that read ### Question: … ### Answer: and then serves the bare customer question gets answers that start mid-sentence, the same format mismatch the Llama 2 chat model showed in lesson 1 of T14.1 without its [INST] wrapper.

Instruction tuning is fine-tuning on pairs of a request and a good response. It is the first step that turned the base Llama 2 of lesson 1 of T14.1 into the chat model. What it teaches is a pattern: text shaped like a request is followed by text shaped like an answer, and then the answer ends.

The notebook streams the first five rows of tatsu-lab/alpaca, an instruction dataset where every row has three fields:

Table 1: The fields of an Alpaca row.

A prompt template turns each row into training text, and the notebook keeps two. Both open with a one-sentence preamble and use ### headers to mark the parts:

Table 2: The notebook's two Alpaca templates, part by part.

Filling a template with a row's values is what the notebook calls hydrating the prompt. The row's input field picks the template:

Figure 1: Hydrating an Alpaca row. All five rows in the notebook take the upper branch.

The result is written to alpaca_processed.jsonl with one object per line: the hydrated prompt under input and the answer under output. The names collide with Alpaca's own input field: read them as "model input" and "model output". The same shape works for Larkspur:

Table 3: A Larkspur support exchange mapped onto the Alpaca fields (illustrative).

The notebook ends by asking the same lamini/lamini_docs test question to two models with the same architecture and tokenizer: the base EleutherAI/pythia-70m, and lamini/lamini_docs_finetuned, which is pythia-70m after fine-tuning on that dataset. Any difference between the two answers comes from the training rows alone. The notebook ships without saved outputs; the two answers appear only when you run it.

A model trained on hydrated rows learns their exact characters, which makes the serving prompt part of the training contract.

A model learns the exact surface form of its training text, and the prompt at inference has to match it. When it does not, we call it template drift: the training rows used one shape and the serving code sends another. Lesson 1 of T14.1 showed the cost on Llama 2, which finished the user's sentence ("on command.") before answering.

The T14.1 notebooks hold a quiet example. Lesson 4 of T14.1 builds ### Question: / ### Answer: prompts, but the lamini/lamini_docs copy that lesson 5 of T14.1 trains on carries none of them. In all 1,260 training rows, the decoded input_ids are the bare question followed immediately by the bare answer (…from Lamini models?There are several metrics…), with no separator and no end token. The inference helper then sends the bare question, which matches that training text, so the two stay consistent by accident.

Changing either side alone breaks that accident. The planted cases below each change one side:

Table 4: Planted template drift. Only the last row is a match by design.

Three rules keep training and inference in step:

• One template, in code, used by both. Write the formatting function once and import it in the data script and the serving code.
• Mark where the answer starts and where it ends. A header such as ### Answer: tells the model to start answering; an end token (<|endoftext|> for Pythia) tells it, and your decoder, that the answer is over.
• Check the tokenized row, not the string. Decode a few prepared rows before training; the lab in lesson 4 of T14.1 shows what to look for.

The end token is the rule the T14.1 data breaks, and it is the one with the loudest consequence. Lesson 4 of T14.1 shows where it has to be added, and lesson 6 of T14.1 shows what a model trained without it does.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Instruction tuning trains on request and response pairs. A prompt template turns each row into training text, with a variant for rows that have no input.
• A model learns the exact surface form of its training text, and the inference prompt must match it. Changing the training format or the serving prompt alone causes template drift.
• Write one formatting function that the data script and the serving code both import, mark where the answer starts and ends, and decode a few tokenized rows before training.