Data preparation

What you'll learn

• Explain how a formatted example becomes one row of token ids in a rectangular batch.
• Trace tokenizing, padding and truncation on real Pythia token ids.
• Diagnose the three silent failures: a cut answer, a cut question and a missing end token.
• Check a dataset for empty rows, duplicates, refusals and length before training on it.

Lesson 3 of T14.1 ended on a rule: decode the tokenized row, not the string. This page shows why. A Larkspur engineer sets a token limit of 48 to save memory, trains, and finds the assistant now stops the refund answer mid-sentence. Nothing raised an error. The limit cut the end off every long answer before the model ever saw it.

Data preparation turns each formatted example into one row of token ids, makes the rows fit a length limit and a common width, and holds some rows back for evaluation. A model trains on token ids, in rectangular batches, and every step below exists to produce one:

Figure 1: The preparation pipeline, from formatted text to a split, labelled dataset.

Tokenize, pad and truncate

The notebook loads the EleutherAI/pythia-70m tokenizer, the one its model was trained with. Text and tokens are different units: the first lamini_docs example is 358 characters and 77 tokens. On a short sentence the three operations look like this:

Table 1: Pythia tokenizer operations on the notebook's sample sentences, checked against the tokenizer in the scratchpad.

Padding exists because a batch is one tensor, so its rows need one width. Pythia's tokenizer has its own <|padding|> token (id 1), but the notebook sets the pad token to the end-of-text token. Padding therefore uses id 0, the end-of-text id. attention_mask marks real tokens with 1 and padding with 0, and attention skips the padding. Truncation caps every row at max_length; the default cuts from the right, and the notebook switches the tokenizer to cut from the left.

Label and split

For a causal language model the target is the input itself, shifted by one inside the model. The notebook adds labels as a copy of input_ids. A shuffled split with 10% held out and seed 123 then turns the 1,400 rows into 1,260 for training and 140 held out. The fixed seed gives the same split on every run, which is what lets lesson 6 of T14.1 compare models on the same 140 questions.

Three failures that raise no error

Each choice above changes what the model learns, and three of them fail silently:

Table 2: Three silent failures in data preparation, one row per guard.

The guards follow from the table. Append the tokenizer's end-of-text token to each training text. Since padding reuses id 0 and labels copies it, set padded positions in labels to -100 so the loss ignores them. And choose max_length from measured lengths, not from a memory budget alone.

The interactive lab builds a three-example Larkspur batch (refund, reboot and export questions in the ### Question: template) with the real Pythia tokens. At max_length = 48 right truncation cuts 11 answer tokens from the 59-token refund example. Left truncation keeps that answer but cuts the 4-token header and 7 question tokens instead. Padding to the longest row pads the two shorter examples by 15 and 3 tokens, 13% of the batch. With the end token appended and padding to a fixed max_length of 64, the batch carries 52 padding positions, 27%. No row ends with <|endoftext|> until you append it.

Those counts depend on the batch, so the next question is what limit this dataset needs:

Table 3: Token lengths of question plus answer, measured with the Pythia tokenizer in the scratchpad.

In the notebook the limit never bites: no row comes near 2,048. Measure your own length distribution before choosing max_length: set it just above the longest rows you want to keep whole, and decide on purpose what happens to the rest.

Two notebook details matter before you reuse its approach. Its tokenizing function reads only the first item of each batch it receives. It works only because the dataset is mapped one example at a time. And with one example per batch there is nothing to pad: the training run in lesson 5 of T14.1 never pads at all.

Tokenizing checks none of the content, which is the job of the next section.

Tokenizing does not check the data. Read the rows before you train on them; in lamini_docs.jsonl a short pass finds four things:

Table 4: Row checks on lamini_docs.jsonl, counted in the scratchpad.

The refusal rows show how a behaviour gets in: the model declines off-topic questions because 39 training answers do. A Larkspur dataset needs its own share of those rows, written in the house style, or the fine-tuned assistant will answer What do you think of Mars? as cheerfully as the base model. We call that gap missing refusals: a behaviour the team wants but no training row shows.

The duplicates point to an ordering rule. Split leakage happens when one copy of a question trains and another sits in the test split, and evaluation rewards memory. Deduplicate before you split, and check that no test question also appears in training.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Data preparation is tokenize, truncate, pad, copy input_ids to labels, then split with a fixed seed. Right truncation cuts the answer's end and left truncation cuts the question, both without an error.
• Append the end token to every training text and set padded label positions to -100. Measure the length distribution before you choose max_length.
• Read the rows before training. Remove empty rows, deduplicate before the split, and include refusal rows in the house style, because the model only declines what its training answers decline.