The same tasks across models

What you'll learn

• Explain why a fixed prompt set and one swappable wrapper make a fair model comparison.
• Choose a check for each kind of task, from exact match to a support judgement.
• Diagnose a comparison that rests on reading replies by eye or on a tiny, leaky test set.
• Adapt a closed-domain question-answering prompt into the answer step of a RAG pipeline.

Lessons 1 and 2 of T3.1 built prompts for one model. A prompt is written against a model, though, and models change: a cheaper one appears, a provider retires an id, a team wants an open model it can host.

Suppose the Larkspur team moves its support assistant from gpt-4o-mini to a cheaper model. Someone sends five handbook questions to the new model, reads the replies, finds them fluent and ships the change. A week later the answer step is filling gaps in the handbook with general knowledge (illustrative). No check measured whether each answer stayed inside the retrieved passages.

The four notebooks of this lesson show the setup that catches it: one fixed prompt set, one wrapper per provider, and a score per task.

The four notebooks run the same ten tasks: zero-shot and few-shot classification, Python and SQL generation, information extraction from a clinical note, closed-domain and open-domain question answering, summarisation, turning a fact sheet into an FAQ, and translation.

Each notebook defines one function, get_completion(prompt, model), and every task calls it. Only that function changes between notebooks:

Table 1: The wrapper is the only part that changes.

All four set temperature 0, and 03d caps replies at 2,048 tokens. 03d's first test call goes to a different model, microsoft/Phi-3.5-mini-instruct; its ten tasks all use the Llama default. The prompt text is identical in 03a, 03c and 03d. 03b adds one prompt, a second LangGraph question ending If you are not able to find the answer in your trained knowledge just say you don't know, and for that task its run answers a different prompt.

The structure in one picture:

Figure 1: The prompt set is the fixed part and the wrapper is the swap point. The last box is the one the notebooks leave to you.

Three practices make the comparison worth trusting.

Score, do not eyeball

The notebooks print each reply. A comparison needs a check per task, decided before the run. Several tasks carry their own answer key in the prompt or its context:

Table 2: A check per task. Expected answers come from the notebooks' own inputs, checked by hand.

Different outputs call for different checks. Ask in this order:

Figure 2: The cheapest check that can fail a wrong reply.

Planted failure: the leaky two-review test

The sentiment tasks show how a test set can mislead. Task 1 asks for Positive, Negative or Neutral. Task 2, the few-shot version, switches the labels to 😊 and 😡 and gives no Neutral example. Its two examples open with the same words as the two reviews under test (Just received the… and Needed a new…), and the test set is two reviews long:

Table 3: Three ways the sentiment tasks would mislead a scorer.

We call this the leaky test: the examples share text with the inputs, and the inputs are too few to separate models. A model comparison needs its own labelled cases, kept apart from any examples in the prompt. That is the T3.1 lab brief on a small scale: 30 labelled pairs, agreement per prompt.

Keep the model id in one place

Provider ids are retired. Groq shut down llama-3.2-90b-vision-preview on 14 April 2025 and names meta-llama/llama-4-scout-17b-16e-instruct as the replacement (Groq deprecations, checked 9 November 2026). Notebook 03c and the Llama half of notebook 04a call that id and fail until it is changed. We call this id rot. With the id inside get_completion, the fix is one line.

Access can also change. The 1B model in 03d is gated: the notebook asks you to request access from Meta on Hugging Face before the first call works.

Read the prompt as each model will

Task 6, closed-domain QA, gives a survey report as context and asks the model to answer to the best of your ability. That wording invites the model's own knowledge. In a RAG pipeline the same prompt needs an instruction to answer only from the context, and to say so when the context does not contain the answer. That is the line 03b adds to its extra prompt.

The closed-domain prompt becomes the answer step of the Larkspur pipeline. It has four parts:

Table 4: The Larkspur answer prompt, adapted from task 6.

To compare candidate models on it:

1. Freeze this prompt and the golden questions with their retrieved passages.
2. Run every candidate through the same wrapper at temperature 0.
3. Score each answer for support: every claim must appear in the passages. Fluency earns nothing.
4. Include the five golden questions the handbook cannot answer. For those, the refusal line is the only correct reply.

Take What happens if my server goes down? with only the reboot passage retrieved. A model that adds status-page advice or service credits from general knowledge fails step 3, however good the advice sounds. The answer either sticks to the reboot instructions or uses the refusal line.

Interview tip: when asked how to choose between models, describe the frozen prompt set and the per-task checks before naming any model.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Keep the prompt set fixed and swap models behind one wrapper, so the model id lives in one place and a retired id is a one-line fix.
• Score each task with its own check (exact match, format and length, support by the context), on enough labelled cases kept apart from the prompt's examples.
• For a RAG answer step, tell the model to answer only from the passages and to say when they do not cover the question. Score answers for support by the passages, not fluency.