The RAG lifecycle and a measured baseline

What you'll learn

• Explain the seven lifecycle steps, the job of each one and its typical failure.
• Diagnose whether a wrong answer is a retrieval failure or a generation failure with one check.
• Measure a plain pipeline on a frozen golden set and record its baseline row.
• Choose whether to keep a change by comparing it with that row, question by question.

Lessons 2 to 4 of T5.1 built the pieces one at a time: an index, a retriever, a vector store. This lesson joins them into one pipeline and gives it a number to beat.

The refund question is still answered wrongly. A teammate spends an afternoon on the prompt: stricter wording, an instruction to quote the handbook, a worked example. The answer does not change, because in the baseline index the refund-timing chunk never reaches the top two. The model cannot quote a passage it never received.

The lifecycle has seven steps in two phases. Each step has one job and one typical failure:

Table 1: The seven lifecycle steps, with the module that fixes each step's typical failure.

When an answer is wrong, one question tells you which half to debug:

Figure 1: The one check that splits retrieval failures from generation failures.

Tuning the prompt cannot fix a retrieval failure, because the model never saw the evidence. We call that the wrong-half fix: effort spent on the half of the pipeline that did not fail. The notebook prints each retrieved chunk with its score for exactly this check. Applied to the three Larkspur questions at the baseline:

Table 2: The check applied to the TF-IDF baseline at k = 2. The teammate's prompt work targeted the refund row, where generation was never the problem.

Every later module in this course changes one stage of the pipeline: chunking, embeddings, hybrid search, reranking, query rewriting. Without a baseline you cannot tell whether a change helped, hurt, or moved the failure to another question.

The method has four parts:

1. Fix a golden set. 20 to 50 real questions, each with its evidence chunks and a reference answer. Freeze it, and add questions when production shows a new failure.
2. Build the plain pipeline. Default loader, recursive splitter, one embedding model, top-k similarity, a simple prompt.
3. Record the baseline row. Retrieval metrics (recall@k, hit rate, MRR), answer metrics (correctness, faithfulness), latency and cost per question.
4. Change one stage, re-run, compare. Keep the change only if the row improves on the questions it targets without breaking the others.

Hit rate is the share of questions with at least one evidence chunk in the top k. MRR (mean reciprocal rank) averages 1 divided by the rank of the first evidence chunk, so it rewards evidence that ranks high. The loop then runs for as long as there are changes to try:

Figure 2: The baseline loop. A change that wins on the mean but breaks a question goes back for rework.

For the handbook, the top-k lab gives the retrieval half of the row:

Table 3: The Larkspur baseline row. Answer metrics, latency and cost join it once the notebook runs generation.

That row already points at the next modules. The export question is solved, the refund question needs better term matching (T5.4, T6.1), and the outage question needs paraphrase-aware retrieval (T5.4, T6.2). The lifecycle notebook ends with the map from symptom to module:

Table 4: Symptom to module. Each later module is a fix for one of these rows.

Lesson 3 of T5.1's switch to embeddings is the first change to run through the loop:

1. Re-run the golden set with embeddings at k = 2.
2. Compare the mean. Recall@2 rises from 0.50 to 0.67, and precision@2 from 0.50 to 0.67.
3. Compare each question. Refund rises from 0.50 to 1.00 and outage from 0.00 to 0.50, while export falls from 1.00 to 0.50.
4. Decide. The change broke a question it was not aimed at. The loop says revert, or rework it (raise k to 3, or add the exact-word signal back with hybrid search in T6.1) and run the golden set again.

Interview tip: when someone reports a mean improvement, ask for the per-question rows. A better average can hide a question type the change broke.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• RAG runs in two phases: indexing (load, split, embed, store) and query (retrieve, augment, generate). Each step has one job and one typical failure.
• Debug a wrong answer by printing the retrieved chunks first. If the evidence is missing, no prompt change can help.
• Freeze a golden set of 20 to 50 questions, record a baseline row, and keep a change only when it helps its target questions without breaking others.