Reliable RAG: relevance checks, hallucination checks, source highlighting

What you'll learn

• Explain what a relevance grader removes before generation, and why it can never add the chunk that holds the answer.
• Diagnose an answer that states a number its kept chunks do not contain.
• Verify highlighted quotes against their source text, and handle the ones that fail.
• Count the extra model calls the three checks add to every answer.

T9.4 made the retriever a tool the agent calls when it decides to. This module asks what happens after the call returns: whether the chunks are on topic, whether the answer stays inside them, and whether the user can see where each claim came from.

A Larkspur customer asks How long does a refund take? The answer comes back fluent and confident, with a number of business days that differs from the handbook's 5 to 10. That duration sits in the refund-timing chunk, and that chunk was never retrieved. The model found a related passage about refunds, saw no duration in it, and filled the gap from memory. Nothing in a plain RAG pipeline notices.

Lesson 1 of T9.5 adds three checks around generation, each catching a different failure:

Figure 1: The reliable RAG pipeline. A grader filters chunks before generation, a second grader checks the answer after it, and highlighting ties each claim to a quote.

A grader before generation

A relevance grader is a small model call that reads one retrieved chunk together with the question and returns a structured verdict, yes or no. Chunks graded no are dropped; only the kept ones reach the generator.

The notebook indexes five posts on agentic design patterns from DeepLearning.AI's The Batch (500-token chunks, no overlap, OpenAI's text-embedding-3-small, Chroma) and retrieves 4 chunks per question. Every grader runs Llama 3.1 8B on Groq at temperature 0 with structured output, so the verdict is a field the code can branch on.

The grading prompt sets a low bar on purpose: a chunk counts as relevant if it shares keywords or meaning with the question, because "The goal is to filter out erroneous retrievals." A strict grader would drop borderline chunks that still help.

What grading can and cannot fix

Run the refund question against the T5.1 TF-IDF index of the Larkspur handbook and the grader's job becomes concrete:

Table 1: The refund question on the T5.1 index, and what a relevance grader sees.

Dropping reboot removes a distractor. But the chunk that holds the answer is absent, and no grader can add it. We call this the candidate ceiling: every check downstream of retrieval works on the candidates retrieval returned, so recall stays where retrieval left it. Fixing the missing chunk is retrieval work (T5.4 to T6.3).

Pitfall: the empty-context answer. The notebook passes the kept chunks to generation without checking whether any survived. If all four are graded no, the model gets no documents and answers anyway, from memory. Branch on the empty case: rewrite the query and retry a bounded number of times (T9.1), fall back to another source (T9.2), or say the handbook does not cover the question.

The grader cleans the input. The next check looks at the output.

Hallucination detection here means a second grader that reads the kept chunks and the finished answer, and returns yes when the answer is "grounded in / supported by the set of facts". A no means the answer says something the chunks do not.

Three limits of one verdict

Three properties decide how far to trust it:

• It checks support, not usefulness. The question is not in the grader's input, so a grounded answer can still miss the question; Self-RAG (T9.2) adds a check for that.
• It returns one verdict for the whole answer. One unsupported sentence fails the answer, and the grader does not say which one. Per-claim checks split the answer into statements and verify each against a cited chunk, so they give the location as well as the verdict.
• It is a model judgement. The same 8B model writes and grades here, so the two can share blind spots. Before relying on it, label a sample of answers by hand and measure how often the grader agrees, for yes and no separately, because a lenient grader fails in one direction.

Planted failure: the remembered number

Back to the refund question. Suppose only refund-window survives grading. That chunk states the 30-day window for an annual plan's refund and says nothing about how long the refund takes. The table shows what the grader should return for three possible answers:

Table 2: Three answers to the refund question when only refund-window is kept (illustrative answers).

The first row is the usefulness gap: grounded, and still no answer. The second row is the remembered number, a fact the model brought from its training data. The notebook's generation prompt invites it, because it says "Answer the question based upon your knowledge". A prompt that says "answer only from the documents, and say so when they do not cover the question" produces the third row instead, and stops most remembered numbers before the grader has to catch them.

What a no leads to

The notebook prints the verdict and stops. In production a no has to go somewhere: regenerate once with the stricter prompt, then retry retrieval with a rewritten query, then answer that the handbook does not say and offer a person. Cap the retries, because each one is another generation call and another grading call on top of these:

Table 3: Model calls the three checks add to one answer, before any retry.

Those calls add latency as well as cost (the four chunk grades can run in parallel; the rest run one after another), so the graders belong on the same dashboard as the answer time. Once an answer passes, the last check shows the user which words it rests on.

Source highlighting asks the model which exact passages of the kept chunks the answer used, so the reader can check the answer against the handbook at a glance.

The notebook asks for the result as four parallel lists, one entry per highlighted passage:

Table 4: The highlight schema.

The prompt asks for segments that are "verbatim snippets from the documents, ensuring a word-for-word match", and tells the model to leave out documents the answer did not use. For Can I export my data after I cancel my plan?, a correct highlight returns two exact sentences:

Table 5: A correct highlight for the export question.

The parser checks shape, not truth

A structured-output parser checks that the response has the four lists with strings in them. It does not check that a segment appears in its chunk. A model can paraphrase, merge two sentences or quote a chunk it was never given, and the output still parses.

So check every segment in your own code with an exact substring search of the cited chunk. A hit returns the character offset where the quote starts, which is what the interface needs to draw the highlight. A miss returns nothing:

Table 6: An exact quote and a paraphrase, checked against the source text.

This checks quotation, not whether the sentence answers the customer. That judgement stays with the hallucination grader.

Planted failures in a highlight

Each way a highlight can go wrong needs its own response:

Table 7: Highlight failures that pass the parser.

Figure 2: Validating a highlight, one check per planted failure.

A rising miss rate in that log means the highlighting prompt or the model has drifted. And a chunk that is retrieved often and never highlighted is a candidate for the grader to drop or for retrieval to stop returning.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Grade each retrieved chunk before generation, at one call per chunk. Grading drops noise but cannot add a chunk retrieval missed (the candidate ceiling), so branch on an empty kept set instead of answering anyway.
• The hallucination grader checks support, not usefulness, and gives one verdict for the whole answer. Measure its agreement with hand labels, prompt the generator to answer only from the documents, and send a no to a capped retry or an abstention.
• A parser checks only the shape of a highlight. Find every segment in its cited chunk, check the ids and list lengths, and fall back to citing the whole chunk when a check fails.