Answers with sources and per-claim citations

What you'll learn

• Explain how to keep the retrieved chunks in a chain's output with one retrieval.
• Design a citation schema whose quotes a machine can check.
• Diagnose three things the schema does not guarantee: support, per-claim links and verbatim quotes.
• Verify citations in code, and track the failure rate.

Lesson 1 of T6.4's chain answered from five chunks and then threw them away. A Larkspur agent reading Yes, annual plans are refundable within 30 days has no way to see which handbook passage said so, or whether one did.

Both lesson 2 of T6.4 notebooks first run the T5.6 contextual retrieval notebook (lesson 5 of T5.6). That run builds the index the citations depend on, and writes four metadata fields on every chunk:

Table 1: The metadata every chunk carries.

Shown as stored, a 0-based page tells a reader "page 0", or sends them to the page before the right one. Add 1 when you display it. We call this the off-by-one page.

Notebook 02a keeps the retrieved documents in the chain's output instead of formatting them away. The retriever's result and the question go into a dictionary, and a pass-through step runs the answer chain on that dictionary and adds the answer beside them. The output holds three things: the five context documents, the question, and the response. The retriever runs once, so the chunks the user sees are the chunks the model read.

This is source attribution at the answer level: the reader gets a list of five sources and has to find the supporting sentence in each.

Notebook 02b asks for more: which chunk supports the answer, and the exact sentences. A citation schema is a structured output the model fills, one record per supporting chunk:

Table 2: The fields of 02b's citation record. The schema returns a list of these.

The citations come from a second model call, after the answer:

Figure 1: Answer first, then citations, then a check in code.

The answer chain prints each chunk's id, source, title and page above its text, so the model can copy real ids into its citations. On the Larkspur handbook, Can I get a refund on my annual plan? should produce one citation (Illustrative: a representative output, not a recorded run):

Table 3: A citation for the refund question (illustrative).

Three gaps remain, each with a name:

• Citation after the fact. The second call looks for support for text that already exists. It shows the answer can be justified from the chunks, not that the model used them. A claim with no support may get a loose citation or none, and nothing marks which.
• The flat list. The schema is a list per answer. Nothing ties a citation to the sentence of the answer it supports. For true per-claim citations, add a claim field (the answer sentence) or have the answer call emit numbered markers that the citations reference.
• The paraphrased quote. The schema asks for exact sentences; the model can still paraphrase or merge two. 02b's display code highlights each quoted sentence in its chunk with a pattern match. A paraphrase highlights nothing and raises nothing, and a citation whose id matches no retrieved chunk comes back with no context. Both fail silently.

So check every citation in code before showing it: look the chunk up by id, normalise whitespace in both texts, and test whether the quote is a substring of the chunk. Here are four planted citations against the refund chunk:

Table 4: Planted citations against the quote check.

Drop or flag the citations that fail, and track the failure rate as a production metric. A rising rate means the prompt, the model or the chunk text changed.

Pitfall: a citation prompt that sees a Python object. 02b's citation chain passes the list of retrieved documents straight into its prompt, so the model reads the documents' programming representation, metadata and all, rather than the formatted text the answer chain saw. Format the chunks the same way for both calls.

1. Keep the context in the output, one retrieval, as 02a does.
2. Ask for the five-field schema, plus a claim field so each citation names the answer sentence it supports.
3. Check every quote in code; show only citations that pass, and display pages from 1.
4. Alert on the failure rate, per model and prompt version.

Citations show which chunk supports an answer. Lesson 3 of T6.4 asks a related question about retrieval itself: why a chunk was retrieved at all.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Keep the retrieved documents in the chain's output, so the retriever runs once and the sources shown are the chunks the model read; display PDF pages from 1, not 0.
• A citation schema with id, page and verbatim quotes makes citations checkable. It is still a flat list per answer, written after the fact, so add a claim field for per-claim citations.
• The model can paraphrase a quote or cite an id that was never retrieved. Verify every quote against its chunk in code, drop failures, and track the failure rate.