Conversational RAG with history-aware retrieval

What you'll learn

• Explain why a follow-up retrieves the wrong chunks even when the model holds the whole history.
• Trace how a history-aware retriever turns a follow-up into a standalone question, and when it skips the rewrite.
• Compare three retrieval inputs on turns 2 and 3: the latest message, appended history and a rewrite.
• Diagnose the misses a rewrite cannot fix.

Lessons 1 to 3 of T10.1 gave the history to the model. In retrieval-augmented generation (RAG) a second component reads the question: the retriever. It reads only the literal text.

A Larkspur customer asks How do I export my data?, then Is that possible after I cancel?, and gets good answers to both. Then they ask How long do I have? The one useful term left after stop words is "long", and no chunk of the Larkspur handbook contains it: every chunk scores zero. The model receives the history and knows the customer means their data after cancelling. The chunks it receives were chosen for a question with no subject.

We will call this subject loss. The notebook shows the same failure on Lilian Weng's post about agents: after What is Self-Reflection?, the follow-up How do we achieve it? retrieves chunks about "it". The fix has to happen before the search.

A history-aware retriever adds one step before retrieval: an LLM reads the chat history and the latest message and writes a standalone question, one that makes sense without the history, and the retriever searches for that. The notebook's prompt tells the LLM to reformulate the question if needed, return it unchanged otherwise, and not answer it.

Figure 1: The history-aware retriever's branch. Turn 1 has no history and costs no rewrite call.

The rewrite turns each follow-up back into a question the retriever can score. Table 1 shows what it does to Larkspur follow-ups from the lab's chats:

Table 1: Three follow-ups from the lab and their rewrites. The rewrites are written by hand to show the shape; the lab's rankings for them are computed.

create_history_aware_retriever builds exactly the branch in Figure 1. create_retrieval_chain then joins it to a chain that stuffs the retrieved documents into an answer prompt, and that prompt carries the chat history too. The answer step sees the conversation as well. Wrapped in RunnableWithMessageHistory with the answer field named as the output to save, the chain loads and saves the history itself, as in lesson 2 of T10.1.

What the rewrite costs, and how it fails

The extra step brings its own failure modes:

Table 2: Four ways a history-aware retriever goes wrong. The last one looks like a retrieval regression and is a memory bug.

The lab measures how much the rewrite buys against the two cheaper inputs.

The interactive lab runs six three-turn Larkspur support chats against the twelve handbook chunks from the T5.1 lab, with the same TF-IDF index (it matches scikit-learn's TfidfVectorizer to within 5e-16 on all 648 scores). It compares three things the retriever can receive on turns 2 and 3: the latest message as typed, the earlier questions with the latest appended, and a standalone rewrite. The customer messages are hand-written and the rewrites are representative; the rankings are computed.

Table 3: Evidence found on the 12 follow-up turns, recomputed from the lab's own scoring code. A hit means the labelled evidence chunk is in the top k.

• The latest message misses the turns that point back. What do I get if you miss it?, How long do I have? and Can I get money back on it? score zero against every chunk, and Is that possible after I cancel? finds the cancel chunk instead of the export evidence. Follow-ups that name their own subject, such as Do my promo credits expire?, hit either way.
• Appending fixes turn 2 and breaks turn 3. Turn 2 usually continues turn 1's topic, and the extra words help. Turn 3 often changes topic, and two earlier questions outweigh one new one: at k = 1 the old topic takes the only slot in 5 of 6 chats. Call this topic drag. At k = 2 appending catches up, by spending a context slot on the old topic every time.
• The rewrite wins at k = 1 and pays for it with 12 LLM calls, one per follow-up. Its two misses are worth reading. How long until the money is back? misses whatever the query says, because the evidence chunk says "Refunds" and "business days" and no phrasing of the question uses those words. A rewrite restores the subject but not the handbook's vocabulary, which is T5.1's representation problem and the reason for embeddings in T5.4. And What happens if I cancel my annual plan instead of getting a refund? is the over-carried rewrite from Table 2: "refund" and "annual plan" pull the refund chunk above cancel at k = 1.

Measuring turns 2 and 3 separately

The lab is the module's lab brief in miniature. On your own corpus, write multi-turn conversations, label the evidence for each turn, and report hit rate on turns 2 and 3 separately from turn 1, with and without the rewrite. A single average over all turns hides the follow-ups, because first questions are usually standalone and score well either way.

The same split catches a regression early. If a refactor stops passing the history, the turn-1 numbers stay put while the turn-2 and turn-3 numbers fall to the latest-message row of Table 3.

Following the export conversation through the full chain:

1. Turn 1. How do I export my data? arrives with an empty history, the branch skips the rewrite, and the retriever finds the export chunk.
2. Turn 2. Is that possible after I cancel? arrives with one exchange of history. As typed it finds the cancel chunk; rewritten as Can I export my data after I cancel my plan? it finds the export evidence.
3. Turn 3. How long do I have? scores zero as typed. The rewrite, How long is my data kept after I cancel my plan?, finds the retention chunk: data is kept for 30 days after cancelling.
4. Answer. The answer prompt receives the retrieved chunk and the history. The reply can then say your data and mean the account from turn 1.
5. Monitoring. The logs keep the raw message, the rewrite and the retrieved chunk ids per turn, which lets the team check the rewrite as well as the retrieval.

What changes next: these chains keep history with the deprecated wrapper from lesson 2 of T10.1. Lesson 6 of T10.1 moves conversation state into a LangGraph thread, where the same bounding choices become nodes.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A retriever reads only the literal question, so a follow-up such as How long do I have? can score zero against every chunk even when the model holds the history.
• A history-aware retriever has an LLM rewrite each follow-up as a standalone question and skips the rewrite on turn 1. Each rewrite adds a call and can drop a constraint or carry an old topic forward.
• In the lab, rewriting raised the k = 1 hit rate from 0.50 to 0.83, while appending earlier questions helped turn 2 and broke turn 3. Report follow-up turns separately from turn 1, which is usually standalone.