MemoRAG: global memory and clue-guided retrieval

What you'll learn

• Explain what corpus memory is and how its clues reach the retriever.
• Trace one question through memory search, clue writing, retrieval and the answer.
• Count the extra searches and model calls that clues add per question.
• Diagnose a fluent clue that points at a product or policy the corpus does not support.

Lesson 1 of T13.1 added summaries to the index, which gave a theme question something to match. MemoRAG leaves the index alone and changes the query instead.

A Larkspur operations lead asks How does Larkspur protect continuity? The question names no section. Searched directly, it matches passages that share its words and misses the ones that answer it: export, retention, availability guarantees and incident reports. A reader who knew the whole handbook would turn the question into better searches first. MemoRAG gives the retriever that reader.

Global memory is a compact, model-readable representation of the whole corpus that turns a vague question into specific search clues before retrieval runs. For the continuity question, memory might suggest export before cancellation, 30-day retention window and availability guarantee and credits. Each clue is a search, not an answer: the final answer still needs source passages that support every claim.

The MemoRAG paper (Qian et al., 2024) builds the memory with a long-range model that compresses the corpus, then has that model draft clues for the retriever. A clue can contain a wrong detail. Its job is to locate evidence, and the generator checks what comes back:

Figure 1: Clue-guided retrieval. Memory shapes the searches; only retrieved passages reach the answer.

The optional notebook implements a smaller approximation. Its memory is a list of readable text records, each a topic and its details, written by a chat model reading the corpus and validated against a schema. The records are embedded and stored in a FAISS index. The memorising prompt says it mimics KV-cache creation; the records are text, not the attention key and value tensors a model keeps inside, and the analogy stops there.

How the notebook reads the corpus sets how much memory each call sees:

Table 1: Memory input and passage index in the notebook. Memory sees large spans; retrieval still returns small passages.

At question time the notebook runs one fixed sequence:

Figure 2: One question through the notebook's pipeline. Clues come from two separate model calls.

The clue step makes two calls: one asks for text spans likely to appear in the corpus, the other for surrogate questions, narrower questions that clarify the original. The notebook splits both replies into lines, drops any line of three words or fewer, and appends the original question. Each surviving line becomes one search for three passages.

That design has a cost that grows with the clue count. The arithmetic for one question:

Table 2: Retrieval work per question (illustrative clue counts). One memory search is added to every row; the three model calls are the two clue calls and the answer.

Removing duplicate passages is the only control on context size. Deduplication drops repeats; it does not cap tokens. Eight clues that each find three new passages put 27 passages in front of the generator. Call that clue sprawl, and set a passage or token budget after deduplication.

Each stage has its own failure, and its own thing to log:

Table 3: Where clue-guided retrieval fails, stage by stage.

The invented-product row is the failure that matters most. Call it the fluent clue: a clue that reads well and names something the corpus does not contain. If retrieval then finds a loosely related passage, the generator may treat the clue's claim as supported. The notebook's answer prompt tells the model to use only the retrieved context and to say when the context is not enough. Logging clues beside the passages they found is what shows whether that instruction held.

The notebook's update path has gaps worth planting:

Table 4: Planted failures in the notebook's memory store.

A topic registry or stable record identifiers fixes the first row. The third needs a fallback that searches with the original question alone.

Memory is derived data, with the same duties as the summaries in lesson 1 of T13.1. When Larkspur changes its retention rules, update the passage index and regenerate every memory record written from those passages. Shared memory must not leak restricted documents through its records or its clues. Test a deletion against the stored memory and against generated answers.

For Larkspur, a memory-assisted retriever would run like this:

1. Memorise the handbook into topic records, each tagged with the passages and versions it came from.
2. For How does Larkspur protect continuity?, retrieve the closest memory records and write clues from them.
3. Search the passage index with each clue and with the original question.
4. Deduplicate, cut to a fixed passage budget, and answer only from what remains.
5. Log the clues, the passages each one found, and the claims each passage supports.

Run the same questions through plain retrieval at the same budget. Report a gain only when the clue-guided run finds supporting passages the plain run missed.

The notebook loads two prebuilt memory stores, one per corpus, and the second corpus is a book PDF. None of those files ships with this repository, and the memorising cells are commented out because they are slow and paid. As shipped, the notebook shows the implementation's shape and does not run end to end.

Its evaluation compares plain RAG with this MemoRAG approximation on the first 5 of 44 climate questions and on 15 of 16 questions about the book. It also scores answers saved from a run of the original MemoRAG on a GPU. Those scores come from a model judge and a different corpus, and they say nothing about the Larkspur handbook. Report a comparison only after running both pipelines on the same questions and budget.

Lesson 3 of T13.1 builds a third derived representation, a graph of entities and relationships, and searches it at two levels.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Corpus memory writes clues that help the retriever find evidence. A clue can contain a wrong detail, so every claim in the answer must rest on a retrieved passage.
• Each extra clue adds a search and its passages, and deduplication alone does not bound the context. Log clue text beside the original question and set a budget after deduplication.
• Treat memory as derived data. Regenerate records when their source passages change, and check that records and clues never expose restricted documents.