Adaptive RAG: vector store vs web, with hallucination grading

What you'll learn

• Explain how a source router decides between the vector store and the web, and what it knows about the index.
• Trace the relevant-fraction rule that sends a weak retrieval to the web.
• Diagnose an inverted grade and an unbounded regeneration loop.
• Compare the LLM calls each path through the graph spends.

Lessons 1 and 2 of T9.3 chose a knowledge base or a person. This lesson chooses between two sources, the vector store and the web, and then adds the two checks from T9.1 and T9.2 behind that choice.

A Larkspur customer asks how to tune Postgres memory on a Larkspur server. The handbook covers server sizing and memory, so a fixed pipeline retrieves those chunks and answers from them. The chunks say how much memory each plan has, and nothing about Postgres settings. The answer sounds grounded and does not help.

A fixed pipeline does the same work for every question. It has no way to say that the evidence is weak, or to look somewhere else.

The graph in this lesson does four jobs a fixed pipeline cannot:

• Choose a source before retrieving.
• Check the evidence and fall back to the web when it is weak.
• Check the answer against the evidence and regenerate when it is unsupported.
• Stop, which the notebook's graph leaves out.

Adaptive-RAG (Jeong et al., NAACL 2024, arXiv:2403.14403) starts from the observation that a fixed pipeline does the same work for an easy question and a hard one. The paper routes by question complexity: no retrieval, one retrieval step or several, chosen by a small classifier trained for the job. The 03a notebook keeps the idea of deciding up front and routes by source instead.

A source router is a classifier whose labels are data sources. Here it is one LLM call with a closed schema whose only field allows vectordb or web_search, the same closed-label pattern as lesson 1 of T9.3.

What the router knows about the index

The router's system prompt lists what the vector store holds: chain-of-thought prompting, transformers and attention, the DINO vision model, diffusion models and parameter-efficient fine-tuning. It asks the model to choose vectordb when the question relates to any of them.

That list is the router's only knowledge of the index. Add documents on a new topic and the router keeps sending those questions to the web until someone edits the prompt. We call this the stale topic list. For Larkspur, the list would be the handbook's sections: pricing, refunds, exports, outages and server sizing.

Each route rewrites for its own engine

Each route rewrites the question before searching. The vector-store rewriter makes it concise for semantic search; the web rewriter makes it a search-engine query. The two paths are set up differently:

Table 1: The two retrieval paths in the 03a notebook.

The whole graph puts the router first and the two graders after retrieval:

Figure 1: The adaptive RAG graph. A failed document grade joins the web path; a failed answer check loops back to generation.

The router can be wrong in both directions, and the document grader is what catches the vector-store side of that.

The graph checks its work twice, once on the evidence and once on the answer. T9.1 and T9.2 teach each check on its own; here they sit behind a router.

The relevant-fraction rule

The document grader asks the LLM, once per retrieved chunk, whether the chunk is relevant, and gets a yes or no. The grading node keeps the chunks graded yes and computes the share that passed. If that share is 0.5 or less, the graph sets a web-search flag. Four cases show where the line falls:

Table 2: The relevant-fraction rule with a threshold of 0.5.

Exactly half counts as failure. On the web route, the search node appends the pages to the chunks that passed, so the answer can draw on both sources.

Planted failure: the inverted grade

The hallucination grader reads the answer and the documents and returns a score and a line of feedback. The score is inverted from what most graders use: yes means the answer may contain unsupported or contradicting content, and no means it is grounded.

Table 3: The hallucination grader's two outcomes.

We call this the inverted grade. Reuse this grader in a graph that expects yes to mean "pass" and every good answer loops while every bad one ships.

Planted failure: the unbounded regeneration loop

Nothing counts the retries. The regeneration edge has no limit, so a grader that keeps saying yes loops until LangGraph's recursion limit stops the graph. In the version this repo locks (langgraph 1.2.18) that limit is 10,007 steps by default. Each round costs two LLM calls, a generation and a check. We call this the unbounded regeneration loop.

The fix is a counter in the graph state and a terminal branch:

Figure 2: The bounded version. Stop after one or two retries and hand the result on.

The escalation in the last box is lesson 2 of T9.3's handoff, triggered by a rejected answer.

The notebook tests the loop by uncommenting a hard-coded wrong answer in the generation node. That line sits in the first-attempt branch only, so the retry, which takes the feedback branch, runs the real chain. The test exercises one rejection and one retry with feedback, never a second rejection.

Counting the LLM calls per path from the graph, before any regeneration:

Table 4: LLM calls per path, counted from the graph's nodes.

Each regeneration adds 2 more. A question the router sends to the vector store and the grader then rejects is the most expensive path. We call that the store-then-web detour. If many questions take it, the router's topic list is wrong or the index is missing content, and both are cheaper to fix than to grade around.

Interview tip: asked how to cut this graph's cost, start with the grades. Five per-chunk grading calls are the largest single item, and one call that grades all chunks together, or skipping grading when the top score is high, removes most of them.

Back to the customer tuning Postgres memory, followed through the graph with a Larkspur topic list in the router's prompt:

1. The source router reads "server" and "memory", which the handbook covers, and picks the vector store.
2. The question is rewritten for semantic search, and the retriever returns up to five chunks scoring 0.35 or more.
3. The grader passes half or fewer of them, so the graph sets the web flag and keeps only the chunks that passed.
4. The question is rewritten for a search engine and Tavily returns up to six pages, appended to the surviving chunks.
5. The model answers from the handbook chunks and the web pages together, and the hallucination grader checks the answer against both.
6. On no, the graph ends. On yes, it regenerates with the feedback, and with Figure 2's counter it stops after two retries.

This is the store-then-web detour, at up to 10 LLM calls. If Postgres questions are common, adding a database-tuning page to the handbook and its topic to the router's list is the cheaper fix.

Next lesson: 30.4 keeps one index and routes the question to a retrieval method instead of a source.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A source router picks the vector store or the web before retrieving. Its prompt's topic list is all it knows of the index, so update the prompt whenever you index a new topic.
• The document grader falls back to the web when half or fewer chunks pass. Frequent store-then-web detours mean the topic list or the index is wrong, which is cheaper to fix than to grade around.
• Bound the regeneration loop with a counter in the graph state, because each round costs two LLM calls and nothing else stops it before the recursion limit. A yes from this hallucination grader means unsupported.