Corrective RAG in LangGraph with web-search fallback

What you'll learn

• Explain what a retrieval evaluator decides and the three verdicts the CRAG paper gives it.
• Trace one question through the CRAG graph, from retrieval to the fallback.
• Diagnose a fallback that fires with both evidence chunks in hand, and one that never fires with half the evidence missing.
• Choose where missing evidence may come from: the web, the index again, or a refusal.

Take the Larkspur question What happens if my server goes down? The T5.1 baseline's TF-IDF retrieval at k = 3 returns three chunks about reboots, server sizing and exports, and the model writes an answer from them. Nothing in that pipeline notices that none of the three is about outages. We call this blind generation: the generator answers from whatever retrieval returned, because nobody checked it first.

T9.1 graded each retrieved chunk and rewrote the question when none passed. Corrective RAG keeps that grader and adds a second exit: when too few chunks pass, it goes outside the index for evidence.

Corrective RAG (Yan et al., Corrective Retrieval Augmented Generation, 2024, arXiv:2401.15884) adds one component to the baseline. A retrieval evaluator is a check that judges the retrieved documents before the generator sees them, and its verdict picks the next step. The paper gives it three verdicts:

• Correct: retrieval looks right: generate from it.
• Incorrect: retrieval looks wrong: find other context.
• Ambiguous: the evaluator is unsure: use both.

The paper trains a small model (T5-large) as the evaluator. Both T9.2 implementations prompt an LLM instead, which costs one call per judgement.

The CRAG graph

The LangGraph implementation turns the evaluator into a node and a conditional edge. Here is the graph it builds:

Figure 1: The CRAG graph. Its nodes are retrieve, grade_documents, rewrite_query, web_search and generate_answer.

The grader keeps the chunks graded yes and drops the rest. When nothing is retrieved at all, the graph also takes the web branch. The graph runs once, with no loop back to retrieval.

Trace the export question through the graph

The T9.2 lab runs this graph on the handbook with TF-IDF retrieval and a stand-in grader that agrees with the golden labels. Follow Can I export my data after I cancel my plan? at k = 5:

Two evaluators side by side

The two implementations build the evaluator differently:

Table 1: The two CRAG implementations in this module.

The graph collapses the paper's three verdicts into two, because a yes-or-no grade has no middle. The function variant keeps all three, because a score has room for a band between the thresholds.

The rule is a fraction of the retrieved set. Hold the number of relevant chunks fixed and vary how many were retrieved:

Table 2: The majority rule on three retrieval sizes. Arithmetic only, no grader or search involved.

Every Larkspur golden question has two evidence chunks. A perfect grader keeps those two, and 2 of k is more than half only when k is 3 or less. The lab's grader agrees with the golden labels: each case below is a routing effect, not a grader mistake:

Table 3: Planted failures of the majority rule, from the T9.2 lab.

• Diluted majority: extra low-ranked chunks lower the fraction, so CRAG falls back with both evidence chunks in hand. The threshold moves with k.
• Thin pass: one relevant chunk out of one passes the rule, and CRAG answers without the export chunk that says how to export. The evaluator grades chunks, not whether the set is complete.
• Wrong fix: the grader rightly rejects all three outage chunks, and CRAG searches the web for a private handbook's outage policy. The answer was in the corpus all along, under the words "outage" and "availability".

Key distinction: the evaluator says whether the context is good enough. It says nothing about where better context lives, and the fallback is a separate design choice.

A correct verdict of "not good enough" still needs a fix that can reach the evidence. The options differ by where they look:

• Web search: suits a corpus of public facts that go stale, such as the research papers the CRAG graph indexes.
• Rewrite and retry inside the index: suits a miss caused by wording, like the outage question (T6.2 rewriting, T9.1 retry loop).
• Refusal: suits a question the corpus cannot answer, where any answer would be invented.

Figure 1 is the web design. Figure 2, an added alternative, keeps the evaluator and replaces the fallback with a bounded retry against the index that ends in a refusal:

Figure 2: Added alternative. The fallback stays inside the index and ends in a refusal.

For a private handbook, replace the web branch or guard it so it runs only for questions about public facts. Knowledge refinement, the other half of CRAG, works on the context the evaluator lets through.

Retrieval returns whole chunks, and a chunk can mix the sentence that answers the question with sentences that do not. Knowledge refinement cuts the context down to the parts that answer the question before generation. The CRAG paper does it with decompose-then-recompose:

Figure 3: Decompose-then-recompose, the paper's refinement step.

The function variant uses a simpler version. Its refinement step sends one document to the LLM with the prompt "Extract the key information from the following document in bullet points" and returns the bullets. Where it runs depends on the verdict:

Table 4: Where the function variant refines its context.

The verdicts form a short chain of decisions on the best score:

Figure 4: The function variant's three verdicts as a chain of thresholds.

The LangGraph graph has no refinement step. It filters at chunk level, keeping only chunks graded yes, and passes web pages as full text. A fallback can put six whole pages into the prompt. We call that context flood: the evaluator removes bad chunks from the index, then the fallback adds pages nobody graded.

Strips on the export question

On the export question at k = 3, the retriever returns retention, export and cancel. Chunk-level filtering keeps the first two whole. Strip-level refinement would go one step further, as this hand-written illustration shows:

Table 5: Strip-level refinement on the export question (illustrative, written by hand, not a measured output).

Refinement has two costs. It adds one LLM call per document it rewrites, and an LLM that rewrites evidence can also change it. We call the second one rewritten evidence: a bullet that reads well and no longer says what the chunk said. Keep the original chunk id beside each bullet so the answer can still cite its source, and so a reviewer can compare the bullet with the text.

Two questions about your corpus settle the design. Retune the fraction whenever k changes:

Table 6: Design choices for a CRAG pipeline.

Log every route with the chunks graded, the chunks kept and the branch taken.

Interview tip: when a CRAG answer is wrong, read the route before the prompt. Which chunks came back, how each was graded and whether the web branch fired usually explain the answer.

• Stand-in grader. The lab's retrieval is real TF-IDF over the T5.1 handbook chunks, but its grader agrees with the golden labels. A real grader makes mistakes these cases do not.
• Different corpus. The notebook's graph indexes research papers, where a web fallback can help. The Larkspur cases test the same routing on a private corpus.
• Illustrations. The strip table is written by hand, and the retry design in Figure 2 is an addition, not the notebook's graph.

Lesson 2 of T9.2 adds the check CRAG lacks: one that runs after generation and looks at the answer itself.

1. Yan et al., Corrective Retrieval Augmented Generation, 2024.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• CRAG grades the retrieved chunks before generation and falls back to a rewritten web search when too few pass. It grades chunks, not answers, so a passing set can still miss the evidence, and a fraction-based rule moves with k.
• A web fallback suits public facts that go stale. For a private handbook, replace it with a retry inside the index and then a refusal, because the web cannot see your documents.
• Knowledge refinement cuts chunks down to the parts that answer the question before generation. An LLM that rewrites evidence can also change it, so carry the source chunk id with every bullet.