Retriever as a tool; grade then rewrite in LangGraph

What you'll learn

• Explain when the agent chooses a retriever tool.
• Trace retrieval, grading and rewriting through the same request.
• Compare routing inside an edge with a grading node that updates state.
• Choose label contracts, stopping rules and evidence checks.

A Larkspur customer asks How long does a refund take? The handbook's first hit, refund-window, says an annual plan can get a full refund within 30 days. That is an eligibility window. It does not say when the money arrives.

The one-shot RAG chain from T5.1 hands refund-window to the model, which answers the wrong question confidently. We call this answering from a near miss. This lesson joins that chain to the tool loop from T7.1: the agent chooses to search, and a grader routes the run to generation or to a rewrite.

Let the model choose the retriever tool

The retriever becomes a tool with a name, a description and an argument schema, the only things the model knows about it. The agent node binds it to the model, which decides whether to call it and with what query. A customer who writes hello gets a greeting with no search. The notebook searches a research paper.

Fixed transitions and runtime routing

A conditional edge is an edge whose destination is chosen at run time: after its source node runs, LangGraph calls a routing function on the state, and the value it returns names the next node. A fixed edge always goes the same way, as the edge from rewrite back to agent does. The graph has two conditional edges:

Table 1: The two routing decisions and how each return value becomes a node.

Put together, the edges form this graph:

Figure 1: The grade-then-rewrite graph. Two edges branch, and the rest are fixed.

Trace the refund question through a rewrite

The T9.1 lab supplies this trace: one failed grade, then one rewrite finds the timing evidence.

Figure 2: One failed grade and one rewrite. Chunks and reworded question come from the lab.

The first search misses refund-timing because TF-IDF matches exact words: "refund" in the question, "Refunds" in the chunk. The rewrite uses the chunk's own words and scores 0.720.

The rewrite returns to agent, not straight to retrieve. The model decides again whether and how to search, at one more model call per round.

Follow the request as a transaction

Added explanation. Here a transaction is one request's lifecycle, not a refund or a database transaction. Figure 3 separates what each step returns from the decision that follows:

Figure 3: Added transaction trace. The stricter answerability criterion drives the no and yes outcomes. Edge-local verdicts select routes without becoming state updates.

A document grader is one LLM call that reads retrieved text next to the question and says whether the text helps answer it. Structured output makes the reply match a declared schema, and the router compares a typed field instead of parsing free text. The prompt sets the bar. The schema decides what the router can trust.

Test answerability as well as relatedness

This lesson's grader passes a document that "contains keyword(s) or semantic meaning related to the user question". Lesson 3 of T9.1's also accepts "context that helps understand the topic". Both approve the refund-window near miss, because it is about refunds. A grader that asks "does this text answer the question?" fails it and sends the query to rewrite. "Related" maximises recall into the prompt. "Answers" protects what the model reads.

Validate the grader label

In this lesson's grader the score is a plain string, and the router compares it exactly with "yes". Three contracts give different outcomes:

Table 2: Planted labels under three contracts. The bold rows are relevant retrievals sent to rewrite.

We call the bold rows the capital-Y rewrite: the grader agreed, and the graph still discarded the evidence. Normalising fixes capitals and whitespace. A fixed set of values (a Literal in Pydantic) makes any other label fail validation instead of choosing a branch, as in lesson 3 of T9.1's topic check. That error still needs a handler.

Similarity scores cannot certify an answer

A top-k retriever returns k chunks for every query, whatever they score. The lab's TF-IDF scores over the T5.1 handbook chunks show that no score threshold can replace the grader:

Table 3: Retrieval scores from the T9.1 lab. A non-answer at 0.503 outscores a real answer at 0.470.

The discount question shares no word with the handbook and still retrieves two chunks. Its rewrite scores 0.503 without finding a discount. A threshold above 0.503 rejects the outage answer at 0.470, and one below it lets the non-answer through.

Start with the same three documents

Added illustrative example. Ask Can I get a refund after 30 days? over three invented snippets, not Larkspur policy:

Table 4: One shared illustrative input. Doc 3 supplies the requested fact.

Approach A: grade inside the conditional edge

The edge's grading function judges the batch and returns generate, because Doc 3 answers the question. The return value chooses a destination. It does not save the verdict or remove Doc 2, and generation receives all three chunks. In this lesson's graph the verdict is only a printed line. We call this the invisible verdict: tracing may capture the call, but shared state and a checkpointer never see the result.

Figure 4: Edge grading combines the evidence check and the route choice. Returning a route does not save a verdict or filter documents.

Approach B: grade in a dedicated node

The node returns an explicit state update: accepted Docs 1 and 3, rejected Doc 2, a verdict and reasons. A router reads that state and chooses generate. Generation sees only accepted evidence when you wire it to that field. Placement alone does not filter documents or improve judgement. With a checkpointer, the state is saved at each step boundary.

Figure 5: A separate grading node filters evidence and updates state before a small router chooses the next action. Generation must read the accepted-evidence field.

Separate placement from grading granularity

Either architecture can grade the batch or each chunk. Moving a call into a node adds a boundary, not an LLM call. The module's two graders differ in both:

Table 5: The two graders side by side.

Grading the block is cheap, but one relevant chunk carries the irrelevant ones into the prompt. Per-chunk grading costs k calls and filters what the model reads.

Choose an edge grader for a compact branch decision. Choose a grading node when verdict reuse, evidence filtering, inspection or a node retry policy earns the boundary. Fixed edges stay active beside dynamic routing, and a leftover unconditional edge can schedule a second destination.

Nothing on this lesson's rewrite path counts the trips. The loop ends only when the grader says yes, which never happens for a question the handbook cannot answer: the run continues until an execution limit or an error stops it. Lesson 3 of T9.1 adds a counter, persisted passing chunks, a should-generate flag and a topic check, at one graph step per retrieval.

Figure 6 adds a bound, checking for a pass before the budget:

Figure 6: Added node-based alternative. The fallback needs both a failed grade and an exhausted budget.

Diagnose failures by their effect on the request:

Table 6: Added diagnosis guide.

An API retry repeats a failed call, and a rewrite changes the search. Each rewrite adds an agent call, a retrieval and a grade: budget attempts, time and model use, and test on answerable, unanswerable and near-miss questions.

• Notebook corpus. The notebook runs on a research paper. The Larkspur paths and scores come from the T9.1 lab.
• TF-IDF scores. Table 3 uses exact-word TF-IDF. Embedding scores differ. Filters, thresholds and corpus size can reduce how many chunks a retriever returns.
• Planted labels. Table 2 tests the comparison, not how often a model writes Yes.
• Illustrations. The walkthrough and comparison are scripted. The three-document example is fictional.

1. LangGraph Graph API, accessed 13 November 2026.
2. Thinking in the Graph API, accessed 13 November 2026.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Let the agent choose the retriever tool, and route after retrieval on the grader's verdict. After a rewrite the model decides again whether and how to search.
• Grade whether the evidence answers the requested fact, and make the verdict a typed field with a fixed set of values. Similarity scores and topical overlap cannot certify an answer.
• An edge only chooses a route. A grading node can record verdicts and filtered evidence for generation to read. Choose batching separately from placement, and bound the rewrite loop with an evidence-gap exit.