Guardrails, query optimisation and a retry loop

What you'll learn

• Distinguish an off-topic question from one the handbook cannot answer, and the check that catches each.
• Trace an unanswerable question through a bounded loop and count its LLM calls.
• Compare two router check orders on the same question.
• Choose a rewrite bound from the golden set.

Lesson 1 of T9.1's graph rewrites when the grade fails, but it cannot stop. Two Larkspur questions show the cost. Do you offer a student discount? is about pricing, but the handbook has no discount policy. What's the weather like today? has nothing to do with Larkspur. In lesson 1 of T9.1's graph both rewrite again and again and never get a reply. Even with a counter, both would end in the same "I couldn't find that" message, which is wrong for the weather question.

The eight-node support agent

The notebook builds this agent for a fictional company's support documents. These pages run it on the Larkspur handbook. Before retrieving, it asks whether the question belongs to the assistant at all:

Figure 1: The support agent. Three nodes end the turn, and the two handlers reply with fixed text and make no LLM call.

A topic guardrail is a classification step before retrieval that decides whether a question is in scope. Here it is one structured-output call whose schema has a classification (RELEVANT or IRRELEVANT) and a confidence (HIGH, MEDIUM or LOW), both typed as fixed sets of values. Its prompt lists what is on topic (services, pricing, support procedures, company information) and what is not (general technology, other companies, weather and news). RELEVANT goes to retrieval. Anything else goes to the off-topic handler, which lists what the assistant can help with.

Off topic or unanswered

The check comes before retrieval because the grader cannot tell "off topic" from "not in the handbook": both look like a retrieval where nothing passes. The lab shows the difference with two rewrites allowed:

Table 1: Three golden questions, two rewrites allowed. The guardrail and the grader catch different failures.

The weather question stops for 1 call instead of 8, and the customer reads "I can help with pricing, refunds and servers" instead of "I couldn't find that in the handbook".

The check costs one call on every question. Over the lab's five golden questions it costs 3 calls at zero rewrites (14 with the check, 11 without), breaks even at one (24 and 24) and saves 3 at two (31 and 34). It pays in calls when the off-topic share of traffic times the cost of a full retry loop exceeds one call per question. It pays in reply quality at any bound.

The classification boundary

• It classifies the enhanced query. The enhancement node first folds the conversation history into a standalone question, and a follow-up such as What about annual plans? carries the refund question it follows.
• Confidence is computed and ignored. The router reads only the classification. Asking a clarifying question on LOW confidence needs a third branch.
• A false rejection is silent. An on-topic question classified IRRELEVANT never reaches retrieval or its logs. We call this the silent rejection. Track the off-topic rate and sample those replies.

Bounded retries means the graph counts its rewrites in the state and takes a different exit when the count reaches a limit. Lesson 1 of T9.1's graph has no counter. When the handbook has no answer, LangGraph's recursion limit (a cap on graph steps per run) is the only stop.

The runaway rewrite loop

In the pinned langgraph 1.2.18 the default recursion limit is 10,007 steps. Lesson 1 of T9.1's graph, run with stub nodes and a grader that always says no:

Table 2: Lesson 1 of T9.1's graph on a question nothing can answer, with stub nodes.

We call this the runaway rewrite loop. Each round costs three LLM calls (agent, grader, rewrite) and three graph steps. Calls track the step limit, and the run still ends in an error, not a reply.

A counter in the state

This lesson's graph bounds the loop in the state:

• The count: a rewrite counter that the enhancement node resets to 0 at the start of every turn. The bound is per question, not per conversation.
• The router: after grading, it picks one of three exits in a fixed order, shown in Figure 2.
• The guard: the rewrite node repeats the limit check and does nothing if it is reached, in case a later edge reaches it directly.

Figure 2: The router's check order. Evidence is checked before the budget, and a hit on the last retrieval is still answered.

Trace the student-discount question

With a bound of 2, a turn makes at most 3 retrievals. Follow the question that never finds evidence:

Check evidence before the budget

The order of the router's two checks changes the outcome on the last retrieval. The outage question, What happens if my server goes down?, finds sla only on its third retrieval, after both rewrites are used.

Evidence first (Figure 2). The router sees a passing chunk and answers. The question costs 10 calls: 1 check, 6 grader calls, 2 rewrites and 1 answer.

Budget first (Figure 3, an added counter-example). The router sees two rewrites used and replies "not found" before it looks at the grades. It spends 9 calls and discards the evidence it has just retrieved.

Figure 3: Added counter-example with the checks swapped. A hit on the last retrieval goes to the no-results reply.

The cost of each attempt

With k = 2 chunks per retrieval, a question costs one call for the topic check, two grader calls per retrieval, one per rewrite and one for the answer if there is one. The handlers and a first-turn enhancement make none. Answered on the first retrieval, a question costs 1 + 2 + 1 = 4. Never answered with a bound of m, it costs 1 + 2(m + 1) + m: 9 at a bound of 2. The interactive lab runs the loop on the five golden questions with real TF-IDF retrieval over the T5.1 handbook chunks and the bound as its slider:

Table 3: The golden set at each bound. The bold row is the notebook's bound.

Choosing the bound

Each step up the bound reaches one more question until 2, where the outage question is answered on its second rewrite. Going to 3 answers nothing new: the extra retrieval and three calls all go to the student-discount question, which no bound can answer. Unanswerable questions always pay the full bound. Set it where the answered count stops rising.

Pitfall: the counter as the only stop. Keep the recursion limit as a backstop under the counter. The counter bounds the loop you designed. The recursion limit bounds the one a later edit creates by accident.

The graph rewrites the query in two places, for different reasons:

Table 4: Enhancement repairs the conversation. Optimisation repairs the retrieval.

When a rewrite helps

A rewrite helps only if it moves toward the documents' words. The outage question shows both cases:

Table 5: The outage question's three retrievals. The rewrites are the lab's hand-written ones.

TF-IDF matches exact words, and "unavailable" is not in the handbook (only "availability" is). The first rewrite retrieves the same chunks at the same scores. We call this the synonym swap: new wording that never reaches the documents' vocabulary. The second rewrite uses the handbook's words, "outage" and "availability".

Drift across rewrites

Each optimisation reads the current query and writes back to the same field. The second rewrite rewrites the first, and two rounds can drift from what the customer asked. Lesson 1 of T9.1's rewrite node rewords the original question every time instead, at the cost of never building on a near miss. T6.2 covers rewriting before retrieval as a stage of its own, including HyDE and multi-query.

Added diagnosis guide. Each failure has a different signal in the logs:

Table 6: Five loop failures, the signal each leaves and its fix.

The no-results rate lists the handbook's gaps, and a jump after a change points at the index or the grader.

Key distinction: this graph knows when to stop, but it never checks its own answer. T9.2 adds a web-search fallback when grading fails and a check on the generated answer.

• Stand-ins in the lab. Retrieval is real TF-IDF, but the grader passes only the golden set's answer chunks, the topic check reads the golden label and the rewrites are hand-written. Real LLMs make mistakes these do not.
• Five questions. A larger golden set may keep gaining answers past a bound of 2.
• Stub nodes. Table 2 measures graph steps, not a model's behaviour.
• Illustrations. The walkthrough and the check-order comparison are scripted from the lab's numbers. Figure 3 is a counter-example, not the notebook's graph.

1. LangGraph Graph API, accessed 13 November 2026.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A topic check before retrieval costs one call per question and keeps off-topic questions out of the retry loop. Track the off-topic rate, because a silent rejection leaves no retrieval log.
• Bound rewrites with a per-question counter in the state, check evidence before the budget, and keep the recursion limit as a backstop. Unanswerable questions always pay the full bound: set it where the golden set's answered count stops rising.
• Enhancement makes a follow-up standalone, and optimisation looks for words that retrieve better. A rewrite helps only when it reaches the documents' vocabulary, and rewriting a rewrite can drift from the question.