Self-RAG: retrieve-or-not, relevance, groundedness, utility

What you'll learn

• Explain the four reflection decisions of Self-RAG and the question each one answers.
• Trace a question the handbook cannot answer through every loop, and count its LLM calls.
• Diagnose a groundedness check that passes without running, and a retry cap that never bites.
• Choose where a loop counter must be updated in a LangGraph graph.

Lesson 1 of T9.2 checked the context before generation and never looked at the answer. Self-RAG checks both, and it can also skip retrieval when a question does not need it.

A Larkspur answer can cite real policy and still miss the question. As an illustrative case, ask Can I export my data after I cancel my plan?, retrieve only the export chunk, and the model can write a well-supported paragraph on how to export that never mentions the 30-day retention window. Every sentence is grounded in the context, and the customer still loses their data. Relevance, support and usefulness are three separate properties, and a pipeline that checks one of them misses the other two.

Self-RAG (Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, 2023, arXiv:2310.11511) trains one model to emit special tokens that judge its own pipeline. Reflection tokens are labels the model produces about its own retrieval and output: whether to retrieve, whether a passage is relevant, whether the answer is supported, and whether it is useful. The table lists the four kinds and how each implementation in this module represents them:

Table 1: The four reflection tokens in the paper and in the two implementations.

In the paper the tokens are part of the model's vocabulary and come out of the same decoding pass as the answer. Neither implementation trains a model. Each judgement is a separate LLM call with a structured-output schema, so "reflection token" here means one more call that returns a label.

Key distinction: in the paper a reflection costs a few extra tokens. In these implementations it costs a full model round trip, and that changes which checks are worth running.

How the graph wires the four checks

The LangGraph implementation wires each judgement to its own conditional edge:

Figure 1: The Self-RAG graph. Two loops go back through a rewrite, one goes straight back to generation.

The three loops have different exits:

• No relevant chunk: rewrite the question and retrieve again.
• Answer not useful: rewrite the question and retrieve again, with the utility feedback as a hint.
• Answer not grounded: generate again, with the groundedness feedback in the prompt.

max_iterations = 2 bounds the rewrite loops, because the rewrite node increments the counter each time it runs. After two rewrites the relevance router sends the question to generation even with no relevant chunk, and the utility router ends the run.

The first token is the one CRAG lacks. What is 9 plus 10? gets need_retrieval = no in the graph and skips the vector store. Every Larkspur question is about the handbook. On this corpus the Retrieve token saves nothing and costs one call.

Trace a question the handbook cannot answer

The loops make the cost depend on the question. The T9.2 lab replays the graph on the handbook with stand-in graders. On How much does the starter plan cost per month? at k = 5 no chunk is ever relevant:

On the export question the same graph stops after one pass in 9 calls. The expensive question is the one the handbook cannot answer, and the run ends with a generation that has nothing to be grounded in. The groundedness check is where that generation should have been caught.

A groundedness check asks whether every claim in the answer is supported by the context the answer was generated from. It is Self-RAG's hallucination guard. In the graph, the groundedness node sends the context and the answer to a "strict fact-checker" prompt. The schema returns a yes only if every claim is supported, and a feedback string naming the unsupported claims.

On a no, the router sends the run back to generation, and the feedback goes into the prompt as "a previous attempt at this answer was rejected for this reason". The regenerate loop is shown here as a sequence of calls:

Figure 2: One regeneration after an unsupported answer.

Groundedness and utility are separate checks because they fail separately. The export answer in the opening example is fully grounded and still not useful. An answer that quotes the retention window correctly and adds an invented export fee (again illustrative) sounds useful and is not grounded.

Planted failures in the hallucination guard

Two details in the graph decide what the check actually catches. Planting one failing case against each shows them:

Table 2: Three planted failures of the groundedness check.

The first row is the empty-context pass. On the starter-plan question, Self-RAG exhausts its rewrites, generates with no context, and the check waves the answer through. The only guard left is the generation prompt's line "If the context does not contain enough information to answer confidently, say so plainly". The function variant is weaker still: its own low-relevance test asks a climate-change corpus how did harry beat quirrell?, and nothing stops the model from answering.

Pitfall: the empty-context pass. A check that returns yes when it has nothing to check guards only the cases where it was least needed. Route "no documents after the cap" to a fixed refusal instead.

The counter that never moves

The second row is the stuck counter. The groundedness router writes iteration + 1 into the state inside the routing function. In LangGraph a conditional-edge router only returns the name of the next node. Only what nodes return is merged into the state. A write inside a router is discarded. The rewrite loops stay bounded because the rewrite node returns the new iteration. The regenerate loop has no node that does:

Figure 3: The notebook's regenerate loop. The router's write is discarded: the counter stays where it was and the yes branch repeats.

The cap still bites if earlier rewrites already pushed the counter to 2, because the router reads that value. On a question that passes relevance on the first pass, the counter starts at 0 and stays there. The fix is to count in a node: have generation return the new iteration when it is regenerating, or add a small regenerate node before it.

Figure 4: Added fix. The node returns the count, and LangGraph merges it into the state.

We replayed the notebook's routers with stub nodes in langgraph 1.2.18 on the export question at k = 5, with a groundedness grader that always says no and recursion_limit = 200:

A test with a grader that always says no proves the fix, and a per-request call budget catches the next loop nobody bounded.

Keeping three support levels

The paper's three support levels are worth keeping when you build this yourself. The function variant generates one candidate answer per relevant passage and ranks them by "fully supported" first and utility second. A yes-or-no grade cannot rank candidates that way, because a partially supported answer and an unsupported one both come back as no.

On Larkspur's questions the cost and the gaps line up: 9 calls for the export question, 20 and an empty-context pass for the starter-plan question, and a stuck counter on any first-pass question when a grader outage returns no on every call. For Larkspur, the out-of-corpus route ends in a refusal and a handoff to support, the regenerate counter moves in a node, and every request carries a call budget. Lesson 3 of T9.2 puts these call counts next to CRAG's and the baseline's.

• Stand-in graders. The lab's relevance grader agrees with the golden labels and its rewrite repeats the question. Its counts are a ceiling for questions a better rewrite could reach.
• Stub nodes. The counter replay measures graph steps and calls, not a model's judgements.
• Illustrations. The opening export answer and the invented export fee are illustrative. Figure 4 is an added fix, not the notebook's graph.

1. Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, 2023.
2. LangGraph Graph API, accessed 13 November 2026.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Self-RAG judges four things: whether to retrieve, whether each chunk is relevant, whether the answer is supported and whether it is useful. In these implementations each judgement is one more LLM call that returns a label.
• The loops make cost depend on the question, from 9 calls on the export question to 20 on an out-of-corpus price question at k = 5. With no documents, the groundedness check passes without running.
• Groundedness and utility fail separately, so check both, and make "no documents" a refusal rather than a pass.
• Update loop counters in a node, because LangGraph discards state written inside a routing function, and back every loop with a per-request call budget.