Self-Correcting RAG: CRAG & Self-RAG: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur's support assistant runs CRAG with a web fallback. A customer asks what happens when their server goes down, and the assistant quotes another provider's availability guarantee. The handbook has the right answer. What went wrong, and what do you change?

What a strong answer covers:

• Reads the trace first: which chunks came back, how each was graded, and whether the web branch fired. In the lab, TF-IDF retrieves reboot, sizing and export for this question, the grader correctly rejects all three, and CRAG goes to the web
• Names the two faults: a retrieval miss on wording ("server goes down" against "outage" and "availability"), and a fallback that searches a source which cannot know a private handbook
• Fixes retrieval inside the corpus first: a rewrite-and-retry against the index (T9.1), hybrid or embedding search (T6.1, T5.4)
• Restricts the web branch to questions about public facts, or replaces it with a refusal that says the handbook does not cover the question

2. Product wants Self-RAG on every Larkspur question "to stop hallucinations". The assistant already carries most of Larkspur's support traffic. How do you decide?

What a strong answer covers:

• Counts calls before arguing: in the lab at k = 5 Self-RAG averages 15.0 LLM calls per question against 1 for the baseline, and the easy export question still costs 9
• Notes that 02a grades chunks one at a time in a loop, so calls add latency in series unless they are batched
• Proposes tiering: the baseline with a cheap check for most traffic, the full loop for high-stakes intents such as billing and refunds
• Measures on the golden set whether the extra calls change answers, and keeps the loop only where they do

3. CRAG has been live for a month. How do you tell whether the retrieval evaluator is helping or just adding cost?

What a strong answer covers:

• Logs the route for every request: chunks graded, chunks kept, relevant fraction, web fallback yes or no
• Tracks the web fallback rate by question type; a rate that jumps after an index change points at retrieval, not at the questions
• Samples cases where the grader kept everything and cases where it kept nothing, and grades them by hand to measure the grader's own precision and recall
• Watches for the threshold coupling with k: in the lab the export question falls back to the web at k = 4 and k = 5 with both evidence chunks kept

4. Someone asks the Self-RAG assistant "How much does the starter plan cost per month?" The handbook lists no prices. The groundedness check passed, yet the answer contains a price. How?

What a strong answer covers:

• Traces the loop: no chunk is relevant, so 02a rewrites and retrieves until the cap, then generates with no documents
• Points at grade_groundedness: with no documents it returns "yes" without calling the grader, so the check never ran
• Notes the only remaining guard is the generation prompt's instruction to say so when the context is missing; 02b drops even that and answers from the model's memory
• Fixes it by routing "no relevant documents after the cap" to a fixed refusal, and by adding out-of-corpus questions to the golden set

5. A cost dashboard shows one Self-RAG request that made about 200 LLM calls before it failed with a recursion error. The iteration cap is 2. Where do you look?

What a strong answer covers:

• Suspects a loop that does not move the counter, and reads the routers
• Finds it in 02a's route_after_groundedness, which writes state["iteration"] inside a routing function; LangGraph keeps only what nodes return, so the counter never moves and the regenerate loop runs until the recursion limit
• Moves the increment into a node (for example, generate returns the new iteration when it regenerates) and adds a test with a grader that always says "no"
• Sets an explicit recursion_limit and a per-request call budget as a second guard