Agentic RAG Foundations: Grading & Query Rewriting: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. A week after the Larkspur handbook moved to a new document system, the support assistant's LLM bill doubled while traffic stayed flat. Traces show most questions now making three retrievals. What happened, and what do you check first?

What a strong answer covers:

• Reads the path log: the share of questions that use rewrites and end in the no-results reply rose, so retrieval or grading changed, not traffic
• Re-runs the golden set and prints retrieved chunks for a failing question: if the right chunk is missing, the migration broke loading, chunking or the index; if it is present and graded no, the grader or its prompt changed
• Explains why cost doubled: with a bound of 2 and k = 2, every failed question pays 9 calls instead of 4
• Adds an alert on the no-results rate and rewrites per question, so the next regression shows up the same day

2. Your retriever moves to k = 8 for better recall. The grader currently reads all eight chunks as one block and answers yes or no. Do you switch to grading each chunk?

What a strong answer covers:

• One block grade lets a single relevant chunk carry seven irrelevant ones into the prompt; per-chunk grading filters them out
• Per-chunk grading costs k calls per retrieval, 8 instead of 1, and multiplies with every retry
• Mitigations: run the grades in parallel, put the grader on a small fast model, or ask for a list of per-chunk verdicts in one structured-output call
• Measures both on the golden set: answer faithfulness, tokens sent to the model, latency and cost per question

3. How do you know the grader is grading correctly in production?

What a strong answer covers:

• Labels a sample of (question, chunk) pairs by hand and measures the grader's precision and recall against them
• Names both errors: a false yes sends a non-answer to the model (refund-window for "how long does a refund take?"); a false no burns a rewrite and can end in "not found" for a question the handbook answers
• Logs the verdict and its reasoning field per chunk, since a grader on a routing edge leaves no trace in the state
• Re-checks after every change to the grader's model or prompt, and when a new document type enters the index

4. A customer asks about refunds, then sends "What about annual plans?" and gets the off-topic reply. What went wrong?

What a strong answer covers:

• The topic check saw the follow-up alone; it should classify the enhanced query, after the history is folded in
• Checks that the history was there: the conversation is checkpointed per thread_id, and a new thread starts empty
• Uses the confidence field the classifier already returns: route low confidence to a clarifying question instead of a refusal
• Adds follow-up questions to the golden set and tracks the off-topic rate, since a false rejection never reaches retrieval logs

5. A product manager asks to raise the retry bound from 2 to 5 "so more questions get answered". How do you respond?

What a strong answer covers:

• Plots answered questions against the bound on the golden set: in the lab, going from 2 to 3 answers nothing new and adds 3 calls
• Points out who pays: questions with no answer in the handbook always use the full bound, 1 + 2(m + 1) + m calls at k = 2
• Looks at the questions still unanswered: a rewrite cannot find content that is not indexed, so the fix is the handbook, or better rewriting toward its vocabulary
• Keeps LangGraph's recursion_limit as a backstop under the counter, since a later edit can add a loop the counter does not see