RAG as an Agent Tool: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur adds a web_search tool to the support agent. A week later a customer asks "Can I get a refund on my annual plan?" and the agent quotes another provider's refund policy. How do you find out what happened?

What a strong answer covers:

• Opens the trace for that conversation and reads the model's tool_calls: which tool it called, with which query, and what came back
• If it called web_search: a routing failure. Compares the two descriptions for overlap ("refund", "plans", "pricing" in both, or in neither)
• If it called handbook_search and still answered wrongly: a retrieval or generation failure, debugged as in T5.1 by printing the returned chunks
• Rewrites the descriptions to state the boundary ("Larkspur's own policies" against "other providers"), adds the question to a routing golden set, and re-runs it

2. The Larkspur support bot today always retrieves before it answers. A teammate wants to replace it with an agent that calls handbook_search only when it decides to. What do you gain, what do you risk, and how do you decide?

What a strong answer covers:

• Gains: no retrieval for small talk or arithmetic, a query the model rewrites from the conversation, and a second call when the first chunks miss
• Risks: the model answers a policy question from memory without calling the tool, or loops on tool calls and adds latency
• Counts the cost per question from the graph: two model calls either way for a handbook question, more when the agent loops
• Decides on the golden set: answer quality, the share of handbook questions that triggered the tool, and p95 latency, against the always-retrieve baseline

3. The agent is live. How do you monitor tool selection in production, before customers notice a wrong answer?

What a strong answer covers:

• Logs every tool call with its name, the model-written query, the number of chunks returned and their top score
• Tracks per-tool call rates and the share of answers given with no tool call; a jump after a prompt, model or description change is the alert
• Samples conversations for an LLM judge or a reviewer to label the right tool, and grows the routing golden set from the misses
• Re-runs the routing golden set and a held-out set on every change to a description, a tool list or the model

4. A customer asks "Is a 4 GB server cheaper at Larkspur or at Hetzner?" What should the agent do, and how can it go wrong?

What a strong answer covers:

• The question needs both tools: handbook_search for Larkspur's price and web_search for the competitor's, then a comparison
• Failure modes: only one tool is called and the other price is filled in from memory, or a web result about Larkspur outranks the handbook
• Guards: descriptions that say which tool owns Larkspur's own facts, answers that cite each figure's source, and a refusal when one side is missing
• Adds multi-tool questions to the routing golden set, scored on whether both tools were called

5. The platform team proposes replacing the hand-built handbook retriever with a managed RAG platform. What do you evaluate before you agree?

What a strong answer covers:

• Runs the same golden set through both and compares answer quality, refusals on out-of-scope questions, and latency, as T5.1 does for any change
• Checks what the platform does with the handbook's tables and policy versions: its parser output, and whether reranker instructions can prefer the current policy
• Reviews data residency, access control on the datastore, per-query cost, and what leaving the platform would take
• Keeps the agent's tool interface fixed (handbook_search(query)), so the swap is one tool's implementation and the routing golden set still applies