Adaptive RAG & Query Routing: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur's support assistant started answering "Apologies, I was not able to answer your question" to customers who ask how long their data is kept after they cancel. The retention chunk is in the general knowledge base. How do you find the fault?

What a strong answer covers:

• Pulls the logged category for those tickets first: a ticket routed to technical or billing never searches the general knowledge base, whatever the retriever does
• Checks that the label matches the stored metadata exactly (General against general), since a filter that matches nothing returns no chunks and triggers the apology
• Replays retrieval with the correct filter to separate a routing failure from a retrieval failure (score below the 0.2 threshold, chunk missing from the index)
• Adds the failing tickets to a labelled routing set, and adds a fallback that searches without the filter when the filtered search returns nothing

2. The team is splitting the handbook into billing, technical and general knowledge bases. One Chroma collection with a category filter, or one collection per department?

What a strong answer covers:

• One collection with a filter (the 01a design) keeps one embedding model and one index to run, and a fallback is the same search with the filter removed
• Separate collections let each department choose its own chunking, embedding model and re-indexing schedule, at the cost of a fan-out query for the fallback
• A misroute costs the same in both designs, because both trust the label
• Access control never rests on a classifier's label: if some documents are restricted, enforce it with collections or filters derived from the user's identity

3. The router has been live for a month. How do you know whether it is still routing well?

What a strong answer covers:

• Keeps a labelled routing set (the lab's eighteen tickets are a small version) and reports accuracy per category and a confusion matrix on every prompt or model change
• Logs the label, its score or confidence, the fallback rate and the escalation rate per day, and watches the label mix: a launch can create a kind of question no knowledge base covers
• Tracks downstream signals per route, such as empty retrievals and apology answers, since a silent misroute shows up there first
• Samples routed tickets for human review, weighted toward low-confidence and fallback cases

4. A ticket reads: "I was charged twice and my server has been down all morning. This is ridiculous." What should the router do?

What a strong answer covers:

• Sentiment runs first in determine_route, so a negative reading sends it to a person; the handoff should carry both topics, the contact details and anything already retrieved
• A single-label classifier must drop one of the two problems; a schema that allows several labels, or decomposition into sub-questions (T6.2), keeps both
• An outage in progress may deserve the on-call path, the way the healthcare notebook separates Distress from Negative
• Measures both error costs: a missed escalation loses a customer, and a needless one costs an agent's time

5. The adaptive RAG graph from lesson 3 of T9.3 is much slower and costlier than the plain pipeline it replaced. Where do the calls go, and what do you cut?

What a strong answer covers:

• Counts LLM calls per path from the graph: on the vector route, one for routing, one rewrite, one grade per retrieved chunk (up to five), one answer and one hallucination check, plus a rewrite and a web search when grading fails
• Grades all chunks in one call, or skips grading when the top retrieval score is high
• Puts a retry counter in the graph state: the regeneration loop has none, so only LangGraph's recursion limit stops it, and that limit is 10,007 steps by default in the version this repo locks (langgraph 1.2.18)
• Moves routing and grading to a smaller model, traces latency per node, and re-checks answer quality against the baseline after each cut