End-to-End RAG Systems: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. A Larkspur customer forwards a support answer about annual-plan refunds. Its citation quotes "Annual plans are fully refundable for 30 days" and points to page 0 of the handbook. Neither the sentence nor a page 0 exists. What happened, and what do you change?

What a strong answer covers:

• Two separate faults: the quote is a paraphrase the model wrote, and the page is the PDF loader's 0-based number shown without adding 1
• Logs the request: the retrieved chunk ids, the answer, the raw QuotedCitations, to confirm the cited id was among the retrieved chunks
• Adds a code check before display: each quote, whitespace-normalised, must appear in the chunk with that id; failures are dropped or flagged, never shown as citations
• Tracks the quote-failure rate per day and alerts on a jump, which signals a model, prompt or chunking change

2. The team wants to cut latency by asking for the answer and its citations in one structured call instead of two. What do you gain and lose?

What a strong answer covers:

• Gains one round trip and the tokens of sending the context twice
• In one call the model writes claims and their support together, which can tie each citation to a claim; in two calls citations are a post-hoc search for support
• Either way the quote check in code stays, since neither design makes quotes verbatim
• Decides on the golden set: citation precision, unsupported-claim rate and latency for both designs, not on intuition

3. After the citation feature ships, how do you know whether answers are better supported than before, and how do you keep watching it?

What a strong answer covers:

• Offline: on the golden set, citation precision (cited chunk supports the claim), citation recall (claims with a citation) and the quote-match rate
• Online: share of answers with at least one verified citation, share of "I don't know" answers, and quote-failure rate
• Samples traffic for an LLM or human judge of whether each claim is supported by its quote
• Watches refusals on questions the handbook does answer, since a stricter prompt can trade hallucinations for missed answers

4. Product adds the server catalogue to the index, and the assistant tells a customer the average Larkspur server costs about $36 a month. The real average is $102.50. Why, and what is the fix?

What a strong answer covers:

• Retrieval read the top 4 of 10 rows, the ones whose descriptions resemble the question most; they average $36.25, and the $480 GPU server ranks last
• No k fixes it below the whole table; aggregation, comparisons, id lookups and rankings need every row or an exact filter
• Routes by question type: aggregates to SQL on a read-only connection, id lookups to a get by id, price limits to a metadata filter, descriptive questions to the vector store
• Adds these four question kinds to the golden set, with the right answer computed from the table, so a regression shows up as a wrong number

5. A bank evaluating Larkspur's assistant requires that no customer data leaves its network. The team says the system is local because it uses bge-base-en-v1.5 and FAISS. Is it, and what changes?

What a strong answer covers:

• Checks every component: embeddings and FAISS run locally, but HuggingFaceEndpoint sends each prompt, retrieved chunks included, to the Hugging Face Inference API
• Moves generation in-process with HuggingFacePipeline (or another self-hosted server inside the network) and removes the API token
• Sizes the hardware: memory for a 7B model, tokens per second on the target machine against the latency requirement
• Keeps the same embedding model for queries and documents, and loads saved FAISS indexes only from files the pipeline wrote, since load_local unpickles them