Why RAG & the RAG Lifecycle: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur's support assistant answers billing questions from the handbook. The policy team changes the refund window on Monday. A teammate proposes fine-tuning the model on the new handbook. What do you recommend, and what has to happen on Monday?

What a strong answer covers:

• Facts that change belong in the index, not the weights; fine-tuning would need repeating at every policy change and gives no citations
• Monday's work is re-indexing the changed document: load, split, embed, upsert, and remove the stale chunks
• Uses date or version metadata so retrieval can prefer the current policy and the old one can be filtered out
• Adds the new policy question to the golden set and checks that the assistant now cites the new chunk

2. A customer asks "Can I get a refund after 45 days?" and the assistant says "No, refunds are only available within 30 days." The handbook says refunds after day 30 are prorated. How do you find out what went wrong?

What a strong answer covers:

• Prints the retrieved chunks with scores before touching the prompt
• If the proration chunk was not retrieved: retrieval failure. Check its rank, whether it was split from its context, and whether "45 days" matches nothing lexically
• If it was retrieved but ignored: generation failure. Check its position in the prompt, conflicting chunks, and the instruction to answer only from context
• Turns the case into a golden-set question so the fix is measured and stays fixed

3. Your team runs at k = 20 "to be safe". Latency is 3.1 s, and the product requirement is 1.5 s. How do you choose a new k without hurting quality?

What a strong answer covers:

• Measures recall@k on the golden set for k = 1 to 20 and finds where the curve flattens
• Separates the decision: retrieve wide (20 to 50), rerank, then send a small k to the model, if a reranker fits in the latency budget
• Reports answer quality, not only recall, because fewer distractors often improves faithfulness
• Watches per-question regressions, not just the mean; one lost question type can matter more than the average

4. The baseline went live last month. How do you know whether retrieval quality is degrading in production before users complain?

What a strong answer covers:

• Logs every query with retrieved chunk ids, scores and the final answer
• Tracks retrieval signals: top-1 score distribution, the share of queries below a score floor, out-of-scope fallback rate
• Samples traffic for offline grading with an LLM judge (faithfulness, relevance) and re-runs the golden set on every index or model change
• Alerts on drift after content updates, such as a new document type that the loader parses badly

5. Someone suggests skipping RAG because the new model has a 1M-token context and the whole handbook fits. When is that right, and when does it break?

What a strong answer covers:

• Right for a small, stable corpus with low query volume, or for one-off analysis
• Cost and latency scale with corpus size on every call, while RAG pays for a few chunks
• Accuracy can fall for facts mid-context, and the model has no ranked evidence to cite
• Access control is coarse: every user's prompt holds every document, unless you filter per user, which is retrieval again