Reliability, Feedback Loops & Diversity: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. A Larkspur customer asks how long a refund takes. The assistant answers with a number of business days that differs from the handbook, and the hallucination grader marked the answer grounded. How do you work out what failed?

What a strong answer covers:

• Pulls the trace: the retrieved chunks, the grader's verdict on each, the kept set, the answer and the hallucination grader's input
• Checks retrieval first: on the T5.1 TF-IDF index refund-timing scores 0 for this question, so the kept set may hold only refund-window, which gives no duration at all
• Checks the generation prompt: the lesson's example tells the model to answer "based upon your knowledge", which invites a remembered number; a grounded system says "only from the documents"
• Treats the grader as a model with an error rate: labels a sample of answers by hand, measures how often the grader agrees, and adds this case to that set

2. Product wants a thumbs-up and thumbs-down on every answer, feeding a re-ranker like the one in lesson 2 of T9.5. What do you tell them it will and will not fix?

What a strong answer covers:

• It reorders the candidates retrieval returns, so it fixes questions whose evidence is retrieved but ranked low (the lab's credit question goes from recall@2 0.00 to 1.00 after one thumbs-down)
• It cannot reach a chunk retrieval never returns: in the lab 40 of the 41 wrong answers in 100 events go to the refund and outage questions, and no rating changes that
• A rating on the answer credits every chunk in it, so unrelated chunks ride along on good answers; per-source signals or a reranker trained on graded pairs are sharper
• Ships it with a holdout: re-runs the golden set before and after, and watches per-question recall so one popular question cannot hide a regression elsewhere

3. The relevance grader and the hallucination grader have been live for a month. How do you know they are working, and what do you alert on?

What a strong answer covers:

• Logs every verdict with the question, chunk ids and answer, so rates can be tracked per question type and per document
• Watches the share of questions where every chunk is graded irrelevant, and the share of answers graded not grounded; a jump after an index or prompt change points at the change
• Samples graded traffic for human labels each week and tracks the graders' agreement with them, separately for "yes" and "no", since a lenient grader fails in one direction
• Counts the cost: five grader calls per answer (one per chunk at k = 4, plus one grounding call) and a sixth for highlighting, and the latency they add belongs on the same dashboard

4. A weekly job has been folding highly rated question and answer pairs into the index. Larkspur then changes its refund rule. What goes wrong, and how do you guard against it?

What a strong answer covers:

• Old answers that quote the old rule are now chunks in the index, and they match refund questions better than the handbook does because they were written as answers to those questions
• The new handbook chunk competes with every highly rated answer that repeats the old rule
• Tags every feedback-derived chunk with its source, date and the handbook chunks it was answered from, and removes it when any of those chunks changes
• Keeps feedback chunks out of the citation path, or ranks them below primary documents, so the assistant never cites a past answer as policy

5. Larkspur ingests the handbook from the help centre, the PDF export and the internal wiki, so the export rule is stored three times. Top-3 retrieval returns it three times. Do you add Dartboard, MMR or deduplication?

What a strong answer covers:

• Deduplicates at indexing time first: exact or near-duplicate chunks waste storage and every later step, and a content hash catches the exact copies
• Uses Dartboard or MMR for the overlap that survives deduplication, such as two documents that state one rule in different words
• Sizes the candidate pool to the duplication: in the dartboard run, a pool of 9 drawn from five copies of each chunk holds only two distinct chunks, so the third pick is a copy again
• Measures on golden questions that need two or more distinct chunks, and checks that single-fact questions do not lose their second supporting chunk