Retrievers & Hybrid Search: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur moved support search from keyword search to embeddings. Agents now report that pasting an order ID such as ORD-88213 returns general refund-policy chunks instead of the one note that mentions that order. What happened, and what do you change?

What a strong answer covers:

• Rare identifiers carry little meaning for an embedding model, so the ID's chunk is not near the query vector; confirms it by printing the dense ranking and the chunk's rank
• Adds BM25 next to the dense retriever and fuses with RRF or weighted scores, keeping the dense half for paraphrased questions
• Checks the tokeniser: whitespace splitting keeps ORD-88213 whole, but ORD-88213? or a lower-cased copy would not match without a normalising preprocess_func
• Adds exact-ID questions to the golden set so a later change cannot drop them silently

2. Your team must choose between weighted min-max fusion with an alpha and Reciprocal Rank Fusion for the handbook search. Which do you ship, and how do you tune it?

What a strong answer covers:

• Min-max is relative to each query's result list: the top chunk always scores 1.0, even when nothing matches, and one outlier compresses the rest
• RRF ignores score scales, needs no per-retriever calibration, and rewards chunks both retrievers agree on; its constant (60 in EnsembleRetriever) and the list weights are the knobs
• Tunes alpha or the RRF weights on the golden set, split by question type (exact-ID, paraphrase, mixed), not on the mean alone
• Joins scores by chunk id, never by list position, and widens each retriever's k beyond the final k

3. Hybrid search is live. How do you know, week to week, that both halves are still pulling their weight?

What a strong answer covers:

• Logs both rankings and the fused one for each query, with chunk ids
• Tracks how often the final top-k comes only from BM25, only from dense, or from both; a half that never contributes is either broken or unneeded
• Re-runs the golden set on every index, embedding-model or tokeniser change and compares recall per question type with the baseline
• Alerts on the share of queries where BM25 returns nothing, which flags tokenisation drift such as new ID formats

4. The handbook index holds the current refund policy and last year's version. With MMR turned on, answers start quoting the old policy. Why, and what is the fix?

What a strong answer covers:

• MMR knows similarity, not dates. The two versions score almost the same against the question, so the old one can be the first pick, and MMR then penalises the current one as redundant
• Diversity is the wrong tool for versioning; filter on version or date metadata before MMR, or remove superseded chunks at index time
• Prints the fetch_k candidates and the picks at the current lambda_mult to confirm which version wins the first pick
• Adds a golden question whose correct answer depends on the current version

5. A teammate copied score_threshold=0.3 from a tutorial into the handbook retriever, then switched embedding models. Now out-of-scope questions return five chunks, and some in-scope ones return none. What is going on?

What a strong answer covers:

• Similarity scores are not comparable across embedding models or distance functions; a threshold is a property of one model on one corpus
• Checks whether the store returns a distance (lower is better, similarity_search_with_score) or a relevance score (higher is better, similarity_search_with_relevance_scores), and how the store converts one into the other
• Re-tunes the threshold from the score distributions of in-scope and out-of-scope golden questions
• Treats an empty result as a valid outcome with a fallback answer, rather than lowering the threshold until something comes back