Post-Retrieval Optimisation & Reranking: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. You add a cross-encoder reranker over the top 50 for Larkspur's support assistant. Latency per question jumps, and answers to “How long does a refund take?” are no better. Where do you look?

What a strong answer covers:

• Checks recall@50 first: if refund-timing is not among the 50 candidates, no reranker can promote it, and the fix belongs in stage 1 (hybrid search in T6.1, query rewriting in T6.2)
• Profiles the rerank step: model loaded once at start-up rather than per request (the walkthrough notebook builds its CrossEncoder inside the rerank function), pairs scored in one batch, long chunks truncated to the model's input limit
• Compares precision@5 with and without the reranker on the golden set, per question, so one regression is not hidden in the mean
• Lowers the depth to where recall@k stops rising, which cuts the pair count and the latency with it

2. The team wants the best possible ranking for a high-traffic support chat and proposes an LLM reranker. What do you recommend?

What a strong answer covers:

• Pointwise LLM reranking costs one call per candidate (up to 30 calls per question in the 18.3 notebook); listwise costs one long call whose output must be parsed back into an order
• A cross-encoder scores every pair locally in one batch, with no parsing step that can fail
• Default to the cross-encoder and reserve the LLM for low-volume, high-stakes questions, or for re-scoring only the top few after the cross-encoder
• Decides on golden-set precision@5 and p95 latency together, not on a demo query

3. How do you choose the first-stage depth: 20, 50 or 100 candidates?

What a strong answer covers:

• Plots recall@k for stage 1 on the golden set and finds where it flattens
• Measures rerank latency at each depth, since cross-encoder cost is one pass per pair
• Picks the shallowest depth on the flat part of the recall curve that fits the latency budget
• Re-checks when the corpus grows or the embedding model changes, because the curve moves

4. “Can a refund be prorated after 45 days?” The reranker puts the refund-window chunk first, but the sentence with the proration rule sits in the next chunk and never reaches the model. What do you change?

What a strong answer covers:

• Recognises a boundary problem: the ranking is right and the unit of context is too small
• Expands each top chunk to its neighbours (18.5) or extracts a contiguous segment (18.6), using a key-value lookup by document and chunk index
• Accounts for the extra tokens in the context budget, for example by keeping fewer chunks
• Checks that compression (18.4) is not the step dropping the qualifier, since an extractor can cut the condition from a rule

5. The reranker has run in production for a month. How do you show it is still earning its latency?

What a strong answer covers:

• Logs, for every answer, the stage-1 rank and the reranked rank of each cited chunk
• Tracks how often a cited chunk came from below the top 5 of stage 1, which is the work only the reranker did
• Re-runs the golden set on every change to the index, the embedding model or the reranker version, and alerts on a fall in precision@5
• Watches rerank latency percentiles separately from generation latency