Retriever patterns compared; hybrid + rerank in a managed vector store

What you'll learn

• Match four retriever patterns to the failure each one fixes.
• Count the LLM calls a nested retriever spends per question.
• Compare hybrid search inside the vector store with an ensemble in the application.
• Judge a vendor's quality claim for a reranker before paying its latency.

Lessons 1 to 5 of T6.1 each fixed one retrieval failure: forced top-k, the dense miss on exact codes, the merge of two lists, redundant chunks, chunks that should never be allowed. This lesson puts the fixes side by side as patterns a team can pick from.

Larkspur's team wants better answers and adds every pattern they have heard of: multi-query in front, compression behind, an ensemble underneath. Answers improve a little, and the cost per question rises several times over. Nobody counted the calls before stacking them.

Notebook 06a runs four patterns over the same eight topic-tagged documents in Chroma. Each fixes a different failure and costs something different:

Table 1: Four patterns, the failure each fixes, and its price.

Choosing starts from the failure seen on the golden set, not from the pattern list. Here is that diagnosis as a chain of checks:

Figure 1: Pick the pattern from the failure, one check at a time.

06a's ensemble demo uses three questions chosen to separate the halves: ACID transactions (keyword-heavy, BM25 should help), How do I store AI model outputs for later retrieval? (worded unlike any document, dense should help), and fast similarity lookup for embeddings (mixed). It prints the top result from BM25, dense and the ensemble for each, side by side. That is the shape a team's own comparison should take: golden questions of each kind, every retriever's answer next to the others.

Because every pattern is a retriever, they nest. 06a's last demo wraps multi-query inside compression and feeds the result to a prompt:

Figure 2: Multi-query inside compression, as 06a builds it.

Multi-query widens the pool for recall and compression narrows each chunk for precision, so the two pull in opposite directions on purpose. The calls add up, and the middle term grows with the union of results:

Table 2: LLM calls per question in the nested pipeline (illustrative: the union size depends on how much the variants overlap).

We call this call stacking: each wrapper's cost is reasonable alone, and the product of them is what the bill shows. Nine calls in sequence also means nine round trips of latency before the user sees a word.

Pitfall: call stacking. Count calls per question before nesting, and measure each layer's gain on the golden set separately. A layer that adds two calls and no recall comes out.

Notebook 06b, an optional notebook, moves hybrid search and reranking into a managed vector store. It embeds the question with BAAI/bge-large-en-v1.5 (1,024 dimensions, normalised) and makes one call per mode, returning 5 results:

Table 3: Three search modes in 06b.

Hybrid in the store and an ensemble in the application do the same job in different places:

Table 4: Two places to run hybrid search.

06b describes the reranker as about 15% better in quality for 1 to 1.5 seconds of extra latency. The notebook states those figures without measuring them. Treat a number like that as a hypothesis: measure the gain and the latency on your own golden set before paying for it. T6.3 covers rerankers and the retrieve-wide, rerank-narrow design.

1. Diagnose first. Run the golden set, and sort misses into Figure 1's boxes.
2. Ensemble for codes. Error codes and plan names were the dense misses in lesson 2 of T6.1; BM25 plus dense with RRF fixes them for one extra search and no LLM call.
3. Hold multi-query until paraphrase misses remain after the ensemble, and count its calls when it goes in.
4. Choose the location by scale. An application ensemble while the handbook is small and changing; the store's hybrid mode once many teams write to it.

T6.2 picks up where multi-query leaves off, with the full family of query rewrites.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Each pattern fixes one failure at its own price: multi-query fixes wording, compression trims noisy chunks at one LLM call each, an ensemble catches exact terms, and parent documents restore missing context.
• Patterns nest because each is a retriever, but calls multiply (call stacking): multi-query inside compression can spend about 9 LLM calls where plain retrieval spends 1.
• Hybrid in the store fuses on the server and stays in sync; an application ensemble is easier to inspect. Measure any reranker's gain and latency on your golden set before enabling it.