Dense + sparse hybrid retrieval with BM25 and EnsembleRetriever

What you'll learn

• Explain why an embedding retriever misses an error code that a keyword retriever finds, and the reverse.
• Compare dense and sparse retrieval on the Larkspur golden questions, rank by rank.
• Explain the two corrections BM25 makes to TF-IDF.
• Diagnose a keyword retriever that misses because of its tokeniser, before touching its parameters.

Lesson 1 of T6.1 treated the retriever as a black box with one method. This lesson opens two different boxes and shows that each one fails on the questions the other gets right.

A Larkspur customer pastes What does E-4021 mean? The handbook has a chunk that starts "Error E-4021: the export stopped at the 50 GB snapshot limit." The embedding retriever ranks it 5th, behind four other error-code chunks that look alike to it, and a top-3 cut drops the one chunk that answers.

A dense retriever turns every chunk and the question into an embedding, a vector in which every position holds a number, and ranks chunks by vector similarity. A sparse retriever describes a chunk by its words: one weight per vocabulary word, zero for almost all of them. The two match on different evidence:

Table 1: Dense and sparse retrieval side by side.

An error code has no learned meaning for an embedding model, so its few characters move the chunk's vector very little. BM25 treats e-4021 as one more token and a rare one, which is the case it weights most.

The module's lab ranks the same 19 Larkspur chunks both ways, with BM25 and with all-MiniLM-L6-v2 cosine similarity. Here is where each retriever puts the evidence for its seven golden questions:

Table 2: Rank of each evidence chunk (computed with the lab's BM25 and all-MiniLM-L6-v2). "None" means BM25 gave the chunk a score of 0.

The pattern is the one Table 1 predicts. The downtime questions share no word with the incidents chunk ("root-cause report", "recovery"), so BM25 ranks it 9th or not at all, while the dense retriever puts it first. The code questions flip it: BM25 finds the code chunk at rank 1 or 2, and the dense retriever finds the look-alike codes first.

Key distinction: neither retriever is the better one. Each fails on a question type the other handles, so the golden set has to contain both types, or a switch from one to the other looks like a pure win.

02a combines the two with EnsembleRetriever:

Figure 1: The hybrid chain in 02a. Both retrievers see the same question; the ensemble merges their lists.

The weights follow the order of the retrievers list: 02a gives 0.7 to dense and 0.3 to BM25, and 02b lists BM25 first with 0.5 each. The ensemble hands its merged list to a chain that stuffs every returned chunk into the prompt. Lesson 3 of T6.1 shows how the merge works and how those weights can bury a keyword hit.

BM25 scores a chunk by summing one term for every query word the chunk contains. Each term multiplies how rare the word is by how often it appears in the chunk, with two corrections that plain TF-IDF lacks:

Table 3: The pieces of a BM25 term. The term-frequency part is tf × (k1 + 1) / (tf + k1 × (1 - b + b × length / average length)).

In plain TF-IDF a word that appears ten times counts ten times as much as one that appears once. A chunk that repeats "refund" ten times is not ten times as relevant, so BM25 bends the curve. Here is that curve for chunks of three lengths:

Figure 2: Term-frequency weight against occurrences, with k1 = 1.5 and b = 0.75.

Reading four points off the curve, as multiples of the word's idf:

Table 4: Saturation and length normalisation in numbers.

The second occurrence adds 0.43, the next eight add 0.74 between them, and the ceiling is 2.5. Length normalisation exists because a long chunk contains more words by chance.

BM25 can only match tokens the tokeniser produces. LangChain's BM25Retriever uses BM25Okapi from rank_bm25 and, by default, splits on whitespace: no lower-casing, punctuation kept on the word.

02a indexes five sentences and asks How can I build an application using LLMs? Here is what the default tokeniser lets match (scores computed with rank_bm25 on that corpus):

Table 5: What whitespace splitting matches.

The question's application does not match application. with its full stop, and LLMs? does not match LLM. Case splits words too: LangChain (one sentence) and Langchain (two sentences) are separate tokens with idf 1.10 and 0.34. The retriever still returns k = 3 sentences, so its third result scored 0 and was picked by tie order. We call this token mismatch: the evidence holds the word, and the index holds a different string.

The fix is a tokeniser that lower-cases, strips punctuation and keeps hyphenated identifiers in one piece, passed to the retriever as its preprocessing function. On the same corpus it lifts the second sentence to 1.947, because application now matches. A code typed at the end of a question (ORD-88213?) becomes ord-88213 and matches the chunk that contains it. The lab's BM25 in Table 2 uses this tokeniser.

Pitfall: tuning around a token mismatch. LLMs still misses LLM after the fix; matching word forms needs a stemmer. Tune k1 and b only after the tokens are right, because the tokens decide what can match at all.

For Larkspur's support search, Table 2 says which questions need which half:

1. Keep the dense half for questions worded unlike the handbook: downtime, "money back", "shut off".
2. Add BM25 for questions that quote an error code, a plan name (LC-408) or an order ID.
3. Tokenise for codes: lower-case, strip punctuation, keep e-4021 and lc-408 whole. Check it on a pasted code with a trailing question mark.
4. Put both question types in the golden set, so a change that helps one type and hurts the other shows up.

Running two retrievers gives two ranked lists for every question. Lesson 3 of T6.1 turns them into one.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Dense retrieval matches meaning and misses rare exact strings; BM25 matches exact tokens and misses paraphrase. On the Larkspur set, dense ranks err-4021 5th and BM25 ranks incidents 9th.
• BM25 is TF-IDF with saturation (k1, each extra occurrence adds less) and length normalisation (b, long chunks scaled down).
• Fix the tokeniser before tuning k1 and b. Whitespace splitting keeps case and punctuation, so application. never matches application (token mismatch).