Retrievers & Hybrid Search: overview

An embedding retriever finds chunks that mean what the question means, and misses chunks that only share a rare exact token: an order ID, a plan name, an error code. A keyword retriever has the opposite failure. T6.1 runs both and merges their rankings. The T6.1 notebooks build on four ideas: dense and sparse retrieval, the BM25 ranking function, Reciprocal Rank Fusion, and the λ that trades relevance against diversity in Maximal Marginal Relevance.

The running example is the Larkspur Cloud handbook from T5.1 to T5.3, now searched with two kinds of question. Some are worded unlike the handbook, such as What happens if my server goes down?, whose evidence talks about "outage" and "availability"; the dense retriever is built for these. Others hang on an exact string, such as an order ID like ORD-88213, which the T5.1 symptom table already sends to this module; BM25 is built for these. The lab's golden set mixes both kinds: paraphrase questions, and questions that name an error code such as E-4021.

• Dense retrieval compares embedding vectors and matches paraphrase; sparse retrieval (BM25) scores shared tokens and matches exact terms. Each misses what the other finds, so production retrieval usually runs both.
• BM25 is TF-IDF with two corrections: repeated terms saturate (k1, 1.5 by default in rank_bm25) and long chunks are discounted (b, 0.75). LangChain's BM25Retriever splits on whitespace only, so application. and application are different tokens.
• Fusing two rankings by weighted min-max scores (03a, alpha = 0.5) depends on each retriever's score scale. Reciprocal Rank Fusion uses ranks only: EnsembleRetriever adds weight / (60 + rank) per list. With equal weights of 1, a chunk ranked 3rd by both retrievers (2/63 = 0.0317) beats one ranked 1st by only one (1/61 = 0.0164).
• MMR picks the most similar chunk first, then trades relevance against redundancy with lambda_mult (0.5 by default) over fetch_k candidates (20 by default).
• Filters on metadata, score, keywords and diversity shrink the context after retrieval. Score thresholds belong to one embedding model; re-tune them when the model changes.

Two retrievers rank the same chunks by different evidence, and fusion turns two ranked lists into one. Fusion only reorders what the two retrievers returned: a chunk that neither puts in its top k is gone before the merge, so give each retriever a wider k than the final one.

Hybrid fusion lab: open the file in a browser. It ranks 19 handbook chunks with real BM25 and with all-MiniLM-L6-v2 cosine similarity, then fuses them. On seven golden questions, mean recall@3 is 0.57 for BM25 alone, 0.64 for dense alone and 0.93 fused at alpha = 0.5.

• Pick E-4102 for the third time. Can I take my files with me another way? and slide alpha from 0 to 1. BM25 alone and dense alone each miss one of the two evidence chunks; the blend in between sends both.
• On What does E-4021 mean?, push alpha toward 1. Dense retrieval ranks the code chunk 5th, behind look-alike codes, so the exact code drops out of the top 3.
• Switch to RRF with E-5013 for the third time. Will my machines get shut off?. With 4 chunks per retriever the suspension chunk is never handed over; with 10 it returns. Changing c barely moves recall here.

• Before: T5.1 gives the baseline and the symptom table that sends exact-code misses here; its TF-IDF lab is the sparse half in miniature. T5.3 sets chunk length, which BM25's b parameter normalises. T5.4 and T5.5 supply the embeddings and the vector store behind the dense half.
• Next: T6.2 rewrites the question before these retrievers see it, and RAG-Fusion applies this module's RRF to several query variants. T6.3 reranks the fused list with a cross-encoder.

• Run sparse and dense retrieval together when questions mix exact strings with free wording. Check the BM25 tokeniser before you trust its misses.
• Prefer RRF when the two retrievers' scores live on different scales; it needs no calibration and rewards agreement. Use weighted scores only if you join them by id and tune the weight on the golden set.
• Fusion and MMR reorder a candidate pool and cannot add to it. Retrieve wider than you return.
• Set λ, score thresholds and fusion weights by measuring them on your golden set, and measure again when the embedding model changes.