Cross-encoder reranking in a RAG chain

What you'll learn

• Explain why reading question and chunk together judges relevance better than comparing two vectors.
• Trace the Larkspur export question as a cross-encoder demotes a distractor that shares its words.
• Place a reranker in a LangChain chain as a document compressor.
• Diagnose three production mistakes: a tiny candidate list, a model loaded per question, and a threshold on uncalibrated scores.

Lesson 1 of T6.3 treated the reranker as a box that lifts evidence from rank 9 to rank 3. This lesson opens it. Both stages read text with a transformer encoder; they differ in what goes into the encoder together:

Figure 1: A bi-encoder encodes each text alone; a cross-encoder encodes the pair.

A bi-encoder turns the question and each chunk into vectors separately and compares them with cosine similarity. The chunk vectors are computed once, when the index is built, so answering a question costs one embedding and a nearest-neighbour search however large the index is. all-MiniLM-L6-v2, the first stage in lesson 1 of T6.3, has about 22M parameters and 384-dimensional vectors.

The price is that each chunk is summarised into one vector before any question exists, and that one vector serves every question. We call the result a near miss: a chunk that shares the question's words and not its answer.

A cross-encoder reads the question and one chunk as a single input, so every question token can attend to every chunk token, and it outputs one relevance score. Nothing can be computed ahead of time, because the input depends on the question: scoring 50 candidates means 50 forward passes. That is why it runs only on the short list.

Can I export my data after I cancel my plan? has a near miss in the lab's handbook: the cancel chunk ("You can cancel any plan from the Billing page. Cancelling stops the next charge…") shares "cancel" and "plan" with the question and says nothing about data. Here are the bi-encoder's top five and what the larger cross-encoder makes of them:

Table 1: The export question's top five, before and after BAAI/bge-reranker-v2-m3 over 10 candidates (the module's rerank precision lab).

The bi-encoder puts cancel 3rd on shared vocabulary. The cross-encoder gives it 0.001 and lifts export, which says how to download the data, into 3rd place. Precision@5 is 0.60 either way here, because export was already 5th; the change is that the top 3 is now all evidence.

The two models compare like this:

Table 2: Bi-encoder against cross-encoder.

The two notebooks put the cross-encoder into a chain in different ways:

• 02a uses LangChain's wrapper. A Hugging Face cross-encoder loads BAAI/bge-reranker-base, a cross-encoder reranker scores the candidates and keeps the top 3, and a contextual compression retriever runs it after the base retriever. In LangChain a reranker is a document compressor: it takes the retrieved documents and the question and returns fewer documents. Lesson 4 of T6.3 plugs an extractor into the same slot.
• 02b writes the step by hand. It builds question-and-chunk pairs, scores the whole list in one call, sorts by score and inserts the result into the chain as a plain function.

Three details in that code matter in production:

Table 3: Three wiring mistakes in the notebooks, and their fixes.

Look at the raw scores for the same export question from the two lab rerankers. The larger model gives retention 0.993 and cancel 0.001; the smaller one gives them 7.769 and minus 0.964. One outputs something like a probability, the other an unbounded logit. Both put the chunks in nearly the same order.

Cross-encoder scores rank the candidates for one question. Nothing guarantees that a score means the same thing across questions or across models. The 18.6 notebook notes that Cohere's reranker returns values "very close to 0 or 1" and spreads them with a beta transform before using them.

Pitfall: a fixed threshold on reranker scores. A cut at 0.5 means something for the larger lab model and nothing for the smaller. Before you drop chunks below a score, check the score distribution on your golden set, for the exact model you run.

1. Rerank 20 candidates from the hybrid first stage (T6.1), as lesson 1 of T6.3's depth curve suggested.
2. Load the model once, score all pairs in one batch, and keep the top 5.
3. Rank, then cut by count, not by a score threshold, until the golden set shows a threshold that holds.
4. Log both ranks for every cited chunk, so the near misses the reranker demotes are visible.

Lesson 3 of T6.3 asks whether an LLM can do the same job better, and what it costs to ask.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A bi-encoder embeds chunks once and scales to the whole index; a cross-encoder reads question and chunk together and needs one forward pass per candidate, so it runs on the short list.
• Reading the pair demotes near misses: for the export question the cancel chunk drops from bi-encoder rank 3 to cross-encoder 0.001, and export rises to 3rd.
• In LangChain a reranker is a document compressor. Raise the base k, load the model once, cut to the top few, and never reuse a score threshold across models.