Retrieve wide, rerank narrow.

A 100-chunk Larkspur Cloud handbook. A bi-encoder returns the top k chunks; a cross-encoder rescores those k pairs and the best five go to the model. Every score here is real model output, computed offline and embedded.

Golden question

First-stage depth, chunks the bi-encoder returns (k)

Second stage, the reranker

Recall@k, first stage
Precision@5, no rerank
Precision@5, reranked
Rerank latency for k pairs

The k candidates, in the order the model sees them

Rank	Chunk	Stage 1	Moved	Cosine	Rerank	Golden set

Stage 1 is the chunk’s rank by cosine similarity. Rerank is the cross-encoder’s score for the (question, chunk) pair: a raw logit for MiniLM-L6, a 0 to 1 sigmoid for bge-reranker-v2-m3. Only the order matters. Rows past rank 10 that hold no evidence are folded.

All four questions at this k

Golden question	Recall@k	P@5 before	P@5 after
Mean

Each question has two or three evidence chunks, so precision@5 tops out at 0.40 or 0.60. Compare the two precision columns; the M10 baseline row works the same way.

What the second stage costs

k pairs	MiniLM-L6, ms	bge-reranker-v2-m3, ms