Dartboard RAG: relevance and diversity together

What you'll learn

• Explain why top-k fills every slot with copies when the index holds duplicates.
• Trace the dartboard method's greedy choice, one chunk at a time.
• Size the candidate pool so a diverse answer is possible at all.
• Choose between deduplication, Dartboard and MMR.

Lesson 2 of T9.5 showed that feedback can only reorder the candidates retrieval returns. This lesson stays inside that limit: a better way to choose among the candidates, when many of them say the same thing.

Larkspur ingests its handbook from three places: the help centre, the PDF export and the internal wiki. Every rule is now stored several times. A customer asks Can I export my data after I cancel my plan? The answer needs two chunks, export (how to export) and retention (how long data is kept after cancelling). Plain top-3 retrieval returns export, export, export, and the answer never mentions the 30-day deadline. The runs below store each chunk five times, so the effect is easy to see.

Why top-k repeats itself

Top-k scores each chunk against the query on its own and keeps the best k. Copies of one passage score identically, so whichever passage scores best takes every slot. We call this duplicate crowding. The notebook creates it on purpose: it loads its climate-change PDF five times over, splits it into 1,000-character chunks with 200 overlap, and shows that the top 3 results for What is the main cause of climate change? are repetitions of one chunk.

The fix is to score the set rather than each chunk. Diversity-aware selection picks chunks so that the set, taken together, covers the query well, which means a chunk adds value only when it brings something the chosen ones do not.

The dartboard picture

The notebook's method comes from Better RAG using Relevant Information Gain (arXiv 2407.12101), the dartboard method. Picture the query as the centre of a dartboard and darts thrown around it, landing more often near the centre. A set of chunks scores by how close its nearest member lies to where each dart lands.

A second copy of a chunk already in the set lies exactly where the first one does, so it brings no landing place any closer and adds nothing. A chunk that covers ground nearby, close to the query but away from the chosen chunks, adds the most.

The greedy rule

Finding the best set exactly is expensive, so the method builds it one chunk at a time from a candidate pool of num_results × oversampling_factor nearest chunks (the factor is 3 by default):

1. Turn distances (1 minus cosine similarity) into log probabilities with a normal curve of width SIGMA = 0.1, for query to chunk and for chunk to chunk.
2. Pick the chunk most probable under the query.
3. Treat every candidate as a place a dart can land, weighted by how close it is to the query (RELEVANCE_WEIGHT). For each remaining candidate, measure how close the set would come to each landing place if that candidate joined (DIVERSITY_WEIGHT), and sum over the landing places. Both weights are 1.0.
4. Add the highest scorer and repeat until num_results are picked.

Figure 1: Greedy dartboard selection. Every pick after the first is scored on the coverage it adds to the set so far.

SIGMA sets how quickly closeness stops counting: a small width means only near neighbours of a landing place matter, so two chunks a little apart already count as covering different ground.

To see both the gain and its limit, we ran the notebook's selection function, unchanged, on the twelve Larkspur chunks stored five times (60 chunks, TF-IDF vectors), for the export question with num_results = 3:

Table 1: Dartboard against plain top-3 on the handbook stored five times, export question.

Plain top-3 has recall 0.50 on this question at every pool size. Dartboard reaches 1.00 once the pool holds retention, from an oversampling factor of 2. At the default of 3 its third pick is still a copy of export, because a pool of 9 drawn from five copies of each chunk holds only two distinct chunks.

That is the pool ceiling, the candidate ceiling of lessons 1 and 2 of T9.5 in a new form: a selector can only choose among the candidates it is given. The arithmetic for sizing the pool is direct when copies rank side by side:

Table 2: Pool size needed when every chunk has the same number of copies, (copies × (wanted − 1)) + 1.

The default pool of 9 sits between the first two rows, which is exactly why Table 1 shows two distinct picks at factor 3 and three at factor 4 (pool 12). Real duplication is uneven, so the reliable fix is to remove the copies before retrieval and keep oversampling for the overlap that remains.

Dartboard and MMR, maximal marginal relevance (lesson 4 of T6.1), solve the same problem the same greedy way and differ in the score:

Table 3: Two greedy selectors compared.

Because Dartboard needs only distances, the notebook's notes point out that it works the same on dense, hybrid or cross-encoder scores. Neither selector replaces deduplication: a content hash at indexing time catches exact copies cheaply, and every later step then works on fewer, distinct chunks.

Figure 2: Choosing a fix for redundant results.

Measure the choice on golden questions that need two or more distinct chunks, such as the export question, and check that single-fact questions do not lose their second supporting chunk.

Limits of the notebook

• The first score is a placeholder. The first pick is recorded with a fixed score of 1.0, not a computed one, so it cannot be compared with the scores after it.
• The last example call fails. It passes the result count under the name k, while the function names that parameter num_results, so Python raises a type error until the name is corrected.
• One question, one corpus. Table 1 is one question on a twelve-chunk handbook with uniform copies. Uneven duplication and near-duplicates in other words were not tested.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Top-k fills every slot with copies of the best match when the corpus holds duplicates. Dartboard selection scores a set by how close its nearest member lies to each landing place, so a copy adds nothing.
• The selector only chooses among its num_results × oversampling_factor candidates. With five copies per chunk the default pool of 9 holds two distinct chunks, so enlarge the pool past the copy count or deduplicate at indexing.
• Dartboard and MMR both select greedily and differ in the score. MMR subtracts a linear redundancy penalty, while Dartboard adds log probabilities and needs only distances, so it works on dense, hybrid or cross-encoder scores.