Relevant segment extraction

What you'll learn

• Explain why relevant chunks cluster inside their documents, and how RSE uses that.
• Trace chunk values through a penalty to the best contiguous segment.
• Compute the best segment for a short list of chunk values by hand.
• Judge a reported benchmark gain that this course did not measure.

Lesson 5 of T6.3's window grows every hit by the same amount. A factoid answer needs no neighbours; a financial statement spread over a dozen chunks needs all of them. A fixed window serves neither well.

Relevant segment extraction (RSE) lets the reranker's scores decide how much to send. It rests on one observation from notebook 06a: "relevant chunks tend to be clustered within their original documents". RSE finds those clusters and sends each as one contiguous segment, in document order, including chunks between two relevant ones that the reranker scored low.

06a runs it on Nike's 2023 annual report, in four steps:

Figure 1: The RSE pipeline in 06a.

1. Chunk with no overlap: 800-character chunks, overlap 0. Without overlap, joining neighbouring chunks rebuilds the original text exactly.
2. Score every chunk: Cohere's hosted reranker scores each chunk against Nike consolidated financial statements. The scores bunch near 0 and 1, so a beta transform (both shape parameters 0.4) spreads them out, and each is multiplied by a decay on its rank (e to the minus rank over 30), so rank counts as well as score.
3. Subtract a penalty: 0.2 per chunk, which makes irrelevant chunks negative and relevant ones positive.
4. Find the best segments: a segment's value is the sum of its chunk values, so finding the best one is a constrained maximum-sum run.

06a's worked example has five chunk values after the penalty:

Figure 2: Five chunk values and the best segment.

Checking each candidate run is quick arithmetic:

Table 1: Candidate segments for 06a's five values (chunks numbered from 0).

The search returns chunks 2 to 3 with a value of 1.2. No other run of chunks outside it has a positive sum, so it stops after one segment.

On the full report the search runs by brute force under three limits: at most 20 chunks per segment, 30 chunks in total, and a minimum segment value of 0.7. It repeats until no segment clears the minimum. Only 7 of chunks 320 to 339 were marked relevant, yet chunks 323 to 333 all hold the consolidated financial statements; the first segment is 323 to 336. Top-k retrieval would have left out the low-scored chunks inside the cluster. We call that the gapped cluster: relevant text interrupted by chunks a reranker scores low on their own.

When only one chunk is relevant, there is no cluster to join, and RSE returns that chunk on its own. It behaves like top-k retrieval on factoid questions and like a large chunk on broad ones, without fixing the chunk size in advance.

06a reports its authors' evaluation on their KITE benchmark: 50 questions across four document collections (AI papers, cloud-company 10-Ks, a company handbook and Supreme Court opinions), each answer graded 0 to 10 by an LLM. RSE is compared with top-k retrieval at k = 20, with the same Cohere reranker and the same generation model in both.

The average score rises from 4.72 with top-k to 6.73 with RSE. The notebook reports the context length as roughly equal in both configurations, so the gain does not come from sending more text.

Pitfall: quoting a benchmark you did not run. These are the authors' numbers on their benchmark, not a measurement in this course. The notebook's per-dataset table also does not add up to its averages (two cells read 45.5 and 45.4 on a 0 to 10 scale), so only the averages are quoted here. Measure RSE on your own golden set before relying on it.

RSE spends the context budget from lesson 4 of T6.3 in a bounded way: at 06a's settings, at most 30 chunks of 800 characters, 24,000 characters in all.

1. Chunk each handbook section with no overlap, so segments rebuild the text exactly.
2. Cap segment length at a section's chunk count: Larkspur's sections are short, so 20 chunks per segment would span several topics.
3. Cap the total to what the generation prompt holds after the instructions.
4. Compare with lesson 5 of T6.3's windows on the golden set; on a handbook of one-sentence rules, the two may send the same text.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Relevant chunks cluster within their documents. RSE subtracts a penalty from each chunk's reranker value and sends the contiguous runs with the highest sums, including low-scored chunks inside a cluster (the gapped cluster).
• Chunk with no overlap, so joined neighbours rebuild the original text exactly. RSE then acts like top-k on factoid questions and like a large chunk on broad ones.
• The KITE gain (average 4.72 to 6.73) is the authors' number, not a course measurement. Cap segment length and total chunks to fit your prompt.