Contextual compression

What you'll learn

• Explain the context budget and which post-retrieval steps save or spend it.
• Trace a chunk through an extractor, and see the compressor slot that rerankers and filters share.
• Diagnose a compressed chunk that lost the condition on a rule.
• Plan saving and spending together for one Larkspur pipeline.

Lessons 1 to 3 of T6.3 decided which chunks reach the prompt. Each one still arrives whole, and most of a chunk is usually not about the question.

The context budget is the share of each prompt that retrieved text may take. Four things set it: the model's context window, the cost of input tokens, the latency of reading them, and the position effect from lesson 1 of T6.3, which makes a long context less reliable than a short one.

Every step after retrieval either saves from that budget or spends it:

Table 1: Post-retrieval steps and the budget.

Saving and spending go together. Here is one budget of 4,000 characters of retrieved text, spent three ways (Illustrative, using 05a's split of 400-character chunks with 200 of overlap):

Table 2: One budget, three shapes (illustrative).

A pipeline that expands each hit to its neighbours can afford it by keeping five hits instead of ten. One that compresses each chunk can keep many more, if it trusts the compressor.

Contextual compression cuts each retrieved chunk down to the parts that bear on the question. Notebook 04a uses an LLM chain extractor, wrapping a small chat model at temperature 0, inside the contextual compression retriever, the same wrapper lesson 2 of T6.3 used for the cross-encoder. For each retrieved chunk the extractor asks the LLM to copy out only the relevant parts; a chunk with nothing relevant comes back empty and is dropped. A question-answering chain then answers from what is left.

All three post-retrieval moves fill the same slot:

Figure 1: The compressor slot. The retriever and the chain stay the same whichever compressor fills it.

Here is what extraction does to one Larkspur chunk for How long does a refund take?:

Table 3: Extraction keeps the answer and drops the rest (illustrative extraction).

Compression has the same risk as T5.3's propositions: an LLM sits between the source and the prompt. Ask Can I get a full refund on my annual plan? and the refund-window chunk comes back. Two extractions are plausible:

Table 4: A planted lossy extraction (illustrative).

The lossy version reads as more relevant, because the traffic condition looks like detail. We call this the dropped qualifier: the extractor keeps the rule and removes the condition that limits it, and the model gives a confident wrong answer to a customer who used 400 GB.

Pitfall: the dropped qualifier. Check compressed context for faithfulness on golden questions whose answers carry conditions, and keep each chunk's id so the citation points at the full source text, not the extract.

Filtering has a simpler failure. The optional 02b notebook asks a small model whether each chunk suits the question and keeps it only if the reply equals the string "True" exactly. A reply of "True." with a full stop drops the chunk. We call this string-match judging. Ask for structured output, as lesson 3 of T6.3's pointwise reranker does, instead of matching text.

1. Rerank and cut to 5 before any compression; never pay an LLM call per chunk on 50 candidates.
2. Compress only long chunks. Larkspur's handbook chunks are one or two sentences, so extraction saves little and risks the dropped qualifier; it pays on long policy pages.
3. Keep chunk ids through compression, and cite the full chunk.
4. Add a golden question with a condition (the 100 GB limit) and check the answer states it.

Compression shrinks a chunk to fit. Lesson 5 of T6.3 does the opposite, for answers that run past a chunk's edge.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Every step after retrieval saves or spends one context budget. Plan them together: expanding each hit means keeping fewer hits, and compressing each chunk lets you keep more.
• An extractor copies out only the relevant parts of each chunk at one LLM call per chunk, and drops chunks with nothing relevant; rerankers, extractors and filters all fill the same compressor slot.
• An extractor can drop the condition that limits a rule (the dropped qualifier). Check compressed context for faithfulness, keep chunk ids for citations, and judge filters with structured output, not string matches.