Context-aware chunks: contextual headers and contextual retrieval

What you'll learn

• Explain how a chunk loses its subject when it is cut from its document.
• Describe a contextual chunk header and what notebook 05a measured with one.
• Compare contextual chunk headers with LLM-written contextual retrieval by cost and limits.
• Choose where to start for the Larkspur handbook, and when to move on.

Lesson 4 of T5.6 added questions as extra keys. The last technique keeps the chunk as its own key and repairs the chunk itself.

Chunks lose their surroundings. A Larkspur chunk that reads "The credits are not added automatically. You claim them by opening a ticket in the Billing category within 30 days of the incident." never says which credits, and the section heading that would tell you, "Outages and service credits", ended up in a different chunk. At child size 150 in the T5.6 lab, that heading becomes a 27-character child of its own and outscores the body text for the outage question, because it holds the question's words and nothing else. Two failures show here: a subject-less chunk that never names what it is about, and a header-only match, a heading that wins on words and carries no answer.

A contextual chunk header puts the missing context back into the chunk's text before embedding: the document title at minimum, and optionally a short document summary and the section path. Notebook 05a does it in three steps:

1. Split the document (Nike's 2023 annual report) with RecursiveCharacterTextSplitter at chunk_size=800, no overlap.
2. Ask gpt-4o-mini for the document's title, passing only the first 4,000 tokens with a note that the text is truncated.
3. Embed f"Document Title: {document_title}\n\n{chunk_text}" instead of the bare chunk.

The notebook measures one chunk, number 86, against the query "Nike climate change impact" with Cohere's rerank-english-v3.0. The chunk is about the effects of climate change but never says "Nike". The notebook reports a relevance of about 0.1 without the header and 0.92 with it.

It also reports results on KITE, an end-to-end benchmark of 50 questions over four document sets, graded 0 to 10 by an LLM, with the header made of the document title and a document summary:

Figure 1: Answer grade on KITE without and with contextual chunk headers, as notebook 05a reports it.

The average rises from 4.72 to 6.04, a 27.9% increase by the notebook's figure. The gain is largest on the BVP Cloud 10-Ks (2.6 to 6.3) and smallest on the Sourcegraph handbook (5.7 to 5.8). On FinanceBench the notebook reports 83% against a 19% baseline, but for CCH and relevant segment extraction together, so the share due to headers is unknown.

Two rules from the notebook: use the same header-plus-chunk text in the reranker that you embedded, and show the header to the LLM with each result so it reads the chunk in context. For the handbook, the header can come from structure you already have: T5.3's MarkdownHeaderTextSplitter puts the section path in metadata, and T5.2 keeps titles in metadata, which is not embedded until you copy it into the text.

A header gives every chunk of a document the same context. Contextual retrieval asks an LLM to write context for each chunk, with the whole document in view: what this passage is about and where it sits. The context is prepended to the chunk, and the combined text is what gets embedded and returned.

Figure 2: Contextual retrieval. An LLM reads the whole document and writes a context for each chunk.

Notebook 05b builds a small RAG system this way. PDF pages are loaded with PyMuPDFLoader and split at chunk_size=3500 with no overlap. For every chunk, generate_chunk_context sends the full document and the chunk to gpt-4o-mini with instructions to "Provide a concise context (3-4 sentences max) for this chunk", phrased as "Focuses on …". The context and chunk go into Chroma with hnsw:space set to cosine, next to Wikipedia documents indexed without context, and a k=5 similarity retriever feeds a "stuff" prompt. The notebook gives context to only two of the twelve sample PDFs (the file names must contain "attention", "transformer" or "vision"), because the whole document has to fit in the prompt; its note says this is "to handle GPT4 out of context error".

Anthropic introduced the technique in Introducing Contextual Retrieval (September 2024). Its example adds "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million." before the chunk "The company's revenue grew by 3% over the previous quarter." Its reported results, on its own datasets:

• "Contextual Embeddings reduced the top-20-chunk retrieval failure rate by 35% (5.7% → 3.7%)."
• "Combining Contextual Embeddings and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 49% (5.7% → 2.9%)."
• "Reranked Contextual Embedding and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 67% (5.7% → 1.9%)."

The post says the context is "usually 50-100 tokens" and is prepended "before embedding it and before creating the BM25 index", so the keyword index (T6.1) benefits as well. On cost, it relies on prompt caching so that each document is loaded into the cache once and reused for its chunks; with its stated assumptions, "the one-time cost to generate contextualized chunks is $1.02 per million document tokens".

Table 1: Contextual chunk headers and contextual retrieval compared.

For the chunk about claiming credits, start with the cheaper fix:

1. Prepend the title and section path, "Larkspur Cloud handbook > Outages and service credits", taken from the metadata T5.2 and T5.3 already keep.
2. Use the same header-plus-chunk text in the reranker, and show it to the LLM with each result.
3. Re-run the golden set. Move to contextual retrieval only if headers fall short.
4. Check the document fits the prompt before contextual retrieval: each handbook page is short enough, a long report may not be.

Next: T6.1 searches these chunks with keywords as well as vectors, and combines the two rankings.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Chunks lose their subject. A contextual chunk header prepends the document title, and optionally a summary and section path, to the text before embedding.
• Use the same header-plus-chunk text in the reranker that you embedded, and show the header to the LLM with each result.
• Contextual retrieval writes context for each chunk with the whole document in view. It costs one LLM call per chunk and needs the document to fit the context window, so start with headers.