Semantic chunking: breakpoints on embedding distance

What you'll learn

• Explain how semantic chunking turns sentence embeddings into chunk boundaries.
• Compare the four threshold types by how they set the cut-off and whether they always cut.
• Diagnose a section with three topic changes that gets cut only once, and a topic that drifts without a cut.
• Choose a threshold per corpus, with a size cap after the chunker.

Lesson 1 of T5.3 cut text by length and used the document's markup where it had any. Semantic chunking cuts on meaning instead: a boundary goes where one sentence is about something different from the next.

The refund section has 13 sentences and three real topic changes: monthly plans, the annual-plan window, refund timing, and credits. Run through the semantic chunker with its default settings, it comes back cut only once, so two of the topics share a chunk with their neighbours.

Semantic chunking places boundaries in five steps:

1. Split the text into sentences.
2. Embed each sentence, combined with its neighbours (a buffer of 1 adds one sentence on each side).
3. Compute the distance between consecutive sentences: 1 minus their cosine similarity.
4. Mark every distance above the breakpoint threshold as a boundary.
5. Group the sentences between boundaries into chunks.

The threshold decides everything that follows. Here it is on the refund section, at two percentile levels:

Figure 1: Distances between neighbouring sentences in the refund section, with the 95th and 75th percentile thresholds (illustrative distances).

The semantic chunker turns the distances into a threshold in one of four ways:

Table 1: The four breakpoint threshold types and their defaults.

All four are computed from the document's own distances. A threshold tuned on one corpus therefore does not transfer to another, and we call reusing one the relative-threshold trap. It also explains the opening example. A 13-sentence section has 12 distances, and only the largest clears their 95th percentile. Lowering the amount to 75 cuts at all three topic changes.

The types also differ in whether they must cut:

Table 2: Which threshold types can return a single chunk.

On a short text, no distance is 3 standard deviations above the mean. The default standard_deviation then returns one chunk. That is correct for a document about one thing, and it leaves chunk size with no upper limit. We call that an unbounded chunk: it can grow past the embedding model's input limit.

Only local change counts. The comparison is sentence i against sentence i + 1. A topic that drifts across five sentences never produces one large distance and never triggers a cut. This is slow drift, and gradient helps with it somewhat. One cut decision looks like this:

Figure 2: One boundary decision. Each sentence is embedded with its neighbours, and only consecutive sentences are compared.

The cost is a full extra embedding pass over every sentence at index time, before the chunks themselves are embedded.

For the refund section, semantic chunking needs four checks against the text:

1. Tune the threshold on this corpus. The 75th percentile finds the three topic changes in the refund section where the default 95th finds one. Check that on other handbook pages before keeping it.
2. Read the chunks before trusting the boundaries, which catches slow drift the distances cannot show.
3. Cap chunk size afterwards with a size-based splitter. A long single-topic section then cannot become an unbounded chunk.
4. Budget the extra embedding pass over every sentence when the handbook is re-indexed.

Next: lesson 3 of T5.3 hands the cutting to an LLM, which rewrites text into standalone facts.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Semantic chunking embeds each sentence with its neighbours and cuts where the distance between consecutive sentences clears a threshold.
• Every threshold type is relative to the document's own distances, so tune it per corpus. percentile always cuts somewhere; standard_deviation may return one chunk.
• Only local change triggers a cut, so slow drift is missed. Read the chunks, cap their size afterwards, and budget an extra embedding pass over every sentence.