Production concerns: batching, caching and similarity behaviour

What you'll learn

• Explain how an embedding cache decides whether a text has been embedded before.
• Trace five indexing runs through the cache: what reaches the model and what the store holds.
• Diagnose a model swap that re-embeds nothing and breaks retrieval.
• Choose batched calls over per-chunk calls for indexing.

Lessons 1 to 3 of T5.4 chose a metric, a model and its dimensions. This lesson is about running that choice every day: re-indexing as the handbook changes, without paying for the same vectors twice or mixing two models' vectors in one index.

The Larkspur team swaps embedding models and re-indexes the handbook. The job finishes in seconds, having sent none of the 12 chunks to the new model, and retrieval quality collapses. The cache answered every request with the old model's vectors, because both models wrote to the same cache keys.

Every re-run of an indexing notebook, and every re-ingest after a change elsewhere in the pipeline, embeds the whole corpus again and pays for it again, for vectors identical to the ones you already had. An embedding cache wraps an embedding model with a key-value store: text it has seen comes back from the store, and only new text reaches the model. LangChain's cache-backed embeddings work this way, and the notebook sets them up with five choices:

Table 1: The cache settings, as notebook 4 of T5.4 sets them.

Each lookup builds a key from the namespace and a hash of the text:

Figure 1: One cache lookup. Only a miss reaches the billed model.

A cache entry therefore matches only the exact text under the exact namespace. To see what that means for the Larkspur handbook, the 12 chunks of the T5.1 lab were run through the real cache-backed embeddings with a counting stand-in for the API (LangChain's deterministic fake embedding, so the counts are exact and free):

Table 2: Five indexing runs through the cache on the 12 handbook chunks.

Each run shows one property of the cache:

• Re-indexing after an edit costs the edit. When T5.3's chunking changes only some chunks, only those are embedded again. Chunks whose text is unchanged hit the cache.
• Old entries stay. Run 3 leaves the pre-edit vector in the store. A file store grows until you clear it. Give it a home you can delete or expire. We call these stale entries.
• The namespace is part of correctness. The cache defaults to an empty namespace. Run 5 swaps the model and gets the old model's vectors back with no error: an index mixing two vector spaces, the mismatch from lesson 1 of T5.4. We call this a shared namespace, and it is the opening failure.

Notebook 4 of T5.4 writes its cache to a temporary folder, and the cache is discarded when the cell ends.

The document method sends many texts in one request, and the query method sends one. Looping the query method over a corpus pays one network round trip per chunk, and on a real corpus that is the difference between minutes and hours. We call this per-chunk round trips. Use embed_documents for the index and embed_query for questions.

Notebook 1 of T5.4 runs both paths on the same six texts and compares the vectors:

Table 3: Batched and single embeddings of the same text.

The difference is numerical noise. Never use exact equality to decide whether two vectors came from the same text. The cache's hash of the text does that job. For an asymmetric model the two methods differ on purpose, which is one more reason to keep the two calls separate (lesson 3 of T5.4).

For the handbook's nightly re-index, the cache and batching settings come down to five decisions:

1. Namespace the cache by model and dimensions, such as text-embedding-3-small at 1536. A model swap then starts a fresh set of keys.
2. Embed in batches with the document method, never a loop of single calls.
3. Expect only edited chunks to reach the model. After a one-chunk edit, 1 of 12 texts is billed.
4. Expire the store on a schedule, or delete a namespace when its model is retired, to clear stale entries.
5. Use a shared byte store, such as Redis, once more than one worker indexes.

Next: lesson 5 of T5.4 compares embedding models on a managed platform with automatic judges, and shows why one metric is not enough.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• An embedding cache keys each vector on namespace plus a hash of the text, so re-indexing after an edit pays only for the changed chunks. Old entries stay until you clear them.
• Put the model name and dimensions in the cache namespace. The default empty namespace silently returns the old model's vectors after a model swap.
• Embed the corpus with batched embed_documents, not a loop of embed_query. The two paths differ by numerical noise, so never compare vectors for exact equality.