Your first LangChain RAG with Chroma

What you'll learn

• Explain which steps of the hand-built index a LangChain vector store takes over.
• Diagnose a duplicated index, a lost index and an unset k from what retrieval returns.
• Explain why Chroma's default distance breaks relevance-score thresholds.
• Choose the distance metric when you create a collection, because it cannot change afterwards.

Lessons 2 and 3 of T5.1 built the index by hand: embed each chunk, keep the vectors in an array, fit a nearest-neighbour search, embed the question, look up its neighbours. This lesson hands that work to a library.

A teammate reruns the indexing cell to pick up an edit to the handbook, then asks How long does a refund take? The top three comes back with refund-timing twice. Nothing raised an error, and the index now holds every chunk twice.

Notebook 04a does the hand-built work through two LangChain interfaces, Embeddings and VectorStore. Each step maps onto one call:

Table 1: The hand-built index from lesson 3 of T5.1 and its LangChain equivalent.

The calls run in this order:

Figure 1: Building and querying a Chroma store. The store holds the embedding object and calls it on both sides.

Because the store keeps the embedding object, the question always goes through the same model as the chunks, and the space mismatch from lesson 3 of T5.1 cannot happen by accident. Embeddings has separate methods for documents and queries because some models encode questions and passages differently. For OpenAI's models the two do the same thing: the query method embeds a one-item list of documents.

Three defaults in notebook 04a surprise people the first time. Each one fails without an error:

Table 2: Three defaults and the failure each one causes.

The second row is silent duplication: every chunk counted twice, so duplicates crowd evidence out of the top k. It is the failure in the opening example.

The notebook imports Chroma from the community integrations package, which now warns that it is being sunset. This repo depends on langchain-chroma, which provides the same class.

Chroma ranks by distance, and its default metric is squared Euclidean (L2) distance. Lower numbers from similarity_search_with_score mean closer chunks. For vectors of length 1 the two measures are tied by distance = 2 - 2 × cosine, so L2 and cosine rank chunks in the same order. On the 12 Larkspur chunks, embedded with all-MiniLM-L6-v2 as in lesson 3 of T5.1 (langchain-chroma 1.1.0, chromadb 1.5.9):

Table 3: The same chunks scored three ways under Chroma's default L2 space.

The relevance-score method converts L2 distance with 1 - distance / √2, which assumes a range the distances do not keep to:

Figure 2: The relevance conversion on the outage question. Any distance above √2 becomes a negative score.

The sla chunk, which holds the evidence for the outage question, gets a negative relevance score, and LangChain warns that relevance scores must be between 0 and 1. A retriever with a similarity-score threshold of 0.2 returns no chunks at all for the outage question, although all three top chunks have a cosine above 0.25. We call this a metric mismatch: a threshold written for one scale applied to scores from another.

Pitfall: metric mismatch. Choose the metric when you create the collection, by setting its HNSW space to cosine in the collection metadata. Then similarity_search_with_score returns 1 - cosine (0.606 for reboot), the relevance score is the cosine itself (0.394), and the same 0.2 threshold returns reboot, incidents and sla. The ranking does not change.

The metric is fixed when the collection is created. Changing it means building a new collection (T5.5).

For the handbook, each guard from this page becomes one decision made before the first run:

1. Name the collection handbook and pass each chunk's id (refund-timing, sla, and so on), so a rerun overwrites instead of duplicating.
2. Set the space to cosine at creation. Relevance scores are then cosines, and thresholds behave.
3. Set a persist directory if the index must survive a kernel restart.
4. Set k = 2 from lesson 2 of T5.1, and check that a 0.2 threshold still returns all three outage candidates.

Next: lesson 5 of T5.1 puts the store into the full lifecycle, adds the generation step, and records the baseline every later module must beat.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A LangChain vector store keeps the embedding object and embeds the question for you, so both sides always use the same model.
• Set k explicitly, pass ids= so a rerun overwrites instead of duplicating, and set persist_directory if the index must survive a restart.
• Chroma ranks by squared L2 distance by default, which breaks relevance-score thresholds. Create the collection with cosine space if you plan to filter on scores.