Embedded stores: Chroma and FAISS

What you'll learn

• Explain how exact search, HNSW and IVF find nearest neighbours, and what each gives up.
• Tune the HNSW and IVF settings that trade recall against speed and memory.
• Diagnose a filtered search that returns nothing although matching entries exist.
• Choose between Chroma and FAISS for an index you run in-process.

Lesson 1 of T5.5 treated the store as a box that returns the nearest k. This lesson opens the box.

Larkspur's index grows from the 12-chunk handbook to years of tickets and runbooks, and the team switches to an approximate index to keep search fast. An approximate index can hand the refund question the next-best chunk instead of refund-timing, with a normal-looking score and no error.

Exact nearest-neighbour search compares the query with every stored vector. FAISS.from_documents builds exactly that, an IndexFlatL2: the answer is always the true top k, and the cost grows with every vector you add. At the size of the T5.5 notebooks (a handful of chunks) exact search is instant, which is why none of them shows a difference between exact and approximate results.

Approximate nearest-neighbour (ANN) indexes give up the guarantee to visit only a small part of the collection. The two families you meet in practice:

• HNSW (hierarchical navigable small world) links each vector to its near neighbours in a graph, and keeps sparser copies of the graph in layers above it. A search enters at the top, walks greedily towards the query, and drops a layer each time no neighbour is closer. Chroma's local collections are HNSW indexes, and pgvector offers it as an index type.
• IVF (inverted file) clusters the vectors into lists around centroids, then searches only the lists whose centroids are nearest the query. pgvector calls the cluster count lists and the number searched probes; FAISS calls them nlist and nprobe.

Figure 1: An HNSW search enters at the top layer, walks greedily towards the query and ends at layer 0 (schematic).

Each knob moves recall against speed or memory:

Table 1: The HNSW and IVF settings, with Chroma's and pgvector's defaults.

The Chroma values are what collection.configuration_json reports for a new collection in the course environment, and the pgvector values are from the pgvector README. Defaults are a starting point. To tune them, measure recall@k of the ANN index against an exact search on your own queries, and raise ef_search until the misses stop mattering for the golden set.

An ANN miss raises no error, and we call it a silent ANN miss. An index that skips the refund chunk returns the next-best chunk with a normal-looking score, so a recall drop after the corpus grows shows up as worse answers, not as an error.

Both run inside your Python process with no server. They differ in what they do beyond the search:

Table 2: Chroma and FAISS compared.

The filter difference shows up as missing results. In a check run for this page, a FAISS store held 25 chunks tagged t="no" that sat close to the query and 3 tagged t="yes" further away. similarity_search(q, k=3, filter={"t": "yes"}) returned 0 results, because none of the 3 was among the 20 nearest candidates; with fetch_k=28 it returned all 3. Chroma, asked for k=5 with a filter only 2 entries match, returned those 2. We call the FAISS case a starved post-filter: the filter runs on too few candidates. The same check in table form:

Table 3: A planted selective filter against both stores, from a check run for this page.

The notebooks' guidance: start with Chroma for anything you might keep, choose FAISS for a static corpus you search often, and move to a hosted store (lesson 3 of T5.5) when you need concurrent writers or more than one machine.

For the handbook on one machine, the choices follow from Tables 1 to 3:

1. Use Chroma for the handbook index, because it is updated nightly and filters inside the store.
2. Measure recall@k of the HNSW index against an exact search on the golden set once the corpus grows, and raise ef_search until the misses stop mattering.
3. Raise fetch_k, or filter elsewhere, for any selective filter on a FAISS archive.
4. Move to a hosted store only when a second writer or machine arrives (lesson 3 of T5.5).

Next: lesson 3 of T5.5 moves the index off your machine, to managed stores, and looks at what decides which entries one query can see.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• ANN indexes such as HNSW and IVF trade the guaranteed top k for speed, and a miss raises no error. Measure recall@k against exact search and raise ef_search until misses stop mattering.
• Chroma filters inside the store before ranking; the FAISS wrapper filters after ranking fetch_k=20 candidates, so a selective filter can return nothing.
• Start with Chroma for anything you will update, use FAISS for a static read-heavy corpus, and move to a hosted store for concurrent writers or several machines.