Semantic retrieval with embedding similarity

What you'll learn

• Explain how an embedding model scores a chunk against a question by the angle between their vectors.
• Compare embeddings with TF-IDF on the three Larkspur golden questions.
• Diagnose why a semantic retriever can rank a related chunk above the one that holds the answer.
• Choose a score threshold from the golden set instead of a fixed number.

Lesson 2 of T5.1 left two questions that no k could fix. The refund-timing chunk scored zero because "Refunds" is a different token from "refund", and the outage evidence scored zero because the customer wrote "server goes down" where the handbook says "outage". Both are representation problems: TF-IDF vectors only count shared words.

An embedding model turns a text into a fixed-length list of numbers, a vector. Texts with similar meaning get vectors that point in similar directions, even when they share no words. Retrieval with embeddings has the same shape as the TF-IDF index; only the vectors change:

Figure 1: Semantic retrieval. Chunks are embedded once at indexing time. The question is embedded when it arrives.

Each chunk is scored by the cosine of the angle between its vector and the question's. Libraries report that comparison in one of two forms:

Table 1: Two forms of the same comparison. Notebook 03a's nearest-neighbour search prints the distance.

When every vector has length 1, the cosine is the plain dot product. OpenAI's embeddings come normalised that way, and sentence-transformers returns them so when asked to normalise.

The question and the chunks must go through the same model. Two models produce vectors in different spaces, so the cosine between a vector from one and a vector from the other means nothing. We call that a space mismatch. Notebook 03a avoids it by sending both sides through one embedding function, and it follows that changing the embedding model means re-embedding the whole corpus (T5.4).

The comparison below uses the 12 Larkspur chunks and three golden questions from the lesson 2 of T5.1 lab. It embeds every chunk and question with all-MiniLM-L6-v2, a small model (384 numbers per vector) that runs locally, so anyone can reproduce the scores. Notebook 03a uses OpenAI's text-embedding-3-small, whose scores will differ.

With two evidence chunks per question, recall moves in steps of 0.50:

Table 2: Recall at k = 2 and k = 3 for TF-IDF and embeddings, with the embedding scores of the top three chunks.

The three questions move in three directions:

• Word-form miss, fixed (refund): the embedding model places "Refunds" and "refund" close together, and refund-timing, which scored 0 under TF-IDF, ranks first.
• Vocabulary gap, narrowed (outage): both evidence chunks now reach the top three, though reboot still ranks first because rebooting a server is close in meaning to a server going down. The top three scores sit within 0.137 of each other, so a small change to the question can reorder them.
• Topic drift (export): embeddings lose a question TF-IDF answered. The question is half about cancelling, so the cancel chunk (0.561) outranks export (0.479). TF-IDF matched the literal word "export" and found both evidence chunks at k = 2.

Topic drift is the general weakness. Embeddings measure how close two texts are in topic, which is a different thing from whether a chunk contains the answer. The two retrievers suit different questions:

Table 3: Which retriever wins, by what the question and the evidence have in common.

Hybrid search combines the two scores. A reranker reorders the candidates with a model that reads the question and each chunk together.

Pitfall: the fixed-threshold trap. Cosine scores are not probabilities. The best outage match scores 0.394, while the best export match scores 0.817 on the same model. A rule such as "drop chunks below 0.5" sends the outage question to the model with no context at all.

Moving the handbook index from TF-IDF to embeddings takes four steps, and the last one decides whether the change stays:

1. Re-embed all 12 chunks with one model, and send every question through the same model.
2. Rerun the golden set at k = 2. Mean recall@2 rises from 0.50 under TF-IDF to (1.00 + 0.50 + 0.50) / 3 = 0.67. At k = 3 it reaches 1.00, against 0.50 for TF-IDF.
3. Check every question, not only the mean. The export question fell from 1.00 to 0.50 at k = 2. Lesson 5 of T5.1's rule keeps a change only when it breaks nothing. This one needs k = 3 or hybrid search before it can become the baseline.
4. Set any score threshold from the golden set. It has to sit below 0.394, the outage question's best match, and it has to be re-checked whenever the model changes.

Next: lesson 4 of T5.1 replaces the hand-built index with a LangChain vector store, which keeps the embedding model and runs the search for you.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Embeddings score chunks by the cosine between their vectors, which lets paraphrases and word forms match. Embed questions and chunks with the same model. Changing it means re-embedding the corpus.
• Embeddings win on paraphrase and TF-IDF wins on rare exact words such as product names or error codes. Topic closeness is not the same as containing the answer.
• Cosine scores are not probabilities and vary by question. Set any threshold per corpus and model from the golden set.