Open-source embeddings with Hugging Face

What you'll learn

• Explain what domain fit means for an embedding model, and how a poor fit shows up.
• Compare open models run locally with hosted ones by cost, data, latency and operations.
• Diagnose an open model that ranks badly because its vectors are not normalised.
• Design a golden set with hard negatives that can separate candidate models.

Lesson 2 of T5.4 sized the hosted OpenAI models. Open models run on your own machines, which widens the choice, and makes the question of which one fits your documents harder to skip.

A Larkspur customer asks How much bandwidth does the Business plan include? The handbook never says "bandwidth". It says "outbound traffic". If the embedding model does not place those two phrases close together, the plans page loses to whatever chunk happens to mention networks, and the answer comes from the wrong page. Nothing raises an error.

A model has domain fit when texts that mean the same thing in your documents land close together and texts that differ land apart. General-purpose models are trained on broad web text. They can place two Larkspur chunks that differ by one condition (a full refund within 30 days, a prorated one after) almost on top of each other, miss a customer's synonym for a handbook term, or fail on a language they were not trained on. We call any of these a domain miss: it shows up only as the wrong chunk ranked first.

The Hugging Face wrapper runs sentence-transformers models on your own machine, with no API key. The models the lesson's notebooks load:

Table 1: Open embedding models used in the lesson's notebooks.

Two settings decide whether a local model behaves like the hosted ones in lessons 1 and 2 of T5.4:

• Normalisation is off by default. Turn it on in the wrapper's encode settings, or score with cosine. Otherwise dot product rewards longer vectors, as the lesson 1 of T5.4 lab shows. We call a local model left this way an unnormalised local model.
• Some models are asymmetric. They are trained so that short questions and long passages meet, and they treat the two sides differently: embed_query and embed_documents can return different vectors for the same text. Keep the two calls separate in your code even when today's model is symmetric.

Hosted or self-hosted is part of the fit, because some corpora may not leave your network. The trade, from notebook 1 of T5.4:

Table 2: Hosted against self-hosted embeddings.

Leaderboards such as MTEB (the Massive Text Embedding Benchmark) average many public datasets. They are good for a shortlist. The decision comes from scoring the shortlist on your own documents and questions, and a short decision chain narrows where to start:

Figure 1: Building a shortlist. Every branch ends in the same scoring on your golden set.

Every shortlist then goes through the same scoring on the golden set: recall@k, margin, cost and latency. A useful golden set for Larkspur holds the T5.1 questions plus hard negatives: chunks on the same topic that do not answer the question, such as cancel for Can I export my data after I cancel my plan? Notebook 1 of T5.4's exercise asks for exactly this, because a six-document toy set separates no models. We call a set without them an easy golden set: every candidate scores well, and the comparison decides nothing.

Score more than one thing, too. Lesson 5 of T5.4 runs three open models through two automatic judges on a managed platform, and the two judges disagree about which model is best.

For the handbook, choosing an open model takes four steps:

1. Shortlist all-MiniLM-L6-v2, all-mpnet-base-v2 and bge-large-en from Table 1, plus the hosted model from lesson 2 of T5.4 as the baseline.
2. Turn on normalisation for every open model, so the metric from lesson 1 of T5.4 behaves.
3. Add hard negatives to the golden set: cancel for the export question, refund-window for the refund-timing question, and a "bandwidth" paraphrase for the plans page.
4. Score recall, margin, latency and cost for each, and keep the documents on Larkspur's machines if policy requires it.

Next: lesson 4 of T5.4 turns to running embeddings in production: batching the calls and caching the results.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Poor domain fit raises no error; it shows up as the wrong chunk ranked first. Use leaderboards for a shortlist and your own golden set, with hard negatives, for the decision.
• The Hugging Face wrapper does not normalise by default. Turn normalisation on or score with cosine, and keep embed_query and embed_documents separate for asymmetric models.
• Hosted models cost per token and send data off your network; self-hosted ones cost hardware and operations. Decide which metric matters before you compare results.