Embeddings & Model Selection: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur's support assistant was fine on Friday. On Monday every answer is vague, and the logged top-1 similarity for golden questions has fallen from its usual range to near zero. Nobody touched the prompt. What do you check first?

What a strong answer covers:

• Suspects a model mismatch before a quality problem: scores collapsing toward zero on questions that clearly have answers is the signature
• Compares the model name and dimensions the query service uses with the ones recorded in the index metadata; matching dimensions prove nothing, since 3-large truncated to 1536 has the same shape as 3-small
• Checks the usual causes: config read from two places, an unnamespaced embedding cache, a provider-side model update
• Adds a startup check that refuses to serve when the query model differs from the index model, and keeps the golden-set run in the deploy pipeline

2. The handbook must stay in the EU, and the team proposes replacing text-embedding-3-small with self-hosted all-MiniLM-L6-v2. What do you need to know before you agree?

What a strong answer covers:

• Data residency may decide it, but the quality cost still has to be measured: 384 dimensions against 1536, on the T5.1 golden set plus hard negatives such as the two refund chunks
• Checks normalisation: sentence-transformers does not normalise by default, so either pass normalize_embeddings=True or make sure the store uses cosine
• Counts the operational cost: serving, scaling, GPU or CPU capacity, model versioning, in exchange for no per-token bill and no provider deprecations
• Plans the migration as a full re-index into a new collection, with a new cache namespace, switched over only after the golden set passes

3. Storage for the vector index is the biggest line on the bill. Someone suggests dimensions=256. How do you decide?

What a strong answer covers:

• Computes the saving first: 256 against 1536 dimensions is one sixth of the raw vector storage and similarity work
• Measures recall@k and the margin between the correct chunk and the best wrong one at each candidate size; recall saturates on easy sets while the margin keeps shrinking
• Tries 3-large truncated to the same size as an alternative: a stronger model at the same index cost
• Remembers that truncation is a re-index, so it is decided once on real traffic samples, not tuned weekly

4. A new embedding model goes live. How would you notice in production, within a day, that it retrieves worse than the old one?

What a strong answer covers:

• Keeps the old index serving in shadow and logs both top-k lists for a sample of real queries
• Tracks the top-1 score distribution and the share of queries under a score floor, recalibrated for the new model because score ranges differ between models
• Runs the golden set and an LLM-judged sample of live traffic on both, comparing recall and answer faithfulness
• Watches slices separately: other languages, product names and numeric questions can regress while the average improves

5. The vector store is configured for dot product, and the embedding model is a sentence-transformers model loaded with default settings. Retrieval keeps returning the longest pages in the handbook. Explain the bug and the fix.

What a strong answer covers:

• Without normalisation, vector length varies, and dot product scores length as well as direction; cosine scores direction only
• Confirms it by printing the norms of the stored vectors, which a normalised model keeps at 1.0
• Fixes it by normalising at embedding time (encode_kwargs={"normalize_embeddings": True}) or by switching the store to cosine, then re-indexes
• Adds a test that asserts unit norms on a sample of stored vectors, so a config change cannot bring the bug back