Query rewriting and expansion

What you'll learn

• Explain the two ways a question differs from the passage that answers it: vocabulary and shape.
• Trace how query expansion sends one text to the retriever and another to the answer prompt.
• Match each T6.2 transformation to the gap it closes and the calls it costs.
• Choose plain retrieval when no transformation can help.

T6.1 improved the retriever: two kinds of search, fused, filtered and de-duplicated. Every one of those techniques still searches with exactly what the user typed. T6.2 changes what is searched for.

A retriever compares the question with each chunk and returns the closest. That works when the two look alike, and a question rarely looks like its answer. In the T5.1 index, What happens if my server goes down? scores 0.000 against both chunks that answer it. Its top three are reboot (0.416), sizing (0.389) and export (0.217), all matched on the word "server". No value of k fixes that: the evidence is not low in the ranking, it is absent from it.

Query–document asymmetry is the name for this mismatch. It has two parts:

• Vocabulary: the user's words are not the handbook's words. The answering chunks say "outage", "availability" and "incident" and never "down".
• Shape: a question is short and interrogative; a chunk is long, declarative and dense with terms. Notebook 05a pairs a nine-word question, What is LangSmith, and why do we need it?, with a 400-token passage that answers it. Even with a good embedding model, the question's vector sits closer to other questions than to the passage.

Here are two Larkspur questions beside the chunks that answer them:

Table 1: Two questions and their evidence. The words barely overlap.

The second pair shows the vocabulary gap at its smallest. The T5.1 TF-IDF index does not stem, so "refund" in the question never meets "Refunds" in the chunk, and the timing chunk scores 0.000 too. Embeddings (T5.4) close part of this gap, because they place paraphrases near each other, and the rest of this module closes more of it at query time.

Query expansion closes the vocabulary gap with one LLM call. Notebook 01a asks the model to "expand the following query to improve document retrieval by adding relevant synonyms, technical terms, and useful context", and the expanded text goes to the retriever. The original question still goes to the answer prompt:

Figure 1: Expansion. The retriever sees the enriched text; the model answers the question the user asked.

The split matters. The expansion exists to find chunks, and an answer prompt built on it would answer a longer, slightly different question.

01a names the two costs. Every retrieval now waits for an LLM call. And expansion can dilute the query: each added term pulls the query vector toward more of the corpus, until it is near everything and close to nothing. Three or four well-chosen terms help; a paragraph of synonyms hurts.

The optional notebook 01b shows the neighbouring moves on one example: a rewrite that makes the query more specific, a step-back query that makes it broader, and decomposition into two to four sub-queries. Each gets its own lesson below.

Every technique in this module adds calls in front of retrieval, and each targets a different gap:

Table 2: The T6.2 transformations. Counts are per question, answer call included, from each notebook's own code.

HyDE's retrieval embeds the hypothetical answer instead of the question, at the same cost as embedding the question.

When a question arrives, a short chain of checks picks the transformation:

Figure 2: Pick a transformation by the question's gap. The lesson numbers in the boxes read 18.x; they mean lessons 2 to 8 of T6.2 of this module.

Plain retrieval is a valid exit. We call the opposite habit transform everything: running a technique on all traffic because it helped some of it.

Pitfall: transform everything. The T5.1 lab already retrieves both evidence chunks for Can I export my data after I cancel my plan? at k = 2. Every transformation on that question adds an LLM call and gains nothing. A router (lesson 7 of T6.2) makes the choice per question.

The three T5.1 golden questions each fall in a different box of Figure 2:

1. How long does a refund take? Short, and worded unlike the timing chunk. Variants or expansion (17.2, 17.3).
2. What happens if my server goes down? Worded unlike both evidence chunks; retrieval finds neither. Variants, with fusion to order the results (17.3).
3. Can I export my data after I cancel my plan? Already retrieved at k = 2. Plain retrieval.

Lesson 2 of T6.2 starts on the first two with the simplest variant technique: ask the question three ways.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A question rarely looks like its answer: it differs in vocabulary (the user's words against the handbook's) and in shape (short and interrogative against long and declarative).
• Expansion sends an LLM-enriched query to the retriever and the original question to the answer prompt. It costs one LLM call per retrieval, and too many added terms dilute the query.
• Match the transformation to the gap, and keep plain retrieval as an exit. Running one technique on all traffic (transform everything) adds cost where nothing is missing.