Query decomposition for multi-hop questions

What you'll learn

• Explain why a compound question's single vector can miss one of its needs.
• Trace a three-part Larkspur question through decomposition, sub-answers and synthesis.
• Count the calls decomposition spends, and recover the wall-clock time.
• Diagnose the list explosion that turned three sub-questions into eighteen.

Lessons 2 to 5 of T6.2 all produced better search keys for one need. Some questions carry two or three needs, and a better key for the whole question is still one key.

A Larkspur customer writes If I cancel my annual plan, do I get a refund, when does the money arrive, and how long is my data kept? Three chunks answer it: refund-window, refund-timing and retention. Dense or sparse, retrieval gives the whole question one vector, which lands between the needs. With a top-k cut, the strongest needs crowd out the weakest. We call this the between-needs vector.

Here is the single query against three sub-questions on the T5.1 TF-IDF index:

Table 1: One compound question against its three sub-questions (T5.1 index, TF-IDF; sub-questions written by hand in the style of the 06a prompt).

The timing need loses inside the compound question because its words ("when", "money", "arrive") are few and common, while "cancel", "plan" and "data" pull toward cancel and retention. Asked alone, each need gets the whole query to itself.

Query decomposition splits the question before retrieval and answers each part on its own:

Figure 1: Decomposition. Each sub-question is a small RAG chain; the synthesis sees only their answers.

The synthesis prompt sees the three question-and-answer pairs, never the retrieved chunks, so its context stays small however many sub-questions there were. Notebook 06a asks What is LangSmith, and why do we need it?, a definition plus a justification. It splits its corpus into 100-token chunks so that each holds about one idea, and counts the unique chunks the three sub-questions reach against the single query's.

Decomposition is the most expensive technique in this module:

Table 2: Calls for three sub-questions, against 1 and 1 for plain RAG.

The sub-questions are independent, so 06a runs their chains as one parallel batch, which recovers most of the wall-clock time at the same token cost.

Errors propagate. A bad sub-question produces a confident wrong sub-answer, and the synthesis has no chunks to check it against. We call this the unchecked sub-answer. Log every sub-question and sub-answer, so a wrong final answer can be traced to the part that caused it.

The first version of 06a asked for prose and split the reply on newlines. The prompt asked for sub-questions and then for "multiple search queries", and the model returned both under markdown headers. The split produced about 18 items: headers, blank strings and bullets, each triggering a retrieval and an LLM call:

Table 3: The list explosion in the first version of 06a.

We call this the list explosion. It came and went, because the same chain returned three clean lines on other runs. The fix is a prompt that asks for one thing and a schema bound with structured output, which returns a list of exactly the sub-questions.

Pitfall: list explosion. Any chain whose call count depends on parsed LLM output needs a schema and a cap. Log the count per request; a jump from 5 calls to 20 is the signature.

T5.1's symptom table sends "half an answer; one fact found, another missed" to decomposition. Not every two-part question needs it:

1. Decompose when a single query misses a need. The three-part cancellation question above loses refund-timing; decomposition finds all three.
2. Skip it when the single query already covers both parts. Can I export my data after I cancel my plan? reads like two needs, yet the T5.1 lab retrieves both evidence chunks at k = 2, because one chunk says "Export your data" and the other "After a plan is cancelled". Decomposing it costs four extra LLM calls for nothing.
3. Find the difference on the golden set, and let a router (lesson 7 of T6.2) send only the compound questions that need it.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• One vector for a compound question lands between its needs (the between-needs vector): the Larkspur cancellation question finds 2 of 3 evidence chunks; its three sub-questions find all 3.
• Three sub-questions cost 5 LLM calls and 3 retrievals. Run them in parallel, and log sub-answers, because a wrong one reaches the synthesis unchecked.
• Get the sub-questions from a structured-output schema; splitting prose on newlines produced about 18 queries instead of 3.