Step-back prompting

What you'll learn

• Explain why a specific question, especially one with a false premise, retrieves the wrong context.
• Trace step-back retrieval: a broader question, two searches, one combined context.
• Explain why two worked examples teach "make it broader" better than an instruction.
• Choose between step-back and plain retrieval by question type and cost.

Lessons 2 and 3 of T6.2 reworded the question to reach the handbook's vocabulary. Rewording keeps the question's scope. Some questions fail because the scope itself is wrong.

Notebook 04a opens with Did Leonardo da Vinci invent the printing press? A retriever searches for exactly that and mostly finds pages about Gutenberg, movable type and 1440. None of them says what Leonardo did invent, which is the knowledge that settles the question. The question carries a false premise, and searching on its words retrieves documents about the premise rather than the answer.

We call this the specificity trap: the narrower and more assumption-laden the question, the more its words steer retrieval toward the assumption.

Larkspur has the same shape of question. A customer on a monthly plan asks Can I get a refund on my monthly plan? The handbook offers refunds on annual plans only, and the answer the customer needs is "no, but cancelling stops the next charge". That second fact sits in the cancel chunk, which shares almost no words with the question.

Step-back prompting (Zheng et al., arXiv:2310.06117) asks an LLM for a more general question, retrieves for both the original and the general one, and answers from the combined context:

Figure 1: Step-back retrieval. Two searches, one for specifics and one for background.

The two contexts do different jobs. The original question's context supplies specifics; the step-back context supplies the background that can correct a premise. 04a feeds both, plus the original question, to a community answer prompt written to take exactly those three inputs.

Here are both questions for each example, with what each one surfaces:

Table 1: Original against step-back retrieval. The Leonardo rows are 04a's live web search; the Larkspur rows are the top 3 from the T5.1 TF-IDF index, with the step-back question written by hand.

On the Larkspur handbook the gain is small and honest. Both questions find refund-window, the chunk that states the annual-plan rule. The step-back question adds cancel, the fact the customer needs next, and the combined context holds five distinct chunks where the original held three. On a twelve-chunk handbook there is little background to miss; on the open web, 04a's search results move from Gutenberg to Leonardo.

"Make this question more general" is hard to specify in prose. Too loose, and the model rewrites the question into something unrelated; too strict, and it barely changes anything. 04a teaches the move with two worked examples instead:

Table 2: The two few-shot examples in 04a.

Both show the same pattern: a yes-or-no question about one specific thing becomes an open question about the category it belongs to. The few-shot template expands into real message pairs (a user turn, then the model's reply) before the real question, so the model imitates the pattern rather than interpreting an instruction.

The pattern can overshoot. We call that over-generalising: a step-back question so broad that it retrieves a vague overview instead of the background that matters. What does Larkspur offer? would be a step back from the refund question too, and it would retrieve almost anything.

Step-back doubles the calls of plain RAG:

Table 3: Plain against step-back RAG, from 04a's comparison.

The two retrievals are independent, so 04a runs them in parallel, and the extra latency is mostly one short LLM call.

Interview tip: step-back helps only when the corpus contains the broader background. On a narrow corpus, generalising returns the same documents, as the Larkspur rows of Table 1 nearly do.

1. Route yes-or-no policy questions here, especially ones that name a plan, a period or an amount the policy might not cover.
2. Write Larkspur few-shots: Can I pause my server for a month? becomes What can I change about a running server?
3. Keep the original retrieval. It supplies the specifics; the step-back retrieval only adds background.
4. Skip direct lookups such as What is the refund window?, where the original question already matches the chunk.

Lesson 5 of T6.2 attacks the other half of the asymmetry from lesson 1 of T6.2: not the question's scope, but its shape.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A specific question, especially one with a false premise, steers retrieval toward its assumption (the specificity trap): Leonardo's question finds Gutenberg.
• Step-back retrieves for the original and a broader question and answers from both. It costs 2 LLM calls and 2 retrievals, and helps only where the corpus holds the background.
• Teach the abstraction with two worked examples, and watch for over-generalising, a step-back question too broad to retrieve anything specific.