Reasoning prompts: chain-of-thought, self-consistency, least-to-most, ReAct

What you'll learn

• Explain how chain-of-thought and least-to-most prompts change what comes before the answer.
• Diagnose a few-shot reasoning prompt whose worked examples contain a wrong step.
• Predict when a self-consistency vote raises accuracy and when it locks in a wrong answer.
• Describe the ReAct loop of thought, tool call and observation, and what actually stops it.

Lesson 1 of T3.1 shaped the reply with examples, personas and templates, each asking for an answer straight away. Some questions need intermediate steps first, and some need facts the model does not have.

A Larkspur customer asks Can I get a refund after 45 days? An assistant that answers in one breath tends to grab the nearest rule it remembers, the 30-day full refund, and reply "No". The handbook has more to say. Monthly plans are never refundable. An annual plan gets a full refund within 30 days, and after day 30 it gets the unused whole months back, minus a 15% early-termination fee. The right answer depends on the plan and on the start date, and the start date is not in the handbook at all.

A chain-of-thought (CoT) prompt asks the model to write its reasoning before the answer. The answer is then conditioned on the steps, and the steps are on the page for you to check.

Notebook 02a uses one test question for every variant: from a game summary, How many yards was the shortest valid field goal? The summary has a 35-yard field goal and a 31-yard one that was blocked. The right answer is 35; a model that skips the word "blocked" says 31.

The notebook asks for the reasoning in three ways:

Table 1: The three ways notebook 02a asks for reasoning.

Planted failure: the copied mistake

In few-shot CoT the model copies the reasoning of the examples as well as their format, so every step of every example has to be right. Two examples in notebook 02a are not:

Table 2: Wrong steps inside the notebook's worked examples (checked by hand against each example's own text).

The field goal example teaches the exact mistake the test question checks for: counting a blocked kick as valid. We call this the copied mistake, and the only guard is to check each worked step, not only each final answer.

Least-to-most on the refund question

Least-to-most fits Larkspur questions that combine rules. Can I get a refund after 45 days? breaks into four subproblems, each answered in order:

Figure 1: The refund question as a chain of subproblems. Each answer feeds the next.

Each subproblem has its own source:

Table 3: Least-to-most decomposition of the refund question.

Subproblems 1 and 4 cannot be solved by reasoning alone. They need a lookup, and a reasoning prompt has no way to make one. ReAct, at the end of this page, adds that.

Self-consistency samples several chain-of-thought paths for the same question and keeps the answer most of them reach (Wang et al., 2022). One path can slip; if the slips land on different answers, the right answer still collects the most votes.

Notebook 02a builds it from three CoT chains run in parallel, all gpt-4o-mini, at temperatures 0, 0.5 and 0.9. A fourth call at temperature 0 reads the three paths and is told to aggregate, take the majority vote as needed:

Figure 2: Self-consistency in notebook 02a: three sampled paths and an LLM that merges them.

That is four calls where plain CoT makes one. The aggregator is itself a model call, and it can be talked round by a confident wrong path. For short answers that compare as strings or numbers, counting the votes in code does the same job with one call fewer.

When the vote helps

Whether the extra calls pay depends on how the wrong answers are spread. The interactive lab computes this exactly. It treats each path as an independent draw that is right with probability p and otherwise gives one of W equally likely wrong answers, then counts how often the most common answer is the right one, with ties split at random:

Table 4: Vote accuracy from the lab (values the lab prints at these settings, checked against a Python port of its formula).

The curves for a path that is right 40% of the time:

Figure 3: One accuracy, two outcomes. Scattered errors lift the vote; a single shared error sinks it.

Three rules follow from the table:

• A vote amplifies whatever answer is most common. If one path is right more often than any single wrong answer, more paths help. If not, the vote makes the wrong answer look certain. We call this the confident wrong vote: a yes/no grader that is right 40% of the time drops to 0.213 with 15 votes.
• Scattered errors are where voting pays. With four distinct wrong answers, a path that is right only 40% of the time still wins the vote 78% of the time at N = 15, because no single wrong answer gathers enough votes. A computed figure such as a prorated refund behaves like this; a yes/no grade never does.
• On two labels, use an odd N. An even N can split evenly, and a split is a coin flip. N = 4 scores exactly what N = 3 scores (0.648 at p = 0.60) for one more call.

The lab's independence assumption is the best case. Paths from one model share its blind spots and agree on the same wrong answer more often than independent draws, which shrinks the gain; a path at temperature 0 repeats the most likely path rather than sampling a new one. For a measured case where the wrong answers clustered on one value, see T8.1's self-consistency page.

ReAct prompts the model to alternate reasoning with actions (Yao et al., 2022): a Thought, an Action with its input, the Observation the action returns, and another Thought, until it can write a Final Answer.

Notebook 02a puts that format in its system prompt, one labelled line per part:

Table 5: The ReAct format as notebook 02a's system prompt states it.

The loop those lines describe:

Figure 4: The ReAct loop. Each Observation goes back into the next Thought until the model answers.

What the framework actually runs

The notebook's tool, search_web, takes the top five DuckDuckGo results, fetches each page and returns its text. The agent is gpt-4o built with create_tool_calling_agent and run by AgentExecutor. That combination changes what the format means:

• The Action is a tool call, not text. The model is given the tool's schema and returns a structured tool call with the tool name and arguments. The agent runs it and feeds the result back as a tool message. Nothing parses an Action: line, and the PAUSE the system prompt mentions has no effect.
• The Thought lines are guidance. They ask the model to plan and to check the Observation before answering. The model may write them, or may call the tool straight away.
• The loop needs a stop. AgentExecutor ends the run when the model replies without a tool call, or after max_iterations (15 by default).

Each of those rules shows up as a failure you can plant:

Table 6: Planted ReAct failures and the rule behind each (behaviour read from the installed agent code and the notebook's tool; no model call).

We call the second case the runaway loop. It usually means the evidence is not where the agent is looking, and the right end is an "I don't know", not a sixteenth search.

The notebook first asks about LangGraph with no tool, and its own comment expects the model to decline or make up an answer; the agent then answers from the pages search_web fetched.

For Larkspur the tool is a handbook search. Can I export my data after I cancel my plan? takes one round: the search returns the export and retention chunks, and the answer is to export from Settings, then Data, within 30 days of cancelling. The refund question takes two tools, the handbook for the rules and the customer's billing record for the plan and purchase date. T7.1 and T7.2 build this loop in LangGraph.

The cheapest prompt that covers the question wins:

Figure 5: Pick a reasoning prompt by what the question needs.

Table 7: The four reasoning prompts compared.

1. Wang, X. et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171.
2. Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Chain-of-thought puts the reasoning on the page before the answer. In few-shot form the model copies the examples' reasoning, so check every step of every example, not only the final answers.
• Self-consistency votes over several paths and amplifies the most common answer. It pays when errors scatter, as computed figures do, and makes a weak yes/no grader worse.
• In a tool-calling agent the action is a structured tool call, a reply with none ends the run, and an iteration limit stops a runaway loop. Use ReAct when a subproblem needs a lookup that reasoning alone cannot supply.