Self-consistency, tree of thoughts and LATS

What you'll learn

• Compare where self-consistency, tree of thoughts and LATS spend their extra LLM calls.
• Diagnose a majority vote that locks in a systematic error.
• Explain why legal moves and two-number states are judged in code, not by the model.
• Choose whether a Larkspur question deserves tree search at all.

Lesson 5 of T8.1 spent extra calls on one better path: a reasoning structure before the answer, or a critic after it. This lesson spends them on several paths at once and keeps the best.

A single chain of thought is one sample from many possible reasoning paths. Sampling more answers helps only if you can pick a better one, and the three patterns here differ in where they spend the extra calls and in what does the picking. The notebooks test them on arithmetic, where every answer can be checked in code. The question for Larkspur is whether its support questions have anything like that check.

Table 1: Three ways to spend extra calls on more reasoning paths.

Tree of thoughts searches over intermediate reasoning states, scoring each and expanding the best. LATS (language agent tree search) does the same search one leaf at a time with Monte Carlo tree search. Both rest on the idea self-consistency tests first: whether more paths plus a selector beat one path.

When agreement hides a shared error

Self-consistency needs short, comparable answers (a number, a label) and nothing else. The notebook samples 5 answers at temperature 0.7, three times per question, and votes. Its prototyping runs, 20 samples per question, show both sides of the vote:

Table 2: Random and systematic errors under a majority vote. Right answers computed in code, rates from the notebook's recorded runs.

We call the second row the confident wrong majority: the samples agree, and they agree on the wrong answer. High agreement is a cheap confidence signal, and consistent is not the same as correct.

Tree of thoughts on the Game of 24

Tree of thoughts (ToT) spends its calls on intermediate steps instead of whole answers. The notebook uses the Game of 24, the benchmark from the ToT paper: combine four numbers with addition, subtraction, multiplication and division to make 24. A state is the numbers still in play, and a thought is one move. Four pieces make the search:

• Rules in code. Moves are applied with exact fractions, and an illegal or mis-computed proposal is dropped and the tree holds only legal states.
• A proposer. The model, at temperature 0.7, suggests up to PROPOSALS = 8 moves per state.
• An evaluator. With three numbers left, the model at temperature 0 judges sure, likely or impossible (scores 3, 1 and 0). With two left, code checks the six possible operations exactly.
• Beam search. An expand and evaluate loop in LangGraph keeps the best BEAM = 5 states per level, and a pure router stops at a solution, an empty frontier or the depth limit.

The two-number rule exists because of a planted failure found while prototyping. The model judged the state with 4 and 24 left as sure, because 24 was in the list. Exact arithmetic says otherwise:

Table 3: Every operation on the state with 4 and 24 left, computed with exact fractions. None gives 24, which makes the state a dead end.

Both remaining numbers must be used in one legal operation, and the target appearing among the inputs is not enough. We call this the lucky-looking state, and it is why code judges every state it can decide: a model saying a state is promising cannot override arithmetic.

In the recorded runs, single-pass chain of thought solved between a twentieth and a quarter of the samples. ToT solved about half of the eight puzzles at roughly 10 to 15 LLM calls per puzzle, and every returned solution is checked in code. A search fails in two ways: a solvable state is scored impossible and pruned, or the proposer never generates the move the solution needs.

LATS: one leaf at a time

LATS replaces the level-by-level beam with Monte Carlo tree search, one leaf per iteration:

Figure 1: One LATS iteration. Rewards flow back to the root, and dead ends become lessons for later proposals.

Selection walks down from the root, at each level taking the child with the best UCB1 score: its average value (total value divided by visits) plus an exploration bonus of c times the square root of ln(parent visits) divided by the child's visits. An unvisited child scores infinity, which tries every child once, and a child visited often earns a smaller bonus. The bonus is what stops the search from only ever deepening its current favourite.

The notebook's settings:

Table 4: The LATS configuration in the notebook.

The second expansion carries the latest reflections in its prompt. Each dead end becomes a one-sentence lesson that later proposals read, which is Reflexion from lesson 3 of T8.1 used inside a search.

On these eight puzzles LATS did not win. The recorded runs show it solved 2 to 4 of 8, the ToT beam about 4, and LATS spent more LLM calls. Its solved runs finished in two or three iterations, because it stops at the first success. LATS was designed for problems with a larger branching factor and a better value estimate than eight small puzzles offer.

All three patterns multiply calls, and each needs something to select with. The decision for any task comes down to the selector:

Figure 2: Search only where an evaluator or a vote can pick the better path.

For a Larkspur support assistant most questions have one good path, and plan-and-execute (lesson 1 of T8.1) or chain-of-verification (lesson 4 of T8.1) are the tools to reach for. Reach for these patterns when one path is not reliable enough and something can score a partial answer: a code check, a test suite or a judge you have measured. A sizing calculation with a checkable result could qualify. A refund question answered from one handbook chunk does not.

Without that evaluator, more branches only multiply the calls.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Self-consistency votes over N sampled answers and needs short, comparable answers. It washes out scattered errors but locks in systematic ones. High agreement signals confidence and says nothing about correctness.
• Tree of thoughts proposes several moves per state, scores them and keeps the best BEAM per level. Enforce the rules in code and let code judge whatever it can decide exactly.
• LATS picks one leaf at a time by UCB1 and learns from dead ends, yet on the Game of 24 it did not win. Use these searches only when something reliable can score a partial answer.