Reasoning vs non-reasoning models

"Think step by step" is a prompt: you ask an ordinary model to write its working into the answer, and you can read every word. A reasoning model is trained to work before it answers. It produces reasoning tokens internally, decides how many to spend, and returns only the conclusion. The notebook sums up the difference in three rows:

The notebook runs the same prompt through gpt-4o-mini (fast) and o4-mini (reasoning) and records four numbers per call: seconds, output tokens, reasoning tokens and whether the answer was right. Its PRICING constants, which it tells you to replace with today's prices, are USD per million tokens:

At those prices o4-mini costs 7.3 times as much per token, in and out, before it spends a single reasoning token. The two tasks are chosen to land on opposite sides of the decision:

• Task A, a constraint puzzle. Five services deploy Monday to Friday; auth before billing, search the day after billing, notify on Monday, reporting not on Friday. Which deploys on Friday? Only search fits, but getting there means holding four constraints at once, and a wrong early step gives a confident wrong answer. The notebook warns that the fast model is often right only some of the time, so a single happy-path test can pass it.
• Task B, an extraction. Pull order_id and total out of "Order #A-4417 shipped 2026-03-02, 3 items, total $148.20, customer 88213" as JSON. The answer is in the input and both models should get it. The reasoning model is slower and dearer, and still spends reasoning tokens deciding there was nothing to decide.

completion_tokens is what you are charged for at the output rate. reasoning_tokens, under usage.completion_tokens_details, is the part of it you never receive. The notebook's cost function makes that explicit:

# reasoning tokens are already inside completion_tokens; they bill at the output rate
cost = (prompt_tokens / 1e6) * input_price + (completion_tokens / 1e6) * output_price
visible = completion_tokens - reasoning_tokens

On a short answer the hidden part is almost all of the bill. The figure prices one o4-mini call on the first problem of lesson 4 of T2.2: 38 prompt tokens counted with o200k_base, a three-token answer (12:25), and 700 reasoning tokens, the lab's representative medium setting.

Two habits follow:

• Never estimate spend from the reply text. Priced from its prompt and reply, that call costs $0.000055; the bill says $0.003135. Log usage per request and sum it.
• Check that reasoning tokens move. The fast model reports 0. If a reasoning model also reports 0, or the count stays flat while you raise the effort in lesson 4 of T2.2, the parameter never reached the model, and any conclusion about effort is drawn from a test that never ran.

The notebook reduces the choice to one question: does the task have a step where being wrong is expensive and pattern-matching is not enough? As a chain:

For the Larkspur generator:

• How long does a refund take? The answer is one sentence of the refund-timing chunk. This is Task B's shape, and a reasoning model only adds latency and cost.
• Can I get a refund after 45 days? The model has to combine the 30-day window, the plan type and the proration rule. A wrong amount turns into a support ticket, so this is Task A's shape.
• Benchmark both kinds. Run your own hard case and your own easy case through both models. A benchmark of hard cases alone recommends the reasoning model for everything, including the lookups that make up most of the traffic.

Sending each question to the right model at run time is routing by model tier, built in T7.3 lesson 2 of T7.3.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A reasoning model decides how much to think and bills hidden reasoning tokens at the output rate. At the notebook's prices o4-mini also costs 7.3 times as much per token as gpt-4o-mini.
• Log usage per request and never estimate spend from the reply text. On a short answer the hidden reasoning can be almost the whole bill.
• Use a reasoning model only where a step is hard and a wrong answer is expensive. Benchmark your own easy and hard cases, since hard cases alone recommend it for every lookup.