Routing by handler and by model tier

What you'll learn

• Explain why a router asks the model for one label from a closed set and dispatches in plain code.
• Compare all-cheap, all-reasoning and routed strategies on accuracy and reasoning calls.
• Diagnose a router from its confusion matrix, and say which of its two errors costs more.
• Choose the direction a router should err, and when a cheap-first fallback is safe.

Lesson 1 of T7.3 sent every request down one chain. Larkspur's tickets are not all the same kind: I want my money back belongs in billing, my server will not boot in technical support. Routing classifies the input once and sends it down exactly one specialist path.

This lesson routes on two axes. The first picks a handler. The second, in the next section, picks a model.

One label from a closed set

The content router in this lesson has the same two-part shape as the 24.1 gate: a model makes a small, constrained decision, and a pure function acts on it.

• A constrained classifier. The model returns structured output whose only field is a content type limited to story, poem or joke. It cannot answer with a label the graph has no edge for.
• A pure condition function. A dictionary maps the stored label to a node name, with story as the default, so even an unexpected label lands on a real node.

Each request then costs exactly two LLM calls, one to classify and one in the chosen handler, however many handlers exist. A fourth category is one more allowed label, one node and one edge.

The recorded routes

The recorded test uses phrasing that implies a category without naming it:

Table 1: Three requests, three correct routes, two LLM calls each.

The LCEL version of the same router counts the specialist chains it invokes and confirms that exactly one runs per request.

Planted failures

Both halves of the router have one guard each:

Table 2: The constrained label and the default route.

For Larkspur the closed set is the ticket queue: billing, technical, sales. The test that matters is the routing decision itself, measured on labelled tickets, because a fluent answer from the wrong queue still reads well.

The second router picks a model, not a handler. A cheap model triages each request as easy or hard. Easy work stays on the cheap model; hard work goes to a reasoning model, so reasoning prices are paid only on the traffic that needs them:

Figure 1: Routing by model tier. The triage call is the only extra cost on every request.

Both answering nodes send the same prompt, so any difference in the answers belongs to the model. Two details matter on the first real call: o-series reasoning models accept only the default temperature, so the reasoning tier runs at 1, and the reasoning-effort setting only means something to a reasoning model.

The workload and the three strategies

A router means nothing without "cheaper than what". The recorded comparison runs seven requests, mixed the way real traffic is, mostly easy:

Table 3: The labelled workload. Each request carries its true difficulty and an expected answer.

Each strategy answers all seven, and an answer counts as correct when the expected value appears as a whole word in it:

Table 4: The recorded payoff table.

Read the table for its shape. Routed accuracy should sit near all-reasoning while its reasoning-call count sits near all-cheap. The time column counts answering calls only; the routed strategy also pays one triage call per request, seven in all.

Pitfall: the nervous router. A router that sends everything to hard is worse than no router. It pays the full reasoning bill and adds a triage call on top of every request.

The router's confusion matrix

The router predicts difficulty from the question text. It never observes it, so it makes mistakes, and they show up in a confusion matrix, a table of predicted against true labels:

Table 5: Routing decisions against ground truth.

The two kinds of mistake cost different things:

• Easy sent to reasoning: overspend. Bounded, and visible on the bill.
• Hard sent to cheap: a confident wrong answer. Invisible until someone acts on it.

We call the second one the silent downgrade. In this run it happened twice and cost one wrong answer. Because it costs more than overspend, bias the triage prompt toward hard on borderline cases and accept some overspend as insurance.

Which way to fall back

The same asymmetry decides the fallback direction. "Answer on the cheap model, detect a bad answer, retry on reasoning" only works with a reliable wrong-answer detector, and that detector can miss exactly the confident errors a silent downgrade produces:

Figure 2: Escalate on uncertainty before answering, not after a bad answer.

A measured detector can support a cheap-first fallback. Without one, escalate before answering.

On Larkspur, Can I get a refund after 45 days? is the case to watch:

1. The handler router sends it to billing, correctly.
2. The tier router sees what looks like a lookup and may label it easy.
3. The answer needs proration arithmetic against the refund-window policy, so the cheap tier can return a confident wrong amount.
4. Add questions like it to the labelled set the router is measured on, and re-read its confusion matrix after every triage-prompt change.

Lesson 3 of T7.3 keeps all the work but runs it at once: several branches in parallel, merged into one answer.

• Seven questions. One question moves accuracy by 14 points, so Table 4 shows a shape, not a ranking of models.
• Token-match grading. An answer that states the right value in other words counts as wrong.
• One run. Times and triage labels come from a single recorded run; a rerun can route differently.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A router asks the model for one label from a closed set and dispatches in plain code, with a default route. Each request costs two calls however many handlers exist.
• Routing by model tier pays reasoning prices only on hard traffic: in the recorded run, 86% correct with 1 of 7 reasoning calls, against 71% all cheap and 100% all reasoning.
• The silent downgrade, hard work sent to the cheap tier, costs more than overspend. Lean triage toward hard on borderline cases and escalate before answering, not after.