Routing: LLM classifier and semantic routing

What you'll learn

• Compare an LLM classifier and an embedding-similarity router on cost, reach and failure.
• Diagnose a classifier that silently returns no route, and one that always returns a route.
• Read route scores and margins, and set the thresholds that send unclear questions onward.
• Design Larkspur's routes and test them on real scores.

Lessons 1 to 6 of T6.2 each fit one kind of question: decomposition for compound ones, HyDE for vague ones, step-back for premises, plain retrieval for direct lookups. Applying all of them to every question is the transform-everything habit from lesson 1 of T6.2. Something has to choose per question.

A router decides where a question goes before anything answers it: which prompt, which data source, which transformation. Notebooks 07a and 07b build the same four-way router twice, over personal finance, book reviews, health and fitness, and travel:

• Logical routing: an LLM reads the question and chooses.
• Semantic routing: the question is embedded and sent to the destination whose example questions it is closest to.

The two fail in different ways:

Table 1: Logical against semantic routing.

The first version of 07a asked for the category as free text and compared it with exact strings. The model answered "health & fitness" and "travel guides", which match neither "Health & Fitness" nor "Travel Guide", so every branch fell through and the router returned nothing. The notebook still appeared to work. We call this the silent fall-through.

The fix binds a schema with structured output whose destination field is a fixed set of the four route names, so no other value can come back, plus a one-sentence reasoning field that makes each decision readable in traces.

A fixed set guarantees a valid route, not a sensible one. Asked What is the capital of France?, the classifier still picks one of the four. We call this the forced route. A production router needs:

• An "other" destination, so out-of-scope questions have somewhere honest to go.
• A regression set of questions with expected routes, because adding a route changes decisions on old questions.
• A small model. Routing is classification; it rarely needs the model that writes answers.

Each route is defined by a few example questions, three per route in 07b, embedded once at start-up. Per query the router embeds the question, takes each route's highest similarity to its examples, and picks the highest route. The routing path never calls an LLM.

07b shows why the winner alone is not enough. With text-embedding-3-small, How can I travel on a budget? is one of the travel examples word for word and scores 1.0000, while personal finance still scores about 0.50 on "budget". Are meal-prep books worth buying? wins by a margin of about 0.002, with every route near 0.28, and book reviews is not even in the top two despite the word "books". The router returns both answers with the same confidence.

Two thresholds turn scores into a confidence check: a minimum top score (did anything match?) and a minimum margin over the runner-up (was the choice distinguishable?). 07b uses 0.35 and 0.02 and calls them illustrative. Questions that fail either go to the LLM classifier, which makes the two routers one system:

Figure 1: A semantic router with an LLM fallback. Most traffic pays for one embedding.

A Larkspur support assistant can route on the handbook's own sections. We gave each route two example questions and scored six test questions with all-MiniLM-L6-v2:

Table 2: Four Larkspur routes and their examples.

Here are the scores, with the decision Figure 1's thresholds make:

Table 3: Route scores for six questions (all-MiniLM-L6-v2, highest similarity to each route's examples).

The guard catches the near-tie and the out-of-scope question. It also shows two limits. Can I download my databases? is a clear data question, but its top score of 0.325 falls under the floor, so it pays for a classifier call it did not need. And the outage-refund question belongs to billing and reliability at once; it passes with a margin of 0.045 and goes to billing alone. A margin floor of 0.05 would send it to the classifier.

Pitfall: borrowed thresholds. 0.35 and 0.02 were set for text-embedding-3-small on 07b's routes. Scores from another model spread differently, as Table 3 shows. Set both thresholds on logged traffic for your own model and routes.

1. Semantic router first, with Table 2's routes and more examples per route drawn from real tickets.
2. Tune the floor and the margin on logged questions with known routes; Table 3 suggests a lower floor and a higher margin for this model.
3. Classifier fallback with an "other" route for anything that fails either check.
4. Send cross-topic questions to both routes when the two top scores sit close, rather than forcing one.

Routing chooses a destination. Lesson 8 of T6.2 makes a related move inside one source: it turns part of the question into a metadata filter.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Logical routing asks an LLM to choose; semantic routing picks the route whose examples are closest by embedding, about 100 times cheaper per query in 07a's comparison.
• A fixed set of route names prevents the silent fall-through but not the forced route: add an "other" destination and a regression set of questions with expected routes.
• Gate semantic routes on a minimum top score and a minimum margin, and tune both per model: on Larkspur's routes the guard catches a 0.008 near-tie but also sends a clear data question to the classifier.