Hardening: edge cases, testing and system-wide error handling

What you'll learn

• Explain why a multi-agent system needs tests at three levels, from a pure router to end-to-end runs.
• Trace the router's three rules on hand-built states, and read which threshold makes which test pass.
• Diagnose a test suite that turns green by hiding a bug.
• Choose a bounded recovery for each kind of failure: retry, fallback model, degraded retrieval or a person.

Lesson 1 of T11.3 built a hotel desk with a coordinator, a compliance checker and a reservation agent, and left two gaps open: the checker runs only when the coordinator names it, and the router forwards on any mention of an agent's name.

A multi-agent system fails quietly. The coordinator skips the compliance check or forwards a booking before it has dates, and the guest still gets a polite reply. System testing for agents means checking the decisions and the side effects, never the tone of the final message.

The build catches these failures at three levels, from cheapest to most realistic:

Table 1: Three levels of tests, cheapest first.

The router as a pure function

The router can be unit-tested because it is a pure function of the state: no model call, no I/O. It checks three rules in order:

Figure 1: The router's three rules. The confidence gate runs first, then the loop guard, then the name check.

The seven tests pin each rule down. We ran the build's router on them at its threshold of 50:

Table 2: The seven router tests at threshold 50, run with the shared hotel module and no model.

The last row is a known issue kept on purpose: lesson 1 of T11.3's mention routing, where the refusal contains the words "reservation assistant" and the name check matches them. The suite keeps it as an expected failure, which documents the gap and flags the day a typed router fixes it.

The confidence gate

The confidence score comes from the coordinator itself. On each new guest message it asks the model for a self-assessment, a score from 0 to 100 and a one-line reason. The assessor's prompt defines three bands:

• 80 to 100: the intent is clear.
• 50 to 79: the intent is clear but details are missing, such as a booking without dates.
• 0 to 49: the model cannot tell what the guest wants.

Only the last band should stop the turn, which puts the threshold at 50. Below it the coordinator apologises and asks the guest to rephrase. The score is a heuristic, not a calibrated probability; its job is to trigger the "ask again" path.

What a threshold can hide

The interactive lab runs the same router and its seven tests in the page, ported from the shared hotel module, and recomputes them at every threshold. We ran the Python router at all 101 thresholds and got the same four bands:

Table 3: Router tests passing at each threshold, from the Python router and the lab.

All seven green at 75 is the wrong result. Mention routing is as fragile as at 50, and every guest whose booking lacks dates, which the assessor scores 50 to 79, now gets an apology instead of a question about the dates. Call this a gate that hides the bug: the suite passes because a different rule fires first.

Regression runs read from state

Unit tests cannot tell whether the model still behaves. The regression table can. Each row is a guest request with its expected outcome, run on a fresh thread after the database is rebuilt:

Table 4: The five end-to-end cases and their expected outcomes.

The outcome comes from the final state, the nodes that ran and a row count on reservations, never from the reply text. Each failure names the field that broke. verdict: expected COMPLIANT, got None means the coordinator skipped the check; reservation: expected False, got True means it forwarded a booking without dates.

Model output varies between runs. A single failure is a lead, and the number to track is the pass rate over several runs after every prompt or model change. Tests show that something broke; the next section covers how the system recovers.

The build pushes on three edges of the core system and puts each safeguard in code:

• Injection-style input. A guest hides ' OR '1'='1 in the room type. Pasted into a WHERE, the condition is true for every row: on a scratch copy of the hotel database, a Deluxe search widened that way returned all 8 rooms. It can only widen a read, because the query tool refuses every write and the booking tool binds its values as parameters. The durable fixes are structural: a read-only connection and a narrow write tool.
• A violation that should reach a person. A human review node pauses the graph with interrupt(): the graph saves a checkpoint and stops. The thread's state shows which node waits, and resuming with the reviewer's decision continues from that point. A resumed node runs again from its first line, so side effects such as writing to the review log go after the pause, where they run once. The checkpoint history replays the thread step by step for investigating a complaint.
• Dates in the past. The model extracts the dates with structured output, given today's date to resolve "next Friday", and a plain function compares them with today. That check is testable without an API call.

Recovery by error type

Some failures do not fix themselves. A failing node catches its exception and writes a small record to an error field in state, with no message added. Routers read that field, so a guest who types the word "error" cannot trigger the error path. One error handler logs every failure with the thread id, and a pure decision router picks the recovery:

Figure 2: The error path. Failures travel through state, and a pure router picks one recovery per error type.

The three recoveries:

Table 5: Recovery per error type.

A fallback that fails raises a different error type, and the router sends it to a person instead of looping. Call the trap this avoids a fallback loop. Every failure is injected with a simulation flag, which lets each recovery path run on every run, and a table computed from the node paths proves which one ran.

Retries for brief outages

Transient failures get a cheaper fix first: a retry policy re-runs a node that raises. The build uses three attempts, a 0.2 s first wait and no jitter. We ran that policy on a tiny graph whose node fails a set number of times:

Table 6: The retry policy on a node that fails twice and three times, measured in the scratchpad with LangGraph's RetryPolicy.

A retry smooths over a brief outage and does not replace the error path. Without any policy, the first failure ends the whole run.

Larkspur's support desk from lesson 1 of T11.3 gets the same three levels:

1. Router unit tests on hand-built states, including the billing refusal that names the billing agent, kept as a known issue until a typed router lands.
2. A regression table of handbook requests (a refund inside the window, a refund outside it, another account's data, an ambiguous "do the same as last month"), read from node paths and the refunds table.
3. A pass rate per case over several runs, tracked after every prompt or model change.
4. Error recovery by type: a second model for a failed call, the hard-coded policy text when the vector store is down, a person for anything else.

Lesson 3 of T11.3 adds agents in parallel and a memory that outlives the context window.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Test at three levels: unit tests on the pure router with no model calls, a confidence gate that stops unclear turns, and an end-to-end regression table read from the nodes that ran and the database.
• A threshold that turns every test green can still be wrong. At 75 all seven router tests pass, yet every booking that lacks dates gets an apology instead of a question about the dates.
• Record failures in an error field in state and let a pure router pick the recovery, with a failed fallback raising a different error so it reaches a person. Keep retries for brief outages.