Tool errors and evaluating tool calls

What you'll learn

• Explain the two places a tool call can fail, and what LangGraph's three error settings do with each.
• Trace an unknown-server question from a crash to a clarifying answer.
• Choose between returning an error from the tool and catching the exception in the loop.
• Score tool selection, tool input and synthesis separately from the same message trace.

Lesson 4 of T7.1 closed the loop from the tool node back to the model. This lesson asks what travels along that edge when a call fails.

A Larkspur customer asks the support agent Is my server web-9 up? The customer has no server called web-9, so the status tool raises an error. With LangGraph's default settings, that error escapes the graph and the customer gets no answer at all.

The loop and its trace

The agent loop has two nodes. The agent node calls the model. The tool node runs every tool call the model asked for and appends one tool message per call. An edge from the tool node back to the agent node lets the model read the results, and a router ends the run when the last model reply has no tool calls:

Figure 1: The agent loop. Every tool result goes back to the model before the run can end.

The message list is the run's trace. For the Larkspur export question it holds four messages: the question, one model reply requesting the handbook search, its tool message, and the final answer. The status-and-reboot question holds five, because one reply requests two tools. Message count helps diagnose extra work, but it does not say whether the answer is correct.

Two places a call can fail

To isolate the tool node, we fed it a hand-written model reply with one call, so every result is deterministic. The tool checks a customer's servers, web-1 and db-1, and raises an error naming the real servers when asked about any other:

Table 1: A valid call and a controlled failure, default settings.

In the full loop the second row means the whole run raises and the model never sees the error. Yet the error text lists the customer's real servers: a model that could read it could ask which one the customer meant.

A tool call can fail in two places:

• Validation error: the model sends bad arguments, which fail the schema check before the tool runs (a call naming the argument id instead of server_id).
• Execution error: the arguments are fine and the tool itself raises (web-9).

Figure 2: The two failure points. The error setting decides what each one becomes.

The three error settings

The tool node's handle_tool_errors setting decides what happens at each failure point. We ran both planted failures under each of its three values, in the LangGraph version this course pins (1.2.18):

Table 2: Two failures under three settings.

Of the three, only True turns both failures into messages the model can read. The default catches the model's mistakes but not the tool's, which is why the web-9 question crashes under it. This behaviour depends on the installed version, so check it again after an upgrade.

What the model does next

An error message lets the model recover, but it does not make it recover: the model still has to read the message and choose a sensible next step. Here is the web-9 run under True. The tool message is LangGraph's real output; the model's two turns are scripted:

Table 3: The web-9 run with errors returned to the model.

The scripted reply asks for clarification using the server names in the error. A live model could choose another action, including an unhelpful retry. Under the default or False, the same run raises after message 1, before another model turn.

Four golden questions, three settings

The interactive lab runs Larkspur's four golden questions through the full loop under each setting. Its tool results and error texts match what LangGraph produces for the same calls byte for byte (12 runs, 0 mismatches); the model's turns are scripted, as in Table 3.

Table 4: The lab's golden questions under each setting.

• The unknown server is an execution error. Only True catches it, and the model can then name the customer's real servers, because the error text lists them.
• The wrong argument name is a validation error. The default and True both return it as a message, the model retries with query, and the run costs one extra round trip (6 messages instead of 4). False re-raises it.
• The two questions with no failure behave the same under every setting. We call this the happy-path blind spot: a test suite of questions that never fail cannot tell the three settings apart.

Deciding where each error is handled

There are two ways to get a failure to the model:

Table 5: Returning an error versus catching an exception.

The setting also accepts a custom message, an exception type or a group of types (catch only those), or a function that formats the message. Either way, write error messages the model can act on: what was wrong and what would work, as the web-9 message lists the valid servers. Return only information the customer is allowed to see.

Pitfall: the successful failure. A tool that returns an error string completes normally, so its tool message has status success. Only a caught exception produces status error. Monitoring that counts error statuses misses every failure a tool reports as text.

Letting an exception escape is also a choice, and it has a cost: your application code must then log the failed run and give the user a fallback reply.

A run can finish and still be wrong. The refund question in the lab answers correctly under the default setting, yet its first call was malformed and cost an extra round trip. An agent that picks the wrong tool, passes the wrong server id, or ignores what the tool returned can also produce a fluent answer. Each of those is a different failure, so score them separately, from the same trace.

Three dimensions

Table 6: The three scores and where each comes from.

Separate scores tell you what to fix: a selection drop after a description change points at the description, an input drop at the schema and the generated arguments. Confirm the cause from the failing calls before changing either.

Scoring selection and input from the trace

The first two dimensions need no model at all. Give each golden question its expected calls:

• Is my server web-1 running, and does a reboot change its IP address? expects the status tool with server web-1, and the handbook search with any query.
• How long does a refund take? expects the handbook search alone.

A small scorer reads every tool call from the model's replies, compares the set of called names with the expected set, and flags any call that is missing a required argument or has the wrong value for a pinned one. On two runs from the lab:

Table 7: The trace scorer on two lab runs.

Both runs chose the right tools. The refund run's first call is flagged because it passed q instead of query; the lab's trace marks the same call "right tool, missing query".

That scorer is deliberately small. It does not check types, query relevance, call order or duplicate calls, and it assumes it knows every tool name. A production scorer validates against the real tool schemas and flags unknown tool names before looking up their required fields.

Judging synthesis

Synthesis measures whether the final answer uses the returned facts correctly, and that needs judgement. An LLM judge is another model asked to grade the trace against explicit criteria: give it the question, the tool requests, the tool results and the final answer, and ask for a score from 1 to 5 on each of the three dimensions plus a short justification.

One recorded judgement shows the shape. The trace asks Calculate 15% of 250 and check weather in Paris; the agent calls a calculator (37.5) and a simulated weather tool (65°F, partly cloudy), and its answer includes both. The judge scored it 5, 5 and 5. That is one recorded judgement on a practice trace, not a Larkspur result, and a new judge call may disagree.

Pitfall: the self-grading judge. In that recording the judge is the agent's own model, which tends to be lenient on its own work. Use a different, ideally stronger, model, and check its scores against a few human-graded runs before you trust a trend.

Two more cautions. The score fields there are plain whole numbers, so nothing rejects a 7 unless you add a range check. And a single trace scored 5 out of 5 proves little: scores mean something across a fixed golden set, compared before and after one change, as T5.1's baseline-first method does for retrieval.

Counting what never finishes

Per-call scores miss the runs that never produce an answer. Log three counts per run as well:

• Rounds per run: the refund question's extra round shows as 6 messages instead of 4.
• Error tool messages per tool: which tool fails, and how often.
• Crashed runs: the web-9 question under the default has no answer to score at all.

Those counts are cheap and catch both of the lab's failures. A web-9 run where the model asks for exactly what the customer said passes selection and input; its failure belongs to the tool's error handling, and only the crashed-run count records it.

• Scripted model turns. Every model reply in Tables 3, 4 and 7 is scripted; only the tool messages and error texts are LangGraph's real output.
• One version. Table 2 is the behaviour of the pinned LangGraph version; the settings have changed between releases.
• One judgement. The synthesis example is a single recorded score, so it shows the method, not the judge's reliability.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Choose error routing explicitly. In the tested version, the default tool node returns the missing-argument error to the model but re-raises the tool's own error; True returns both.
• A tool that returns an error string produces a success-status message. Write error messages the model can act on, and remember that happy-path tests never show the difference between settings.
• Score tool selection, tool input and synthesis separately. The first two come straight from the trace; judge synthesis with a different model, across a fixed golden set, and count crashed runs too.