The Agent Loop & Tool Calling: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur's support agent returns a 500 whenever a customer mistypes a server name ("Is web-9 up?"). The logs show ValueError: Unknown server 'web-9'. The tool works for valid names. What is wrong, and what do you change?

What a strong answer covers:

• The exception escapes the tool-running step. ToolNode's default turns invalid arguments into messages but re-raises exceptions from inside the tool, so the whole run fails and the model never sees the error
• Two fixes: set handle_tool_errors=True, or have the tool catch the failure and return a descriptive error string. Either way the model receives a tool result marked as an error and can respond to it
• Makes the error text useful to the model by listing the customer's real servers, so the model can ask "did you mean web-1?" instead of apologising
• Adds the typo to the golden set and alerts on the share of error results per tool, so a new failure mode shows up before customers report it

2. A product manager wants to drop search_handbook and give the agent the provider's hosted web search, since "the handbook is public anyway". What do you weigh before agreeing?

What a strong answer covers:

• Where a tool runs decides how much control you have. With a client-side tool you see the arguments before anything runs, can refuse or redact, and own the raw result. With hosted web search you see a status and the query only after the search has happened
• Accuracy and scope: the handbook index returns Larkspur's own current policy with chunk ids the answer can cite, while web search can return third-party pages or an old cached copy
• Testing and cost: a client-side tool can be replaced by a stub that returns fixed results, which makes tests repeatable. A hosted call hits the live web and is billed per call on top of tokens
• A reasonable split: client-side for anything that touches customer data or policy, hosted search only for questions the handbook cannot answer, and a log of which path produced each answer

3. You change the description of get_server_status and swap in a new model in the same release. How do you know the agent still calls the right tool with the right arguments?

What a strong answer covers:

• A golden set of questions with the expected tool calls, run before and after. Change one thing at a time, so a regression points at its cause
• Scores tool selection and tool arguments directly from the trace (the tool names match, the arguments validate against the schema). These checks are deterministic and need no judge
• Uses an LLM judge only for the final answer, from a different and ideally stronger model than the agent, checked against a few traces graded by people, because a model grading its own run tends to be lenient
• Watches production too: how often each tool is called, the share of error results, and rounds per run, split by question type

4. A trace shows the agent calling search_handbook eight times with slightly different queries for "How long does a refund take?" before it gives up. What is going on, and how do you stop it?

What a strong answer covers:

• Checks the tool results first. TF-IDF matches exact tokens, so "refund" never matches "Refunds" in the refund-timing chunk. The tool returned the wrong chunk or nothing, and the model kept rephrasing in the hope of a better result
• Fixes the retriever (embeddings or query rewriting, T5.4 and T6.2) rather than the loop, and makes an empty result say so plainly, so the model can stop searching and say it found nothing
• Caps the loop anyway, with a round limit in a hand-written agent or LangGraph's recursion_limit, and returns a fallback answer when the cap is hit
• Monitors for repeated near-identical calls in traces, since each round costs a model call and adds latency

5. The model sometimes calls get_server_status with "my database server" instead of an id like db-1. The schema says server_id: string. How do you make wrong arguments rarer, and what catches the ones that remain?

What a strong answer covers:

• The schema is all the model reads. Describe the argument in the schema with the id format and an example, and say in the tool description when to use the tool
• Constrains the type where the values are known, for example an enum of the customer's server ids, so a free-text value fails validation instead of reaching the lookup
• Sends validation errors back as messages (the ToolNode default does this), so the model can retry with a valid id
• Measures the tool-argument score on the golden set before and after the description change, and keeps the version that scores better