Hosted vs client-side tools: code execution and image generation

What you'll learn

• Explain the difference between a client-side tool and a hosted tool, call by call.
• Compare what each path lets you see, refuse, test and pay for.
• Diagnose where a policy check can sit, and why a hosted tool has no place for it.
• Choose where each of Larkspur's tools should run.

Every tool in lessons 1 to 5 of T7.1 ran in your process. This lesson moves a tool to the provider's side and looks at what you give up.

Larkspur has to confirm that a customer owns a server before showing its status. That check has to happen before the lookup runs, in code Larkspur controls. A public web search has nothing comparable to enforce. Where a tool runs decides whether such a check is even possible.

Two execution paths

A client-side tool is the kind built so far: the model returns a call, your code runs it, and you send the result back. A hosted tool runs on the provider's infrastructure instead, and the reply arrives with the result already folded in. The change in the request is small. The change in what you can see and control is large.

The recorded comparison answers What is the current population of Lisbon? both ways with the same small model:

Figure 1: The same question on both paths. Only the client-side path passes through your code between the call and the result.

Table 1: One recorded run of each path.

One run proves little about latency, but it does show that one round trip is not automatically faster. The hosted reply carries a record of the search with its status and the query the provider chose. That record is your only window into the execution: you did not choose the query, you cannot see the raw results, and nothing of yours ran between retrieval and generation.

Table 2: What each path gives you.

The policy gap

The gap between "the model asked" and "the model sees the result" is where policy lives. We call it the policy gap. In the recorded example, a guarded version of the client-side lookup refuses any city outside an allow-list:

Table 3: An allow-list in the policy gap.

Allow-lists, tenant scoping, redaction of personal data and spend caps all sit in that gap. Choosing a hosted tool means choosing not to have it.

Key distinction: a hosted tool can be well run and still impossible to govern. What decides the placement is whether anything must happen between the call and the result.

What each hosted tool shows you

Hosted tools expose different amounts of their execution:

Table 4: Visibility by hosted tool.

Code interpreter is also configurable: the recorded example sets a memory ceiling and turns network access off, so the sandbox works only on data you supplied. What you still cannot do is approve the code before it runs, veto one call, or replace it with a stub in a test.

Image generation adds a different caution. The agent describes what it meant to draw, and only showing the image to a vision model reveals what it actually drew. We call that narrated output: a description of the result that nobody checked against the result. The model also has to leave the tool alone when a turn asks for advice rather than an image, and the recorded example checks that it does.

The decision follows the data the tool touches:

Figure 2: Three yes/no questions. Any yes keeps the tool in your process.

For Larkspur's support agent, the split is:

1. The handbook search and the server-status lookup stay client-side. They read the customer's account and Larkspur's own policy, need tenant scoping and logging, and must be replaceable with stubs in tests. The T7.1 lab runs them deterministically for this reason.
2. A hosted web search fits questions the handbook cannot answer, about public software or third-party services, where there is nothing to enforce between the call and the result.
3. Hosted code execution fits arithmetic on numbers the customer pasted into the chat. A model that runs three lines of code computes the answer; a model that answers directly produces plausible-looking digits.

In OpenAI's API, hosted tools are offered through the Responses API, while Chat Completions carries client-side function calls. Check which API your integration and provider version support before choosing a wrapper. A third placement, a tool on a server you run that the model reaches over a protocol, is MCP (T12.1 to T12.3), and it gives back much of the control that hosted tools remove.

• One run per path. The latencies in Table 1 come from a single recorded run each; measure on your own traffic before deciding anything on speed.
• One provider. The visibility in Table 4 is what OpenAI's hosted tools returned in the recorded examples; other providers expose more or less.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A hosted tool runs on the provider's side and returns the result folded into the reply. You lose the policy gap, where your code can log, refuse, redact or cache a call.
• Hosted tools vary in visibility: web search shows roughly the query, code interpreter shows the code and output, image generation shows its revised prompt. None can be replaced with a stub in a test.
• Keep tools that touch customer data or need tenant scoping client-side. Use hosted tools for public lookups and arithmetic on data the user supplied.