Tool-Using Agent with Safety Controls

Ops automation is turning a messy email into a real system change: CRM (customer relationship management) note, refund, lookup.

G03 covers one slice: the model proposes the next small action; other services decide if it may run. The model never holds the keys.

End to end, as a “refund this VIP” email:

1. Support agent Sara delegates the case.
2. The planner proposes one step — read the email, or check eligibility, not “do everything.”
3. Registry and validation check the tool and arguments.
4. Policy says allow, block, or needs a human.
5. A broker mints a short, narrow token. The gateway runs once and writes a receipt.
6. If it is looping or a bad refund is in flight, operators kill new calls, queues, and tokens.

That’s it: propose → validate → decide → maybe approve → execute once → record. Shell access and unrestricted refunds stay out.

Core idea: The model proposes one bounded action. Deterministic services validate, authorise, approve, execute, and record it. The agent never holds broad credentials or decides whether its own action is allowed.

Use this for the spoken design. The Deep Dive holds the mechanics, and the unchanged source case holds the full reference.

“We want an agent that reads email, checks internal systems, updates CRM, and sometimes issues refunds. The useful part is interpreting the request; the dangerous part is letting that interpretation become authority. I’ll let the model propose actions, while identity, policy, approval, and an idempotent gateway decide what can actually happen.”

The same request matters differently to operations, security, finance, and tool owners. Ask what the agent may do alone, what requires review, and what it must never do. Treat customer email and tool responses as untrusted input.

Questions to ask the interviewer

The source discovery table, §1 gives the full set. These seven change the controls. Email and tool output stay untrusted either way, so that is assumed rather than asked.

If approval policy is unspecified, assume a human gate for irreversible actions; do not invent a permissive refund threshold.

The anchor for its variants

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Authenticate the delegating actor — record their rights.
2. Propose one bounded next action — the planner does not perform it.
3. Resolve a declared tool — validate typed arguments and data class.
4. Decide allow / block / needs-approval — deterministic policy, not the model.
5. Mint a short-lived least-privilege credential — only for an approved action.
6. Execute once through an idempotent gateway — record the receipt; stop safely on uncertainty.

The first usable version also needs a tool registry, a real approval record, and a tested kill switch.

Non-functional requirements — how well / under what constraints

These are illustrative interview numbers, not measured production demand. Risk is impact × likelihood × irreversibility.

First release

Exclude free-form code or shell, open-ended tool discovery, unbounded loops, cross-tenant access, self-modifying policy, and unrestricted refunds.

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, the model proposes one bounded action; policy, approval, a scoped token, and an idempotent gateway decide and execute. Non-functionally, fail closed on writes, bound loops and spend, keep a ledger, and keep a kill switch.”

This is a whiteboard version of the source architecture, §4.

Step-by-step architecture

• Step 1. Establish delegation. Authenticate the user and determine which actions the user may delegate.
• Step 2. Plan one bounded step. The LLM (large language model) agent planner reads task state and proposes one next action within its step and cost budgets.
• Step 3. Validate the tool call. Resolve the tool's typed schema and check arguments, data class, and action limits.
• Step 4. Apply policy. Block and record a disallowed action, allow a safe one, or request human approval for the exact payload before expiry.
• Step 5. Grant narrow authority. For an allowed or approved action, the broker issues a short-lived scoped token; budgets and the kill switch still apply.
• Step 6. Execute once through the gateway. The idempotent gateway calls the target system and prevents a retry from becoming a second side effect.
• Step 7. Close the loop. Store the action receipt in the audit ledger and task state; continue planning only from the recorded result.

Model and agent role: The LLM is the agent's planner, not its executor. It consumes task state and tool results, proposes one bounded action, and can continue only after the policy, approval, gateway, and receipt path completes.

Say the boundaries: the planner can propose, the registry defines legal tool shapes, validation rejects malformed or over-limit arguments, policy decides, approval binds to the exact state, the broker supplies one narrow capability, and the gateway is the only write path. A proposal is not proof of execution; the receipt is. The task state store remembers workflow progress, while downstream systems remain the business systems of record.

For “read an email, check refund eligibility, update the CRM note, and refund if allowed”:

1. Verify the support agent’s delegated rights.
2. Propose the next smallest step, such as reading the email or checking the account.
3. Resolve the approved tool; validate schema, amount, account, data class, and destination.
4. Ask the policy point to allow, block, or request approval.
5. Bind any approval to tenant, record version, amount, and expiry.
6. Execute with a scoped credential and an idempotency key; record the result.
7. Continue only if the new task state and evidence justify another bounded step.

The four core security controls are untrusted observations, narrow credentials, validated proposals, and hard ceilings. Prompt instructions alone do not enforce any of them.

The kill switch must stop new tool calls, cancel queued work, revoke or expire tokens, and mark in-flight tasks for downstream rejection. A dashboard flag that merely blocks new user requests is insufficient.

At the source’s illustrative 20 QPS and 10 actions/task, plan for about 200 tool actions/s, with 40–60 model calls/s during iterative planning bursts. The source estimates roughly 5 KB of decision record plus 20 KB audit trail per task, or about 8.6–34.6 GB/day of raw append-only data at its assumed range. Ask whether planner time, downstream tool time, or human approval is the actual bottleneck.

At 10×, partition by tenant or workflow, cap concurrency per user/tool/globally, queue slow actions, and keep ordered actions for one account together. REST executes; function calling selects among a small set; MCP (Model Context Protocol) can expose tool discovery; an agent framework coordinates multi-step work. A registry loads only relevant schemas instead of placing 1,500 functions from 100 apps in one prompt. Backend services trust enterprise identity, not the LLM.

If the agent is slow or expensive: trace → route common intents deterministically → bound steps, retries, tokens and tool time → cache safe read results within the request → move slow side effects async. Never lengthen an unbounded loop to make it “more capable.”

Release gates: zero unsafe actions and zero duplicate effects. Track policy denials, approval delay, tool success, task completion, overrides, latency, and cost beside those gates. Red-team indirect instructions in emails, CRM rows, tickets, and fetched pages; stale approvals; over-broad tokens; over-limit refunds; lost responses; and runaway loops.

Roll out in layers: read-only triage and drafting → narrow reversible CRM writes with proven revert/idempotency → financial actions behind human approval → adversarial tests and canary → expand only while the two zero-count gates hold. The control plane, gateway, broker, ledger, and approval service are reusable; customer-specific CRM fields and thresholds live in adapters/configuration.

In the source incident, §13, the planner proposed 18 refunds for delayed VIP shipments, including a €740 refund above a €250 approval limit. The gateway blocked six calls, but 12 were allowed. No unauthorised money moved; the planner still failed by proposing a risky bulk action.

Detect: read planned, blocked, and allowed calls with amount, role, approval state, policy version. Contain: disable bulk refund planning, route VIP refunds to review tickets, and audit the 12 allowed calls. Root cause: action-oriented planner examples omitted bulk approval boundaries; the gateway policy still worked. Prevent: risk-aware tool schemas, policy simulation, a separate action-risk check, and non-bypassable gateway enforcement.

Spend roughly 50 minutes: opening/scope (12), architecture and one-request walk (10), trade-offs and failure drills (16), rollout/measurement (6), close (6). Keep the technical detail in reserve; the interviewer should hear the control boundary early.

“The model interprets requests and proposes one bounded action. A registry and validator check the tool and arguments; deterministic policy returns allow, block, or approval. A broker mints a short-lived scoped token, and an idempotent gateway executes once and writes a receipt. Tool output is untrusted, writes fail closed, and budgets cap loops and spend. I would ship read-only first, add reversible writes with a revert plan, keep money behind human approval, and gate expansion on zero unsafe or duplicate effects.”

Final mental model: Propose → validate → decide → approve if needed → scope credential → execute once → record → stop or repeat within budget.

The Main guide is the spoken design. This is its technical backup; the unchanged source case keeps the complete synthesis and references.

Model and agent roles: The LLM (large language model) agent planner produces one bounded ActionProposal from task state and tool observations. The registry, validator, policy point, approver, broker, and idempotent gateway remain separate deterministic authorities; the planner never executes directly.

AgentTask(id, actor, goal, state, step_budget) tracks the workflow, not customer or financial truth. ToolDefinition(name, schema, data_class, scopes) limits the legal tool space. ActionProposal(id, task_id, tool, args_hash, policy_decision) is the model’s recommendation. PolicyDecision(proposal_id, verdict, scopes, reason) is the deterministic verdict. ActionReceipt(proposal_id, idempotency_key, outcome) is proof of the side effect. Canonicalize arguments before hashing, and return the stored receipt when the same business action is retried.

Keep the order explicit: authenticate → propose → resolve registered tool → validate typed arguments/data class/limits → evaluate policy → obtain approval if required → mint scoped credential → execute through the gateway → persist receipt → decide whether to continue. In this case, validation runs before policy so malformed or over-limit proposals never enter authorisation. If the policy or broker cannot be reached, a write cannot proceed.

An approval is a durable record of reviewer, time, rationale, exact payload, tenant, record version, amount, and expiry. A later state change invalidates it. The gateway, not the planner, owns exactly-once effect semantics within the idempotency contract. A lost response is an unknown outcome that needs reconciliation; neither a prompt nor a generic retry makes it safe.

| Component | Control it owns | If unavailable | |---|---| | Planner | Next bounded proposal, stop condition, step budget | Pause and escalate. | | Task state store | Durable workflow progress | Stop writes; no untracked continuation. | | Tool registry | Declared schemas, data class, risk label | Undeclared tool cannot run. | | Policy decision point | Allow, block, or approval with reason/scope | Writes fail closed. | | Approval service | Payload-bound human decision | Wait, expire, or re-request. | | Credential broker | Short-lived capability for one action | Writes fail closed; no broad-secret fallback. | | Execution gateway | Scoped, validated, idempotent effect | Reconcile uncertain writes. | | Audit ledger | Intent, proposal, verdict, approval, receipt | Stop if evidence cannot be made durable. | | Kill switch | Stop autonomous paths and revoke authority | Tested before launch. |

The user boundary ends at authentication, planner boundary at proposal, policy boundary at verdict, and execution boundary at the downstream side effect. CRM (customer relationship management), email, internal data, and refund systems remain systems of record. A cache may speed safe reads but never becomes authority for permissions or refunds.

Use Impact × Likelihood × Irreversibility as a qualitative risk aid, not a numeric proof. A public FAQ read may be autonomous; a CRM update may need preconditions; a refund may need human approval; some exports or permission changes are blocked. Refund reversibility in software does not erase customer, finance, or abuse costs.

The source’s planning anchor is 50k users, 20 QPS (queries per second) peak, 10 actions/task: roughly 200 tool actions/s. Iterative plans may reach 40–60 model calls/s. With about 1 s of planner compute/task, the planner needs roughly 20 core-seconds per second at peak, plus slack for tails. Illustrative 5 KB decision + 20 KB audit/task gives about 100–400 KB/s and 8.6–34.6 GB/day raw before indexing/replication. At 10× peak, ask what saturates first: planner, approval queue, adapter, ledger, or refund service. At 10× lower load, simplify deployment but keep the same safety boundary.

Cap concurrency by user, tool, and globally; also cap value and destination. Partition ordered work by account or workflow so conflicting writes do not race. Reads can run in parallel; writes and approvals must follow state transitions. Long workflows return a job ID and continue asynchronously.

The layers have different jobs: REST/SOAP/SQL execute; function calling lets a model select a tool; MCP (Model Context Protocol) advertises tools/resources; an agent framework coordinates multi-step work. MCP does not replace the underlying API. Direct REST is enough for a fixed deterministic workflow, while a planner earns its overhead only when a request spans systems.

Putting 1,500 functions from 100 applications into every prompt makes selection and schema maintenance brittle. A registry loads only task-relevant tool descriptions and carries typed arguments, data class, risk, and scopes. At 500+ applications/200k employees, scale stateless planner/gateway services, use per-user and per-app quotas, keep long work async, and place region-local gateways/tool servers where data residency requires it.

The model never receives a broad API key. A trusted service propagates enterprise identity and attaches a narrow OAuth/RBAC/ABAC capability to the approved call. Every downstream application evaluates that identity. Task state and audit are separate from cross-session business truth.

The four controls are a set: untrusted tool observations, no broad long-lived credentials, validation of proposals, and ceilings on steps/spend/amount/destinations. Add defense in depth: policy, broker, idempotent gateway, ledger, and kill switch. Scope failures by tenant, region, workflow, and dependency.

The runbook must locate prompts, versions, proposal/validation/policy records, approval artifact, token scope, calls, and receipts. The kill switch must halt new calls, cancel queued work, revoke/expire delegated tokens, and make downstream reject in-flight completions. A read-only replay can be safe; a write replay needs its receipt/idempotency contract.

Zero unsafe actions and zero duplicate effects gate expansion. Track policy denial rate, approval rate/delay, tool success, task completion, override rate, p95, step count, retry count, and cost per task. A denial spike can mean an attack or a broken schema; an override spike can mean model quality or policy calibration. Inspect examples before changing the gate.

The red-team set covers indirect injection in tool output; over-broad credentials; over-limit or malformed proposals; response-loss retries; stale approvals; tool loops; cross-tenant requests. A passing demo is not evidence that these paths are safe.

Rollout: read-only triage/draft → narrow reversible CRM writes with owner/revert plan → money behind a human → injection tests → small canary → expansion if gates hold. Shared product services are policy, broker, approval, execution, and ledger; customer wiring remains configuration and adapters. Train users on refusals, approvals, escalation, and recovery.

Agent steps and tool calls, including retries after rejected or failed actions, are the common cost driver. Profile model planning versus downstream tool wait versus approval. If planner-heavy, route common intents deterministically and combine safe reads; if tool-heavy, cache safe reads in-request and prefetch; if approval-heavy, improve proposal clarity and risk classification, never bypass approval.

For a tool loop, inspect the trace for repeated tool/observation pairs and missing state transitions, then impose max steps, result cache, clear stopping criteria, and tool-selection tests. For a tool timeout, find the critical path, parallelise independent reads, set per-tool deadlines, and move slow side effects to an async job with status. Measure steps, tool latency, retries, timeouts, and cost before and after.

The source incident’s support agent asked to handle delayed VIP shipments. The planner selected refund_customer for 18 orders. A €740 request exceeded a €250 no-approval threshold; six calls were blocked, 12 allowed, and no unauthorised money moved. A planner prompt designed to be “more action-oriented” omitted bulk approval boundaries. The unchanged gateway policy blocked the high-value actions.

Contain by disabling bulk refund planning, forcing VIP refunds into review-ticket mode, adding user confirmation, and auditing the 12 allowed actions. Prevent with threshold-aware schemas, policy simulation, a separate action-risk check, better planner examples, and non-bypassable gateway enforcement. Treat it as a near miss even though the money gate held; the proposal distribution deteriorated.

The source retains the original 50-minute script, full diagrams, trace excerpt, and reference table.

Use the Main guide to practice speaking and the Deep Dive for mechanisms. The source is the full reference.

The model proposes; deterministic systems decide; the idempotent gateway executes once and records a receipt.

Authenticate → bounded proposal → registered typed tool → validate → policy: allow/block/approve → scoped token → idempotent gateway → system of record → receipt/audit → continue or stop.

LLM/agent role: The agent's LLM (large language model) planner proposes one bounded tool call; policy, approval, and the gateway control execution.

Actions reversible? Money boundary? Delegated identity? What needs approval? Systems of record? Audit/retention? How to stop in-flight work? Peak task and tool load?

1. Planning separated from execution.
2. Tool, arguments, identity, and policy validated.
3. Narrow short-lived credentials.
4. Human approval for high-risk effects.
5. Idempotent, auditable writes.
6. Safe stop on uncertainty.

Untrusted tool observations · no broad credentials · validated proposals · hard ceilings on steps, spend, amount, destination, and concurrency.

Proposal ≠ receipt. Confidence ≠ permission. Approval binds to payload + record version + expiry.

50k users · 20 QPS (queries per second) peak · 10 actions/task → ~200 tool actions/s. Iterative planning may mean 40–60 model calls/s. Qualitative risk aid: impact × likelihood × irreversibility.

Zero unsafe actions; zero duplicate effects. Read-only → reversible writes with revert/idempotency → financial actions behind approval → red-team → canary. Kill switch halts new calls, queued work, and in-flight authority.

Trace model steps and tool latency; route common tasks deterministically; cap steps/retries; cache safe read results; run slow side effects asynchronously. Measure cost and steps per task.

Planner proposed 18 refunds; six blocked, 12 allowed; €740 exceeded €250 no-approval threshold. Gateway prevented unauthorised transfer, but planner bulk-action behaviour failed. Disable bulk path, audit allowed calls, add risk-aware schemas and simulation, keep gateway enforcement.

“I would let the model interpret and propose, keep authority in policy and approval, execute only through a scoped idempotent gateway, and expand autonomy only when unsafe and duplicate effects stay at zero.”

Let the model act on email, CRM (customer relationship management) and refunds without ever letting it decide, because usefulness comes from acting and trust comes from never deciding.

The hard part of this system is not getting a model to call a tool. It is that an agent which reads email, queries systems, updates CRM and issues refunds has to be useful without holding any authority. So the model proposes and deterministic systems decide. This page consolidates the group's prompts into one read for the day before. Everything else in the group is a delta on it.

Open with the disagreement, not the feature list. The customer asks for an agent that reads email, queries internal systems, updates CRM records and issues refunds. Everyone nods at the feature list, then disagrees completely. Operations wants speed. Security wants hard approval gates. The support manager wants routine cases handled without friction. Finance fears irreversible money movement. That disagreement is the design problem. So restate the ask as a testable outcome: enable useful automation while deterministic controls govern identity, permissions, risk and irreversible effects. The consequence is that the model is not the authority. It interprets and proposes. Deterministic systems decide who may act, what may change and whether an action can be reversed.

"Let me restate the problem first. We need a tool-using agent that can read email, query internal systems, update CRM records, and sometimes trigger refunds, but the model itself should not have unchecked authority. So I'd frame the outcome as enabling useful automation while deterministic controls govern identity, permissions, risk, and irreversible effects."

The three answer tiers show what that framing buys. The weak answer connects the agent to email and CRM and lets it issue refunds when it decides the customer deserves one. It makes the model the authority, hands it credentials instead of scoped capabilities, and has no defence against instructions hidden in a ticket. The average answer plans tool calls, checks permissions before each one, requires approval for large refunds, logs everything and adds retries. It still treats tool output as trusted, never bounds loops or spend, and adds retries without idempotency, which is how one refund quietly becomes two. The strong answer has the planner propose one bounded action. A typed schema validates it. A deterministic policy engine returns allow, block or needs-approval. A credential broker mints a short-lived token scoped to that single action. An idempotent gateway guarantees exactly one effect. Tool observations are untrusted data, never instructions. Loops, spend, refund amounts and destinations all have hard ceilings.

Then ask the questions that change what may be built, because a useful question is not the one that sounds thorough.

Map the people, because each stakeholder defines a different success condition.

State the assumptions where the interviewer withholds answers, ranked by risk. With no approval process defined, assume a conservative human gate on every irreversible action. With retention unspecified, assume enough is stored to reconstruct any action, and leave the exact period to customer policy. With tenancy unclear, assume strict isolation and a separate authorisation check at every tool boundary. The CRM is the source of truth for customer context. Refunds above a threshold require approval. The agent may draft actions but cannot execute high-risk steps unless the policy engine allows it.

A requirement the customer cannot test is a preference. "It should be safe" is a preference; "no arbitrary command execution" is a constraint, and only the second one changes the architecture. Distinguish what the system must do, how safely it must do it, and what it must never do.

The six must-have functional requirements are a set, and interviewers ask for them cold. Separate planning from execution, so the model proposes a plan and a separate control layer decides whether any tool call is allowed. Validate tool name, arguments, identity and policy against an allowlist, a parameter schema, the caller's identity and the policy rules before anything runs. Use scoped, short-lived credentials, so the agent never holds a broad secret and receives only the minimum for the next approved step. Require approval for high-risk actions, meaning refunds, account changes and any irreversible external write cross a human boundary unless the customer accepts a narrower automated threshold. Make effects idempotent and auditable, so a retry cannot double-refund and every decision leaves a replayable trace. Stop safely under uncertainty: poor confidence, a malformed tool response, unclear policy or an inconsistent environment means pause, escalate or fail closed, never improvise.

Three should-haves belong in the first usable version. A declared registry with schema and data class for every tool, so undeclared tools cannot be called. A tested kill switch that halts autonomous paths without a deploy. Approval captured as a first-class record. The could-haves wait: richer policy expression, adaptive risk scoring, queued multi-step orchestration, replay tooling, operator dashboards, anomaly detection on action patterns, per-tenant policy overlays.

Declare the non-goals, because the strongest candidates draw a narrower boundary than the prompt suggests:

• free-form code execution or shell access
• open-ended tool discovery
• unbounded autonomous loops
• automatic refunds without approval thresholds
• cross-tenant data access
• self-modifying policy logic

The constraints are the lines that cannot be crossed, and each is stated so a test can fail it.

Every must-have then needs an owner in the architecture, and the traceability table is the proof.

One request is not one unit of work. Use the planning anchor of 50,000 users, 10 actions per task and 20 QPS peak. Then separate three layers of load. How often tasks arrive. How many tool calls, model calls and policy checks each task triggers. How many actions can actually change records, money or permissions.

The estimate that most affects the design is not CPU. It is the ratio of model time to tool time, plus the irreversibility of tool effects. Planning fast and tools slow means invest in async orchestration, caching and queueing. Tool effects high-risk means invest in policy checks, staged execution and audit logging even at the cost of latency. Human review as the bottleneck means clearer escalation rules and fewer ambiguous cases.

Derive the risk tier before deciding autonomy. The decision aid is Risk = Impact × Likelihood × Irreversibility, not literal arithmetic. A low-impact read of a public FAQ can be fully autonomous. A CRM update can be partially reversible and still costly if it misroutes a customer. A refund may be technically reversible and still meaningfully irreversible through customer dissatisfaction, accounting noise or abuse. The tiers are fully autonomous, autonomous with preconditions, requires human approval, and blocked entirely.

Show the sensitivity range instead of false precision. The 10× growth case is the real test: which part breaks first, the planner, the approval queue, a tool adapter, the audit log or the refund service?

Concurrency limits are a safety feature, not only a performance one. Cap per user, so nobody floods the system with parallel tasks; per tool, so a noisy downstream is not overwhelmed; and globally. Encode value limits too, so many low-risk reads are allowed while the number and value of refunds in flight stays bounded. A well-designed limit stops a prompt injection from becoming a cascade.

One diagram carries the whole design, and the organising split is control plane against data plane. The control plane decides what the agent is allowed to do, under what conditions and with which approvals. The data plane performs the bounded business work once those decisions are made. The model sits entirely inside the control plane's proposal step and never touches the data plane directly.

 ╔════════════════════════════════ CONTROL PLANE (decides what may happen) ═════════════════════════════════╗
 ║                                                                                                            ║
 ║  user ─> AuthN/AuthZ edge ─> AGENT PLANNER <──> task state store                                           ║
 ║            (delegation rights)   │ proposes ONE bounded action; forbidden from executing                   ║
 ║                                  v                                                                         ║
 ║                             TOOL REGISTRY   (allowed tools, arg schemas, data class, risk label)           ║
 ║                                  │ undeclared tool cannot be called; schema validation BEFORE policy       ║
 ║                                  v                                                                         ║
 ║                        POLICY DECISION POINT  ──> allow | block | needs-approval  + scopes + reason        ║
 ║                             │         │                                                                    ║
 ║                             │         └──> APPROVAL SERVICE (human sign-off bound to a snapshot + expiry)  ║
 ║                             v                                                                              ║
 ║                       CREDENTIAL BROKER  mints token: one tenant · one workflow · one action · one expiry  ║
 ║                             │                                                                              ║
 ║        KILL SWITCH ─────────┼── halts new tool calls, cancels queued work, revokes tokens, marks in-flight ║
 ║                             │                                                                              ║
 ╚═════════════════════════════╪══════════════════════════════════════════════════════════════════════════════╝
                               │ scoped token + idempotency key
 ╔═════════════════════════════╪═════════════ DATA PLANE (performs the bounded work) ═════════════════════════╗
 ║                             v                                                                              ║
 ║              IDEMPOTENT EXECUTION GATEWAY  (exactly one effect per proposal; replay returns the receipt)   ║
 ║                             │                                                                              ║
 ║                             v                                                                              ║
 ║        CRM · email · refund service · internal data  (systems of record; tool output is UNTRUSTED data)   ║
 ║                                                                                                            ║
 ║  AUDIT LEDGER (tamper-evident): request received · proposal · policy verdict · approval · receipt          ║
 ║  OBSERVABILITY: policy denial rate · unsafe actions · duplicate effects · approval delay · override rate   ║
 ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════╝

The same flow as a rendered diagram, for viewers that draw Mermaid:

Read the nine components in dependency order, because that is the order they have to exist and the order they fail.

Four trust-boundary sentences are worth saying verbatim. The user boundary ends at authentication. The planner boundary ends at proposing intent, not action. The policy boundary ends at allow, deny or approve. The execution boundary ends at a scoped, logged side effect in a downstream system of record. Mark the systems of record explicitly. CRM (customer relationship management) owns customer attributes and refund history. The email system owns transport and inbox state. The internal data source owns account and entitlement data. The task state store owns workflow progress but not business truth. Caches may sit beside the planner or registry for low-risk lookups and must never become the source of truth for permissions or refunds.

Not every boundary is synchronous. "Plan the next action" and policy evaluation are synchronous, because a human waits on a gating decision. Tool execution may be synchronous for one low-latency action, but bulk and slow steps go through queues. Audit writes can be asynchronous as long as the log never falls behind the point where a failure becomes unrecoverable. Use backpressure when risk rises faster than capacity: if approvers are saturated, slow autonomous execution rather than letting risky actions accumulate. Partition by workflow or account, so all actions for one account share a key and two conflicting refunds cannot run in parallel.

Walk the representative request in order without skipping a handoff. The request is: "Read the customer's email, check whether the account qualifies for a refund, update the CRM note, and issue the refund if policy allows." The interviewer should hear where control moves from the model to deterministic services and back.

1. Authenticate the delegating user. The front door verifies who is asking and what delegation rights they hold; a support agent is not a finance approver.
2. Plan the bounded next action. The planner selects the next smallest step, such as "retrieve the latest email thread", not the whole workflow.
3. Resolve the tool schema. The registry returns the exact contract, including field names and data sensitivity labels.
4. Validate arguments and data classification, for shape, missing fields, dangerous content and regulated data that changes handling.
5. Evaluate policy and risk. The decision point compares action, user role, data class and downstream effect against current policy; a note update may pass, a refund may need approval.
6. Obtain approval if required, captured as a record bound to the exact payload.
7. Execute with a scoped credential and an idempotency key, so a retry cannot duplicate the refund.
8. Record the result and decide whether to continue. The ledger stores input, policy result, approval outcome, tool result and side effects; the planner picks the next bounded action or stops.

The data model follows from the walk, and two records must never be conflated. AgentTask(id, actor, goal, state, step_budget). ToolDefinition(name, schema, data_class, scopes). ActionProposal(id, task_id, tool, args_hash, policy_decision). PolicyDecision(proposal_id, verdict, scopes, reason). ActionReceipt(proposal_id, idempotency_key, outcome). The proposal is the model's recommendation; the receipt is the only proof a side effect occurred. Arguments are canonicalized before hashing, so the same logical input always yields the same idempotency seed. A repeated key returns the stored receipt instead of calling the tool again, which is what makes retries safe. Schema validation runs before policy evaluation, so malformed arguments are rejected before any authorisation logic executes.

The tool layer is where the "agent over 100+ applications" case joins this one. REST, function calling, MCP (Model Context Protocol) and agent frameworks are layers, not competing technologies, and the test is whether they are treated that way.

 agent framework     coordinates more than one decision across a multi-step task
       ▲
 MCP                 how the model DISCOVERS which functions exist, without 1,500 hardcoded schemas
       ▲
 function calling    the model's mechanism for DECIDING to invoke one thing
       ▲
 REST / SOAP / SQL   what actually EXECUTES

REST is ideal when the interface is stable, operations are deterministic and no reasoning is needed. But a model does not natively understand authentication, endpoint discovery or request schemas, so REST alone never adds up to an assistant. Function calling works while the tool count is small. The model chooses reasonably among twenty functions. Scale to 1,500 functions across 100 applications and it breaks: prompt size explodes, selection gets inaccurate, every schema has to be maintained and re-embedded. MCP solves discoverability, one server per application advertising its tools and resources, so the assistant asks what is exposed instead of hardcoding schemas. MCP does not replace REST; MCP itself runs JSON-RPC over stdio or Streamable HTTP, and an MCP server may call REST, SOAP (Simple Object Access Protocol), SQL or a legacy system underneath. The axis that separates REST from function calling is who decides to make the call: fixed application code, or the model from natural language. Same wire call, two layers, because another REST endpoint is cheap and another function the model must choose between costs prompt space and accuracy.

Agent frameworks earn their place when a request needs multi-step reasoning. "Prepare a renewal report for our top customers and email their managers" means finding customers, retrieving CRM data, generating a report, storing a PDF and sending an email. No single function coordinates that. A single agent suits one tool, a short workflow and deterministic execution. A multi-agent shape suits planning across many domains with independent specialists. "Create a Jira ticket" with an orchestrator would be overhead.

The tool registry is what makes this scale. Loading every MCP server up front does not scale. The registry sits between the planner and the universe of servers and loads only the ones relevant to the current task, keeping the prompt small. In this design the registry also carries the arg schema, the data class and the risk label. That is what lets the policy point decide without asking the model. At 500+ applications and 200,000 employees the rest of the scaling list is short:

• caching of employee profiles, org hierarchy and holiday calendar
• stateless horizontal scaling of planner, gateway and LLM (large language model) routing
• per-user and per-application quotas, with circuit breakers against runaway agents
• async execution that returns a job ID for long workflows
• regional gateways, region-local MCP servers and data-residency controls

The security principle in one sentence: no application trusts the LLM; applications trust enterprise identity. Every request propagates the user's identity, an OAuth token, RBAC (role-based access control) or ABAC (attribute-based access control), and an audit record, and the gateway attaches the identity so no backend ever trusts the model directly. That is identity propagation, and it is the same rule as the credential broker: the model receives context to reason with, never a secret to act with.

The reported Salesforce prompt asks for the loop, the tool interfaces, the memory design, the orchestration and the safety in one architecture, and the shape above answers it. The loop is propose, validate, decide, execute, record, repeat, with a step budget. Tool interfaces are the registry's typed schemas with data class and risk. Memory is the task state store for workflow progress plus the ledger for what happened, and neither is business truth. Orchestration is the planner choosing one bounded action at a time over a queue partitioned by account. Safety is every layer below the planner.

Security starts from the hostile case, not the happy path. A message that looks like ordinary customer email is a prompt injection asking the agent to ignore policy, pull a privileged record and issue an unauthorised refund. The right move is not to debate whether the model understands the instruction. It is to contain the blast radius, preserve evidence and keep every irreversible action behind deterministic checks. The model sees information; it does not inherit trust.

Four controls answer almost any "how do you secure an agent with tool access" question, and they are memorised as a set. Treat tool observations as untrusted. A CRM response, ticket thread, database row or fetched page can carry hostile instructions. If the model can read it, the model can be manipulated by it, so tool output is never a command, a policy override or an approval signal. Never give the model broad, long-lived credentials. A broker mints capability tokens tied to one tenant, one workflow, one action type and one expiry window. If the broker is down, writes fail closed. Validate output before downstream use. The model may propose "refund $5,000", and a $500 task limit rejects it however confident the model sounds. Validation is where policy becomes enforceable. Bound the dangerous dimensions with hard ceilings on tool calls, per-task spend, maximum refund amount, allowable CRM objects and destinations.

Blast radius is a design variable along four axes: tenant, region, workflow and dependency. A single-tenant outage beats a cross-tenant incident. A regional backlog beats a global write failure. A refund-workflow failure should not block email triage. A broker outage should not take down read-only analytics that can safely degrade. Defence in depth means no single control is load-bearing. The model is constrained by policy, the action layer validates output, the broker scopes identity, the gateway tracks idempotency, the ledger preserves evidence, and the operator can kill the workflow. When asked for the single most important control, name all six.

Design the failure path with the happy path. Five rules organise the table. Reads that support a recommendation may degrade or queue. Writes that change customer state fail closed when authorisation, validation or identity is uncertain. Irreversible effects require approval or a second deterministic check. A write is never retried without proven idempotency. Incomplete evidence means stop and escalate.

Replay the path under the one failure that matters most. A malicious instruction arrives inside an email. The planner still extracts a bounded action. The registry marks the email content as untrusted. The policy decision point rejects any instruction that attempts to modify scope. The approval service is never reached. The ledger records the attempted injection. The kill switch can disable the autonomous tool path if the pattern repeats. This is where the control-plane split becomes concrete: the model may read adversarial content in the data plane, but that content must not rewrite policy in the control plane.

Before launch the team needs evidence and a runbook. Evidence shows which prompt version ran and which proposals were generated. It shows which validations passed or failed, which approval artifact was attached, which identity token was used and which external calls were made. The runbook tells an operator how to quarantine a workflow, revoke a capability, replay a safe read-only step and reconcile uncertain writes without making things worse. That record is what answers, after the fact, what happened, what was allowed, what was blocked and what the recovery path was.

The two counts that gate every expansion are unsafe actions and duplicate effects, and both must be zero. High tool success with a rising duplicate-effect count is a deployment blocker, not a product victory.

Report four dashboard layers, because the customer buys workflow improvement under control, not model accuracy. Technical health is tool success, duplicate effects, error rate, queue latency, denial rate. Model quality is task completion, approval rate and delay, and how often a proposed action is later corrected. Adoption is override rate, usage by team, share of eligible requests routed through the agent. Business outcome is time to resolution, refund cycle time, backlog reduction and the customer KPI (key performance indicator) the workflow exists to move. A healthy service can still fail the business. A high denial rate may mean the guardrails work or the schema is too restrictive. A high override rate may mean the model underperforms or policy needs calibration.

Red-team before granting more power, as a release gate rather than a stunt. Feed the system emails, documents and ticket text with adversarial instructions hidden in legitimate content. Ask whether it can be steered into calling tools it should not, leaking context, or skipping approvals. The go/no-go for autonomy has three parts. The obvious attack paths were exercised. The policy layer rejected them. The agent degraded to read-only or human review instead of improvising around the control plane. The red-team list covers five attack families:

• indirect injection in CRM (customer relationship management) records, tickets, database rows and fetched pages
• over-broad credentials that turn one manipulated plan into fleet-wide access
• unvalidated proposals, such as a confident $5,000 refund against a $500 limit
• lost responses that a naive retry turns into a second refund
• stale approvals or action loops that authorise the wrong state or retry into a bill

Autonomy is earned in phases, and the go/no-go at each phase is a control claim, not a model claim.

A practical approval gate includes reviewer identity, decision timestamp, policy rationale and the exact payload that will execute. An approval that ages out expires rather than executing later. A policy change updates the gate before the next rollout step, not after an incident.

Keep a risk register with the shape risk, owner, mitigation, trigger, rollback, and reuse it for any de-risking question. Prompt injection causes an unauthorised tool call: security reviewer; injection tests, allowlists, policy checks, read-only fallback; any confirmed unauthorised attempt; disable autonomous tool calls and revert to human review. Duplicate CRM writes after retries: reliability engineer; idempotency keys, reconciliation jobs, replay-safe contracts; any duplicated side effect; pause write traffic and drain the queue. Refund approvals stall operations: finance operations lead; approval SLAs (service level agreements), escalation path, reviewer training; delay exceeds target or queue depth climbs; narrow the eligible refund scope or route more to manual.

The rollout is also a training and support plan. Users need to know what the agent can do, what it will refuse, how approvals work and how to escalate when a result looks wrong. Support needs a runbook for inspecting logs, finding the policy rule that fired and recovering a stuck workflow. Build the approval service, idempotency layer, policy engine and ledger as shared services, and keep the customer-specific pieces, CRM fields, refund rules, routing and thresholds, in configuration and adapters. The durable product asset is the control plane.

Organise the sketch around trust boundaries, not the model, and spend minutes where the stakes are: identity, authorisation, duplicate execution and malicious instructions. A weak answer covers prompts, vector search, OCR (optical character recognition), channels and regions at once; a better one narrows by risk.

The four trade-offs each have a resolved shape, "give a little here to protect X", and interviewers probe exactly these pairs.

The follow-ups arrive in a predictable order, and each has a prepared answer.

The ninety-second decision memo that closes the round:

"We're building an agent that turns unstructured email into safe operational action, but we are not giving the model unchecked authority. The model handles interpretation and recommendation; deterministic services handle identity, authorisation, idempotency, approvals, audit, and execution. The key trade-off is flexibility versus control: we accept some workflow rigidity so that refunds and CRM writes are gated, logged, and reversible where possible. The first gate is not 'the agent can do everything'; it is that the control plane demonstrably prevents unauthorised actions, duplicate execution, and uncontrolled side effects."

The interviewer will challenge the riskiest assumption: "what if the email itself is trying to steer the agent into a refund it should not make?" Answer calmly and specifically. The email is untrusted input for classification and extraction only. The policy engine outside the model decides whether the request is eligible for any write path. The workflow requires a durable action record plus approval for risky cases. Content that looks like injection or conflicting instructions fails closed.

The two-minute spoken answer, for when the whole design has to fit in a summary:

I would not start with the model. The customer asks for an agent that reads email, queries internal systems, updates CRM records, and issues refunds, and everyone nods at the feature list before disagreeing completely: operations wants speed, security wants hard gates, finance fears irreversible money movement. That disagreement is the design problem. So I would restate the outcome as enabling useful automation while deterministic controls govern identity, permissions, risk, and irreversible effects. The crucial consequence is that the model is not the authority: it can interpret and propose, but deterministic systems decide who may act, what may change, and whether an action can be reversed. Concretely, the planner proposes one bounded action, a typed schema validates the arguments before policy even runs, a deterministic policy engine returns allow, block, or needs-approval with the scopes it grants, a credential broker mints a short-lived token tied to one workflow and one action type, and an idempotent execution gateway guarantees exactly one side effect no matter how many retries occur. Tool observations are untrusted data, never commands, so a prompt injection hidden in a ticket fails at the policy gate rather than depending on the model to notice it. Loops, per-task spend, refund amounts, and destinations all carry hard ceilings. I would roll out read-only first, prove reversibility before enabling any write path, keep money behind a human, and red-team the prompt before granting more autonomy. Success is fewer handling minutes with zero unauthorised or unreviewed irreversible changes.

The lines that carry the round:

1. "The model proposes; it never decides."
2. "Flexible interpretation, deterministic execution."
3. "Tool output is untrusted data, never instructions."
4. "The model sees information; it does not inherit trust."
5. "No application trusts the LLM (large language model). Applications trust enterprise identity."
6. "Validation is where policy becomes enforceable, and it runs before policy does."
7. "A retry without idempotency is how one refund becomes two."
8. "Fail closed on writes. Degrade only the lowest-risk reads. Never fall back to a broad secret to keep the demo alive."
9. "Ceilings on steps, spend, refund amount and destinations are guardrails, not tuning knobs."
10. "The first production gate is that the control plane prevents unauthorised actions, duplicate execution and uncontrolled side effects."

Repair the common weak answers on the spot. Talking about the model too long becomes re-centring the control plane. Optimising for elegance becomes acknowledging the cost of fine-grained scopes and approval queues. Assuming email is trustworthy becomes stating untrusted-input handling early. Describing the kill switch as a UI flag becomes tying it to cancellation, token revocation and downstream rejection. Drowning the interviewer in components becomes saying what ships first and what is deferred.

The pivot after a good design is "the agent is slow and expensive". On this system the dominant driver is agent steps and tool calls, plus retries after failed or rejected actions. An agent that acts must be bounded before it is optimised. The bounds already exist in the design. Cap the steps. Make the common paths deterministic. Cache read-only tool results within the request. Let the approval gate stop the loop. Do not give the loop a bigger model and a longer budget. Prove it with steps per request, tool latency, retry count and cost per request.

Two production scenarios arrive as the same pivot in different words. "The agent keeps calling tools repeatedly": it repeats a CRM (customer relationship management) lookup or retrieval and times out. Ask which tool repeats, what observation triggers it and whether state is preserved. The causes are a poor stopping condition, ambiguous tool descriptions, missing memory, a failed parse or no result cache. Inspect the trace, find the loop trigger and the missing state transition. Now: set max steps, cache the tool result, improve the schema, add a deterministic route. Later: agent budget tests and trace-based loop detection. Prevent with unit tests for tool selection and loop prevention. Say: "I would treat this as a control-loop bug. The fix is step budgets, better tool routing, cached results, and explicit stopping criteria."

"The tool-calling workflow times out": the response fails because an external workflow exceeds the timeout. Ask which tool is slow, whether it can be async and whether the result is needed immediately. The causes are a slow external API, serial calls, no timeout budget or synchronous side effects. Trace the calls and find the critical-path dependency. Now: set tool timeouts, parallelise read-only calls, return a partial answer or an async job. Later: dependency SLAs (service level agreements) and circuit breakers. Prevent with a workflow design review before adding tools. Say: "A tool timeout is not always an LLM (large language model) issue. I would redesign slow side effects as async and keep the user informed."

Every strong cost answer is generated by four verbs in order. Measure, by tracing and attributing first. Route, matching model and path to risk. Bound, with limits on steps, tokens, timeouts and budgets. Cache safely, with tenant, permission and version in the key. Deliver it in six moves: frame the business impact, decompose the path, name the largest measured driver, fix safely, prove with before and after, prevent recurrence. Where the interviewer wants unit economics, the model-time to tool-time ratio decides the lever. Batch reads into one planning pass when planning is expensive. Cache safe reads and prefetch context when tool calls dominate. Tighten the threshold so only high-value cases reach a person when human time is the cost.

The incident on this design is a near miss, and the trap is to say there was no problem because the gateway blocked it. A retail support assistant may summarise orders, draft replies, check policy and recommend refunds. It may not execute refunds above €250 without manager approval. It may only create a proposed refund ticket. On 2026-07-08 an agent asked it to "handle all delayed VIP shipments from yesterday". The planner selected the refund tool for 18 orders, including several above the threshold. The tool gateway blocked execution, so no money moved.

2026-07-08T16:07:51.119Z level=warn service=agent-planner
  trace_id=trc_tool_9031 request_id=req_retail_22018 tenant_id=shopline
  user_role=support_agent user_intent="handle delayed VIP shipments from yesterday"
  planned_tool_calls=18 selected_tool=refund_customer
  policy_doc_version=refund_policy_v3 planner_model=agent_planner_2026_07

2026-07-08T16:07:51.522Z level=critical service=tool-gateway
  trace_id=trc_tool_9031 tool_call_id=tc_77881 tool=refund_customer
  order_id=ord_884120 customer_tier=vip refund_amount_eur=740.00
  approval_state=missing user_role=support_agent max_allowed_without_approval_eur=250.00
  tool_call_blocked=true block_reason=approval_required policy_decision=deny

2026-07-08T16:07:51.800Z level=warn service=agent-safety-evaluator
  trace_id=trc_tool_9031 event=unsafe_bulk_action_near_miss
  bulk_action=true affected_orders=18 blocked_calls=6 allowed_calls=12
  safer_alternative=create_refund_review_ticket
  planner_policy_compliance_score=0.42

Detect from the telemetry: 18 planned calls, 6 blocked and 12 allowed, a €740 refund against a €250 limit with approval state missing, and a planner compliance score of 0.42. The gateway's critical line is enforcement working; the safety evaluator's line is the planner failing. Contain immediately: disable bulk refund planning, force all VIP refund actions into review-ticket mode, and add a user confirmation step for financial actions. Then audit the 12 allowed calls to confirm the smaller refunds were legitimate.

The root cause is a planner prompt, agent_planner_2026_07, deployed to make the agent "more action-oriented", whose examples emphasised completing workflows end to end and omitted bulk-action approval boundaries. The planner over-generalised "handle delayed shipments" into direct refund execution, retrieved the refund policy, and failed to apply the threshold during planning. The gateway policy was unchanged and blocked correctly. The distinction that matters is planner failure versus enforcement success: a near miss, not a completed unauthorised transaction.

Prevent with defence in depth rather than a prompt edit:

• risk-aware tool schemas that encode the approval threshold
• pre-execution policy simulation
• planner training examples for thresholds
• a separate action-risk classifier before tool calls
• hard enforcement kept in the gateway regardless of planner confidence

The same pattern applies to account deletion, data export and permission changes. A malicious user can phrase any of them as operational cleanup.

"This is a near miss. The gateway prevented loss, but the planner attempted unauthorised high-value refunds. I would inspect the planned calls, policy retrieval, and gateway decisions. Then I would disable bulk refund execution, route VIP refunds to review tickets, encode approval thresholds in the tool schema, add policy simulation before execution, and keep the gateway as a non-bypassable enforcement layer."

The weak answer is "the tool was blocked, so there is no problem, I would tell the agent to be more careful". The scorecard rewards four things. Reading planned, blocked and allowed calls with role, amount and approval state. Separating planner failure from gateway success. Auditing plan, policy context, schema and gateway logs rather than editing the prompt. Naming the fraud and unauthorised-action risk.

On this design the levels separate on how far the control plane reaches past a single request, not on how well the planner picks tools.

• The model proposes and never decides, and the disagreement between operations, security and finance is the design problem to name in the first two minutes.
• Requirements are the six must-haves as a set, the constraints stated so a test can fail them, and a component owner for every one.
• Sizing is workload shape and risk: 20 QPS (queries per second) becomes about 200 tool actions per second, and Risk = Impact × Likelihood × Irreversibility sets the autonomy tier.
• One diagram splits the control plane that decides from the data plane that performs, with nine components in dependency order and four trust boundaries.
• One narrated request shows where control moves from the model to deterministic services, and the proposal is never the receipt.
• REST, function calling, MCP (Model Context Protocol) and agent frameworks are layers; the registry keeps the prompt small; no application trusts the LLM, only enterprise identity.
• Four security controls are held as one set: untrusted observations, no broad credentials, validated output, bounded dimensions.
• Writes fail closed and only reads degrade, with injection, lost responses, stale approvals, loops and broker outage each rehearsed.
• The release gate is zero unsafe actions and zero duplicate effects, reported across technical health, model quality, adoption and business outcome.
• Rollout is read-only first, reversible writes second, money behind a human, red-team before power, canary before scale.
• The fifty minutes are organised around trust boundaries, with four resolved trade-offs and the four follow-ups rehearsed.
• The cost pivot is answered by bounding before optimising: step caps, deterministic routes, in-request caches, the approval gate as loop breaker.
• The refund near miss is a planner failure and an enforcement success, prevented by encoding thresholds in schemas and keeping the gateway non-bypassable.

1. Why is "the model refuses" not a security control? Because the model sees information and does not inherit trust; enforcement lives in the policy point, the broker and the gateway, so an injected instruction fails there even if the model is fooled.
2. What makes a retry safe? A canonical argument hash as the idempotency seed and a gateway that returns the stored receipt for a repeated key instead of calling the tool again. Without that, do not retry a write.
3. What does an approval bind to, and why? A snapshot of tenant, record version, amount and expiry, so a change to the record before execution invalidates it and it is re-requested.
4. Where does the policy boundary end? At allow, deny or approve. The execution boundary ends at a scoped, logged side effect in a system of record.
5. Why does validation run before policy? So malformed or over-limit arguments, such as a $5,000 refund on a $500 task, are rejected before any authorisation logic executes, whatever the model's confidence.
6. What happens to writes when the credential broker is down? They fail closed; only reads with a cached, non-sensitive path that policy explicitly allows degrade, and never to a broad static secret.
7. At what scale does function calling break, and what replaces it? Around 1,500 functions across 100 applications, when prompt size and selection accuracy collapse; MCP gives discovery and a tool registry loads only the servers a task needs.
8. Which two counts gate every autonomy expansion? Unsafe actions and duplicate effects, both zero.
9. What is the refund near miss's root cause, and what is its fix? A planner prompt that omitted bulk-action approval boundaries; risk-aware schemas, policy simulation before execution, a risk classifier, and enforcement kept in the gateway.
10. What is the first production gate? That the control plane demonstrably prevents unauthorised actions, duplicate execution and uncontrolled side effects, not that the agent can do everything.

Fact-check sources (checked 27 Sep 2026)

• MCP specification 2025-11-25 – Transports — MCP is JSON-RPC over stdio or Streamable HTTP, not REST
• Claude docs – Tool search tool — selection degrades with many tools; load tools on demand
• OWASP GenAI LLM01:2025 Prompt Injection — indirect injection from any content the model reads