Middleware, including human-in-the-loop

What you'll learn

• Explain where middleware hooks run in the agent loop, and how they show up in the compiled graph.
• Compare message, token and context-fraction triggers for summarising history.
• Trace an approval through approve, edit and reject, and what each leaves in the message list.
• Diagnose the provider error that a new message on a paused thread causes.

Lesson 2 of T7.2 capped the loop with one middleware. This lesson opens the mechanism up and adds the guard every agent with a write tool needs: a person who can stop the call.

Larkspur's support agent may look up an account on its own, but sending the customer an email or issuing a refund must wait for a support engineer. The loop from lesson 2 of T7.2 has no place to wait. Rebuilding it as hand-wired graph nodes for every guard would mean rewriting the agent each time a rule is added.

Middleware attaches hooks at fixed points of the agent loop: before the model call, after it, and around a model or tool call. Teams use it for observability (logging, timing), transformation (rewriting the prompt, filtering tools), control flow (retries, early exit) and guardrails (call limits, redaction, human approval).

Hooks are graph nodes

Each hook becomes a node of the compiled graph, named after the middleware and the hook. Listing the nodes of three recorded agents shows where each one sits:

Table 1: Hook nodes added to the model and tools nodes, recorded graphs.

The built-ins used in this lesson:

Table 2: Four built-in middlewares.

Middleware runs in the order you list it. A production stack of redaction, a call limit and a custom timing logger produced its hook nodes in exactly that order.

Summarising history: three triggers

Summarisation needs a trigger, the point at which it fires, and a keep setting, how much recent history survives verbatim. Each can be counted in messages, tokens or a fraction of the model's context window. Three recorded runs, with thresholds set deliberately low so they fire within a few turns:

Table 3: History size per turn under each trigger. Token figures are a display estimate of characters divided by four.

• A message count is predictable but a poor proxy for pressure: ten short arithmetic turns and ten long tool results count the same.
• Tokens track what actually runs out. The token run's history collapsed twice as long hotel listings piled up. The table's estimate is rough, so it does not show the exact point where the middleware's own counter crossed 550.
• A fraction keeps its meaning when the model changes. Seven tenths of the window means the same on a 128,000-token model and on one with a million; a hard-coded token count does not. The demo's tiny fractions only make it fire in six turns.

A checkpointer keeps history across invocations, which is what gives summarisation a long history to work on and what lets an approval pause survive. Within one run, summarisation works on whatever history the run has.

Planted failure: over-redaction

Email redaction ran on a user message and on a tool result. The recorded trail:

Table 4: Redaction before and after the model call.

The model never saw either address, which is the point. But the user asked for the email on file and received a placeholder. We call this over-redaction: a redaction rule that covers a field the user is entitled to see. Decide per field whether the model needs it, and return entitled data to the user through code that bypasses the model.

Writing your own

A custom middleware subclasses the agent middleware class and overrides the hook it needs. Two rules are easy to break, and both fail quietly or late:

• Every hook takes the state and the runtime. A hook defined without the runtime argument raises a type error when the agent runs, not when it is built.
• A hook returns a state update; it never edits the state it was given. To hand a value from before the model to after it, declare the key on a custom state schema and return it. The recorded timing logger did this and reported 1.15 s and 0.81 s for the two model calls of one run.

To end a run early, a hook declares that it may jump to the end and returns that jump with a final message, as the call-limit middleware does. The T7.2 lab's off-topic guard uses the same shape.

The approval middleware turns a consequential tool call into a decision. Gate writes, such as sending email or issuing a refund. Authorised reads can run freely, though a sensitive read may still need access control.

Figure 1: An approval round trip. The thread is checkpointed at the interrupt and nothing runs until the app resumes it.

The pause is a real suspension. The state is saved under the thread id, the agent returns an interrupt naming the pending calls, and the app resumes with one decision per gated call. The three decisions, from the recorded trails:

Table 5: Three decisions on the same send-email request.

The edit case started from a request with a wrong address. The reviewer replaced the recipient, subject and body, and the model's answer reported the reviewer's values, because the tool message told it what had happened. Prefer edit to reject when the call is right in kind and wrong in detail.

Planted failure: the orphaned tool call

When the middleware interrupts, the AI message carrying the tool call is already saved on the thread. If the app sends a new user message on that thread instead of resuming it, the history becomes:

Table 6: The history after a new message on a paused thread.

The provider rejects the next model call (OpenAI, error 400): an assistant message with tool calls must be followed by tool messages responding to each call id, and the error points at index 2. We call this the orphaned tool call. The only valid next step on a paused thread is a resume with decisions. If the user wants a separate conversation, start a new thread.

1. Gate the email and refund tools with human approval; leave the account lookup ungated so approvals stay rare.
2. Show pending interrupts per thread in the support console and block new messages on that thread until a reviewer decides.
3. Trigger summarisation on a fraction of the context window, with a checkpointer for every conversation.
4. Redact payment details before the model, and decide per field what a customer may see in answers.
5. Keep the model-call limit from lesson 2 of T7.2, sized from the golden questions, on the same agent: it bounds whatever the other hooks let through.

• Toy tools. The email, hotel and account tools are stubs; the approval trails show the mechanism, not a review workload.
• Rough token counts. Table 3's tokens are characters divided by four, for display only.
• One version. Hook names and node names come from the installed LangChain 1.x; they have changed between releases.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Middleware hooks run before, after or around model and tool calls, appear as nodes in the compiled graph, and run in the order you list them. A custom hook takes the state and the runtime and returns a state update.
• Trigger summarisation on tokens or a fraction of the context window rather than a message count, and redact only what the model does not need, or the user loses data they are entitled to.
• Gate consequential writes with human approval and resume a paused thread with one decision per call. A new user message on that thread orphans the tool call, and the provider rejects the next request.