Distributed execution: think like a staff engineer

Track accepted jobs, queue age, claim latency, lease expiries, attempts, unknown/dead counts, tool rejection, and completion age. Separate acceptance latency from job turnaround. Metadata-only traces join request ID, job ID, and attempt epoch; avoid storing source text or tokens in ordinary logs. Exporters and alerts are proposed controls, not installed by the fixture.

When oldest-job age rises, first compare arrival and completion rates. Check available workers and provider quotas, then inspect DB lock time and slow tools. A growing queue with idle workers suggests claim/configuration failure; busy workers at a dependency limit need demand control rather than more claimers. T18.6 will teach those admission/fairness controls in depth.

For a dead job, fix the poison input/tool bug before replay. For unknown work, preserve its operation identity and provider evidence; do not delete the row to make dashboards green. Cancellation should prevent unstarted actions and report whether a started action remains uncertain. This slice does not implement cancellation, heartbeat renewal, an outbox, or provider reconciliation. Deployment also needs durable storage, backup/restore, real identity, process supervision, and bounded tool networking.

Prompt: A teammate says async solves the 20-second task. Defend the new path.

Spoken answer: “Async lets a process overlap waiting work; it does not make a task survive that process. The customer needs a durable identifier and eventual inspection after a restart. I commit a job before acknowledging acceptance, then let a worker claim it with bounded authority. I keep the original short drafting route. The extra queue state costs operations, but it addresses lost work rather than merely reducing blocked threads.”

Trade-off: More recoverable state versus simpler request handling. Changed constraint: If all tasks take 100 ms and can be safely abandoned, when would you remove this queue?

Prompt: Result commit succeeds but delivery acknowledgement is lost.

Spoken answer: “The delivery can repeat, so I use the saved job state and scoped key to avoid repeating already completed work. A stale epoch cannot overwrite current state. None of that proves a separate provider executed once: a crash after its call but before our commit leaves uncertainty. I expose unknown state and require a reconciliation policy before retrying that boundary.”

Trade-off: Conservative unknown states need operator work. Changed constraint: The provider supports a documented durable idempotency key; explain what guarantee it adds and its retention limits.

Prompt: Six arrivals/minute, 20 seconds each, two workers. Is capacity adequate?

Spoken answer: “Two workers yield six jobs/minute only under constant runtimes and no overhead. That is full nominal utilisation, with no margin for bursts or retries. Three workers give nine jobs/minute in the simplified model. I would validate the runtime distribution and provider budget before scaling. Polling and persistence can independently saturate the API/database.”

Trade-off: Headroom costs capacity. Changed constraint: If 80% of time is CPU parsing, compare thread concurrency with process workers and memory limits.

Prompt: The model suggests sending the supplier email as a fourth tool call.

Spoken answer: “The current customer requirement keeps sending with Alice. Trusted code rejects that tool regardless of model instructions. The workflow has a three-step budget, scoped inputs, and durable progress. A new product requirement for sending would need a separate approval and external-effect contract. A checkpoint alone cannot prove whether an irreversible call happened.”

Trade-off: Bounded workflows sacrifice open-ended autonomy. Changed constraint: If approval arrives after the job deadline, what must be revalidated before resuming?

• When the oldest job's age rises, compare arrival and completion rates first. A growing queue with idle workers suggests a claim or configuration failure, while busy workers at a dependency limit need demand control rather than more claimers.
• Fix the poison input or tool bug before replaying a dead job. Keep an unknown job's operation identity and provider evidence instead of deleting the row to make dashboards green.
• async overlaps waiting work but does not make a task survive its process. Committing the job before acknowledging acceptance and claiming it with bounded authority addresses lost work, at the cost of more queue state to operate.
• Saved job state and the scoped key stop a repeated delivery from redoing completed work. They cannot prove a separate provider ran once, because a crash after its call but before the commit leaves uncertainty.