SRE Incident Response Agent

Incident response is: pages go off, someone finds a cause, someone changes production. Wrong rollback at 3 a.m. is worse than a slow summary.

G04 covers one slice: collapse the alert storm, look with read-only tools, then suggest a fix. Humans still approve writes.

End to end, as checkout 5xx after a deploy:

1. 400 alerts hit. Rules or a small model fold them into one investigation.
2. We attach service, last deploy, owner, freeze.
3. We read metrics, logs, traces in parallel and mark empty vs down vs denied.
4. A strong model ranks causes from that pack, with citations and coverage gaps.
5. It proposes a rollback command, blast radius, runbook, dry-run — it does not run it.
6. Owner/commander click yes. One gateway executes once. Timeline records it.

That’s it: dedupe → read → hypothesize → propose → human → maybe write. Auto-restart of prod is later, if ever.

Core idea: Dedupe the alert storm, investigate with read-only tools, show evidence and missing sources, then stage any mitigation for human approval. The agent runs outside the failure it is investigating.

Use this guide for the interview. The Deep Dive contains the technical drills, and the unchanged source case is the full reference.

“The agent may query anything it is authorised to read and change nothing on its own. A ten-second approval is acceptable during an incident; an autonomous rollback based on a wrong hypothesis is not. I’ll optimise for a useful, evidence-backed summary within 30 seconds, even when telemetry is degraded.”

The dependency inversion is the distinctive risk: the observability stack the agent queries may be degraded by the same incident. It must disclose source coverage and distinguish “nothing found” from “source unavailable.”

Questions to ask the interviewer

The source discovery, §1 gives the full set. Ask these before choosing tools:

If the interviewer cannot provide all numbers, state assumptions for throughput, latency, horizon, accuracy, cost, autonomy, data class, and recovery. Name the oracle: post-incident review of the hypothesis and outcome.

The anchor for its incident variants

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Correlate the alert — logs, metrics, traces, deploys, flags, and approved runbooks.
2. Rank hypotheses — evidence, uncertainty, and source coverage.
3. Run read-only diagnostics — only authorised tools, automatically.
4. Propose a mitigation — exact command, blast radius, evidence, runbook, freeze, simulation where possible.
5. Route writes through humans — service owner / incident commander, then one tool gateway.
6. Record the timeline — policy decisions, source gaps, and outcomes for replay.

Non-functional requirements — how well / under what constraints

These are the source’s interview assumptions, not observed load.

First release

One alert class, read-only investigation, staged proposals. No autonomous restarts or rollbacks, no destructive command without a dry run, no action during a freeze without its owner, and no source with unmapped ACL metadata.

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, I dedupe, read telemetry, rank cited hypotheses, and stage a mitigation for approval. Non-functionally, first useful output under 30 seconds in a burst, fail closed on writes, and never hide a dead source.”

This diagram condenses the source architecture, §4.

Step-by-step architecture

• Step 1. Collapse alerts first. Rules or a small model correlate and deduplicate alerts; cap concurrent investigations per service before any strong-LLM call.
• Step 2. Assemble incident context. Add service ownership, recent deploys, and flag changes to the investigation.
• Step 3. Read telemetry within bounds. Query relevant sources in parallel with limits; record whether each result is available, empty, stale, or denied.
• Step 4. Form an evidence-backed view. The strong LLM (large language model) synthesizes ranked hypotheses from bounded telemetry; the agent presents a cited first summary with source coverage.
• Step 5. Prepare a mitigation. If action is warranted, propose the exact command, blast radius, runbook, and simulation result.
• Step 6. Get the right approval. Apply read/propose/execute and freeze policies, then ask the owner and commander; a declined proposal is recorded without a write.
• Step 7. Execute and check. Route an approved command through the authorised tool gateway, execute idempotently, verify its effect, and add the decision to the incident timeline.

Model and agent role: The incident agent uses read-only tools to gather evidence. Rules or a small model handle alert clustering, while a strong LLM synthesizes hypotheses and proposes mitigation. The agent cannot execute a production change without the human approval and tool gateway shown above.

Three boundaries: dedupe before model calls; read-only investigation before approval; all approved writes through one gateway. Keep the agent outside the affected service’s blast radius. The summary states which sources were reached, stale, empty, or unavailable, so a confident hypothesis cannot conceal missing evidence.

Collapse related alerts into one incident before spending model calls. Attach new alerts to an active investigation and cap concurrent runs per service. Build context from service topology, deploys, flags, owners, and freeze status. Query telemetry in parallel with bounded lookback; pre-aggregate log signatures, metric buckets, trace samples, and deploy diffs before they become model tokens. Use the strong model for a small set of ranked hypotheses, not raw-log search.

Every tool returns one typed state: SUCCESS, EMPTY, UNAVAILABLE, DENIED, INVALID. EMPTY means the query worked and found nothing; UNAVAILABLE means the source did not answer. Neither is evidence for the same conclusion. DENIED is a dead end to the model; details go into the human audit, not a workaround hint.

The mitigation is always a proposal first. Compute blast radius from the service catalog/topology, cite evidence and runbook, check change freeze and simulation, then route to the right humans. Approval is bound to the proposal and expires. The gateway authorises before validating arguments so unauthorised callers cannot learn a protected tool schema from validation errors.

At the source’s burst of 400 alerts/90 s, one investigation per alert would overload the telemetry stack and produce noisy summaries. Dedupe/correlation should take under a second; context building ~1–2 s; parallel telemetry fan-out ~5–10 s for the slowest source; strong-model synthesis ~10–15 s in this budget, streamed; verification completes the under-30-second first output. Track time to first useful hypothesis, not only final response time.

At 10×, provider quotas and telemetry rate limits likely bind first. Per-service caps and pre-aggregation keep strong-model calls tied to real incidents rather than alert count. Profile input tokens and serial steps before choosing a faster model. Cache versioned service catalogs/runbooks, shrink lookback and top-k, and parallelise independent read-only calls. Spend is per resolved investigation; it remains visible even when a SEV-1 justifies a larger budget.

Use a visible degradation ladder chosen from system health, with the selected rung in the trace and responder view. In this design, circuit breakers use a windowed failure rate with a minimum sample, because agent traffic is bursty.

The falsifying metric is hypothesis precision, confirmed in post-incident reviews: responders stop reading a tool that repeatedly points them the wrong way. Gate with time to first useful output <30 s under the alert burst, source coverage reported on every run, permission and tool-call safety at zero violations, and review of any harmful proposed mitigation. The source also gives example groundedness ≥90%, citation accuracy ≥95%, high-risk escalation ≥95%, and task completion ≥80%; agree thresholds per alert class with the customer.

Start with historical incidents and one alert class. Build an SME-approved golden set, red-team logs/runbooks/roles, run in shadow mode, then show live read-only summaries with citations and coverage. Add approved low-risk actions only after precision, latency, and safety hold. Pin model versions, canary changes, and roll back on leaks, acted-on wrong hypotheses, missed latency, or responders ignoring output.

For a 45-minute round: requirements and oracle (8), records/action space (4), architecture (8), failure ladder and gates (8), scale/cost arithmetic (7), measurement/rollout (6), questions back (4).

“I’d start with a read-only incident assistant. It deduplicates an alert storm before any model call, builds service/deploy context, queries telemetry in parallel, and presents ranked hypotheses with evidence and explicit source coverage within 30 seconds. Since telemetry can fail during the incident, it distinguishes empty from unavailable and runs outside the affected blast radius. A mitigation is an exact, blast-radius-aware proposal; only the service owner and incident commander can approve a write through a gateway. I’d validate precision against post-incident reviews, shadow it first, then ship live read-only summaries before allowing approved low-risk actions.”

Final mental model: Dedupe → read-only evidence → ranked hypothesis + coverage → proposal → human approval → safe gateway → verified outcome.

The Main guide is the spoken answer. This is its technical backup; the unchanged source case has the complete reference and interview scripts.

Model and agent roles: Rules or a small model collapse alerts before the strong LLM (large language model) synthesizes ranked hypotheses from bounded, read-only telemetry. The incident agent may propose a mitigation, but only human approval plus the gateway permits a write.

The agent runs outside the affected blast radius and assumes any telemetry source may be partially unavailable. Every record needs stable ID, owner, updated time, and ACL (access control list) metadata; exclude an unmapped source. Redact sensitive log fields before context assembly. Feedback stays separate from ground truth until SME/post-incident approval.

Core records: Alert, Signal, Deployment, Runbook, ActionProposal, IncidentTimeline, and SourceCoverage(investigation_id, source, status). Source coverage is part of the answer, not just operational telemetry.

The source’s 1,200 alerts/day average is less important than 400 in 90 s. Group by service and time window; attach new alerts to an active cluster; cap concurrent investigations per service. Rules or a small model do this before any strong-model call. One good investigation per correlated incident beats hundreds of shallow ones and protects the telemetry backend.

Pre-aggregate before prompting: bucket metrics, count log signatures, sample traces, diff deployments, cap lookback. Keep raw slices available through read tools when a hypothesis needs them. The strong model synthesizes ranked causes from a bounded evidence set, with citations and missing-source disclosure. A citation-free cause is a refusal or a low-confidence hypothesis, not a finding.

Use five typed states: SUCCESS (data), EMPTY (query succeeded, no match), UNAVAILABLE (source did not answer), DENIED (caller lacks authority), INVALID (bad arguments). A string result hides the difference. EMPTY versus UNAVAILABLE prevents false confidence and retry storms. DENIED must not reveal a workaround; log the human-readable reason outside model context.

An approved write takes one route: authorise → validate → execute with timeout and idempotency key → classify → trim to declared fields → log. Authorisation comes first to avoid revealing a protected schema to a denied caller. An example idempotency seed is a hash of run ID, tool name, and canonical arguments. Read tools use scoped read-only credentials. Write tools appear only for the current rollout stage and remain behind approval.

An action proposal contains exact command, service/owner, computed blast radius, supporting signals, runbook, change-freeze status, simulation or dry-run result, rollback path, and expiry. A missing blast radius or required simulation blocks the proposal. Service owner and incident commander review; approval and decision time are logged. A freeze owner must approve work during a declared freeze. The model may recommend; the gateway and approvers decide.

The agent’s timeline records alert IDs, queried signals, reached/missed sources, hypotheses, confidence, proposals, approvals, executed command, outcome, model/prompt version, and policy state. That supports post-incident review of both correct and wrong calls.

First useful output target is under 30 s from alert receipt; interactive follow-up 3–8 s. Example slices from the source: dedupe/correlation <1 s, context ~1–2 s, parallel queries ~5–10 s for the slowest source, strong synthesis ~10–15 s with streaming. Pre-aggregation runs as results arrive. Citation verification stays inline for proposals; explanatory text can use an async check if policy allows it.

For a slow prototype, trace auth, retrieval, rerank, model, tools, verification. Parallelise independent reads, cache versioned catalog/deploy context, reduce candidate count, stream useful partials, and remove diagnostics that do not change the hypothesis. Avoid serial reasoning loops in a live incident. At 10× alert volume, correlation load rises but strong-model calls should follow incident clusters, not raw alert count. Provider quota and telemetry rate limits likely bind; source coverage exposes degradation early.

Choose the degradation rung by a deterministic health snapshot, disclose it to the responder, and emit it as a trace attribute. Trip a source breaker on windowed failure rate with minimum sample size; consecutive-failure triggers can misbehave under bursty incident traffic. Permission uncertainty and writes fail closed. Telemetry gaps degrade visibly.

The oracle is post-incident review. Hypothesis precision is the falsifying metric: low precision makes responders ignore the assistant. Also track first useful output, coverage on every run, approval time, later reversal rate, groundedness, citations, task completion, permission violations, unsafe tool calls, and cost/investigation. Example source targets include ≥90% grounded claims, ≥95% citations, ≥95% correct high-risk escalation, ≥80% task completion, and zero permission or unapproved-write violations; agree per-class thresholds before launch.

Red-team wrong-service commands, unsupported causes, leaked secrets/PII, alert noise, injected runbooks/logs, different user roles on the same alert, freezes, and stale contradictory runbooks. Offline cases cannot detect every small regression: shadow live alerts, pin model versions, canary changes, and roll back on disagreement or a harmful proposal.

Week one: one alert class, named source owners, historical incidents and SME (subject-matter expert) labels, no write path. Then shadow summaries, live read-only with citations/coverage, draft timelines or approved low-risk actions, and expansion only while precision, latency, and safety hold. A responder ignoring output is a rollback signal, not merely an adoption metric.

• “Should it restart production?” Read-only investigation → proposed remediation → approved execution → limited autonomous low-risk reversible actions after proof. Never start with auto rollback.
• “What breaks first?” Example: metrics returns EMPTY during a partial outage, assistant infers “nothing wrong” and cites nothing. Distinguish EMPTY from UNAVAILABLE; track citation-free hypothesis rate.
• “18-second prototype?” Decompose the stack, parallelise independent telemetry, pre-aggregate and stream; keep verification for proposals.
• “Too expensive?” Rules/small model for correlation, one strong synthesis, bounded lookback, cached versioned catalog/runbooks. Track cost per resolved investigation.
• “Wrong citation?” Verify against the actual retrieved signal/runbook span; reject or repair before a proposal.

The source retains the full diagrams, 45-minute script, levelling rubric, recovery moves, and references.

Use the Main guide to practice speaking and the Deep Dive for technical follow-ups. The source is the full reference.

Query anything authorised, change nothing autonomously; dedupe first, show evidence and source gaps, stage mitigation for human approval.

Alerts → dedupe/correlate → cap per-service investigations → parallel read-only telemetry → typed coverage → ranked hypothesis + evidence → proposal → owner/commander approval → gateway → verify effect/timeline.

LLM/agent role: Rules or a small model dedupe alerts; the strong LLM (large language model) synthesizes hypotheses. The incident agent proposes mitigation, and humans approve production writes.

Which alert class? Who approves? Which sources and owners? What is read-only? Burst and first-output target? Required evidence/simulation? Pilot oracle? Audit and rollback?

1,200 alerts/day average; 400 alerts/90 s burst; <30 s first useful summary; 3–8 s interactive follow-up. A ~10-second human approval can be acceptable mid-incident.

SUCCESS data · EMPTY searched, no match · UNAVAILABLE source did not answer · DENIED no permission · INVALID bad arguments.

EMPTY ≠ UNAVAILABLE. Every summary reports source coverage. Do not explain a denial to the model.

1. Dedupe before any model call.
2. Read-only investigation before human approval.
3. One gateway for every approved write; run outside affected blast radius.

Exact command + computed blast radius + evidence + runbook + change-freeze check + simulation/dry run + rollback path. Missing approval, permission, or required simulation means no execution.

Telemetry down → report UNAVAILABLE and lower confidence. Alert storm → attach to cluster, visible queue. Stale runbook → warn/block proposal. Injection in logs → data, not instructions. Approval/policy down → fail closed on writes.

Hypothesis precision from post-incident review is the falsifying metric. Track <30 s under burst, coverage on every run, reversal rate, citations, zero unauthorised tools/permission violations. Historical golden set → shadow → live read-only → approved low-risk actions.

Send less raw telemetry: pre-aggregate, cap lookback, parallelise reads, stream ranked hypotheses, reserve the strong model for synthesis. At 10×, telemetry quotas and provider limits bind before average daily alert count.

“The agent buys responders a fast, evidence-backed first view while the observability stack may be failing. It suggests the next safe check or mitigation; humans approve production changes.”

A write-capable agent under a 30-second latency ceiling: fast enough to matter mid-incident, never fast enough to skip the human.

The hard part of this system is not correlating logs with deploys. It is that the agent depends on the same observability stack that is most likely to be degraded during the incident it is investigating. And it can propose changes to production while the people who would catch a wrong hypothesis are the busiest they will ever be. This page consolidates the group's prompts into one read for the day before. Everything else in the group is a delta on it.

Open with the risk profile, not the components. A research agent that is wrong wastes a query; an incident agent that is wrong restarts the wrong service during a payments outage. So the first sentence separates what the agent may query from what it may change, and everything after it is justified by that line.

"The agent may query anything and change nothing on its own. A ten-second human approval is acceptable mid-incident. An autonomous rollback on a wrong hypothesis is not."

The three answer tiers show what that opening buys. The weak answer connects the LLM (large language model) to logs and metrics, lets it find the root cause, and restarts services from runbooks. It treats the model as the security boundary and never separates guidance from action. The average answer builds a triage assistant that queries observability tools and retrieves runbooks, requires approval for actions and evaluates root-cause accuracy. It has the right direction but no numbers, no failure design and no rollout gates. The strong answer designs an evidence-first incident copilot. It never claims a root cause without supporting telemetry. It presents hypotheses, confidence, blast radius, recent deployments and the recommended next checks. Remediation stays approval-gated with service ownership and change-risk validation.

Restate the problem in one sentence, ask for the eight numbers, then name the oracle and cut scope. The eight numbers are throughput, latency, horizon, accuracy, cost, autonomy, data class and recovery. The oracle is the answer to "how will we know a run succeeded". The scope cut is "I'll treat X as out of scope unless you want it in". Close the requirements phase by restating everything as one sentence and asking "is that the system?" It costs 20 seconds and is consistently marked as a strong signal.

Map the people, because each notices a different failure first.

Scope out loud before the first box. Read-only investigation over the observability stack, staged mitigation with approval, one alert class first, and no autonomous production changes. Say the practice prompt's own progression as the roadmap: read-only investigation, then suggested remediation, then human-approved execution, then limited autonomous action only for low-risk, reversible operations.

This OpenAI-style practice prompt lists what an interviewer would expect to hear discussed. Use it as a coverage check on the way out:

• logs, metrics, traces, alerts, tickets and runbooks
• time-window correlation
• service topology
• retrieval of historical incidents
• hypothesis generation
• tool access
• proposed versus executed remediation
• approval workflow
• incident timeline
• auditability
• false-positive cost
• safe rollback

A requirement the responder cannot test is a preference. "Fast during incidents" is a preference; "first useful summary under 30 seconds at 400 alerts in 90 seconds" is a constraint, and only the second one changes the architecture. Split the functional list with MoSCoW so the launch gate is the smallest set that still helps an on-call engineer.

The must-haves are four. Correlate logs, metrics, traces, deployments and runbooks for a fired alert, and propose a likely cause with the evidence that supports it. Execute only safe read-only diagnostics automatically; every write is a proposal with an exact command, its blast radius and the evidence behind it. Apply user, role and service-ownership permissions before retrieval and before generation, and hold only scoped read credentials. Show confidence, missing evidence and which sources could not be reached, because source coverage is an output field, not an afterthought.

The should-haves belong in the first usable version. Draft the incident timeline as events arrive. Ground every step in an approved runbook and cite it. Route proposals to the service owner and the incident commander with change-freeze awareness. Simulate a command before proposing it where the platform allows. Collect responder feedback and post-incident review outcomes for evaluation, never for direct training. Admin controls for tool allowlists, source inclusion, blocked actions and audit export. Later come automatic low-risk reversible actions, historical-incident similarity search, and per-service tuning.

Declare the non-goals, because "tell us what is broken and fix it automatically" is the request and not the first release:

• no autonomous production changes, restarts or rollbacks
• no writes to incident tooling beyond draft timeline entries and Jira creation behind approval
• no destructive commands, even proposed, without a simulated or dry-run result
• no action during a declared change freeze without the freeze owner's approval
• no ingestion of a source that lacks ACL (access control list) metadata

Every must-have then needs an owner in the architecture.

The agent's inputs are the systems that break during incidents, so map each source with how it fails, not only what it holds. A source that is down is a fact the agent must report, never a gap it silently fills.

State the integration assumptions aloud. Every source record has a stable ID, an owner, a last-updated timestamp and ACL metadata. A source that lacks ACL metadata is excluded until it is mapped. Embeddings are never the authority for permissions. Source freshness varies by system, so the summary carries stale-source warnings when a record is older than the approved threshold. Responder feedback is stored separately from ground truth, and post-incident reviews become evaluation data only after approval.

The core records follow from that. Alert(id, service, severity, labels, start_time). Signal(id, alert_id, source, metric or log or trace, timestamp, value). Deployment(id, service, version, commit, deploy_time). Runbook(id, service, symptom, steps, risk_level). ActionProposal(id, action_type, blast_radius, approval_status). IncidentTimeline(event_id, incident_id, timestamp, source). Add one field the answer key does not name and the anchor insists on: SourceCoverage(investigation_id, source, status) with the five typed states.

The dependency inversion is the fact to say while the map is up. The agent depends on the observability stack, which is disproportionately likely to be degraded during the incident it is investigating. So run outside the affected blast radius, treat every telemetry query as potentially EMPTY or UNAVAILABLE, and report which sources could not be reached.

One diagram carries the design, and the sections after it zoom in. The organising split is control plane against data plane. Policy, tool allowlists, credentials, runbook and catalog sync, model versions and evaluation rules live in the control plane; every alert, query, hypothesis, proposal and approval lives in the data plane. The second split is the read-only line inside the data plane: everything to the left of the approval gate may query, nothing to its left may change.

 ╔═══════════════════════════════ CONTROL PLANE (changes are releases) ═══════════════════════════════╗
 ║  tool allowlist + policy (read / propose / execute) · scoped read credentials · service-ownership map  ║
 ║  runbook + service-catalog sync · change-freeze calendar · model + prompt versions · eval rules        ║
 ╚═════════════════════════════════════════╤═══════════════════════════════════════════════════════════╝
                                           │ configures every box below
 ╔═══════════════════════════════ DATA PLANE (runs outside the affected blast radius) ═════════════════╗
 ║                                                                                                       ║
 ║  INTAKE — before any model call                                                                       ║
 ║   alerts (~1,200/day, 400 in 90 s at peak) ─> DEDUPE + CORRELATE (rules, small model) ─> per-service ║
 ║                                                  cap on concurrent investigations                     ║
 ║        │ one investigation per correlated cluster                                                     ║
 ║        v                                                                                              ║
 ║  INVESTIGATION — read-only                                                                            ║
 ║   context builder ─> PRE-AGGREGATE telemetry ─┬─> metrics  ┐                                          ║
 ║   (service, deploys, flags, owners)           ├─> logs     │ parallel, bounded lookback,               ║
 ║                                               ├─> traces   │ typed results: SUCCESS / EMPTY /         ║
 ║                                               ├─> k8s      │ UNAVAILABLE / DENIED / INVALID           ║
 ║                                               └─> runbooks ┘                                          ║
 ║        │ evidence + source coverage                                                                   ║
 ║        v                                                                                              ║
 ║   HYPOTHESIS (strong model) ─> evidence ranker ─> first useful summary (< 30 s, streamed)             ║
 ║        │ ranked causes · confidence · missing evidence · sources not reached                          ║
 ║        v                                                                                              ║
 ║  MITIGATION — staged                                                                                  ║
 ║   proposal: exact command + blast radius + evidence + runbook + freeze check + simulation             ║
 ║        │                                                                                              ║
 ║        v                                                                                              ║
 ║   ══ APPROVAL GATE (owner + commander, ~10 s) ══ ─> TOOL GATEWAY (authorize → validate → idempotent   ║
 ║                                                     call with timeout → classify → trim → log)        ║
 ║        │                                                                                              ║
 ║        v                                                                                              ║
 ║   execute ─> verify effect ─> incident timeline ─> postmortem draft                                   ║
 ║                                                                                                       ║
 ║  OBSERVABILITY — every stage writes                                                                   ║
 ║   trace store (alert → signals → hypotheses → proposal → decision → outcome) ─> eval service          ║
 ║                                 ─> dashboards (time to first output · hypothesis precision · source    ║
 ║                                    coverage · approval decision time · later reversal rate)            ║
 ╚═══════════════════════════════════════════════════════════════════════════════════════════════════════╝

The same flow as a rendered diagram, for viewers that draw Mermaid:

The anchor's own diagram is the four-box version to draw when the board is small. Reproduce it exactly; it is the shape interviewers recognise.

Read the components in dependency order, because that is the order they have to exist and the order they fail.

Three boundaries are worth pointing at while the diagram is up. The model-call boundary sits after dedupe and correlation, so a burst never becomes 400 model invocations. The write boundary sits at the approval gate, so nothing left of it can change production, and the gateway behind it is the single doorway for anything that can. The blast-radius boundary sits around the whole data plane, which runs outside the systems it investigates.

The design point is the burst, not the daily average. 1,200 alerts a day is trivial. 400 alerts in 90 seconds during an incident is not, and an agent that starts 400 investigations produces 400 shallow answers while the metrics store it is querying falls over. One thorough investigation beats 400 shallow ones.

So deduplicate and correlate with rules and a small model before any strong-model call, and cap concurrent investigations per service. A cluster of alerts on one service within one window becomes one investigation. Alerts that arrive while it runs attach to it as new signals rather than opening new runs.

The same rule governs what enters the model's context. Raw logs, traces and deploy diffs are input tokens, and under a latency ceiling the win is in what is not sent. Pre-aggregate telemetry first: bucket the metric, diff the deploy, count the log signature, sample the trace. Bound the lookback window. Send the summary and keep the raw slice reachable by a tool call if the hypothesis needs it. Reserve the strong model for hypothesis synthesis and use smaller models or rules for clustering and correlation.

Four decisions matter more than the diagram, and the first is the whole risk model. Read-only investigation, staged mitigation: the agent may query anything and change nothing on its own. A ten-second human approval is acceptable mid-incident. An autonomous rollback on a wrong hypothesis is not.

A proposal has a fixed shape: the exact command, the blast radius, the evidence, the runbook it follows, the change-freeze status, and a simulation or dry-run result where the platform allows one. A proposal without a blast radius is not a proposal. Route it to the service owner and the incident commander, record the decision and the decision time, and let it expire when the incident window closes.

Give every tool a typed result with five states, because a string return cannot tell the agent what happened. SUCCESS carries the data. EMPTY means the query ran and nothing matched. UNAVAILABLE means the source is down. DENIED means the caller may not. INVALID means the arguments were wrong and says what to fix. Mixing up EMPTY and UNAVAILABLE can cause retry storms during a real outage, which is exactly when this agent runs. Mixing up DENIED and INVALID teaches the agent to work around policy instead of respecting it.

Never explain a denial. A DENIED that says "restarts of this service need commander approval" can read to the model as a hint, and the next attempt may be the same restart phrased as a scale-to-zero. Denials are a dead end to the model, with the reason logged for humans.

Every write, once approved, walks through one gateway in one order. Authorise before validating, otherwise a denied caller learns the tool's schema from the validation error. Validate the arguments against the schema. Execute with a timeout and an idempotency key, which is a hash of run id, tool name and arguments, so a retry after a timeout runs the restart once. Classify the result into the five states. Trim it to the declared fields before it returns. Log all of it.

Hold credentials to match. Read tools run with scoped read-only credentials. Write tools exist only behind the gate and only for the allowlisted, reversible actions of the current rollout stage. The blast radius on a proposal is computed from the service catalog and topology, not estimated by the model, and a rollback path is part of the proposal or the proposal is refused.

Thirty seconds is a ceiling, not a target. A first useful summary must land inside it while 400 alerts arrive, because I'd expect a responder who has already started typing a manual query to stop reading the agent. Say the budget in slices and where each second goes.

Dedupe and correlation are rules, so they cost under a second. Context building reads a cached service catalog, deploy metadata and flag history in parallel, another second or two. Telemetry queries fan out in parallel with bounded lookback windows and a per-query timeout, budgeted at five to ten seconds for the slowest source, with UNAVAILABLE returned rather than waited for. Pre-aggregation runs on the query results as they arrive. In my budget, hypothesis synthesis on the strong model is the largest slice, ten to fifteen seconds, streamed so the first ranked cause appears before the last. Citation and evidence verification runs inline for anything that will become a proposal, and asynchronously for explanatory text.

The mock's objection is the one to rehearse: the prototype takes 18 seconds and users will not adopt it. The answer is not a faster model. Break the 18 seconds into authentication, retrieval, reranking, generation, tool calls and verification, then fix the slice that dominates. Parallelise independent queries. Cache embeddings and the service catalog. Reduce top-k before reranking. Stream partial responses. Route low-risk requests to a smaller model. Remove tool calls that do not change the hypothesis. Keep citation verification. It is a safety feature, and it can be made asynchronous for low-risk explanatory answers without being removed for proposals. Avoid long chain-of-thought loops in a live incident; a bounded set of parallel checks beats a long serial reasoning chain.

Partial dependency degradation is the normal operating condition here, not an edge case, because the agent's inputs are the systems that are failing. So the failure table is organised by one rule: authorisation and writes fail closed, and every telemetry gap degrades visibly with the gap disclosed.

Instrument the ladder. A degradation ladder is an ordered set of rungs from full capability to honest refusal, selected by a pure function of a health snapshot. The selected rung is disclosed to the responder and emitted as a span attribute. In this design, a breaker per telemetry source trips on a windowed failure rate with a minimum sample size, not on a run of consecutive failures. Agent traffic is bursty, so a consecutive-failure trigger tends to trip too often or too late.

Then say what breaks first at 10×. The binding constraint is provider quota and the telemetry sources' rate limits during a storm, so the per-service cap and the pre-aggregation are what scale, not the model. Ten times the alerts means ten times the correlation load on rules and small models, which is cheap, and the same number of strong-model calls per real incident. Source coverage per investigation is the metric that shows the sources falling behind before the responders notice.

The falsifying metric is hypothesis precision, confirmed in post-incident review. Below a threshold, I'd expect responders to stop reading the output, and I'd argue an ignored incident tool is worse than none because it still consumes attention during the incident. Measure that first and gate on it.

Red-team the boundary with the attacks specific to this system:

• an unsafe remediation command proposed against the wrong service, or above its blast radius
• a wrong root cause stated with confidence and no telemetry behind it
• secrets or PII (personally identifiable information) leaking from log lines into the summary or the timeline
• alert fatigue: an agent that adds noise instead of collapsing it
• prompt injection planted in logs, runbooks, tickets or comments that instructs the model to ignore policy
• the same alert queried by users with different roles and service ownership
• unsafe automation attempts: restart, scale, flag flip, rollback without approval, or during a freeze
• staleness attacks where an old runbook contradicts the current one

An offline gate of a few hundred cases usually can't detect small regressions with confidence, so the rollout itself becomes the detector: shadow evaluation on live alerts with disagreement rate as the signal, then a canary with guardrails and automatic rollback. Pin model versions, evaluate before adopting a new one, and keep a second provider behind an adapter.

Week one at a customer is not the whole diagram. It is one alert class, the telemetry sources with named owners, a golden set built from the last quarter's incidents, and a shadow run against history that proves hypothesis precision before anyone sees a proposal.

State the rollback conditions before the pilot: a leak, a high-risk wrong hypothesis acted upon, latency over the ceiling during a real incident, or responders ignoring the output. The practice prompt's follow-up asks whether the agent should restart production services automatically, and the answer is the progression above: read-only investigation, suggested remediation, human-approved execution, then limited autonomous action only for low-risk, reversible operations.

Time is the binding constraint. Twenty minutes on requirements produces an incomplete design; two minutes produces a generic one. Rehearse against a clock until the pacing is automatic.

Miss the first 8 minutes and every later answer is guesswork. Miss the last 5 and the levelling signal is forfeited. Volunteer the safety analysis before being asked; naming the security boundary unprompted reads as senior-level threat modelling, not box-checking.

The two-minute spoken answer, adapted from the answer key:

I would not start with the model. I would start by clarifying the broken incident response workflow, who uses the system, what decision they need to make, and what risk we cannot automate. For an SRE incident agent I would design a permission-aware assistant around the workflow: an alert fires, the agent deduplicates and correlates before any model call, correlates logs, metrics, traces, deployments and runbooks, proposes the likely cause with the evidence, drafts the incident timeline, and executes only safe read-only diagnostics unless a human approves. It ingests approved sources such as Prometheus, Grafana, Datadog, logs, traces, Kubernetes, CI/CD (continuous integration and continuous delivery), feature flags, incident management and the runbook repository, preserves metadata, freshness and ACLs, and pre-aggregates telemetry before it becomes tokens. It runs outside the affected blast radius and treats every telemetry source as possibly unavailable, reporting what it could not reach. The model produces cited hypotheses with confidence and escalates when evidence is missing. Tool use is allowlisted: read-only tools run automatically, and any write is a proposal with an exact command and blast radius that a human approves in about ten seconds. I would evaluate with hypothesis precision from post-incident review, time to first useful output under thirty seconds, source coverage, approval decision time and the reversal rate on approved mitigations. Rollout is staged: shadow summaries on history, live read-only summaries, then approved low-risk actions, never auto-remediation first. The goal is not a demo; it is an incident tool responders keep reading.

The lines that carry the round:

1. "Read-only investigation, staged mitigation. Query anything, change nothing on its own."
2. "A ten-second approval is fine mid-incident. An autonomous rollback on a wrong hypothesis is not."
3. "The agent depends on the observability stack that is most likely degraded during the incident it is investigating."
4. "Empty and unavailable are different facts. Source coverage is an output field."
5. "Deduplicate and correlate before any model call. One thorough investigation beats 400 shallow ones."
6. "The falsifying metric is hypothesis precision. An ignored incident tool is worse than none."
7. "Under a latency ceiling the win is in what you do not send."
8. "Trip circuit breakers on a windowed failure rate with a minimum sample, not on a run of consecutive failures."

The follow-ups arrive from a known bank, and each has a prepared shape.

Cover the whole design at consistent depth first, then offer depth explicitly: "I can go deeper on the evaluation layer or the cost model, which is more useful to you?" Candidates who go deep unprompted pick the component they know best. They run out of time before failure and measurement, which is where the levelling signal lives.

Recover a round that has gone sideways with a named move for each failure.

Interviewers respect explicit prioritisation and penalise silent overruns. The failure mode is not running out of time; it is running out of time without saying so.

Repair the common weak answers on the spot. "Connect the LLM (large language model) to logs and let it restart services from runbooks" becomes read-only investigation with staged, approved mitigation. "Put the user's role in the prompt and tell the model not to reveal restricted data" becomes permissions enforced at retrieval and at the gateway. "Use a faster model if it is slow" becomes a decomposed latency budget with parallel queries and pre-aggregation. "Test a few examples manually" becomes a golden set from historical incidents, red-team cases and hypothesis precision from review. "Optimise cost later" becomes tiered routing and a cost per resolved investigation from day one.

Ask them something at the end:

• Which telemetry source is least reliable during an incident today, and does anyone measure that?
• How are service ownership and change freezes represented, and who can approve a mitigation at 3 a.m.?
• What did the last post-incident review say about the tools responders actually used?
• Where do platform-generic and customer-specific stop in an incident deployment?

The interviewer's pivot after a good design is "it misses its 30-second ceiling." Answer it in the same sitting, on the same architecture, with the self-drill card.

Every strong cost or latency answer is generated by four verbs in order. Measure, by tracing and attributing first. Route, matching model and path to risk. Bound, with limits on steps, tokens, lookback windows, timeouts and budgets. Cache safely, with the service catalog, runbooks and deploy metadata keyed on version. Deliver it in six moves: frame the impact, decompose the path, name the largest measured driver, fix safely, prove with before and after, prevent recurrence.

The levelling rubric describes what to say, not who anyone is.

• The read-only boundary is named in the first sentence: query anything, change nothing on its own, because a wrong autonomous rollback mid-incident is the failure that ends the project.
• Requirements are stated as numbers a replay can fail: 30 seconds to first useful output at 400 alerts in 90 seconds, four must-haves, and a non-goals list that excludes auto-remediation.
• Every telemetry source is mapped with how it fails during an incident, and the agent depends on the stack that is most likely degraded.
• One end-to-end diagram puts the model-call boundary after correlation, the write boundary at the approval gate, and the whole data plane outside the blast radius.
• Deduplicate and correlate before any model call, and pre-aggregate telemetry so a burst never becomes 400 investigations or a million tokens.
• Investigation is read-only and mitigation is a typed proposal with an exact command, blast radius and evidence, executed only through one gateway after approval.
• The latency arithmetic is said aloud in slices, and the fix for a slow prototype is decomposition, parallelism and streaming, never a faster model.
• Telemetry gaps degrade visibly as EMPTY or UNAVAILABLE with coverage disclosed; authorisation and writes fail closed; breakers trip on windowed rates.
• The release gate is hypothesis precision from post-incident review, because an ignored incident tool is worse than none.
• Rollout is shadow summaries, then live read-only, then approved low-risk actions, never auto-remediation first.
• The forty-five minutes are paced by the script, the safety analysis is volunteered, and the levelling signal lives in failure and measurement.
• The latency pivot is answered by sending less: pre-aggregate, parallelise, and spend the strong model on the hypothesis only.

1. Why is a ten-second approval acceptable but an autonomous rollback not? The approval costs ten seconds of a responder's time; a rollback on a wrong hypothesis changes production during the incident and can make it worse, and the humans who would catch it are at their busiest.
2. What is the dependency inversion and what does it force? The agent depends on the observability stack that is most likely degraded during the incident it investigates. So it runs outside the blast radius, treats every query as possibly EMPTY or UNAVAILABLE, and reports which sources it could not reach.
3. Why deduplicate before any model call? 400 alerts in 90 seconds would otherwise become 400 shallow investigations; one thorough investigation per correlated cluster, capped per service, is the design point.
4. What is the falsifying metric? Hypothesis precision, confirmed in post-incident review. Below a threshold responders stop reading, and an ignored tool still consumes attention.
5. What breaks first, in the strong shape? The metrics store returns empty during a partial outage, the agent treats empty as valid, ships a confident hypothesis with no evidence; the signal is citation-free hypothesis rate; the fix is a typed result distinguishing empty from unavailable.
6. Why never explain a denial? The model can read the reason as a hint and look for a workaround, such as splitting one refund into two or phrasing a restart as a scale-to-zero. Denials are a dead end; the reason is logged for humans.
7. What is in a proposal? The exact command, the blast radius, the evidence, the runbook, the change-freeze status and a simulation result, routed to the service owner and the commander.
8. Why does the gateway authorise before it validates? Otherwise a denied caller learns the tool's schema from the validation error it receives.
9. What is the latency answer to an 18-second prototype? Decompose the budget, parallelise independent queries, cache the catalog and embeddings, reduce top-k, stream partial output, route low-risk requests down, and keep citation verification, made asynchronous for explanatory text only.

Fact-check sources (checked 27 Sep 2026)

• Grafana roles and permissions — Grafana org, team and folder permissions
• Prometheus security model — Prometheus has no fine-grained authorisation
• Datadog API rate limits — query APIs are rate-limited
• AWS Post-Event Summary, us-east-1, 7 Dec 2021 — monitoring degraded during the incident it should observe
• OWASP GenAI LLM06:2025 Excessive Agency — limit what an agent can do around a denial
• OWASP Authorization Cheat Sheet — check authorisation first on every request
• Resilience4j – CircuitBreaker — windowed failure-rate breaker with a minimum call count
• Miller 2024, Adding Error Bars to Evals — small eval sets can't detect small differences
• MSRC: How Microsoft defends against indirect prompt injection attacks — prompt defences reduce risk but are not boundaries
• OWASP GenAI LLM01:2025 Prompt Injection — prompt-level defences are probabilistic