Interview Guide · 8 min — Start here. The answer you would talk through in the interview, readable on its own.
Delivery process is: customer wants an agent, you ship something real in two weeks, with a named person who can say stop.
G12 covers one slice: intake → gates → a small live agent → rollback. Not “we’ll figure production out later.” The source uses a Northwind support-triage demo.
End to end, as Northwind tickets:
- Intake: success is “first response under five minutes,” not “launch an agent.” SME (subject-matter expert) and sources named now.
- Security and data access — Zendesk + Confluence + Salesforce read. If that’s a six-week ticket, two weeks is a lie.
- Configure, golden set, baseline eval.
- Shadow, then a tested rollback (kill send in two minutes).
- Limited production. Humans still send. Autonomy is earned.
- Sponsor go/no-go and handover. A failed gate does not become a quiet exception.
That’s it: name the outcome → prove each gate → ship small → own rollback. Open-ended custom planners wait.
Source: Sources and Full Design. The Deep Dive covers gate mechanics and limitations; the Cheat Sheet is for rehearsal.
The case in one sentence
Turn a customer request into a governed, two-week agent deployment with measurable value, named approvers, evidence-backed gates, a working rollback and a clear owner. The source demonstrates this with a Northwind support-triage agent; its day-14 numbers are scripted demo results, not live production evidence.
Questions to ask the interviewer
| Question to ask | What it's really asking | What you then decide |
|---|---|---|
| What customer outcome and baseline define success by week four? | Are we cutting first-response time from 40 minutes to 5, or just “launch an agent”? | Whether the build has a measurable goal. |
| Who is the sponsor, subject-matter expert and go/no-go owner? | If the Confluence owner is on vacation, who can still say ship or stop? | Approvals and escalation. |
| Which data sources, credentials and policies are available now? | Do we already have Zendesk + Salesforce read access, or is that a six-week ticket? | Whether two weeks is even possible. |
| Which accelerator parts can be reused, and what is bespoke? | Can we reuse the support template, or are we writing a custom planner? | Delivery time and later maintenance. |
| What happens when a gate fails or an SME misses a deadline? | If golden-set review slips two days, do we ship anyway? | No-go, escalation, and schedule impact. |
| Which agent actions require human review, and how is rollback tested? | If drafts go weird, can we kill send in two minutes? | Production autonomy and containment. |
Do not start the engagement clock without measurable success metrics, a named SME and named sources. The source uses those as intake requirements.
Requirements: Functional + Non-Functional
The easiest way to frame requirements in an interview is:
Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.
Functional requirements — what the system must do
Delivery
- Intake and scope — measurable outcome, named SME, named sources.
- Evidence-backed security and data access.
- Configure the agent and sign a golden set.
- Baseline evaluation, shadow, tested rollback.
- Limited production, sponsor go/no-go, handover — runbook, dashboard, owner.
- Advance only after each gate — authorised role and evidence recorded.
Northwind agent (demo)
- Triage the top three Zendesk categories.
- Use Confluence plus Salesforce read-only context.
- Redact sensitive data; draft grounded replies; apply escalation rules.
- Keep people in control of sending while autonomy is earned.
Non-functional requirements — how well / under what constraints
| Requirement | Example target / constraint |
|---|---|
| Schedule | Two-week delivery target. |
| Governance | Auditability, separation of duties; failed gate stops advancement. |
| Reuse | Accelerators over bespoke planners. |
| Security | Tenant isolation, secure credentials. |
| Reliability | Reliable escalations; demo rollback under two minutes. |
| Value | Measurable first value. Northwind demo: first response <5 min; ≥60% zero-edit sends by week four. |
Interview shortcut
If asked “What are the requirements?”, say:
“Functionally, gated two-week delivery plus a small triage agent that drafts, not sends. Non-functionally, two weeks, tested rollback, and a number for first-response time — not ‘launch an agent.’”
Architecture: delivery gates and the deployed agent
The delivery state machine is deterministic; an LLM (large language model) does not approve its own release. Within the delivered support agent, model calls can classify and draft replies. The latter architecture is a source-grounded design sketch, not a claim that every runtime component was implemented in the source demo.
Step-by-step architecture
- Intake records target outcome, baseline, sponsor, SME and data sources; incomplete intake does not start the delivery clock.
- Security and data-access reviewers sign their own gates with evidence before the agent touches customer data.
- The team pulls reusable connectors, prompts, evals and dashboards, then builds only the customer-specific escalation policy; the SME signs the golden set.
- A baseline evaluation tests quality and safety. Shadow mode compares agent behaviour without granting unrestricted sending, and the team proves rollback.
- Limited production measures agreed success metrics. The sponsor makes the day-14 go/no-go decision; the team hands over a runbook, dashboard, owner and credentials plan.
- At runtime, a Zendesk ticket is redacted, classified by a model, enriched with scoped KB/CRM context, drafted by an LLM, checked for grounding and escalation, and sent only under the chosen human approval policy.
Gates, evidence and ownership
| Stage | Source gate | Signer |
|---|---|---|
| Scoping, days 1–2 | security_review_passed |
Security reviewer |
| Data readiness, days 3–4 | data_access_granted |
Customer SME; pending access escalates on day 3 |
| Configure, days 5–7 | golden_set_signed_off |
Customer SME |
| Evaluate, days 8–9 | eval_baseline_met |
FDA (Forward Deployed Architect) |
| Shadow, days 10–11 | rollback_tested |
FDA |
| Limited production, days 12–13 | success_metrics_met |
Sponsor |
| Day 14 | Go/no-go and handover | Named decision owner |
The gate API should reject wrong roles, missing evidence and skipped prior stages; deny decisions override optimistic state and every attempt is logged. Free-text evidence is weak: require structured checklist items and links to evaluation or rollback proof. A no-go is a valid outcome.
Trade-offs, cost and honest limits
Northwind reuses five of six accelerator components (about 83%); the bespoke part is escalation policy. The demo script hard-codes 4m12s first response, 63% zero-edit sends, a .83 evaluation score against a .75 baseline, and a rollback note saying under two minutes; none of these was measured or timed. Week-four retention was not observed; None does not mean zero. The source's deterministic gate demo has 17 tests but does not implement full agent provisioning, eval-harness wiring, scope-change approval, versioned rollback or multi-engagement operations. State those gaps plainly in the interview.
For cost, agree on cost per resolved ticket at intake; bound model steps and tokens, reuse common connectors, and sample evaluation proportionally to risk. Keep customer credentials in a vault with scoped access and revoke or transfer them at handover. Record tenant boundaries and override rate. A low-cost pilot that fails its outcome metric is not success.
Evaluation and rollout
Golden tickets must cover the top categories, retrieval misses, PII (personally identifiable information), policy escalations and cases where a draft should not be sent. Measure response time, zero-edit sends, groundedness, escalation correctness, human overrides, rollback time, cost per resolved ticket and week-four retention when it actually exists. Shadow before limited production; refuse broad rollout until sponsor-approved metrics and rollback evidence hold.
Two-minute interview answer
“I start with a measurable business result, a named SME and real data sources; without those, the two-week clock is fiction. I run seven gated stages from scoping and security through data access, configuration, evaluation, shadow, limited production and sponsor go/no-go. Each gate needs evidence from the right owner and cannot be skipped. For the support-triage agent, a ticket is redacted, classified, enriched from read-only knowledge and CRM (customer relationship management), drafted by an LLM, checked for grounding and escalation, then human-approved as needed. I reuse standard components, measure first response and zero-edit sends, prove rollback, and hand over ownership. If a gate fails, I stop or escalate rather than calling an unverified demo deployed.”
Technical Deep Dive · 4 min — Read after the Interview Guide. The detail behind each component, for follow-up questions.
Read with the unchanged source. The Main guide is the interview route; the Cheat Sheet is the recall card.
1. Two distinct systems
The primary system is a deterministic delivery state machine: intake, gated progress, sign-off, evidence and handover. The second is the customer support-triage agent that the process delivers. An LLM (large language model) helps classify and draft support replies, but it does not adjudicate its own security review, eval baseline or production approval. The source's agent runtime diagram is a design construction; the gate demo is the part actually exercised by 17 deterministic tests.
2. Entry contract and stages
Intake requires a measurable target and baseline, named SME (subject-matter expert) and named sources. Ask whether the two-week SLA (service level agreement) is a target or commitment; who owns go/no-go; what is reused; what a failed gate means; and whether scaling is per team or across an organisation. These answers change staffing, access and the schedule. A missing SME or data source is an intake failure, not a hidden implementation task.
The seven-stage sequence is security review, data access, configuration/golden set, baseline evaluation, shadow/rollback, limited-production success metrics and go/no-go. The source's day ranges are 1–2, 3–4, 5–7, 8–9, 10–11, 12–13 and day 14. Data access still pending on day 3 auto-escalates. Each gate records signer role and evidence, and advances only to the next stage when the previous gate is complete. A deny overrides an earlier optimistic state. Audit every attempt, including rejected sign-offs.
Free-text evidence can rubber-stamp a gate, so the production form should require structured security controls, dataset lineage, signed golden set, baseline report and rollback evidence. Separate the people who build from those who approve risk. Sponsor approval should cite actual metrics, not only a checkbox.
3. Accelerator reuse and agent runtime
Reusable connectors, prompt templates, eval harness, guardrails and dashboards shorten the calendar. Northwind uses five of six standard components; custom escalation logic is the exception. Record pull_or_build for each component so teams can see where bespoke work accumulates. Reuse must still be adapted to tenant identity and customer policy.
The proposed runtime path is Zendesk ticket → PII (personally identifiable information) redaction → classification of the top three categories → Confluence KB plus Salesforce read-only retrieval → grounded LLM draft → groundedness and escalation checks → human review/send policy → telemetry. Keep source credentials scoped, secrets in a vault and customer data segregated. A retrieval miss should trigger abstention or escalation. The source does not demonstrate an autonomous action loop; avoid calling the LLM draft an independent release authority.
4. Outcomes and limitations
The demo reports 4m12s first response, 63% zero-edit sends, .83 evaluation score against .75 baseline, a rollback note saying under two minutes (never timed), .27 override rate, first value in one day and deployed True at day 14. These are scripted demonstration results. The actual week-four retention field is unavailable, so it cannot support a retention claim. The 17 tests demonstrate gate enforcement, not agent quality in live production.
The source calls out missing scope-change sponsor approval, real provisioning, eval-harness integration, versioned rollback and concurrent-engagement scaling. A production plan should add these before promising the same calendar broadly. If data access, security or golden-set agreement fails, a no-go is safer than silently compressing later stages.
5. Evaluation, cost and operations
Build a golden set with top ticket classes, ambiguous cases, PII, retrieval gaps, escalation triggers and policy exceptions. Test classification, grounded replies, refusal/escalation behaviour, response latency and human-edit rate. Shadow mode collects evidence without broad sending. Limited production uses a defined approval policy and rollback trigger. Track cost per resolved ticket, token/step budget, human review load and post-handover retention separately from demo results.
Credentials should be scoped during the engagement and transferred or revoked at handover. A runbook should name the ongoing owner, how to inspect dashboards, how to stop sends, and which version to roll back to. Measure override reasons: a high rate can reveal weak retrieval, an overly conservative policy or poor classification.
Quick Review · 2 min — The whole case on one page, for recall before an interview. Assumes the guide.
Ask first
Success metric and baseline? Sponsor and SME (subject-matter expert)? Data sources and credentials? Reuse versus bespoke? Failed-gate policy? Human approval and rollback?
Seven gated stages
Intake → security → data access → golden set → baseline eval → shadow + rollback → limited production → sponsor go/no-go / handover
Days 1–2 / 3–4 / 5–7 / 8–9 / 10–11 / 12–13 / 14. Right signer, evidence and prior gate required. Pending access escalates on day 3.
Agent runtime
Zendesk → PII redaction → LLM classification → KB + read-only CRM → LLM grounded draft → escalation/grounding checks → human approval/send → telemetry
The delivery gates are deterministic; the model cannot approve itself.
Demo facts and caveat
Northwind: five of six components reused; 4m12s first response; 63% zero-edit sends; eval .83 versus .75 baseline; rollback note “<2 min” (not timed); 17 gate tests. These are scripted/demo results; week-four retention is unobserved. Real provisioning, eval wiring and versioned rollback remain gaps in the source.
Close
“I start the clock only when outcome, SME and sources are real. Each phase has evidence and an owner. I shadow, prove rollback, then limit production and let the sponsor make an informed go/no-go decision.”
Full Design 1 · 11 min — The complete, referenced write-up behind the other tabs. The sources for every tab live here.
Two weeks is fast enough to lose the customer's trust by cutting a corner, and slow enough to prove the process never needed one.
This is not a software architecture question, even though it is asked in a system design round. It is an operating-model question, and the design is a state machine whose gates are code rather than trust. The page consolidates the group's prompts into one read for the day before.
| Case in the group | What it contributes here |
|---|---|
| #9 Scoping Doc to Deployed Agent in Two Weeks, 60-minute whiteboard script (anchor) | Sections 1 to 11 and 14: framing, clarifying questions, gates, the library, risks, failure modes, scale, the close |
| #65 Northwind Logistics tier-1 support triage, Module 10 running case and the AI_Engineer interview script and code project | The fourteen-day table, the five metrics, the evidence-bar critique and the verified numbers; the delta is in section 13 |
| Self-drill on #9, Drill Add-ons tab | Section 15 |
Module 10 doc 4 and the project's docs/05 |
Sections 6, 9 and 10 |
Sections and tables marked (own construction) were built for this page from the sources' arguments and are not in the sources verbatim. The triage agent's internals in section 4 are the largest such addition, because the sources design the delivery pipeline and name the agent without drawing it.
1. Frame the Round as an Operating-Model Question
Say the framing sentence in the first two minutes. It names the real test before any box is drawn.
"This isn't really a software architecture question — it's an operating-model question. Anyone can draw a pipeline with seven boxes. The signal is whether each box is a real, enforced gate or just a status someone reports honestly. I designed it so the gates are code, not trust."
The naive answer is "work faster" or "hire more people", and that is the trap. Two weeks is only possible if most of the work already existed before the customer showed up. If every engagement starts from a blank page, two weeks is not a schedule, it is a wish. So speed has to be a side effect of reuse and hard gates, never a target chased by cutting corners.
That splits the answer into two questions, and a strong answer spends its time on both:
| QUESTION THE ROUND IS REALLY ASKING | WHAT ANSWERS IT |
|---|---|
| How do we know each stage is safe to leave, not just that time passed? | Gates: a named, checkable condition, signed off by the right role, never by the person closest to the work |
| How do we make two weeks achievable for a different customer every time? | A library of reusable assets that most of each stage is assembled from, not written |
A weak answer draws seven boxes labelled scoping, data, build, test and launch, then stops. That is a checklist, and checklists get skipped under deadline pressure. Restate the problem in one breath when asked:
"We're designing a repeatable delivery pipeline — not a one-off project plan — where speed comes from a reusable accelerator library, and safety comes from hard, role-gated checkpoints between stages, so that two weeks is an achievable outcome of the process, not a deadline we hope to hit by cutting corners."
2. Ask the Five Questions That Change the Design
A clarifying question earns its minute only if a different answer changes a box. These five each do.
| QUESTION | WHAT CHANGES WITH THE ANSWER |
|---|---|
| Is "2 weeks" a hard SLA or a target? | Whether a gate failure means "slip the date" or "abort and re-scope" |
| Who owns the go/no-go decision: the FDA, the account team, or the customer? | Who the success_metrics_met gate's allowed role should be |
| What is genuinely reusable across customers versus inherently one-off? | Whether the library covers day 1 of a new engagement type, or only the 50th customer of a proven one |
| What happens if a hard gate fails outright, not late but genuinely? | Whether the engagement pauses or terminates when a security review finds a real blocker |
| Is this framework for one team or an org-wide standard? | Whether the library is a shared asset with ownership and versioning, or tribal knowledge |
Then scope out loud. Design the pipeline as a state machine with named, role-gated checkpoints and a reusable asset library behind each stage. Name the infrastructure-provisioning layer under "configure, do not code" as a real system in its own right, say what it must do, and move on.
3. Write the Success Metrics Before Day 1
A two-week clock started against something nobody can measure is worse than no clock. So the requirements of this case are the scoping document's contents, and intake refuses the engagement if they are missing. The running case is Northwind Logistics: a tier-1 support triage agent over Zendesk, Confluence and Salesforce.
Success metrics, agreed at intake, in writing (verbatim from the intake record):
| Metric | How it is measured |
|---|---|
| Tier-1 ticket first-response time drops below 5 minutes for the top 3 ticket categories | Ticket timestamps |
| Agent-drafted replies are sent with no human edit on at least 60% of eligible tickets by week 4 | Edit history |
Both are measurable from data the customer already has, not "the agent should be helpful". Say that aloud: if a metric cannot be measured this cleanly, intake refuses rather than starting a clock against it.
Functional requirements of the delivery framework. Most are the source's; the MoSCoW split is (own construction).
| Priority | Requirement |
|---|---|
| Must | Intake refuses a request with no measurable success metric, no named customer SME, or no named data sources; no engagement object is created |
| Must | Seven stages in fixed order, each blocked by a gate; no code path skips a stage |
| Must | Every gate is signed by an allowed role, with evidence, in order; deny overrides |
| Must | Every stage pulls assets from the accelerator library before building custom, and every custom build is logged |
| Must | Data access still pending on day 3 raises an escalation automatically |
| Must | Every sign-off attempt, stage move, escalation and pull is written to a replayable event log, allowed or denied |
| Must | Day 14 hands over a runbook, dashboards and a named owner |
| Should | Scope changes after intake go through the sponsor's authority (named, not yet built) |
| Should | Gate evidence has a required shape, a checklist rather than a sentence (section 10) |
| Could | A portfolio view across engagements per FDA |
Functional requirements of the agent being delivered (own construction, from the case's metrics and library assets): classify incoming tier-1 tickets and handle the top 3 categories; draft a grounded reply from the Confluence knowledge base and the customer's Salesforce record; redact PII (personally identifiable information); escalate anything outside policy to a human; in shadow mode draft without sending, and in limited production send only after human approval.
Non-functional requirements (own construction; figures are the source's):
| Attribute | Target | Why it is the bar |
|---|---|---|
| Time to deploy | Day 14, or a named gate explains why not | The promise is the process, not the date |
| Agent latency | First response under 5 minutes on the top 3 categories | The customer's own success metric |
| Quality at handover | Eval score clears the agreed baseline (the demo scripts 0.83 against 0.75; no real eval harness runs) | The number true when responsibility transfers |
| Safety of rollout | Rollback tested, not documented (the demo's sign-off text says under 2 min; scripted, not timed) | The first real action must not be the first unwatched one |
| Auditability | Every gate decision replayable with signer, role, evidence and reason | "Why did it stall at day 3?" has one answer |
| Data access | Read-only access (for Zendesk, OAuth with the read scope; its API tokens are all-or-nothing and being retired), scoped to the engagement, revoked at handover |
The delivery process holds live customer credentials |
| Repeatability | Accelerator reuse rate tracked per engagement (83% on Northwind) | The measure of productised versus bespoke |
Constraints. Two calendar weeks. A customer SME who has other work. Security review before any connection is made. Three of the six gates need a customer signature, so customer-side delay is the default risk.
Non-goals, said out loud. Real infrastructure provisioning under "configure, do not code". A running eval harness, since the Evaluate stage references one. More than one engagement at a time. A versioned rollback mechanism, since the gate attests that rollback was tested elsewhere. The honest limit sits above all four: some engagements should not be two weeks, and knowing which ones to reject is part of the framework.
4. Draw the Architecture End to End
Two systems sit on this board, and naming both is the move. The delivery pipeline is the answer. The triage agent is what the pipeline produces, and one diagram of it shows the gates are about something real.
Write the nouns before the boxes:
Engagement · Stage · Gate · Principal (role) · Artifact · AcceleratorAsset · Escalation · Metric
Narrate one engagement end to end before drawing:
"A scoping request comes in. Intake checks it's actually measurable and has an SME assigned, or refuses it outright. Accepted, it enters day 1 at the Scoping stage. Each of the next six stages is blocked behind a named gate, signed off only by the role authorised for that gate — the FDA cannot sign the security gate, no matter how senior. Every stage pulls what it needs from a reusable accelerator library before building anything custom. On day 14, if the agreed success metrics were actually met, the sponsor signs the go/no-go gate and the engagement deploys with a runbook and dashboards handed over."
The source's pipeline, kept verbatim. Label every gate with who signs it.
SCOPING ──> gate:security_review ──> DATA READY ──> gate:data_access ──> CONFIGURE
(1-2) (security reviewer) (3-4) (customer SME) (5-7)
|
day 3, still pending?
AUTO-ESCALATE
v
gate:golden_set ──> EVALUATE ──> gate:eval_baseline ──> SHADOW
(customer SME) (8-9) (FDA) (10-11)
|
gate:rollback_tested ──> LIMITED PROD ──> gate:success_metrics ──> GO/NO-GO
(FDA) (12-13) (sponsor) (14)
|
DEPLOYED
Both layers on one board, with the agent inside the pipeline (own construction):
╔══════════════════ DELIVERY CONTROL PLANE (the framework) ═══════════════════════╗
║ scoping request ─> INTAKE ──fails──> REFUSED (no engagement, no clock) ║
║ │ passes ║
║ v ║
║ ENGAGEMENT state ─> 7-stage state machine ─> 6 role-gated sign-offs ─> DEPLOYED ║
║ │ stage, day, gates, artifacts, pulls, escalations ║
║ ├─> ACCELERATOR LIBRARY: connectors · prompt templates · eval harness · ║
║ │ guardrail policies · dashboards (pull, or build custom + log) ║
║ ├─> ESCALATION CHECK: data access pending on day 3 -> raise once ║
║ ├─> EVENT LOG: every attempt, allowed or denied, replayable ║
║ └─> METRICS: time-to-value · eval score · override rate · week-4 retention · ║
║ reuse rate ║
╚══════════════════════════════════╤═════════════════════════════════════════════╝
│ configures and gates
╔══════════════════ THE DELIVERED AGENT (Northwind tier-1 triage) ════════════════╗
║ Zendesk ticket ─> webhook ─> PII redaction ─> classify (top 3 categories?) ║
║ │ no -> route to human queue ║
║ v yes ║
║ retrieve: Confluence KB (read-only) + Salesforce account (read-only) ║
║ v ║
║ draft reply (support_triage_prompt) ─> groundedness check ─> escalation policy ║
║ v ║
║ SHADOW (days 10-11): store draft, human writes own reply, compare ║
║ LIMITED PROD (12-13): human approves or edits, then send; rollback tested ║
║ v ║
║ telemetry: first-response time · zero-edit sends · override rate · cost ║
╚═════════════════════════════════════════════════════════════════════════════════╝
The delivery pipeline for viewers that render Mermaid, compressed from the source's state machine:
The delivered agent (own construction):
Read the components in dependency order. The pipeline rows are the source's; the agent rows and the failure column are own construction.
| Component | Job | What goes wrong without it |
|---|---|---|
| Intake validation | Refuses unmeasurable, SME-less or source-less requests | A clock starts against a goal nobody can prove |
| Engagement state | The one piece of mutable state, threaded through every stage | Status lives in five trackers and disagrees |
Stage state machine (advance_stage) |
Moves only to the immediate next stage | A stage is skipped under deadline pressure |
Gate engine (sign_off) |
Role, evidence and order checks, deny overrides | The FDA certifies their own security review |
| Escalation check | Raises data-access delay on day 3, once, logged | The delay is noticed on day 6 |
Accelerator library (pull_or_build) |
Pulls a named asset or logs a custom build | Week two turns into bespoke code |
| Event log | Every attempt, allowed or denied | "It's taking longer than expected" |
| Metrics | Computed on demand from the engagement's history | Success is argued, not read |
| Connectors (Zendesk, Confluence, Salesforce) | Read-only ingestion from the customer's systems | Data readiness slips; credentials sprawl |
| PII redaction policy | Strips personal data before the model sees it | Customer data leaks into prompts and logs |
| Triage classifier | Keeps the agent to the top 3 categories | The agent answers what it was never evaluated on |
| Draft plus groundedness check | Reply built from retrieved evidence, judged before a human sees it | A fluent, wrong reply reaches a customer |
| Custom escalation policy | Northwind's own rules for what goes to a human | The one custom asset is skipped because it is custom |
| Human approval step | The override signal in shadow and limited production | Trust is assumed rather than earned |
| Handover bundle | Runbook, dashboards, named owner | Day 15 has nobody to call |
Point at two lines while the diagram is up. "This is deliberately the same decision shape as the RAG (retrieval-augmented generation) project's ABAC (attribute-based access control) engine: a named rule, a role check, deny overrides, and an explicit reason." And "there is no code path that jumps a stage, the same way the RAG project has no code path that queries the vector store without the ACL (access control list) filter."
Full Design 2 · 8 min — The complete, referenced write-up behind the other tabs. The sources for every tab live here.
5. Walk the Fourteen Days in Order
The order is not optional. Configuring against unconnected data fails, evaluating against an unsigned golden set proves nothing, and limited production without a tested undo is a gamble. So transitions are a state machine with hard gates, not a checklist.
| Days | Stage | What has to be true to leave it | Northwind evidence in the demo | Signed by |
|---|---|---|---|---|
| 0 | Intake | Measurable metrics, a named SME, named sources; otherwise refused | Intake record accepted | — |
| 1–2 | Scoping and qualification | Metrics written down and measurable; SME assigned; security review started | scoping_questionnaire.pdf, success_metrics_signoff.pdf; Zendesk and Confluence connectors pulled; "SEC-2026-0142 review report, no blocking findings" |
Security reviewer |
| 3–4 | Data readiness | Sources connected; access live and verified, not "requested" | "Read-only Zendesk (OAuth read scope), Confluence (scoped token) and Salesforce (read-only integration user) access verified live"; escalation check runs on day 3 | Customer SME |
| 5–7 | Configure, do not code | Assembled from the library and configured for this customer | Triage prompt, golden-set harness, PII policy pulled; Northwind escalation policy built custom; golden_set_v1.json |
Customer SME ("golden_set_v1.json reviewed, 40 cases approved") |
| 8–9 | Evaluate and iterate | Golden set signed off as representative; baseline clears the bar | "eval_baseline_report.pdf - 0.83 vs 0.75 agreed baseline" (scripted value, no eval harness run) | FDA |
| 10–11 | Shadow mode | Agent sees real traffic, decides, takes no action; humans compare | 5 shadow approvals, 2 overridden; "rollback_runbook.md - tested in staging, verified <2min" (scripted text, not timed) | FDA |
| 12–13 | Limited production | Agent acts for real with a human approving, and a tested rollback | 6 approvals, 1 overridden; cumulative override rate 0.27 | — |
| 14 | Go/no-go and handover | The day-1 metric was met; runbook, dashboards, owner handed over | "Week-2 metrics: first-response 4m12s, 63% zero-edit sends" (scripted evidence text in the demo, not measured); handover_runbook.md, dashboards_live.url |
Sponsor |
The table's last two evidence columns come from scripts/run_engagement_demo.py. Shadow mode and limited production are the same two-step trust ladder as Module 05's SHADOW and LIVE. Never let a system's first real action also be the first time nobody is watching.
7. Refuse at Intake Rather Than Start a Clock
Immeasurable-metric refusal is the cheapest, highest-signal move in the design, so say it before drawing a single box. Intake raises ScopingRefused and creates no engagement at all. It is the delivery version of Module 04's refusal to index a document with no usable ACL (access control list).
The SME requirement is refused at intake for a structural reason. Two of the six gates need an SME signature, and every later gate waits on them. An engagement with no SME cannot finish, and it is better to know on day 0 than on day 8.
The trade-off has a revisit condition. A strategic customer may be accepted despite the risk. Then intake needs an explicit override path with its own sign-off, never a silent bypass.
8. Turn Reuse Into a Number
"Productised process, not bespoke heroics" is the thesis of this problem type, and a thesis on a slide proves nothing. So every asset a stage needs is either pulled from the library or built custom and logged, with no third option.
Every stage needs an asset (a connector, a prompt template, a guardrail policy...)
Pull it from the library? -> reused = true
Not there yet? -> build it custom, reused = false, LOGGED
accelerator_reuse_rate = reused pulls / total pulls
The library holds connectors, prompt templates, an eval harness, guardrail policies and dashboard templates. By name: zendesk_connector, confluence_connector, salesforce_connector, support_triage_prompt, groundedness_prompt, golden_set_harness ("recall@k / MRR (mean reciprocal rank) / groundedness / leak-rate harness"), pii_redaction_policy, destructive_action_gate, eval_baseline_dashboard, cost_attribution_dashboard.
In the Northwind demo run the rate is 83%: 5 of 6 assets came straight from the library, and one guardrail policy was custom. A rising custom-build count is the early signal that two weeks is about to slip. Tracked across engagements, the ratio says whether the framework works or is quietly turning bespoke again.
Be honest about what the library is not. The entries are named assets, not the connector code or the eval harness. The real harness exists one project over, in the RAG (retrieval-augmented generation) platform, and wiring the Evaluate stage to call it is on the punch list. The gate mechanism is real; the number behind one piece of evidence is not yet.
9. Make Risk Mitigations Checks, Not Reminders
A reminder fires only if someone is watching. A check fires anyway, and lands in the permanent event log.
| Risk | Mitigation | How it is enforced | Built? |
|---|---|---|---|
| Data access delays | "Start day 1, escalate day 3" | check_escalation_triggers() raises it once if data_access_granted is pending on day 3 |
Yes |
| Scope creep | Change-control on the signed scope | A request_scope_change() requiring the sponsor role, reusing sign_off()'s pattern |
No, sized as low effort |
| Unmeasurable success criteria | Refuse to start | intake() raises ScopingRefused |
Yes |
| No customer SME | Contractual prerequisite | Also enforced at intake() |
Yes |
"I'd rather show you three of four real and name the honest gap on the fourth than claim all four and get caught on it under a follow-up."
10. Raise the Evidence Bar Behind Each Signature
A gate can be structurally real and still be a rubber stamp. The right role must sign, which is a genuine control. But "does the gate exist" and "what counts as sufficient evidence" are separate questions.
| What a thorough security review should include | Does a generic "security review passed" checkbox prove it happened? |
|---|---|
| Basic output validation | No, a quality check, not a security check |
| Retrieval and tool-selection accuracy | No |
| Task success on real test cases | No, that is the evaluation gate |
| Adversarial testing: prompt injection, permission-boundary probes | This is what a security review should mean, and free-text evidence cannot prove it happened |
| Ongoing production monitoring | No, continuous, not a gate |
The fix is a checklist, not a sentence. Adversarial testing done, permission boundaries tested and data agreements signed must all be present before the gate passes. That is intake's discipline applied to what a signature attests to.
Three more gaps sit in the delivery process itself, easy to miss because attention is on what gets deployed. The process holds live customer credentials in two windows: when sources first connect, and when the agent first runs against the real environment. Those credentials need a vault, engagement scope and revocation at handover. The golden set built with the SME often contains real customer data, so it follows that customer's sensitivity tier rather than one default. And many engagements at once is a tenancy decision, not only a capacity one: are two customers' gate evidence and test sets separated by a tag or physically?
"A gate proving access was granted proves a human attested it's live. It doesn't prove the credentials enabling that access were ever handled safely during the engagement — stored in a vault, scoped only to what this engagement needs, revoked cleanly at handover."
Full Design 3 · 12 min — The complete, referenced write-up behind the other tabs. The sources for every tab live here.
11. Treat a Stalled Engagement as a Visible Outcome
The framework does not guarantee every engagement deploys in 14 days. It guarantees that when one does not, everyone knows which gate stopped it and why.
| Fails | Behaviour |
|---|---|
| A hard gate genuinely fails, not late but denied | The engagement stays at its current stage; the failure is a first-class event, not a silent stall |
| The wrong person attempts a sign-off | Denied, logged, state unchanged; no partial progress |
| The golden set never gets customer sign-off | Evaluate is blocked; a customer-side failure the framework surfaces rather than routes around |
| Success metrics are not met by day 14 | The go/no-go gate does not pass; "no-go" is a legitimate outcome |
Scale here is not ten million chunks. It is one FDA (Food and Drug Administration) running six engagements at once. The state machine does not change with portfolio size; capacity does. Gates start queueing behind a person's calendar rather than the customer. Designing that needs real engagement-duration data first, the same way embedding-cache hit rates need real query logs.
12. Read the Northwind Numbers Honestly
Five metrics are computed on demand from the engagement's own history, never tracked separately.
| Metric | What it tells | Northwind demo |
|---|---|---|
| Time-to-first-value | How long until the customer saw anything real | 1 day |
| Eval score at handover | The number true when responsibility transferred | 0.83 |
| Human-approval override rate | Falling means trust earned; flat means not ready for less supervision | 0.27 |
| Week-4 retention | Whether the deployed thing is still used a month later | None until recorded, never a fake 0.0 |
| Accelerator reuse rate | Productised process versus bespoke heroics | 0.83 |
The run also shows 6 of 6 gates passed, deployed True on day 14, and one escalation, auto-raised on day 3 and resolved when the gate passed. The project has 17 gate-enforcement tests, deterministic, with no LLM (large language model) anywhere.
Read these as one engagement, run once. The eval score and time-to-value are demo-scripted, not measured on a real delivery. What is proven is that the pipeline enforces its own gates. Saying so before the interviewer asks is itself the signal.
13. Read the Same Case in Prose
#65 is not a second design. It is the anchor's Northwind running case, written three ways, and each adds something to rehearse.
| Source in #65 | What it adds beyond the anchor script |
|---|---|
| Module 10 doc 3, Scoping to Production in Two Weeks | The fourteen-day table with exit conditions, the trust-ladder link to Module 05, the one-breath restatement |
| FDE delivery framework reference implementation, case-study data | The two success metrics verbatim, the three connectors, the SME id |
AI_Engineer INTERVIEW_SCRIPT.md |
Identical to the anchor; the anchor was lifted from it, so reading one covers both |
The code project (run_engagement_demo.py, demo_gate_failure.py, 17 tests) |
The evidence strings and figures in section 5, and the negative-control demo |
Module 10 doc 4 and docs/05 |
The five metrics, the two gate pairs, the evidence-bar critique, credentials, residency and tenancy |
The negative-control demo is the most persuasive artefact in the group. It shows intake refusing an unmeasurable request, then three denials: wrong_role, no_evidence and prior_gate_incomplete. It then attempts a stage skip that has no code path, and the day-3 escalation fires by itself. A demo of things being stopped proves more than a demo of things working.
14. Deliver It in Sixty Minutes
Write the time budget in the corner of the board.
| Minutes | Phase |
|---|---|
| 0–8 | Clarify and scope |
| 8–15 | High-level pipeline |
| 15–35 | Deep dive: gates and the reuse system |
| 35–45 | Cross-cutting: risk mitigation, metrics, CI/CD parallels |
| 45–55 | Failure modes and scale, portfolio not chunks |
| 55–60 | Close: trade-offs and what to build first |
Announce the risk before the deep dive. Seven stages in order is not a hard problem. The hard part is making "done" mean something checkable instead of "time passed", and making the library real enough that week two does not turn into bespoke code.
Summarise in three sentences at minute 55:
"A seven-stage pipeline where every transition is blocked behind a named, role-gated decision, not a status field. Reusable assets are pulled before anything is custom-built, and that ratio is tracked as the actual measure of whether the process is repeatable. Two of four named risk mitigations are automatic checks in code, not reminders on a tracker — and I can tell you exactly which two aren't, yet."
The source's summary says two of four, while its own risk table marks three of four as built. Quote the table: the day-3 escalation, unmeasurable-metric refusal and SME refusal are built, and change-control is not.
| Decision | Chose | Would revisit if |
|---|---|---|
| Role-gated sign-off over self-certification | Six named gates, six named authorities | A single-person delivery team where role separation is impossible; gates compress, they do not disappear |
| Hard refusal at intake over "start and hope" | Refuse unmeasurable or SME-less requests | A strategic customer the business accepts anyway; add an explicit override with its own sign-off |
| Automatic escalation over manual tracking | Day-3 auto-escalate on data access | Mitigations multiply and a dashboard is clearer than N checks; then it becomes a rules engine |
Close forward-deployed. Week one of an internal rollout is not all seven stages. Pick the two gates most often skipped under deadline pressure, the golden-set sign-off and the rollback test, and make only those structurally unskippable. That proves gates as code change behaviour before anyone argues about the other five.
The lines that carry the round:
- "A gate is a decision with a named authority behind it, not a checkbox anyone can tick."
- "There's no code path that skips a stage — the same way the RAG (retrieval-augmented generation) project has no code path that skips the ACL (access control list) filter."
- "Refuse to start an engagement with no measurable success metric — the two-week clock is a consequence of readiness, not a target."
- "The reuse-vs-custom ratio is the actual measure of 'productised process,' not a slide bullet."
- "An automatic escalation on day 3 beats a reminder on a tracker — it fires whether anyone was watching or not."
- "A 'no-go' at day 14 is a legitimate outcome, not a failure of the framework."
- "Some engagements shouldn't be two weeks — knowing which to reject is part of the design, not a gap in it."
Questions to ask them: what the accelerator library looks like today, whether code, templates or tribal knowledge; where the two-week target actually broke and at which gate; who owns go/no-go in practice; and how many engagements one Forward Deployed Architect carries at once.
15. Answer the Pilot-Spend Pivot in Ten Minutes
The pivot after a good answer is "who pays for the pilot, and what stops it running away?" There is no playbook drill for this group, so the answer is the self-drill card.
| Dominant driver | Evaluation and logging overhead during the pilot, plus an unbounded agent in front of real tickets |
| Cheapest lever first | Agree cost per resolved ticket at intake alongside the quality metrics; sampled evals not full-corpus; step and token caps from day one; budget alert per pilot |
| Metric that proves it | Cost per resolved ticket vs the intake baseline; eval spend per day; P95 |
| Do not | Instrument cost after the pilot proves value |
| 60-second line | Cost is a success metric agreed at intake, not a cleanup after go-live. Cap the agent, sample the evals, and report cost per resolved ticket in the same dashboard as quality. |
The card fits the framework without a new box (own construction). Cost per resolved ticket becomes a third success metric at intake, so the refusal rule covers it: no baseline, no clock. cost_attribution_dashboard is already in the library, so it is pulled on day 1, not built on day 13. And success_metrics_met then certifies cost alongside first-response time and zero-edit sends.
Every strong cost answer follows four verbs in order. Measure cost per resolved ticket from day 1. Route only the top 3 categories to the agent. Bound steps, tokens and eval volume. Cache safely, never across customers.
Level calibration
In this round the level shows in whether each gate is a decision with a named authority, and in how far the answer reaches past one engagement.
| Level | Sounds like | Missing |
|---|---|---|
| Senior | Seven stages in fixed order, each behind a role-gated sign-off with evidence, and intake that refuses an unmeasurable metric or a missing SME. The Northwind agent drafts, a human sends, and rollback is tested in shadow before limited production. | Who acts when the SME misses a deadline, what free-text evidence fails to prove, and the customer credentials the delivery process itself holds. |
| Staff | Adds the people around the gates: the day-3 auto-escalation, an explicit override path for a strategic customer, a checklist evidence bar for the security gate, and credentials vaulted and revoked at handover. Reads the Northwind numbers as scripted and says so before being asked. | The portfolio view: one FDA running six engagements, where gates queue behind a person's calendar rather than the customer. |
| Principal | Treats the accelerator library as an owned, versioned asset shared across teams, and the reuse rate across engagements as the test of whether delivery is productised. Decides which engagements should not be two weeks, and whether customers' gate evidence and test sets are separated by a tag or physically. | Real engagement-duration data from a portfolio of deliveries, which is needed before sizing how many engagements one FDA can carry. |
Key Takeaways
- The round is an operating-model question, and speed is a side effect of reuse and hard gates.
- Five clarifying questions each change a box: SLA (service level agreement) or target, go/no-go owner, reusability, hard failure, scope of adoption.
- The requirements are the scoping document: two measurable metrics, a named SME, named sources, constraints and stated non-goals.
- Two systems share the board: the gated delivery pipeline, and the triage agent it produces.
- Fourteen days run in a fixed order, and each stage exits only when a checkable thing is true.
- Every gate is a decision by an allowed role, with evidence, in order, and seniority is not an override.
- Intake refuses rather than starting a clock against something unmeasurable.
- The accelerator reuse rate, 83% on Northwind, turns "productised" into a number.
- Risk mitigations are checks in code, with change-control named as the honest gap.
- A real gate still needs an evidence bar with a required shape, and the delivery process meets the same security bar as the product.
- A stalled engagement names its gate, and scale means FDA capacity, not data volume.
- The Northwind numbers are one engagement run once; the proof is that the gates enforce themselves.
- #65 is the anchor's running case in prose, code and verified output.
- The hour goes to gates and reuse, and the close makes two gates unskippable first.
- Pilot cost is a success metric agreed at intake, capped, sampled and reported beside quality.
Check Yourself
- Why is "work faster" the trap? Two weeks is achievable only if most of the work existed before the customer arrived; speed has to come from the library and the gates.
- Name the seven stages, their days, and what blocks each. Scoping 1–2, data readiness 3–4, configure 5–7, evaluate 8–9, shadow 10–11, limited production 12–13, go/no-go 14; blocked in turn by security review, data access, golden set, eval baseline, rollback tested, success metrics.
- State the three deny rules.
wrong_role,no_evidence,prior_gate_incomplete, deny overriding, every attempt logged. - What separates
golden_set_signed_offfromeval_baseline_met? The first certifies the test is representative, signed by the SME; the second certifies the agent passed it, signed by the FDA. - Why refuse at intake instead of warning? A clock against an unmeasurable goal is worse than no clock, and an engagement without an SME cannot pass the two SME gates, so nothing after them.
- What does the reuse rate measure, and what is Northwind's? Reused pulls over total pulls; 83%, five of six, with one custom guardrail policy.
- How can a structurally real gate still be a rubber stamp? Free-text evidence cannot prove adversarial testing happened; require a checklist.
- What breaks first at scale? FDA capacity, with gates queueing behind a calendar rather than the customer.
- Which risk mitigation is not built? Scope-creep change-control, which would reuse
sign_off()with the sponsor role. - What is the sixty-second pilot-spend answer? Cost is a success metric agreed at intake: cap the agent, sample the evals, report cost per resolved ticket beside quality.
References
Fact-check sources (checked 27 Sep 2026)
- Demo script
run_engagement_demo.py(reference implementation) — the 0.83 eval score, "verified <2min" rollback and week-2 metrics are hard-coded evidence strings, not measurements - Gate definitions
gates.py(reference implementation) — the customer SME signs two gates; the FDA signs the eval and rollback gates - Zendesk: Announcing the removal of API tokens — API tokens are all-or-nothing; no new tokens from 27 Oct 2026, all off 30 Apr 2027
- Zendesk: Creating and using OAuth tokens with the API — read-only access needs the OAuth
readscope - Atlassian: Scoped API tokens in Confluence Cloud — Confluence supports scoped tokens