Interview prompt
Design a GenAI FDE (Forward Deployed Engineer) solution for a Reliable Workflow Orchestration System. One business process spans a document approval, a payment, an email and three internal systems, and any step can fail or sit waiting for days. Here, "exactly once" refers to the business outcome, not to individual messages.
Clarify the customer problem
- Which event tells us a workflow has truly finished?
- When a step fails, do we undo it or run a compensating action?
- How long can each step wait: minutes, hours or days?
- Who has authority to resume, rerun or override a single case?
- Will the payment provider accept an idempotency key?
Users and workflows
| User | Workflow | Current pain | AI assist opportunity | Human approval needed? |
|---|---|---|---|---|
| Workflow participant | ||||
| Operations team | ||||
| Application developer | ||||
| Auditor |
Requirements
Functional
Non-functional
- Latency target
- Availability target
- Cost budget
- Security/privacy constraints
- Audit/compliance requirement
Data and integration map
| Data source | Format | Owner | Freshness | Permission model | Risk |
|---|---|---|---|---|---|
Proposed architecture
Fill in each part of your architecture for this case:
- Durable state and history
- Scheduling
- Timers and signals
- Activity workers
- Compensation
- Operator repair
Evaluation plan
| Metric | Good threshold | Bad threshold | Test dataset | Owner |
|---|---|---|---|---|
| Duplicate business effects | 0 | Even one repeat | Idempotency key logs | Reliability team |
| Replay determinism | Same decisions on every replay | One divergent decision | Replay jobs over stored history | Runtime engineering |
| Age of stuck workflows | Inside the expected wait | Past the agreed limit | Queue and state inspection | Operations |
| Manual repairs | Inside agreed tolerance | Over tolerance | Operator ticket records | Support operations |
Failure modes
- A worker dies just after the payment goes through
- The email sends but the CRM (customer relationship management) write fails
- An approval sits untouched for seven days
- A new definition ships while old instances still run
- An external API never returns
- Two workers try to move the same instance forward
Rollout plan
- Start with a single workflow; skip the platform.
- Make crash recovery the first gate.
- Show that no side effect happens twice.
- Give operators visibility early.
- Practise resume and compensation drills.
- Publish new versions; never edit live ones.
Weak vs strong answer
Weak: "I'd write an orchestrator that calls each API one after another and retries when something fails."
Strong: "I'd guarantee exactly once for the business effect. An append-only history writes the intent before each step runs. Receipts are stored by idempotency key. Waiting is a proper timer state. Every step has its own compensation. Each instance stays pinned to the definition version it started on."
Candidate scorecard
Final 2-minute spoken answer
Rehearse this out loud before you look at it. Summarising the whole design in two minutes is its own skill, and it is usually the last thing you are asked for.