Configurable Platform for Customer-Specific Workflows

As the FDE (Forward Deployed Engineer), your job is to shape a Configurable Platform for Customer-Specific Workflows. Ten customers share one platform, yet each wants its own fields, approval steps, branding and integrations. The thing to avoid is ending up with ten separate forks.

• Which of the differences keep coming back, and which really happen only once?
• Who writes the configuration: our engineers, the customer's admins, or both?
• Does each configuration have to keep working, unchanged, across core releases?
• Where is custom code allowed: in the core, in a sandbox, or only inside adapters?
• How long should it take to configure a new customer?

Functional

Non-functional

• Latency target:
• Availability target:
• Cost budget:
• Security/privacy constraints:
• Audit/compliance requirement:

Start from a rendered diagram and shape it to fit this case. Cover each part below:

• Workflow runtime:
• Configuration registry:
• Schema validation:
• Adapter SDK:
• Sandbox and flags:
• Migration and rollback:

• An approval graph contains a cycle and never ends
• An adapter's contract shifts on the downstream side
• An upgrade breaks a field mapping that older configs rely on
• Nobody tested a particular mix of feature flags
• Custom code gets out of its sandbox
• One validation rule eats all the available resources

1. Spot the pattern that keeps repeating.
2. Build it into shared primitives.
3. Have the first customer's admin set it up.
4. Run a pilot that goes through the real support desk.
5. Record a baseline for launch speed and defect count.
6. Widen only while the fork count stays at zero.

Weak: "I'd make a flexible system with forms, approvals and hooks you can configure."

Strong: "I'd run every customer on a single engine that ships on a single release train. Each customer's variation would live in a declarative config that is versioned, immutable and validated before it can be published. I'd deny arbitrary code unless it is explicitly allowed, and scope adapters and secrets to a single tenant. I'd count success as variations shipped with no fork."

Try the worksheet before you read this key. Use it to see what a strong GenAI FDE (Forward Deployed Engineer) candidate asks, designs, measures, secures and explains while taking a demo to production.

• Across the ten customers, which differences keep coming back and which happen only once? The first kind earns a platform capability; the second goes down an exception path.
• Is the variation limited to fields, approvals, branding and integrations? Or does it reach into business logic and the states a workflow can pass through?
• Who writes the configuration: our engineers, the customer's admins, or both, with the vendor signing off before anything reaches production?
• Which integrations must exist on day one? If an adapter fails, does the transition stop, or does the workflow carry on in a reduced mode?
• What do customers expect from an upgrade? Must their configs keep working, unchanged, through every core release?
• Where does custom code stop? Is it allowed in the trusted core, confined to a sandbox, or limited to integration points we have approved?
• How fast must a new customer be configured? A target of hours means self-service tooling. A target of a week leaves room for manual review.
• What should happen when someone misconfigures a workflow? Do we need a fallback approval path that always works?

• Core workflow: ten customers each change fields, approvals, branding and integrations, all on one shared platform and one codebase with no forks.
• Keep a single, stable workflow engine. Every customer runs the same core lifecycle, and every customer ships on the same release train.
• Describe variation as data: schemas, rules and UI metadata. Most customisation should then need no code change at all.
• Offer an adapter interface. External systems plug into it, and the engine itself never needs rewriting.
• Give every configuration a version and validate it. A change can then be reviewed, tested and rolled back.
• Provide migration and rollback tools, so no version change leaves a customer stuck on a config the platform no longer supports.

• Latency: split the budget by action. A save should feel instant, a validation preview may take a moment, and a full deployment can run slower as long as it is reliable.
• Availability: this case assumes about 1,000 configuration versions a day, with peaks at 3x. Validation and deployment therefore need separate paths, not one synchronous chain.
• Security: arbitrary code stays off unless explicitly allowed. Adapters and secrets belong to one tenant each. We also check the validation rules themselves, because a rule can be a denial-of-service risk.
• Compliance: keep a lasting record of who published each change, what it altered, which approval gate it passed and which version went live.
• Reliability: isolate configuration by tenant. A setting or test change from one tenant must never change how another tenant's workflows run.
• Cost: track spend per validated version and per activated template. Include the support load that a very flexible primitive brings with it.

• Configuration truth lives in the control plane, and workflows execute in the data plane. That separation is what stops the product splitting into ten forks.
• For every tenant, the workflow runtime handles instances, step transitions, timers and retries in exactly the same way.
• Published config versions are immutable. The configuration registry stores them with their release metadata and acts as the system of record.
• On every save and publish, the schema and rule validator runs synchronously. Syntax, semantic constraints, permissions and compatibility all get checked there.
• The UI renderer builds each tenant's forms, labels and layouts from the approved config. Apart from a cache, it stores nothing.
• CRM (customer relationship management), ERP (enterprise resource planning), ticketing and webhook connectors all sit behind one standard interface, the integration adapter SDK. Each tenant's credentials are held by reference.
• Rollout runs through a feature flag service, by tenant, cohort or percentage. The few extensions we approve run in a sandbox with strict limits.
• The migration service moves older configs onto new schema versions. Before a release ships, the tenant test harness replays each tenant's contract tests as a gate.

• WorkflowTemplate(id, engine_version, schema); TenantConfig(tenant_id, template_id, version, values); AdapterDefinition(name, contract_version, permissions); ConfigDeployment(tenant_id, version, state).
• ConfigDeployment acts as the deployment ledger. It is the source of truth for the draft, validated, deployed, rolled_back and failed states.
• A rollback adds a new deployment state. It never edits the earlier record, so we can always rebuild the full history.
• Each write states the version it expects to replace. Two operators racing on the same config then cannot overwrite each other without noticing.
• We tag every validation, deployment, adapter test and rollback with the tenant, the config version, a request ID and an idempotency key.

• approval loops that never end, drift in adapter contracts, upgrades that break old field mappings, flag combinations nobody tested, escape from the sandbox
• An approval graph with a cycle in it keeps looping at runtime. That harm is deterministic, not a transient fault that a retry fixes.
• A downstream service renames a field or changes its response shape. That adapter contract drift breaks every tenant on the adapter at the same moment.
• A tenant upgrade breaks old field mappings, and integrations still pointing at the previous version are left stranded.
• Two feature flags are each safe alone, but nobody tested them together, and the combination opens a bad path.
• Customer-defined logic escapes the sandbox. This is the reason arbitrary code is denied by default instead of only being reviewed.

• Week 0-1: find out which differences recur, who owns the configuration and where the line for custom code falls.
• Week 1-2: show that one recurring pattern fits into shared primitives, with no one-off fork needed.
• Week 2-3: keep the gate closed until the team agrees that this pattern is stable enough to become a platform capability.
• Week 3-4: have the first customer admin configure their own workflow, then run a pilot through the real support channel.
• Week 5: before going wider, check that the customer accepts both the support model and the rollback procedure.
• Week 6-8: record a baseline for launch speed, defects and validation failures; scope grows only after that.
• After pilot: grow only if configuration is clearly cutting custom work, not hiding rework somewhere else.

I picked these thresholds for this case. They are not industry standards, so be ready to justify them rather than recite them.

I'd build a flexible workflow system where forms, approvals, themes and API hooks are all configurable. It falls short because flexibility with no governance turns into fragmentation. It never says who may write configuration, what an upgrade must preserve, or where custom code has to stop.

I'd keep one shared engine, push customer differences into versioned configuration, validate it before publish and put integrations behind an adapter layer. That is an improvement, but gaps remain. It never decides which differences should become platform primitives. It sets no limit on what a validation rule may cost to run. And it has no plan for a tenant left behind on an old schema.

I'd split the feature they asked for, configuration, from the result they need: customer-specific workflows that don't break the product into ten incompatible forks. We should be able to test it on three counts: safe, versioned and supportable. The number I'd track is the share of new customer variations that launch without a fork, a hotfix or an escalation to support. In the design, that becomes one stable engine on one release train. Declarative schemas carry most of the variation, and an adapter interface handles integrations. Configs are immutable, versioned and validated before publish. A narrow sandbox appears only where declarative config really can't reach, and arbitrary code is denied by default. First I'd show that shared primitives can express one pattern that keeps recurring. Then a real customer admin configures and pilots it, and only after that do we expand.

I wouldn't open with a list of features. People agree on the headline request and disagree on everything beneath it. So my first job is to pull apart what they asked for, configuration, from what the business needs: ten customers served without the product splitting into ten incompatible forks. I'd make that outcome testable with three words. Safe: no change skips authorisation or creates runtime behaviour nobody reviewed. Versioned: each customer knows which definition is live and what changed in it. Supportable: someone can diagnose a problem and roll a bad configuration back. My own scorecard is simple: count the new customer variations that go live and never needed a fork, a hotfix or a call to support. A count of configurable fields would be a vanity number. The first real design choice is which differences recur. A difference in the approval chain that keeps coming back should become a platform capability. A single seasonal override belongs on an exception path. For the architecture, every tenant shares a single stable engine and a single release train. Declarative schemas carry most of the variation, and an adapter interface handles integrations. Configs are immutable and versioned, and we validate them before publish. A narrow sandbox exists only where declarative config can't go. Arbitrary code is off by default. Adapters and secrets belong to one tenant each, and we check validation rules too, in case one exhausts resources. The failure I'd name is an infinite approval loop. That version fails closed instead of carrying on. My first step is to show one recurring pattern working, and then a real customer admin pilots it.

• 1. The Customer Problem and Discovery
  • Reframing the Request
  • Stakeholder Map
  • Discovery Question Tree
  • Assumption Ledger
  • Weak vs. Strong Answers
• 2. Clarifying Questions, Requirements, and Constraints
  • Requirement vs. Preference
  • The Six-Question Interview Tree
  • Must-Have Functional Requirements
  • Measurable Non-Functional Requirements
  • MVP Non-Goals / Scope Fence
  • Requirement-to-Component Traceability
  • Handling Partial Answers
  • The Strong Candidate Move
• 3. Scale Estimates, SLOs, and Capacity
  • Load Shape, Not Marketing Shape
  • The Four-Step Capacity Envelope
  • SLOs Tied to Customer Pain
  • Leverage and What Belongs in Core
  • A Whiteboard Estimate That Changes Component Choice
  • Sensitivity Table Across Growth Scenarios
  • Average vs. Peak: Headroom as a Product Feature
  • Decision Rule to Carry Forward
• 4. Architecture and End-to-End Flow
  • Control Plane vs. Data Plane
  • Architecture Diagram and Trust Boundaries
  • Core Components in Dependency Order
  • Failure-Path Overlay: Infinite Approval Loop
  • Happy-Path Sequence: Author to Promotion
  • Systems of Record, Caches, Queues, and Backpressure
  • MVP vs. Later Evolution
  • 90-Second Interview Summary
• 5. Data Model, APIs, and Working Code
  • Four Core Records and Ownership
  • Four API Contracts
  • Idempotent Deployment Creation Walkthrough
  • The Smallest Safe Code Path: Config Publication
  • What the Whiteboard Version Omits on Purpose
  • Contract and Failure-Injection Tests
• 6. Security, Reliability, and Failure Handling
  • The Red-Team Question
  • Four Security Controls
  • Failure-Policy Decision Table
  • Blast-Radius Containment Levels
  • Failure Drill: Infinite Approval Loop
  • Other Named Failure Cases
  • Six Operational Primitives
  • Evidence, Runbooks, and Launch Gates
  • Interview-Sized Production Sketch
• 7. Delivery Plan, Observability, and Business Impact
  • Rollout as a Series of Proofs
  • The Four-Phase Rollout Plan
  • The Six-Metric Scorecard
  • Four Layers of Metrics
  • Dashboard in the Language of the User
  • Ownership Before Launch
  • Go/No-Go and Rollback Triggers
  • Config vs. Adapter vs. Shared Service vs. Core
  • The Risk Register
• 8. Interview Walkthrough, Trade-Offs, and Practice
  • Minute-Zero Opening
  • The 50-Minute Pacing Plan
  • Four Trade-Off Pairs
  • Expected Follow-Up Questions
  • Weak Answers and Their Repairs
  • Scoring Rubric
  • Interview Rehearsal Checklist
  • 90-Second Architecture Summary
  • Practice Plan
• Coverage Notes
  • My Perspective on the Gaps

Reframing the Request

• The room agrees on the headline, "one platform for similar workflows", and on little else. Fields, approval paths, audit depth and release process all differ by stakeholder.
• So don't start by drawing services. The first move is to turn a vague ask into a measurable outcome: one configurable workflow platform on which ten customers can each change their fields, approvals, branding and integrations, while the shared core stays stable, versioned and supportable.
• The key reframe splits the feature someone asked for ("configuration") from the result the business needs: each customer gets its own workflow, and the product does not splinter into ten forks that cannot talk to each other. Saying that split aloud is what makes you sound like an FDE (Forward Deployed Engineer), not an implementer.
• Make each word in the outcome testable:
  • Safe: no change can skip authorisation or switch on runtime behaviour that nobody reviewed.
  • Versioned: every customer can see which workflow definition is live and what changed in it.
  • Supportable: support and release staff can trace a fault, undo a bad configuration and explain behaviour tenant by tenant.
• Avoid vanity measures such as the number of configurable fields. Track how often a new customer variation ships with no fork, no hotfix and no support escalation.
• Think of it as a job to be done. The customer is hiring the platform to make custom forks unnecessary while each tenant keeps its own process; they are not buying "a workflow engine". That pushes the design towards a stable core with a few controlled extension points, not a codebase that changes shape per account.

🎯 Interview Pointer: Learn the outcome statement by heart: "keep one stable product core, and allow customisation that is safe, versioned and supportable." Interviewers listen for those three words, because each one drives a later architecture decision.

Stakeholder Map

• One request means something different to each group:
  • Customer administrators ask who may change a workflow, how much they may customise, and how fast a change reaches users.
  • FDE teams want a real customer use case in production without leaving one-off code behind.
  • Core platform engineers want a lasting model for workflow definitions, validation, permissions and how the runtime executes them.
  • Support and release teams want observability, rollback, a clear incident process and a quick answer to "what changed?"
• Drop a group and the design tilts: without support you ship flexibility nobody can operate; without platform engineers the customer layer never generalises.
• The concrete map, one line per role:
  • End user: a customer employee who starts a workflow, approves it, or is on the receiving end of its result.
  • Operator: on the support or release team; watches health, reverts a bad config, handles incidents.
  • Security owner: answers for permissions, audit and least privilege between tenants.
  • Executive sponsor: the business leader paying for one platform rollout instead of a run of forks and long customisation projects.

Discovery Question Tree

• Time is short, so skip the twenty generic questions. Ask the few that remove the most doubt.
• Workflow shape:
  • Which parts are the same for every customer, and which really differ?
  • Is the variation only in fields, routing, branding and integrations, or does it also touch business logic and lifecycle states?
• Ownership and control:
  • Who may write configuration changes, and who signs them off?
  • Do customer admins change configuration themselves, or does an FDE or platform team handle every change?
• Risk and recovery:
  • What happens when someone misconfigures a workflow?
  • Can the customer live with a broken approval path, or must a fallback always exist?
• Success:
  • What counts as "done" for the first customer: one live workflow, several teams onboarded, or every variation migrated?
  • How will the customer judge it: less manual handling, faster turnaround, fewer errors or simpler audits?
• The answers become the design's inputs: scope, assumptions, risks, owners and success measures.
• If the interviewer holds back, state your assumption aloud: "I'll assume only customer admins can configure, and the vendor approves each change before production." That is a clear boundary, not hedging.

Assumption Ledger

• A useful discovery pass leaves four things on the table:
  1. Scope: what is configurable and what stays fixed.
  2. Assumptions: what you accept for now because the prompt left it open.
  3. Risks: where a bad configuration, a permission slip or a failed integration could hurt a customer.
  4. Success criteria: the measurable change that shows the platform works.

Weak vs. Strong Answers

• A weak restatement leads with features: "Let's build a flexible workflow system: forms, approvals, themes and some API hooks."
• A strong restatement leads with the outcome: "We need a single shared workflow platform. Each customer can tailor its fields, approvals, branding and integrations without a separate codebase, and the core stays stable enough to version, support and roll out safely." The second version exposes the real tension: flexibility against fragmentation.
• A short spoken opening: "Let me frame it first. One workflow product, ten customers, each wanting its own fields, approval chains, branding and integrations. I want a stable shared core with safe, versioned, supportable customisation, so we never run ten forks. First I'd find out who owns configuration, which variations are really needed, which failures we can't accept, and how we'll measure the first rollout. Then the design rests on validation, rollback and a few controlled extension points, not per-customer code."
• Why it lands: it names the problem and the people, and it holds the architecture back until someone owns the workflow, someone owns the risk and a success metric exists. Bound the problem first; choose the design second.

Requirement vs. Preference

• The quickest way to lose control of this design is to take "we need flexibility" at face value. Ask who needs it, under what governance, and with what promise about upgrades.
• Sort requirements from preferences and hard constraints from conveniences. Every customer wants a slightly different mix of fields, approvals, branding, integrations and sometimes rules. The first design fork is spotting which differences repeat.
• Skip that, along with who authors configuration and where custom code ends, and you fall into one of two holes. Either the platform is too thin, or you build a bespoke rules engine nobody can support safely.

The Six-Question Interview Tree

• Six questions keep the conversation away from a feature wish list:
  1. Which differences keep coming back between customers? This separates a shared product pattern from one-offs. Approval chains that differ again and again are a platform capability; one seasonal override goes down an exception path. It also tells you what the core abstraction is: steps, policy rules, form schemas or integration orchestration.
  2. Who will write the configuration? It sounds like staffing but it is architecture. Internal engineers can handle expressive power; customer admins and support staff need guardrails such as validation, previews, narrow building blocks and safe defaults. The answer reshapes the UX, the validation model and the authorisation model.
  3. Which integrations and extensions can't be dropped? Typical items are webhook callbacks, outbound notifications, CRM (customer relationship management) sync, document generation, ticket creation and auth hooks. The list fixes the adapter interface, settles sync versus async, and decides whether a failed call blocks a transition or degrades gracefully.
  4. What upgrade promise do customers expect? If configs must outlive core releases, you need versioning, migration tooling, backward-compatible schema changes and rollback paths. Without them the "product core" is really a services project with no end.
  5. Where does the security boundary for custom code sit? Options are the trusted core itself, a sandbox, or approved integration points only. The choice sets tenant isolation and supply-chain scrutiny. Default to a declarative core, with arbitrary code kept beyond the main trust boundary.
  6. How fast must a configuration change go live? Changes needed within hours call for self-service tooling with validation built in. A week allows more manual review. The target decides where you invest: admin UX, reusable templates, automated testing or partner onboarding.

🎯 Interview Pointer: Expect follow-ups on question 5, the custom-code boundary. Have the default ready (a declarative core, with any arbitrary code kept outside the trust boundary) and link it to the sandbox component in the architecture section and the controls in the security section.

Must-Have Functional Requirements

• A stable workflow engine that runs one core lifecycle for all customers.
• Declarative configuration (schemas, rules, UI metadata) for labels, fields, visibility and validation, so no customer needs a code fork.
• An adapter interface, so external systems plug in and the engine stays untouched.
• Versioned configuration with validation, so every change can be reviewed, tested and reverted.
• A small set of controlled extension points for the rare cases declarative configuration cannot cover.
• Migration and rollback tooling, so a version change never strands a customer on an incompatible config.
• How they fit: the engine is the centre, so there is one release train, one way to operate and one thing to support. Most variation lives in declarative configuration, not code. Integrations go through the adapter interface instead of each becoming a special case. Versioning and validation make customisation supportable. Extension points stay narrow and explicit, never a blank cheque.

Measurable Non-Functional Requirements

• One core release train: a single product ships, never ten customer branches.
• Configuration isolation: no customer's settings, data or test changes can leak into how another customer's workflows run.
• Backward-compatible upgrades: an existing configuration keeps working until a planned migration runs.
• Observable customer-specific behaviour: when a workflow goes wrong, operators can see the config version involved and which rule or adapter fired.
• These shape the system; they are not wishes. No observability means no support. No backward compatibility turns configuration into a liability. Weak isolation turns customisation into a steady stream of multi-tenant incidents.

MVP Non-Goals / Scope Fence

• The MVP (minimum viable product) leaves out:
  • Code written by customers and run inside the workflow engine.
  • Endless per-customer UI branches to meet branding requests.
  • A general-purpose rules language that tries to model any business process.
  • Bespoke integration behaviour that goes around the adapter contract.
  • Automatic, unreviewed migration of every legacy configuration shape.
• These cuts protect the core. The MVP proves that one platform can serve ten customers with safe variation, not that it can model every workflow on day one.

Requirement-to-Component Traceability

• Give every requirement a likely home. The table shows a clean path from requirements to components:

Handling Partial Answers

• If only some questions get answers, don't stall. Assume sensibly and guard the riskiest constraint.
• Say the customers look alike but nobody says who authors configuration. Assume a mix: the customer's admins own the safe declarative fields, and our ops team signs off any extension or schema change. That guards the most fragile line, between safe customisation and unsafe code.
• This tests discovery under ambiguity, prioritisation with gaps, and the habit of protecting delivery by narrowing scope.

The Strong Candidate Move

• The strong move is not "make everything flexible". It runs in order: find the recurring differences, settle who owns configuration, state the integration and upgrade promise, draw a firm line for custom code, then commit to a stable engine backed by declarative configuration, versioning and strict validation.
• That order turns discovery into a buildable plan, not a feature list. Ask only questions that change the design, then proceed on stated assumptions.

Load Shape, Not Marketing Shape

• A tidy first sketch has a single workflow engine, config store, validator and deployment pipeline, all shared. It holds up at average load. Add a deadline and it falls over.
• Customer count is the wrong thing to estimate. Ask instead how many edits, validation runs, activations and approval events land together, and what happens to them as a deadline nears.
• A working assumption: 100 tenants and 50 workflow templates, producing 1,000 configuration versions per day. That traffic mixes small edits, bulk template updates and bursts around release windows.
  • Spread evenly, that is ~42 versions/hour.
  • If half the day's changes arrive inside a 2-hour business window, the peak is ~250 versions/hour. Retries, failed validations and resubmissions come on top.
  • Size well above the average, often by several times, with a growth factor and a headroom target. Then a release-day spike does not turn configuration into an outage.

The Four-Step Capacity Envelope

• Work through the estimate in the order that moves the architecture:
  1. Version intake and validation throughput. Validation means schema, policy and dependency checks plus a dry run. Say they cost 2–10 seconds of CPU-bound work (my rough guess) or a few network calls. Quick user feedback must then split from slower checks that run asynchronously. The near-term SLO (service level objective): the edit saved and came back with a clear result, fast. The deeper promise: validation finished, and promoted or rejected the version, before the deployment window shut.
  2. Deployment throughput. Perhaps only some of the 1,000 versions/day get promoted, but each promotion fans out across tenants, regions or integrations. One template update can reach dozens of workflow instances or queue downstream sync jobs. Fan-out sets deployment capacity, not the number of human edits.
  3. Limits for expressive power. Hard caps on rules, custom fields and plugins pin down runtime cost early. Think of a moderate number of fields per form, a bounded count of rule clauses per transition, and short plugin runs with memory and network limits. Caps stop one customer's customisation from eating the shared budget every other tenant relies on.
  4. State growth and retention. Event history outgrows configuration history, yet config history still counts. Record volume is half the question. The other half is what audit, rollback and support require you to keep, and for what period. Keep the version graph small, store deltas where they pay off, and treat rollback metadata as a first-class record.

SLOs Tied to Customer Pain

• Availability: When users need to, can they open the workflow editor, validate a change and activate a version?
• Latency: How long does the editor sit waiting on a save, a validation preview or an approval routing result?
• Freshness: Once a configuration is approved, how soon does the runtime path start using it?
• Quality: What share of deployments fail validation, get rolled back, or need someone to step in by hand?
• Security: Are audit trails, plugin and approval permissions, and tenant boundaries enforced at the right control points?
• Cost: What does each validated version, activated template or thousand workflow executions cost, and how does adding rules or plugins change that?
• Split the latency budget by step instead of quoting one figure. A save should feel instant and a validation preview may lag a little. Full deployment may take longer, provided you can trust and watch it. Making every step synchronous because it seems simpler is a mistake.

Leverage and What Belongs in Core

• Leverage tells you where a capability lives: shared product code, configuration, or an adapter around one customer's integration.

$$ Leverage=\frac{Customers\ served\ by\ shared\ capability}{Engineering\ effort} $$

• High leverage earns first-class core support. An approval model that 80 tenants use is an example.
• Low leverage, such as one customer's brittle edge-case approval chain, goes into configuration, an adapter or a tightly scoped extension. Don't bend the whole platform around it.
• Count support cost as well. A feature can look reusable yet bring costly onboarding, hard debugging or painful upgrades. Its real leverage may then trail a simpler, narrower design.

A Whiteboard Estimate That Changes Component Choice

• The naive plan validates inside the API that also saves the configuration. That looks fine until you estimate peak load. Worst-case request time then tracks the slowest rule set or plugin call, and one slow tenant holds capacity everyone else needs. The control plane turns brittle.
• With peak validation throughput and deployment fan-out on the board, the design changes:
  • Save → fast and durable.
  • Validation → queued, bounded, observable.
  • Deployment → idempotent and retryable, kept apart from authoring traffic.
• Tenant count matters less than you might think. Component choice and partitioning depend most on how often validation and deployment run, and what each run costs.

Sensitivity Table Across Growth Scenarios

I picked these volumes and peak factors for this case. They are not industry benchmarks: justify them, don't cite them.

• Growth of 10x raises more than cost; it can demand a new partitioning strategy. Custom fields and rules behave the same way. Deeply branched forms full of computed fields may call for cached execution plans, partial evaluation, or checks done at compile time. Plugins that run often may need a sandbox or an adapter boundary instead of running in-process.

🎯 Interview Pointer: Expect a follow-up on why 10x growth calls for new partitioning and not just extra servers. The reason is noisy-neighbour risk. Past a certain point, a shared validator or queue can no longer keep one misbehaving tenant's blast radius contained. Interviewers value that reasoning more than a precise spreadsheet.

Average vs. Peak: Headroom as a Product Feature

• Average load shows whether the system makes economic sense. Peak load shows whether customers can rely on it on their worst day.
• Headroom is the gap between average and peak, and it is not waste. It stops a release window, a rush of approvals or a big tenant onboarding from becoming the day the platform breaks its promise.
• Widen the unit economics question. Running cost is one part; the support load the design creates is the other. A cheap design that needs constant hands-on work can lose to a slightly dearer one that is simpler to roll back, explain and support.

Decision Rule to Carry Forward

• Every estimate should act as a design lever. Does the number move a queue, a quota, a latency budget, a support process or a component boundary? If not, it is likely trivia. If so, record it and be ready to defend it.

Control Plane vs. Data Plane

• To present this system, follow one tenant request end to end, then replay it with a dependency in poor health.
• A hidden constraint: a customer's own approval logic becomes dangerous if it can change how the shared runtime behaves.
• So the design has two halves. The control plane is where configuration gets written, checked, versioned and promoted. The data plane runs live workflow instances. Keeping them apart moves risky change management off the hot path.
  • Control plane duties: editing configs, checking schemas, running tenant contract tests, promoting releases.
  • Data plane duties: workflow execution, runtime UI rendering, connector calls and audit events.
• Asked where the trust boundary sits? Point at this split first.

Architecture Diagram and Trust Boundaries

• Read the diagram top to bottom. A tenant's change request enters the control plane. It passes the config editor, then the schema/rule validator and tenant test harness, then the configuration registry (the system of record), then the feature flag service that gates exposure. The data plane follows: workflow runtime, UI renderer, integration adapter SDK, and approved extensions in a sandbox. External systems sit last: customer APIs, the identity provider and notification services.

• There are two trust boundaries. One lies between content a tenant writes and execution the platform owns. The other lies between platform execution and outside dependencies.
  • Config written by a tenant never runs as code with full platform trust.
  • Even approved extensions stay untrusted enough to need isolation and quotas.
  • You do not control external systems. Give every adapter call timeouts, retries, idempotency where it applies, and a backoff policy that does not make a failure worse.

Core Components in Dependency Order

• The components below follow their dependencies, not the order in which they come to mind.

• Each box earns its place. The validator keeps junk out of the registry. The flag service exists because publishing a version and showing it to every tenant are two different actions. The sandbox exists because approved extensions are valuable, yet letting arbitrary code run at runtime is how customisation ends in incident response.

Failure-Path Overlay: Infinite Approval Loop

• Here the architecture is marked up for the infinite-approval-loop failure drill, so the blast radius is visible. The validator, the test harness and the flags each shrink it at a separate stage. If one layer misses the defect, a later one still prevents a tenant-wide incident.

Happy-Path Sequence: Author to Promotion

• The live release path, one step at a time:
  1. Author config against a versioned schema. The workflow definition, approval graph, field layout and adapter mappings are edited by a customer admin, an implementation engineer or a product manager.
  2. Validate syntax and semantic constraints. Checks cover structure, permitted field types, whether the approval graph makes sense, reference integrity, permissions, and fit with the features switched on.
  3. Run tenant contract tests. It replays typical scenarios: form submission, approval routing, odd edge inputs and integration stubs.
  4. Publish an immutable version. The registry keeps that precise artifact plus its version tag, schema version, release metadata and who wrote it, so it records exactly what was approved.
  5. Roll out behind a tenant flag. One tenant sees it first, then a cohort, then wider traffic once it proves stable.
  6. Observe behaviour. Watch step latency, failed validations, adapter errors, approval-loop detection and how often users finish the workflow.
  7. Promote or rollback. A release that clears the acceptance gate goes active. A release that fails has its flag switched back, or its tenant stays pinned to the earlier immutable version.
• The subtle point: publishing a version is not switching it on everywhere. Keeping those apart stops one bad config from causing a wide outage.
• Some steps should block and some should not. Validation and contract tests are synchronous gates, because they must hold back promotion. Bulk replay, migration and some integration recovery suit async work, so the user's request does not stay open. The runtime usually does both: quick step decisions inline, and external side effects pushed to async work when a dependency is slow or flaky.

Systems of Record, Caches, Queues, and Backpressure

• System of record: published versions belong to the configuration registry; execution history normally belongs to the workflow audit store.
• Caches: recent flag decisions, adapter metadata, form layouts once rendered, and compiled schema descriptors.
• Queues: test harness batches, migration tasks, event delivery, retries and adapter jobs.
• Backpressure and flow control: as adapter queues grow, the runtime should drop or defer noncritical work, not let retries stampede external systems.
• The partitioning key is normally tenant ID, sometimes paired with workflow definition ID or environment. It stops a noisy tenant from spoiling latency for the rest, makes replay easier, and gives quotas an obvious place to be enforced. A surge of approvals from one tenant should stay inside its partition.

MVP vs. Later Evolution

• For the MVP (minimum viable product), stay ruthlessly simple: the runtime, a versioned registry, the validator, feature flags, the tenant test harness, a small adapter SDK, and only as much sandbox as approved extensions strictly need. Prefer immutable config versions and an obvious rollback over a fancier live-editing system.
• Leave for later: config templates shared across tenants, more automated migration, a wider plugin marketplace and deeper visibility into tenant behaviour. Add them once the core release path is stable and easy to support.

90-Second Interview Summary

• "I'd split the system into two planes. The control plane is where people write configs, where schemas and rules get checked, where tenant contract tests run, and where versions are frozen and rolled out with feature flags. The data plane runs workflows, renders the UI and calls adapters. Published versions live in the configuration registry, which is the system of record, and the runtime only reads approved releases. I'd partition mainly by tenant ID, with queues and backpressure shielding external dependencies. The hardest trade-off is how far customer logic may go inside approved extensions, versus keeping it all as declarative config. For the MVP I'd keep extensions narrow, insist on versioned configs, and send every release through a tenant test harness and a staged flag rollout. My first production gate is one tenant, with a rollback I can run straight away."

We reviewed this guide once against our 20-point checklist for design case studies. Almost every point was met on that first pass, so the notes below record the result and flag where the guide is thin.

Phase 1 — Problem Framing & Discovery

• Item 1 (turning the feature request into a business outcome): Met. Section 1 states what the business gains before it names any technology.
• Item 2 (who the stakeholders are): Met. Section 1 lists 8 stakeholder groups.
• Item 3 (questions whose answers reshape the design): Met by the interview tree of six questions in Section 2.
• Item 4 (splitting and ranking requirements): Met. Section 2 keeps must-have functional needs apart from non-functional targets that can be measured.
• Item 5 (what is out of scope): Met. Section 2 names five things the MVP (minimum viable product) leaves out.

Phase 2 — Estimation & Architecture

• Item 6 (rough sizing and capacity arithmetic): Met. Section 3 runs a worked estimate, a four-step envelope and a sensitivity table.
• Item 7 (cost per unit, split by driver): Partly met. Section 3 lists cost per validated version among its SLO (service level objective) dimensions, yet no table separates cost into its drivers.
• Item 8 (the whole architecture and how data moves through it): Met. Section 4 gives the architecture diagram, a table of components and a sequence diagram.
• Item 9 (records and API contracts): Met. Section 5 specifies the four records and the four API contracts, and each contract states its auth, idempotency and error codes.
• Item 10 (whether to build or buy, and which vendor or model): Not met. The guide compares configuration with code and core with adapters, but never weighs vendors or BPM products against each other.

Phase 3 — Trade-offs, Security & Reliability

• Item 11 (named trade-offs, each with a fair verdict): Met by the four pairs in Section 8.
• Item 12 (threats and the controls that answer them): Met by Section 6, which sets out four security controls.
• Item 13 (drills for failure modes and reliability): Met. Section 6 runs a loop drill in four parts, four more failure cases and a policy table with six rows.
• Item 14 (how the system is tested): Met. Section 5 includes contract tests and failure-injection tests; Section 6 adds a test for the publish-path invariant.

Phase 4 — Delivery, Governance & Communication

• Item 15 (metrics in layers, plus observability): Met. Section 7 names 6 metrics inside a framework of four layers.
• Item 16 (staged rollout, risk register and gates for rollback): Met. Section 7 sets out four rollout phases, a risk register with four rows and the conditions that trigger rollback.
• Item 17 (depth on regulation and governance): Not met. Beyond the general audit trail, the guide says nothing about data residency or industry audit regimes.
• Item 18 (responsible AI and risks past the obvious failure): Not met. No AI component exists in this design; every risk it frames concerns safe configuration and keeping tenants apart.
• Item 19 (how people adopt the change): Met. Section 7 tells the rollout story, assigns owners and tracks adoption and the number of forks.
• Item 20 (a structured talk plan and a rubric to score yourself): Met. Section 8 provides the minute-by-minute plan, the 90-second summary, a scoring rubric, a rehearsal checklist and a practice plan.

Four items fall short: 7, 10, 17 and 18. Each gap is a deliberate choice of scope or emphasis in the design, not something lost while writing it up, so a single review pass was enough.

My Perspective on the Gaps

These notes reach beyond the core design. Use them if an interviewer presses on one of the gaps.

Item 7 — cost per unit, split by driver.

• Base the cost model on Section 3's four-step capacity envelope. Four lines are enough. Line one prices each validated version: the CPU and network spent on schema, policy, dependency and dry-run checks. Line two prices each deployment; fan-out to tenant instances and downstream sync jobs drives it, while raw edit volume barely matters. Line three is the price of each second a plugin runs in the sandbox. Line four is storage for the version graph and the append-only ConfigDeployment ledger.
• Say one driver out loud: tenants, not the platform design, set validation cost. Forms with deep branching or rules with many clauses make validation dearer even when nothing on the platform side moves. So the limits on expressive power in Section 3 (maximum fields, maximum rule clauses, bounded plugin windows) control cost as much as they control safety.
• Report cost per tenant, line by line. One noisy tenant then shows up clearly instead of disappearing into a platform total. That view turns "should this customisation sit in config or in an adapter?" into a debate about real figures rather than about leverage in the abstract.

Item 10 — whether to build or buy, and which vendor or model.

• Here the obvious build-or-buy question is the workflow and rules engine. You could build the declarative engine laid out in Sections 4 and 5. Or you could buy a BPM or workflow-orchestration product (a commercial rules engine, say, or workflow-as-a-service) and keep the same split between control plane and data plane around it.
• A rule of thumb: buy what is already solved and commoditised, where the vendor owns its security record and competes on it. Identity and SSO (single sign-on) federation is the standard example, and it fits here if customer admins must log in through an enterprise IdP. Build the workflow engine, though. What sets this platform apart is its constrained, tenant-safe configuration model, with validation, versioning and sandboxing. No vendor can take responsibility for that in your place. It is also what the interviewer wants to see you design.
• The adapter SDK falls between the two. Where a connector framework already exists for a common system, such as CRM (customer relationship management) or ticketing, reuse or buy it. Keep the adapter contract and its permission model in-house: the security design in Section 6 depends on that trust boundary.

Item 17 — depth on regulation and governance.

• Section 4 already isolates tenants: it separates control plane from data plane, partitions data by tenant ID and scopes each adapter to one tenant. Data residency extends that. When customers live under different jurisdictions, the configuration registry and the workflow audit store may need storage fixed to a region. Section 5's TenantConfig and ConfigDeployment records would each gain a region field. The deployment ledger could then show where every tenant's data was held and where its workflows ran.
• Tie this to the audit control in Section 6, which logs who published each config version. That durable log already answers most of what a SOC 2 auditor (criterion CC8.1) or a review of SOX (Sarbanes-Oxley Act) IT general controls asks about change management: who signed off, what was tested and what shipped. It does less for HIPAA (Health Insurance Portability and Accountability Act) if the workflow handles health data, because HIPAA's audit-control rule targets activity on any system storing that data, beyond config changes alone. The real gap is therefore modest. What it mostly lacks is an explicit retention period per regulation, added to the retention rules Section 5 already sets.
• Bring this up in an interview before anyone asks. Pick one regulation that matches the customer's industry; SOX-style segregation of duties for approval workflows in financial services is a good example. Then show how the existing approval roles and audit trail satisfy it, with no new infrastructure.

Item 18 — responsible AI and risks past the obvious failure.

• Nothing in this design is AI. Its "intelligence" consists of deterministic validation, policy checks and rule evaluation. Bolting on a responsible-AI story would sound hollow. Tell the interviewer that directly instead of reaching for generic AI-safety wording.
• That may change. A later release might add an AI helper for authoring, say one that drafts an approval graph from a plain-language description. The fail-closed stance of Section 6 applies unchanged. Whatever the AI proposes is untrusted input. It passes through the same schema, policy and cycle-detection gates as config a person wrote, and it never gets a route around them.
• Under pressure from the interviewer, state the trade-off. AI-assisted authoring adds risk at the very boundary this design already treats as hostile: the path from config editor to validator in Section 4. It raises the volume of untrusted input that reaches the validator. The trust model stays the same.

• Reframe "configuration" as the outcome: one stable product core with customisation that is safe, versioned and supportable, so ten customers never become ten forks.
• Measure how often a new customer variation ships with no fork, no hotfix and no support escalation, not how many fields are configurable.
• Who writes the configuration is an architecture question, because customer admins need validation, previews, narrow building blocks and safe defaults.
• Size for the peak: 1,000 config versions a day is about 42 an hour, but half of them inside a 2-hour window is about 250 an hour before retries.
• Keep validation out of the save API, so save stays fast and durable, validation is queued and bounded, and deployment is idempotent and retryable.
• Deployment capacity is set by fan-out across tenants, regions and integrations, not by the number of human edits.
• Split the control plane, where configs are written, checked, versioned and promoted, from the data plane that runs live workflows, so risky change stays off the hot path.
• Publishing an immutable version is not switching it on: a tenant flag exposes it to one tenant, then a cohort, and rollback flips the flag back to the previous version.
• The append-only ConfigDeployment ledger, not the config record, says which version is live, so a rollback is a new event and never an edit in place.
• Publish runs parse, validate, authorise-by-policy, persist in that order, so cheap structural checks fail fast and nothing is written until every check passes.
• Create-style calls carry an idempotency key: the same key and body return the same deployment, a changed body gets 422 and a retry still in flight gets 409.
• Arbitrary code is off by default, and any approved extension runs in a sandbox with least privilege, resource caps and limits on outbound traffic.
• An infinite approval loop is caught by cycle detection at publish or a max-hop guard at runtime, and fails closed for that version while the tenant falls back to the last known-good one.
• Partition by tenant ID so one tenant's approval surge stays in its partition; at 100 times the baseline that becomes per-tenant quotas and sharded queues.
• Widen the configuration surface only after a second and third customer reuse the same mechanism, and steer by fork count and upgrade time per tenant.

On a shared workflow platform for ten customers, the levels separate on how far the answer goes past "make it configurable".

• The Idempotency-Key HTTP Header Field (IETF draft -07) — 422 for a reused key with a different body, 409 for a retry while the first request is still running, replay returns the original result
• RFC 9110: HTTP Semantics — meanings of 409 Conflict, 412 Precondition Failed for a failed If-Match, and the 422 Unprocessable Content name
• AICPA 2017 Trust Services Criteria (revised points of focus, 2022) — SOC 2 change-management criterion CC8.1
• PCAOB AS 2201: An Audit of Internal Control Over Financial Reporting — SOX audits test IT controls, including program changes and segregation of duties
• 45 CFR 164.312(b), HIPAA audit controls — HIPAA's audit rule covers activity on systems holding health data