Skip to content
Forward Deployed

Forward Deployed
Lessons and contents
Staff-level platform architectureLesson 4 of 6
6 lessons
Learn

Staff-level platform architecture: think like a staff engineer

Source
From the Staff AI System Design course
By
Sourav Banerjee
Last updated
8 Nov 2026

Report an issue

Apply Earlier Decisions to the Platform

1.1 Requirements (T18.2). T18.2 turned Alice's need into measurable acceptance criteria before choosing components. Here the customers are four app teams and three tenants, so the criteria name onboarding, rejected violations and rollback evidence, not only latency.

11.1 Fine-tuning fundamentals (T18.11). T18.11 ran an adapter challenger only after prompting, retrieval and schema fixes failed on measured behaviour. The platform keeps that order: a manifest may reference an approved adapter, but the template does not offer tuning as a default step.

11.2 Fine-tuning platform architecture (T18.11). T18.11 created an adapter only through Fine-Tuning Jobs, a versioned dataset and a registry approval gate, and T18.12 kept each approval in the tenant region's authority store. The platform stays on the consuming side of that workflow. A manifest may reference only a registry-approved adapter digest, and the cell rechecks that approval before it applies a route. Creating or retraining an adapter never runs from the app template.

11.3 Distributed training infrastructure (T18.11). T18.11 kept distributed training at orientation depth, and T18.12 copied finished adapters by digest rather than training in each region. It stays deferred here: the platform routes approved artefacts and has no measured training workload that would justify multi-accelerator jobs.

14.1 Evaluation architecture (T18.9). T18.9 froze the bundle and dataset that a release gate judged. Shadow evidence applies the same rule to routes: it binds to a baseline generation, and a changed bundle needs fresh evidence.

14.3 Model and data lifecycle (T18.11). T18.11 traced each model to its dataset version and recipe. Because manifests pin the bundle, withdrawing a dataset now names every app route that must move before its descendant model retires.

15.2 SRE and production operations (T18.8). T18.8 set latency and reliability budgets and a bounded incident procedure. Centralising contracts does not move regional on-call to the platform team: the platform owns contract and control-plane availability, and each cell's SRE owner still owns recovery.

16.2 Cloud networking and deployment (T18.12). T18.12 kept private endpoints and controlled egress inside each cell. Platform templates deploy only into those approved network boundaries, and a manifest cannot add an endpoint outside them.

17.1 AI unit economics (T18.12). T18.12 priced cost per successful workflow, not per request. Platform showback keeps that denominator, so retries and failed calls stay visible in each team's cost.

17.2 Capacity planning (T18.12). T18.12 sized each cell's request, token and slot capacity. Onboarding an app adds demand, not capacity, so the pilot review checks the target cell's headroom before cutover.

Define Ownership and Review Authority

Boundary Accountable owner What the owner proves
Customer task, prompt and tool behaviour, and outcome App team Supplier acceptance cases, safe business effects, support escalation
Identity, data purpose, policy and exception rules Security and data owner Current resource access, retention, approved provider and region constraints
Shared API, gateway, fair budgets and manifest compatibility Platform team Contract versions, rollout compatibility, operational limits
Model datasets, candidate lineage and evaluations Model operations with app reviewer Approved examples, reproducible evidence, retirement descendants
Regional data durability, promotion and restoration Database and SRE owner Writer fencing, watermarks, restore exercises and capacity
Architecture decision and migration gates Named decision owner Alternatives, dissent, hard constraints, evidence and reversal trigger

The app owner accepts customer behaviour, the platform owner accepts contract and capacity changes, the security and data owner accepts purpose and isolation, and the incident commander coordinates an outage. Keep one named accountable owner per decision rather than a committee. In this hypothetical operating model, hold a weekly pilot review and a monthly cost and capacity review. Emergency changes still record identity, current eligibility, generation, reason and post-incident follow-up.

An SLO needs a specified population, window and valid outcome. App completion and quality, platform request availability, policy and audit freshness, and recovery exercises have different owners and indicators. An exhausted error budget pauses risky rollout under an agreed policy; it never disables authorisation. Agree escalation and support handoffs before an app leaves the pilot.

An ownership failure, illustrated. In the week-4 pilot the canary cell's error rate rises. The app team assumes the platform team will roll back; the platform team waits for the app owner's outcome call. Nobody acts for 40 minutes. The signal is an open incident with no named commander and a rollback request with no approver in the audit log. The fix is a runbook line: during a pilot the release owner alone decides rollback, and the incident commander takes over once an incident is declared.

Decision Exercise: Veto the Top Score and Write the ADR

Changed constraint: Tenant B's regulator now requires that no record controlling B's processing, even an opaque route generation or the key that signs it, is stored or operated outside the EU. The global control plane runs in the US and holds both. Use the same weights as before: delivery 0.2, isolation 0.3, operability 0.3, evolvability 0.2.

Option Delivery Isolation Operability Evolvability
Shared library plus templates 5 2 2 2
Regional contracts, one global control plane 3 5 4 4
Regional contracts, separate EU control plane for EU tenants 2 5 3 3
  1. Check each option against the hard constraints (residency including the new rule, and current access in trusted runtime) before computing any score.
  2. Compute weighted totals for the options that survive.
  3. Write a one-page ADR: context, options, decision, consequences and reversal criteria. Add an evidence ledger that marks each number measured, assumed or proposed, and a change record naming what it supersedes.
  4. Write a memo section of at most 150 words for Tenant B's data protection officer.

Rubric. Pass only when every line is observably true:

  • Every option gets a pass or veto verdict before any total, and the new residency rule removes the global-control-plane option.
  • The weighted arithmetic is shown and the scores are labelled as assumptions.
  • The decision names one accountable owner.
  • Consequences include one new cost and one new failure mode.
  • Each reversal criterion is an observable event or threshold.
  • The evidence ledger separates measured, assumed and proposed items and claims no measurement that does not exist.
  • The change record supersedes ADR-012 in part without editing it.
  • The memo answers the reader's question first, uses no unexplained acronym and stays under 150 words.
Model ADR and memo, reveal after writing yours

Model exercise ADR (unnumbered, not an accepted decision): EU control plane for EU tenants. Status: proposed. Change record: supersedes ADR-012 in part, for control-plane placement of Tenants A and B only.

Context. Tenant B's regulator forbids storing or operating B's control records outside the EU, even opaque ones. Under ADR-012 and ADR-013 the global control plane in the US stores and signs B's route generations, which name B's apps only by opaque IDs, and holds the signing keys.

Options. Hard constraints first. Templates only: pass; templates add no record outside the EU, and T18.12's regional runtime still checks current access. One global control plane: veto, because B's route generations and signing keys sit in the US. Separate EU control plane: pass; B's control records and keys stay in the EU, and runtime checks are unchanged. Totals for the survivors: templates 0.2×5 + 0.3×2 + 0.3×2 + 0.2×2 = 2.6; EU control plane 0.2×2 + 0.3×5 + 0.3×3 + 0.2×3 = 3.4. The vetoed option would have won at 4.1.

Decision. Run a second control-plane instance in the EU, with EU-held signing keys, for Tenants A and B. Tenant C stays on the US instance. Cells, runtime checks and migration gates do not change. Owner: the platform owner, with approval from the security and data owner.

Consequences. Two control planes to operate and keep on one contract version; platform releases now pass two canary cells. A new failure mode: the two instances drift to different validator versions. An EU control-plane outage freezes EU changes only.

Reversal criteria. Revisit if the regulator accepts EU-held keys on a shared control plane, or if version drift between the instances blocks a release twice in one quarter.

Evidence ledger. Measured: none yet. Assumed: scores, weights and the current control plane's US location. Proposed: the EU instance, EU keys and two-canary rollout. Needed before approval: the regulator's written confirmation and a pilot of contract-version sync.

Memo section for Tenant B's data protection officer. Your processing records will stay in the EU. Your app settings, approvals and policy are already held only in the EU. Today a US system signs the route records that tell EU servers which approved setup to use; those records name your apps by reference numbers only, and the US system holds the signing keys. From the pilot onward, an EU-only system will sign them, with keys kept in the EU. Customer content was never in either system, and that does not change. We need your written confirmation that this meets the regulator's rule before the pilot starts.

A full design memo uses the same structure at greater length: incident and requirements, scope and assumptions, before and after diagrams, options, budgets and TCO, failure and security boundaries, owners, migration gates, reversal criteria and evidence gaps. A product leader needs customer effect and staged cost; an SRE needs failure and recovery; a security reviewer needs every data copy and authority boundary. Keep one factual core and change only the emphasis, never the diagrams.

Decision portfolio Still accepted constraint Trigger for a new decision
ADR-001, ADR-002 Commit saved success; durable jobs and unknown-effect recovery Changed persistence or execution guarantees
ADR-003, ADR-004 Current authorised source versions; immutable caches; shared authority Measured retrieval or persistence bottleneck
ADR-005, ADR-006 Finite fair budgets and trusted audited disclosure New isolation or workload policy
ADR-007, ADR-008 Data-class consistency and version-bound quality releases Stronger recovery or changed gate evidence
ADR-009, ADR-010 Approved inference route and governed model lineage Measured serving or training requirement change
ADR-011, ADR-012, ADR-013 Eligible regional cells and controlled platform adoption Zero-loss or global-write requirement, control-record residency rule, or failed adoption economics

Accepted ADRs stay immutable apart from supersession metadata. A new decision records changed assumptions and references the old one. Do not rewrite history to make the current design look inevitable.

Key Takeaways

  • Keep one named accountable owner per decision. When nobody rolled back a degrading canary for 40 minutes, the fix was a runbook line making the release owner the sole pilot rollback decider until an incident commander takes over.
  • If Tenant B's regulator forbids control records outside the EU, that hard constraint vetoes the global control plane even though it would have scored 4.1. Among the survivors, a separate EU control plane scores 3.4 against 2.6 for templates.
  • An exhausted error budget pauses risky rollout but never disables authorisation. Centralising contracts does not move regional on-call, because each cell's SRE owner still owns recovery.
  • Accepted ADRs stay immutable apart from supersession metadata. A new decision records the changed assumptions and references the old one rather than rewriting history.

Rate your confidence and keep notes on the module page.