Staff-level platform architecture: observe and experiment

The offline walkthrough traces migration intent through a controlled route change, including mismatch, revoked-previous-route and control-plane-outage branches.

Migration sequence. CAS changes a pointer only if its expected generation still matches. It prevents stale metadata updates; it does not stop an unfenced old region from writing.

Use a strangler migration: route selected new work through the new contract while the old path serves unconverted work. Keep one authoritative writer and stable operation IDs. Avoid dual writes to independent result stores; an outbox plus a replayable projection is safer where eventual views suffice. Schema changes add compatible fields first, migrate readers and writers, verify backfill, then remove old fields after the reversal window. A route rollback cannot undo a destructive schema migration, erase a tombstone or revoke a remote side effect.

Production shadow evidence binds manifest, release bundle, population, evaluator, source policy, time window, and the baseline route and generation it was compared against. It must not send protected content to an unapproved route, consume unbounded quota or invoke business tools twice. Evidence is append-only: a later clean report cannot erase a recorded mismatch. If the route changes after the comparison, the evidence is stale and the change needs a fresh comparison under a new change key. The local gate needs 100 matched examples. Zero mismatches in 100 independent, representative samples bounds the true mismatch rate below about 3% at 95% confidence (the rule of three, 3/100). It is a demonstration gate, not proof of safety.

TCO adds operating cost, migration, support, the overlap while old and new paths both run, and expected change effort over an explicit horizon. It differs from T18.12's per-success cost because it compares organisational choices. It counts labour cost and keeps opportunity cost qualitative. Count shared costs the same way in both options, and do not claim savings from costs that remain.

Assume a three-year horizon without discounting, common serving spend of $20,000 a month in both options, and a loaded planning labour rate of $100 an hour. Four teams spend 30 hours a month each on duplicated platform work: 120 × $100 = $12,000 a month. A shared platform needs 50 hours a month of operations plus 20 of adoption support, $7,000, plus $1,000 of control-plane infrastructure: $8,000 a month. Once the old paths are retired, the saving is $12,000 − $8,000 = $4,000 a month. Migration takes 600 one-time hours, $60,000.

If the old paths stopped on day one, break-even would be $60,000 ÷ $4,000 = 15 months and the three-year saving $84,000. The strangler migration says otherwise. While cohorts move, the old duplicated work and the new platform both run. Call those months the dual-run overlap. Assume six.

With a six-month overlap and the inputs above, predict the break-even month and the three-year saving before you open the lab.

At these inputs, break-even month = 15 + 3 × overlap months, and each overlap month cuts the three-year saving by $12,000. A seven-month overlap erases it. If migration doubles to 1,200 hours, break-even moves to month 48 and the platform costs $48,000 more over three years. If support rises from 20 to 60 hours a month, the monthly saving falls to zero and break-even never arrives. Shortening the overlap, by setting retirement dates per cohort, is the strongest lever. A team that ships customer work sooner has opportunity value, but do not invent a revenue figure; add measured adoption, incident load and delivery evidence before claiming it. These assumptions exclude tax, discounts, financing and revenue. Ask finance which costs are avoidable and which only move between teams.

ADR-012 records the no-overlap figures of 15 months and $84,000. Accepted ADRs are immutable, so ADR-013 records the dual-run figures and ADR-012 carries only an amendment note.

Open the offline lab. Change migration effort (0–1,200 hours), shared operations and support (30–150 hours a month), dual-run overlap (0–12 months) and planning horizon (12–60 months). Fixed assumptions: 120 baseline hours a month, $100 an hour, $1,000 a month of control infrastructure and $20,000 a month of common serving. Outputs: monthly saving after the overlap, up-front cost, break-even month, both horizon TCOs and their difference. Zero or negative savings report no break-even rather than dividing by zero.

Inputs read: migration hours, shared hours a month, overlap months, horizon months. Output columns use the lab's labels, shortened; all values are USD except break-even.

Run this stage's Python lab in your terminal to see each idea in the stage hold. It needs Python 3.10 or later and nothing else: the standard library and SQLite only.

From the root of your local clone of the repository, run:

python3 10_AI_System_Design/stages/12-staff-platform/labs/test_platform.py -v

-v prints one line per test, and each test name states one behaviour. Read a test, change one value in the lab code, predict which test fails, and run it again.

The SQLite controller stands in for the regional migration controller's change records and one cell's route record. It has no signatures, pull loop or staleness timer. It persists proposal identity and appends shadow reports bound to the route and generation they were compared against. It rejects side-effectful comparison, applies a fixed sample and mismatch floor, and changes route plus generation atomically with a history row. apply() stands in for a pulled, signed generation: it needs current eligibility and refuses any generation not above the current one. Rollback needs actor and reason and writes generation N+2 carrying the previous route, so a later attempt to apply N+1 is refused. It also clears the previous pointer, so a rollback cannot itself be rolled back; going forward again needs a fresh, gated change. inspect() reports the change's state and the cell's route separately: after a rollback the change still reads cutover, while route and generation show what the cell serves. Fourteen tests cover durable restart, lost acknowledgement, stale generation, stale evidence, mismatch retention, history, inspection after revocation, policy blockers, cross-tenant lookup, rollback eligibility, refusal of a rolled-back generation and injected pre-commit failure. Eligibility booleans stand in for trusted evaluation. The controller never migrates a result row or executes a customer request.

Inject fail_before_commit=True at cutover. Expect injected pre-commit failure, with route, generation and history unchanged after reopening the database, and a later valid cutover to succeed. After a committed cutover with a lost acknowledgement, call inspect() or repeat the same-key proposal; both return the persisted state, even after the bundle is revoked. Do not run another cutover with the old generation. Observe stale-update and stale-evidence counts, blocked proposals, shadow mismatch by bounded slice, time in each phase, pilot outcomes, incident burden and cost reconciliation.

A canary fails, but the old model's data approval has been revoked. Should rollback ignore current eligibility to restore availability?

• A strangler migration routes selected new work through the new contract while the old path serves the rest, with one authoritative writer and stable operation IDs. A route rollback cannot undo a destructive schema migration, erase a tombstone or revoke a remote side effect.
• Shadow evidence is append-only and bound to its baseline route and generation, so a route change makes it stale. Zero mismatches in 100 representative samples bounds the mismatch rate below about 3% at 95% confidence, by the rule of three.
• With no overlap, $60,000 of migration against a $4,000 monthly saving breaks even at 15 months. A six-month dual-run overlap moves break-even to month 33 and cuts the three-year saving from $84,000 to $12,000.
• Rollback still checks current eligibility. If the previous model's data approval is revoked, block the rollback and choose another approved route, because a historical pointer is not present permission.