Global multi-region platform: think like a staff engineer

The incident commander owns promotion authority. The authority owner proves the new lease and the rejected old epoch. The database owner proves the loss bound at isolation. Security verifies current policy and audit, model operations verifies the approved generation and quota readiness, and the application owner checks customer outcomes. Observe regional success, authority quorum health, placement denials, route convergence, the loss bound at isolation, unknown operations, token exhaustion, and per-success cost. Redact content and avoid tenant IDs in unrestricted metric labels.

Before failback, discard or reconcile EU-primary's quarantined post-isolation writes, rebuild its stores, indexes and caches from the new authority's versions and tombstones, reconcile operation/usage evidence, test readiness, and move the lease once through the quorum. Do not reverse DNS while both regions accept writes. Test quarterly under the hypothetical operating plan and after changes to replication, policy, networking, or model startup. A tabletop pass does not establish RTO; a timed integrated exercise is still required.

Prompt: A/B need recovery and C needs a US home.

Spoken answer: “I would place each tenant in an approved home cell with a same-geography recovery target: EU-primary and EU-secondary for A and B, US-home for C. Different tenants are active in different regions, but each tenant has one writer at a time. A lease and epoch from an EU-only 2-of-3 quorum enforce that. Global concurrent writes for one tenant would add conflict resolution, global policy freshness, shared quota and operation deduplication, and would copy A/B content outside the EU. My design costs recovery delay plus an idle EU reserve of $3,000 a month in our assumptions. I would accept multi-writer only for a measured local-write requirement with clear residency and ownership rules.”

Trade-off: one writer per tenant makes ownership provable but adds recovery delay and latency for distant users.

Follow-up: If Tenant A opens an office in Singapore and its users complain about latency, what changes? Not the writer, and not the endpoint. Singapore users still call the EU hostname, whose TLS ends only in EU ingress, so their location never decides where content is decrypted. Faster service needs a new placement policy agreed with Tenant A. A read cache outside the EU would still be a copy of restricted content.

Prompt: The route switches in thirty seconds but the model loads for ten minutes.

Spoken answer: “The RTO runs from incident start to validated customer success, not to the DNS change. Our plan is 60 seconds of detection, 30 of fencing, 180 of restore and model load, 30 of route convergence and 60 of validation. That is 360 seconds against a 600-second objective. The 30-second route switch is one substep. If the model takes ten minutes, the plan becomes 780 seconds and misses the RTO before traffic moves. I would report the missed objective, not relabel it. Then I would fix the binding resource: keep the approved adapter warm in EU-secondary, or agree a longer RTO with the customer. Safety gates still decide promotion, because a late but safe recovery beats a fast one with two writers.”

Trade-off: warm readiness costs idle capacity, which sits in the reserve line of the cost per success.

Follow-up: If the customer offers a smaller fallback model that loads in 60 seconds, what changes? The plan fits at 240 seconds. The fallback still needs its own approval, evaluation evidence and EU provider profile; it is a different release, not a faster copy.

Prompt: Finance reports cost per API request while customer completion falls.

Spoken answer: “I would keep request and token costs for diagnosis, but the business denominator is successful customer workflows. Here 1,250,000 charged attempts at $0.004 cost $5,000. Add $17,500 of fixed infrastructure, reserve, networking and support, and 800,000 successes cost about $0.028 each. Dividing by the 1,000,000 accepted requests shows $0.0225 and hides that one in five failed. A failover month with 20,000 retried attempts adds $80 and no new successes. The ledger keys usage by tenant, operation and attempt, so a redelivered record counts once and a cross-region retry counts as extra spend. At zero successes, unit cost is undefined, not zero. I would investigate quality and retry amplification before buying more capacity.”

Trade-off: workflow attribution needs operation and attempt IDs on every usage record, which is more work than counting HTTP requests.

Follow-up: If the provider invoice arrives 30 days late and differs from the ledger, what changes? Record an adjustment against the original attempts with the invoice as evidence. Never silently replace the earlier figure, because the difference may reveal a duplicate or missing attempt.

Prompt: The primary region fails and a healthy US endpoint exists.

Spoken answer: “I would enforce allowed processing and storage regions in trusted services: a residency hostname whose TLS ends only in EU ingress, a placement check that rejects other tenants before reading the body, scoped data paths, and approved EU provider profiles at the Model Gateway. Authority records stay in the EU too, on a quorum of EU-primary, EU-secondary and an EU witness. A healthy US endpoint is not a valid target for A or B, and neither is the failed EU-primary. If EU-secondary cannot get a lease from 2 of 3 voters or read current policy, I return unavailable and escalate, even if that misses the 600-second RTO.”

Trade-off: strict residency lowers availability during a wide EU outage.

Follow-up: The customer wants one global address for all its users. What changes? A shared name with DNS geo-routing lets user location pick the ingress, so a traveller's request could be decrypted at US ingress; decline it. Offer anycast with EU-only edges for the EU host: one fixed address per residency, at the cost of auditing the provider's edge set and key placement. One address for EU and US tenants together would decrypt before placement is known.

Prompt: EU-secondary's provider quota is 2 million tokens/minute, and the A/B peak is 60 RPS at 1,000 tokens per request.

Spoken answer: “I would size each budget separately. The peak needs 3.6 million tokens a minute, five app replicas at 14 RPS each, and about 120 active slots by Little's law. A 2-million quota admits only about 33 RPS, so app replicas are not the bottleneck. I would choose partial warm capacity with shedding: admit 33 RPS split by contract share, roughly 20 for A and 13 for B. The other 27 RPS is rejected with a retry-after signal instead of queued. If customers need the full 60 RPS during recovery, I would raise the reserved quota, pay for it in the fixed reserve line, and rerun the cost per success. Scaling on failover is cheapest, but acquisition and model load can exceed the 600-second RTO.”

Trade-off: shedding keeps the reserve cost down but turns an outage into partial rejection for both tenants.

Follow-up: If Tenant B's contract guarantees full service during recovery, what changes? Reserve B's full share first, 24 RPS (40% of the 60 RPS peak), and give A the remaining 9 RPS of the 33 RPS, or buy more quota. The contract changes the split, not the quota.

• Different tenants can be active in different regions, but each tenant has one writer at a time. Global active-active for one tenant adds conflict resolution, shared quota and operation deduplication, and copies A/B content outside the EU.
• The RTO runs to validated customer success, not to the DNS change. A ten-minute model load misses the 600-second objective before traffic moves, so report the miss and keep the adapter warm or agree a longer RTO.
• Before failback, reconcile or discard EU-primary's quarantined writes, rebuild its stores, indexes and caches from the new authority, and move the lease once through the quorum. Never reverse DNS while both regions accept writes.
• A late provider invoice becomes an adjustment against the original attempts and never silently replaces the earlier figure. At zero successes unit cost is undefined, not zero.