Global multi-region platform: validate and carry forward
- Source
- From the Staff AI System Design course
- By
- Sourav Banerjee
- Last updated
- 8 Nov 2026
Recall and Exit Gates
- What does RPO 60 seconds allow, and how is the loss bound computed?
- What must precede writer promotion, in order?
- What limits a 60 RPS workload besides app replicas?
- Why does the EU hostname, not the user's location, keep A/B content in the EU, and what does a 20-second DNS TTL buy?
- The usage record for attempt 2 arrives twice, then a third record changes its cost. What does the ledger do?
Reveal after answering
1: Up to 60 seconds of acknowledged writes may be lost; it is not zero loss. The bound is last renewal + 20-second lease − watermark, frozen at isolation, because a partitioned primary can acknowledge until its lease expires.
2: Lease expiry plus drift margin, a new quorum lease (epoch 8), epoch 8 installed in every store the new writer will use with each refusing an epoch 7 probe, and EU-primary's stores quarantined. Then an allowed destination that is not the failed region, current policy and approval generation, audit, complete artefacts and capacity.
3: The 3.6 million tokens/minute and request quotas, 120 assumed active slots, database/search, network, and model readiness.
4: Geo-routing picks by the resolver's location, so a user in the US could reach US ingress, where TLS ends. The EU hostname resolves only to EU ingress, and its key exists only there. A 20-second TTL plus 5 seconds of update fits the 30-second budget for moving between EU ingress points, for clients that honour it.
5: It counts the attempt once and rejects the changed record as conflicting evidence for investigation.
60-second card: bind each tenant to its residency hostname; keep authority and tombstones on a 2-of-3 EU quorum; promote after lease expiry, epoch 8 installed and epoch 7 refused; fail closed without quorum; bound loss as last renewal + lease − watermark; measure end-to-end RTO; reserve allowed-region tokens and slots; hold ambiguous operations; count every attempt per successful workflow; fail back with one lease move.
Exit gates for this stage:
- Explain, in two minutes without notes, why a healthy US endpoint is ineligible for Tenant A and why 1 of 3 EU voters blocks promotion.
- Draw the after-architecture: residency hostnames, the US-home sibling, the EU authority quorum with its witness, async EU replication, the promotion gate and the fail-closed branch.
- Defend two alternatives with this stage's numbers: residency hostnames against an anycast load balancer with EU-only edges (16.2), and partial warm capacity against a full-peak reserve under the 2 million tokens/minute quota (17.2).
- Diagnose a failed promotion from an observable signal (an accepted epoch 7 probe, a loss bound over 60 seconds, or an approval generation mismatch) and name the mitigation.
- Build and debug the 17.1 slice (build depth): run
labs/test_regions.py, then add a test that asserts your 700,000-success prediction withunit_cost. AddLedger.adjust, which appends a late-invoice adjustment against an existing attempt without changing its record and raises for an unknown attempt. Test that +500 micro-USD on attempt 2 keeps the original record and raises the total by 500. Explain each ledger result.
Evidence and Carry Forward
Read the architecture snapshot, decisions and ADR-013. Local content verification and learner completion are separate. ADR-013 amends ADR-011 with authority placement, endpoint residency and fencing order; ADR-011's text is otherwise unchanged. T18.13 inherits these regional cells, the EU authority quorum, model lineage, recovery gates, and explicit deployment gaps to define shared platform contracts and migration.
Review after about one, seven, and 21 days. Stop when the five gates pass. These learner gates remain unassessed. Production integrations and deployment controls described here are proposals unless the validation report explicitly records an executed check.
Key Takeaways
- An RPO of 60 seconds allows up to 60 seconds of acknowledged writes to be lost. The bound is last renewal plus the 20-second lease minus the watermark, frozen at isolation, because a partitioned primary can acknowledge until its lease expires.
- Promotion order is lease expiry plus drift margin, a new quorum lease, the new epoch installed with every store refusing the old one, and EU-primary's stores quarantined. Then come allowed destination, current policy, approval generation, audit, artefacts and capacity.
- A 60 RPS workload is limited by token and request quotas, active slots, database and search, network and model readiness, not by app replicas alone.
- The usage ledger counts a twice-delivered attempt once and rejects a record that changes an existing attempt's cost as conflicting evidence for investigation.