Reliability and consistency: validate and carry forward
- Source
- From the Staff AI System Design course
- By
- Sourav Banerjee
- Last updated
- 8 Nov 2026
Recall and Exit Gates
- Why can a saved answer be absent on a replica? Its commit may not yet be applied there; use the trusted minimum or primary.
- What is atomic in the outbox fixture? Result and local notification intent, not external delivery or model execution.
- What does a contiguous watermark exclude? Missing earlier events despite later observations.
- Why retain unknown provider state? A lost response is not proof of no external work.
- Which recovery claims are untested? Real consensus, replication, database promotion, restore and production error/latency objectives.
The 60-second card: consistency per class; one authoritative writer; minimum-version reads; transactional intent; duplicate-safe contiguous projection; bounded attempts/probes; critical-path diagnosis; recover with evidence. Review at about one, seven and 21 days.
- Explain lag, local atomicity and external uncertainty without notes in two minutes.
- Draw primary, outbox, projection and protected-authority read paths.
- Defend current authority versus eligible replica reads and a recovery alternative.
- Diagnose rollback, lost response, lag, outage and latency with concrete state/signals.
- Run/modify the fault and probe tests and write a deployment recovery drill; learner completion remains unassessed.
Decisions and Next Dependency
ADR-007 and the stage snapshot preserve the evidence.
A future T18.9 adds versioned quality evaluation and release controls. It inherits these telemetry, transaction and recovery boundaries, and must roll back a model/retrieval change without reviving stale permissions or overwriting durable operation identity.
Primary References
- PostgreSQL standby documentation: real replication/acknowledgement choices absent from the fixture.
- AWS transactional outbox: local dual-write intent and duplicate delivery boundaries.
- OpenTelemetry traces: span/context vocabulary for the proposed diagnostic path.
- Google SRE SLO guidance: objectives and error-budget reasoning, not a claim of measured attainment.
Sources checked 2 November 2026. Timings, populations and fault clocks are authored examples. Validation records actual local checks and remaining deployment gaps.
Key Takeaways
- A saved answer can be missing on a replica because its commit is not yet applied there. Use the trusted minimum version or read the primary.
- The outbox makes the result and local notification intent atomic. It does not make external delivery or model execution atomic.
- A contiguous watermark excludes missing earlier events even when later ones have arrived. A lost response is not proof that no external work happened, so unknown provider state is kept.
- Real consensus, replication, database promotion, restore and production error or latency objectives remain untested.