Skip to content
Forward Deployed

Forward Deployed
Lessons and contents
Reliability and consistencyLesson 5 of 5
5 lessons
Learn

Reliability and consistency: validate and carry forward

Source
From the Staff AI System Design course
By
Sourav Banerjee
Last updated
8 Nov 2026

Report an issue

Recall and Exit Gates

  1. Why can a saved answer be absent on a replica? Its commit may not yet be applied there; use the trusted minimum or primary.
  2. What is atomic in the outbox fixture? Result and local notification intent, not external delivery or model execution.
  3. What does a contiguous watermark exclude? Missing earlier events despite later observations.
  4. Why retain unknown provider state? A lost response is not proof of no external work.
  5. Which recovery claims are untested? Real consensus, replication, database promotion, restore and production error/latency objectives.

The 60-second card: consistency per class; one authoritative writer; minimum-version reads; transactional intent; duplicate-safe contiguous projection; bounded attempts/probes; critical-path diagnosis; recover with evidence. Review at about one, seven and 21 days.

  • Explain lag, local atomicity and external uncertainty without notes in two minutes.
  • Draw primary, outbox, projection and protected-authority read paths.
  • Defend current authority versus eligible replica reads and a recovery alternative.
  • Diagnose rollback, lost response, lag, outage and latency with concrete state/signals.
  • Run/modify the fault and probe tests and write a deployment recovery drill; learner completion remains unassessed.

Decisions and Next Dependency

ADR-007 and the stage snapshot preserve the evidence.

A future T18.9 adds versioned quality evaluation and release controls. It inherits these telemetry, transaction and recovery boundaries, and must roll back a model/retrieval change without reviving stale permissions or overwriting durable operation identity.

Primary References

Sources checked 2 November 2026. Timings, populations and fault clocks are authored examples. Validation records actual local checks and remaining deployment gaps.

Key Takeaways

  • A saved answer can be missing on a replica because its commit is not yet applied there. Use the trusted minimum version or read the primary.
  • The outbox makes the result and local notification intent atomic. It does not make external delivery or model execution atomic.
  • A contiguous watermark excludes missing earlier events even when later ones have arrived. A lost response is not proof that no external work happened, so unknown provider state is kept.
  • Real consensus, replication, database promotion, restore and production error or latency objectives remain untested.

Rate your confidence and keep notes on the module page.