Skip to content
Forward Deployed

Forward Deployed
Lessons and contents
Architecture decision recordsLesson 7 of 13
13 lessons
Learn

ADR-007: Data-class consistency and evidence-based recovery

Source
From the Staff AI System Design course
By
Sourav Banerjee
Last updated
8 Nov 2026

Report an issue

Status: accepted educational architecture. Module: T18.8. Date: 2026-11-02. Supersedes: none; preserve security and execution semantics from ADR-001–006.

Situation and Alternatives

A saved answer may be absent on a lagging read copy; lost responses invite duplicate attempts, and authority outages invite unsafe stale permission. Compare primary-only reads, minimum-version eligible projections, stale-authority fallback and blind retry/failover. Choose current authoritative policy/budget/ownership decisions, eligible history projection reads with primary fallback, and no stale authorisation or uncertain-provider replay.

Decision

Keep result persistence and outbox intent atomic locally. Deliver asynchronously with duplicate-safe transactional projection updates and a contiguous applied watermark. Reuse a stable scoped operation key after a lost acknowledgement; absence of a result after an external attempt still requires uncertainty handling. Carry remaining deadlines, bounded retry/probe policies, telemetry correlation and per-boundary recovery runbooks.

Real replication/failover must specify writer fencing, acknowledgement durability, RPO/RTO, minimum-version reads and restore/promotion tests. Quorum counts alone are not a consensus implementation. Security audit/current-policy rules remain mandatory even when a history data leg can read a projection.

Evidence and Limits

Two SQLite files and explicit event delivery test rollback, committed replay, conflict/scope, lag fallback, duplicates/gaps, authority unavailability and restart persistence. A locked local breaker tests cooldown and a single probe per object. It is a sequential-result fixture, not a production breaker handling arbitrary late/in-flight result generations or distributed state. No model call, network replication, consensus, actual failover, collector/alert backend or measured objective is delivered.

Operations and Revisit

Diagnose commit versus projection state, queue/pool/dependency waits, outbox age, gap watermarks, unknown effects and audit/policy availability. Fence old writers before promotion; reconcile before deleting evidence. T18.9 may add quality/release versions and rollback while preserving operation identity and current access authority.

Key Takeaways

  • Policy, budget and ownership decisions always read current authority, while saved-answer history may read an eligible projection with primary fallback. Result persistence and outbox intent commit atomically, and projections apply duplicate-safe updates behind a contiguous watermark.
  • Stale-authority fallback and blind retry or failover are rejected because an outage would grant stale permission and a lost response would trigger duplicate attempts. A lost acknowledgement reuses the stable scoped operation key instead.
  • Real replication needs writer fencing, acknowledgement durability, RPO/RTO, minimum-version reads and tested restore and promotion, and quorum counts alone are not consensus. Old writers are fenced before promotion and evidence is reconciled before deletion.

Rate your confidence and keep notes on the module page.