Chapter 14: Incident Case Study: The Silent Data Corruption

Why this matters

Chapter 13's cascading failure was loud, immediate, and impossible to miss — metrics spiked, an alert fired, users were affected visibly and immediately. This case study covers a genuinely different, arguably more dangerous shape of incident: one that produces no availability signal at all, silently serving incorrect data for an extended period before anyone notices, specifically because the observability practice (Chapter 8) that would have caught a latency or error-rate spike has no equivalent signal for data correctness.

The scenario

An online inventory management system uses a read replica (companion Database book, Chapter 10) to serve inventory-count read queries, reducing load on the primary database — a standard, reasonable Chapter 6 scaling decision. The primary handles all writes; the replica serves reads with a small, generally acceptable amount of replication lag.

What went wrong

Day 1, 09:00 — A routine database maintenance operation on the replica is performed, involving a brief pause and resync of replication. A subtle bug in the resync tooling (not caught in the maintenance runbook's checklist, companion SDLC book Chapter 8) causes the replica to silently resume replication from an incorrect position in the primary's write log — the replica is now missing a specific window of writes entirely, but reports itself as healthy and "caught up," since the tooling has no way to detect this specific class of resync error.

Day 1-4 — The replica continues serving reads normally — fast, low-latency, no errors, no timeouts. Every availability and latency metric (Chapter 8) looks completely healthy throughout this entire period, because nothing about the availability of the system has actually degraded at all — the replica is up, responsive, and answering queries correctly from its own perspective. But because of the missed write window, a specific subset of inventory count updates from Day 1 were never applied to the replica — the replica is now serving silently, systematically stale data for those specific items, indefinitely, with zero indication anything is wrong.

Day 4, 16:00 — A customer support ticket arrives: a customer was able to purchase an item that the website showed as "in stock," but the warehouse reports zero actual inventory. This is initially treated as an isolated, one-off customer service issue, not a systemic incident — there's no metric or alert that would have connected this single report to a broader, systemic problem.

Day 5, 11:00 — A second, similar report arrives, this time from a different product category entirely. An engineer investigating notices both affected items were recently updated (a stock adjustment) on Day 1, and — after specifically comparing the replica's data against the primary directly, item by item, something no automated system was doing continuously — discovers a real, systematic discrepancy: every inventory count updated during a specific window on Day 1 is now stale, silently.

Day 5, 13:00 — The replica is failed over to a fresh resync from the primary; the stale data is corrected. Total impact: an unknown, never-fully-quantified number of oversold items over roughly 4 days, only two of which were ever specifically reported and identified.

The postmortem: why this was so much harder to detect than Chapter 13's incident

Proximate cause: "A bug in replication resync tooling caused the replica to silently miss a window of writes."

Continuing to ask why (Chapter 9's discipline), specifically focused on detection, not just the trigger:

• Why did this go undetected for 4 days? → No monitoring existed for data consistency between primary and replica — only for the replica's own self-reported health and standard availability/latency metrics (Chapter 8), none of which are designed to catch this specific class of problem.
• Why did the replica report itself as healthy despite missing data? → The resync tooling's own health check only verifies that replication is running, not that the replica's actual data content matches the primary correctly.
• Why wasn't this caught by the maintenance process itself? → The maintenance runbook didn't include a post-maintenance data-consistency verification step — it only checked that replication had resumed and appeared to be "caught up" per the tooling's own (flawed) self-report.

What the postmortem changed

## Action items from this postmortem
1. Add an ONGOING, automated data-consistency check comparing a
   sample of records between primary and replica continuously --
   a NEW metric type this system never had: not availability or
   latency, but actual data-correctness verification (directly
   extending Chapter 8's observability discussion into a dimension
   it hadn't previously covered).
2. Add an explicit post-maintenance verification step to the
   maintenance runbook: after ANY replication resync, verify actual
   data consistency for a sample of recently-written records, not
   just that the tooling self-reports "caught up."
3. Treat isolated customer reports of data discrepancies as a
   potential SYSTEMIC signal worth immediate cross-checking, not
   automatically as one-off, isolated incidents -- update the
   support-escalation process to flag a SECOND similar report within
   a short window as an automatic trigger for engineering investigation.
4. Reconsider whether inventory-count reads (specifically, given their
   direct connection to overselling, a genuinely costly failure mode)
   should read from the PRIMARY rather than a replica, accepting the
   replica's scaling benefit is not worth this specific consistency
   risk for this specific, business-critical data (directly Chapter
   12's consistency-vs-availability trade-off, now revisited with the
   benefit of this incident's hard-won evidence).

The deeper lesson: observability has blind spots by default

[SENIOR INSIGHT] This incident's defining lesson, distinct from Chapter 13's: standard observability (metrics, logs, traces) is built to detect availability and performance problems by default — it has no inherent mechanism for detecting data correctness problems, since a system can be perfectly available, fast, and error-free while silently serving wrong data the entire time. This is a genuine blind spot in most systems' default observability posture, not a failure of the specific tools used, and it's precisely why data-correctness verification needs to be deliberately, separately designed in as its own observability dimension for any system where silent data corruption would be a genuinely costly failure mode — it does not come for free alongside standard availability/latency monitoring, no matter how comprehensive that monitoring otherwise is.

Common mistakes

1. Assuming comprehensive availability/latency monitoring also covers data correctness, missing that these are genuinely different observability dimensions requiring separate, deliberate instrumentation.
2. Trusting a replication tool's self-reported health status without independent, direct verification, especially after a maintenance operation that specifically touches the replication mechanism itself.
3. Treating isolated customer reports as automatically one-off rather than a potential systemic signal, missing an early opportunity to detect a silent, ongoing problem before it accumulates significant additional impact.
4. Not revisiting a consistency-vs-availability trade-off (Chapter 12) after a real incident provides new, concrete evidence about its actual risk — the original decision to read inventory counts from a replica may have been entirely reasonable when made, but this incident is exactly the kind of new information that should prompt revisiting it.

Discussion / interview questions

1. "Why might a data corruption incident go undetected for days, even with comprehensive availability and latency monitoring in place?" Strong answer: Standard observability is built to detect availability and performance problems by default — a system can be perfectly available, fast, and error-free while silently serving incorrect data, since data correctness is a genuinely separate observability dimension that doesn't come for free alongside availability/latency monitoring and needs to be deliberately, separately instrumented.

2. "How would you design monitoring to catch a silent data-consistency problem between a primary and a read replica?" Strong answer: An ongoing, automated process comparing a sample of records between primary and replica continuously, verifying actual data content matches — not just checking that the replication tooling self-reports as "running" or "caught up," which this incident shows can be true even when data has silently diverged.

3. "A customer reports a data discrepancy that seems like an isolated incident. How would you decide whether to investigate it as a potential systemic issue?" Strong answer: Treat a second, similar report (especially from an unrelated area) as a signal worth immediate cross-checking rather than assuming both are unrelated one-offs — this case study's actual detection came from noticing exactly this pattern, and building an explicit process trigger for it prevents relying on an engineer happening to notice the pattern by luck.

4. "After this kind of incident, would you reconsider using a read replica for this specific data, or just fix the specific bug that caused it?" Strong answer: Both — fix the specific triggering bug and add consistency verification, but also explicitly revisit the original consistency-vs-availability trade-off (Chapter 12) for this specific, business-critical data type with the new, concrete evidence this incident provided, rather than treating the original architectural decision as permanently settled regardless of new information.

Chapter summary / cheat sheet

• This incident produced zero availability or latency signal — the replica was fast, healthy, and error-free while silently serving stale, incorrect data, because standard observability doesn't inherently cover data correctness as a dimension.
• Detection required noticing a pattern across isolated customer reports and manually comparing primary-vs-replica data directly — no automated system was checking for this specific failure mode.
• The postmortem's key structural fix: add ongoing, automated data-consistency verification as a genuinely new observability dimension, not assume it's covered by existing availability/latency monitoring.
• This incident is a concrete instance of Chapter 12's consistency-vs-availability trade-off, and a real postmortem should explicitly revisit that original trade-off decision with the new evidence the incident provides, not just patch the specific triggering bug.

---