Chapter 10: The CAP Theorem and Consistency Models in Practical Terms

Why this matters

The companion Database book's Chapter 9 covered CAP and consistency models in depth at the database level. This chapter's job is applying that same theorem at the whole-system level — because CAP's real, practical decision (CP vs. AP during a partition) doesn't just apply to a single database, it applies to every distributed component in a system design, and a system design conversation needs to make this trade-off explicit for the specific system being designed, not just recite the theorem's definition.

Restating CAP at the systems-design level

[BEGINNER] Recall (Database book, Chapter 9): during a network partition (an inevitable fact of distributed systems, not a choice), a distributed system must choose between Consistency (every read reflects the latest write, or an error) and Availability (every request gets a non-error response, possibly with stale data). The systems-design-level question this chapter adds: for THIS specific system, given its specific requirements, which side of this trade-off actually matters more, and does the answer differ for different parts of the same system?

The answer often differs by component within one system

[SENIOR INSIGHT] A single system frequently makes genuinely different CP/AP choices for different pieces of its own data, and a strong system design answer names this explicitly rather than treating "CP or AP" as one binary choice for the entire system:

Example: an e-commerce system

Inventory count during checkout: leans CP -- briefly refusing a checkout
   rather than risking an oversold item (companion Database book, Chapter
   3's oversell scenario) is the right trade-off here; a wrong "yes you
   can buy this" answer has a real, costly consequence.

Product recommendation feed: leans AP -- serving a slightly stale
   recommendation list during a partition is far preferable to showing
   an error page; the cost of staleness here is negligible compared to
   the cost of unavailability.

Both trade-offs exist within the SAME overall system -- the "CP or AP"
question is answered per-component, based on each component's specific
cost of staleness versus cost of unavailability, not once for the whole system.

Consistency models as a systems-level design lever

[BEGINNER] Beyond the binary CP/AP framing, the companion Database book's Chapter 9 consistency spectrum (strong, eventual, read-your-own-writes, causal) is directly usable as a systems design tool: a strong system design answer picks a specific consistency model per component, justified by that component's actual requirement, rather than defaulting to either extreme everywhere.

User's own profile updates: read-your-own-writes consistency is often
   the right choice -- a user should always see their OWN recent change
   reflected back, even if other users see it with some propagation delay.

Global "trending posts" ranking: eventual consistency is fine -- no
   individual user has a strong expectation about the EXACT real-time
   state of a global ranking that's inherently aggregate and approximate anyway.

Applying this during a system design conversation

[IN PRACTICE] A strong system design answer, when consistency comes up, explicitly walks through: "this specific piece of data has [X] cost if stale, and [Y] cost if unavailable — given that, I'd choose [specific consistency model] for it," repeated per component as needed, rather than a single blanket statement like "we'll use eventual consistency for scalability" applied uniformly without regard to which specific data actually tolerates it.

Weak: "This system will be eventually consistent for better scalability."
      (No connection to which SPECIFIC data this applies to, or why
      that specific data can tolerate staleness.)

Strong: "For the inventory count specifically, I'd want strong
        consistency, since overselling has a real cost -- I'd use
        [Chapter 7's discussion of] a transactional database with row
        locking for that specific operation. For the product view
        counter, eventual consistency is fine -- a slightly stale
        count has essentially zero cost, and eventual consistency lets
        us scale that specific counter far more cheaply."

Common mistakes

1. Treating CAP as a single, whole-system decision rather than recognizing that different components within the same system often warrant genuinely different CP/AP choices based on each one's specific cost of staleness versus unavailability.
2. Defaulting to "eventual consistency for scalability" as a blanket, unjustified statement, without naming which specific data this applies to and why that specific data can actually tolerate the staleness.
3. Choosing strong consistency everywhere "to be safe," without recognizing the real availability/performance cost this imposes on components that didn't actually need such a strong guarantee.
4. Not connecting the consistency-model choice back to a concrete, stated cost (what happens, specifically, if this data is stale? what happens if it's unavailable?) — a consistency choice without this justification is asserted, not reasoned.

Discussion / interview questions

1. "Does every component of a distributed system need to make the same CP-vs-AP choice?" Strong answer: No — different components within the same system often warrant genuinely different choices based on each one's specific cost of staleness versus cost of unavailability (e.g., inventory counts leaning CP, a recommendation feed leaning AP within the same e-commerce system).

2. "How would you decide what consistency model to use for a specific piece of data in a system you're designing?" Strong answer: Name the specific cost of that data being stale, and the specific cost of it being unavailable, then choose a consistency model (strong, eventual, read-your-own-writes, causal) that fits — not a blanket, system-wide default applied without regard to the specific data's actual requirements.

3. "A candidate says 'this system will use eventual consistency for scalability.' What follow-up question would you ask?" Strong answer: "Eventual consistency for which specific data, and why can that specific data tolerate staleness?" — a blanket, unjustified consistency statement doesn't demonstrate genuine trade-off reasoning; a strong answer connects the choice to a specific component's specific tolerance.

4. "Why might a user's own profile updates need read-your-own-writes consistency, even in a system that's otherwise eventually consistent?" Strong answer: A user expects to see their own recent change reflected immediately, even if other users' visibility of that same change can lag — serving the user their own stale data (before their own write) would be confusing and feel broken, even though the same staleness might be entirely acceptable for other users viewing the same data.

Chapter summary / cheat sheet

• CAP's CP-vs-AP choice, and the broader consistency-model spectrum, should be applied per-component within a system, not as one blanket, whole-system decision.
• A strong system design answer names each component's specific cost of staleness versus cost of unavailability, and chooses a consistency model justified by that specific trade-off.
• "Eventual consistency for scalability" as an unqualified, blanket statement is a weak answer — always connect the choice to which specific data it applies to and why that data tolerates it.
• Read-your-own-writes and causal consistency (companion Database book, Chapter 9) are genuinely useful middle-ground tools, not just "strong" or "eventual" as the only two options.

---