Chapter 9: The CAP Theorem and Consistency Models

Why this matters

The CAP theorem is one of the most frequently cited — and most frequently oversimplified — concepts in distributed database discussions. "Pick two of three" is the version everyone remembers, and it's precise enough to be dangerous: it invites choosing an option that isn't actually available in practice. Understanding what CAP actually constrains (and doesn't) is what separates a real architectural decision from a slogan.

The theorem, precisely

[BEGINNER] The CAP theorem states that a distributed data system (one where data lives on more than one node, connected by a network) can provide at most two of the following three guarantees simultaneously, during a network partition:

• Consistency — every read receives the most recent write, or an error; all nodes see the same data at the same time.
• Availability — every request receives a (non-error) response, without guaranteeing it contains the most recent write.
• Partition tolerance — the system continues operating despite network failures preventing some nodes from communicating with others.

[SENIOR INSIGHT] The critical, commonly-missed detail: partition tolerance isn't really a choice. Network partitions will happen in any real distributed system — a network cable fails, a switch misbehaves, a datacenter loses connectivity — this is a fact about physical networks, not a design decision an architect makes. This means the real, practical choice CAP presents isn't "pick any two of three" — it's "when a partition happens (and it will), do you choose Consistency or Availability?" CP or AP is the actual decision; choosing "CA" isn't a real option for a genuinely distributed system, because it would require partitions to simply never happen, which isn't within an architect's control.

What this looks like concretely, during a partition

Normal operation: Node A <----> Node B   (both reachable, data replicated between them)

Partition occurs: Node A    X    Node B   (network link between them fails)

A client sends a write to Node A. Node A cannot confirm Node B has received it.
Now Node A must choose:

CP choice: Node A REFUSES the write (or the read, if it can't confirm it has
the latest data), returning an error, until the partition heals -- prioritizing
correctness over responding at all.

AP choice: Node A ACCEPTS the write anyway, responding successfully -- prioritizing
staying responsive, accepting that Node B (and any client reading from B) may now
see STALE or DIVERGENT data until the partition heals and the two nodes reconcile.

Mapping real systems onto CP and AP

[IN PRACTICE]

System	Typical choice	Why
A traditional single-primary relational database (PostgreSQL with synchronous replication)	CP-leaning	A write isn't acknowledged until it's confirmed durable/replicated — prioritizes correctness over availability during a replica failure
MongoDB (with default settings)	Configurable, but often tuned toward strong consistency for primary reads	Reads from the primary are consistent; reading from replicas can trade consistency for availability, configurable via read preference
DynamoDB, Cassandra (classic wide-column, AP-leaning)	AP-leaning by design/default	Prioritize staying available and accepting writes even during a partition, resolving conflicts afterward (e.g., "last write wins" or application-level conflict resolution)
DNS (companion Networking book, Chapter 5)	AP	A DNS resolver serving a cached, possibly-stale answer during a partition is considered far better than refusing to answer at all

Beyond binary CP/AP: consistency as a spectrum, not a switch

[SENIOR INSIGHT] Real systems rarely implement pure, absolute Consistency or pure, absolute Availability — most offer a tunable spectrum of consistency models between the two extremes, and understanding this spectrum matters more day-to-day than the binary CAP framing:

• Strong consistency — every read reflects the most recent write, globally, immediately (closest to CAP's "C").
• Eventual consistency — after writes stop, all nodes will eventually converge to the same value, but reads during that window may return stale data.
• Read-your-own-writes consistency — a specific, practical middle ground: a client is always guaranteed to see its own recent writes (even if it might see other clients' writes with some delay) — this specific model solves a huge share of the user-facing confusion eventual consistency can otherwise cause (e.g., a user updating their own profile and immediately seeing the old version reflected back).
• Causal consistency — writes that are causally related (one happened because of, or after seeing, another) are seen by everyone in that same causal order, even if unrelated writes might be seen in different orders by different observers.

// MongoDB: choosing a read preference is choosing a point on this consistency spectrum explicitly.
db.orders.find().readPref("primary")            // strong consistency: always the latest write
db.orders.find().readPref("secondaryPreferred")  // may read slightly stale data from a replica,
                                                   // in exchange for better read availability/scaling

Common mistakes

1. Treating CAP as "pick any two of three" as if CA were a realistic option for a genuinely distributed system — partitions are a fact of networked systems, not a choice, making CP vs. AP the only real decision.
2. Assuming a database is either "CP" or "AP" as a fixed, permanent label, without recognizing many real systems are configurable along the consistency spectrum (as the MongoDB read preference example shows) — the actual behavior depends on specific configuration choices, not just the product name.
3. Choosing strong consistency by default everywhere "to be safe," without considering whether the specific use case can tolerate eventual consistency in exchange for better availability/performance — many use cases (a social media like count, a view counter) genuinely don't need strong consistency at all.
4. Not recognizing "read-your-own-writes" as a specific, valuable, achievable middle ground — instead choosing between "fully strong" and "fully eventual" consistency as if those were the only two options, when many practical systems benefit from this specific intermediate guarantee.

Discussion / interview questions

1. "Explain the CAP theorem, and why 'CA' isn't a realistic choice for a distributed system." Strong answer: States the three properties (Consistency, Availability, Partition tolerance), then explains that partitions are a fact of real networks the architect doesn't control — making the real decision "CP or AP when a partition occurs," not "pick any two" as if all three combinations were equally available.

2. "Give an example of a system that would reasonably choose availability over consistency during a network partition, and explain why." Strong answer: DNS, or a social media "like" counter, or a shopping cart service — cases where serving a possibly-stale response is far preferable to an error, since the cost of slight staleness is low and the cost of unavailability (a broken user experience) is comparatively high.

3. "What is eventual consistency, and what's a common user-facing problem it can cause?" Strong answer: All nodes eventually converge to the same value after writes stop, but reads during the convergence window may be stale; a common user-facing problem is a user not seeing their own very-recent write reflected back immediately, which "read-your-own-writes" consistency specifically exists to solve.

4. "Is a database always either fully 'CP' or fully 'AP'? Give an example that complicates this framing." Strong answer: No — many real systems (like MongoDB with configurable read preferences) let you tune the consistency/availability trade-off per-operation rather than having one fixed system-wide behavior, meaning the label "CP" or "AP" often depends on specific configuration choices rather than being an immutable property of the product itself.

Chapter summary / cheat sheet

• CAP theorem: during a network partition (which is a fact of real networks, not a choice), a distributed system must choose between Consistency and Availability — "CA" isn't a realistic option.
• Map real systems: traditional relational databases with synchronous replication lean CP; DNS, and many wide-column stores by default, lean AP.
• Consistency is a spectrum, not a binary: strong, eventual, read-your-own-writes, and causal consistency are all distinct, useful points along it — many real systems let you choose per-operation.
• Choose strong consistency where correctness genuinely can't tolerate staleness (financial balances); choose weaker models where availability/performance matters more and staleness is tolerable (counters, feeds, caches).

---