Chapter 10: Replication: Leader-Follower and Multi-Leader

Why this matters

Replication — keeping copies of the same data on multiple nodes — is the foundation underneath both high availability (surviving a node failure) and read scalability (spreading read load across copies), and it's also where Chapter 9's abstract CAP/consistency discussion becomes concrete, operational reality. Understanding the specific replication topology a system uses explains exactly what failure modes and consistency trade-offs are actually possible.

Leader-Follower (Primary-Replica) replication

[BEGINNER] In leader-follower replication, one node (the leader/primary) accepts all writes; one or more follower/replica nodes receive a continuous stream of changes from the leader and apply them, staying in sync. Reads can be served from the leader (always current) or from followers (potentially slightly behind, in exchange for spreading read load across more nodes).

Clients --- WRITES ---> Leader
                            |
                 (replication stream)
                    /       |       \
               Follower  Follower  Follower
                  ^          ^          ^
                  |          |          |
             Clients ---- READS (can be spread across followers) ----

-- PostgreSQL: checking replication lag from the primary's perspective
SELECT client_addr, state, sent_lsn, replay_lsn,
       (sent_lsn - replay_lsn) AS lag_bytes
FROM pg_stat_replication;

Synchronous vs. asynchronous replication — a direct, concrete instance of Chapter 9's CP/AP trade-off:

• Synchronous: the leader waits for at least one follower to confirm receipt before acknowledging the write to the client — stronger consistency/durability (a confirmed write is guaranteed to exist on more than one node), at the cost of higher write latency and reduced availability (if the required follower is unreachable, writes block or fail).
• Asynchronous: the leader acknowledges the write immediately, without waiting for any follower — lower latency, higher availability, but a leader failure immediately after acknowledging a write (before it replicated) means that write can be lost entirely.

[SENIOR INSIGHT] This is CAP's abstract CP/AP choice, made completely concrete and configurable: synchronous replication is choosing consistency (a write is guaranteed durable across nodes before being acknowledged, at an availability/latency cost); asynchronous replication is choosing availability/performance (faster acknowledgment, at the risk of data loss on a leader failure before replication completes).

Failover: what happens when the leader dies

[BEGINNER] If the leader fails, the system needs failover — promoting a follower to become the new leader. This process has real, non-trivial complexity:

1. Detect the leader has actually failed (not just slow/temporarily unreachable --
   distinguishing a true failure from a transient network partition, Chapter 9,
   is itself a hard problem).
2. Choose which follower becomes the new leader (typically the one with the
   most up-to-date replicated data).
3. Reconfigure that follower to accept writes, and reconfigure remaining
   followers to replicate from the NEW leader instead of the old one.
4. Reconfigure clients/load balancers to send writes to the new leader's address.

[IN PRACTICE] With asynchronous replication specifically, failover carries a real risk: if the old leader had acknowledged writes that hadn't yet replicated to the promoted follower before the failure, those writes are permanently lost the moment failover completes — the new leader has no record of them, and if the old leader ever comes back online, its own extra, un-replicated writes must be discarded (or reconciled through some conflict-resolution process) to avoid the system having two divergent histories.

Multi-leader (Multi-Master) replication

[BEGINNER] Multi-leader replication allows more than one node to accept writes, with each leader replicating its changes to the others — used specifically when writes need to happen close to users in multiple, geographically distant locations (avoiding the latency of every write traveling to one single, far-away leader), or when a single leader's write throughput becomes a genuine bottleneck.

Leader (US) <----replicates----> Leader (EU) <----replicates----> Leader (Asia)
   ^                                  ^                                  ^
   |                                  |                                  |
Clients (US write locally,      Clients (EU write locally,      Clients (Asia write locally,
 low latency)                    low latency)                    low latency)

The genuinely hard problem this introduces: write conflicts. If two different leaders accept conflicting writes to the same piece of data before either replicates to the other, the system must decide how to resolve the conflict once both writes are known to both sides — there's no single obviously-correct answer, and different systems (and different applications) resolve this differently:

Leader US: UPDATE product SET price_cents = 1999 WHERE id = 1;  (at 10:00:00.100)
Leader EU: UPDATE product SET price_cents = 2499 WHERE id = 1;  (at 10:00:00.150)
-- Both writes happened before either leader knew about the other's write.
-- Once they replicate to each other, WHICH price wins? This must be decided
-- by an explicit conflict resolution strategy -- it isn't obvious or free.

Common conflict resolution strategies:

• Last write wins (LWW) — the write with the later timestamp wins, simple but can silently discard a legitimate concurrent write with no warning.
• Application-level merge logic — custom logic specific to the data (e.g., for a shopping cart, merge both sets of added items rather than picking one side's version entirely).
• Conflict-free replicated data types (CRDTs) — specially-designed data structures (certain counters, sets) that can be merged automatically and deterministically without losing information from either side, for the specific, narrower set of operations they support.

Common mistakes

1. Choosing asynchronous replication without understanding the specific data-loss window it introduces — appropriate for many use cases (where losing the last few milliseconds of writes on a rare leader failure is an acceptable risk), but a real, concrete risk that should be a deliberate decision, not an unexamined default.
2. Adopting multi-leader replication without a real, deliberate conflict resolution strategy, discovering only in production that concurrent writes to the same record silently produced an unexpected, unintended result.
3. Assuming failover is instantaneous and lossless by default — real failover takes time (detecting failure, promoting a follower, reconfiguring clients) and, with asynchronous replication, carries genuine data-loss risk that needs to be understood and accepted (or mitigated via synchronous replication) deliberately.
4. Introducing multi-leader replication's complexity without a genuine need for it (multi-region low-latency writes, or a single-leader throughput bottleneck) — single-leader replication is simpler to reason about and should remain the default absent a specific, real reason to add multi-leader complexity.

Discussion / interview questions

1. "What's the difference between synchronous and asynchronous replication, and how does this relate to the CAP theorem?" Strong answer: Synchronous replication waits for follower confirmation before acknowledging a write (stronger consistency/durability, at an availability/latency cost); asynchronous acknowledges immediately (better availability/latency, at the risk of data loss on leader failure) — this is CAP's abstract CP/AP trade-off, made into a concrete, configurable replication setting.

2. "What can go wrong during failover with asynchronous replication?" Strong answer: If the failed leader had acknowledged writes that hadn't yet replicated to the promoted follower, those writes are permanently lost — the new leader has no record of them, since acknowledgment happened before replication completed.

3. "Why would a system use multi-leader replication instead of a single leader?" Strong answer: To allow writes to happen with low latency close to users in multiple geographically distant regions, or to scale write throughput beyond what one leader can handle — at the cost of needing an explicit conflict resolution strategy for writes that conflict across leaders.

4. "What is 'last write wins' conflict resolution, and what's a risk of using it?" Strong answer: The write with the later timestamp is kept when two leaders' writes conflict; the risk is silently discarding a legitimate, concurrent write from the other leader with no warning or merge, which can be surprising or incorrect depending on the specific data and use case.

Chapter summary / cheat sheet

• Leader-follower replication: one node accepts writes, others replicate and can serve reads — synchronous (stronger consistency, lower availability/higher latency) vs. asynchronous (better availability/latency, real data-loss risk on failover) is a direct, concrete instance of Chapter 9's CP/AP trade-off.
• Failover (promoting a follower to leader) has real complexity and, under asynchronous replication, a genuine risk of losing recently-acknowledged, not-yet-replicated writes.
• Multi-leader replication: multiple nodes accept writes (for multi-region low latency or throughput scaling), introducing the genuinely hard problem of write conflict resolution (last-write-wins, application-level merging, or CRDTs).
• Default to single-leader replication unless there's a specific, real need (multi-region writes, throughput bottleneck) justifying multi-leader's added complexity.

---