The replication problem.

"Just replicate the data" sounds like a solved problem — copy it to a second machine, done. It isn't, because the moment you have two copies, you've created a new question that didn't exist before: what happens when they disagree? Every replication strategy is really an answer to that one question, dressed up in different terminology. This article builds the full picture: why a single machine is unacceptable at any real scale, the replication spectrum from single-leader to leaderless, the precise mechanics of how split-brain actually corrupts data (not just the visual intuition from 1.1), and why "replicas disagreeing" is not an edge case to handle — it's the central design problem the rest of this series exists to solve.

Why a single machine is not an option

Start with the boring case: one database, one server, no replicas. It's the simplest system to reason about, and for genuinely small, low-stakes workloads it's a legitimate choice — fewer moving parts, no consensus protocol to misconfigure, no split-brain to worry about because there's only one brain. But it has a hard ceiling, and the ceiling isn't subtle:

• Availability is capped at that one machine's uptime. If it's down — hardware failure, OS crash, a bad deploy, the datacenter losing power, someone tripping over a cable — your service is down, in its entirety, until that specific machine (or a restore from backup) comes back. There is no partial degradation; it's binary.
• Durability rests on that one machine's disk. A backup taken every few hours means every write since the last backup is unrecoverable if the disk dies between backups. (This is exactly the gap continuous WAL archiving closes for PostgreSQL — see the pgBackRest guide — but that only protects against data loss, not availability during the outage.)
• Read capacity is capped at one machine's throughput. You cannot add read capacity by adding servers if there's only ever one server that has the data.
• Geography is fixed. A user in Singapore querying a database in Virginia pays the full round-trip latency of that distance, every single request, with no way to serve them from somewhere closer.

Replication is the answer to all four — copy the data onto multiple machines, and you get survivability past a single failure, more read throughput, and the option to place copies close to users. But the fix creates the problem this whole series is about: now that there are multiple copies, what guarantees do you make about how closely they agree with each other, and who's allowed to accept a write when they can't all talk to each other? Every replication architecture is a specific, named answer to that question.

The replication spectrum

Replication strategies aren't a binary choice — they sit on a spectrum defined by two independent questions: who is allowed to accept writes, and how is agreement enforced before those writes are considered safe.

Single-leader replication

One node (the leader/primary) accepts all writes; the rest (followers/replicas) apply the same writes in the same order and serve reads. This is by far the most common pattern in production systems — PostgreSQL streaming replication, MySQL replication, MongoDB replica sets in their default configuration. It's conceptually simple: there's never a question of two nodes disagreeing about write order, because only one node ever decides that order.

The catch is entirely about how writes propagate to followers, and this single design decision is one of the most consequential in all of distributed systems:

• Synchronous replication — the leader waits for at least one follower to confirm the write before acknowledging it to the client. Safer (a leader crash right after acknowledging doesn't lose the write — a follower has it), but the leader's write latency now includes a network round-trip to a follower, and if that follower is unreachable, the leader can't accept writes at all (or must fall back to a degraded mode).
• Asynchronous replication — the leader acknowledges the write immediately and propagates to followers in the background. Fast, and doesn't block on follower availability, but if the leader crashes before a write reaches any follower, that write is gone — acknowledged to the client, then silently lost. This exact failure mode is why "my database has replicas" is not automatically the same claim as "my database doesn't lose acknowledged writes."
• Semi-synchronous — a middle ground (used by MySQL, among others): wait for confirmation from at least one follower, but not all of them, trading some safety for latency better than full-synchronous.

Multi-leader replication

More than one node accepts writes, typically one leader per datacenter/region, with each leader replicating to the others. This buys write availability during a full datacenter partition (each side keeps accepting writes locally) at a steep cost: the same piece of data can now be written differently and simultaneously by two leaders that haven't yet heard from each other — a write conflict. Resolving these conflicts (last-writer-wins, custom merge logic, CRDTs) is a real, ongoing engineering burden, not a one-time setup cost, and it's precisely the trade-off eventual consistency systems (mentioned in 1.1) accept deliberately.

Leaderless replication

No fixed leader at all — a client writes to (and reads from) several replicas directly, using quorum math (write to W replicas, read from R replicas, where W+R > total replicas guarantees at least one overlap) to keep reads and writes consistent enough. Amazon's original Dynamo popularized this; Cassandra and Riak are direct descendants. It trades the simplicity of "one node decides order" for very high write availability — there's no leader to be a bottleneck or single point of failure for writes at all.

Consensus-backed single-leader: the pattern this series builds toward

There's a fourth row worth calling out separately, because it's where nearly every algorithm in Phases 4–7 lands: single-leader replication where the leader itself is elected and writes are only acknowledged once a majority durably has them — not "one arbitrarily-designated leader whose crash can silently lose data," but "whichever node currently holds a majority's trust, provably." This gets you single-leader's simplicity (one order, no conflict resolution needed) while structurally closing the async-replication data-loss hole, because promotion after a crash can only ever happen from a node that's part of a majority that already has the data. Article 3.3 (Quorums and Majority Voting) proves exactly why majority overlap makes this safe; for now, hold onto the shape of the idea — it's the single mechanism nearly every algorithm from here forward is a variation on.

Split-brain, mechanically this time

Article 1.1 showed split-brain as a scenario. Here's exactly how it corrupts data, step by step, using a simple key-value store as the running example:

1. t=0. Replica A is leader, Replica B is follower. Both agree: x = 1.
2. t=1. The network link between A and B fails. Neither crashes — each is fully healthy and reachable by clients on its own side of the partition.
3. t=2. A client on A's side writes x = 5. A accepts it (it has no way to know B is unreachable due to a partition rather than a crash — the ambiguity from article 1.1's "Lie #1" is exactly this moment). A now has x = 5.
4. t=3. B, having not heard a heartbeat from A in some timeout window, decides A must be down and promotes itself to leader. This is a correct, reasonable local decision given what B can observe — it has no way to distinguish "A crashed" from "the network between us broke" (again, article 1.1's central ambiguity).
5. t=4. A client on B's side writes x = 9. B, now believing itself leader, accepts it. B now has x = 9.
6. t=5. The network heals. A and B can talk again. Both believe they are the current leader. Both have accepted client-facing writes since the partition began. There is no shared history to consult that says which write should win — from either replica's local perspective, its own write sequence is completely valid.

This is the mechanical version of Fig 2 from article 1.1, and it's worth sitting with the specific failure: it is not that A or B did anything wrong. Each followed a sensible local rule. The corruption is a systemic property — the system as a whole had no rule that could have prevented two nodes from simultaneously believing they were sole leader, because no rule based on local information alone can distinguish a partition from a crash.

A first look at why majorities specifically

It's worth previewing the actual arithmetic here, informally, because it explains a design choice you'll see repeated throughout this series: consensus systems are almost always deployed with an odd number of nodes (3, 5, 7), and tolerate (N-1)/2 failures.

With 5 nodes, a majority is any 3. If a partition splits the cluster into a group of 3 and a group of 2, the group of 3 has a majority and can keep operating; the group of 2 does not, and correctly refuses to accept writes. There is no possible partition of 5 nodes into two non-overlapping groups where both groups have 3 or more members — 3+3=6 > 5, so it's arithmetically impossible. That's the whole mechanism. With an even number, say 4 nodes, a 2-2 split is possible, and neither side has a majority — which is actually the safer failure mode (nobody accepts writes, rather than the risk of miscounting) but it does mean you paid for a 4th node that bought you zero additional fault tolerance over 3 (both tolerate exactly 1 failure). This is why production consensus clusters are conventionally sized 3 or 5, not 4 or 6 — the even number is pure waste. Article 3.3 makes this fully rigorous; this is the intuition to carry forward.

What replication (even done right) doesn't solve

Worth being honest about the limits before moving on, so the rest of this series doesn't oversell itself:

• Correlated failures. If all your replicas are in the same datacenter and that datacenter loses power, having 5 replicas didn't help — they all failed together. Real fault tolerance requires replicas to fail independently, which is why production consensus clusters are spread across availability zones or regions (Phase 9 covers the added latency cost of doing this across real geographic distance).
• Bad data replicated correctly is still bad data. Consensus guarantees replicas agree on what happened, not that what happened was correct. If your application logic issues a wrong write (the DELETE FROM orders without a WHERE clause example from the pgBackRest guide), a perfectly functioning consensus system will faithfully replicate that mistake to every single replica, instantly and durably. This is exactly why point-in-time recovery from backups remains necessary even in a fully consensus-backed system — consensus and backup solve different problems.
• Client-perceived consistency still needs care. Even with a correctly majority-committed write, a client reading from a follower that hasn't yet applied that write can see stale data — unless the system specifically provides read guarantees (linearizable reads, leases — article 3.4) on top of the underlying replication.

FAQ

Takeaways

• A single machine has a hard ceiling on availability, durability, read throughput, and geography — replication exists to lift all four, but creates the "what if replicas disagree" problem in exchange.
• The replication spectrum runs single-leader → multi-leader → leaderless, trading write-availability-during-partition against conflict-resolution complexity.
• Asynchronous single-leader replication can silently lose acknowledged writes if the leader crashes before propagating them — a real, common production risk, not a theoretical edge case.
• Consensus-backed single-leader replication (where the leader is elected by majority and writes need majority durability before acknowledgment) is the pattern nearly this entire series builds toward — single-leader's simplicity without async replication's data-loss hole.
• Split-brain, mechanically, is two replicas each making a locally reasonable decision that turns out to be globally wrong — because neither can distinguish "peer crashed" from "network partitioned" using only local information.
• Majorities specifically work because any two majorities of the same set must overlap — which is why clusters are conventionally sized 3 or 5 (odd), not 4 or 6.
• Consensus is not a cure-all: correlated failures, application-level bad writes, and read-side staleness are all separate problems it doesn't automatically solve.

References & further reading

• Kleppmann — Please Stop Calling Databases CP or AP — directly relevant here: why coarse spectrum labels undersell the real design space, previewing article 1.3.
• DeCandia et al. — Dynamo: Amazon's Highly Available Key-value Store (SOSP 2007) — the canonical leaderless replication paper; W/R quorum reads and writes, vector clocks, hinted handoff.
• PostgreSQL docs — Log-Shipping Standby Servers — sync vs. async streaming replication in a real, widely-deployed system.
• cvam.sight — pgBackRest with S3 — replication and backup solve different problems; this guide covers the durability half.
• cvam.sight — Consensus 1.1: Why Distributed Systems Are Hard — the failure vocabulary this article builds directly on.