Why distributed systems are hard.

A single computer either does a thing or it doesn't, in an order you can reason about, with a clock you can (mostly) trust. A distributed system throws all three of those guarantees away: messages arrive late, out of order, or not at all; every machine's clock disagrees with every other machine's clock; and any node can die at the single worst possible moment — mid-write, mid-vote, mid-anything. Consensus algorithms exist because engineers got tired of pretending those three problems don't happen. This article is the "why" that every algorithm in this series is a specific, principled answer to. Skip it and you'll memorize Raft; read it and you'll understand why Raft looks the way it does.

Start with what a single machine gives you for free

Before appreciating what's hard about many machines, name what's easy about one. A single process running on a single computer gives you, essentially without asking:

• A total order of operations. If your code does x = 1 then y = 2, every observer of that machine's state sees x set before y. There is one timeline, and everything sits on it.
• Atomic failure. When the process crashes, it crashes entirely. It doesn't execute half of a function and leave the other half hanging in some purgatory state visible to callers. (Yes, torn writes to disk exist — but the CPU/memory model itself gives you clean all-or-nothing execution up to that boundary.)
• A shared, trustworthy clock. Every part of your program reads the same system clock. "Five seconds ago" means the same five seconds to every function in the process.
• Synchronous communication. A function call returns, or it doesn't (crash), and you know which happened by whether you're still executing.

Every one of those four guarantees quietly evaporates the instant you have two machines talking over a network instead of one machine talking to itself. That evaporation is the entire subject of this series. Consensus algorithms are not an optimization or a nice-to-have abstraction — they are the machinery engineers built specifically to manufacture a working substitute for those four guarantees, out of a network that provides none of them naturally.

The three lies a distributed system tells you

There's an old, still-accurate list — sometimes called the "fallacies of distributed computing" — of assumptions programmers make that are all false. Three of them matter enough to build this whole series around:

Lie #1 — "The network is reliable"

It isn't, and not in a rare-edge-case way — in a constant, expected, budget-for-it way. Packets get dropped by an overloaded switch. A fiber gets backhoed. A load balancer's connection table fills up and starts silently resetting connections. A misconfigured BGP route sends half your traffic through Singapore for four minutes. None of these are exotic; large-scale production systems see network blips measured in the thousands per day, and it is a documented operating assumption at every major cloud provider that "the network" is a probabilistic service, not a guaranteed one.

The consequence that actually matters for consensus: when a message from node A to node B doesn't arrive, node A cannot tell whether B never received it, B received it and crashed before replying, or the reply itself got lost on the way back. Those three situations demand completely different responses (retry safely / don't retry, you might duplicate an effect / the recipient already knows) and from A's point of view they are indistinguishable. This single fact — that failure and slowness look identical from the outside — is the seed that grows into the FLP impossibility theorem two articles from now.

Lie #2 — "Clocks are synchronized"

Every machine has a local clock, and every local clock drifts — typically tens of milliseconds per day for consumer-grade crystal oscillators, and non-negligible drift even on the better hardware datacenters use, which is precisely why Google built an entire custom hardware+software stack (TrueTime, covered in Phase 9) just to bound clock uncertainty instead of eliminating it. NTP helps, but NTP synchronization can itself jump backward or forward when it corrects drift, and NTP daemons can silently fail while looking fine.

The practical result: if machine A's clock says "14:00:03.100" and machine B's clock says "14:00:03.050" for events that were sent and received in that order, you cannot use wall-clock timestamps to determine which event actually happened first across machines. This isn't a precision problem you fix by buying better clocks — it's a fundamental limit, because information about "now" itself takes time to propagate across a network, so there is no single global "now" two machines can both observe simultaneously. Special relativity has the same flavor of problem, for the same underlying reason: information has a finite propagation speed. Article 2.1 builds the physical-clock picture in detail, and 2.2–2.3 show how logical clocks (Lamport, vector) sidestep the whole mess by not trying to measure time at all — only order.

Lie #3 — "A node either works or it's down"

The comforting mental model is binary: a server is either serving requests correctly, or it's off. Reality offers a much uglier menu of partial-failure modes:

• Crash failure — the node stops entirely and stays stopped. The "nice" failure, and the only one many textbook algorithms assume.
• Crash-recovery — the node stops, then comes back later, possibly having lost in-memory state but kept disk state (or vice versa, if you're unlucky and it's the other way around).
• Omission failure — the node is alive and correct internally but selectively fails to send or receive some messages (a flaky NIC, a congested queue silently dropping under backpressure).
• Timing failure — the node is correct but slow: a GC pause, a noisy neighbor stealing CPU on a shared host, swap thrashing. From the outside, indistinguishable from a crash until the pause ends.
• Byzantine failure — the node sends actively wrong or contradictory information: different answers to different peers, corrupted data that still passes a basic sanity check, or (the adversarial case) a genuinely malicious or compromised participant. Phase 6 is entirely about this failure class, because it needs fundamentally stronger algorithms to survive.

Split-brain, visualized

Put lies #1 and #3 together and you get the scenario that makes "just replicate the data" so much harder than it sounds: split-brain. Two replicas of the same service, connected by a network link that fails — not either replica crashing, just the link between them dying. Each side is alive, healthy, and can no longer see the other.

Notice what's not the problem here: neither machine misbehaved. Neither had a bug. Each one, in complete isolation, made a locally reasonable decision — "my peer seems gone, someone has to keep serving traffic." The bug, if you can call it that, is architectural: the system had no mechanism that could tell either replica, correctly, whether it was safe to keep accepting writes alone. That mechanism — a way for a group of unreliable, un-synchronized, partially-failing machines to agree on one fact even when they can't all talk to each other — is precisely what a consensus algorithm provides. Not "prevent partitions" (impossible, you can't out-engineer physics) but "guarantee that during a partition, at most one side keeps accepting writes, and everyone agrees afterward on which writes actually count."

Why not something simpler than consensus?

A reasonable question at this point: split-brain sounds like it has an easy fix. Why not just: "whichever replica has more recent data wins" or "the one that was primary longest wins" or "just don't let it happen, use a really reliable network"? Every simple-sounding fix here has a well-known, well-documented failure mode:

• "Most recent write wins" (last-writer-wins) — requires trustworthy synchronized timestamps to compare "most recent" across machines, which lie #2 already ruled out. And even with perfect clocks, silently dropping a write because it lost a timestamp race is a correctness decision users didn't sign up for.
• "Whoever was primary longest wins" — both sides can make this exact same claim about themselves after a long enough partition, and there's no way for either to independently verify the other's claim without... communicating, which is the thing that's broken.
• "Use a more reliable network" — reduces the frequency of partitions, never their possibility. At sufficient scale (thousands of machines, multiple datacenters, the public internet), partition events aren't a tail risk you can insure away; they are a statistical certainty over any sufficiently long time window. This is the empirical grounding behind the "P" in CAP, which article 1.3 formalizes properly.
• "A human resolves conflicts after the fact" — works for low-volume systems, catastrophic for anything processing thousands of writes per second. You cannot page a human for every ambiguous write.

What all the failed fixes have in common: they try to resolve the ambiguity locally, using information available to one node in isolation. Consensus algorithms take a structurally different approach — they require a majority of nodes to agree before any write is considered "durable" or "committed," which mathematically guarantees that two disjoint groups can never both believe they have a majority at the same time (any two majorities of the same set must overlap by at least one node). That single mathematical fact — majority overlap — is the load-bearing idea underneath Paxos, Raft, and nearly everything else in this series. Article 3.3 (Quorums and Majority Voting) proves it properly; for now, just notice that it's a genuinely different category of solution from "guess based on local information," which is why it needed a body of research to discover rather than being obvious from day one.

Real production failures, briefly

These aren't hypotheticals — they're the reason the field takes this seriously:

What consensus is not (yet) — clearing space for the rest of the series

Two adjacent ideas people conflate with consensus, worth separating early so later articles land cleanly:

• Consensus is not the same as "eventual consistency." Systems like DNS, or Amazon's original Dynamo, deliberately accept that replicas can disagree temporarily and provide a mechanism (vector clocks, CRDTs, last-writer-wins with defined tie-breaks) to reconcile disagreement later, in exchange for never blocking a write while replicas are unreachable. That's a legitimate, different design point — optimizing for availability over immediate agreement — not a lesser version of consensus. Consensus algorithms instead guarantee agreement before a write is acknowledged as durable, at the cost of sometimes being unable to accept writes at all during a bad enough partition. Article 1.3 (CAP) formalizes this exact trade-off.
• Consensus is not the same as a distributed transaction (2PC/3PC). Two-Phase Commit (article 4.4) coordinates whether a set of different operations across different systems all happen or none happen — think "debit account A and credit account B, atomically." Consensus coordinates whether a set of replicas holding the same data agree on a single sequence of operations. They're related, historically 2PC came first and directly informed early consensus thinking, and 2PC's specific failure mode (a blocked coordinator) is exactly what article 4.4 uses to motivate why real consensus needed something structurally different.

Where this series goes from here

This article deliberately stayed at the "why" level — no algorithms yet, on purpose. The next three articles in Phase 1 build the remaining foundational vocabulary before any algorithm appears: 1.2 walks through the replication problem and split-brain in more mechanical detail (what exactly a "replica" needs to promise), 1.3 gives CAP its full, careful formal treatment (most explanations you've read are subtly wrong — this one earns the "properly" in its title), and 1.4 extends past CAP into PACELC, which fixes CAP's biggest blind spot: it says nothing about what happens when the network is fine but merely slow, which is the far more common case in practice.

From there, Phase 2 (Time and Failure) gives you the precise vocabulary — logical clocks, failure detectors, the FLP impossibility theorem — that every algorithm's design decisions are justified in terms of. Only after that does Phase 4 introduce the first real consensus primitives, and Phase 5 the classical algorithms (Paxos, Raft) themselves. By the time you reach an algorithm, you'll already know exactly which problem it's solving and why the obvious simpler solutions don't work — which is the difference between memorizing Raft's rules and actually understanding why they're the rules.

FAQ

Takeaways

• A single machine gives you, essentially for free: a total order of operations, atomic failure, a trustworthy clock, and synchronous communication. Distance and independence destroy all four.
• Three specific lies underlie every distributed systems problem: the network is not reliable, clocks are not synchronized, and failure is not binary.
• The failure spectrum runs from crash through crash-recovery, omission, timing to Byzantine — and every algorithm in this series makes an explicit choice about how far along that spectrum it defends against.
• Split-brain happens when a partition (not a crash!) leaves two replicas each independently, reasonably believing they should keep serving writes — with no local information either could use to know it's the wrong one.
• Simple fixes (last-writer-wins, "longest primary wins," "just use a better network") all fail for structural reasons, not implementation sloppiness — which is why a dedicated field of algorithms exists.
• Real incidents (GitHub 2018, AWS DynamoDB 2015) show the blast radius of these failures is wildly disproportionate to the triggering event — seconds of network trouble, hours or days of consequence.
• Consensus is a distinct concept from eventual consistency (which accepts temporary disagreement) and from distributed transactions / 2PC (which coordinate different operations across systems, not the same data across replicas).

References & further reading

• The Fallacies of Distributed Computing — the original list (Deutsch/Gosling, Sun Microsystems), still the best short framing of what programmers wrongly assume.
• GitHub — October 21 Incident Post-Mortem — the 43-second-partition, 24-hour-recovery incident referenced above, in the company's own words.
• AWS — Summary of the Amazon DynamoDB Service Disruption (Sep 2015) — the membership-storm cascading failure.
• Kleppmann — A Critique of the CAP Theorem — required reading before article 1.3; sets up why "properly" is earned.
• Chandra & Toueg — Unreliable Failure Detectors for Reliable Distributed Systems — the formal paper behind article 2.4's failure-detector treatment.
• cvam.sight — PagedAttention — a different kind of systems problem (GPU memory), same discipline of naming the exact failure mode before fixing it.