Consensus Algorithms · Phase 3

Replication

Article 3.4 of 4 — Phase 3 complete after this

Jul 10, 2026 · devops · 19 min read · 3900 words advanced

Leases and linearizable reads.

devops distributed-systems leases series-consensus

Everything in this phase so far has focused on writes — getting a majority to agree, replicating a log, proving overlap makes it safe. But a subtle question remains, one that trips up a surprising number of otherwise-correct system designs: is reading from the leader alone actually safe? The intuitive answer ("the leader has the most up-to-date data, so just read from it") is wrong in one specific, important way — and understanding exactly why closes out Phase 3 by connecting directly back to article 1.3's precise definition of linearizability. This article covers the naive approach's flaw, the fully-safe-but-expensive fix (quorum reads), and the two major optimizations production systems actually use to get linearizable reads without paying a full consensus round-trip on every single read: leader leases and the read-index protocol.

Why reading from "the leader" alone isn't automatically safe

Here's the scenario that breaks the naive intuition. A node believes it's the leader (it was, as of the last time it successfully communicated with a majority) and serves a client's read request directly from its local state, without any further coordination. The flaw: believing you're the leader and actually still being the leader are not the same thing, and the gap between them is exactly the ambiguity article 1.1 and article 2.4 have been building vocabulary around this whole series. If this node has been partitioned from the rest of the cluster (its outgoing/incoming messages to peers are failing, but it hasn't yet detected this — or has detected it, but a client's read request arrives in the narrow window before it steps down), a new leader may already have been elected by the majority partition and may already be serving fresher writes that this stale, isolated node has no way of knowing about.

The stale-leader-read problem Node A: "I'm still leader" (partitioned, doesn't know it's been replaced) Node B: new leader elected by majority, already serving newer writes Client reads from A → gets STALE data, violates linearizability (1.3)

Fig 1 — A node can believe it's leader after actually being replaced — reading from it directly can silently violate linearizability.

This is a direct, concrete instance of article 1.3's formal Consistency definition being violated: a linearizable read must return the most recent write, and a stale isolated leader has no way to guarantee that by itself. It's also a subtler failure than split-brain (article 1.2) — no conflicting write happens (majority quorums already prevent that, per article 3.3), but a read can still return outdated information, which is its own real correctness bug for any application that needs read-your-writes or stronger guarantees.

The fully-safe fix: quorum reads

The unambiguously correct solution, directly following from article 3.3's overlap proof: treat a read like a write, and require confirmation from a majority before answering. Concretely, before serving a read, the leader (or any node) confirms with a majority of the cluster that it's still actually the current leader (or, in a leaderless system, directly applies the W+R>N generalized quorum logic from article 3.3 to the read itself). Because any majority is guaranteed to overlap with whatever majority most recently committed a write (article 3.3's core proof), this approach is provably safe — no stale read is possible.

The cost is exactly what you'd expect: a full round-trip to a majority of the cluster, for every single read, even though nothing is actually being written. For read-heavy workloads (the overwhelming majority of real production traffic in most systems), paying a full quorum round-trip cost on every read — often to a cluster spread across multiple datacenters, where that round-trip could mean tens of milliseconds — is a serious, real performance tax, exactly the kind of cost article 1.4's PACELC framework would flag as an EC (consistency-over-latency) choice worth optimizing if possible.

Optimization 1: leader leases

A lease is a time-bounded, exclusive grant: the cluster majority explicitly agrees, "node L is the leader, and this agreement is valid until time T." As long as the current node holds an unexpired lease, it can serve reads entirely locally — no per-read quorum round-trip needed — because the lease itself already encodes the majority's confirmation, valid for a bounded window.

The safety argument hinges on one critical, easy-to-get-wrong detail: the lease holder must never serve a local read after the lease has expired from its own clock's perspective, and — critically — every other node must never grant or recognize a new leader's lease until the old lease has definitely expired from a global perspective, accounting for clock skew between machines. This directly reconnects to article 2.1's entire treatment of physical clocks: if the old leader's clock runs even slightly slow (thinks less time has passed than actually has), it might believe its lease is still valid and serve a stale read after a new leader has already been safely granted a fresh lease elsewhere — precisely the stale-read scenario this whole article is trying to prevent, now reintroduced via clock skew instead of a naive missing-check.

This is exactly why lease durations in real systems are chosen conservatively, with an explicit safety margin for clock skew. A common, safe pattern: the leader treats its own lease as expiring slightly before the nominal lease duration (accounting for its own clock potentially running fast, wrongly extending its perceived validity), while other nodes wait slightly past the nominal duration before considering the old lease expired and granting a new one (accounting for the old leader's clock potentially running slow). This asymmetric safety margin is a direct, practical application of article 2.1's core lesson — measure and bound the clock uncertainty, then design correctness around the bound, rather than assuming clocks are perfectly synchronized.

Optimization 2: the read-index protocol

Raft's specific refinement (covered fully in Phase 5.6) avoids leases' clock-dependency entirely, trading it for one lightweight network round-trip instead of a full consensus write — a middle ground between "quorum read on every request" and "trust a time-based lease." The mechanism: when a read request arrives, the leader records the current commit index (its position in the log, from article 3.2) as the read index, then sends a single round of heartbeats to a majority of followers without proposing any new log entry — just confirming "are you still following me as leader?" If a majority responds affirmatively, the leader is confirmed to still genuinely be the leader (by the same quorum-overlap logic as a full quorum read), and can safely serve the read locally once its own state machine has applied everything up to that recorded read index.

The advantage over a full quorum read: this heartbeat-confirmation round-trip is much cheaper than proposing and committing an actual new log entry (no disk write, no log replication, just a lightweight liveness check), while still being fully safe — it doesn't depend on synchronized clocks or bounded clock skew the way lease-based reads do, avoiding article 2.1's entire clock-uncertainty problem. The trade-off versus leases: read-index still requires a network round-trip per read (or per batch of reads, since the confirmation can be amortized across multiple pending reads that arrive close together), whereas a valid, unexpired lease allows a read to be served with zero network communication at all.

ApproachSafety basisCost per readClock-dependent?
Naive leader readnone — unsafezero (but wrong)no
Quorum readmajority overlap (3.3), every timefull round-trip to majorityno
Leader leasemajority-granted, time-bounded exclusivityzero, while lease validyes — needs bounded clock skew
Read-index (Raft)majority overlap, confirmed via lightweight heartbeatone cheap round-trip (amortizable)no

Closing Phase 3

Phase 3 built the concrete machinery every classical consensus algorithm operates on top of. 3.1 established the abstract SMR guarantee — agree on commands, get matching state for free. 3.2 made that concrete with the actual append-only log data structure, including the real engineering (snapshotting, catch-up) it demands. 3.3 proved the single mathematical fact — majority overlap — that makes the whole thing safe against split-brain. This article closed the loop by showing that safety extends to reads too, but requires its own explicit mechanism (quorum reads, leases, or read-index) rather than following automatically from the write-side guarantees alone.

Phase 4 (Consensus Basics) now introduces the first real consensus-specific vocabulary — safety vs. liveness as formal properties, leader election as its own studied problem, atomic broadcast, and the two commit protocols (2PC, 3PC) that predate and directly motivate real consensus algorithms, closing out the "basics" before Phase 5 introduces Paxos and Raft themselves with the complete foundation (Phases 1-4) already in place.

FAQ

Do all consensus-backed systems need to worry about this stale-read problem, or only some?

Any system claiming linearizable (or even just read-your-writes) consistency needs one of these mechanisms — a system that's explicitly designed for weaker guarantees (eventual consistency, as discussed in article 1.2's leaderless row) has already accepted some staleness by design and doesn't need to solve this problem the same way, since it never promised the stronger guarantee in the first place.

Can leases and read-index be combined in the same system?

Yes, and some production systems do exactly this — using lease-based fast-path reads when clock synchronization is trusted to be tight (e.g., within a single datacenter with good NTP), falling back to read-index or full quorum confirmation for cross-region reads where clock skew assumptions are riskier. This mirrors the tunable, per-situation trade-offs seen throughout this series (CAP's per-data-type choice in 1.3, PACELC's per-operation choice in 1.4).

Is TrueTime (article 2.1) essentially a very sophisticated version of a leader lease?

They share the same underlying idea — bound the clock uncertainty and design correctness around the bound — but TrueTime is used for a different, broader purpose (globally ordering transactions across Spanner, not specifically leader-lease-based reads). It's fair to say they're philosophical cousins: both convert "we can't have a perfectly synchronized clock" into "we can have a precisely bounded and honestly-accounted-for uncertainty," then build correctness on top of the bound rather than the impossible ideal.

What happens to in-flight reads if a lease expires mid-request?

A correctly implemented system checks lease validity at the moment of serving the read (or re-validates if enough time has passed during processing), not just at the moment the request arrived — a request that started under a valid lease but would be served after expiration must either wait for lease renewal or fall back to a safer mechanism (quorum confirmation) rather than serving potentially-stale data on a technicality of when the check happened.

Takeaways

  • "Read from the leader" is not automatically safe — a node can believe it's still leader after actually being replaced by a majority elsewhere, and serve a stale read that violates linearizability.
  • Quorum reads (confirm current leadership, or apply W+R>N directly to the read) are fully safe, by the same majority-overlap proof as article 3.3, at the cost of a full round-trip per read.
  • Leader leases let a confirmed leader serve reads with zero network cost while the lease is valid — but require careful, conservative handling of clock skew (article 2.1) to remain safe, with asymmetric safety margins on both the leader and the followers' sides.
  • The read-index protocol (Raft) avoids clock dependency entirely — a lightweight heartbeat-confirmation round-trip, cheaper than a full quorum write but not free like a valid lease.
  • Phase 3 is now complete: SMR's abstract guarantee (3.1), the concrete log (3.2), the majority-overlap proof that makes it all safe (3.3), and the read-side extension of that safety (3.4) — the full replication machinery Phase 5's classical algorithms build directly on top of.

References & further reading

← 3.3 Quorums and Majority Voting Phase 3 complete — back to series hub →
© cvam — written in plaintext, served warm