Articles 9.1 through 9.4 each showed a specific system's answer to geo-distribution, but this article steps back and names the underlying constraint all four are actually fighting: the speed of light. A round trip between two points roughly the width of a continent apart has a hard physical floor — tens of milliseconds — that no amount of engineering cleverness removes, only works around. This article makes that floor explicit and quantitative, then covers the specific techniques real systems use to keep reads fast despite it: quorum leases and follower reads (both direct extensions of article 3.4's leases-and-linearizable-reads material), witness and non-voting replicas as a quorum-shaping tool distinct from article 7.2's Flexible Paxos but philosophically related, and the geo-partitioning strategies that let a write for a specific region avoid crossing an ocean at all.
The physical floor: what geography actually costs
Light in vacuum travels roughly 300,000 kilometers per second; in the fiber-optic cable that actually carries internet traffic, the effective speed is closer to two-thirds of that, because of the refractive index of glass. A round trip between, say, a datacenter on the US East Coast and one in Western Europe — a great-circle distance of roughly 6,000 kilometers each way — has a physical lower bound of somewhere around 60 milliseconds for the round trip, before accounting for any of the real-world detours, switching delays, and congestion that push actual measured latency higher still. This is not a limitation of current technology that a faster fiber or a cleverer protocol will fix; it is dictated by the physical constant this series' entire Phase 9 has been implicitly working around since article 9.1 first introduced TrueTime.
Recall precisely why this matters for consensus specifically, connecting back to article 3.3's majority-overlap mechanism: every write that needs to reach a majority quorum has to wait for messages to travel to and from at least a majority of replicas. If those replicas are spread across continents, the write's latency is bounded below by the round-trip time to the farthest replica needed to form that majority — a single-digit-millisecond floor within one datacenter becomes a tens-of-milliseconds floor the moment the quorum spans oceans. This is the concrete, physical version of the abstract "latency" cost article 1.4's PACELC framing named back in Phase 1, now fully cashed out in real numbers.
Fig 1 — No Phase 5, 6, or 7 algorithm changes this floor — it's set entirely by which replicas a majority must include.
Quorum leases: extending article 3.4's leases across a whole quorum
Article 3.4 covered leader leases: a single elected leader, holding a time-bounded promise from a majority that it remains leader, can serve reads locally without a network round-trip, as long as its lease hasn't expired. Quorum leases generalize this idea a step further, specifically for the geo-distributed setting: rather than only the leader being allowed to serve fast local reads, a lease can be granted to a specific subset of replicas — typically, the replicas in the same region as a particular access pattern's dominant read traffic — letting reads for that data be served locally from whichever nearby replica holds the lease, without contacting a leader that might be a continent away. This directly attacks the physical floor described above: if most reads for a given piece of data come from users near one specific region, granting that region's replica a quorum lease turns an ocean-crossing read into a local one, at the cost of the same bounded staleness risk article 3.4's clock-skew safety margins were built to manage, just now applied per-lease-holder rather than only per-leader.
Follower reads: trading a small staleness bound for locality
A closely related, somewhat simpler technique — used directly by CockroachDB (9.2) and conceptually available in most systems this series has covered since Phase 8 — is the follower read: rather than requiring every read to go through the current Raft or Paxos leader, a client can read directly from a nearby follower replica, accepting a small, bounded staleness (the follower might be a few log entries behind the leader) in exchange for avoiding the leader round-trip entirely. This is exactly article 8.1's etcd "serializable read" concept (a local read, possibly stale, versus a linearizable one that always contacts the leader), specialized for the geo-distributed case where "possibly stale" specifically means "possibly a few milliseconds behind, because the follower hasn't yet received the leader's latest AppendEntries batch" rather than an unbounded, unpredictable staleness. Some systems tighten this further with bounded staleness reads, where the client specifies an acceptable staleness window (e.g., "no more than 5 seconds old") and the system picks whichever nearby replica can satisfy it, giving an explicit, tunable dial between article 1.4's EL and EC extremes rather than a fixed binary choice.
Witness and non-voting replicas: shaping the quorum without shaping the data
A different lever entirely, worth distinguishing clearly from Flexible Paxos's quorum-size relaxation (article 7.2): a witness replica (sometimes called a non-voting or arbiter replica) participates in the majority-overlap voting process — casting a vote in leader elections and acknowledging log entries for the purpose of reaching a majority — without storing the full data itself, or without being eligible to become leader. This is genuinely useful in geo-distributed deployments for a specific, practical reason: it lets an operator place a lightweight tie-breaking vote in a third region specifically to avoid an even-numbered majority requirement across two "real," full-data regions, without paying the cost of a third full data replica (and its associated storage and bandwidth cost) in that third region. A classic three-region deployment might place full replicas in two regions and a witness-only replica in a third, cheaper location purely to guarantee majority quorums remain resolvable if either of the two full-data regions goes down — the same underlying majority-overlap logic (article 3.3) this series has repeated throughout, just applied with an asymmetric mix of full and lightweight participants.
Geo-partitioning: keeping a write from crossing an ocean at all
The most direct answer to the physical floor, when the workload allows it: don't put a majority quorum across an ocean in the first place for data that doesn't need it. Geo-partitioning (sometimes marketed as "table localities" or "row-level geo-partitioning" in specific products, including CockroachDB's own implementation of the idea) lets an operator pin a specific shard, range, or table's Raft/Paxos group entirely within one region's replicas, based on the data's actual access pattern — a European customer's records live in European replicas, an Asian customer's in Asian replicas — so that the common case write for that data never needs a cross-ocean round-trip at all, while still retaining the option of a genuinely global quorum for the smaller subset of data (shared reference tables, cross-region coordination state) that actually needs it. This is, in effect, applying article 1.2's replication-topology thinking deliberately and non-uniformly across a single logical database, rather than accepting one fixed replication topology for every piece of data a system holds.
| Technique | What it trades | Where it's used |
|---|---|---|
| Quorum leases | Bounded staleness risk for fast local reads on a specific data subset | Research-lineage technique, echoed in several production systems' locality-aware read paths |
| Follower / bounded-staleness reads | A tunable staleness window for lower read latency | CockroachDB (9.2), etcd's serializable reads (8.1), most Phase 8-9 systems in some form |
| Witness/non-voting replicas | Extra storage cost avoided, in exchange for that region never serving full reads or becoming leader | CockroachDB, YugabyteDB (9.4), and most production Raft deployments crossing 3+ regions |
| Geo-partitioning | Global flexibility for a given row, in exchange for near-local latency on its common-case access pattern | CockroachDB locality features, Spanner's own partitioning options (9.1) |
Leader placement: the other half of the WAN latency story
Everything covered so far in this article addresses read latency. Write latency in a geo-distributed deployment has its own dedicated lever, worth naming explicitly: leader placement. Because every write in a Raft or Multi-Paxos group must be proposed by that group's current leader (articles 5.3, 5.5-5.6), and a client's write latency is dominated by its round trip to whichever region currently holds leadership, deliberately placing (and, when access patterns shift, deliberately re-electing) the leader in the region generating the most write traffic for that specific shard directly minimizes the common-case write latency, even though the underlying majority-quorum replication cost to the other regions is unchanged. Several production systems expose explicit leader-placement hints or preferences for exactly this reason — CockroachDB's locality-aware lease preferences and YugabyteDB's tablet leader affinity settings both let an operator express "prefer to keep this shard's leader in region X," directly trading a small amount of manual operational configuration for a real, measurable write-latency win on the dominant access pattern, the write-side mirror of the quorum-lease and follower-read techniques already covered for reads.
FAQ
Can any of these techniques eliminate the physical latency floor entirely?
No — every technique in this article works around the floor for a specific subset of operations (reads that can tolerate staleness, writes that can be geo-partitioned to stay local) rather than eliminating it for operations that genuinely require a fresh, globally-agreed answer touching data spread across regions. A truly global, always-fresh, cross-region write is always going to pay something close to the physical round-trip cost described in this article's opening section — no algorithm changes the speed of light.
Is a witness replica the same thing as Flexible Paxos's asymmetric quorums?
Related in spirit but a different specific mechanism — Flexible Paxos (7.2) relaxes which nodes must be in a Phase 1 versus Phase 2 quorum, among otherwise-equal participants; a witness replica is a participant that is structurally lighter-weight (no full data, ineligible for leadership) but still counts toward quorum size. Both are examples of this series' broader theme — quorum composition is a genuine, tunable design space — realized through different specific levers.
Does geo-partitioning conflict with the strong consistency guarantees this series has spent so much time building?
No — a geo-partitioned row's Raft or Paxos group still enforces exactly the same majority-overlap safety guarantee (article 3.3) as any other group; it simply chooses to place all of that group's replicas within one region rather than spreading them globally. The consistency guarantee for that specific row is unchanged; what changes is the latency profile and the fault-tolerance scope (a full regional outage affects that row's availability more than it would if replicas were spread across regions) — a real, honest trade-off, not a compromise on correctness.
Takeaways
- The speed of light, via fiber's real-world propagation speed, sets a hard physical floor on cross-region quorum latency — tens of milliseconds for continent-spanning round trips, unfixable by any algorithm.
- Quorum leases extend article 3.4's leader-lease concept to arbitrary regional replica subsets, letting locally-dominant read traffic be served without crossing the ocean.
- Follower reads and bounded-staleness reads trade a small, tunable staleness window for lower read latency — a direct, more finely-tunable extension of article 8.1's linearizable-vs-serializable read distinction.
- Witness/non-voting replicas shape quorum composition without full data-storage cost — a distinct lever from Flexible Paxos's quorum-size relaxation (7.2), aimed at cheap tie-breaking presence rather than cost/latency optimization of the primary quorum.
- Geo-partitioning avoids the floor altogether for data whose access pattern is genuinely regional, by keeping that data's entire replica set within one region — applying article 1.2's replication-topology thinking non-uniformly across a single database.
References & further reading
- CockroachDB documentation — Topology Patterns — follower reads, geo-partitioning, and locality configuration described directly.
- Corbett et al. — Spanner: Google's Globally-Distributed Database (OSDI 2012) — Spanner's own directory/partitioning options, referenced in article 9.1.
- cvam.sight — Consensus 3.4: Leases and Linearizable Reads — the foundational lease mechanism quorum leases extend.
- cvam.sight — Consensus 7.2: Flexible Paxos — the related but distinct quorum-composition flexibility.