Phase 8 covered consensus systems operating within a single datacenter or a handful of nearby zones. Phase 9 asks what changes when replicas are spread across continents — and Google's Spanner is the sharpest possible starting point, because it makes the boldest possible claim under those conditions: externally consistent transactions, globally, ordered consistently with real-world wall-clock time, at planet scale. This article covers how Spanner is built — Multi-Paxos (article 5.3) replicating each of many data shards independently — and the specific piece of custom infrastructure, TrueTime, that makes the strong global guarantee affordable rather than merely theoretically possible. This is the direct, concrete payoff of article 1.4's PACELC discussion, which used Spanner as its worked example before this series had built any of the machinery to actually explain how.
Spanner's architecture: many Paxos groups, one system
Spanner shards data into many small pieces, each independently replicated by its own Multi-Paxos group (article 5.3) — the same fast-path/leader-election machinery this series built in Phase 5, just running many instances in parallel, one per shard, rather than one instance for an entire dataset. This is directly the sharding pattern this series has referenced without deep coverage since Phase 8 — the interesting, genuinely hard problem isn't any single shard's replication (that's ordinary Phase 5 material) but coordinating transactions that span multiple shards, potentially in different datacenters, and making the whole system behave as if it had one single, globally-consistent clock — which is precisely what article 1.3's linearizability definition demands, extended to multi-shard transactions.
TrueTime: bounded uncertainty instead of a single number
Recall article 2.1's entire argument: no clock can give you a trustworthy single "current time" value, because clocks drift and synchronization has fundamental limits. TrueTime doesn't try to defeat that limit — it accepts it and makes it precise. Instead of returning a single timestamp, TrueTime's API returns an interval [earliest, latest] that is guaranteed to contain the true current time — backed by real hardware (atomic clocks and GPS receivers) deployed across Google's datacenters specifically to keep that interval's width (the uncertainty, typically single-digit milliseconds) as small and as reliably bounded as possible.
Fig 1 — TrueTime's contribution isn't eliminating uncertainty — it's making the uncertainty explicit, bounded, and small.
Commit-wait: turning a bound into a guarantee
Here's the mechanism that actually cashes TrueTime's interval into a correctness guarantee, and it's genuinely elegant. When Spanner commits a transaction, it assigns it a timestamp — and then, before acknowledging the commit to the client, it waits until TrueTime's interval guarantees that the assigned timestamp is definitely in the past (waiting out the uncertainty window, latest - assigned_timestamp). This commit-wait is a deliberate, small, bounded latency cost paid specifically to guarantee that any transaction that could have observed this commit (because it started after this commit's timestamp, in true wall-clock time) actually does — delivering external consistency (a stronger version of article 1.3's linearizability, ordering transactions consistently with real-world time, not just some internally-agreed order) even though no two machines' clocks are ever perfectly synchronized.
FAQ
Does Spanner use Paxos or Raft for its per-shard replication?
Multi-Paxos (article 5.3), predating Raft's 2014 publication and remaining Spanner's replication mechanism — a reminder from article 5.7's Phase 5 closing comparison that both algorithm families remain live, actively-deployed technology, not one having strictly superseded the other.
What happens if TrueTime's uncertainty bound is ever wrong (wider than reported)?
This would be a serious correctness risk — commit-wait's guarantee depends entirely on the reported interval genuinely containing the true time. Google's TrueTime infrastructure is specifically engineered (redundant time sources, continuous monitoring) to make this failure mode extremely unlikely, but it's worth noting explicitly that the entire external-consistency guarantee rests on this one hardware/infrastructure assumption holding.
Could a company without Google's infrastructure budget replicate Spanner's guarantees?
Approximately, with more latency cost — without dedicated atomic-clock/GPS hardware, a system must use a looser, NTP-derived uncertainty bound, making commit-wait proportionally more expensive. This is exactly the trade-off CockroachDB (article 9.2, next) makes differently, worth reading immediately after this article for the direct contrast.
Takeaways
- Opens Phase 9: consensus at planet scale, where geographic distance turns latency into a first-class design constraint (article 1.4's PACELC, now with real mechanism behind it).
- Spanner shards data into many independent Multi-Paxos (5.3) groups — the hard problem is coordinating multi-shard transactions consistently, not any single shard's replication.
- TrueTime returns a bounded uncertainty interval instead of a false single timestamp, backed by real atomic-clock/GPS hardware — honest about clock imprecision (article 2.1) rather than pretending it away.
- Commit-wait turns that bound into a correctness guarantee: pause until the assigned timestamp is certainly in the past, delivering external consistency at a small, bounded latency cost.
- This is the concrete mechanism behind article 1.4's PACELC example — the atomic-clock investment exists specifically to keep commit-wait's cost small.
References & further reading
- Corbett et al. — Spanner: Google's Globally-Distributed Database (OSDI 2012) — the primary source; TrueTime and commit-wait described in full.
- cvam.sight — Consensus 1.4: PACELC and Beyond CAP — the theoretical framing this article's TrueTime mechanism cashes out concretely.
- cvam.sight — Consensus 2.1: Physical Clocks and Why They Lie — the clock-uncertainty problem TrueTime is engineered specifically to bound.