Physical clocks and why they lie.

Every computer has a clock, and every one of those clocks is quietly, constantly wrong — not broken, just imprecise, in a way that compounds the moment you try to compare "now" across two machines. This article makes that imprecision concrete: what actually causes clock drift, why NTP helps but doesn't solve the problem, why you cannot use timestamps to reliably order events across machines, and how Google's Spanner responded — not by pretending the problem away, but by building custom hardware to measure and bound exactly how wrong the clock might be. This is the mechanical detail behind article 1.1's "Lie #2," and the last piece of vocabulary before logical clocks (2.2) show a genuinely different way to sidestep the whole mess.

What a computer's clock actually is

Every computer has a small quartz crystal oscillator — the same basic technology as a quartz wristwatch — that vibrates at a specific frequency when a voltage is applied to it (usually around 32,768 Hz for the low-power real-time clock chip that keeps time even when the machine is off, and much higher frequencies for the main system clock). The operating system counts these vibrations to derive "how much time has passed," and adds that to a stored reference point to produce the wall-clock time your applications read.

This is already a hint at the whole problem: a clock isn't measuring time directly — it's counting oscillations of a physical crystal and trusting that the crystal vibrates at a perfectly constant, known rate. It doesn't. Temperature changes the resonant frequency of quartz measurably; manufacturing tolerances mean no two crystals, even from the same batch, oscillate at exactly the same frequency; and the crystal ages, drifting further from its nominal frequency over months and years. None of this is a defect — it's the physical reality of every quartz oscillator ever made, consumer-grade or otherwise.

Clock drift, with real numbers

The practical consequence is called clock drift (or clock skew when comparing two clocks against each other): a machine's clock runs measurably faster or slower than true time, accumulating error linearly. Typical consumer-grade crystal oscillators drift on the order of tens of parts-per-million — commonly cited as roughly 1 second every 11–12 days for a cheap crystal, though quality varies. Server-grade hardware does meaningfully better, but "better" here means less bad, not solved: even a well-specified server clock can drift several milliseconds per hour without correction, and that's before accounting for temperature swings in a datacenter, which measurably shift oscillator frequency in real time.

0.18 seconds might sound trivial. It is not trivial for a system trying to determine which of two events, on two different machines, happened first — especially when those events might be only microseconds apart in real time, which is routine for anything processing thousands of operations per second.

NTP: helps enormously, solves nothing structurally

The Network Time Protocol (NTP) is the standard fix — a hierarchical system of time servers (traceable back to atomic clocks and GPS receivers at the top, called "stratum 0/1" sources) that periodically corrects a machine's clock by exchanging timestamped messages and estimating network round-trip delay to compensate for it. Well-configured NTP keeps machines within single-digit milliseconds of true time under good conditions, and this is genuinely valuable — it's why your laptop's clock doesn't drift into obvious wrongness over weeks of use.

But NTP has real, structural limits that matter enormously for distributed systems design:

• NTP corrections can jump the clock, not just smoothly adjust it. If a machine's clock has drifted significantly and NTP detects this, depending on configuration it may step the clock — an instantaneous jump forward or backward — rather than gradually correcting ("slewing") it. A backward jump is exactly as bad as it sounds for any code assuming time only moves forward, which turns out to be an assumption baked deep into a surprising amount of software (timeout logic, cache expiry, log ordering).
• NTP itself can silently fail. A misconfigured firewall blocking NTP's UDP port, a machine that can't reach any time server, an NTP daemon that's crashed — all of these leave a machine's clock free-running and drifting with no indication anything is wrong, unless you specifically monitor NTP sync status (a real, commonly-missed operational gap).
• Network asymmetry breaks NTP's core assumption. NTP estimates one-way delay by assuming the round trip is symmetric (equal delay each direction) and halving it. On networks where the path there and the path back have meaningfully different latency (common with asymmetric routing, congestion in one direction, or satellite/cellular links), this assumption is simply false, and NTP's correction inherits that error.
• Precision is still bounded by network jitter. Even under ideal conditions, NTP's accuracy over the public internet is typically single-digit to tens of milliseconds — vastly better than unsynchronized drift, but still far too imprecise to safely order two events that might be microseconds apart.

Why you still can't compare timestamps across machines, even with good NTP

Suppose you have excellent NTP sync — both machines within 1ms of true time. Can you now safely say "event A (timestamped 14:00:03.100 on machine 1) happened before event B (timestamped 14:00:03.099 on machine 2)"? No — and this is the crux of the whole article. Machine 1's clock could be 0.5ms fast and machine 2's could be 0.5ms slow (both well within the "1ms accuracy" bound, since that bound is typically a ± figure), meaning the "0.001s gap" you observed could easily be entirely clock error rather than real elapsed time. The two events could have happened in the opposite real-world order from what their timestamps suggest, and nothing about the timestamps alone tells you that.

This isn't a corner case you can engineer around with better hardware alone — it's the reason article 1.1 called clock synchronization a fundamental limit rather than a precision problem. There is a real, physics-grounded lower bound here too: information about "what time is it right now, exactly" takes nonzero time to propagate from any reference clock to any machine that wants it, so there is no way to eliminate uncertainty entirely, only to shrink and bound it.

Google's answer: bound the uncertainty instead of pretending it's zero

This is the single most important idea in this article, and it's the bridge to article 9.1's full Spanner treatment: instead of treating "current time" as a single trustworthy number, treat it as an interval of uncertainty, and design your algorithm to be correct no matter where within that interval the true time actually falls.

Google's TrueTime API does exactly this. Rather than now() returning a single timestamp, TrueTime's TT.now() returns an interval [earliest, latest] that is guaranteed to contain the true current time. Every datacenter running Spanner has GPS receivers and atomic clocks (multiple, for redundancy — a "time master" per datacenter), so the uncertainty bound stays small — typically a few milliseconds in practice, occasionally more if a time master is temporarily unreachable and the bound has to widen conservatively to remain honest.

The genuinely clever part is what Spanner does with that interval: for a transaction that needs to be ordered with certainty relative to another, it simply waits out the uncertainty window before committing — this is literally called the "commit wait." If TT.now() returns [100ms, 107ms] uncertainty, Spanner waits until it's certain the interval has fully passed before allowing the transaction to be visible, guaranteeing that any transaction which started after this one will get a TrueTime interval entirely after this one's commit. It trades a small, bounded amount of latency (the width of the uncertainty interval, typically single-digit milliseconds) for a correctness guarantee that no amount of "just synchronize better" NTP tuning could ever provide, because the guarantee comes from knowing and respecting the bound, not from making the bound zero (which is physically impossible).

What everyone else does (most systems aren't Google)

TrueTime-grade infrastructure — atomic clocks and GPS receivers in every datacenter — is a genuinely enormous investment that makes sense for a company operating Spanner at Google's scale, and it is not what most systems building on this series' algorithms actually have available. The overwhelmingly more common approach, and the one nearly every algorithm from Phase 4 onward actually relies on, is to not use physical clocks for ordering at all. This is precisely the motivation for logical clocks — the subject of the next two articles — which derive a correct ordering of events purely from the pattern of messages sent and received, with zero dependence on any physical clock's accuracy. It is, in a real sense, a cleverer and cheaper solution to the same problem TrueTime solves with hardware: Lamport's insight (1978, predating TrueTime by decades) was that you don't need to know real time at all if all you actually need is a consistent order.

FAQ

Takeaways

• Every clock is a quartz oscillator whose frequency isn't perfectly constant — clock drift is physical reality, not a hardware defect, and it compounds continuously without correction.
• NTP narrows drift dramatically but has real structural limits: it can jump the clock, can fail silently, assumes symmetric network delay, and is still bounded by network jitter to single-digit-to-tens of milliseconds under real conditions.
• Even with good NTP sync, you cannot safely order two events on different machines by comparing timestamps — the accuracy bound is wide enough that closely-spaced events can show the wrong order.
• Google's TrueTime doesn't eliminate clock uncertainty (impossible) — it measures and bounds it precisely (via atomic clocks + GPS per datacenter), then designs correctness (commit wait) around the known bound rather than pretending the bound is zero.
• This "measure and design around the limit, don't fight it" pattern reappears throughout this series, most sharply in FLP (article 2.5).
• Most systems don't have TrueTime-grade infrastructure, which is exactly why logical clocks (Lamport, vector — next two articles) exist: they derive correct event ordering from message patterns alone, with zero dependence on physical clock accuracy.

References & further reading

• Corbett et al. — Spanner: Google's Globally-Distributed Database (OSDI 2012) — the primary source for TrueTime; full Spanner treatment in article 9.1.
• NTP documentation — clock discipline and step vs. slew — the mechanics of how NTP corrects a drifting clock.
• Kulkarni et al. — Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases (2014) — a bridge paper between physical-clock and logical-clock thinking, relevant heading into article 2.2.
• cvam.sight — Consensus 1.1: Why Distributed Systems Are Hard — "Lie #2," which this article makes mechanically precise.