Kubernetes and etcd — Consensus 8.5

Article 8.1 covered etcd on its own terms. This article covers the specific relationship that makes etcd one of the most consequential consensus-backed systems in modern infrastructure: every single object in a Kubernetes cluster — every Pod, every Deployment, every Service, every ConfigMap — lives in etcd, and nowhere else. Kubernetes has no separate database; etcd is the cluster's entire persistent state. This article covers why that architecture works, how Kubernetes's defining "controller" pattern maps directly onto etcd's watch mechanism (article 8.1), and the optimistic-concurrency mechanism that lets thousands of controllers safely read and write shared state without explicit locking.

etcd as the single source of truth

Kubernetes's API server is, architecturally, close to a thin translation layer in front of etcd: when you kubectl apply a Deployment, the API server validates and normalizes the request, then writes it as a key in etcd (article 8.1's flat, MVCC-versioned key-value store) — there is no separate relational database, no secondary source of truth to keep in sync. Every piece of desired and observed cluster state — what you asked for, and what's actually running — is a key in etcd's namespace, replicated and made durable by the exact Raft mechanism (article 5.5-5.6) this series built from first principles. This radical simplicity (one storage system, one consistency model, for literally everything) is a deliberate architectural bet, and it's one that pays off directly because of the specific mechanism covered next.

The controller pattern: watch, diff, reconcile

Kubernetes's defining architectural idiom — the controller pattern — is a direct, almost one-to-one application of etcd's watch mechanism (article 8.1). A controller (the Deployment controller, the ReplicaSet controller, hundreds of custom controllers in real clusters) does exactly three things in a loop: watch etcd (via the API server) for changes to the resources it cares about, diff the observed state against the desired state encoded in those resources, and reconcile by issuing whatever API calls (which themselves become new etcd writes) are needed to close the gap. This loop runs continuously and asynchronously across potentially hundreds of independent controllers, all watching overlapping slices of the same etcd-backed state, all reacting to changes as etcd's Raft-backed log commits them and streams them out via watches.

Fig 1 — Hundreds of independent controllers run this loop concurrently, all fed by the same underlying etcd watch stream.

Optimistic concurrency: resourceVersion instead of locks

With potentially many controllers reading and writing overlapping resources concurrently, Kubernetes doesn't use explicit locking (article 8.2's ZooKeeper-style sequential-ephemeral-znode locks would be one option, but a heavyweight one for this specific access pattern) — it uses optimistic concurrency control built directly on etcd's MVCC revision numbers (article 8.1). Every Kubernetes object carries a resourceVersion field, which is, under the hood, exactly etcd's MVCC revision for that key. When a controller updates an object, it includes the resourceVersion it last read; etcd's underlying compare-and-swap semantics (itself a direct, practical application of this series' repeated "propose a value, only accept if nothing changed underneath you" pattern, echoing article 5.2's Phase 2 acceptance logic) reject the write if the resource has changed since — forcing the controller to re-read the latest state and retry its reconciliation, rather than silently overwriting a concurrent change and losing information.

This is a clean, production-scale instance of a theme this series introduced abstractly and now gets to see fully cashed out: rather than pessimistically locking a resource before every read (expensive, and a real bottleneck at Kubernetes's scale — potentially thousands of controllers across a large cluster), Kubernetes lets every controller optimistically compute its intended change, and lets etcd's underlying consensus-backed compare-and-swap mechanism be the single, authoritative arbiter of whether that change is still valid by the time it's actually applied. This is exactly the kind of design decision that only works because the underlying storage layer (etcd, backed by Raft, backed by the majority-overlap safety argument of article 3.3) is trustworthy enough to be the sole point of truth for a conflict check this consequential.

Why etcd's specific design fit Kubernetes better than ZooKeeper's

Revisiting article 8.2's comparison table with Kubernetes's specific access pattern in mind: the controller loop's core need — a live, precisely-resumable stream of changes across a broad set of resources, with a clean way to know exactly what changed and in what order — maps far more directly onto etcd's watch + MVCC revision history (article 8.1) than onto ZooKeeper's older, more callback-oriented watch API (article 8.2), which was originally designed more around the ephemeral-znode presence/locking recipes than around Kubernetes-style continuous, high-volume state reconciliation. This isn't a claim that ZooKeeper is deficient in general — article 8.2 covered plenty of things it does well — it's a specific statement about fit-for-purpose: Kubernetes's designers picked the coordination service whose particular feature set (streaming watches anchored in revision history) most directly matched their specific architectural pattern.

FAQ

What happens to a Kubernetes cluster if etcd becomes unavailable?

The cluster's control plane (API server, controllers, scheduler) can't read or write any state, so no new scheduling or reconciliation decisions can be made — but already-running Pods keep running on their nodes (kubelet, the per-node agent, operates independently for already-scheduled workloads). This is a direct, practical instance of article 4.1's safety/liveness framing: etcd unavailability is a liveness failure for the control plane, not a safety violation or an immediate outage for already-running workloads.

Does every kubectl read go through etcd directly?

Not necessarily — the API server maintains in-memory caches (informers, built on the same watch mechanism) to serve many reads without round-tripping to etcd for every single request, an optimization layered on top of, not replacing, etcd's role as the authoritative source of truth for writes and the ultimate backing store.

Can Kubernetes run with a non-Raft-backed etcd alternative?

Kubernetes's API is technically storage-agnostic in principle, and alternative backends have been explored in various projects, but etcd (and by extension Raft, article 5.5-5.6) is overwhelmingly the standard, battle-tested, officially-supported choice — the watch/MVCC/resourceVersion machinery this article describes is specifically etcd's design, and substituting a different backend means reimplementing equivalent guarantees.

Takeaways

Kubernetes has no separate database — etcd (8.1) is the single source of truth for every object in a cluster, with the API server as a thin translation layer in front of it.
The controller pattern (watch, diff, reconcile) is a direct, near-literal application of etcd's watch mechanism, run concurrently by hundreds of independent controllers across a large cluster.
resourceVersion is etcd's MVCC revision number exposed directly to Kubernetes's API — the foundation of an optimistic concurrency control scheme that avoids explicit locking at scale.
etcd's specific watch + MVCC revision history design was a better architectural fit for Kubernetes's continuous, high-volume reconciliation pattern than ZooKeeper's (8.2) older, more locking/presence-oriented API.
etcd unavailability is a control-plane liveness failure (article 4.1), not an immediate safety violation or workload outage — already-running Pods keep running independently.

References & further reading

Kubernetes documentation — Cluster Architecture — the API server / etcd relationship described directly.
Kubernetes documentation — Controllers — the watch/diff/reconcile pattern in the project's own words.
cvam.sight — Consensus 8.1: etcd — the watch/MVCC/lease mechanisms this article builds directly on.
cvam.sight — Consensus 8.2: Apache ZooKeeper — the comparison this article's fit-for-purpose argument depends on.