Consensus Algorithms · Phase 8

Consensus in Production

Article 8.2 of 6

Jul 21, 2026 · devops · 20 min read · 4200 words intermediate

Apache ZooKeeper.

devops distributed-systems zookeeper series-consensus

Before etcd (8.1), before Consul (8.3), there was ZooKeeper — the system that established the entire "consensus-backed coordination service" product category most of Phase 8 covers, built on Zab (article 5.7's closing algorithm). This article covers ZooKeeper's data model (a filesystem-like hierarchical namespace, a genuinely different presentation from etcd's flat key-value store), ephemeral nodes as its own answer to article 8.1's lease concept, and the specific coordination patterns (locks, leader election, service discovery) it directly enabled — patterns so influential that etcd's own design (8.1) can be read, in significant part, as a deliberate response to ZooKeeper's strengths and rough edges.

The znode namespace: a filesystem, not a flat key-value store

ZooKeeper organizes data as znodes in a hierarchical namespace — genuinely path-structured, like a filesystem (/services/payments/leader), rather than etcd's conceptually flat keyspace (which happens to support prefix queries but isn't structurally hierarchical the way ZooKeeper's actually is). This isn't just an aesthetic difference: ZooKeeper's client API includes operations that specifically leverage the hierarchy — listing a znode's direct children, watching for changes anywhere within a subtree — that map naturally onto coordination patterns organized around logical groupings (all workers registered under /workers/, say), a structural fit that flat key-value stores need to simulate via naming conventions rather than getting for free from the data model itself.

Ephemeral znodes: ZooKeeper's answer to article 8.1's leases

Where etcd uses explicit leases (article 8.1) attached to arbitrary keys, ZooKeeper's equivalent mechanism is the ephemeral znode — a znode tied directly to the client session that created it, automatically deleted when that session ends (either a clean disconnect or a timeout-detected failure, echoing article 2.4's failure-detection machinery once again). The core coordination pattern this enables: a service registers its presence by creating an ephemeral znode under a well-known path; if the service crashes, its session times out, and ZooKeeper automatically removes the znode — any client watching that path (ZooKeeper's own watch mechanism, conceptually parallel to article 8.1's etcd watches, though with different specific semantics around one-shot vs. continuous notification) is notified immediately.

Ephemeral znodes: presence tied directly to session liveness Service registers /workers/worker-1 (ephemeral) Session alive znode persists session ends (crash/timeout) znode auto-deleted watchers notified instantly

Fig 1 — Ephemeral znodes make "is this service alive" a structural property of the namespace itself, not a separately-maintained TTL.

The coordination pattern library ZooKeeper established

ZooKeeper's design directly enabled, and its documentation explicitly codified, a small set of coordination "recipes" that became the de-facto reference patterns for the entire product category — worth knowing by name since they reappear, in spirit, across every system Phase 8 covers:

  • Distributed locks — clients create sequentially-numbered ephemeral znodes under a lock path; the client holding the lowest-numbered znode holds the lock; others watch the next-lowest znode and are notified when it's their turn — a clean, fair queuing mechanism built entirely from znode creation, sequencing, and watches.
  • Leader election — the same sequential-ephemeral-znode pattern, just interpreted as "lowest number is leader" instead of "lowest number holds the lock" — directly implementing this series' article 4.2 leader-election concept on top of ZooKeeper's primitives.
  • Service discovery — services register ephemeral znodes with their connection details under a well-known path; clients watch that path's children to maintain a live view of currently-available service instances.
This pattern library is arguably ZooKeeper's most lasting influence on the field — even systems that don't use ZooKeeper directly (etcd, Consul, and others) implement conceptually identical recipes on top of their own primitives (leases instead of ephemeral znodes, but the same underlying "presence tied to a renewable liveness signal, watched by interested parties" shape). Reading etcd's lease-based leader-election pattern (article 8.1) after this article should feel like recognizing the same idea in different clothing, not learning something new.

ZooKeeper vs. etcd: what actually differs

PropertyZooKeeperetcd (8.1)
Consensus algorithmZab (5.7)Raft (5.5-5.6)
Data modelHierarchical znode namespaceFlat key-value (with prefix queries)
Liveness/TTL mechanismEphemeral znodes (tied to session)Leases (explicit, attachable to any key)
Client API era/styleOlder, Java-centric, callback-based watchesNewer, gRPC-based, streaming watches
Notable production usersKafka (historically, pre-KRaft — see 8.4), Hadoop ecosystemKubernetes (8.5)

FAQ

Is ZooKeeper's Zab genuinely different from Raft in a way that matters for these coordination patterns?

Not for the patterns themselves — the lock/election/discovery recipes rely on ordering and ephemeral-liveness guarantees that both Zab (5.7) and Raft (5.5-5.6) provide equivalently, per their shared majority-overlap safety foundation (article 3.3). The difference that matters more in practice is the surrounding data model and client API, covered above.

Why did Kafka move away from ZooKeeper (covered fully in article 8.4)?

Briefly here, full treatment in 8.4: Kafka historically used ZooKeeper purely for cluster metadata and controller election, but running a separate ZooKeeper ensemble alongside a Kafka cluster was real operational overhead: two systems to deploy, monitor, and scale independently. KRaft replaces that external dependency with Raft consensus built directly into Kafka itself.

Does ZooKeeper support the same linearizable/serializable read distinction as etcd (8.1)?

Conceptually yes — ZooKeeper distinguishes between a full linearizable read (via a sync call before reading, ensuring the client sees the latest committed state) and a plain local read (potentially stale, served directly from whichever server the client is connected to) — the same underlying trade-off article 3.4 built up theoretically, exposed through ZooKeeper's own specific API shape rather than etcd's.

Takeaways

  • ZooKeeper, built on Zab (5.7), established the coordination-service product category that etcd (8.1) and Consul (8.3) both directly follow and respond to.
  • Its hierarchical znode namespace is a genuinely different data model from etcd's flat key-value store, with API operations (child listing, subtree watches) that leverage that structure directly.
  • Ephemeral znodes (tied to client session liveness) are ZooKeeper's answer to article 8.1's leases — the same underlying "presence tied to a renewable liveness signal" idea, different implementation shape.
  • ZooKeeper's lock/leader-election/service-discovery recipe library is its most lasting influence — the same conceptual patterns reappear across every coordination service this phase covers, including systems that don't use ZooKeeper directly.

References & further reading

← 8.1 etcd next: 8.3 Consul →
© cvam — written in plaintext, served warm