Apache Kafka's original architecture used ZooKeeper (8.2) purely for cluster metadata and controller election — the actual message data never went through ZooKeeper, only coordination about which broker owns which partition. KRaft (Kafka Raft) removes that external ZooKeeper dependency entirely, replacing it with a Raft implementation (article 5.5-5.6) built directly into Kafka's own brokers — treating cluster metadata as just another Kafka-style partition log. This article covers why the ZooKeeper dependency became a real operational liability worth removing, how KRaft folds metadata into Kafka's existing log abstraction, and a fact worth making explicit for the first time in this phase: Kafka's own data-plane partitions were already a real-world instance of article 4.3's atomic broadcast, years before KRaft touched metadata consensus at all.
Why ZooKeeper was there in the first place, and why it became a liability
Kafka's original design used ZooKeeper for a genuinely bounded set of responsibilities: tracking which brokers are alive, storing topic/partition configuration, and — critically — electing a controller broker responsible for coordinating partition leadership across the cluster. This is a legitimate, correctly-scoped use of a coordination service exactly like article 8.2 described. The problem, articulated directly in Kafka's own KIP (Kafka Improvement Proposal) process: running Kafka meant operating two separate distributed systems — Kafka itself, and a ZooKeeper ensemble alongside it — each with its own deployment, monitoring, upgrade, and capacity-planning lifecycle. This operational overhead, and specific scaling limits ZooKeeper's design imposed on the total number of partitions a cluster could support (driven by how much metadata needed to round-trip through ZooKeeper's own consensus), motivated removing the dependency entirely rather than continuing to work around its limits.
Metadata as just another partition log
KRaft's core design move: rather than bolting on a separate embedded consensus module, represent cluster metadata itself as a special internal Kafka topic (__cluster_metadata), replicated among a small set of controller-eligible brokers using Raft — meaning metadata changes are literally just entries appended to a partition log, using the exact same log-replication machinery (article 5.6's AppendEntries, log-matching property, commitment rules) Kafka's designers already needed to build and maintain for ordinary data partitions. This is a clean, elegant unification: instead of "Kafka's data path" and "Kafka's metadata path" being two conceptually different systems (one Kafka-native, one ZooKeeper/Zab-based), KRaft makes them the same underlying mechanism, just applied to a different specific log.
Fig 1 — KRaft doesn't just swap Zab for Raft — it eliminates the conceptual and operational split between data-plane and metadata-plane replication entirely.
Kafka's data partitions were always atomic broadcast
Worth pausing on a fact this series can now state precisely, having built article 4.3's equivalence result: a Kafka partition, even in the pre-KRaft architecture, was already a direct real-world instance of Total Order (Atomic) Broadcast — every producer's messages to a given partition are appended in a single, agreed order (Kafka's own in-sync-replica, ISR-based replication protocol enforcing exactly article 4.3's atomicity and total-order properties among a partition's replica set), and every consumer reading that partition sees messages in that identical order. This was true well before KRaft ever touched metadata consensus — it's worth stating explicitly because it directly confirms article 4.3's claim that atomic broadcast shows up as a practical, everyday production mechanism, not just an abstract equivalence result confined to research papers.
FAQ
Does KRaft change how Kafka's actual message data is replicated?
No — KRaft specifically replaces the ZooKeeper-based metadata/controller-election path; ordinary topic partition replication (the ISR mechanism mentioned above) is a separate, pre-existing part of Kafka's architecture and is unaffected by the KRaft migration.
Is KRaft's Raft implementation identical to etcd's (8.1)?
Conceptually the same algorithm (article 5.5-5.6's Raft), but implemented independently within Kafka's own codebase and specifically integrated with Kafka's partition-log abstraction — not a shared library with etcd, though both are correctly described as "Raft implementations" in the sense this series has built that term.
Did removing ZooKeeper actually solve the partition-count scaling limit mentioned above?
Yes, substantially — by folding metadata into Kafka's own optimized log-replication path (rather than round-tripping through a separate system with its own, differently-tuned scaling characteristics), KRaft-based clusters support meaningfully larger numbers of partitions than the pre-KRaft ZooKeeper-based architecture could comfortably handle, a concrete, measured operational improvement.
Takeaways
- Kafka originally used ZooKeeper (8.2) narrowly, for metadata and controller election — a correctly-scoped use that nonetheless became a real operational liability (two systems to run, real partition-count scaling limits).
- KRaft replaces that dependency with a Raft implementation built directly into Kafka, representing cluster metadata as just another partition log —
__cluster_metadata, replicated with the same log-replication machinery (5.6) Kafka's data partitions already used. - This unifies the data-plane and metadata-plane replication mechanisms into one conceptual system, rather than two separately-operated ones — a direct operational and architectural simplification.
- Kafka's data partitions were already a real-world atomic broadcast instance (article 4.3) well before KRaft — a concrete confirmation that the equivalence this series proved isn't just academic.
References & further reading
- Apache Kafka — KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum — the original proposal and motivation for KRaft.
- Apache Kafka documentation — KRaft — the current architecture reference.
- cvam.sight — Consensus 4.3: Atomic Broadcast — the equivalence this article shows Kafka's partitions have always instantiated.
- cvam.sight — Consensus 8.2: Apache ZooKeeper — the system KRaft removes as an external dependency.