KRaft & the Cluster
The idea in one sentence
Section titled “The idea in one sentence”A Kafka cluster needs somewhere to store cluster metadata — which broker leads which partition, what topics exist, who is in the ISR — and in Kafka 4.x that job is done by KRaft, a built-in Raft-based controller quorum, with ZooKeeper fully removed.
Why this changed
Section titled “Why this changed”For most of Kafka’s history, an external ZooKeeper ensemble stored cluster metadata. It was a separate system to deploy, secure, tune, and reason about — and it became a scalability bottleneck as clusters grew to many partitions. KRaft (Kafka Raft) moves metadata management inside Kafka itself. As of Kafka 4.0, KRaft is the only mode — ZooKeeper mode was removed entirely, and the --zookeeper CLI flag is gone.
Metadata is itself a log
Section titled “Metadata is itself a log”KRaft applies Kafka’s own philosophy to cluster coordination: it stores metadata as a replicated log. A set of brokers act as controllers forming a quorum; one is the elected leader. Every metadata change — create a topic, elect a new partition leader, update the ISR — is an event appended to the internal metadata log and replicated to the other controllers via the Raft protocol.
The payoff is consistency and speed: brokers catch up on metadata the same way consumers catch up on data — by reading a log from an offset. Leader failover and metadata propagation are far faster than the ZooKeeper-era watch model.
flowchart TB
subgraph quorum["Controller quorum (KRaft)"]
lead["Active controller (leader)"] -->|replicate metadata log| f1["Controller (follower)"]
lead -->|replicate metadata log| f2["Controller (follower)"]
end
lead -->|metadata updates| brk["Brokers (data plane)"]
brk --> cli["Clients discover leaders via bootstrap-server"] What it means for you as a developer
Section titled “What it means for you as a developer”Day to day, KRaft is mostly invisible from the client side — you still connect with --bootstrap-server, produce, and consume exactly the same way. What changes is the mental model and the operations:
- One system, not two. No ZooKeeper to deploy or monitor. Brokers and controllers are Kafka processes; small clusters can even combine both roles.
- Faster recovery. Controller failover and metadata propagation are quicker, so partition leadership settles faster after a broker dies.
- Modern defaults. Kafka 4.x also ships the KIP-848 next-generation consumer rebalance protocol as generally available and strengthens transactions (KIP-890) — both covered later in this course.