KRaft & the Cluster

The idea in one sentence

A Kafka cluster needs somewhere to store cluster metadata — which broker leads which partition, what topics exist, who is in the ISR — and in Kafka 4.x that job is done by KRaft, a built-in Raft-based controller quorum, with ZooKeeper fully removed.

Why this changed

For most of Kafka’s history, an external ZooKeeper ensemble stored cluster metadata. It was a separate system to deploy, secure, tune, and reason about — and it became a scalability bottleneck as clusters grew to many partitions. KRaft (Kafka Raft, KIP-500) moves metadata management inside Kafka itself. As of Kafka 4.0, KRaft is the only mode — ZooKeeper mode was removed entirely, and the --zookeeper CLI flag is gone.

Metadata is itself a log

KRaft applies Kafka’s own philosophy to cluster coordination: it stores metadata as a replicated log. A set of brokers act as controllers forming a quorum; one is the elected leader. Every metadata change — create a topic, elect a new partition leader, update the ISR — is an event appended to the internal metadata log (topic __cluster_metadata, partition 0) and replicated to the other controllers via the Raft protocol.

The payoff is consistency and speed: brokers catch up on metadata the same way consumers catch up on data — by reading a log from an offset. Leader failover and metadata propagation are far faster than the ZooKeeper-era watch model.

flowchart TB
  subgraph quorum["Controller quorum (KRaft)"]
    lead["Active controller (leader)"] -->|replicate metadata log| f1["Controller (follower)"]
    lead -->|replicate metadata log| f2["Controller (follower)"]
  end
  lead -->|metadata updates| brk["Brokers (data plane)"]
  brk --> cli["Clients discover leaders via bootstrap-server"]

KRaft: a controller quorum replicating a metadata log

Proof: describing the quorum on a single combined node

On this course’s own single-broker container (process.roles=broker,controller — a combined node acting as both), kafka-metadata-quorum.sh talks to the quorum directly:

echo "=== describe --status ==="
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server $BOOTSTRAP describe --status

echo "=== describe --replication ==="
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server $BOOTSTRAP describe --replication

Real, unedited output:

=== describe --status ===
ClusterId:              5L6g3nShT-eMCtK--X86sw
LeaderId:               1
LeaderEpoch:            1
HighWatermark:          185
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   0
CurrentVoters:          [{"id": 1, "endpoints": ["CONTROLLER://localhost:29093"]}]
CurrentObservers:       []

=== describe --replication ===
NodeId  DirectoryId             LogEndOffset  Lag  LastFetchTimestamp  LastCaughtUpTimestamp  Status
1       AAAAAAAAAAAAAAAAAAAAAA  187           0    1790636449483       1790636449483          Leader

One voter, itself, as Leader — exactly what a single combined broker+controller node looks like: it is a quorum of one, so MaxFollowerLag/MaxFollowerLagTimeMs are trivially 0 and CurrentVoters lists only itself.

Proof: a real 3-node controller quorum (leader, followers, lag)

To see an actual quorum with followers, a temporary 3-node compose (all three nodes process.roles=broker,controller, no host ports published — this course’s own scratch stack, torn down immediately after this proof) was brought up, then the exact same command run against it:

# kafka-metadata-quorum.sh --bootstrap-server broker1:9092 describe --status
ClusterId:              5L6g3nShT-eMCtK--X86sw
LeaderId:               2
LeaderEpoch:            1
HighWatermark:          48
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   182
CurrentVoters:          [{"id": 1, "endpoints": ["CONTROLLER://broker1:9093"]}, {"id": 2, "endpoints": ["CONTROLLER://broker2:9093"]}, {"id": 3, "endpoints": ["CONTROLLER://broker3:9093"]}]
CurrentObservers:       []

# kafka-metadata-quorum.sh --bootstrap-server broker1:9092 describe --replication
NodeId  DirectoryId             LogEndOffset  Lag  LastFetchTimestamp  LastCaughtUpTimestamp  Status
2       AAAAAAAAAAAAAAAAAAAAAA  50            0    1790636489291       1790636489291          Leader
1       AAAAAAAAAAAAAAAAAAAAAA  50            0    1790636489121       1790636489121          Follower
3       AAAAAAAAAAAAAAAAAAAAAA  50            0    1790636489121       1790636489121          Follower

Node 2 won the election and shows Leader; nodes 1 and 3 show Follower, and MaxFollowerLagTimeMs: 182 is a real, nonzero number this time — the time since the slowest follower last caught up, on an otherwise idle 3-node cluster on this machine. This is the same shape of information kafka-topics.sh --describe’s Leader/Isr/high-watermark reporting gives for data partitions — proven in Brokers, Topics & Partitions — because the metadata partition is replicated by the exact same Raft-backed mechanism KRaft was built to generalize.

Proof: what’s actually inside `__cluster_metadata`

The metadata log is a real Kafka log — same segment/index files as any other partition (see The Log Abstraction) — just with a special decoder. kafka-dump-log.sh --cluster-metadata-decoder reads the raw __cluster_metadata-0 segment and prints the actual typed records:

# Redirect the full dump to a file first (this log grows for as long as the
# broker runs), then grep the first real example of each record type --
# keeps output bounded no matter how large the log has grown.
/opt/kafka/bin/kafka-dump-log.sh --cluster-metadata-decoder \
  --files /tmp/kafka-logs/__cluster_metadata-0/00000000000000000000.log --print-data-log \
  > /tmp/meta-dump.txt 2>&1

for t in LeaderChange FEATURE_LEVEL_RECORD CONFIG_RECORD REGISTER_CONTROLLER_RECORD \
         REGISTER_BROKER_RECORD BROKER_REGISTRATION_CHANGE_RECORD NO_OP_RECORD \
         TOPIC_RECORD PARTITION_RECORD; do
  grep -m 1 "$t" /tmp/meta-dump.txt
done

Real output — the first real example of each record type found in this course’s own metadata log:

| offset: 0 ... LeaderChange: {"version":0,"leaderId":1,"voters":[{"voterId":1}],"grantingVoters":[{"voterId":1}]}
| offset: 2 ... payload: {"type":"FEATURE_LEVEL_RECORD","version":0,"data":{"name":"metadata.version","featureLevel":30}}
| offset: 8 ... payload: {"type":"CONFIG_RECORD","version":0,"data":{"resourceType":4,"resourceName":"","name":"min.insync.replicas","value":"1"}}
| offset: 10 ... payload: {"type":"REGISTER_CONTROLLER_RECORD","version":0,"data":{"controllerId":1,"incarnationId":"SW2n63rERL2UUhHUFJ3VkQ","zkMigrationReady":false,"endPoints":[{"name":"CONTROLLER","host":"localhost","port":9093,"securityProtocol":0}], ...}}
| offset: 11 ... payload: {"type":"REGISTER_BROKER_RECORD","version":4,"data":{"brokerId":1,"incarnationId":"riTwzbFkR2qHiCWR29Fbzw","brokerEpoch":11,"endPoints":[{"name":"PLAINTEXT","host":"localhost","port":9092, ...},{"name":"PLAINTEXT_HOST","host":"localhost","port":19092, ...}], ..., "fenced":true, ...}}
| offset: 12 ... payload: {"type":"BROKER_REGISTRATION_CHANGE_RECORD","version":0,"data":{"brokerId":1,"brokerEpoch":11,"fenced":-1}}
| offset: 14 ... payload: {"type":"NO_OP_RECORD","version":0,"data":{}}
| offset: 26 ... payload: {"type":"TOPIC_RECORD","version":0,"data":{"name":"consumers-consumer-groups-and-rebalancing-orders","topicId":"obGo8usiSIiFSzu8Acbl-w"}}
| offset: 27 ... payload: {"type":"PARTITION_RECORD","version":2,"data":{"partitionId":0,"topicId":"obGo8usiSIiFSzu8Acbl-w","replicas":[1],"isr":[1],"leader":1,"leaderEpoch":0,"partitionEpoch":0, ...}}

(the TOPIC_RECORD/PARTITION_RECORD example is from another lesson’s own topic on this same shared harness broker — the metadata log carries every topic ever created on it, which is itself a real illustration of “one broker, one metadata log, every change ever made to the cluster shape” — proven above by the fact that the very first records at offset 0-14 are this broker’s own bootstrap, before any topic existed yet.)

These are the actual event types the “metadata is a log” claim above refers to, not an abstraction: a LeaderChange control record for the controller quorum itself, FEATURE_LEVEL_RECORD entries pinning metadata.version and per-feature versions, a CONFIG_RECORD for the default min.insync.replicas, REGISTER_CONTROLLER_RECORD/REGISTER_BROKER_RECORD as this node joined both roles, a BROKER_REGISTRATION_CHANGE_RECORD un-fencing the broker once it was ready to serve, periodic **NO_OP_RECORD**s (quorum heartbeat/liveness padding), and finally — only once kafka-topics.sh --create was run — a TOPIC_RECORD and a PARTITION_RECORD carrying exactly the replica/ISR/leader assignment that kafka-topics.sh --describe reports from the data-plane side. Every broker joining the cluster replays this same log from offset 0 to build its in-memory view of the cluster — this is what “metadata is itself a log” means, down to the bytes.

Combined vs isolated controllers

Every node above ran process.roles=broker,controller — a combined node doing both jobs, the only shape this course’s single- and 3-node proofs used (and the shape small clusters typically run). Kafka also supports isolated controllers: a node with process.roles=controller only (never serves client traffic or hosts partitions) alongside separate nodes with process.roles=broker only. Isolated controllers trade operational simplicity (more processes to run) for blast-radius isolation — a broker under heavy client load cannot starve the controller quorum of CPU/disk I/O it needs to keep electing partition leaders promptly, which matters more as cluster size and partition count grow. This course’s proofs are all combined-mode; the choice between the two is an operational sizing decision covered further in Running Kafka Locally and the production module.

`controller.quorum.voters` vs `controller.quorum.bootstrap.servers`, and dynamic quorum (KIP-853)

Every node above was started with a static controller.quorum.voters=1@broker1:9093,2@broker2:9093,3@broker3:9093 — every voter’s node ID and address fixed at startup, verified directly in the container’s own server.properties. This is the original KRaft model: to change quorum membership you edit and restart every node’s config.

KIP-853 adds a dynamic quorum alternative, confirmed present and usable on this course’s own Kafka 4.3.1 image:

controller.quorum.bootstrap.servers is a real, documented broker config key (verified against the cached official config reference this course’s harness uses) — a bootstrap-style address list for discovering the current quorum, instead of a fixed, closed voter set.
kafka-metadata-quorum.sh’s own --help output on this image lists three subcommands, not one: describe, add-controller, and remove-controller — real, present commands for changing quorum membership at runtime (via a --bootstrap-controller connection) rather than editing controller.quorum.voters and restarting every controller.

What it means for you as a developer

Day to day, KRaft is mostly invisible from the client side — you still connect with --bootstrap-server, produce, and consume exactly the same way, proven throughout this course. What changes is the mental model and the operations:

One system, not two. No ZooKeeper to deploy or monitor. Brokers and controllers are Kafka processes; small clusters can even combine both roles, as every proof in this lesson did.
Faster recovery. Controller failover and metadata propagation are quicker, so partition leadership settles faster after a broker dies.
Modern defaults. Kafka 4.x also ships the KIP-848 next-generation consumer rebalance protocol as generally available (its own banner appeared, unprompted, in the very first proof of this course — see What Is Kafka) and strengthens transactions (KIP-890) — both covered later in this course.

Reading AI-generated code

🐛 Spot the bug

kafka-topics.sh --zookeeper zk1:2181,zk2:2181,zk3:2181 --list

kafka-topics.sh --zookeeper zk1:2181,zk2:2181,zk3:2181 \
--create --topic events --partitions 6 --replication-factor 3

Show the answer

This is a very common pattern for an AI assistant to generate — it comes straight from pre-4.0 training data, where --zookeeper was the normal way to run admin commands. On Kafka 4.x, verified directly against this course’s own broker, there is no ZooKeeper process running at all (KRaft removed it, proven above by every metadata operation in this lesson going through --bootstrap-server and the __cluster_metadata log instead). Running this script as written fails immediately — kafka-topics.sh in 4.x does not even accept a --zookeeper flag anymore. The fix is a straight swap to --bootstrap-server localhost:9092 (or the real bootstrap address), which talks to a broker, which resolves the request against the KRaft-managed metadata itself; no ZooKeeper connection string, and no ZooKeeper ensemble to keep running, is needed anywhere.

Mental Model

Before KRaft, “what does the cluster look like right now” was a question you asked a different system (ZooKeeper) than the one doing the work (the brokers) — two systems that had to agree, and could drift apart under partition or bug. KRaft’s mental model collapses that into one idea already used everywhere else in this course: cluster shape is just another log. The controller quorum is a tiny Kafka cluster (proven above: it has a Leader, Followers, a high watermark, replication lag — all the same vocabulary as a data partition) whose one job is replicating the log of “what changed about the cluster.” Every broker is a consumer of that log. There is no second source of truth to fall out of sync with, because there is no longer a second system.

Trade-offs

Combined (broker+controller) nodes — fewer processes to run and monitor (every proof in this lesson used this shape), at the cost of a broker’s client-traffic load sharing CPU/disk with its own controller responsibilities.
Isolated controllers — better blast-radius isolation on larger clusters, at the cost of running and monitoring a separate fleet of controller-only nodes.
Static controller.quorum.voters — simple, explicit, and what every proof in this lesson used; changing membership means editing config and restarting every controller.
Dynamic quorum (KIP-853) — proven above to exist as real tooling in 4.3.1 (add-controller/remove-controller, controller.quorum.bootstrap.servers); the payoff is membership changes without a full-quorum restart, at the cost of a less-explicit, more-operationally-complex bootstrap-discovery model to reason about.

Common Misconceptions

“Kafka 4.x still needs ZooKeeper unless you opt into KRaft.” No — proven above and stated directly by KIP-500: as of Kafka 4.0, KRaft is the only mode; ZooKeeper mode was removed, not just deprecated.
“kafka-metadata.sh is the tool for inspecting KRaft state.” No — checked directly against /opt/kafka/bin/ on this course’s own 4.3.1 image: that script does not exist. The real tools are kafka-metadata-quorum.sh and kafka-metadata-shell.sh.
“The controller quorum and data replication (leader/ISR) are two unrelated mechanisms.” No — proven above: describe --replication on the metadata partition reports Leader/Follower/lag in the exact same shape kafka-topics.sh --describe reports for a data partition, because KRaft’s controller quorum is a Raft-replicated Kafka partition, just one carrying cluster metadata instead of application records.
“Every Kafka 4.x cluster must run isolated controller nodes.” No — every real proof in this lesson (single-node and 3-node) used combined process.roles=broker,controller nodes; isolated controllers are an available, larger-cluster option, not a requirement.