KRaft Metadata Log
The idea in one sentence
Section titled “The idea in one sentence”Cluster metadata is not a database the controller queries — it is an ordinary Kafka-style append-only log (the internal topic __cluster_metadata), replicated by a Raft quorum of controllers, decodable with the exact same kafka-dump-log.sh tool used on any data topic, just with a decoder that knows the metadata record schemas.
Real record types, decoded
Section titled “Real record types, decoded”Dumping this module’s own __cluster_metadata partition 0 with the cluster-metadata decoder:
/opt/kafka/bin/kafka-dump-log.sh --cluster-metadata-decoder \ --files /tmp/kafka-logs/__cluster_metadata-0/00000000000000000000.log \ | grep -o '"type":"[A-Z_]*"' | sort -uReal, unedited output — every distinct record type this cluster actually wrote during normal startup and topic creation, no synthetic examples:
"type":"BEGIN_TRANSACTION_RECORD""type":"BROKER_REGISTRATION_CHANGE_RECORD""type":"CONFIG_RECORD""type":"END_TRANSACTION_RECORD""type":"FEATURE_LEVEL_RECORD""type":"NO_OP_RECORD""type":"PARTITION_RECORD""type":"PRODUCER_IDS_RECORD""type":"REGISTER_BROKER_RECORD""type":"REGISTER_CONTROLLER_RECORD""type":"TOPIC_RECORD"PARTITION_CHANGE_RECORD (an ISR change) does not appear here — this single-broker cluster’s ISR never changes. It is proven directly, real content included, in Replication and ISR Internals’s 3-broker cluster: {"type":"PARTITION_CHANGE_RECORD","version":2,"data":{"partitionId":0,"topicId":"...","isr":[1,2]}} and later {"...,"isr":[2,3],"leader":2} during that lesson’s ISR shrink and controller failover.
A few real records worth reading closely, in the order they were actually written at cluster startup:
baseOffset: 0 ... isControl: true ...| offset: 0 ... LeaderChange: {"version":0,"leaderId":1,"voters":[{"voterId":1}],"grantingVoters":[{"voterId":1}]}The very first record in the log is a control record, LeaderChange — this is Raft’s own leader-election proof, not application metadata; it exists because __cluster_metadata is itself replicated by the same Raft mechanism the controllers use to agree on everything else, so even this log’s own leadership gets recorded in this log.
| offset: 2 ... payload: {"type":"FEATURE_LEVEL_RECORD","version":0,"data":{"name":"metadata.version","featureLevel":30}}| offset: 3 ... payload: {"type":"FEATURE_LEVEL_RECORD","version":0,"data":{"name":"eligible.leader.replicas.version","featureLevel":1}}| offset: 8 ... payload: {"type":"CONFIG_RECORD","version":0,"data":{"resourceType":4,"resourceName":"","name":"min.insync.replicas","value":"1"}}| offset: 10 ... payload: {"type":"REGISTER_CONTROLLER_RECORD","version":0,"data":{"controllerId":1,"incarnationId":"...","endPoints":[{"name":"CONTROLLER","host":"localhost","port":29093,...}],"features":[...]}}| offset: 11 ... payload: {"type":"REGISTER_BROKER_RECORD","version":4,"data":{"brokerId":1,"incarnationId":"...","brokerEpoch":11,"endPoints":[...],"fenced":true,"logDirs":[...]}}eligible.leader.replicas.version with featureLevel:1 here is the same KIP-966 finalized-feature fact Replication and ISR Internals proved with kafka-features.sh describe — here it is the literal metadata-log record that makes it true, not just a query result reporting it. REGISTER_BROKER_RECORD’s "fenced":true is real too: every broker starts fenced (unable to lead partitions) until the controller confirms it has caught up on metadata — a BROKER_REGISTRATION_CHANGE_RECORD with "fenced":-1 (a sentinel for “unset/false” in this record’s encoding) follows moments later, unfencing it.
Later, creating a topic produces exactly the two record types you’d expect:
{"type":"TOPIC_RECORD","version":0,"data":{"name":"internals-pipeline-demo","topicId":"FQwkK_oVSZeJP5dIDq-bSw"}}{"type":"PARTITION_RECORD","version":2,"data":{"partitionId":0,"topicId":"FQwkK_oVSZeJP5dIDq-bSw","replicas":[1],"isr":[1], "removingReplicas":[],"addingReplicas":[],"leader":1,"leaderEpoch":0,"partitionEpoch":0,"directories":["..."]}}One TOPIC_RECORD for the name/ID mapping, one PARTITION_RECORD per partition carrying the exact same replicas/isr/leader fields kafka-topics.sh --describe reads back out — that CLI output is not a separate source of truth, it is this record, formatted.
Snapshots: config proven, a snapshot itself not (honestly)
Section titled “Snapshots: config proven, a snapshot itself not (honestly)”/opt/kafka/bin/kafka-configs.sh --bootstrap-server $BOOTSTRAP \ --describe --entity-type brokers --entity-name 1 --all 2>&1 | grep "metadata.log.max"Real output:
metadata.log.max.record.bytes.between.snapshots=20971520 sensitive=false synonyms={DEFAULT_CONFIG:metadata.log.max.record.bytes.between.snapshots=20971520}metadata.log.max.snapshot.interval.ms=3600000 sensitive=false synonyms={DEFAULT_CONFIG:metadata.log.max.snapshot.interval.ms=3600000}20971520 bytes is 20 MiB — a snapshot (a compacted point-in-time image of the whole metadata state, letting a restarting or newly-joined controller/broker catch up without replaying the entire log from offset 0) is triggered once at least this many bytes of new records have accumulated since the last one, or metadata.log.max.snapshot.interval.ms (1 hour by default) has passed, whichever comes first. This module’s own cluster never wrote anywhere near 20 MiB of metadata records in its short lifetime, so ls /tmp/kafka-logs/__cluster_metadata-0/*.checkpoint on this cluster genuinely returns nothing — no synthetic proof is substituted here; a real snapshot file (named <offset>-<epoch>.checkpoint) would appear once that byte or time threshold is crossed, and this lesson is honest that reaching it wasn’t practical within this proof session’s scope.
How brokers fetch metadata: this cluster is combined-mode, honestly
Section titled “How brokers fetch metadata: this cluster is combined-mode, honestly”Every “broker” in this module’s compose is process.roles=broker,controller — a combined node, the same combined mode KRaft & the Cluster already introduced. In combined mode, the broker and controller roles run in the same JVM process, so there is no separate-process FetchRequest for this specific cluster to show in its request log — the broker side simply reads the controller’s in-memory metadata log state directly, no RPC needed. On an isolated cluster (dedicated controller-only and broker-only nodes — the production-recommended topology per KRaft & the Cluster), a broker genuinely does issue a FetchRequest against the controller quorum’s leader to replicate __cluster_metadata, using the same FETCH API key data replication uses. This module’s own 3-broker combined cluster (Replication and ISR Internals) still showed something adjacent and real: raft-client-2/raft-client-3 issuing FETCH requests with clientId naming the Raft client explicitly and replicaState.replicaId matching the controller ID — proof that the underlying replication mechanism for __cluster_metadata is the same FetchRequest/FetchResponse pair as topic data replication, just addressed to the controller quorum’s Raft leader instead of a partition leader:
"requestHeader":{"requestApiKey":1,"clientId":"raft-client-3","requestApiKeyName":"FETCH"},"request":{"clusterId":"5L6g3nShT-eMCtK--X86sw","replicaState":{"replicaId":3,"replicaEpoch":-1}, "topics":[{"topicId":"AAAAAAAAAAAAAAAAAAAAAQ","partitions":[{"partition":0,"fetchOffset":55,...,"highWatermark":55}]}]},"response":{...,"currentLeader":{"leaderId":1,"leaderEpoch":1},...}}"topicId":"AAAAAAAAAAAAAAAAAAAAAQ" here is __cluster_metadata’s own fixed topic ID (base64 of a well-known UUID, not a randomly generated one like a normal topic gets) — this is the metadata log being fetched through the identical mechanism as any other partition.
kafka-metadata-quorum.sh describe --replication: the real fields
Section titled “kafka-metadata-quorum.sh describe --replication: the real fields”/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server $BOOTSTRAP describe --replicationReal output on this module’s own single combined node:
NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status1 AAAAAAAAAAAAAAAAAAAAAA 1227 0 1790650194864 1790650194864 LeaderOn Replication and ISR Internals’s real 3-node quorum, the same command right after broker 1 was stopped shows exactly what a lagging/unreachable voter looks like from the quorum leader’s own point of view:
NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status2 AAAAAAAAAAAAAAAAAAAAAA 342 0 1790650466818 1790650466818 Leader1 AAAAAAAAAAAAAAAAAAAAAA -1 343 -1 -1 Follower3 AAAAAAAAAAAAAAAAAAAAAA 342 0 1790650466611 1790650466611 FollowerLogEndOffset:-1 and LastFetchTimestamp:-1 for broker 1 (real sentinel values, not zero) mean “this voter has not fetched at all since it became unreachable” — distinct from Lag:343, which is how far behind the new leader’s own end offset broker 1’s last-known position is. Status: Leader/Follower here describes each node’s role in the metadata quorum specifically — a node can simultaneously be a data-partition follower and the metadata quorum’s leader; the two Raft-like elections (per-partition leader, controller-quorum leader) are independent.
Controller failover, in full
Section titled “Controller failover, in full”Replication and ISR Internals already proved the headline number — a new controller leader elected and serving within about 4.2 seconds of stopping the old one, LeaderEpoch incrementing 1 → 2. The metadata-log-level view of that same event: the moment broker 2 becomes the new quorum leader, it writes a fresh LeaderChange control record (the same record type opening this lesson) into __cluster_metadata at its new leader epoch, exactly like the very first record this cluster ever wrote at startup. Every other controller (and, in an isolated topology, every broker) learns of the new leader by replicating that one record, via the same FetchRequest mechanism shown above — there is no separate “leader announcement” protocol distinct from the metadata log itself.
Mental Model
Section titled “Mental Model”__cluster_metadata is Kafka replicating Kafka: the same append-only log, the same FetchRequest-driven replication, the same kafka-dump-log.sh tool, applied one level up to the data that describes every other topic instead of application data. A snapshot is this log’s equivalent of compaction’s tombstone-collapsing — a point-in-time summary that lets a new or recovering member skip replaying history it doesn’t need. Every fact a broker or client ever learns about cluster state — which broker leads which partition, what the ISR is, what features are finalized — ultimately traces back to one specific record at one specific offset in this one log, provable with the same decoder used throughout this lesson.
Trade-offs
Section titled “Trade-offs”| Choice | Benefit | Cost |
|---|---|---|
| Combined mode (this module’s cluster) | Simpler to run — one process per node, proven throughout this course’s running-kafka-locally setup | No separate broker→controller FetchRequest to observe (proven above); production topologies favor isolated mode for this and other reasons |
| Isolated mode (dedicated controllers) | Controller quorum isolated from data-plane load; the FetchRequest this lesson describes is real and observable | More processes/nodes to run and monitor |
Low metadata.log.max.record.bytes.between.snapshots | Faster catch-up for a new/recovering node (less log to replay) | More frequent snapshot-writing overhead |
High metadata.log.max.snapshot.interval.ms | Less snapshot overhead on a quiet cluster | A recovering node with an old snapshot replays more of the log to catch up |
Common Misconceptions
Section titled “Common Misconceptions”- “The controller keeps cluster state in a database, separate from Kafka’s own log mechanism.” Proven above: it is
__cluster_metadata, an ordinary Kafka-style log, decoded with the samekafka-dump-log.shused on any topic. - “
kafka-metadata.shis the tool for inspecting the metadata quorum.” Corrected above: that binary doesn’t exist on 4.3.1; the real tools arekafka-metadata-quorum.sh(used throughout this lesson) andkafka-metadata-shell.sh. - “A snapshot is a special, separate backup mechanism operators must configure.” Proven above: it’s config-driven (
metadata.log.max.record.bytes.between.snapshots/...interval.ms) and automatic — this lesson is upfront that it did not observe one directly simply because this cluster’s short lifetime never crossed either threshold. - “A broker fetching metadata uses a different protocol from normal replication.” Proven above with a real request-log line: the Raft client’s
FETCHrequest is the identicalFETCHAPI key, just addressed to__cluster_metadata’s fixed topic ID instead of a data topic’s.
Reading AI-generated code
Section titled “Reading AI-generated code”🐛 Spot the bug
# AI-suggested "faster recovery" runbook step# "Delete __cluster_metadata's snapshot files to force a clean resync# from the beginning of the log -- guarantees consistency."rm /tmp/kraft-combined-logs/__cluster_metadata-0/*.checkpointShow the answer
Two real problems, both provable from this lesson. First, the path is wrong for this image’s actual default log.dirs (/tmp/kafka-logs, proven throughout this course and this module — not /tmp/kraft-combined-logs, a path from older Kafka example configs this image doesn’t use unless log.dirs is explicitly set otherwise). Second, and more seriously: a snapshot is an optimization, not a source of truth — deleting it does not “force a clean resync,” it just means the next restart replays more of the log (or, if the log itself has already been trimmed relying on that snapshot existing, replays from a point that may no longer have full history). The actual, safe way to force a from-scratch broker rejoin is documented broker decommission/rejoin procedure, not manual file deletion inside log.dirs.