Glossary
Kafka’s vocabulary spans several layers at once: a storage layer (log, segment, index, offset) that every other layer sits on top of; a coordination layer (KRaft, controller quorum, __cluster_metadata) that replaced ZooKeeper in Kafka 4.0; topics, partitions and replication (leader, follower, ISR, high watermark) that make the log durable and scalable; producers and consumers (acks, idempotence, transactions, consumer groups, rebalancing) that read and write it; and a processing/integration layer (share groups, Kafka Streams, Connect, Schema Registry) built on top of all of it. When a lesson uses a term you’re unsure about, or unsure whether it still holds for Kafka 4.3.1, look it up here — every definition below is sourced from a lesson that proved it against a real broker or a real client, not from memory.
| Term | Meaning | Lesson |
|---|---|---|
| acks | The producer setting for how many replicas must confirm a write: acks=0 (no wait, an empty response with no offset — proven to give zero delivery confirmation either way), acks=1 (leader only), or acks=all/acks=-1 (every in-sync replica, KafkaJS’s real default). | Acks and Durability |
| ACL (Access Control List) | A rule binding a principal, an operation (Read/Write/…), and a resource (topic, group, …). ACLs do nothing at all until an authorizer is configured — kafka-acls.sh --list on a broker with none throws a real SecurityDisabledException; once enabled, a principal with no grant gets a real TopicAuthorizationException on produce/consume. | Security and Production |
| acknowledgement mode (ACCEPT / RELEASE / REJECT) | How a share-group consumer disposes of a record it acquired: ACCEPT (the default) marks it done and it is never redelivered; RELEASE (“couldn’t finish, give it back”) increments the record’s delivery count on every release until group.share.delivery.count.limit (default 5) is hit; REJECT marks it permanently bad, consumed once and never redelivered. | Choosing a Processing Layer |
| apache/kafka vs apache/kafka-native | Apache ships two broker images: apache/kafka (JVM, ships every bin/*.sh CLI tool) and apache/kafka-native (GraalVM-compiled, ~7x faster to “Kafka Server started” on one measured Mac, but /opt/kafka/ contains only the compiled binary — no kafka-topics.sh, no CLI at all). | Running Kafka Locally |
| at-least-once | Every record is delivered one or more times, never lost. Achieved by committing the consumer offset after processing — proven with a real kill mid-gap: one record applied twice after a crash between the durable write and the commit. | Delivery Semantics |
| at-most-once | Every record is delivered zero or one times, never duplicated. Achieved by committing the offset before processing — proven with a real kill: 3 of 4 records survived, one permanently lost because the offset had already advanced past it. | Delivery Semantics |
| authorizer (StandardAuthorizer) | The broker component that actually enforces ACLs; Kafka ships with none configured by default. authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer is the real, KRaft-native authorizer (the ZooKeeper-era AclAuthorizer is gone) — super.users lists principals that bypass every ACL check. | Security and Production |
| auto.create.topics.enable | Broker default true: a produce/consume against a missing topic races a real creation (a transient UNKNOWN_TOPIC_OR_PARTITION, then success). Set to false (a static config, needs a restart) the same error repeats forever with the topic genuinely never created. | Brokers, Topics & Partitions |
| auto.offset.reset / share.auto.offset.reset | Where a client starts reading when it has no committed position. KafkaJS’s subscribe({ fromBeginning }) picks earliest (true) or latest (false, the default). A share group’s equivalent, share.auto.offset.reset (default latest), is a group-level broker config set with kafka-configs.sh --entity-type groups — passing the ordinary client auto.offset.reset property to a share consumer throws a real ConfigException. | Offsets & Commits |
| Avro / Protobuf / JSON Schema | The three serialization formats Schema Registry commonly manages: Avro (compact binary, the most common choice), Protobuf (Google’s schema language, strong cross-language tooling), and JSON Schema (human-readable). Not managed: a raw, undeclared format like plain CSV. | Schema Registry |
| Term | Meaning | Lesson |
|---|---|---|
| batch.num.messages | A librdkafka-only batch-size cap measured in record count rather than bytes, exposed by @confluentinc/kafka-javascript’s native surface — no equivalent name exists in KafkaJS or the plain Java client. | Batching and Compression |
| bootstrap server | The initial broker address a client connects to (--bootstrap-server, or brokers: [...] in a client config); from there the client discovers the full cluster and which broker leads each partition. Modern Kafka has no --zookeeper equivalent — a tool that still accepts that flag predates KRaft. | Brokers, Topics & Partitions |
| broker | A single Kafka server process; several brokers form a cluster. Each broker stores some of the data and serves reads/writes for the partitions it leads. | What Is Kafka |
| buffer.memory | The Java client’s producer-side send buffer size; when full, send() blocks (up to max.block.ms) as back-pressure. KafkaJS has no equivalent setting — it does not pre-buffer records ahead of send() the way the Java client does. | Batching and Compression |
| Term | Meaning | Lesson |
|---|---|---|
| changelog topic | The Kafka topic backing a Kafka Streams state store: every state-store update is also written to it, so a crashed instance restarting elsewhere rebuilds its local RocksDB state store by replaying the changelog. | Kafka Streams |
| cleanup.policy | The per-topic setting that picks a topic’s lifecycle: delete (the default — age/size-based segment deletion), compact (keep only the latest value per key), or compact,delete (both — a hard age ceiling on top of a rebuildable-by-key snapshot). | Retention & Compaction |
__cluster_metadata | The internal, KRaft-managed topic (partition 0) that is the cluster’s metadata log — every topic creation, leader election, and ISR change is an event appended to it and replicated across the controller quorum. kafka-dump-log.sh --cluster-metadata-decoder reads it directly, showing real typed records like TOPIC_RECORD, PARTITION_RECORD, FEATURE_LEVEL_RECORD, and REGISTER_BROKER_RECORD. | KRaft & the Cluster |
| cluster.id | The cluster’s identity, written into every broker’s meta.properties at format time. Confirmed on this course’s own image: a fresh apache/kafka container with no CLUSTER_ID env var set gets the same baked-in default every time — kafka-storage.sh random-uuid, run twice, genuinely differs, proving the container’s own default (not the tool) is what’s static. | Running Kafka Locally |
| co-partitioning | Giving two topics the same partition count and the same partitioner so that a shared key (e.g. orderId) always lands on the same partition number in both — the requirement Kafka Streams needs for a key-based join to see both sides of a join locally, with no cross-partition shuffle. | Keys & Partitioning |
| combined controller / isolated controller | A combined node runs process.roles=broker,controller (both jobs on one process — the shape every proof in this course’s foundations module used); an isolated controller runs process.roles=controller only, trading more processes to run for blast-radius isolation so client load can’t starve controller-quorum CPU/disk I/O. | KRaft & the Cluster |
| commit (offset commit) | The consumer’s record of “everything before this offset is done” — Kafka stores it as a real, decoded record in __consumer_offsets. Committing the record’s own offset instead of offset + 1 is a proven, reproducible bug: the last-processed record gets redelivered on restart. | Offsets & Commits |
| compaction | The cleaner-thread process that keeps only the most recent record per key on a compact topic, garbage-collecting older values. Proven cross-segment with real kafka-dump-log.sh before/after: a superseded value is only collected once the cleaner actually runs (governed by min.cleanable.dirty.ratio, default 0.5), and the active segment is never compacted, no matter how stale a key’s value there is. | Retention & Compaction |
| compatibility mode (BACKWARD / FORWARD / FULL) | The rule Schema Registry enforces before accepting a new schema version for a subject: BACKWARD (the default — new schema can read old data; upgrade consumers first), FORWARD (old schema can read new data; upgrade producers first), or FULL (both directions hold). A violating registration is rejected outright. | Schema Registry |
@confluentinc/kafka-javascript | Confluent’s actively maintained Node.js client, built on librdkafka. Ships two surfaces from one package: a KafkaJS-compatible promise API (.KafkaJS.Kafka) and the native librdkafka callback API (Producer/KafkaConsumer) using raw librdkafka config keys. Carries KIP-848 (group.protocol=consumer) support since librdkafka 2.12.0 (Oct 2025). | The Producer API |
| compression.type | The codec a batch is compressed with before sending: lz4, zstd, snappy, gzip, or none (the default). KafkaJS ships only gzip natively (verified from source) — Snappy/LZ4/ZSTD throw KafkaJSNotImplemented unless a separate codec package is installed and registered. Measured on realistic JSON: gzip cut a 500-record batch from 108,177 to 4,741 bytes (22.8x); on incompressible random bytes, lz4 was ~4% slower than no compression. | Batching and Compression |
| config source (DEFAULT_CONFIG / STATIC_BROKER_CONFIG / DYNAMIC_BROKER_CONFIG / DYNAMIC_TOPIC_CONFIG) | The real provenance tag kafka-configs.sh --describe --all prints in a config’s synonyms={...} — proven with a real three-layer chain: a topic’s own DYNAMIC_TOPIC_CONFIG wins over a broker’s DYNAMIC_BROKER_CONFIG, which wins over the built-in DEFAULT_CONFIG. A value being DEFAULT_CONFIG does not mean it’s eligible for dynamic --alter — auto.create.topics.enable and log.dirs both reject --alter with a real InvalidRequestException: Cannot update these configs dynamically. | Brokers, Topics & Partitions |
| connector (source / sink) | The reusable, config-only Kafka Connect unit: a source connector pulls from an external system (a database, a file) into Kafka; a sink connector pushes from Kafka to an external system (a search index, a warehouse). Deployed to a distributed Connect cluster over its REST API (POST /connectors). | Kafka Connect |
__consumer_offsets | The internal, compacted topic holding every committed offset, keyed by (group, topic, partition). Read directly with kafka-console-consumer.sh --formatter org.apache.kafka.tools.consumer.OffsetsMessageFormatter — the decoded value carries offset, leaderEpoch, and commitTimestamp. | Offsets & Commits |
| consumer | A client that reads records from one or more topics, in a consumer group, tracking its own position (offset) per partition. | What Is Kafka |
| consumer group | A set of consumers sharing a groupId; Kafka assigns each partition to exactly one member of the group at a time, so partition count is the ceiling on parallelism — a 7th consumer on a 6-partition topic sits fully idle. Different groups are independent, each tracking its own offsets over the full stream. | Consumer Groups & Rebalancing |
| controller | A broker acting as part of the KRaft metadata quorum, replicating the cluster’s metadata log; one is the elected leader (the active controller). Since Kafka 4.0 removed ZooKeeper, this quorum — not an external system — is where cluster shape lives. | KRaft & the Cluster |
| controller quorum | The Raft-replicated group of controller voters that agrees on the metadata log; kafka-metadata-quorum.sh describe --status reports a real LeaderId, CurrentVoters, and MaxFollowerLagTimeMs — proven on both a single-node quorum (a quorum of one) and a real 3-node quorum with one Leader and two Followers. | KRaft & the Cluster |
| controller.quorum.voters / controller.quorum.bootstrap.servers | controller.quorum.voters is the original, static voter list (1@broker1:9093,2@broker2:9093,...) fixed at startup — changing membership means editing config and restarting every controller. controller.quorum.bootstrap.servers is KIP-853’s dynamic alternative, confirmed as a real config key on Kafka 4.3.1 alongside real add-controller/remove-controller subcommands. | KRaft & the Cluster |
| converter | The Kafka Connect component that translates between Connect’s internal record format and the bytes on a topic — an AvroConverter serializes with a Schema Registry schema id; a JsonConverter writes plain JSON. This is where Schema Registry plugs into a Connect pipeline. | Kafka Connect |
| Term | Meaning | Lesson |
|---|---|---|
| dead-letter topic (DLQ) | A topic a poison message is produced aside to before its offset is committed past. Order proven to matter with a real kill: committing first, then producing to the DLQ, loses the record permanently on a crash in the gap (0 entries found anywhere); producing to the DLQ first and awaiting the ack, then committing, only ever risks a harmless duplicate DLQ entry on the same crash. | Error Handling & Retries |
| delete.retention.ms | How long a tombstone (a record with valueSize: -1, the real on-disk marker for a deleted key) survives after compaction first processes it — proven directly in kafka-dump-log.sh output: the tombstone outlives the value it superseded, specifically so a lagging consumer still sees the deletion before the marker itself disappears. | Retention & Compaction |
| delivery.timeout.ms | The Java/librdkafka client’s single wall-clock deadline (default 120000) covering the whole send lifecycle — batching, network, and every retry — from send() to acknowledgement. KafkaJS has no equivalent single deadline; it bounds retries by count and backoff (retries, initialRetryTime, maxRetryTime) instead, which only approximates a total time budget. | Batching and Compression |
| distributed mode (Connect) | Multiple Connect workers coordinating like a consumer group, with connector configs and offsets stored in Kafka topics — if a worker dies, its tasks rebalance onto survivors. This is the production mode, managed over a REST API; standalone mode is a single worker with local-file config, fine for a one-off demo. | Kafka Connect |
| Term | Meaning | Lesson |
|---|---|---|
| eachBatch | The KafkaJS handler that receives a whole batch at once and a heartbeat function you must call yourself for legitimately slow processing — proven live: 8 seconds of eachBatch work (longer than a 6-second sessionTimeout) produced no rejoin once a manual heartbeat() call sat partway through. | The Consumer API |
| eachMessage | The simplest KafkaJS consume handler, one record at a time. Heartbeats are sent synchronously right after eachMessage returns, in the same loop iteration — not on an independent timer — so a handler that blocks past sessionTimeout starves its own heartbeat and triggers a real rebalance (The coordinator is not aware of this member, re-joining the group). | The Consumer API |
| errors.deadletterqueue.* / errors.tolerance | The Connect sink-connector configs that route a record Connect cannot process to a real dead-letter topic instead of killing the task. errors.tolerance defaults to none — verified live against a running worker’s own /config/validate endpoint — under which a bad record kills the task outright regardless of what errors.deadletterqueue.topic.name is set to; only errors.tolerance=all activates the DLQ. Meaningful only on sink connectors — setting it on a source connector validates and does nothing. | Kafka Connect |
| exactly-once (EOS) | Every record affects the outcome once — no loss, no duplicates — reached in a consume-transform-produce loop by putting the consumer’s input offsets inside the producer’s transaction via sendOffsets. Proven end to end with a real SIGKILL mid-transaction (2 of 5 sends buffered, zero committed): the restart’s own fencing aborted the hanging transaction and reprocessed the whole batch, ending at 5 unique output records for 5 input records. | Exactly-Once Processing |
| exactly_once_v2 | The Kafka Streams processing.guarantee setting that wraps each read-process-write cycle — including the state-store changelog and consumer offsets — in a single Kafka transaction, so a retry never applies an update twice. | Kafka Streams |
| Term | Meaning | Lesson |
|---|---|---|
| fencing | The mechanism a transactionalId exists for: a second producer connecting with the same id bumps the transaction epoch on the broker, and the first (zombie) instance’s next commit() fails with a real KafkaJSProtocolError (“Producer attempted an operation with an old epoch…”, retriable: false) — its earlier send() calls had already succeeded locally (buffered), only the commit is rejected. | Transactions |
| FEATURE_LEVEL_RECORD / metadata.version | The __cluster_metadata record type pinning a feature’s version level (e.g. metadata.version itself, featureLevel: 30 in this course’s own broker), read live with kafka-dump-log.sh --cluster-metadata-decoder. kafka-features.sh describe reports the same feature-level information for a running cluster (e.g. share.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1). | KRaft & the Cluster |
| follower | A partition replica that does nothing but continuously fetch from the leader to stay identical; if the leader’s broker dies, a follower already in the ISR is promoted — proven live with a real kafka-topics.sh --describe showing Leader: 3 becoming Leader: 1 after broker 3 was stopped, no manual failover command. | Replication & ISR |
| Term | Meaning | Lesson |
|---|---|---|
| group coordinator | The broker-side component a consumer group’s members register with and rebalance through; under KIP-848’s consumer protocol it also computes assignment (a diff, not a full recompute) instead of leaving that to a client-elected group leader the way the classic protocol does. | Consumer Groups & Rebalancing |
| group.protocol | The client opt-in for which rebalance protocol to speak: classic (KafkaJS’s only option — it ships exactly one assignor, round-robin) or consumer (KIP-848, broker-driven, incremental — a top-level librdkafka config key on @confluentinc/kafka-javascript, not nestable under kafkaJS: {}; doing so throws a real KafkaJSError: The 'groupProtocol' property is not supported). Client-side default is classic even though the broker enables all protocols by default. | Consumer Groups & Rebalancing |
| group.share.delivery.count.limit | The broker-wide default (5) capping how many times a share group will redeliver a RELEASEd record before it becomes terminal — proven live: one record released repeatedly showed Delivery:1 through Delivery:5, then stopped being redelivered at all. | Choosing a Processing Layer |
| Term | Meaning | Lesson |
|---|---|---|
| health metrics (UnderReplicatedPartitions / OfflinePartitionsCount / ActiveControllerCount) | The non-negotiable JMX broker metrics: under-replicated partitions and offline partitions must both read zero, and there must be exactly one active controller across the cluster — any deviation is an immediate alerting signal, not just a dashboard number. | Monitoring & Consumer Lag |
| heartbeat | The signal a consumer group member sends to stay alive in the group. In KafkaJS’s classic protocol it is client-side, sequenced right after eachMessage/eachBatch returns — not on a background timer. Under KIP-848 the cadence moves to broker config (group.consumer.heartbeat.interval.ms, default 5000ms, vs classic’s client-side default 3000ms). | The Consumer API |
| high watermark (HW) | The offset up to which every in-sync replica has replicated a partition — the real boundary between “written” and “committed”: a consumer at the default isolation level never reads past it, and acks=all is really a promise about the HW advancing past your record. | Acks and Durability |
| Term | Meaning | Lesson |
|---|---|---|
| idempotent producer | A producer stamping every record with a producer id (PID), epoch, and per-partition sequence number so the broker can discard retry duplicates — exactly-once delivery per partition. Default true in the Java client since Kafka 3.0; KafkaJS defaults the other way (idempotent: false) and requires an explicit opt-in. | The Idempotent Producer |
| index (offset index) | The .index file mapping offset → byte position within a segment’s .log file, used for fast seeks. Proven genuinely sparse: 8 records with index.interval.bytes=4096 (the real default) produced exactly one index entry, not eight — a read binary-searches the sparse index for the nearest position at or before the target, then scans forward a short, bounded distance. | The Log Abstraction |
| index.interval.bytes | The config (default 4096) controlling how many bytes of log data must be appended before the sparse .index/.timeindex files get a new entry. Setting it far too high (proven with a 100 MiB SpotTheBug example against a 1 GiB segment.bytes) forces long linear scans on lookups — the opposite of the “reduce overhead” intent behind widening it. | The Log Abstraction |
| ISR (in-sync replica set) | The set of replicas fully caught up to the leader right now, eligible to become leader without data loss — proven live, not static: it shrinks within replica.lag.time.max.ms (default 30000ms) of a broker going quiet, and expands back once it rejoins. acks=all waits on whatever is currently in the ISR, which can be smaller than the topic’s nominal replication factor. | Replication & ISR |
| Term | Meaning | Lesson |
|---|---|---|
| JMX | The interface Kafka brokers publish hundreds of metrics through, scraped by tools like Prometheus’s JMX exporter or an APM. A healthcheck reusing the broker’s own CLI tool alongside JMX can break: kafka-broker-api-versions.sh starts its own short-lived JVM that inherits KAFKA_JMX_OPTS and tries to bind the same port the running broker already holds — proven with a real ExportException: Port already in use. | Monitoring & Consumer Lag |
| KafkaJS | The most widely deployed Node.js Kafka client (pinned 2.2.4 in this course) — and unmaintained since 2023-02-27 (verified via npm view kafkajs time.modified; its maintainer opened “Looking for maintainers” on GitHub). It still runs fine against Kafka 4.3.1, has no separate initTransactions() call, ships only gzip natively, has no linger.ms/batch.size, and never gained KIP-848 or share-group support. | The Producer API |
| key (message key) | The optional field on a record that decides its partition: the default partitioner hashes the key so the same key always lands on the same partition, giving per-entity ordering. Proven bit-for-bit compatible between KafkaJS’s DefaultPartitioner and the real Java kafka-console-producer.sh for identical keys. | Keys & Partitioning |
| KIP-848 | The next-generation, broker-driven consumer rebalance protocol, GA since Kafka 4.0. Moves assignment to the group coordinator and makes rebalances incremental — proven live: under classic, a membership change pauses every member (even one whose final assignment doesn’t change); under group.protocol=consumer, unaffected partitions keep consuming uninterrupted while only the moving ones pause. librdkafka reached GA support in v2.12.0 (Oct 2025); KafkaJS never implemented it. | Consumer Groups & Rebalancing |
| KIP-890 | ”Transactions Server-Side Defense,” finalized in Kafka 4.x — hardens the transactional protocol against edge cases like hanging transactions and zombie producers, enabled automatically once the 4.0 upgrade is finalized with no code change required on the client side. | Transactions |
| KIP-932 (Share Groups / Queues for Kafka) | The proposal delivering share groups — queue semantics natively on a Kafka topic, no separate broker. Confirmed GA in Kafka 4.2.0 via the release announcement and live on this course’s 4.3.1 broker (kafka-features.sh describe shows share.version FinalizedVersionLevel: 1, on by default); the 4.3 release adds the RENEW acknowledgement type and adaptive share-coordinator batching on top of that GA base. | Choosing a Processing Layer |
| KIP-1071 (Streams Rebalance Protocol) | The new broker-side group.protocol=streams rebalance protocol for Kafka Streams — per the Kafka 4.2.0 release announcement, GA “with a limited feature set”: sticky assignment only (no warm-up tasks, no rack-aware placement), no static membership, no on-the-fly topology changes, and migrating a group between the classic and new protocol is offline-only. | Kafka Streams |
| KRaft | Kafka’s built-in, Raft-based metadata quorum (KIP-500) that stores cluster metadata as a replicated log instead of relying on an external ZooKeeper ensemble. As of Kafka 4.0, KRaft is the only mode — ZooKeeper support and the --zookeeper CLI flag are both gone entirely, not just deprecated. | KRaft & the Cluster |
| KStream | A Kafka Streams view of a topic as an unbounded sequence of independent events — every record is a new fact, nothing is overwritten. KTable is the other half of the same stream-table duality: a table is the snapshot you get by replaying a stream of updates, and toStream() turns a table back into a stream of its changes. | Kafka Streams |
| KTable | A Kafka Streams view of a topic as a changelog interpreted as the latest value per key — each record upserts the row for its key. Built by a stateful operation like count()/aggregate(), and backed by a state store whose changelog topic makes it durable across a crash. | Kafka Streams |
| Term | Meaning | Lesson |
|---|---|---|
| lag (consumer lag) | log-end-offset − committed offset for a partition — the single most important application-level health signal, read with kafka-consumer-groups.sh --describe --group. Watch the trend: flat or falling is healthy, steadily rising is the earliest sign a consumer group cannot keep up. | Monitoring & Consumer Lag |
| leader | The one replica handling all reads and writes for a partition at any moment; the others (followers) only fetch from it. Leadership is spread round-robin across partitions so no single broker leads everything on a healthy cluster. | Replication & ISR |
| leader epoch / leader-epoch-checkpoint | The leader epoch is a monotonically increasing counter marking which leader was in charge as of which offset; leader-epoch-checkpoint is the real on-disk file recording it, used to detect and truncate divergent data after an unclean leader change — one of the five real files in every segment directory alongside .log, .index, .timeindex, and partition.metadata. | The Log Abstraction |
| LEO (log-end-offset) | The position of the newest record in a partition — the “numerator” in the consumer-lag formula (lag = log-end-offset − committed offset), and the same field kafka-metadata-quorum.sh describe --replication reports per voter for the controller quorum’s own Raft log (LogEndOffset). | Monitoring & Consumer Lag |
| linger.ms / batch.size | The Java/librdkafka batching knobs: linger.ms (default 0) is how long a producer waits for more records before sending a not-yet-full batch; batch.size (bytes) caps how big that batch can grow. KafkaJS has neither (verified from source/types) — it batches implicitly, whatever you pass to one send() call, with no timer-based accumulation at all. | Batching and Compression |
| listeners / advertised.listeners | listeners is where the broker binds (e.g. PLAINTEXT://:9092); advertised.listeners is the address it hands back to clients in every metadata response after the first. Getting the second one wrong produces a working bootstrap connection followed by a real getaddrinfo ENOTFOUND on the next hop, proven live. CONTROLLER is a third listener kind regular clients never dial directly — only broker processes acting as controllers use it. | Brokers, Topics & Partitions |
| log (append-only log) | Kafka’s core primitive: an ordered, immutable sequence of records where a new record is always appended at the tail and assigned the next integer offset, never rewritten in place — proven as real files on disk (.log segments) holding record batches, not an abstraction. | The Log Abstraction |
| log.retention.check.interval.ms | The broker’s own polling interval for time/size-based retention deletion — default 300000ms (5 minutes), confirmed both from kafka-configs.sh --describe and by a real InvalidRequestException proving it cannot be changed dynamically without a restart. | Retention & Compaction |
| log.segment.delete.delay.ms | How long after aging past retention a segment is actually, physically removed from disk — default 60000ms; segments first get renamed to .deleted (e.g. 00000000000000000000.log.deleted), confirmed live on this course’s own broker. | Retention & Compaction |
| Term | Meaning | Lesson |
|---|---|---|
| max.in.flight.requests.per.connection | Idempotence requires this at most 5 (not exactly 1) — the broker retains up to five batches per producer for sequence tracking while still preserving order. The Java client enforces this ceiling itself; KafkaJS does not (verified from source: idempotent: true only affects the retry default, never touches maxInFlightRequests, which stays null/unlimited unless you set it yourself). | The Idempotent Producer |
| metadata log | The replicated log KRaft’s controller quorum maintains — the literal mechanism behind “cluster shape is just another log”: every broker joining the cluster replays it from offset 0 to build its in-memory view, the same way a consumer replays a data partition. | KRaft & the Cluster |
| min.cleanable.dirty.ratio | The threshold (default 0.5) controlling how eligible a compacted partition is for the cleaner thread to run — set low (e.g. 0.01) the cleaner runs almost continuously, good for a demo or for topics where stale duplicate values must disappear fast, at the cost of constant cleaner I/O. | Retention & Compaction |
| min.insync.replicas | The minimum number of in-sync replicas that must be available for an acks=all write to be accepted. Proven with a genuine ISR shrink on a real 3-broker cluster to reject writes with the broker’s own NotEnoughReplicasException (“Messages are rejected since there are fewer in-sync replicas than required”) — but proven to not reject writes at all when the shortfall is structural (min.insync.replicas set above replication.factor itself, e.g. on a single broker); that case only shows up under --under-min-isr-partitions, never as a write-time error. | Acks and Durability |
| murmur2 / murmur2_random | The hash algorithm the default (Java-compatible) partitioner uses for a keyed record: hash(key) % numberOfPartitions. @confluentinc/kafka-javascript’s librdkafka layer’s real documented default is actually consistent_random (CRC32-based) — its .KafkaJS.Kafka compat class overrides this to murmur2_random specifically for Java/KafkaJS interoperability, proven with a live 3-way comparison landing on different partitions for consistent (CRC32) than for murmur2_random. | Keys & Partitioning |
| Term | Meaning | Lesson |
|---|---|---|
| num.partitions | The broker-wide default partition count applied when a topic is auto-created or created without an explicit --partitions value — real default 1 on this course’s own broker, visible in kafka-configs.sh --describe --all’s DEFAULT_CONFIG source tag. | Brokers, Topics & Partitions |
| offset | A record’s position within a partition — a monotonically increasing integer a consumer tracks to know what it has read. Reading a record never removes it; the offset is a cursor the consumer owns, never something the broker mutates on delivery. | What Is Kafka |
| Term | Meaning | Lesson |
|---|---|---|
| page cache | The OS-level cache Kafka relies on for read speed instead of maintaining its own large in-process cache — free RAM left for the page cache (not consumed by an oversized JVM heap) is what lets the kernel serve recent reads from memory. | Performance & Tuning |
| partition | The unit both of parallelism (partitions can live on different brokers, scaling writes/reads) and of ordering (Kafka guarantees order within a partition, never across a whole topic) — proven live: real-time-interleaved writes to two partitions were consumed as “all of one partition, then all of the other,” with zero relationship to write order across them. | Brokers, Topics & Partitions |
| partitioner (DefaultPartitioner) | The producer component turning a record’s key into a partition number, hash(key) % numberOfPartitions for a non-null key. Proven from KafkaJS’s own source: JavaCompatiblePartitioner is a deprecated alias for DefaultPartitioner — the murmur2 implementation switched at KafkaJS 2.0.0, silently reassigning every existing key’s partition for an app that upgrades without pinning Partitioners.LegacyPartitioner. | Keys & Partitioning |
partitionsConsumedConcurrently | The KafkaJS run() option letting up to N partitions’ eachMessage calls run genuinely at the same time — proven live: 3 partitions each with a 1-second handler finished in ~1 second total (not ~3) with partitionsConsumedConcurrently: 3, identical start timestamps across all three. | The Consumer API |
| PID / producer epoch / sequence number | The three fields an idempotent producer stamps on every record batch — real, on-disk data confirmed with kafka-dump-log.sh (producerId: 0 producerEpoch: 0 baseSequence: 0 ...), not an abstraction. The broker rejects a retry whose sequence it has already recorded for that (producerId, partition) instead of comparing payload content. | The Idempotent Producer |
| poll loop | KafkaJS’s internal consume loop, driven by consumer.run() — cooperative and single-threaded per partition slot: eachMessage/eachBatch for record N must return before the loop can heartbeat, commit, or fetch record N+1 on that slot; nothing interrupts a slow handler preemptively. | The Consumer API |
| producer | A client that builds a message (topic, optional key, value) and writes it to a partition via send(), which returns a Promise and completes asynchronously — never blocking until the record actually reaches the broker. | What Is Kafka |
| Term | Meaning | Lesson |
|---|---|---|
| quota | A per-client cap on bandwidth or request-rate that keeps one producer or consumer from starving the rest of the cluster of capacity — part of a real production-readiness checklist alongside ACLs, TLS, and replication. | Security and Production |
| read_committed / readUncommitted | The consumer isolation level: read_committed (KafkaJS default readUncommitted: false) never surfaces records from an open or aborted transaction; readUncommitted: true genuinely exposes them — proven live by deliberately aborting a transaction and reading both ways: the default consumer saw nothing, the readUncommitted: true one saw the ghost record. | Transactions |
| rebalance (classic vs KIP-848) | Redistributing partitions among a consumer group’s members when membership changes. Under classic, every member stops and rejoins on any change — proven live with a real, if short, pause even for a member whose final assignment ends up identical. Under KIP-848’s consumer protocol, only the moving partitions revoke; unaffected members’ partitions keep consuming through the whole rebalance. | Consumer Groups & Rebalancing |
| record batch (v2 format) | The real on-disk unit kafka-dump-log.sh shows one line of per batch: a header (baseOffset/lastOffset, producerId/producerEpoch/baseSequence, magic: 2 for the v2 format, crc) shared by every individual ` | offset: …` record line it contains. |
| replica | A copy of a partition kept on a broker — one leader replica handles all reads/writes; the rest are followers. Replicas: in kafka-topics.sh --describe lists every broker holding a copy, whether or not it’s currently caught up (that subset is the ISR). | Replication & ISR |
| replication factor | The number of copies Kafka keeps of a partition — --replication-factor 3 means the partition lives on three brokers. The default cluster-wide value (default.replication.factor) applies to auto-created topics; a mismatched min.insync.replicas above the actual replication factor is a real, silent misconfiguration trap. | Replication & ISR |
| RequestHandlerAvgIdlePercent | The kafka.server:type=KafkaRequestHandlerPool JMX MBean measuring how idle a broker’s request-handler thread pool is — near 1.0 (proven live at 0.97) means plenty of spare capacity; trending toward 0 means every handler thread is constantly busy and produce/fetch latency is about to climb. Read directly with JmxTool, alongside UnderReplicatedPartitions. | Monitoring & Consumer Lag |
| retention (retention.ms / retention.bytes) | The default cleanup.policy=delete age/size limits: retention.ms (default 168 hours) and retention.bytes (a 50 GiB cap is exactly 53687091200, not 53687456000, an off-by-364,800-bytes figure that shows up in some older notes). Enforced on whole segments, never individual records, and the active segment is exempt — a topic’s real on-disk size can exceed retention.bytes by up to one full segment at any moment. | Retention & Compaction |
| Term | Meaning | Lesson |
|---|---|---|
| SASL/SCRAM | An authentication mechanism whose credentials live in the cluster’s own KRaft __cluster_metadata log — no ZooKeeper, no external credential store needed. Provisioned dynamically with kafka-configs.sh --alter --entity-type users --add-config 'SCRAM-SHA-256=[password=...]'; only credential metadata like iterations=4096 is ever stored or shown back, never the plaintext password. | Security and Production |
| schema id (wire format) | The tiny integer a serializer gets back after registering a schema once; every record’s value is then prefixed with one magic byte, the 4-byte schema id, and the payload — the consumer reads the id, fetches (and caches) the matching schema from the registry, and deserializes. | Schema Registry |
| Schema Registry | A shared, versioned contract for record shape: producers register a schema and get back a schema id; consumers fetch the schema by id so both sides always agree on how to read the bytes — turning an implicit assumption about record shape into an explicit, enforced contract. | Schema Registry |
| segment (segment.bytes / segment.ms) | The unit a partition’s log is physically split into — one .log/.index/.timeindex file triple per segment, named by its base offset. segment.bytes has a real, enforced 1 MiB floor (anything smaller is rejected at topic-creation time); segment.ms forces a roll on a time basis regardless of size, and a segment cannot be deleted or have retention checked until it rolls (closes). | The Log Abstraction |
| sendOffsets | The KafkaJS transaction call (txn.sendOffsets({ consumerGroupId, topics })) that commits a consumer’s input offsets inside the producer’s transaction, atomic with the output records. Committing offsets separately instead reopens the exact crash-sized gap transactions exist to close — proven live: 3 input records became 6 duplicated output records after a crash between a separate commit and the transaction commit. | Transactions |
| session.timeout.ms / heartbeat.interval.ms | Under the classic protocol these are client-side KafkaJS consumer options (defaults 30000ms / 3000ms) — the client sets its own liveness cadence. Under KIP-848 they move to broker-controlled config (group.consumer.session.timeout.ms default 45000ms, group.consumer.heartbeat.interval.ms default 5000ms), verified directly against a running 4.3.1 broker. | Consumer Groups & Rebalancing |
| share group | A KIP-932 consumer group where multiple members can all read the same partition, each record acquired and acknowledged individually — no partition-per-consumer ownership, the native queue semantics of RabbitMQ/SQS on a plain Kafka topic. Proven live with kafka-console-share-consumer.sh across all three acknowledgement modes; neither KafkaJS nor @confluentinc/kafka-javascript speaks the share-group protocol (verified with a zero-match grep across both packages’ type definitions) — only the Java client’s KafkaShareConsumer and the CLI tools do today. | Choosing a Processing Layer |
| SMT (Single Message Transform) | An inline, no-deploy Kafka Connect record edit (transforms=..., e.g. InsertField$Value) — no separate job needed. Proven live to fail with a real DataException: Only Struct objects supported for [field insertion] against FileStreamSourceConnector’s plain-string values: SMTs like InsertField/ReplaceField require Struct-schema’d records (typically JDBC, Debezium, or Avro/JSON-Schema sources), not a schemaless or plain-string value. | Kafka Connect |
| sparse index | The design of Kafka’s .index/.timeindex files: one entry per index.interval.bytes of log data, not one entry per record — proven directly: 8 records produced exactly one index entry. This keeps the index small enough to binary-search cheaply while the resulting forward scan stays bounded. | The Log Abstraction |
| standalone mode (Connect) | A single Kafka Connect worker with connector config in a local file — simple, no fault tolerance, fine for a laptop demo. Real production use is distributed mode instead. | Kafka Connect |
| state store | A local key-value store (RocksDB by default) living inside a Kafka Streams app instance, holding the running result of a stateful operation (aggregate, count, reduce, join, windowing). Backed by a changelog topic so a crashed instance restarting elsewhere rebuilds it by replay, without you managing a separate database. | Kafka Streams |
| sticky partitioner (KIP-480) | The Java producer’s default (since Kafka 2.4) strategy for null-key records: batch-stick to one partition until it fills or linger.ms elapses, for fewer, larger batches. KafkaJS’s DefaultPartitioner never implemented this — proven from its own source and a live run (1, 2, 3, 4, 5, 0, 1, 2) that it instead round-robins null keys one partition per call. | Keys & Partitioning |
| subject (Schema Registry) | The key a schema is stored and versioned under, usually <topic>-value (or <topic>-key) — e.g. orders-value. A subject’s compatibility mode (BACKWARD/FORWARD/FULL) governs which new versions the registry will accept for it. | Schema Registry |
| Term | Meaning | Lesson |
|---|---|---|
| task (Connect) | The unit of parallelism a Kafka Connect connector is split into, up to tasks.max — workers spread tasks across a distributed cluster (e.g. one task per table, or per partition range). | Kafka Connect |
| timeindex | The sparse timestamp → offset mapping file (.timeindex) sitting alongside every segment’s .log/.index pair, powering time-based lookups and --offsets-for-times. Pre-allocated at log.index.size.max.bytes (default 10 MiB) regardless of how little is actually written — a real, if surprising, ls -la artifact. | The Log Abstraction |
| TLS / mTLS | Broker-side encryption (listeners=SSL://:9093, a keystore for the broker’s own identity, a truststore for who it trusts) that by itself only proves the broker’s identity to the client. ssl.client.auth=required turns this into mutual TLS, demanding and verifying a client certificate too; ssl.endpoint.identification.algorithm=https enables hostname verification against the certificate’s SAN. | Security and Production |
| tombstone | A record with valueSize: -1 and no payload — the real, on-disk marker that a key is deleted on a compacted topic, proven directly in kafka-dump-log.sh output. It is not removed the instant a newer write supersedes it; it survives at least delete.retention.ms so a lagging consumer still sees the deletion. | Retention & Compaction |
| topic | A named, logical stream of records (e.g. orders) that Kafka physically splits into partitions — one topic, several independent ordered logs underneath it. | What Is Kafka |
| TopologyTestDriver | An in-memory, broker-free Kafka Streams test harness — proven live running a real word-count topology, count() emitting 8 update records for 2 input lines via an auto-inserted repartition topic. Needs a Struct-free default serde or it throws immediately at construction; exercises the topology’s logic but not real partition assignment or co-partitioning enforcement. | Kafka Streams |
| transaction | A unit of work letting a producer write to multiple partitions (and commit consumer offsets) atomically — everything becomes visible together on commit(), or none of it does on abort(). Every commit/abort in this course is a real, readable record in __transaction_state, walked through the real state progression 1 → 2 → 4 (began → prepare → complete). | Transactions |
| transactional id | A stable identifier a transactional producer registers on connect() (KafkaJS has no separate initTransactions() call — it’s implicit), letting Kafka fence out a zombie instance the moment a replacement with the same id begins its own transaction. Requires idempotent: true and, in the Java client, max.in.flight.requests.per.connection <= 5. | Transactions |
| transaction.timeout.ms / transaction.max.timeout.ms | The producer’s requested transaction timeout (KafkaJS config key transactionTimeout) versus the broker’s real ceiling on it, transaction.max.timeout.ms (default 900000, 15 minutes) — exceeding it fails immediately at InitProducerId/producer.transaction(), not at commit time, with a real KafkaJSProtocolError whose message text swaps the key’s word order (max.transaction.timeout.ms) from the actual config key. | Transactions |
__transaction_state | The internal, compacted topic holding every transaction’s commit/abort record, readable with kafka-console-consumer.sh --formatter org.apache.kafka.tools.consumer.TransactionLogMessageFormatter — each record carries the transactionalId, producerId/producerEpoch (the exact pair fencing compares), and a numeric transactionStatus. | Transactions |
| Term | Meaning | Lesson |
|---|---|---|
| unclean leader election | Electing a stale, out-of-sync replica as leader when the ISR has dropped to zero — brings the partition back online but loses every record it never received. Kafka’s default, unclean.leader.election.enable=false, refuses that trade (confirmed live), choosing durability over availability; a partition whose whole ISR is down instead stays offline until an in-sync replica returns. | Replication & ISR |
| worker (Connect) | The JVM process that actually runs connectors and their tasks; one or more workers form a Connect cluster. In distributed mode, if a worker dies its tasks rebalance onto the survivors. | Kafka Connect |
| ZooKeeper (removed) | The external coordination system Kafka relied on before KRaft, storing cluster metadata in a separate ensemble that had to be deployed, secured, and tuned on its own. Fully removed as of Kafka 4.0 — not deprecated, gone: there is no ZooKeeper process to start, and --zookeeper is not even an accepted flag on this course’s Kafka 4.3.1 broker’s tools. | KRaft & the Cluster |