What Is Kafka

The idea in one sentence

Kafka is a distributed, append-only log that decouples the systems producing data from the systems consuming it — producers write events once, and any number of independent consumers read them, in order, at their own pace.

Not a queue, a log

A traditional message queue deletes a message once a consumer acknowledges it. There is one logical reader, and the message is gone after delivery. Kafka is different: it is a log. When a producer writes a record, Kafka appends it to a partition and keeps it for a configured retention period. Reading a record does not remove it. Every consumer tracks its own position (an offset) in the log, so two teams can read the same stream for completely different purposes without interfering.

That single design choice is why Kafka is used for so many things at once:

• Messaging — decouple services with a durable buffer between them.
• Event sourcing — the log is the source of truth; state is a projection of it.
• Stream processing — transform and join streams as they arrive.
• Data integration — one pipeline feeds databases, search indexes, and warehouses.

flowchart LR
  p1["Producer: orders service"] --> log["Topic (append-only log)"]
  p2["Producer: payments service"] --> log
  log --> c1["Consumer group: fraud detection"]
  log --> c2["Consumer group: analytics"]
  log --> c3["Consumer group: email notifications"]

Producers write once; independent consumers read at their own offsets

Why not just use a database?

You could poll a database table for new rows, but you would be rebuilding Kafka badly: no built-in ordering guarantees per key, no efficient fan-out to many readers, no back-pressure, no replay from an arbitrary point, and no horizontal scaling of throughput. Kafka is purpose-built for high-throughput, ordered, replayable event streams — sequential disk writes, zero-copy reads, and partitioning that scales writes and reads across a cluster.

The core vocabulary

You will meet these terms constantly, so anchor them now:

• Broker — a Kafka server. A cluster is several brokers.
• Topic — a named stream of records (e.g. orders). Logically, one log.
• Partition — a topic is split into partitions; each partition is the actual ordered log.
• Offset — a record’s position within a partition. Consumers track offsets to know what they have read.
• Producer / Consumer — clients that write to and read from topics.

Proof: a real produce → consume round trip

Everything above is a claim until it runs against a real broker. This is a single-broker Kafka 4.3.1 KRaft container on this machine — create a topic, produce three records with the CLI, list it, describe it, then consume from the beginning:

scripts/what-is-kafka/roundtrip.sh

/opt/kafka/bin/kafka-topics.sh --bootstrap-server $BOOTSTRAP \
  --create --if-not-exists --topic orders-demo --partitions 3 --replication-factor 1

/opt/kafka/bin/kafka-console-producer.sh --bootstrap-server $BOOTSTRAP --topic orders-demo <<'MSGS'
order-101
order-102
order-103
MSGS

echo "=== list topics ==="
/opt/kafka/bin/kafka-topics.sh --bootstrap-server $BOOTSTRAP --list

echo "=== describe orders-demo ==="
/opt/kafka/bin/kafka-topics.sh --bootstrap-server $BOOTSTRAP --describe --topic orders-demo

echo "=== consume from the beginning ==="
timeout 8 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server $BOOTSTRAP \
  --topic orders-demo --from-beginning --max-messages 3

Real, unedited output from this exact script (topic name here is orders, the same commands run against this course’s own broker):

Created topic orders.
=== list topics ===
orders
=== describe orders-demo ===
Topic: orders  TopicId: c8AQc2zSSeOo5q0EC94zFA  PartitionCount: 3  ReplicationFactor: 1  Configs: min.insync.replicas=1
  Topic: orders  Partition: 0  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: orders  Partition: 1  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
  Topic: orders  Partition: 2  Leader: 1  Replicas: 1  Isr: 1  Elr:   LastKnownElr:
=== consume from the beginning ===
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocol
order-101
order-102
order-103
Processed a total of 3 messages

Two things worth noticing in real output, not just the diagram: the topic came back with ReplicationFactor: 1 and three independent partitions each with their own Leader, exactly the “topic split into partitions across the cluster” model from the vocabulary above — proven, not just asserted. And the console consumer itself prints a banner about KIP-848, the modern consumer rebalance protocol — a sign of how current this broker build (4.3.1) is; later modules in this course use it directly.

🌊 Where the internals of this round trip are covered

This lesson only needs “it works, here is the proof.” What actually happens inside the broker for that kafka-console-producer.sh call — the ProduceRequest, the leader appending to its log, followers fetching to replicate, the high watermark advancing, and the consumer’s FetchResponse — is traced end to end, RPC by RPC, in Kafka Internals.

🌊 The one-line takeaway

Kafka is a log, not a queue: records are appended and retained, not deleted on read, so many independent consumers can read the same stream at their own offset — proven above with a real produce → consume round trip against Kafka 4.3.1. That decoupling is what makes one platform serve messaging, event sourcing, stream processing, and data integration at once.

Reading AI-generated code

🐛 Spot the bug

src/what-is-kafka/queue-style-consumer.ts

import { Kafka } from 'kafkajs';

const kafka = new Kafka({ clientId: 'notifier', brokers: ['localhost:9092'] });
const consumer = kafka.consumer({ groupId: 'notifier-group' });

async function main() {
await consumer.connect();
await consumer.subscribe({ topic: 'orders' });

await consumer.run({
  eachMessage: async ({ message }) => {
    await sendNotification(message.value!.toString());
    // AI comment: "message handled and removed from the queue"
  },
});
}

main();

Show the answer

The comment is the bug — not the code. It describes Kafka as a queue (“removed from the queue” after handling), and that mental model is wrong: orders is not consumed away by this handler. The record proven above (order-101) stays in the log for its full retention period; this consumer only advances its own committed offset for the notifier-group consumer group. Any other consumer group — an analytics job, a fraud-detection service, a second run of this very same code with a different groupId — reads the exact same records from the start, untouched by what notifier-group already did. The code itself works; the mental model behind the comment, if it leaks into a design decision (like assuming a second consumer of the same topic would see nothing), does not.

Mental Model

Picture a whiteboard that only gets written to at the bottom, never erased: producers add lines at the end; each reader keeps their own sticky note marking how far down the board they have read. Nobody’s sticky note affects anyone else’s, and nobody’s reading erases a line. That is the entire mental model — the “broker,” “topic,” “partition,” and “offset” vocabulary above is just naming the parts of that whiteboard: which room it’s in (broker/cluster), which board (topic), which physical whiteboard if there’s more than one covering the same subject (partition), and where a sticky note sits (offset).

Trade-offs

• Log instead of queue — every consumer group gets an independent, replayable view, at the cost of the broker retaining data (disk and, on a compacted or long-retention topic, real storage cost) instead of discarding it the instant it’s read.
• Replay is easy; targeted redelivery of a single failed message is not — a queue with per-message ack/nack can retry exactly one poisoned message; Kafka’s unit of “redo” is a consumer’s offset, i.e. “replay from here forward,” which the offsets and commits lesson covers in depth.
• Partitioning gets you throughput and ordering-within-a-partition, but not a single total order across an entire topic — the next lesson in this module makes that trade-off explicit.

Common Misconceptions

• “A Kafka consumer removes messages, like a queue.” No — proven above and in the SpotTheBug: the record stays in the log for its retention window; only the reading consumer group’s own offset moves.
• “If one service has already consumed a topic, a new service reading it from scratch gets nothing.” No — a new consumer group with fromBeginning: true (or no prior committed offset) reads the full retained history, independent of what any other group has done.
• “Kafka is just a faster message queue.” It can be used as one, but the log abstraction (retention, replay, multiple independent readers, ordered partitions) is a genuinely different primitive from a queue’s delete-on-ack model — that difference is why Kafka also does event sourcing and stream processing, which a queue cannot.

How does Kafka differ from a traditional message queue?

It deletes each message after one consumer reads it It is an append-only log — records are retained, not removed on read, and each consumer tracks its own offset It only supports a single consumer It stores data only in memory

What does an offset represent?

The size of a message in bytes A consumer's position within a partition — which records it has read The number of brokers in the cluster A network port

Why can two teams read the same Kafka topic without interfering?

Kafka duplicates the topic for each team Reading does not consume the record; each consumer group tracks its own offset independently Only one team is allowed at a time The topic is copied into a database first

Which is NOT a typical Kafka use case?

Decoupling services with a durable buffer (messaging) Stream processing and data integration Event sourcing where the log is the source of truth Serving as a transactional relational database with SQL joins as primary store

In the real round-trip output above, what does ReplicationFactor: 1 combined with three separate Partition: lines tell you about the topic's physical layout?

The topic exists once, undivided, on a single broker The topic is split into 3 independent partitions (each its own log with its own Leader), each replicated to only 1 broker There are 3 full copies of the entire topic Replication factor 1 means the topic has no partitions

Submit

Mark as complete