The Log Abstraction

The idea in one sentence

Everything Kafka does rests on one data structure — an append-only, totally-ordered log where each record gets a monotonically increasing offset, and reads are just a scan starting from an offset.

What a log actually is

A Kafka log is not a text file of debug lines. It is an ordered, immutable sequence of records. New records are appended to the end; existing records are never modified in place. Each record is assigned the next integer offset:

offset:   0      1      2      3      4      5   →  (append here)
record: [ev0]  [ev1]  [ev2]  [ev3]  [ev4]  [ev5]
                        ▲
              a consumer reading from offset 2

Because writes only ever go to the end, Kafka turns message storage into sequential disk I/O, which is dramatically faster than the random I/O a mutable store needs. There is no in-place update, no page to find and rewrite — just append and advance.

Proof: the actual files on disk

A partition’s log is not one abstract thing — it is a directory of real files, split into segments. Creating a one-partition topic and producing 8 keyed records to it, then listing that directory inside the broker container:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server $BOOTSTRAP \
  --create --if-not-exists --topic log-demo --partitions 1 --replication-factor 1 \
  --config index.interval.bytes=4096

/opt/kafka/bin/kafka-console-producer.sh --bootstrap-server $BOOTSTRAP --topic log-demo \
  --property parse.key=true --property key.separator=: <<'RECORDS'
k1:order-101 created
k2:order-102 created
k1:order-101 paid
k3:order-103 created
k2:order-102 shipped
k1:order-101 shipped
k3:order-103 paid
k3:order-103 shipped
RECORDS

echo "=== ls -la <log.dirs>/log-demo-0 ==="
ls -la /tmp/kafka-logs/log-demo-0

Real, unedited output — this course’s own broker’s actual log.dirs for this partition:

=== ls -la <log.dirs>/log-demo-0 ===
total 20
drwxr-xr-x  2 appuser appuser     4096 Sep 28 23:00 .
drwxr-xr-x 57 appuser appuser     4096 Sep 28 23:00 ..
-rw-r--r--  1 appuser appuser 10485760 Sep 28 23:00 00000000000000000000.index
-rw-r--r--  1 appuser appuser      263 Sep 28 23:00 00000000000000000000.log
-rw-r--r--  1 appuser appuser 10485756 Sep 28 23:00 00000000000000000000.timeindex
-rw-r--r--  1 appuser appuser        8 Sep 28 23:00 leader-epoch-checkpoint
-rw-r--r--  1 appuser appuser       43 Sep 28 23:00 partition.metadata

Five real files, one active segment (the log is small enough here that it never rolled to a second one):

00000000000000000000.log — the segment itself: the actual record bytes, named by its base offset (the first offset this segment holds — 0 here, hence the twenty zero-padded digits).
00000000000000000000.index — the offset index: a sparse mapping from offset → byte position within the .log file (proven below — it is not one entry per record).
00000000000000000000.timeindex — a sparse mapping from timestamp → offset, used by time-based lookups (kafka-console-consumer.sh --offset by timestamp, retention-by-time).
leader-epoch-checkpoint — records which leader epoch was in charge as of which offset, used to detect and truncate divergent data after an unclean leader change.
partition.metadata — the partition’s topic ID, so a broker can tell which topic a log directory belongs to even if the topic is later recreated.

Proof: dumping the segment — real record batch v2 fields

kafka-dump-log.sh --print-data-log decodes the actual bytes of .log:

/opt/kafka/bin/kafka-dump-log.sh --files /tmp/kafka-logs/log-demo-0/00000000000000000000.log --print-data-log

Real, unedited output for the exact 8 records produced above:

Dumping /tmp/kafka-logs/log-demo-0/00000000000000000000.log
Log starting offset: 0
baseOffset: 0 lastOffset: 7 count: 8 baseSequence: 0 lastSequence: 7 producerId: 1 producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 0 CreateTime: 1790636428911 size: 263 magic: 2 compresscodec: none crc: 3117226911 isvalid: true
| offset: 0 CreateTime: 1790636428907 keySize: 2 valueSize: 17 sequence: 0 headerKeys: [] key: k1 payload: order-101 created
| offset: 1 CreateTime: 1790636428911 keySize: 2 valueSize: 17 sequence: 1 headerKeys: [] key: k2 payload: order-102 created
| offset: 2 CreateTime: 1790636428911 keySize: 2 valueSize: 14 sequence: 2 headerKeys: [] key: k1 payload: order-101 paid
| offset: 3 CreateTime: 1790636428911 keySize: 2 valueSize: 17 sequence: 3 headerKeys: [] key: k3 payload: order-103 created
| offset: 4 CreateTime: 1790636428911 keySize: 2 valueSize: 17 sequence: 4 headerKeys: [] key: k2 payload: order-102 shipped
| offset: 5 CreateTime: 1790636428911 keySize: 2 valueSize: 17 sequence: 5 headerKeys: [] key: k1 payload: order-101 shipped
| offset: 6 CreateTime: 1790636428911 keySize: 2 valueSize: 14 sequence: 6 headerKeys: [] key: k3 payload: order-103 paid
| offset: 7 CreateTime: 1790636428911 keySize: 2 valueSize: 17 sequence: 7 headerKeys: [] key: k3 payload: order-103 shipped

This is exactly two layers, and both are real:

One line starting baseOffset: — this is a record batch (Kafka’s on-disk record format is v2, magic: 2): a header shared by every record it contains — baseOffset/lastOffset (the batch’s offset range, 0..7 here — all 8 records landed in a single batch because the console producer flushed them together), producerId/producerEpoch/baseSequence (idempotent-producer bookkeeping — idempotent-producer.mdx covers what these prevent), position: 0 (the batch’s own byte offset in the .log file), size: 263 (matching the file size from ls -la above, confirming this one batch is the entire file so far), and crc (a checksum over the batch).
One | offset: line per record inside that batch — each with its own offset, keySize/valueSize, and the decoded key/payload (only shown because --print-data-log was passed; without it, kafka-dump-log.sh prints just the batch headers).

Proof: the index is sparse, not one entry per record

The textbook claim is “the index makes lookups fast.” What it does not usually say precisely is how sparse it is. Dumping the .index file itself (no --print-data-log, since it is a different, binary offset→position format) and separately checking its internal consistency:

echo "=== index entries ==="
/opt/kafka/bin/kafka-dump-log.sh --files /tmp/kafka-logs/log-demo-0/00000000000000000000.index

echo "=== index sanity check ==="
/opt/kafka/bin/kafka-dump-log.sh --files /tmp/kafka-logs/log-demo-0/00000000000000000000.index --index-sanity-check

Real output:

=== index entries ===
Dumping /tmp/kafka-logs/log-demo-0/00000000000000000000.index
offset: 0 position: 0
Mismatches in :/tmp/kafka-logs/log-demo-0/00000000000000000000.index
  Index offset: 0, log offset: 7

=== index sanity check ===
Dumping /tmp/kafka-logs/log-demo-0/00000000000000000000.index
/tmp/kafka-logs/log-demo-0/00000000000000000000.index passed sanity check.

Exactly one index entry for 8 records — offset: 0, position: 0, the mandatory first entry every segment gets. That is the proof: this topic was created with index.interval.bytes=4096 (the config that controls how many bytes of log data must be appended before the index gets a new entry — the default is also 4096), and the entire segment is only 263 bytes, so the threshold was never crossed after the first entry. The “mismatch” line is the tool’s own diagnostic comparing the highest indexed offset (0) against the highest offset actually in the log (7) — not a corruption; it is exactly what a sparse index looks like on a small segment, and --index-sanity-check (which validates the index’s internal ordering and file-length invariants, not “one entry per record”) passes clean.

🌊 Why sequential appends AND a sparse index both matter for reads

A read for a specific offset does not scan the whole index looking for an exact match — it binary-searches the index for the largest indexed offset ≤ the target, seeks the .log file to that position, then scans forward record-by-record from there. A sparse index (one entry per index.interval.bytes, not per record) keeps the index itself small enough to memory-map and binary-search cheaply, while the final short linear scan stays cheap too because it never has to cross more than index.interval.bytes worth of records. This is also why appends being sequential matters for more than just write speed: the index and the log grow in lockstep at the tail, so both the write path and this lookup path stay simple, contiguous disk operations instead of random-access ones.

Reads are position-based, not destructive

A consumer reads by saying “give me records starting at offset N.” Kafka streams them in order using exactly the index → position → scan path proven above. The consumer’s offset is a cursor it controls, not something the broker mutates on delivery. This is the whole trick:

Move the cursor backward to replay — reprocess yesterday’s events after fixing a bug.
Move it forward to skip.
Keep two cursors (two consumer groups) reading the same log at different speeds.

# Read a topic from the very beginning — replaying all retained history
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders --from-beginning

flowchart LR
  prod["Producer"] -->|append| tail["log tail (next offset)"]
  subgraph log["Partition log: offsets 0..N"]
    tail
  end
  log --> curA["Group A cursor at offset 40"]
  log --> curB["Group B cursor at offset 12 (replaying)"]

Append at the tail; each consumer holds its own read cursor

Ordering is per-log

A single log gives you a total order: offset 4 always comes after offset 3. This ordering guarantee is the foundation for correctness — but it holds within one partition, not across a whole topic. The next module unpacks that distinction, because it shapes how you design keys and scale throughput. For now, hold the core idea: the log is ordered, append-only, and read by position — proven above down to the exact bytes.

Reading AI-generated code

🐛 Spot the bug

# section: topic
# "Reduce index overhead by writing fewer index entries"
index.interval.bytes=104857600
segment.bytes=1073741824

Show the answer

This looks like a reasonable performance tweak — larger index.interval.bytes (100 MiB here, versus the real default of 4096 proven above) means fewer index entries, so a smaller .index file. The bug is what that trade costs: the sparse index proven above already means every lookup ends with a linear scan forward from the nearest indexed position — set index.interval.bytes to 100 MiB on a segment.bytes of 1 GiB (the default, also shown here) and most lookups will only ever find the one entry at the start of the segment, forcing a scan through up to 100 MiB of records to find a target offset or timestamp. The real trade-off is index file size (smaller, cheaper to memory-map) against scan length on every read (longer) — this config takes it to an extreme that would show up as slow --from-beginning seeks and slow time-based lookups on any topic with real traffic, exactly the opposite of “reduce overhead.” The default 4096 proven above is already a considered balance; widening it should be a deliberate, measured choice per topic, not a blanket “fewer entries is always better.”

Mental Model

Think of the .log file as the actual ledger pages and the .index as a thin, hand-written table of contents someone jots down every few pages, not next to every line. To find “line 4200,” you check the table of contents for the closest page number at or before it, flip straight there, then read forward a little — you never scan the whole ledger, and the table of contents never has to be as big as the ledger itself. That is precisely what the sparse index proven above does, and it is why “append-only” and “sparse index” are not two separate features — the second only works efficiently because the first guarantees new data always lands at a predictable, growing tail.

Trade-offs

Sparse index (index.interval.bytes) — small index files, cheap to hold in memory and binary-search, at the cost of a short forward scan on every lookup (proven above: the default 4096 bytes keeps that scan short; the SpotTheBug example shows what happens when it is set far too high).
Segment size (segment.bytes) — larger segments mean fewer files and fewer index-file allocations, but a segment cannot be deleted or have its retention checked until it rolls (closes) — a segment that never fills stays open (and un-collectible by time-based retention within it) indefinitely, which is a real operational knob, not just a performance one.
Append-only, immutable records — this is what makes sequential I/O and the sparse index both possible, but it means “editing” a record does not exist; correcting bad data means writing a new, corrected record (and, for keyed topics, relying on compaction rather than in-place update).

Common Misconceptions

“The .index file has one entry per record, which is why lookups are fast.” No — proven above: 8 records produced a single offset: 0 index entry. Lookups are fast because the index is small enough to binary-search cheaply, and the resulting scan is short because index.interval.bytes bounds it — not because every record is individually indexed.
“A bigger .index/.timeindex file on disk (10 MiB shown above) means Kafka is storing a lot of index data.” No — proven above: both files are pre-allocated to log.index.size.max.bytes regardless of how much is actually used; the real, used portion here was a single entry.
“Kafka logs are just files you could tail -f and read.” No — proven above: .log holds binary record batches (v2 format) that need kafka-dump-log.sh (or a client’s own deserializer) to decode; raw cat/tail output is not meaningful.
“Sequential writes and the sparse index are two independent optimizations.” No — the index’s sparseness is only cheap to search because new records always append at a predictable, growing tail; if records could be written anywhere, a sparse offset→position index would not reliably find the right neighborhood to scan from.