Retention & Compaction

The idea in one sentence

Kafka does not keep data forever by default — a topic’s cleanup policy decides whether old records are deleted after a retention window, or compacted so that only the latest value per key survives.

Retention: delete by time or size

With the default policy cleanup.policy=delete, Kafka removes old log segments once they cross a threshold:

# Keep records for 7 days (whichever limit hits first)
cleanup.policy=delete
retention.ms=604800000
# ...or cap the partition at 50 GB
retention.bytes=53687456000

Retention is enforced on whole segments, not individual records, and it is about age or size — not about whether anyone has consumed the data. A consumer that falls further behind than the retention window will miss records that age out. This is the right policy for event streams where old events lose value: clickstreams, logs, metrics.

Compaction: keep the latest value per key

Set cleanup.policy=compact and Kafka changes strategy entirely. Instead of dropping old data by age, it keeps the most recent record for each key and garbage-collects the older values for that key over time:

before compaction:  (a,1)(b,1)(a,2)(c,1)(b,2)(a,3)
after compaction:   (c,1)(b,2)(a,3)     ← latest value per key survives

This turns a topic into a durable, replayable snapshot of current state by key — perfect for changelogs, configuration, and the state stores Kafka Streams keeps. A brand-new consumer can read the whole compacted topic and rebuild the latest state for every key.

flowchart TB
  topic["Topic cleanup.policy"] --> del["delete: drop segments past retention.ms / retention.bytes"]
  topic --> comp["compact: keep latest record per key, GC older values"]
  del --> use1["event streams: logs, clicks, metrics"]
  comp --> use2["state/changelog: config, snapshots, Streams stores"]

Delete drops old segments; compact keeps the latest per key

Tombstones: deleting a key

Under compaction, how do you delete a key entirely? Write a tombstone — a record with the key and a null value. Compaction treats it as “this key is gone,” retains it long enough for all consumers to observe the deletion, then removes both the tombstone and the key’s history.

// Tombstone: null value tells a compacted topic to forget this key
await producer.send({
  topic: 'user-config',
  messages: [{ key: 'user-42', value: null }],
})

You can also combine policies with cleanup.policy=compact,delete to compact by key and still age out very old data.

Choosing a policy

delete — the record is the point; history matters, current-per-key does not. Event streams.
compact — the key’s latest value is the point; you want a rebuildable snapshot. State, changelogs, config topics.