The Idempotent Producer

The idea in one sentence

An idempotent producer tags every record with a producer id, epoch, and per-partition sequence number so the broker can discard duplicates created by retries — giving exactly-once delivery per partition.

The duplicate problem retries create

Without idempotence, a retry can silently duplicate a record. The producer sends a record, the broker writes it, but the acknowledgement is lost on the way back. The producer times out, retries, and the broker writes the same record a second time. You now have two copies and no way to tell them apart.

Idempotence fixes this: the broker assigns each producer a producer id (PID) and an epoch, and the producer stamps every record with a monotonically increasing sequence number per partition. When a record arrives, the broker checks the sequence number: if it has already seen it for that PID and partition, it acknowledges but does not write the record again.

# The Java client: idempotence is on by default; shown explicitly here
enable.idempotence=true
# acks=all is required and applied automatically once idempotence is on
acks=all
# retries must be > 0 (defaults high) so the producer actually retries
retries=2147483647
# ordering is preserved only if in-flight requests per connection stays <= 5
max.in.flight.requests.per.connection=5

KafkaJS requires an explicit opt-in on the producer. It then enforces the equivalent requirements internally — you do not set acks or in-flight limits separately:

// idempotent: true implies acks: -1 (all) internally, and caps in-flight requests
const producer = kafka.producer({ idempotent: true, maxInFlightRequests: 5 })

What it requires, and what it guarantees

Idempotence is not free-standing — it needs a specific configuration to hold its guarantee:

acks=all — the record must be confirmed by all in-sync replicas.
retries > 0 — retries are what create duplicates in the first place; without them there is nothing to dedupe.
max.in.flight.requests.per.connection <= 5 — up to five in-flight batches are allowed while still preserving ordering, because the broker tracks sequence numbers and rejects anything out of order.

These are protocol-level requirements, not just Java-client trivia: in KafkaJS, passing idempotent: true makes the client apply the equivalent guarantees internally, which is why you do not configure acks yourself on an idempotent KafkaJS producer.

When those hold, you get exactly-once delivery per partition on retries: no matter how many times a record is retried, it is written once. The epoch is what makes this safe across producer restarts and network partitions — a newer epoch fences an older, possibly zombie, producer instance so stale writes are refused.

flowchart LR
  send1["Send seq=7 (PID 100, epoch 3)"] --> write["Broker writes seq=7"]
  write --> lostack["Ack lost on the way back"]
  lostack --> retry["Producer retries seq=7"]
  retry --> check["Broker: already saw seq=7 for this PID"]
  check --> dedupe["Acknowledge but do NOT write again"]

The broker dedupes a retried record by its sequence number

Idempotence is not transactions

This is a common mix-up. Idempotence guarantees a record is written once per partition despite retries — but that is the only thing it does. It does not make a multi-record, multi-partition write atomic, and it does not tie a producer write to a consumer offset commit.

That stronger guarantee — atomic writes across partitions and true exactly-once processing — is what transactions provide, built on top of the idempotent producer. The Delivery Semantics and Transactions module covers transactionalId, producer.transaction(), and txn.commit(). For now, hold the boundary: idempotence removes retry duplicates; transactions add atomicity.