Error Handling & Retries

The idea in one sentence

Robust Kafka code sorts every failure into one of three buckets — retriable (let the KafkaJS retry policy retry it), abortable (roll the transaction back and try the batch again), or fatal (disconnect the producer and let another instance take over) — and dead-letters the records that can never succeed.

Producer retries are automatic and bounded

KafkaJS retries retriable errors — leader elections, transient network blips — on its own, governed by the client’s retry policy: a bounded number of retries with exponential backoff (each wait starts at initialRetryTime and grows up to maxRetryTime). You set it once on the Kafka client, and it covers producing, consuming, and admin calls.

const kafka = new Kafka({
  clientId: 'orders-app',
  brokers: ['localhost:9092'],
  retry: {
    retries: 8,             // bounded automatic retries for retriable errors
    initialRetryTime: 300,  // first backoff in ms; grows exponentially
    maxRetryTime: 30000,    // cap on any single backoff
  },
})

const producer = kafka.producer({ idempotent: true })  // keeps those retries duplicate-free

With an idempotent: true producer, these retries never create duplicates in the partition, so you can retry aggressively without corrupting the stream.

Abortable vs fatal in a transaction

Inside a transactional loop, KafkaJS throws when an operation fails, and the thrown error’s retriable flag decides the recovery:

retriable / abortable (e.retriable === true) — the transaction cannot commit but the producer is still healthy. Call txn.abort() and retry the batch from the last committed offsets.
fatal (e.retriable === false) — a newer producer has claimed your transactionalId (a zombie was fenced) or the idempotent sequence is broken. You cannot recover this producer; disconnect it and shut down so a healthy instance continues.

const txn = await producer.transaction()
try {
  for (const message of batch.messages) {
    await txn.send({ topic: 'orders-enriched', messages: [{ key: message.key, value: transform(message) }] })
  }
  await txn.sendOffsets({ consumerGroupId: 'enricher', topics: offsetsFor(batch) })
  await txn.commit()
} catch (e) {
  await txn.abort()                  // ABORTABLE: roll back and retry the batch
  if (!e.retriable) {                // FATAL: a zombie was fenced — give up this instance
    await producer.disconnect()
    throw e
  }
}

Poison messages, dead-letter topics, and the safety net

A poison message is a record the consumer can never process — malformed payload, an unparseable schema, a permanently failing downstream. Retrying it forever stalls the whole partition behind it. The standard fix is a dead-letter topic (DLT): after N failed attempts, produce the record plus error context to a separate orders-dlq topic, commit past it, and keep the main flow moving. A human or a separate job inspects the DLT out of band.

try {
  await process(message)
} catch (e) {
  // Route the bad record aside instead of blocking the partition
  await dlqProducer.send({ topic: 'orders-dlq', messages: [{ key: message.key, value: message.value }] })
}
// commit past the record either way, so the partition keeps advancing

Underneath all of this, idempotent consumers are the safety net. Because at-least-once means a record can be redelivered after any crash, make your processing safe to repeat — an UPSERT keyed by an event id, a conditional write, a dedupe table. Then a retry, an aborted-and-retried batch, or a redelivery after a fenced producer all converge on the same correct state.

flowchart TB
  fail["failure during send or process"] --> retriable{"which kind?"}
  retriable -->|"transient"| retry["KafkaJS retries per its retry policy"]
  retriable -->|"abortable (e.retriable)"| abort["txn.abort() then retry batch"]
  retriable -->|"fatal (not retriable)"| fatal["disconnect producer and shut down"]
  retriable -->|"poison message"| dlq["send to dead-letter topic, commit past it"]

Sort each failure: retry, abort-and-retry, dead-letter, or shut down