Offsets & Commits

The idea in one sentence

A consumer records how far it has processed by committing an offset per partition — Kafka stores it in the internal __consumer_offsets topic — and where you commit relative to where you process decides whether you can lose or duplicate records.

Committed vs current position

Two offsets travel with each partition:

• Position — the next offset the consumer will fetch (advances as you consume).
• Committed offset — the last offset the consumer durably recorded as “done.”

On restart or rebalance, a consumer resumes from the committed offset. So a commit is a promise: “everything before this is processed; don’t send it again.”

Auto-commit is convenient and risky

By default KafkaJS commits automatically (autoCommit: true): it periodically commits the offsets it has delivered, every autoCommitInterval ms (or after autoCommitThreshold messages). It is easy, but it commits on a schedule regardless of whether your processing succeeded — a crash after an auto-commit but before processing loses those records. For anything that matters, set autoCommit: false and commit after processing:

const consumer = kafka.consumer({ groupId: 'order-processor' });
await consumer.connect();
await consumer.subscribe({ topic: 'orders' });

await consumer.run({
  autoCommit: false,                                    // take control of commits
  eachMessage: async ({ topic, partition, message }) => {
    await process(message);                             // do the work FIRST
    // then commit the NEXT offset to read (offset + 1) — at-least-once
    await consumer.commitOffsets([
      { topic, partition, offset: (Number(message.offset) + 1).toString() },
    ]);
  },
});

Manual commits in KafkaJS

With autoCommit: false you commit yourself with consumer.commitOffsets([...]), which returns a Promise. Await it to be sure the commit was acknowledged before you move on — the safe, slightly slower choice, like a synchronous commit. Skip the await to fire it in the background for throughput, accepting that a failure won’t be caught inline. Either way you commit the next offset to read (offset + 1), because that is where the consumer should resume.

Where to start, and replaying with seek()

A committed offset only helps if one exists. The first time a group reads a partition — or if its committed offset has aged out of retention — KafkaJS’s fromBeginning flag on subscribe() decides where to begin:

• fromBeginning: true — start at the beginning of the partition and reprocess all retained history (the earliest policy).
• fromBeginning: false — start at the end, reading only records that arrive from now on (the default, latest).

You can also override the position yourself with consumer.seek() — move a partition to any offset to replay old records or skip ahead:

// where to begin when there is no committed offset:
// fromBeginning: true = earliest (replay all history), false = latest (default)
await consumer.subscribe({ topic: 'orders', fromBeginning: true });

// Replay partition 0 of "orders" from offset 100 (call after run() has started)
consumer.seek({ topic: 'orders', partition: 0, offset: '100' }); // next fetch starts at offset 100
// use offset '0' to replay the whole partition from the start

flowchart LR
  poll["poll batch"] --> proc["process records"]
  proc --> commit["commitOffsets() after success"]
  commit --> poll
  proc -.->|crash before commit| replay["restart re-reads from last commit (duplicates)"]

Commit after processing gives at-least-once

Why this is at-least-once by default

Commit after processing and a crash between the two means those records are re-read and reprocessed on restart — at-least-once, possible duplicates. Commit before processing and a crash means they are skipped — at-most-once, possible loss. At-least-once is the common default; making it effectively exactly-once takes idempotent processing or transactions, which the next module covers.

🌊 The one-line takeaway

A consumer commits offsets to mark progress (stored in __consumer_offsets), and resumes from the committed offset after a restart. Auto-commit (autoCommit: true) is easy but commits on a schedule and can lose records; for correctness set autoCommit: false and commit after processing with consumer.commitOffsets([...]). Committing after processing yields at-least-once (possible duplicates) — the default to design around. Use fromBeginning to decide where a brand-new group starts (earliest/latest) and seek() to replay.

Where does Kafka store committed consumer offsets?

On the consumer's local disk only In the internal `__consumer_offsets` topic In ZooKeeper In a relational database you must provide

Why is auto-commit risky for important processing?

It commits too rarely to be useful It commits on a schedule regardless of whether processing succeeded, so a crash can lose records It duplicates every record It disables the consume loop

Committing offsets AFTER processing gives which delivery semantic?

At-most-once (records may be lost) At-least-once — a crash before commit means records are reprocessed (possible duplicates) Exactly-once with no extra work No delivery guarantee at all

With manual commits in KafkaJS, what does awaiting consumer.commitOffsets(...) give you?

Nothing; commits are always instant and unacknowledged It waits until the commit is acknowledged before continuing — the safe choice, vs firing it in the background for throughput It deletes old offsets from the topic It creates a new consumer group

Submit

Mark as complete