Consumer Groups & Rebalancing

The idea in one sentence

A consumer group is a set of consumers that share a topic’s partitions between them — each partition is handled by exactly one consumer in the group — and rebalancing is the process of reassigning partitions when membership changes.

Groups divide the work

Every consumer declares a groupId. Kafka gives each partition to exactly one consumer within a group, so a topic with 6 partitions can be processed by up to 6 consumers in parallel. Add a 7th consumer and it sits idle — partitions, not consumers, are the ceiling on parallelism.

Different groups are independent: analytics and billing each get their own full copy of the stream, tracking their own offsets. Within one group, the partitions are divided.

You can inspect a group, its members, and its lag from the CLI:

# Show members, assigned partitions, and lag for a group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group billing

flowchart TB
  subgraph t["Topic orders: 3 partitions"]
    p0["orders-0"]
    p1["orders-1"]
    p2["orders-2"]
  end
  subgraph g1["Group billing"]
    c1["consumer A"]
    c2["consumer B"]
  end
  p0 --> c1
  p1 --> c1
  p2 --> c2
  t --> g2["Group analytics (own offsets, full copy)"]

One group divides partitions; another group reads the same topic independently

Rebalancing

When a consumer joins, leaves, or dies — or partitions are added — the group must rebalance: redistribute partitions among the current members. The classic protocol does this with a stop-the-world step: every consumer stops processing, gives up its partitions, and waits for a new assignment. On large groups that pause hurts.

KIP-848: the new rebalance protocol (Java client)

Kafka 4.0 ships the next-generation consumer rebalance protocol (KIP-848) as generally available for the Java client. It moves assignment logic to the broker-side group coordinator and makes rebalances incremental — only the partitions that actually move are revoked, and the rest keep processing. The result is shorter, less disruptive rebalances, especially for large groups.

# Java-client opt-in: the new consumer group protocol (KIP-848), GA in Kafka 4.0
group.protocol=consumer
# The classic (older) protocol is still available:
# group.protocol=classic

KafkaJS consumers — the ones this course’s examples use — currently speak the classic group protocol only; there is no group.protocol setting to flip. That’s not a lost benefit, though: KIP-848’s redesign lives mostly on the broker-side coordinator, so a KRaft 4.x cluster still benefits from the new coordinator’s efficiency even when talking to classic-protocol consumers like KafkaJS’s. Support for the new protocol in JavaScript clients (e.g. Confluent’s @confluentinc/kafka-javascript) will arrive as it lands upstream.