Partitions & Ordering

The idea in one sentence

The partition is Kafka’s unit of both parallelism and ordering: more partitions mean more throughput, but Kafka only guarantees order within a partition — never across the whole topic.

One guarantee, one scope

Kafka gives you a strong ordering guarantee, but it is scoped to a single partition. Within orders-0, offset 5 always follows offset 4, forever. Across partitions there is no defined order: offset 5 of orders-0 and offset 5 of orders-1 could have been written in either real-time order, and consumers may read them in either order.

orders-0:  [a0][a1][a2][a3]      ← totally ordered
orders-1:  [b0][b1][b2]          ← totally ordered
                                  ← a2 vs b1: NO defined order

This is not a limitation to work around so much as the price of horizontal scale. A single totally-ordered log cannot be written faster than one machine can append. Splitting into partitions lets many machines append in parallel — you trade global order for throughput.

Partitions bound consumer parallelism

Within a consumer group, a partition is read by exactly one consumer at a time. That makes the partition count the ceiling on how many consumers can work in parallel:

6 partitions, 3 consumers → each reads 2 partitions.
6 partitions, 6 consumers → each reads 1 (maximum parallelism).
6 partitions, 8 consumers → 6 work, 2 sit idle with nothing to assign.

If you want to scale consumption later, you must have provisioned enough partitions up front — consumers cannot subdivide a single partition.

flowchart LR
  subgraph topic["Topic: orders (4 partitions)"]
    p0["orders-0"]
    p1["orders-1"]
    p2["orders-2"]
    p3["orders-3"]
  end
  p0 --> c1["Consumer 1"]
  p1 --> c1
  p2 --> c2["Consumer 2"]
  p3 --> c3["Consumer 3"]
  idle["Consumer 4 (idle: no partition left)"]

Partition count caps how many consumers can work in parallel

The design tradeoff

Choosing a partition count is choosing a point on a curve:

More partitions → higher throughput and more consumer parallelism, but more open files, more metadata, and longer rebalances.
Fewer partitions → simpler and cheaper, but a lower throughput ceiling.
Ordering need → if a set of events must stay strictly ordered relative to each other, they must land on the same partition (the next lesson shows how keys do this).

The practical rule: partition for your target throughput plus headroom, and make sure related events that need ordering share a partition.