Keys & Partitioning

The idea in one sentence

The key on a record decides its partition — the default partitioner hashes the key so that the same key always lands on the same partition, which is exactly how you get ordering guarantees per entity (per user, per order, per account).

Key in, partition out

Every message you send can carry an optional key. The producer’s partitioner turns that key into a partition number:

// Same key "user-42" → same partition, every time → ordered per user
await producer.send({
  topic: 'orders',
  messages: [
    { key: 'user-42', value: 'placed order #918' },
    { key: 'user-42', value: 'cancelled order #918' },
  ],
})

For a non-null key, the default partitioner computes roughly hash(key) % numberOfPartitions. Deterministic hashing means every record for user-42 goes to the same partition, so all of that user’s events stay in one totally-ordered log. Different keys spread across partitions, balancing load.

Null keys and the sticky partitioner

If the key is null, you do not care which partition a record lands on — only about even spreading. Kafka uses the sticky partitioner: it sends a batch of null-key records to one partition until that batch is full or linger.ms elapses, then switches to another. Sticking to one partition per batch produces larger, more efficient batches (fewer, bigger requests) while still spreading evenly over time.

flowchart TB
  k1["key = user-42"] -->|hash % N| p1["partition 1 (always)"]
  k2["key = user-99"] -->|hash % N| p2["partition 2 (always)"]
  k3["key = null"] -->|sticky batch| p3["one partition per batch, then rotate"]

Keyed records hash to a fixed partition; null keys batch stickily

The add-partitions caveat

Because the mapping is hash(key) % numberOfPartitions, changing the partition count changes where keys land. Add a partition to a live topic and user-42 may hash to a new partition — its future events go somewhere different from its past ones, breaking the per-key ordering you relied on.

Practical consequences:

Provision enough partitions up front for keyed topics where ordering matters.
If you must grow, understand that historical per-key ordering is preserved only within each old partition, not across the boundary.

Co-partitioning for joins

When you want to join two streams by the same key (say orders and payments, both keyed by orderId), give both topics the same partition count and the same partitioner. Then orderId=918 lands on partition 3 in both topics, and a single consumer or stream task sees both sides locally — no cross-partition shuffle. This is co-partitioning, and Kafka Streams requires it for key-based joins.