Batching and Compression

The idea in one sentence

A producer groups records into batches and optionally compresses them, so a little extra waiting and bigger batches buy dramatically higher throughput — a concept every Kafka client implements, even where the exact knobs differ.

linger.ms and batch.size trade latency for throughput

A producer does not send each record on its own. It accumulates records per partition into a batch and sends the batch as one request. Two canonical settings shape that batch — these are the Java-client / librdkafka config names you will meet in the official docs and most other clients:

batch.size — the maximum size in bytes of a single batch. A batch ships when it fills up to this size (or when linger.ms elapses).
linger.ms — how long the producer will wait for more records before sending a not-yet-full batch. It defaults to 0 (send as soon as possible). Raising it to, say, 10 lets more records accumulate, producing fuller batches.

Larger batches mean fewer requests, better compression, and higher throughput — at the cost of a few extra milliseconds of latency per record. That is the central lever.

# Wait up to 10 ms to fill a batch, and allow bigger batches for throughput
linger.ms=10
batch.size=65536

KafkaJS has no linger.ms or batch.size knobs. It batches the messages you pass to a single send() call internally and transmits them together — you control batching by how you group messages into send() calls (e.g. buffering records yourself and sending them in one array), not by a linger timer.

Compressing batches

Compression is applied per batch, so bigger batches compress better. Every client picks a codec along the same speed-versus-ratio curve:

lz4 — very fast, good ratio; a common default choice.
zstd — excellent ratio, still fast; great when network or storage is the bottleneck.
snappy — fast, modest ratio.
gzip — best ratio, but the most CPU-heavy and slowest.
none — no compression (the default).

In the Java client this is the compression.type property. In KafkaJS, compression is a per-send option using CompressionTypes:

import { Kafka, CompressionTypes } from 'kafkajs'

// KafkaJS ships GZIP natively; snappy/lz4/zstd need an extra codec
// package registered via CompressionCodecs before you can use them.
await producer.send({
  topic: 'orders',
  compression: CompressionTypes.GZIP,
  messages,
})

buffer.memory, back-pressure, and the delivery deadline

Because send() is asynchronous, records sit in an in-memory buffer before the I/O layer transmits them. In the Java client, buffer.memory caps the total size of that buffer; when it fills, send() blocks (up to max.block.ms), applying back-pressure so a fast producer cannot exhaust memory. KafkaJS exposes no equivalent buffer setting — it does not pre-buffer records ahead of send() the way the Java client does, so there is no buffer.memory/max.block.ms pair to tune.

The Java client also bounds the whole journey with a deadline: delivery.timeout.ms (default 120000) is the total deadline from the moment send() returns until the record is acknowledged — it covers batching delay, network sends, and all retries. It should be >= request.timeout.ms + linger.ms. If a record cannot be delivered within it, the send fails permanently.

# 32 MiB send buffer; when full, send() applies back-pressure and blocks
buffer.memory=33554432
# Total deadline covering batching + retries; must be >= request.timeout.ms + linger.ms
delivery.timeout.ms=120000
request.timeout.ms=30000

KafkaJS achieves the same end goal — bounding how long a send may keep retrying — differently: via the client-wide retry policy (retries, initialRetryTime, maxRetryTime), covered in Error Handling & Retries. There is no single deadline property; retries simply stop once the policy’s retry count is exhausted.

flowchart LR
  send["send() adds record to buffer"] --> buf["'buffer.memory': full means back-pressure"]
  buf --> batch["Batch fills to 'batch.size' or 'linger.ms' elapses"]
  batch --> comp["Compress batch: lz4 / zstd / snappy / gzip"]
  comp --> io["I/O thread sends; retries allowed"]
  io --> deadline["Must finish within 'delivery.timeout.ms' (120000)"]

Records batch, compress, and drain within the delivery deadline

Tuning for throughput

In the Java client, push throughput up by raising linger.ms a little (5-20 ms), enlarging batch.size, and turning on lz4 or zstd compression — the three work together, since more waiting makes fuller batches and fuller batches compress better. Keep buffer.memory comfortable so back-pressure does not stall you, and make sure delivery.timeout.ms is generous enough to absorb retries during a broker hiccup. In KafkaJS, since there are no batching knobs to tune, focus your throughput work on: sending larger arrays of messages per send() call, choosing compression: CompressionTypes.GZIP (or a registered LZ4/ZSTD codec), and sizing the retry policy generously enough to ride out broker hiccups.