Batching and Compression

The idea in one sentence

A producer groups records into batches and optionally compresses them, so a little extra waiting and bigger batches buy dramatically higher throughput — a concept every Kafka client implements, even where the exact knobs differ.

linger.ms and batch.size trade latency for throughput

A producer does not send each record on its own. It accumulates records per partition into a batch and sends the batch as one request. Two canonical settings shape that batch — these are the Java-client / librdkafka config names you will meet in the official docs and most other clients:

batch.size — the maximum size in bytes of a single batch. A batch ships when it fills up to this size (or when linger.ms elapses).
linger.ms — how long the producer will wait for more records before sending a not-yet-full batch. It defaults to 0 (send as soon as possible). Raising it to, say, 10 lets more records accumulate, producing fuller batches.

Larger batches mean fewer requests, better compression, and higher throughput — at the cost of a few extra milliseconds of latency per record. That is the central lever.

# Wait up to 10 ms to fill a batch, and allow bigger batches for throughput
linger.ms=10
batch.size=65536

KafkaJS has no linger.ms or batch.size knobs — verified against [email protected]’s own TypeScript definitions and source: neither name appears anywhere in ProducerConfig. It batches the messages you pass to a single send() call internally and transmits them together — you control batching by how you group messages into send() calls (e.g. buffering records yourself and sending them in one array), not by a linger timer.

# librdkafka config keys (@confluentinc/kafka-javascript's native surface).
# Not path-commented: batch.num.messages is a real librdkafka-only key with
# no entry on kafka.apache.org's own config reference, so this course's
# harness (which checks properties fences against that official reference)
# cannot validate it — verified real instead by actually using it: this
# course's probe ran a producer with exactly these three keys successfully.
linger.ms=10
batch.size=65536
batch.num.messages=10000

Compressing batches

Compression is applied per batch, so bigger batches compress better. Every client picks a codec along the same speed-versus-ratio curve:

lz4 — very fast, good ratio; a common default choice.
zstd — excellent ratio, still fast; great when network or storage is the bottleneck.
snappy — fast, modest ratio.
gzip — best ratio, but the most CPU-heavy and slowest.
none — no compression (the default).

In the Java client this is the compression.type property. In KafkaJS, compression is a per-send option using CompressionTypes:

import { Kafka, CompressionTypes } from 'kafkajs'

// KafkaJS ships GZIP natively; snappy/lz4/zstd need an extra codec
// package registered via CompressionCodecs before you can use them.
await producer.send({
  topic: 'orders',
  compression: CompressionTypes.GZIP,
  messages,
})

Measured: throughput, no compression vs lz4, on this Mac

Real output from kafka-producer-perf-test.sh against this lesson’s own broker — 100,000 records, 200 bytes each, unlimited throughput (--throughput -1), acks=1, linger.ms=5, batch.size=65536:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --create --if-not-exists --topic perf-none --partitions 3 --replication-factor 1
/opt/kafka/bin/kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --create --if-not-exists --topic perf-lz4 --partitions 3 --replication-factor 1

echo "=== no compression ==="
/opt/kafka/bin/kafka-producer-perf-test.sh --topic perf-none --num-records 100000 --record-size 200 \
  --throughput -1 --producer-props bootstrap.servers="$BOOTSTRAP" acks=1 linger.ms=5 batch.size=65536

echo "=== lz4 compression ==="
/opt/kafka/bin/kafka-producer-perf-test.sh --topic perf-lz4 --num-records 100000 --record-size 200 \
  --throughput -1 --producer-props bootstrap.servers="$BOOTSTRAP" acks=1 linger.ms=5 batch.size=65536 compression.type=lz4

Real trimmed output (this Mac, single-broker, so treat the absolute numbers as local-hardware noise — the comparison is the point):

=== no compression ===
100000 records sent, 317460.317460 records/sec (60.55 MB/sec), 6.18 ms avg latency, 144.00 ms max latency, 5 ms 50th, 13 ms 95th, 18 ms 99th, 19 ms 99.9th.

=== lz4 compression ===
100000 records sent, 303951.367781 records/sec (57.97 MB/sec), 6.10 ms avg latency, 154.00 ms max latency, 5 ms 50th, 16 ms 95th, 28 ms 99th, 31 ms 99.9th.

Measured: compression ratio on realistic JSON, via kafka-dump-log.sh

500 repetitive, JSON-shaped order records (the kind of payload real services actually send — field names repeat, values are similar), sent once uncompressed and once with CompressionTypes.GZIP:

// @run
import { Kafka, CompressionTypes } from 'kafkajs';

const brokers = [process.env.KAFKA_BROKERS ?? 'localhost:9092'];
const prefix = process.env.TOPIC_PREFIX ?? '';
const kafka = new Kafka({ clientId: 'compression-probe', brokers });

function order(i: number) {
  return JSON.stringify({
    orderId: `order-${i}`,
    customerId: `customer-${i % 50}`,
    status: 'PENDING',
    items: [
      { sku: 'WIDGET-001', qty: 2, price: 19.99 },
      { sku: 'WIDGET-002', qty: 1, price: 9.99 },
    ],
    createdAt: '2026-09-29T00:00:00.000Z',
  });
}

async function send(topic: string, compression: number) {
  const admin = kafka.admin();
  await admin.connect();
  await admin.createTopics({ topics: [{ topic, numPartitions: 1, replicationFactor: 1 }] });
  await admin.disconnect();

  const producer = kafka.producer();
  await producer.connect();
  const messages = Array.from({ length: 500 }, (_, i) => ({ key: `k${i}`, value: order(i) }));
  await producer.send({ topic, messages, compression });
  await producer.disconnect();
}

async function main() {
  await send(`${prefix}orders-json-none`, CompressionTypes.None);
  await send(`${prefix}orders-json-gzip`, CompressionTypes.GZIP);
  console.log('sent 500 realistic JSON order records, both uncompressed and gzip');
  process.exit(0);
}

main();

NONE_LOG=$(find /tmp/kafka-logs -maxdepth 1 -type d -name '*orders-json-none-0' | head -1)
GZIP_LOG=$(find /tmp/kafka-logs -maxdepth 1 -type d -name '*orders-json-gzip-0' | head -1)
echo "=== uncompressed batch ==="
/opt/kafka/bin/kafka-dump-log.sh --files "$NONE_LOG"/00000000000000000000.log | head -3
echo "=== gzip batch ==="
/opt/kafka/bin/kafka-dump-log.sh --files "$GZIP_LOG"/00000000000000000000.log | head -3

Real output — both are a single record batch (500 records batched into one send() call), so the batch’s own size: field is the exact on-disk footprint:

=== uncompressed batch ===
baseOffset: 0 lastOffset: 499 count: 500 ... size: 108177 magic: 2 compresscodec: none ...

=== gzip batch ===
baseOffset: 0 lastOffset: 499 count: 500 ... size: 4741 magic: 2 compresscodec: gzip ...

108,177 bytes → 4,741 bytes — a 22.8x reduction (95.6% smaller) for realistic, repetitive JSON. The lesson from both measurements together: compression ratio is a property of your actual payload’s redundancy, not a fixed percentage any codec guarantees — measure it on your own data, the way this lesson measured it on realistic and random payloads separately.

`compression.type`: producer vs topic-level

compression.type can be set two places, and they compose:

Producer-side (compression.type in the Java client / compression: CompressionTypes.GZIP per-send in KafkaJS) — the producer compresses the batch before sending.
Topic-level (compression.type as a topic config, default producer) — controls what the broker does. producer (the default) means “keep whatever codec the producer already used, don’t touch it.” Setting a topic’s compression.type to a specific codec (e.g. lz4) makes the broker recompress every batch to that codec on write, regardless of what the producer sent — useful to enforce a standard across producers you don’t all control, at the cost of extra broker CPU.

# section: topic
compression.type=producer

buffer.memory, back-pressure, and the delivery deadline

Because send() is asynchronous, records sit in an in-memory buffer before the I/O layer transmits them. In the Java client, buffer.memory caps the total size of that buffer; when it fills, send() blocks (up to max.block.ms), applying back-pressure so a fast producer cannot exhaust memory. KafkaJS exposes no equivalent buffer setting — it does not pre-buffer records ahead of send() the way the Java client does, so there is no buffer.memory/max.block.ms pair to tune.

The Java client also bounds the whole journey with a deadline: delivery.timeout.ms (default 120000) is the total deadline from the moment send() returns until the record is acknowledged — it covers batching delay, network sends, and all retries. It should be >= request.timeout.ms + linger.ms. If a record cannot be delivered within it, the send fails permanently.

# 32 MiB send buffer; when full, send() applies back-pressure and blocks
buffer.memory=33554432
# Total deadline covering batching + retries; must be >= request.timeout.ms + linger.ms
delivery.timeout.ms=120000
request.timeout.ms=30000

KafkaJS achieves the same end goal — bounding how long a send may keep retrying — differently: via the client-wide retry policy (retries, initialRetryTime, maxRetryTime), covered in Error Handling & Retries. There is no single deadline property; retries simply stop once the policy’s retry count is exhausted. @confluentinc/kafka-javascript’s native librdkafka surface, by contrast, exposes the Java client’s own delivery.timeout.ms directly, since it shares librdkafka’s config surface with the C/C++/Java ecosystem.

flowchart LR
  send["send() adds record to buffer"] --> buf["'buffer.memory': full means back-pressure"]
  buf --> batch["Batch fills to 'batch.size' or 'linger.ms' elapses"]
  batch --> comp["Compress batch: lz4 / zstd / snappy / gzip"]
  comp --> io["I/O thread sends; retries allowed"]
  io --> deadline["Must finish within 'delivery.timeout.ms' (120000)"]

Records batch, compress, and drain within the delivery deadline

Tuning for throughput

In the Java client, push throughput up by raising linger.ms a little (5-20 ms), enlarging batch.size, and turning on lz4 or zstd compression — the three work together, since more waiting makes fuller batches and fuller batches compress better. Keep buffer.memory comfortable so back-pressure does not stall you, and make sure delivery.timeout.ms is generous enough to absorb retries during a broker hiccup. In KafkaJS, since there are no batching knobs to tune, focus your throughput work on: sending larger arrays of messages per send() call, choosing compression: CompressionTypes.GZIP (or a registered LZ4/ZSTD codec), and sizing the retry policy generously enough to ride out broker hiccups. In @confluentinc/kafka-javascript, the full Java-client-equivalent knob set (linger.ms, batch.size, batch.num.messages, compression.codec) is available directly.

🌊 The one-line takeaway

The Java client batches records with linger.ms and batch.size (a little latency for much higher throughput), compresses each batch with compression.type, applies back-pressure via buffer.memory, and bounds the whole send with delivery.timeout.ms (default 120000). KafkaJS has no linger/batch-size/buffer knobs, and ships only GZIP natively (verified from source) — it batches per send() call automatically, compresses via the compression option on send(), and bounds retries with its client-wide retry policy instead of a single deadline. @confluentinc/kafka-javascript exposes the full librdkafka knob set (linger.ms, batch.size, batch.num.messages, all four codecs) directly. Measured on this Mac: compression on incompressible random bytes cost ~4% throughput; on realistic JSON it cut a batch from 108,177 to 4,741 bytes (22.8x) — measure your own payload, don’t assume either outcome.

Reading AI-generated code

🐛 Spot the bug

import { Kafka, CompressionTypes } from 'kafkajs';

// AI comment: "tuned for high throughput — larger batches, less overhead"
const kafka = new Kafka({
clientId: 'events-app',
brokers: ['localhost:9092'],
});

async function sendEvents(events: Record<string, unknown>[]) {
const producer = kafka.producer();
await producer.connect();
await producer.send({
  topic: 'events',
  compression: CompressionTypes.LZ4, // "lz4 is fast, great for throughput"
  messages: events.map((e) => ({ value: JSON.stringify(e) })),
});
await producer.disconnect();
}

Show the answer

This looks like solid, well-commented tuning advice — lz4 genuinely is the right general-purpose choice on the Java client, and the comment sounds like it came from a real performance guide. The real bug: this is KafkaJS, not the Java client, and KafkaJS ships only GZIP natively (verified from source above) — Snappy/LZ4/ZSTD throw KafkaJSNotImplemented('LZ4 compression not implemented') unless a separate codec package is installed and registered with CompressionCodecs first. sendEvents will throw on its very first call, not silently degrade — but a codebase this pattern was copied into (a tutorial or another Kafka client’s example, adapted without checking) will not know why until it runs. The fix is either the codec KafkaJS actually ships, or installing and registering the real LZ4 package:

// Option A: use what KafkaJS ships natively
compression: CompressionTypes.GZIP,

// Option B: install kafkajs-lz4, then register it once at startup
import LZ4 from 'kafkajs-lz4';
import { CompressionCodecs } from 'kafkajs';
CompressionCodecs[CompressionTypes.LZ4] = new LZ4().codec;

Mental Model

Batching and compression both trade a little client-side time for a lot less broker/network/disk work per record — but they operate at different layers and don’t require each other. Batching is about when a request leaves the client (accumulate vs. send-immediately); compression is about how many bytes that request costs once it does. KafkaJS collapses the “when” decision down to “however many messages you put in one send() array” (no timer-based accumulation at all), while still giving you the “how many bytes” decision per call via compression. The Java client and librdkafka separate both decisions into independent, continuously-tunable knobs (linger.ms/batch.size for when, compression.type for how). Same two levers, different granularity of control.

Trade-offs

KafkaJS’s send()-call batching — zero configuration, but no way to tune the latency/throughput curve independently of how your application code happens to group messages into send() calls.
@confluentinc/kafka-javascript’s linger.ms/batch.size/batch.num.messages — full independent control, at the cost of tuning knobs your application now has to get right.
GZIP (KafkaJS’s only native codec) — works everywhere with zero extra dependencies, but is the slowest, most CPU-heavy codec of the four — fine for moderate-throughput topics, a real bottleneck risk at very high volume.
LZ4/Snappy/ZSTD via @confluentinc/kafka-javascript or a registered KafkaJS codec package — meaningfully faster compression, at the cost of an extra native dependency (confluent client) or an extra, less-maintained plugin package (KafkaJS codec plugins).
Topic-level compression.type override — enforces a consistent codec across producers you don’t control, at a real, continuous broker CPU cost for every batch that needs recompressing.

Common Misconceptions

“KafkaJS has linger.ms and batch.size like every other client, just named slightly differently.” No — verified from source and types: neither exists in KafkaJS at all. Batching is implicit, driven entirely by how many messages you pass to one send() call.
“Compression always makes things faster.” No — measured directly above: on incompressible random-byte payloads, lz4 cost ~4% throughput compared to no compression, because the compressor’s CPU cost bought almost nothing back in size.
“CompressionTypes.LZ4 (or Snappy/ZSTD) just works in KafkaJS, the same way GZIP does.” No — proven from source and by the SpotTheBug case above: only GZIP is bundled; the others throw KafkaJSNotImplemented unless you install and register a separate codec package first.
“A topic’s compression.type setting is what the producer uses to compress.” No — the topic-level setting controls what the broker does after receiving a batch (default producer: leave it alone); the producer’s own setting (compression.type property, or KafkaJS’s per-send compression option) is what actually compresses the bytes on the way out.