Kafka Streams

The idea in one sentence

Kafka Streams is a Java library — not a separate cluster — that you embed in your own application to transform, aggregate, and join Kafka topics as continuous streams, with state and exactly-once guarantees handled for you.

KStream, KTable, and the duality

Kafka Streams gives you two views of a topic:

KStream — an unbounded sequence of independent events. Every record is a new fact (“user clicked”). Nothing is overwritten.
KTable — a changelog interpreted as the latest value per key. Each record updates the row for its key, like UPSERTs into a table.

This is the stream-table duality: a table is the snapshot you get by replaying a stream of updates, and a stream is the sequence of changes you get by watching a table. The same topic can be read either way depending on what you need — a running feed of events, or the current state per key.

flowchart LR
  s["KStream (events)"] -->|"aggregate / reduce"| t["KTable (latest per key)"]
  t -->|"toStream()"| s2["KStream (changes)"]

A stream of updates folds into a table; a table emits a stream of changes

Stateless vs stateful operations

Operations split into two families:

Stateless — each record is handled on its own: map, filter, flatMap, branch. No memory of past records is needed.
Stateful — the result depends on records seen so far: aggregate, count, reduce, join, and windowing (grouping records into time buckets). These need somewhere to keep the running result.

That “somewhere” is a state store — a local key-value store (RocksDB by default) living inside your app instance. To survive a crash, every state store is backed by a changelog topic in Kafka: each update is also written to the changelog, so if the instance restarts elsewhere, Streams rebuilds the store by replaying it. Your local state is durable without you managing a database.

A word-count topology

Here is the classic example — count how often each word appears across an input topic:

StreamsBuilder builder = new StreamsBuilder();

// Read the input topic as a stream of (key, line) events
KStream<String, String> lines = builder.stream("text-input");

KTable<String, Long> counts = lines
    .flatMapValues(line -> Arrays.asList(line.toLowerCase().split("\\W+"))) // split into words
    .groupBy((key, word) -> word)     // re-key by the word itself
    .count();                         // stateful: running total per word -> a KTable

// Write the changelog of counts out to an output topic
counts.toStream().to("word-counts");

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

The count() builds a KTable backed by a state store and its changelog; toStream().to(...) emits every updated count downstream.

Exactly-once and the streams protocol

Stream processing that reads, updates state, and writes is easy to get wrong on failure — you can double-count. Kafka Streams makes this a one-line setting. With processing.guarantee set to exactly_once_v2, Streams wraps each read-process-write cycle (including the state-store changelog and consumer offsets) in a single Kafka transaction, so a retry never applies an update twice:

// Turn on end-to-end exactly-once for the whole topology
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, "exactly_once_v2");

// Modern Kafka: use the dedicated broker-side rebalance protocol for Streams groups
props.put(StreamsConfig.GROUP_PROTOCOL_CONFIG, "streams");