Streams & I/O

The problem with loading everything at once

Imagine reading a 4 GB log file. If you load it all into memory first, Node has to allocate 4 GB before you can inspect a single line. On a machine with 2 GB of RAM, the process crashes before it begins.

Streams solve this by processing data in chunks — you start reading the first chunk while the rest is still arriving from disk or network. Memory stays bounded regardless of how large the source is.

The same insight applies to writing: instead of buffering an entire response in memory and sending it all at once, a stream pipes each chunk to the destination as it arrives.

The four stream types

Node.js defines four abstract stream types in the node:stream module:

Type	Direction	Example
`Readable`	Data flows out	`fs.createReadStream`, `process.stdin`
`Writable`	Data flows in	`fs.createWriteStream`, `process.stdout`
`Duplex`	Bidirectional	TCP socket (`net.Socket`)
`Transform`	Bidirectional with transformation	`zlib.createGzip` (compresses in-flight)

Every stream is also an EventEmitter. The stream base classes emit standard events (data, end, error, drain, finish) that you attach handlers to.

What this module covers

Lesson	Topic
Readable Streams	`data`/`end`/`error` events, `Readable.from`, async iteration
Writable Streams	`write()`, `end()`, and the `drain` event
Pipe & Pipeline	Connecting streams with `.pipe()` and `stream.pipeline()`
Backpressure	What it is, why it matters, and how pipelines handle it

A first look at streaming

The demo below uses Readable.from to create a readable stream from an array of strings, then consumes it chunk by chunk with for await. Notice that no full array is ever held in a buffer — each item is yielded on demand.

Node.js

const { Readable } = require('node:stream');

async function main() {
  const source = Readable.from(['hello', ' ', 'streaming', ' ', 'world']);

  let result = '';
  for await (const chunk of source) {
    result += chunk;
    console.log('chunk: ' + JSON.stringify(chunk));
  }
  console.log('assembled: ' + result);
}

main();

Output (verified on Node 24)

chunk: "hello"
chunk: " "
chunk: "streaming"
chunk: " "
chunk: "world"
assembled: hello streaming world

Mental Model

flowchart LR
  Readable["Readable — data flows OUT"] -->|".pipe() / pipeline()"| Writable["Writable — data flows IN"]
  Readable -->|"through a"| Transform["Transform — reshapes each chunk"]
  Transform --> Writable
  Duplex["Duplex — both directions independently"]
  All["Readable, Writable, Duplex, Transform"] -->|"all extend"| EE["EventEmitter"]

data flows from a Readable, optionally through a Transform, into a Writable — every stream type is also an EventEmitter

Trade-offs

Choice	Benefit	Cost
Streaming (chunked)	Memory use stays bounded regardless of total data size; can start processing before the source finishes	More moving parts than a single `readFile()` call — events or async iteration, backpressure, error propagation
Loading everything into memory first	Simplest possible code — no event wiring, whole result available at once	Peak memory scales with input size; a large enough input crashes the process outright

Common Misconceptions

“Streams are always faster than reading a whole file” — fact: streams save memory, not necessarily wall-clock time. For a file that comfortably fits in RAM, readFile() can be simpler and just as fast.
“Every stream needs a data event listener to produce output” — fact: for await...of (async iteration) consumes a Readable without ever touching the data event, and handles backpressure automatically in the process.
“Transform streams and Duplex streams are the same thing” — fact: both are bidirectional, but a Transform’s output is derived from its input (one side feeds the other); a Duplex’s readable and writable sides are independent — a TCP socket reads and writes unrelated data on each side.

Reading AI-generated code

Does code call fs.readFileSync() or fs.readFile() to load a file only to immediately write it back out or forward it somewhere else, when a stream (or pipeline) would never hold the whole file in memory?
Does a snippet attach a data listener to a Readable without ever checking backpressure downstream, assuming .pipe()’s automatic pause/resume behavior without actually calling .pipe()?
Does code treat Transform and Duplex as interchangeable — extending Duplex when the real intent is “take X in, produce derived Y out” (a Transform)?

🐛 Spot the bug

const fs = require('node:fs');

// Loads the whole file into memory just to copy it elsewhere
var data = fs.readFileSync('./huge-input.log');
fs.writeFileSync('./huge-output.log', data);

Show the answer

This reads the entire file into a single in-memory Buffer before writing it back out, even though nothing here needs to inspect or transform the content — for a multi-gigabyte log file this can exhaust available memory or simply waste it. fs.createReadStream('./huge-input.log').pipe(fs.createWriteStream('./huge-output.log')) (or stream/promises’ pipeline, covered later in this module) copies the file in bounded chunks without ever holding more than one chunk in memory at a time.