io and bufio

The io.Reader and io.Writer interfaces

The io package defines two small but foundational interfaces:

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

These two interfaces are the lingua franca of I/O in Go. Dozens of types implement them: files, network connections, HTTP request and response bodies, in-memory buffers, and more. Functions that accept io.Reader or io.Writer work with all of them without modification.

strings.NewReader — an in-memory Reader

strings.NewReader(s) wraps a string as an io.Reader. This is invaluable in tests and examples — you can exercise any function that reads from a stream without touching the filesystem.

r := strings.NewReader("hello, world")
data, _ := io.ReadAll(r)
fmt.Println(string(data)) // hello, world

bytes.Buffer — a read/write buffer

bytes.Buffer implements both io.Reader and io.Writer. You write into it and read back out:

var buf bytes.Buffer
fmt.Fprintln(&buf, "first line")
fmt.Fprintln(&buf, "second line")
content, _ := io.ReadAll(&buf)
fmt.Print(string(content))

fmt.Fprintln accepts any io.Writer, so you can redirect formatted output to a buffer, a file, or a network connection with zero code changes.

bufio.Scanner — line-by-line reading

bufio.Scanner wraps any io.Reader and provides convenient line-by-line (or token-by-token) reading. The default split function is bufio.ScanLines.

scanner := bufio.NewScanner(r)
for scanner.Scan() {
    fmt.Println(scanner.Text()) // one line at a time
}
if err := scanner.Err(); err != nil {
    log.Fatal(err)
}

scanner.Scan() returns false when the input is exhausted or an error occurs. Always check scanner.Err() after the loop — a false return alone does not distinguish EOF from an error.

io helper functions

The io package also provides useful helpers:

io.ReadAll(r)           // read until EOF, return []byte
io.Copy(dst, src)       // copy from Reader to Writer
io.MultiReader(r1, r2)  // chain readers sequentially
io.Discard              // Writer that discards all bytes (like /dev/null)
io.LimitReader(r, n)    // read at most n bytes

Run ▸ Open in Playground

package main

import (
  "bufio"
  "bytes"
  "fmt"
  "io"
  "os"
  "strings"
)

// printLines accepts any io.Reader — works with files, buffers, or strings
func printLines(r io.Reader) {
  scanner := bufio.NewScanner(r)
  for scanner.Scan() {
    fmt.Println(">", scanner.Text())
  }
  if err := scanner.Err(); err != nil {
    fmt.Println("scan error:", err)
  }
}

func main() {
  // strings.NewReader implements io.Reader
  r := strings.NewReader("first line\nsecond line\nthird line")
  printLines(r)

  // bytes.Buffer implements both io.Reader and io.Writer
  var buf bytes.Buffer
  fmt.Fprintln(&buf, "written to buffer")
  fmt.Fprintln(&buf, "another line")
  content, _ := io.ReadAll(&buf)
  fmt.Print(string(content))

  // io.MultiReader chains two readers sequentially
  r1 := strings.NewReader("hello ")
  r2 := strings.NewReader("world\n")
  combined := io.MultiReader(r1, r2)
  fmt.Print("MultiReader output: ")
  io.Copy(os.Stdout, combined)

  // bufio.Scanner over in-memory CSV
  csv := "alice,30\nbob,25\ncharlie,35"
  scanner := bufio.NewScanner(strings.NewReader(csv))
  for scanner.Scan() {
    parts := strings.Split(scanner.Text(), ",")
    fmt.Printf("name=%s age=%s\n", parts[0], parts[1])
  }
}

The io.Reader contract in depth

The two-line Read(p []byte) (n int, err error) signature hides three rules that trip up AI-generated code constantly:

1. Read can return n > 0 and a non-nil err in the same call. The caller must process the n bytes returned before checking err — discarding them because err != nil silently drops the last chunk of data.
2. A “short read” is legal. Read may fill fewer bytes than len(p) even when more data is coming and err == nil. Never assume one Read call fills your buffer — loop, or use a helper that loops for you.
3. io.EOF is not a failure. It is the normal signal that there is no more data. Treat err == io.EOF as “stop looping, no error to report” — never log.Fatal it or return it up the stack unwrapped as if something broke.

n, err := r.Read(buf)
if n > 0 {
    process(buf[:n]) // always handle n bytes first
}
if err != nil {
    if err == io.EOF {
        return nil // clean end of stream, not an error
    }
    return err
}

When you need an exact or minimum number of bytes and don’t want to hand-write the loop, io.ReadFull(r, buf) reads until buf is full or returns io.ErrUnexpectedEOF if the stream ends first; io.ReadAtLeast(r, buf, min) is the same idea for a minimum length instead of the full buffer.

bufio.Scanner’s 64 KB token limit

bufio.Scanner caps each token (line) at bufio.MaxScanTokenSize — 64 KB — by default. When a line exceeds that, Scan() returns false and Err() returns bufio.ErrTooLong. This looks exactly like clean EOF unless you check Err() after the loop — the rest of the input is silently dropped.

scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024) // raise the max to 1 MB
for scanner.Scan() {
    process(scanner.Text())
}
if err := scanner.Err(); err != nil {
    log.Fatal(err) // catches ErrTooLong, not just I/O errors
}

scanner.Buffer(initial, max) sets both the starting buffer and the ceiling it is allowed to grow to. Call it before the first Scan() if a single line might legitimately exceed 64 KB (a long JSON line, a base64 blob).

bufio.Writer, Flush, and Close errors

bufio.Writer buffers writes in memory and only reaches the underlying io.Writer when the buffer fills, or when you call Flush() explicitly. Nothing is guaranteed to have reached disk or the network until Flush() returns.

A Close() error matters for the same reason: a type built on top of a buffered writer (or a file) can surface a delayed write failure — such as a disk-full error — only when the last flush happens inside Close. Ignoring defer f.Close()’s return value silently swallows that failure.

w := bufio.NewWriter(f)
fmt.Fprintln(w, "important data")
if err := w.Flush(); err != nil {
    return fmt.Errorf("flush: %w", err) // the write may not have failed until now
}
if err := f.Close(); err != nil {
    return fmt.Errorf("close: %w", err) // a writer's Close error is not decorative
}

io.Copy, TeeReader, Pipe, and NopCloser

io.Copy(dst, src) reads from src and writes to dst until src returns io.EOF, relying on the Writer contract: Write must return a non-nil error if it wrote fewer than len(p) bytes, so io.Copy can trust a nil error to mean the full chunk landed.

Three small io helpers round out the toolkit:

• io.TeeReader(r, w) — returns a Reader that mirrors every byte it reads into w as a side effect (checksumming or logging a stream while it is consumed, without buffering it all in memory first).
• io.Pipe() — an in-memory, synchronous io.Reader/io.Writer pair with no buffer; a write blocks until a matching read drains it, useful for streaming data from one goroutine to another without an intermediate buffer.
• io.NopCloser(r) — wraps an io.Reader with a no-op Close, satisfying io.ReadCloser. This is exactly what http.Request.Body needs in a test, or what you hand back when you don’t want a caller to close a shared reader.

tee := io.TeeReader(resp.Body, hasher)     // reading tee also feeds hasher
req.Body = io.NopCloser(bytes.NewReader(replayed)) // satisfy io.ReadCloser in a test

Run ▸ Open in Playground

package main

import (
  "bufio"
  "bytes"
  "fmt"
  "io"
  "strings"
)

// shortReader returns at most 3 bytes per Read call, and delivers n>0
// together with io.EOF on the final call — a legal io.Reader.
type shortReader struct {
  data []byte
}

func (r *shortReader) Read(p []byte) (int, error) {
  if len(r.data) == 0 {
    return 0, io.EOF
  }
  n := copy(p, r.data[:min(3, len(r.data))])
  r.data = r.data[n:]
  if len(r.data) == 0 {
    return n, io.EOF // n > 0 AND err != nil in the same call
  }
  return n, nil
}

func main() {
  // 1. short reads: n>0 with err!=nil on the last call
  sr := &shortReader{data: []byte("hello!")}
  buf := make([]byte, 10)
  for {
    n, err := sr.Read(buf)
    fmt.Printf("Read: n=%d err=%v data=%q\n", n, err, buf[:n])
    if err != nil {
      break
    }
  }

  // 2. io.ReadFull loops for you
  sr2 := &shortReader{data: []byte("hello!")}
  exact := make([]byte, 6)
  n, err := io.ReadFull(sr2, exact)
  fmt.Printf("ReadFull: n=%d err=%v data=%q\n", n, err, exact[:n])

  // io.ReadFull when the stream has fewer bytes than requested
  sr3 := &shortReader{data: []byte("hi")}
  short := make([]byte, 6)
  n, err = io.ReadFull(sr3, short)
  fmt.Printf("ReadFull short: n=%d err=%v\n", n, err)

  // 3. bufio.Scanner ErrTooLong when a token exceeds the buffer max
  longLine := strings.Repeat("x", 20)
  scanner := bufio.NewScanner(strings.NewReader(longLine + "\ntail\n"))
  scanner.Buffer(make([]byte, 0, 16), 16) // small max on purpose
  for scanner.Scan() {
    fmt.Println("scanned:", scanner.Text())
  }
  fmt.Println("scanner.Err():", scanner.Err())

  // 4. bufio.Writer needs Flush
  var out bytes.Buffer
  w := bufio.NewWriter(&out)
  fmt.Fprint(w, "buffered, not flushed yet")
  fmt.Println("before Flush, out.Len():", out.Len())
  w.Flush()
  fmt.Println("after Flush, out.Len():", out.Len())

  // 5. io.TeeReader mirrors what it reads into a side buffer
  var tee bytes.Buffer
  tr := io.TeeReader(strings.NewReader("mirrored"), &tee)
  got, _ := io.ReadAll(tr)
  fmt.Printf("TeeReader read=%q sideBuffer=%q\n", got, tee.String())

  // 6. io.NopCloser satisfies io.ReadCloser for a plain Reader
  var rc io.ReadCloser = io.NopCloser(strings.NewReader("wrapped"))
  data, _ := io.ReadAll(rc)
  fmt.Println("NopCloser Close err:", rc.Close(), "data:", string(data))
}

Real output from this program on Go 1.26.5:

Read: n=3 err=<nil> data="hel"
Read: n=3 err=EOF data="lo!"
ReadFull: n=6 err=<nil> data="hello!"
ReadFull short: n=2 err=unexpected EOF
scanner.Err(): bufio.Scanner: token too long
before Flush, out.Len(): 0
after Flush, out.Len(): 25
TeeReader read="mirrored" sideBuffer="mirrored"
NopCloser Close err: <nil> data: wrapped

Note the scanner loop above: scanned: never prints. The 20-byte line exceeds the 16-byte max on the very first token, so Scan() returns false immediately and Err() reports bufio.Scanner: token too long — the "tail" line is never seen.

The Reader/Writer pipeline

flowchart LR
  File["os.File / net.Conn"] --> BR["bufio.Reader"]
  BR --> Scanner["bufio.Scanner"]
  Scanner --> App["your code"]
  App2["your code"] --> BW["bufio.Writer"]
  BW -- "Flush()" --> Out["os.File / net.Conn"]
  Src["io.Reader"] -- "io.Copy" --> Dst["io.Writer"]
  Src -- "io.TeeReader" --> Side["side io.Writer"]

Reading batches through bufio, writing needs an explicit Flush, TeeReader forks a copy

bytes.Buffer must not be copied

bytes.Buffer holds its data in an internal slice; copying a bytes.Buffer by value (assigning it, passing it by value, or storing it in a slice you then append to) copies that slice header without copying a mutex or lock — but the two copies now alias the same backing array while each tracks its own read/write offsets independently, so writes and reads on one copy corrupt what the other sees. The package doc says so explicitly: “The zero value for Buffer is an empty buffer ready to use… Do not copy a non-zero Buffer.”

Unlike sync.Mutex, there is no go vet copylocks-style check for this — bytes.Buffer does not embed a noCopy marker, so the compiler and go vet stay silent. Always pass *bytes.Buffer, never bytes.Buffer by value, once it holds data.

Mental Model

flowchart LR
  subgraph R["io.Reader interface"]
    Src["Source"] --> Read["Read(p []byte)"] --> Buf["buffer p"] --> Consumer["consumer"]
  end
  subgraph C["Chaining readers"]
    F["os.File (raw)"] --> BRd["bufio.NewReader (buffered)"] --> GZ["gzip.NewReader (decompressed)"] --> JD["json.NewDecoder (decoded)"]
  end
  subgraph W["bufio.Writer batching"]
    Small["small writes (many syscalls)"] --> WB["buffer 4 KB"] --> Flush["one Flush (one syscall)"] --> Out["os.File"]
  end

Reader pipeline, reader chaining, and Writer batching

Trade-offs

What you get	Benefit	Cost
bufio.Reader	Fewer syscalls (batches reads)	Extra memory for the buffer
bufio.Writer	Fewer syscalls (batches writes)	Must call Flush() before close
io.Reader chain	Composable, zero-copy pipeline	Error handling gets complex when the chain is long
io.Copy	Efficient transfer between Reader and Writer	Cannot inspect the data in transit

Common Misconceptions

• An unbuffered read is fast enough — every Read() call with no buffer is one syscall — very slow for many small reads.
• bufio.Writer flushes automatically — it does not; you must call Flush() yourself before closing or sending the response, or the data can be lost.
• io.Reader always reads exactly n bytes — Read() may return fewer than n bytes; loop until you have what you need when you require an exact read.
• strings.NewReader and bytes.NewReader are the same — strings.NewReader is for a string literal; bytes.NewReader is for a []byte you already have.

💡 Real-world example

Go HTTP servers: net/http automatically wraps the TCP connection in a bufio.Reader — request parsing does not need a syscall per byte.

🟦 Program to io.Reader and io.Writer, not to concrete types

Write your functions to accept io.Reader or io.Writer rather than *os.File or *http.Request. This single habit makes your code universally testable — pass a strings.NewReader in tests, a real file in production, and an HTTP body in handlers, all without touching the function. It also means your function participates in the entire ecosystem of Go I/O composition: bufio.NewReader, io.MultiReader, io.LimitReader, gzip.NewReader, and so on.

Reading AI-generated code

AI assistants write io/bufio code that compiles and looks reasonable far more often than code that is actually correct on the edges. Check for these before trusting a review:

• Does a bufio.Scanner loop check scanner.Err() after the loop, or does it treat false as always meaning clean EOF?
• If the input could contain a line longer than 64 KB, did it call scanner.Buffer(...) to raise the limit, or will it silently truncate?
• Does anything written through a bufio.Writer get Flush()ed before the function returns or the response is sent?
• Is f.Close()’s return value checked when f was written to, not just when it was read from?
• Does a custom Read implementation assume the caller always gets a full buffer, or does the caller correctly loop / use io.ReadFull?
• Is a bytes.Buffer ever copied by value (passed as an argument, stored in a slice, returned from a function) after it has data in it?

🐛 Spot the bug

func loadConfig(path string) (map[string]string, error) {
  f, err := os.Open(path)
  if err != nil {
    return nil, err
  }
  defer f.Close()

  cfg := map[string]string{}
  scanner := bufio.NewScanner(f)
  for scanner.Scan() {
    k, v, _ := strings.Cut(scanner.Text(), "=")
    cfg[k] = v
  }
  return cfg, nil
}

Show the answer

The loop never checks scanner.Err(). If one line in path is longer than the default 64 KB limit — a long value, a stray binary file — Scan() returns false, the loop ends, and loadConfig returns a partial config with a nil error, as if the file had been read completely. Fix it by checking the error after the loop:

if err := scanner.Err(); err != nil {
    return nil, fmt.Errorf("scan %s: %w", path, err)
}
return cfg, nil

If lines longer than 64 KB are expected and valid, call scanner.Buffer(make([]byte, 0, 64*1024), maxLineSize) before the loop instead of just detecting the failure.

What does io.ReadAll(r) return?

string, error []byte, error int, error io.Reader, error

After bufio.Scanner.Scan() returns false, what must you check?

scanner.Text() for any remaining data scanner.Bytes() for raw bytes scanner.Err() to distinguish EOF from a real error Nothing — false always means clean EOF

Which type implements both io.Reader and io.Writer?

strings.Reader bytes.Buffer bufio.Scanner io.Discard

Why accept io.Reader in a function instead of *os.File?

*os.File causes memory leaks on Linux io.Reader is faster than *os.File Accepting io.Reader makes the function work with any source: files, buffers, strings, network — without code changes Go does not allow *os.File as a function parameter

What does this print?

sc := bufio.NewScanner(strings.NewReader(strings.Repeat("x", 100000)))
sc.Scan()
fmt.Println(sc.Err())

<nil> EOF bufio.Scanner: token too long It panics

What does this print, and why?

var out bytes.Buffer
w := bufio.NewWriter(&out)
fmt.Fprint(w, "hi")
fmt.Println(out.Len())

2 — bufio.Writer writes through immediately 0 — nothing reaches out until w.Flush() is called It does not compile — Fprint needs an io.Writer, not a bufio.Writer -1 — out is unset until Close

Submit

Mark as complete