Generators

What is a generator?

A generator is a function that can pause its execution and resume from where it left off. Instead of computing all values and returning them in a list, a generator produces values one at a time — only when asked. This is called lazy evaluation.

The key ingredient is the yield keyword. When Python encounters yield, it pauses the function, hands the value to the caller, and freezes the local state (variables, the instruction pointer). The next call to next() resumes from that exact line.

def count_up(start: int, stop: int):
    current = start
    while current <= stop:
        yield current       # pause here and emit current
        current += 1        # resume here on the next next() call

gen = count_up(1, 4)
print(next(gen))   # 1
print(next(gen))   # 2
print(next(gen))   # 3
print(next(gen))   # 4
# next(gen) would now raise StopIteration

Generator functions vs regular functions

Aspect	Regular function	Generator function
Returns	Once, with return	Many times, with yield
State	Discarded after return	Frozen between yields
Memory	Builds full result	One value at a time
Type	The return value	A generator object

A function becomes a generator function the moment it contains even one yield statement. Calling it does not execute any code — it returns a generator object immediately.

iter() and next()

Every generator object implements the iterator protocol: it has __iter__() and __next__() methods.

• next(gen) advances to the next yield and returns that value.
• When there are no more values, next(gen) raises StopIteration.
• A for loop calls next() automatically and catches StopIteration to stop.

def three_words():
    yield "Python"
    yield "is"
    yield "expressive"

# Equivalent — for loop handles next() and StopIteration for you
for word in three_words():
    print(word)

Generators are exhausted once

A generator object is a one-shot iterator: once every yield has run and the function returns, that object is permanently empty. Calling the generator function again produces a brand-new generator object — but iterating the same, already-exhausted object a second time silently produces nothing at all, with no error:

def three_words():
    yield "Python"
    yield "is"
    yield "expressive"

gen = three_words()
print([w for w in gen])   # ['Python', 'is', 'expressive']
print([w for w in gen])   # []  — same object, already exhausted, no error raised

This is different from a list, which you can loop over any number of times. If a function needs to iterate a value more than once, either pass a list (or re-call the generator function to get a fresh generator), not a spent generator object.

yield from: delegating to a sub-generator

yield from hands control to another iterable, forwarding every value it yields as if the outer generator had yielded them directly — and it also evaluates to the sub-generator’s return value:

def inner():
    yield 1
    yield 2
    return "inner done"

def outer():
    result = yield from inner()   # forwards 1, then 2
    print("inner returned:", result)
    yield 3

print(list(outer()))
# inner returned: inner done
# [1, 2, 3]

yield from is the idiomatic way to compose generators — it is shorter and correctly forwards .send(), .throw(), and the sub-generator’s return value, none of which a manual for x in inner(): yield x loop does for you.

.send(): two-way communication

yield is not just an exit — it is also an expression that receives whatever value is passed to .send(). The first next() (or .send(None)) is required to advance the generator to its first yield, priming it, before any real value can be sent in:

def echo_upper():
    received = None
    while True:
        received = yield received       # yield is an expression here
        if received is not None:
            received = received.upper()

gen = echo_upper()
print(next(gen))             # None — priming call, runs up to the first yield
print(gen.send("hello"))     # HELLO
print(gen.send("world"))     # WORLD

return inside a generator and StopIteration.value

A return statement inside a generator function does not produce a value the normal way — it still raises StopIteration, but attaches the returned value to that exception’s .value attribute. A plain for loop never sees it (it just stops), but manual iteration or yield from can retrieve it:

def summing():
    total = 0
    while True:
        x = yield
        if x is None:
            return total
        total += x

gen = summing()
next(gen)
gen.send(10)
gen.send(20)
try:
    gen.send(None)
except StopIteration as e:
    print(e.value)   # 30

.close() and finally

Calling .close() on a generator raises GeneratorExit inside it at the point where it is currently paused — this is exactly what lets a try/finally block around a yield run its cleanup, the same guarantee a with block gives a context manager:

def resource_gen():
    print("acquiring resource")
    try:
        yield 1
        yield 2
    finally:
        print("releasing resource")

gen = resource_gen()
print(next(gen))   # acquiring resource \n 1
gen.close()         # releasing resource

A generator that is garbage-collected before being exhausted also has .close() called on it automatically, so finally blocks still run — but relying on garbage collection timing for cleanup is fragile; call .close() explicitly (or drive the generator through a with contextlib.closing(gen): block) when early termination is expected.

Type hints for generators

A generator used only for its yielded values can be annotated as Iterator[YieldType]. When .send() or a return value matter too, the full three-parameter Generator[YieldType, SendType, ReturnType] says exactly what flows in both directions:

from typing import Generator, Iterator

def count_up(start: int, stop: int) -> Iterator[int]:
    current = start
    while current <= stop:
        yield current
        current += 1

def echo_typed() -> Generator[str, str, str]:
    received = ""
    while True:
        received = yield received.upper()
        if received == "STOP":
            return "done"

Iterator[int] is shorthand for “yields int, does not care what is sent in, does not care about the return value” — use it unless the generator genuinely uses .send() or a meaningful return.

Generators vs lists: memory

A generator does not materialize its values — it holds only enough state to produce the next one. sys.getsizeof makes the difference concrete: a generator’s footprint stays constant no matter how many values it will eventually produce, while a list’s footprint grows with every element:

import sys

big_list = [x for x in range(100_000)]
big_gen = (x for x in range(100_000))
print(sys.getsizeof(big_list))   # 800984 (bytes; grows with element count)
print(sys.getsizeof(big_gen))    # 200 (bytes; constant, regardless of range size)

This is why sum(x * x for x in range(10_000_000)) does not need to build a ten-million-element list first — the generator expression feeds sum() one value at a time.

itertools.chain and itertools.islice

itertools.islice takes a bounded slice of any iterable — including an infinite generator — without ever calling list() on it first, which would never return. itertools.chain iterates several iterables back to back as if they were one, without concatenating them into a new sequence:

from itertools import islice, chain

def fibonacci():
    a, b = 0, 1
    while True:        # infinite loop — but never blocks
        yield a
        a, b = b, a + b

first_5 = list(islice(fibonacci(), 5))
print(first_5)   # [0, 1, 1, 2, 3]

combined = list(chain([1, 2], (3, 4), range(5, 7)))
print(combined)  # [1, 2, 3, 4, 5, 6]

Generator expressions

Just as list comprehensions build lists, generator expressions build generators — using parentheses instead of brackets. They are memory-efficient because no list is materialised.

squares_list = [x ** 2 for x in range(1_000_000)]   # allocates a list
squares_gen  = (x ** 2 for x in range(1_000_000))   # allocates almost nothing

# Consume lazily
for sq in squares_gen:
    if sq > 25:
        break

Generator expressions compose naturally with built-ins that accept iterables:

total = sum(x ** 2 for x in range(10))   # no brackets needed inside sum()

Infinite generators

Because generators are lazy, you can define sequences that never end — as long as the consumer stops asking.

def fibonacci():
    a, b = 0, 1
    while True:        # infinite loop — but never blocks
        yield a
        a, b = b, a + b

Full runnable demo

Run ▸ Open in an online REPL

from itertools import islice

def fibonacci():
    """Infinite Fibonacci sequence generator."""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Take only the first 10 values — safe because islice stops early
first_10 = list(islice(fibonacci(), 10))
print("First 10 Fibonacci numbers:")
print(first_10)

# Generator expression: squares of even numbers
even_squares = (x * x for x in range(1, 11) if x % 2 == 0)
print("\nSquares of even numbers 1-10:")
for val in even_squares:
    print(val, end=" ")
print()

# Demonstrate single-use nature
gen = (x * 3 for x in range(5))
print("\nFirst pass:", list(gen))
print("Second pass:", list(gen))   # exhausted — empty list

# Manual next() / StopIteration
counter = iter(range(3))
print("\nManual iteration:")
print(next(counter))
print(next(counter))
print(next(counter))
try:
    print(next(counter))
except StopIteration:
    print("StopIteration raised — generator exhausted")

Depth demo: yield from, .send(), .close(), and memory

Run ▸ Open in an online REPL

import sys
from itertools import islice, chain
from typing import Generator, Iterator


# --- exhausted once: second loop is empty ---
def three_words():
    yield "Python"
    yield "is"
    yield "expressive"

gen = three_words()
print("--- exhausted once ---")
print("first pass:", [w for w in gen])
print("second pass:", [w for w in gen])

# --- yield from ---
def inner():
    yield 1
    yield 2
    return "inner done"

def outer_yield_from():
    result = yield from inner()
    print("inner returned:", result)
    yield 3

print("\n--- yield from ---")
print(list(outer_yield_from()))

# --- .send() ---
def echo_upper():
    received = None
    while True:
        received = yield received
        if received is not None:
            received = received.upper()

print("\n--- .send() ---")
gen2 = echo_upper()
print(next(gen2))          # prime the generator, prints None
print(gen2.send("hello"))  # HELLO
print(gen2.send("world"))  # WORLD

# --- return value via StopIteration.value ---
def summing():
    total = 0
    while True:
        x = yield
        if x is None:
            return total
        total += x

print("\n--- return + StopIteration.value ---")
gen3 = summing()
next(gen3)
gen3.send(10)
gen3.send(20)
try:
    gen3.send(None)
except StopIteration as e:
    print("StopIteration.value:", e.value)

# --- .close() / finally ---
def resource_gen():
    print("acquiring resource")
    try:
        yield 1
        yield 2
    finally:
        print("releasing resource")

print("\n--- .close() / finally ---")
gen4 = resource_gen()
print(next(gen4))
gen4.close()
try:
    next(gen4)
except StopIteration:
    print("StopIteration after close()")

# --- type hints: Iterator[int] / Generator[Y, S, R] ---
def count_up(start: int, stop: int) -> Iterator[int]:
    current = start
    while current <= stop:
        yield current
        current += 1

def echo_typed() -> Generator[str, str, str]:
    received = ""
    while True:
        received = yield received.upper()
        if received == "STOP":
            return "done"

print("\n--- typed generators still run the same ---")
print(list(count_up(1, 3)))

# --- generator vs list memory ---
print("\n--- memory: list vs generator ---")
big_list = [x for x in range(100_000)]
big_gen = (x for x in range(100_000))
print("list getsizeof:", sys.getsizeof(big_list))
print("generator getsizeof:", sys.getsizeof(big_gen))

# --- itertools.chain / islice ---
print("\n--- itertools.chain / islice ---")
def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

first_5 = list(islice(fibonacci(), 5))
print("first 5 fib:", first_5)

combined = list(chain([1, 2], (3, 4), range(5, 7)))
print("chained:", combined)

🐍 Generators are lazy and single-use

Two important properties to remember:

Lazy: A generator computes the next value only when next() is called. A generator expression (heavy_compute(x) for x in data) does zero work until you start iterating.

Single-use: Once a generator is exhausted, it stays exhausted. Iterating over it a second time produces nothing. If you need to iterate multiple times, convert to a list or call the generator function again to get a fresh generator.

gen = (x for x in range(3))
print(list(gen))   # [0, 1, 2]
print(list(gen))   # []  — already exhausted

Mental Model

A generator is a tiny state machine with exactly these states — it starts Created, and every next()/.send() moves it forward until it reaches Returned (raising StopIteration with the return value) or Closed (via .close(), running any finally block on the way):

stateDiagram-v2
  [*] --> Created: call the generator function
  Created --> Suspended: first next()/send(None)
  Suspended --> Suspended: next()/send(v) resumes to the next yield
  Suspended --> Returned: function body reaches return or falls off the end
  Suspended --> Closed: .close() raises GeneratorExit at the paused yield
  Returned --> [*]: StopIteration(value=...)
  Closed --> [*]: finally blocks have already run

Generator lifecycle: Created -> Suspended (looping on yield) -> Returned or Closed

Trade-offs

What you get	Benefit	Cost
Generator (lazy)	Constant memory regardless of how many values are produced (sys.getsizeof stays ~200 bytes)	Single-use — a second iteration silently produces nothing, no error
List (eager)	Re-iterable any number of times, supports indexing and len()	Memory grows with every element — must fit in memory all at once
yield from	Forwards values, .send(), and the sub-generator’s return value correctly	Only meaningful when delegating to another generator/iterable, not a plain loop body
.send() two-way generators	Lets a generator receive values, not just produce them (coroutine-style)	Requires a priming next() first, and the flow is harder to follow than a plain producer generator
itertools.islice/chain on generators	Take a bounded slice of, or combine, infinite/lazy sequences without materializing them	Consumes the underlying generator — you cannot “rewind” past what islice already pulled

Common Misconceptions

• “Iterating an exhausted generator raises an error” — fact: it silently produces zero values and no exception — no StopIteration leaks out of a for loop or list comprehension either way, so this is easy to miss in code review.
• “yield from inner() is the same as for x in inner(): yield x” — fact: yield from additionally forwards .send()/.throw() into inner() and evaluates to inner()’s return value; a manual loop does neither.
• “A generator’s return value is what a for loop gets at the end” — fact: a for loop never sees it — the value is attached to StopIteration.value, retrievable only via manual iteration or yield from.
• “.close() just stops the generator with no cleanup” — fact: it raises GeneratorExit at the paused yield, so any try/finally around it still runs, exactly like closing a context manager.
• “A generator expression is basically a list, just written with ()” — fact: sys.getsizeof shows the generator’s footprint stays constant (~200 bytes) no matter the range size, while the list’s footprint scales with element count — the generator has not computed or stored any of the values yet.

Reading AI-generated code

• Is a generator (not a list) iterated more than once — in two separate loops, or passed to two functions that each loop over it? The second consumer silently gets nothing.
• Does code delegate to a sub-generator with a manual for x in sub(): yield x where the sub-generator’s return value or .send() behavior actually matters? That should be yield from sub().
• Does a generator’s return <value> look like it is supposed to reach a for loop? It only reaches StopIteration.value — a plain for loop discards it silently.
• Is a resource (file, lock, connection) acquired before a yield with no try/finally? If the generator is abandoned early, .close() (or garbage collection) still raises GeneratorExit at that yield, and without finally the resource is never released.
• Is list(...) called on an infinite generator, or on one with no upper bound in practice? It should be wrapped in itertools.islice(..., n) first.

🐛 Spot the bug

def read_lines(path):
  with open(path, encoding="utf-8") as f:
      for line in f:
          yield line.strip()


def process(path):
  lines = read_lines(path)
  total_lines = sum(1 for _ in lines)
  non_empty = [line for line in lines if line]
  print(f"{path}: {total_lines} lines, {len(non_empty)} non-empty")

Show the answer

lines is a generator, and sum(1 for _ in lines) fully consumes it while counting. By the time the list comprehension [line for line in lines if line] runs, lines is already exhausted, so non_empty is always [] regardless of the file’s contents — no exception is raised, so this bug ships silently and only shows up as a wrong count. Verified on Python 3.14: printing list(lines) a second time after the sum(...) call returns []. The fix is to only iterate the generator once — read it into a list first (lines = list(read_lines(path))) if it must be scanned more than once, or compute both total_lines and non_empty in a single pass over the generator.

What happens when Python encounters yield inside a function?

The function raises an exception The function exits permanently The function pauses, emits the value, and resumes on the next next() call The function runs in a separate thread

What does this print? def words(): yield "a" yield "b" gen = words() first = list(gen) second = list(gen) print(first, second)

['a', 'b'] ['a', 'b'] ['a', 'b'] [] [] ['a', 'b'] Raises StopIteration on the second list(gen)

What does this print? def gen(): x = yield 1 y = yield x return y g = gen() print(next(g)) print(g.send("A")) try: g.send("B") except StopIteration as e: print(e.value)

1, A, B 1, None, B 1, A, None TypeError on the first send()

Why is yield from inner() generally preferred over for x in inner(): yield x?

It runs measurably faster in every case It also forwards .send()/.throw() into inner() and evaluates to inner()'s return value, which the manual loop does not The manual loop form is a SyntaxError in Python 3.14 yield from can only be used with infinite generators

A generator has a try: ... yield ... finally: cleanup() block. If the generator is garbage-collected before being fully iterated, what happens to cleanup()?

It never runs — finally blocks are skipped when a generator is abandoned Python raises GeneratorExit at the paused yield (via an implicit .close()), so finally still runs It only runs if the generator was iterated with a for loop, not manually It raises a ResourceWarning instead of running cleanup()

What does sys.getsizeof show about [x for x in range(1_000_000)] versus (x for x in range(1_000_000))?

They are the same size, since both eventually visit the same number of elements The list's size scales with the number of elements; the generator's size stays roughly constant regardless of range size The generator is always larger because it stores extra state for lazy evaluation sys.getsizeof cannot measure generator objects

Submit

Mark as complete