Multiprocessing

Why processes, not threads?

As covered in the GIL lesson, CPython’s Global Interpreter Lock prevents multiple threads from executing Python bytecode in parallel. For CPU-bound work — numerical computation, image processing, data parsing — threads provide no speedup.

multiprocessing solves this by spawning separate OS processes. Each process has its own Python interpreter, its own GIL, and its own memory space. True parallelism becomes possible: a machine with 8 cores can run 8 Python processes simultaneously.

The if __name__ == "__main__" guard

On macOS and Windows, the default process start method is spawn — the child process imports the parent’s module from scratch. Without the guard, the module-level code (including the Pool(...) call) re-executes in every child, causing an infinite process-spawning loop.

from multiprocessing import Pool

def square(x: int) -> int:
    return x * x

if __name__ == "__main__":        # REQUIRED on macOS/Windows
    with Pool(processes=4) as pool:
        results = pool.map(square, range(10))
    print(results)
    # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Since Python 3.14, Unix platforms other than macOS default to forkserver instead of fork (the default from 3.3–3.13) — see the 3.14 What’s New entry for multiprocessing and gh-84559. macOS and Windows continue to default to spawn. Because every platform can now re-run your module in a fresh process on pool startup, keep the guard everywhere, not just on macOS/Windows.

Pool.map — parallel map over an iterable

Pool.map(fn, iterable) distributes the items across workers and returns results in input order. It blocks until all items are processed.

from multiprocessing import Pool
import math

def is_prime(n: int) -> bool:
    if n < 2:
        return False
    for i in range(2, int(math.sqrt(n)) + 1):
        if n % i == 0:
            return False
    return True

if __name__ == "__main__":
    candidates = list(range(1, 1_000_001))
    with Pool() as pool:              # Pool() defaults to cpu_count() workers
        flags = pool.map(is_prime, candidates)
    primes = [n for n, flag in zip(candidates, flags) if flag]
    print(f"Found {len(primes)} primes up to 1,000,000")

Pool.starmap — multiple arguments per call

When the worker function takes more than one argument, use starmap with an iterable of tuples:

from multiprocessing import Pool

def power(base: int, exp: int) -> int:
    return base ** exp

if __name__ == "__main__":
    args = [(2, 10), (3, 5), (5, 4), (7, 3)]
    with Pool(processes=4) as pool:
        results = pool.starmap(power, args)
    print(results)
    # [1024, 243, 625, 343]

ProcessPoolExecutor — the modern high-level API

concurrent.futures.ProcessPoolExecutor provides the same interface as ThreadPoolExecutor, making it easy to switch between thread-based and process-based parallelism.

from concurrent.futures import ProcessPoolExecutor
import math

def cpu_work(n: int) -> float:
    # Simulate CPU-intensive computation
    return sum(math.sin(i) for i in range(n))

if __name__ == "__main__":
    inputs = [3_000_000] * 8   # 8 CPU-bound tasks, large enough that compute dominates pool startup

    # Sequential baseline
    import time
    start = time.perf_counter()
    seq_results = [cpu_work(n) for n in inputs]
    seq_time = time.perf_counter() - start

    # Parallel with ProcessPoolExecutor
    start = time.perf_counter()
    with ProcessPoolExecutor() as executor:
        par_results = list(executor.map(cpu_work, inputs))
    par_time = time.perf_counter() - start

    print(f"Sequential: {seq_time:.2f}s")
    print(f"Parallel:   {par_time:.2f}s")
    print(f"Speedup:    {seq_time / par_time:.1f}x")

Measured three times on an 11-core machine (python3.14): 3.0x, 3.3x, 3.6x — nowhere near a clean 8x, and that is expected, not a bug in the measurement. The smaller 500_000-element version used to show here spent so much of its time on pool startup (ProcessPoolExecutor spawns fresh interpreter processes) that the speedup barely cleared 1x; even with a workload six times larger, process startup, pickling the arguments/results, and scheduling across a mix of performance and efficiency cores all eat into the theoretical maximum. Numbers here are machine-dependent — measure on your own hardware before quoting a specific multiplier.

Picklability: why a lambda breaks ProcessPoolExecutor

Every argument, and the callable itself, has to travel from the parent process to a worker process as bytes — multiprocessing and ProcessPoolExecutor both do this with pickle. A lambda, or any function defined locally inside another function, cannot be pickled, because pickling a function stores a reference to it by name (module and qualified name) for the worker to re-import — and a lambda’s name is literally <lambda>, not something importable:

from concurrent.futures import ProcessPoolExecutor

def main() -> None:
    with ProcessPoolExecutor(max_workers=2) as pool:
        fut = pool.submit(lambda x: x * x, 5)   # lambdas cannot be pickled
        try:
            print(fut.result())
        except Exception as e:
            print(f"{type(e).__name__}: {e}")

if __name__ == "__main__":
    main()

PicklingError: Can't pickle local object <function main.<locals>.<lambda> at 0x...>

The same problem hits a plain def nested inside another function — “local object” in the error covers both. The fix is always the same: define the callable at module level (or as a staticmethod on a class defined at module level) so its qualified name resolves to something the worker process can import on its own. A functools.partial wrapping a module-level function is picklable, so it is the usual way to pass extra fixed arguments without writing a lambda.

Inter-process communication

Processes do not share memory. To exchange data between them, use multiprocessing.Queue or multiprocessing.Pipe.

from multiprocessing import Process, Queue

def producer(q: Queue, items: list) -> None:
    for item in items:
        q.put(item)
    q.put(None)    # sentinel

def consumer(q: Queue) -> None:
    while True:
        item = q.get()
        if item is None:
            break
        print(f"consumed: {item}")

if __name__ == "__main__":
    q: Queue = Queue()
    p1 = Process(target=producer, args=(q, [1, 2, 3, 4, 5]))
    p2 = Process(target=consumer, args=(q,))
    p1.start(); p2.start()
    p1.join();  p2.join()

Shared memory in Python 3.8+

multiprocessing.shared_memory.SharedMemory allows processes to access a shared block of memory without serialization overhead — useful for large NumPy arrays.

from multiprocessing import shared_memory
import numpy as np

# Create a shared memory block
shm = shared_memory.SharedMemory(create=True, size=400)   # 100 float32 values
arr = np.ndarray((100,), dtype=np.float32, buffer=shm.buf)
arr[:] = np.arange(100, dtype=np.float32)
print(arr[:5])    # [0. 1. 2. 3. 4.]
shm.close()
shm.unlink()

Subinterpreters: concurrent.interpreters (Python 3.14)

PEP 734 exposes subinterpreters to Python code. Each interpreter gets its own GIL — that groundwork landed in 3.12 — so they run on separate cores like processes, but they live inside one OS process, which makes startup far cheaper than fork or spawn.

from concurrent import interpreters

interp = interpreters.create()
interp.exec("print(sum(range(1_000_000)))")
interp.close()

There is a matching executor for pool-shaped work:

from concurrent.futures import InterpreterPoolExecutor

with InterpreterPoolExecutor(max_workers=4) as pool:
    print(list(pool.map(str, range(4))))

The trade-off is the boundary: interpreters do not share objects, so arguments and results have to cross as serialisable data, exactly like multiprocessing. What you save is process startup time and memory.

Processes vs threads: decision guide

Criterion	multiprocessing	threading	asyncio
CPU-bound work	Best	No benefit (GIL)	Wrong tool
I/O-bound, blocking libs	Overkill	Good	run_in_executor
I/O-bound, async libs	Overkill	OK	Best
Memory overhead	High (full copy)	Low (shared)	Minimal
Communication	Queue / Pipe	Shared state + Lock	Channels / queues
Startup time	Slow (fork/spawn)	Fast	Zero

🐍 Run these examples locally

This lesson has no in-browser runner, and the browser sandbox has no real processes anyway. Run the examples above locally:

python3 multiprocessing_demo.py

Remember the if __name__ == "__main__" guard — without it the script loops forever on macOS and Windows (spawn); see the guard/start-method section above for how this changed in Python 3.14.

Mental Model

A Pool is a small factory with its own separate buildings (processes), not extra workstations in your building (threads). Nothing walks between buildings except what you deliberately mail — pickle is the mail service, and it can only ship things with a return address someone else’s building can look up (a module-level name). A lambda has no such address; a Queue or Pipe is how you actually mail data back and forth once the buildings are up and running.

flowchart LR
P["Parent process"] -->|"pickle(fn, args)"| W1["Worker process 1"]
P -->|"pickle(fn, args)"| W2["Worker process 2"]
W1 -->|"pickle(result)"| P
W2 -->|"pickle(result)"| P
L["lambda / local def"] -.->|"PicklingError — no importable name"| P

Every argument, callable, and result crosses the process boundary as pickled bytes

Trade-offs

What you get	Benefit	Cost
Separate process per worker	True parallelism, own GIL, immune to one worker crashing the rest	Full interpreter startup cost per process; no shared memory by default
pickle-based argument passing	Works transparently for most data	Callables must be importable by name — no lambda, no local closures
shared_memory	Skips serialization for large arrays	Manual lifecycle management (close()/unlink()); no automatic safety net
Subinterpreters (3.14)	Cheaper startup than a full process	Still no shared objects across the boundary — same serialization cost as data crosses
forkserver default (non-macOS Unix, 3.14+)	More predictable than fork around threads/locks held at fork time	Still needs the __main__ guard, unlike old-style fork

Common Misconceptions

• “multiprocessing shares memory like threading does.” It does not, by default — every argument and result is pickled and sent across a pipe. shared_memory and Manager exist specifically because plain multiprocessing does not share state.
• “Any function works with Pool.map or ProcessPoolExecutor.submit.” Only functions the worker process can import by name work — module-level functions and picklable callables. A lambda or a function defined inside another function fails immediately with PicklingError. Nothing about a lambda’s logic is the problem; it is purely that it has no importable name. Passing extra arguments to a module-level function is the usual reason someone reaches for a lambda here — functools.partial does the same job and is picklable.
• “The if __name__ guard only matters on Windows.” As of Python 3.14, non-macOS Unix also defaults to forkserver, which re-imports the module too — the guard is needed on every platform now, not just Windows/macOS.
• “More processes always means more speed.” Beyond the number of CPU cores, extra processes add scheduling and memory overhead without additional parallel compute capacity; for small or I/O-heavy workloads, process startup cost alone can erase any benefit — see the measured pool startup vs. workload trade-off above.

Reading AI-generated code

Before trusting generated multiprocessing code, check for these:

• Does Pool(...), Process(...), or ProcessPoolExecutor(...) construction happen at module level, outside an if __name__ == "__main__": guard? On spawn/forkserver platforms this re-imports the module in every worker.
• Is a lambda, or a function defined inside another function, passed to pool.map, pool.submit, or as a Process target? It will fail with PicklingError the moment it actually runs in a worker — not at the point it was written.
• Is the workload actually CPU-bound? Reaching for multiprocessing for an I/O-bound task adds serialization and process-startup overhead for no parallelism benefit asyncio/threading wouldn’t already give more cheaply.
• Does code assume child processes share module-level mutable state with the parent (a shared cache, a counter) without Manager, shared_memory, or a Queue? Each process has its own copy.

🐛 Spot the bug

from concurrent.futures import ProcessPoolExecutor

def main() -> None:
  multiplier = 3
  with ProcessPoolExecutor() as pool:
      results = list(pool.map(lambda x: x * multiplier, range(5)))
  print(results)

if __name__ == "__main__":
  main()

Show the answer

This has the if __name__ == "__main__": guard in place, which fixes the most commonly taught multiprocessing bug — so it is easy to assume the rest is fine. But the lambda closing over multiplier fails immediately with PicklingError: Can't pickle local object <function main.<locals>.<lambda> at 0x...> the moment pool.map tries to send it to a worker process, guard or no guard. The fix is a module-level function plus functools.partial to bind multiplier: from functools import partial then pool.map(partial(multiply, multiplier), range(5)) with def multiply(m: int, x: int) -> int: return x * m defined at module scope.

🐍 If a worker process needs it, it must have a name

Anything crossing into a multiprocessing worker — the callable, its arguments, its return value — travels as pickled bytes and must be reconstructible by name on the other side. Module-level functions and simple data qualify; lambdas, local closures, and open file handles or sockets do not.

Why does multiprocessing achieve true CPU parallelism while threading does not in CPython?

Processes use a faster bytecode compiler than threads Each process has its own Python interpreter and GIL, so they run truly in parallel multiprocessing bypasses the OS scheduler that throttles threads multiprocessing uses C extensions which release the GIL automatically

This is missing the if __name__ == '__main__': guard. What happens when run on macOS or Windows?

from multiprocessing import Pool

def square(x):
    return x * x

with Pool(processes=4) as pool:
    results = pool.map(square, range(10))
print(results)

It is a Python style guideline with no functional effect -- the code runs fine Without it, the Pool cannot determine how many workers to create, so it raises a TypeError The spawn start method re-imports the module in every child process, which re-runs the Pool(...) call and spawns processes without end It prints results correctly but leaks memory

Which method distributes an iterable over a process pool and returns results in input order?

Pool.apply() Pool.map() Pool.run() Pool.scatter()

What mechanism should you use to share large NumPy arrays between processes without serialization overhead?

multiprocessing.Queue threading.shared_dict multiprocessing.shared_memory.SharedMemory pickle.dumps across a Pipe

What happens when this runs?

from concurrent.futures import ProcessPoolExecutor

def main():
    with ProcessPoolExecutor() as pool:
        fut = pool.submit(lambda x: x * x, 5)
        print(fut.result())

if __name__ == "__main__":
    main()

Prints 25 -- the guard fixes the only real issue here PicklingError -- lambdas have no importable name, so they cannot be sent to a worker process Runs correctly but slower than a module-level function Raises NameError inside the worker process

A CPU-bound benchmark using 8 ProcessPoolExecutor workers on an 11-core machine measures a 3.3x speedup, not 8x. What is the most likely honest explanation?

The measurement is wrong -- 8 processes should always give exactly 8x Process startup cost, argument/result pickling, and scheduling across a mix of core types all eat into the theoretical maximum ProcessPoolExecutor silently falls back to threads after the first few workers Python caps process-based speedup at 4x regardless of core count

Submit

Mark as complete