Python Multiprocessing vs Multithreading: Which One and Why

The short answer

Threads for work that waits. Processes for work that computes.

If your script spends its time on HTTP requests, database round trips or reading files, use threads — they will overlap and your wall-clock time drops. If it spends its time doing arithmetic in pure Python, threads will buy you nothing at all, and you need processes.

The reason is one lock. In CPython — the interpreter you get from python.org, the one nearly everyone runs — there is a mutex called the Global Interpreter Lock, or GIL. A mutex is just a lock that only one holder may have at a time. Only the thread holding the GIL may execute Python bytecode. Ten threads, ten cores, one lock: nine of them are parked.

Processes dodge this entirely. Each process gets a fresh Python interpreter with its own GIL, its own memory, its own everything. Four processes on four cores really do run four things at once. What you pay for that is startup time and the fact that nothing is shared — every argument and every return value has to be serialised and shipped across a pipe.

So the question is never "which is faster". It is: what is your program waiting on?

Note

There is a third model, asyncio, which handles very high I/O concurrency in a single thread using cooperative scheduling. It solves the same problem threads do, with different ergonomics. This article compares the two thread/process models only.

The GIL, precisely

The Python glossary defines the GIL as the mutex that allows only one thread to hold control of the Python interpreter and execute Python bytecode at any one time. It is a CPython implementation detail, not part of the Python language specification — another implementation is free not to have one.

It exists because CPython manages memory with reference counting — every object carries a count of how many names point at it, and that count is incremented and decremented constantly. Those updates are not thread-safe. Without the GIL, two threads touching the same object's refcount could corrupt it and either leak memory or free a live object. The GIL makes the whole object model implicitly safe with one lock instead of a lock per object, and it makes it easy to embed C libraries that were never written with threads in mind.

Threads do not hold the GIL forever. CPython asks the holding thread to drop it periodically — the default switch interval is 0.005 seconds (5 ms), readable with sys.getswitchinterval() and adjustable with sys.setswitchinterval(). That is what makes threads concurrent: they take turns. It is not what makes them parallel, because only one runs bytecode at a time regardless.

Here is the part people miss. The GIL is released around blocking I/O. When a thread calls into the OS to read a socket, read a file, or sleep, it hands the GIL back before it blocks and reacquires it when the data arrives. So while thread A waits on a slow API, threads B through Z are free to run. For I/O work, the GIL costs you almost nothing.

The same escape hatch is available to C extension modules, and the good ones use it. NumPy releases the GIL around many array computations. If your hot loop is a big matrix operation rather than a Python for loop, threads may already be parallelising it. Check before you reach for processes.

The plain consequence: N threads running a pure-Python arithmetic loop on N cores finish no faster than one thread, and often slightly slower, because you have added context switching to the bill. The threading docs say so directly and point you at multiprocessing or ProcessPoolExecutor for compute.

What multiprocessing actually costs

The multiprocessing module side-steps the GIL by using subprocesses. Real parallelism, real cores. Now the bill.

Nothing is shared. Arguments go in, results come out, and both are pickled — converted to bytes, sent down a pipe, rebuilt on the other side. Lambdas and functions defined inside other functions cannot be pickled, so they cannot be sent to workers. This is the number one surprise for people whose first ProcessPoolExecutor attempt dies with a pickling error.

Starting a worker is not free. There are three start methods:

Start method	How it works	Cost
fork	Duplicates the parent with os.fork(). POSIX only.	Cheapest. Unsafe if the parent has threads
spawn	Starts a fresh interpreter, re-imports the main module	Slowest; arguments must be picklable
forkserver	A single-threaded helper process forks workers on request	Middle ground, avoids forking a threaded parent

Defaults have moved, which is why code that works on your Linux box breaks on a colleague's Mac. spawn is the default on Windows and on macOS — macOS switched from fork to spawn in Python 3.8, so globals are no longer inherited by the child there. In Python 3.14 the default on other platforms (Linux included) changed from fork to forkserver. Part of the reason: since 3.12, calling os.fork() directly in a process that already has threads is deprecated and raises a DeprecationWarning, because the child can deadlock on a lock that was held by a thread that does not exist in the child.

Because spawn and forkserver re-import your main module in the child, you must guard the code that launches processes:

## worker.py
from concurrent.futures import ProcessPoolExecutor

def square(n: int) -> int:
    return n * n

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        print(list(pool.map(square, range(10))))

Without the if __name__ == "__main__": guard, the child re-imports worker.py, hits the pool creation again, starts more children, and you have fork-bombed yourself.

Common mistake

Assuming fork means memory is free. The child does inherit the parent's objects, but anything it needs to keep alive it must eventually account for in its own address space, and any state the two processes diverge on is duplicated rather than shared. Treat inherited data as a convenience, not as a guaranteed zero-cost share.

If you genuinely need shared state, multiprocessing gives you Queue and Pipe for messages, Value and Array for shared memory primitives, Manager for a server process holding real Python objects, and multiprocessing.shared_memory (added in 3.8) for a raw block of memory shared without pickling. Each of them costs either serialisation or IPC.

A worked example: where the crossover sits

Say you have 10,000 items. Take two versions of the job.

Version A — each item is an HTTP GET that takes 200 ms, almost all of it waiting on the network. Single-threaded, that is 10,000 × 0.2 s = 2,000 seconds, about 33 minutes. With a thread pool of 32, the waits overlap: roughly 2,000 / 32 ≈ 63 seconds as a best case. The GIL is released around the blocking part of each call, so the threads genuinely stack up; the request setup and response decoding still run under the GIL, so expect somewhat worse than the arithmetic suggests. Processes would also work here, but you would be paying 32 interpreter startups and pickling every response for zero additional benefit.

Version B — each item is 200 ms of pure-Python number crunching. Threads: still 2,000 seconds, because only one thread holds the GIL and runs bytecode; you have added switching overhead for nothing. On 8 cores with 8 processes: roughly 2,000 / 8 ≈ 250 seconds, plus startup and pickling.

Now shrink the work. Same 10,000 items, but each one takes 0.1 ms of computation. The total real work is one second. Send that to a process pool with Pool.imap(), whose default chunksize is 1, and every single item is pickled, pushed through a pipe, unpickled, computed in a tenth of a millisecond, pickled back, and shipped home. The IPC dwarfs the work and the parallel version is slower than the serial one. Raise chunksize so each trip carries hundreds of items and the overhead amortises — the multiprocessing docs note explicitly that a larger chunksize can make long iterables complete much faster. Pool.map() computes a chunk size for you; imap does not.

That is the whole trade-off in one place: processes win when per-item work is large relative to the cost of moving the item.

Switching is one line

ThreadPoolExecutor and ProcessPoolExecutor are both subclasses of concurrent.futures.Executor and expose the same submit, map and shutdown interface, so you can benchmark both by changing the class name — the same way Docker and Kubernetes solve different-sized problems with overlapping vocabulary, these two share an API but not a cost model.

## bench.py
import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def work(n: int) -> int:
    total = 0
    for i in range(n):
        total += i * i
    return total

def run(executor_cls, items):
    start = time.perf_counter()
    with executor_cls() as pool:
        results = list(pool.map(work, items))
    return time.perf_counter() - start, len(results)

if __name__ == "__main__":
    items = [400_000] * 16
    for cls in (ThreadPoolExecutor, ProcessPoolExecutor):
        elapsed, count = run(cls, items)
        print(f"{cls.__name__:22} {elapsed:.2f}s  ({count} results)")

pool.map yields results in submission order, and any exception a worker raised is re-raised when you pull that result out of the iterator.

Defaults are chosen to match intent, per the concurrent.futures docs:

Executor	Default max_workers
ThreadPoolExecutor	min(32, os.process_cpu_count() + 4)
ProcessPoolExecutor	os.process_cpu_count(); must be ≤ 61 on Windows
multiprocessing.Pool()	os.cpu_count()

The thread default is capped at 32 deliberately, so an I/O pool does not explode on a 128-core machine, and the +4 gives you a few extra threads for the waiting. That formula was os.cpu_count() * 5 before 3.8, and the count switched from os.cpu_count() to os.process_cpu_count() in 3.13 — which respects CPU affinity, so a container pinned to two cores gets two, not the host's full count. That distinction matters as much for right-sizing a pool as it does for choosing between Lambda and EC2.

Use the context-manager form so shutdown happens; Executor.shutdown() defaults to wait=True, and cancel_futures was added in 3.9. One failure mode to know: if a ProcessPoolExecutor worker dies abruptly — OOM killer, segfault in a C extension — you get BrokenProcessPool and the executor is permanently unusable. That is closer to a Kafka consumer dropping out of its group than to a thread raising an exception; there is no recovery in place, you rebuild the pool.

What free-threaded Python changes

PEP 703 made the GIL optional. A free-threaded build of CPython runs without it, so threads execute Python bytecode in parallel and the CPU-bound case for threads finally works.

It shipped experimentally in 3.13 as a separate build, offered as an optional component in the Windows and macOS installers, invoked as python3.13t and identified by the Py_GIL_DISABLED flag. PEP 779 moved it to officially supported in 3.14 — still an optional, separate build, not the default interpreter.

One honest caveat. C extension modules must be rebuilt and explicitly declare support; one that does not opt in causes the GIL to be re-enabled at import time, silently returning you to where you started.

3.14 also adds a third model: concurrent.interpreters (PEP 734) exposes multiple isolated interpreters in one process, each with its own GIL — the per-interpreter GIL machinery landed in 3.12 but was C-API only. There is an InterpreterPoolExecutor to go with it. It sits between threads and processes: real parallelism, no separate OS process.

If you are shipping on 3.11 or 3.12 today, none of this changes your decision. Threads for I/O, processes for CPU, measure both. Treat free-threading as something to test against, not something to ship on.