`multiprocessing`

The Python multiprocessing package allows you to run code in parallel by leveraging multiple processors on your machine, effectively sidestepping Python’s Global Interpreter Lock (GIL) to achieve true parallelism.

This package provides an interface similar to the threading module but uses processes instead of threads.

Here’s a quick example:

import multiprocessing

def worker(num):
    print(f"Worker: {num}")

if __name__ == "__main__":
    process = multiprocessing.Process(target=worker, args=(1,))
    process.start()
    process.join()

Running that script prints the worker’s message:

Worker: 1

Key Features

Spawns processes using an API similar to the threading module
Supports local concurrency and remote concurrency with multiprocessing.managers and explicit setups
Offers process pools for simple parallel task management
Provides shared data structures such as queues and pipes
Starts workers with the spawn, fork, or forkserver start method, defaulting to forkserver on POSIX and spawn on Windows and macOS as of Python 3.14, so fork is no longer the default on any platform

Frequently Used Classes and Functions

Object	Type	Description
`multiprocessing.Process`	Class	Spawns a new process
`multiprocessing.Pool`	Class	Manages a pool of worker processes
`multiprocessing.Queue`	Class	Provides a queue implementation for process-safe data sharing
`multiprocessing.Pipe`	Function	Returns a pair of connection objects
`multiprocessing.Lock`	Class	Provides a synchronization primitive for process-safe locking

Examples

Creating a process:

from multiprocessing import Process

def worker():
    print("Worker process")

if __name__ == "__main__":
    process = Process(target=worker)
    process.start()
    process.join()

Using a pool of worker processes to compute squares:

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool(4) as pool:
        print(pool.map(square, [1, 2, 3, 4]))

The pool returns the squares in the order you passed them in:

[1, 4, 9, 16]

Because the four calls to square() run at the same time rather than one after another, the pool finishes the batch in a fraction of the time. Drag the number of worker processes below to watch a batch of equal-length tasks fan out across them and finish in fewer rounds:

Adding processes keeps paying off until you run out of tasks, or CPU cores to run them on.

Common Use Cases

Running CPU-bound tasks in parallel
Speeding up tasks by utilizing multiple processors
Managing parallel execution of independent tasks

Real-World Example

Suppose you have a directory of images and you want to apply a computationally expensive transformation to each image in parallel. You can use the multiprocessing package to speed up this task:

import multiprocessing
import pathlib

def process_image(image_path):
    print(f"Processing {image_path}")

if __name__ == "__main__":
    image_dir = pathlib.Path("images")
    with multiprocessing.Pool() as pool:
        pool.map(process_image, list(image_dir.glob("*.jpg")))

In this example, the multiprocessing package helps you distribute the workload across multiple processes, significantly reducing the time needed to process all images in the directory.

Tutorial

Speed Up Your Python Program With Concurrency

In this tutorial, you'll explore concurrency in Python, including multi-threaded and asynchronous solutions for I/O-bound tasks, and multiprocessing for CPU-bound tasks. By the end of this tutorial, you'll know how to choose the appropriate concurrency model for your program's needs.

advanced best-practices

For additional information on related topics, take a look at the following resources: