Parallel Processing With multiprocessing: Overview

In this section, you’ll learn how to do parallel programming in Python using functional programming principles and the multiprocessing module. You’ll take the example data set based on an immutable data structure that you previously transformed using the built-in map() function. But this time you’ll process the data in parallel, across multiple CPU cores using the Python multiprocessing module available in the standard library.

You’ll see, step by step, how to parallelize an existing piece of Python code so that it can execute much faster and leverage all of your available CPU cores. You’ll also learn how to use the multiprocessing.Pool class and its parallel map implementation that makes parallelizing most Python code that’s written in a functional style a breeze.

First, you’ll build a little testbed program that we can use to measure the execution time with the time.time() function, so that we can compare the single-threaded and multithreaded implementations of the same algorithm.