Parallel Processing With concurrent.futures: Overview

In this section, you’ll learn how to do multithreading and parallel programming in Python using functional programming principles and the concurrent.futures module.

You’ll take the example data set based on an immutable data structure that you previously transformed using the built-in map() function. But this time, you’ll process the data in parallel, across multiple threads using Python 3’s concurrent.futures module, which is available in the standard library.

You’ll see, step by step, how to parallelize an existing piece of Python code so that it can execute much faster and leverage all of your available CPU cores and computing power. You’ll learn how to use the ProcessPoolExecutor and ThreadPoolExecutor classes and their parallel map implementations that make parallelizing most Python code written in a functional style a breeze.

By knowing the difference between both executors available in the concurrent.futures module, you’ll be able to parallelize your Python functions across multiple threads and across multiple processes. You’ll get a brief introduction to the Python Global Interpreter Lock, also known as the GIL, and see how you can work around its limitations by using the correct executor implementation.

Once again, you’ll use your little testbed program from the last video to measure the execution time with the time.time() function. This allows you to compare the single-threaded and multithreaded implementations of the same algorithm.