In this section, you learned how to do parallel programming in Python using functional programming principles and the multiprocessing
module. You used the example data set based on an immutable data structure that you previously transformed using the built-in map()
function. But this time, you processed the data it in parallel, across multiple CPU cores using the Python multiprocessing
module available in the standard library.
You saw, step by step, how to parallelize an existing piece of Python code so that it can execute much faster and leverage all of your available CPU cores. You learned how to use the multiprocessing.Pool
class and its parallel map
implementation, which makes parallelizing most Python code that’s written in a functional style a breeze.
You built a little testbed program that you used to measure execution time with the time.time()
function, so that you could compare the single-threaded and multithreaded implementations of the same algorithm. Stay tuned for the next section in the course, where you’ll learn how to make your Python programs multithreaded using the concurrent.futures
module as an alternative way to implement concurrency.
Jishnu Banerjee on Feb. 27, 2020
Thanks Dan for exposing this functional programming approach. In this particular context I guess it is worth mentioning that if “lambda” function is used inside the “map” function then we cannot re-use the same code inside Pool.map as it will fail to unpickle. So it is always good to use regular function and pass it to the map() as we exercise this functional programming approach.