Join us and get access to thousands of tutorials and a community of expert Pythonistas.
This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.
In this lesson, you’ll see why you might want to use concurrent.futures rather than multiprocessing. One point to consider is that concurrent.futures provides a couple different implementations that allow you to easily change how your computations are happening in parallel.
In the next lesson, you’ll see which situations might be better suited to using either concurrent.futures or multiprocessing. You’ll also learn about how that ties in with the Global Interpreter Lock (GIL).
00:00
So, that was pretty easy to do. You may be wondering, “Okay, why should I use this thing here over multiprocessing, or over a multiprocessing.Pool?” Well, what’s kind of nice is that the concurrent.futures module, it provides a couple of different implementations that allow you to change very easily how your computations are happening in parallel.
00:23
So, this was the ProcessPoolExecutor, and I can just swap out the word Process for the word Thread, and then we’re going to get a different result here.
00:33
Now you can see here that the ThreadPoolExecutor actually performed all of these calculations within a single process spread out across multiple threads. In this case, it’s even faster, but this is just a toy example. It’s not really doing anything, it’s just sleeping for a second.
00:50 But you can already see here that the big difference is that now, everything is running within a single process. Parallel computation is happening because we have multiple threads within that single process that are working on this data in parallel.
Zarata on May 7, 2020
Well! Per comment two(?) videos ago, now my simple concept of one process per thread, one thread per core is truly all topsy-turvey. The ThreadPoolExecutor is running 7 threads concurrently in one process. Is it fair to presume on one core, or is the mother process doling threads across cores? Is the OS time slicing between the threads so that all complete with near simultaneity? Is the child, Melissa, Dr. Stuben’s long lost daughter? The inquiring mind wants to know! (Who is that man behind the curtain? I can’t ignore him …)
macro84 on Oct. 14, 2021
Strange… it works perfectly fine with with “concurrent.futures.ThreadPoolExecutor() as executor:” but I get error message when using with “concurrent.futures.ProcessPoolExecutor() as executor:” Error message: BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Bartosz Zaczyński RP Team on Oct. 14, 2021
@hlaret It works for me with both pools. What kind of operating system are you on, and which Python version are you using?
macro84 on Nov. 10, 2021
Windows 10 and Python 3.8.3
Bartosz Zaczyński RP Team on Nov. 10, 2021
@macro84 I haven’t used Windows in a long time, but this rings a bell. Did you make sure to wrap your code in the if __name__ == "__main__": condition?
There’s an explanation in the official documentation of the multiprocessing module:
Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).
(…)
Instead one should protect the “entry point” of the program by using
if __name__ == '__main__':as follows:(…)
macro84 on Dec. 20, 2021
thx
Become a Member to join the conversation.
tinachoudhary on April 6, 2020
Thanks a lot for the tutorial, it is explained well.