Locked learning resources

Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Locked learning resources

This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

concurrent.futures vs multiprocessing

In this lesson, you’ll see why you might want to use concurrent.futures rather than multiprocessing. One point to consider is that concurrent.futures provides a couple different implementations that allow you to easily change how your computations are happening in parallel.

In the next lesson, you’ll see which situations might be better suited to using either concurrent.futures or multiprocessing. You’ll also learn about how that ties in with the Global Interpreter Lock (GIL).

Avatar image for tinachoudhary

tinachoudhary on April 6, 2020

Thanks a lot for the tutorial, it is explained well.

Avatar image for Zarata

Zarata on May 7, 2020

Well! Per comment two(?) videos ago, now my simple concept of one process per thread, one thread per core is truly all topsy-turvey. The ThreadPoolExecutor is running 7 threads concurrently in one process. Is it fair to presume on one core, or is the mother process doling threads across cores? Is the OS time slicing between the threads so that all complete with near simultaneity? Is the child, Melissa, Dr. Stuben’s long lost daughter? The inquiring mind wants to know! (Who is that man behind the curtain? I can’t ignore him …)

Avatar image for macro84

macro84 on Oct. 14, 2021

Strange… it works perfectly fine with with “concurrent.futures.ThreadPoolExecutor() as executor:” but I get error message when using with “concurrent.futures.ProcessPoolExecutor() as executor:” Error message: BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Avatar image for Bartosz Zaczyński

Bartosz Zaczyński RP Team on Oct. 14, 2021

@hlaret It works for me with both pools. What kind of operating system are you on, and which Python version are you using?

Avatar image for macro84

macro84 on Nov. 10, 2021

Windows 10 and Python 3.8.3

Avatar image for Bartosz Zaczyński

Bartosz Zaczyński RP Team on Nov. 10, 2021

@macro84 I haven’t used Windows in a long time, but this rings a bell. Did you make sure to wrap your code in the if __name__ == "__main__": condition?

There’s an explanation in the official documentation of the multiprocessing module:

Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).

(…)

Instead one should protect the “entry point” of the program by using if __name__ == '__main__': as follows:

(…)

Avatar image for macro84

macro84 on Dec. 20, 2021

thx

Become a Member to join the conversation.