Join us and get access to thousands of tutorials and a community of expert Pythonistas.

This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

When to Use concurrent.futures or multiprocessing

From Python's Functional Primitives to Parallelism Dan Bader 02:24

In this lesson, you’ll see which situations might be better suited to using either concurrent.futures or multiprocessing. You’ll also learn about how that ties in with the Global Interpreter Lock (GIL).

Because of the GIL, no two threads can execute Python code at the same time. So even if you have multiple threads running in your Python program, only one of them can execute at a time. The best way to get around this is to use process-based parallel programing, or process-based parallelism.

Congratulations, you made it to the end of the course! Make it stick: Mentor AI will check what you’ve got down, help you build something with it, and find your #1 takeaway to share in the discussion.

Or go straight to the discussion section and share your #1 takeaway and how you’ll put your new skills to use.

00:00 Now, of course, the question is, “Well, when should you use one over the other?” The dark secret of Python is something called the Global Interpreter Lock.

00:10 What it means, basically, is that no two threads can execute Python code at the same time. Even if you have multiple threads running in your Python program, only one of them can execute at a time. Now, this sounds like a huge limitation and in some cases it is, but, really, what happens most of the time is that your thread will be waiting on I/O to complete. In this case, if I’m calling time.sleep()—that’s an I/O operation. That’s blocking this thread. That means while that time.sleep() call is blocking this one thread, other threads can execute, and then at the end, the thread will resume and they will all finish their processing.

00:52 So in this case, it doesn’t really make a difference. However, if I was doing heavy number crunching in these threads, I would run into the global interpreter lock problem because the end result wouldn’t really be faster than running a single-threaded version.

01:06 So, there’s lots more to say about this, but really what you need to remember is that the best way to get around this is to use process-based parallel programming in Python, or process-based parallelism.

01:18 And this is where this concurrent.futures module is extremely handy because you know—see? I did it again. I just switched from thread-based execution, or thread-based parallelism, to process-based parallelism.

01:31 This gets around the global interpreter lock problem because every single process has its own interpreter. Therefore, they can all run in parallel, you can actually spread them out across multiple CPU cores, and this solves the global interpreter lock problem.

01:48 But this is definitely something that you need to keep in mind when you’re writing parallel programs in Python. This is also where this concurrent.futures module is kind of nice, because you can change the execution strategy very, very easily.

02:02 And, really, the ProcessPoolExecutor is just a wrapper around the multiprocessing.Pool, but if you’re using this interface, it just becomes so simple to swap out the different execution strategies here.

02:17 Now, we’re back to a ThreadPoolExecutor again, and we’re getting a different result.

Divyanshu Sharma on July 27, 2020

Can you explain how does multiprocessing.dummy compare to concurrent.futures.ThreadPoolExecutor ?

Bartosz Zaczyński RP Team on Aug. 3, 2020

The concurrent.futures package came with Python 3.2, which was years after the multiprocessing.dummy. It was modeled after the Execution Framework from Java 5 and is now the preferred API for implementing thread pools in Python. That said, you still might want to use multiprocessing.dummy as an adapter layer for legacy code.

Ghani on Oct. 14, 2020

This functional programming course is really excellent! Although I need to revise it and chew it again, I learned plenty of ways I can make my code more efficient. Thanks Dan.

paulagm12 on Nov. 15, 2020

Great course! It is explained in a very clear way and I have learnt a lot of new and useful things to put into practise in my programming. I really appreciate all the effort put into this.

squeakyboots on April 27, 2021

Thanks so much for this course! I’m excited to try applying this to my own API calls when something might be running more slowly than I’d like.

MOSTA on June 4, 2021

Great contents and clear presentation. Now I have to do my own practicing.

samuelebright on Jan. 30, 2022

Thank you Dan for this course. I’m looking forward to storing data from Excel sheets in immutable data structures and then using some of the strategies from the videos to manipulate the data for use in my programs.

MarkYoung on Sept. 25, 2023

Great course. Number 1 takeway for me was an answer as to why to keep functions (that will be parallelized) small and with no side-effects. An open question I had is that if I don’t use map() to apply a function to an iterator, can that still be parallelized?

mihail-t on Dec. 1, 2024

You recommend using process-based parallelism, but I have seen advice in multiple other places, where they recommend process-based for CPU-heavy tasks and thread-based for IO-heavy tasks. Also, in the Python docs it says ‘the GIL is always released when doing I/O.’ What about that?

Bartosz Zaczyński RP Team on Dec. 2, 2024

(…) but I have seen advice in multiple other places, where they recommend process-based for CPU-heavy tasks and thread-based for IO-heavy tasks.

@mihail-t That’s perfectly reasonable advice, which doesn’t contradict what Dan is already explaining in the video lesson.

In a nutshell, the GIL prevents multiple threads from running at the same time in Python, which means that only one thread can execute at a time. This renders Python threads useless for CPU-bound tasks that you wish to run in parallel. On the other hand, threads can still be beneficial for I/O-bound tasks, which mostly wait for data to arrive, e.g., through the network. Historically, the only way to run multiple tasks in parallel with Python has been to run them in separate interpreter processes.

If you want to learn more about these topics, including the GIL, then you’ll find plenty of Real Python resources that explore them:

Note: The concurrent.futures package provides a high-level interface on top of both threading and multiprocessing, which was modeled after Java’s executor framework API.

Become a Member to join the conversation.