When to Use concurrent.futures or multiprocessing
In this lesson, you’ll see which situations might be better suited to using either
multiprocessing. You’ll also learn about how that ties in with the Global Interpreter Lock (GIL).
Because of the GIL, no two threads can execute Python code at the same time. So even if you have multiple threads running in your Python program, only one of them can execute at a time. The best way to get around this is to use process-based parallel programing, or process-based parallelism.
Congratulations, you made it to the end of the course! What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment in the discussion section and let us know.
What it means, basically, is that no two threads can execute Python code at the same time. Even if you have multiple threads running in your Python program, only one of them can execute at a time. Now, this sounds like a huge limitation and in some cases it is, but, really, what happens most of the time is that your thread will be waiting on I/O to complete. In this case, if I’m calling
time.sleep()—that’s an I/O operation. That’s blocking this thread. That means while that
time.sleep() call is blocking this one thread, other threads can execute, and then at the end, the thread will resume and they will all finish their processing.
00:52 So in this case, it doesn’t really make a difference. However, if I was doing heavy number crunching in these threads, I would run into the global interpreter lock problem because the end result wouldn’t really be faster than running a single-threaded version.
01:06 So, there’s lots more to say about this, but really what you need to remember is that the best way to get around this is to use process-based parallel programming in Python, or process-based parallelism.
And this is where this
concurrent.futures module is extremely handy because you know—see? I did it again. I just switched from thread-based execution, or thread-based parallelism, to process-based parallelism.
01:31 This gets around the global interpreter lock problem because every single process has its own interpreter. Therefore, they can all run in parallel, you can actually spread them out across multiple CPU cores, and this solves the global interpreter lock problem.
But this is definitely something that you need to keep in mind when you’re writing parallel programs in Python. This is also where this
concurrent.futures module is kind of nice, because you can change the execution strategy very, very easily.
And, really, the
ProcessPoolExecutor is just a wrapper around the
multiprocessing.Pool, but if you’re using this interface, it just becomes so simple to swap out the different execution strategies here.
Become a Member to join the conversation.