When to Use concurrent.futures or multiprocessing

Awesome lesson and thanks for sharing

Good tutorial. I’d read about python multiprocessing/threading, but had not yet implemented it. Seeing it in action in this tutorial - wow - quite easy to get going. No reason not to implement when it will help.

You’ve cleared my doubt about functional programming. Now, the time for question. Can we say that a piece of functional program should only contain function calls with no procedural logic and it should ideally act on immutable data ?

Glad you liked the course!

Can we say that a piece of functional program should only contain function calls with no procedural logic and it should ideally act on immutable data?

Great question. Well, my take on this is that I’ll use whatever makes my life and the lives of my colleagues easier :) I’m not a purist when it comes to functional programming.

I find it useful as a technique that I can use when appropriate, but I’m not going to lock myself into writing only pure FP code with 100% immutable data structures.

It might make for a fun exercise to try and attain that, but at the end of the day I’m usually writing code to solve a problem. So I’ll use whatever tools and techniques that make me the most effective in getting to my goal. I don’t feel bad about mixing functional, procedural, and object-oriented programming styles.

I liked taking builtin map from python3 and connecting it with a parallel map idiom from two separate modules to speed up code execution; great place to use FP on a big data structure.

Great tutorial in functional programming. I learned a lot. I’d like to share a newbie mistake I made in the last section. I typed in the code from the video but I named it concurrent.py (bad mistake) and when I tried to run, it caused an error:

ModuleNotFoundError: 
No module named 'concurrent.futures'
'concurrent' is not a package

I fumbled with this for most of the day trying to figure out the problem before I finally found a post from a few years back on stackoverflow that had the same error and one of the answering comments included ‘…Either that or you’re shadowing concurrent. Do you have a concurrent.py?’

Umm, yes I do.

Once I renamed the file it worked as expected.

So I learned that you shouldn’t name your file the same name as a package name. Of course it seems so obvious now.

Thanks for a great intro to the functional programming style.

Thanks a lot for this course, very well explained :)

Got is now! That concurrent.futures bit was very useful particularly when GIL is in the picture. Thanks and kudos!

nice contents.

is there any project you can suggest to apply the lesson learnt from this course

Thanks a lot for this course, very well explained is there any project you can suggest to apply the lesson learnt from this course plz

That was cool !

Thank you Dan for this great tutorial! :)

Awesome, thanks Dan! I finally understand what GIL blocks. It was always a black box for me, I knew that there was something wrong in Python parallelism but I didn’t know that it was restricted to threads while computing. Good to know, Good to know!

Good tutorial I really enjoyed watching it and picking up some cool techniques from it Great work keep it up

Thanks for the course, Dan! key takeaway was definitely parallel processing. the discussion was “pure gold” to jump start its use in my coding

Never fully understood the GIL problem and how concurent.futures solves it for us. Thanks you very much for that.

I’m super impressed with the clarity of the explanations. And finally I understand what functional programming is, and how handy it can be for multiprocessing. Many thanks!

Honestly thought it would be harder than this. Many thanks to you for the awesome tutorials.

Thank you for this course. I learned a lot from it!

That was excellent! You do a great job presenting this info.

Your examples were working with a small dataset. How would you populate this type of immutable/named tuples data structure? Would you import to pandas first and then set this up?

Really nice overview and explanation! congrats.

Great tutorial, thanks!

This was a very nice present! Thanks Dan and Real Python!

has anyone idea about given error like below while using multiprocessing pool?

**cls(buf, protocol).dump(obj)

TypeError: can’t pickle SSLContext objects

Thanks for the course, it is very clear and concise and helpful.

Great course and even greater presentation. Keep it up.

Processes and Threads and GIL, oh my! Processes and Threads and Gil, oh my! … It’s a mind stretcher!! The fact the (implied superior) not-GIL-limited ProcessPoolExec executes in 2 sec while the ThreadPoolExec different result is 1 sec (half) teaches much … but it’s going to be awhile gaining the skill and insight to know precisely what :) ! Wow! Thanks DB. BTW, roughly how much resource hit giving each process its own Py interpreter??

Excellent course! Thank you!

Nice & great presentation. It helps to understand multiprocessing, concurrent feature, and GIL block better. Is any way to get the codes and supporting material for this course? Thanks!

Excellent !

nicely teached even an intermidiate like me can also understand 👍👍

Great practical applications possibilites for one of most efficient derived data types - namedtuples. For SEO tasks I did namedtuples classes to automate mapping of scrapped data keywords with their attributes. This course is great supplement how to process, map, join, combine and save time running processing data with multiple attributes.

Can you explain how does multiprocessing.dummy compare to concurrent.futures.ThreadPoolExecutor ?

The concurrent.futures package came with Python 3.2, which was years after the multiprocessing.dummy. It was modeled after the Execution Framework from Java 5 and is now the preferred API for implementing thread pools in Python. That said, you still might want to use multiprocessing.dummy as an adapter layer for legacy code.

This functional programming course is really excellent! Although I need to revise it and chew it again, I learned plenty of ways I can make my code more efficient. Thanks Dan.

Great course! It is explained in a very clear way and I have learnt a lot of new and useful things to put into practise in my programming. I really appreciate all the effort put into this.

Thanks so much for this course! I’m excited to try applying this to my own API calls when something might be running more slowly than I’d like.

Great contents and clear presentation. Now I have to do my own practicing.

Thank you Dan for this course. I’m looking forward to storing data from Excel sheets in immutable data structures and then using some of the strategies from the videos to manipulate the data for use in my programs.

Great course. Number 1 takeway for me was an answer as to why to keep functions (that will be parallelized) small and with no side-effects. An open question I had is that if I don’t use map() to apply a function to an iterator, can that still be parallelized?

You recommend using process-based parallelism, but I have seen advice in multiple other places, where they recommend process-based for CPU-heavy tasks and thread-based for IO-heavy tasks. Also, in the Python docs it says ‘the GIL is always released when doing I/O.’ What about that?

(…) but I have seen advice in multiple other places, where they recommend process-based for CPU-heavy tasks and thread-based for IO-heavy tasks.

@mihail-t That’s perfectly reasonable advice, which doesn’t contradict what Dan is already explaining in the video lesson.

In a nutshell, the GIL prevents multiple threads from running at the same time in Python, which means that only one thread can execute at a time. This renders Python threads useless for CPU-bound tasks that you wish to run in parallel. On the other hand, threads can still be beneficial for I/O-bound tasks, which mostly wait for data to arrive, e.g., through the network. Historically, the only way to run multiple tasks in parallel with Python has been to run them in separate interpreter processes.

If you want to learn more about these topics, including the GIL, then you’ll find plenty of Real Python resources that explore them:

• Speed Up Your Python Program With Concurrency
• Python 3.12 Preview: Subinterpreters
• Bypassing the GIL for Parallel Processing in Python
• What Is the Python Global Interpreter Lock (GIL)?
• Python 3.13 Preview: Free Threading and a JIT Compiler
• Synthesize an Image in Chunks Using Parallel Processing
• Async IO in Python: A Complete Walkthrough

Note: The concurrent.futures package provides a high-level interface on top of both threading and multiprocessing, which was modeled after Java’s executor framework API.