How Functional Programing Makes Parallel Processing Simple
In this lesson, you’ll see how functional programing makes parallel processing simple. If you can write your program in such a way that it uses a
map operation to transform some input data into some output data, then it’s quite simple to parallelize it.
You just need to import
concurrent.futures and add two lines of code instead of a straight-up
map() call, and your code is running in parallel! This can lead to huge improvements in speed.
00:00 All right. Again, only scratching the surface in this video, but I hope it whetted your appetite for doing more parallel processing in Python and to also do it in a functional programming style, because I think one of the key advantages of functional programming is that it makes it very easy to parallelize your programs, right?
If you can write your program in a way where it’s using a
map() operation to transform some input data into some output data, or, let’s say, if a part of your program can be written that way, it is extremely simple to parallelize this like crazy. As you’ve seen here, all you need is you need to import
concurrent.futures and you need these two lines of code instead of a straight-up
map() call that’s built into Python, and
01:00 So, highly recommended for any kind of number crunching that you do, or if it’s a super I/O bound program, like most web scrapers are, where you’re waiting a long time for the web requests to finish so that we can parse out the data.
01:13 This is extremely handy for those kinds of situations. I would encourage you to play with it and just get a better understanding of how this works and what the difference is between thread-based parallelism and process-based parallelism, and how the global interpreter lock works in Python. Have fun playing with parallel programming. All right, talk to you soon.
Become a Member to join the conversation.