How to Use multiprocessing.Pool()
In this lesson, you’ll dive deeper into how you can use
multiprocessing.Pool. It creates multiple Python processes in the background and spreads out your computations for you across multiple CPU cores so that they all happen in parallel without you needing to do anything.
You’ll import the
os module in order to add some more logging to your
transform() function so you can see what’s going on behind the scenes. You’ll use
os.pid() to see which process is working on which record, so you’ll be able to see that the same processes are working on different data in different batches. You can even set the number of processes you want to be working at once.
Comments & Discussion
Become a Member to join the conversation.