Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

This lesson is for members only. Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Hint: You can adjust the default video playback speed in your account settings.
Sorry! Looks like there’s an issue with video playback 🙁 This might be due to a temporary outage or because of a configuration issue with your browser. Please see our video player troubleshooting guide to resolve the issue.

How to Use multiprocessing.Pool()

Give Feedback

In this lesson, you’ll dive deeper into how you can use multiprocessing.Pool. It creates multiple Python processes in the background and spreads out your computations for you across multiple CPU cores so that they all happen in parallel without you needing to do anything.

You’ll import the os module in order to add some more logging to your transform() function so you can see what’s going on behind the scenes. You’ll use os.pid() to see which process is working on which record, so you’ll be able to see that the same processes are working on different data in different batches. You can even set the number of processes you want to be working at once.

Comments & Discussion

Michal on Aug. 5, 2019

On my Windows machine, I had to move the top-level code into a main() function, otherwise I was getting this error.

Michal on Aug. 5, 2019

Update: Also, the script works fine after moving the top-level code into an if __name__ == '__main__' block, i.e. without defining a main() function.

But not as a plain top-level code that we can see in the video.

Is it not possible to parallelize reduce function because of the accumulator which needs to be shared across multiple processses? Is that the reason why multiprocessing.Pool doesn’t have a reduce function as opposed to map? Can somebody please let me know if my thought process is correct? Thanks in advance.

Dan Bader RP Team on Jan. 12, 2020

@Diwanshu:

Is it not possible to parallelize reduce function because of the accumulator which needs to be shared across multiple processses?

Yep I’d say that’s accurate. Parallelizing the reduce step might be possible depending on the exact use case (and you could do it by splitting up the work manually and taking care of it with another multiprocessing.Pool) but there’s no general implementation of a parallel reduce that the multiprocessing module provides.

somanathsankaran on March 30, 2020

I did some experiments with multi processing .Is the number of processes must be equal to no of cores in our machine ,But when I experimented with more no of process than number of cores the output came fast ,But I have a question is this way of giving more process than number of cores in machine not recommended

I’m experimenting with simple squaring function like this:

def fungsi_kuadrat(n):
    return n**2

a = tuple(range(5000))

waktu_mulai = time()
pool = multiprocessing.Pool()
dikuadratkan = tuple(pool.map(fungsi_kuadrat, a))
#dikuadratkan = tuple(map(fungsi_kuadrat, a))
waktu_selesai = time()
lama_eksekusi = waktu_selesai - waktu_mulai

print('waktu yg dibutuhkan {} detik'.format(lama_eksekusi))

Using conventional map it takes 0.0102119445801 secs, while multiprocessing pool map it needs 0.0106949806213 secs.

Why is multiprocessing.Pool() a bit slower in this case?

Jim Anderson RP Team on March 31, 2020

Sid - The slow down on the multiprocessing pool is probably due to a couple of things. Mainly, spinning up processes takes time, and passing data to and from these processes is relatively slow. The time it takes to pass n from one process to the worker process could well be longer than then time it takes to compute n**2. (I haven’t measured so i don’t know definitively)

If you make your function more compute intensive, you’ll likely see the multiprocessing pool do better.

Hope this helps. jima

Rombituon on April 1, 2020

Hi, i try running the code but got error like

PicklingError: Can't pickle <class '__main__.scientists'>: it's not the same object as __main__.scientists

do you have any idea?

Rombituon on April 1, 2020

Hi sorry, solved. it was because of my namedtuple

amicablezebra on April 17, 2020

multiprocessing is great, but be aware, if your transformation function fails, you have to code up a bunch more stuff to get access to the stacktrace…

Chris James on April 20, 2020

On Python 3.8, I needed to change the code to this

if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        start = time.monotonic()
        result = pool.map(transform, scientists)
        end = time.monotonic()
        print(f"Time taken: {end - start:.2f}")
        pprint.pprint(result)

The pool needs to be setup before you can use it, the easiest way is to use the context manager, otherwise you will get the bootstrapping error on Mac/Linux too.

Trying to run the code in Jupyter Notebook will deadlock, not entirely sure why, Jupyter and multiprocessing don’t really work togther.

(time.monotonic() always gives you the correct result with this type of usage, even if the system clock gets screwed with inbetween readings. Useful for production code!)

Zarata on May 7, 2020

Well that was unexpected … the mid video set of the pool size to 7 led ~1 sec completion. Yet, later the cores available are shown to be 4. I’d been thinking the pool manages 1 thread per core, one process per thread, so despite the large pool size I thought the problem would have been core bound, i.e. the time would be ~2. So is the pool creating 7 threads, (I see there ARE seven PIDS), doling the 7 threads over the 4 cores, and the OS takes over managing multiple concurrent threads on some of the single cores? Or what? (there was a mention of the hardware supporting ‘hyperthreading’ … is that related to my conundrum?)

Varun Vaddiparty on May 11, 2020

Can someone clarify whats up with the number of processes being more than the number of cores. In that scenario how do all the processes run at the same time and finish in 1sec.

Zarata on May 11, 2020

I think it begins to get answered in a later tutorial. There are two different types of pools in the Futures package: Thread and Process. Futures bears some similarity with the multiprocessing package, and what is revealed there thus may have some relevance here. At least, it made me think further. I suggest: The transform() functions that are used here (and in the future Futures :) ) only contain a “sleep(1)”, so each is very similar as a threaded task, and each largely consists of a hold time doing nothing – the OS is pulling off the wait. Somehow the pool here in multiprocessing is cooperating with the OS and knows “launched a thread, it’s in hold, no CPU being used, I can launch another, …” so that eventually there are 7 threads out there. I can’t say how they are actually distributed across the cores, but they are all held and in wait. Then, after the 1 sec., the OS starts un-waiting the threads, all probably sequentially, but maybe divided between cores; regardless, all release at about the same time. Other than the “huge” wait, very little time is used. So, the threads clean up, and the total exec. is only ~1.00 sec.

Become a Member to join the conversation.