How to Use multiprocessing.Pool()

On my Windows machine, I had to move the top-level code into a main() function, otherwise I was getting this error.

Update: Also, the script works fine after moving the top-level code into an if __name__ == '__main__' block, i.e. without defining a main() function.

But not as a plain top-level code that we can see in the video.

Is it not possible to parallelize reduce function because of the accumulator which needs to be shared across multiple processses? Is that the reason why multiprocessing.Pool doesn’t have a reduce function as opposed to map? Can somebody please let me know if my thought process is correct? Thanks in advance.

@Diwanshu:

Is it not possible to parallelize reduce function because of the accumulator which needs to be shared across multiple processses?

Yep I’d say that’s accurate. Parallelizing the reduce step might be possible depending on the exact use case (and you could do it by splitting up the work manually and taking care of it with another multiprocessing.Pool) but there’s no general implementation of a parallel reduce that the multiprocessing module provides.

I did some experiments with multi processing .Is the number of processes must be equal to no of cores in our machine ,But when I experimented with more no of process than number of cores the output came fast ,But I have a question is this way of giving more process than number of cores in machine not recommended

I’m experimenting with simple squaring function like this:

def fungsi_kuadrat(n):
    return n**2

a = tuple(range(5000))

waktu_mulai = time()
pool = multiprocessing.Pool()
dikuadratkan = tuple(pool.map(fungsi_kuadrat, a))
#dikuadratkan = tuple(map(fungsi_kuadrat, a))
waktu_selesai = time()
lama_eksekusi = waktu_selesai - waktu_mulai

print('waktu yg dibutuhkan {} detik'.format(lama_eksekusi))

Using conventional map it takes 0.0102119445801 secs, while multiprocessing pool map it needs 0.0106949806213 secs.

Why is multiprocessing.Pool() a bit slower in this case?

Sid - The slow down on the multiprocessing pool is probably due to a couple of things. Mainly, spinning up processes takes time, and passing data to and from these processes is relatively slow. The time it takes to pass n from one process to the worker process could well be longer than then time it takes to compute n**2. (I haven’t measured so i don’t know definitively)

If you make your function more compute intensive, you’ll likely see the multiprocessing pool do better.

Hope this helps. jima

Hi, i try running the code but got error like

PicklingError: Can't pickle <class '__main__.scientists'>: it's not the same object as __main__.scientists

do you have any idea?

Hi sorry, solved. it was because of my namedtuple

multiprocessing is great, but be aware, if your transformation function fails, you have to code up a bunch more stuff to get access to the stacktrace…

On Python 3.8, I needed to change the code to this

if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        start = time.monotonic()
        result = pool.map(transform, scientists)
        end = time.monotonic()
        print(f"Time taken: {end - start:.2f}")
        pprint.pprint(result)

The pool needs to be setup before you can use it, the easiest way is to use the context manager, otherwise you will get the bootstrapping error on Mac/Linux too.

Trying to run the code in Jupyter Notebook will deadlock, not entirely sure why, Jupyter and multiprocessing don’t really work togther.

(time.monotonic() always gives you the correct result with this type of usage, even if the system clock gets screwed with inbetween readings. Useful for production code!)

Well that was unexpected … the mid video set of the pool size to 7 led ~1 sec completion. Yet, later the cores available are shown to be 4. I’d been thinking the pool manages 1 thread per core, one process per thread, so despite the large pool size I thought the problem would have been core bound, i.e. the time would be ~2. So is the pool creating 7 threads, (I see there ARE seven PIDS), doling the 7 threads over the 4 cores, and the OS takes over managing multiple concurrent threads on some of the single cores? Or what? (there was a mention of the hardware supporting ‘hyperthreading’ … is that related to my conundrum?)

Can someone clarify whats up with the number of processes being more than the number of cores. In that scenario how do all the processes run at the same time and finish in 1sec.

I think it begins to get answered in a later tutorial. There are two different types of pools in the Futures package: Thread and Process. Futures bears some similarity with the multiprocessing package, and what is revealed there thus may have some relevance here. At least, it made me think further. I suggest: The transform() functions that are used here (and in the future Futures :) ) only contain a “sleep(1)”, so each is very similar as a threaded task, and each largely consists of a hold time doing nothing – the OS is pulling off the wait. Somehow the pool here in multiprocessing is cooperating with the OS and knows “launched a thread, it’s in hold, no CPU being used, I can launch another, …” so that eventually there are 7 threads out there. I can’t say how they are actually distributed across the cores, but they are all held and in wait. Then, after the 1 sec., the OS starts un-waiting the threads, all probably sequentially, but maybe divided between cores; regardless, all release at about the same time. Other than the “huge” wait, very little time is used. So, the threads clean up, and the total exec. is only ~1.00 sec.

After moving everything into a main function and adding an

if __name__ == '__main__':
    main()

block at the bottom, I continued to get an error:

AttributeError: Can't pickle local object 'main.<locals>.transform'

The solution was to move the definition of the transform function, the collection.namedtuple and the place where I defined the scientists variable outside of the main() function.

Can anyone please post a complete script to fix the AttributeError: Can’t pickle local.... ?

Nevermind. I managed to fix it. Sorry about that.

Great examples for multiprocessing.