If you’re doing I/O-bound concurrency, this typically isn’t a problem. You can still get speed-up because of the long latency waiting for network or disk access. Python has a library called
multiprocessing that allows you to spin up an interpreter per CPU.
This is the multiprocessor variant on our download program. It looks remarkably similar to the
threading example. Line 2 imports the
multiprocessing library. Line 8 sets a global
session. Because each processor will get its own interpreter, each interpreter has its own memory footprint.
Unless you specify otherwise, you will get one interpreter per CPU inside of your computer. The
initializer parameter specifies a function to call once each of the instances is set up. In this case, this is where I’m setting up the
And just like the thread library, a map happens between the function that does the processing and the data in question. The
Pool.map() function takes care of assigning the site values to one of these functions, spreading out the computation. Let me just scroll down.
02:57 Keep in mind, this doesn’t have to do with the scheduling of the processes, but when the web server comes back. Printing of the CPU number happens when the web page has finished downloading, so depending on connection speed and network latency, things move back and forth between the CPUs. Because there are four interpreters in this case, four things are happening simultaneously.
Each process has its own memory space and the
initializer parameter is called once per process within the local memory of that space. In the case of this example, there are still 160 things to download, so as a single
download_site() function finishes, the
Pool assigns the next one to whatever CPU is currently idle.
This could account for some of the numbers repeating themselves. If CPU 4 happens to be freed up while 3, 2, and 1 are still waiting on the I/O, 4 would get the next
download_site(), and if for whatever reason it happened to be able to download quickly, it might print out a result before one of the other three processes finished downloading their site. And once again, because each CPU gets its own instance of the interpreter, you no longer have the problem of the GIL.
04:49 The implementation of a process happens at the operating system level, so you will also see behavior and scheduling differences between operating systems in your code. Because each process gets its own copy of the interpreter, it tends to require more memory than threading does as well.
05:16 In fact, threads were introduced into operating systems as a lightweight way of getting around the overhead involved in processes. Because each process has its own interpreter and does not share memory footprints, communicating between the processes must be done with explicit constructs.
multiprocessing library comes with a few that can help you do that.
Pipe are ways of sending data from one process to another, and the
Array constructs allow you to share memory between processes.
multiprocessing library includes locking mechanisms to make sure that you don’t end up with race conditions or deadlocks when two or more processes are trying to access the same chunk of shared memory.
The number of threads that you instantiate generally can be as many as you want. More is not always better, but it’s under your control.
multiprocessing typically is only used to map processes to CPUs.
06:14 You can instantiate more interpreters than CPUs, but it doesn’t really make sense because then all you’re doing is swapping out those and the overhead of the swap tends to be more expensive and you don’t actually gain any speed-up.
asyncio tend to be beneficial in I/O-bound situations. They’re not beneficial in CPU-bound situations. That’s where multiprocessing reigns. So in the next lesson, I’ll show you the differences.
Become a Member to join the conversation.