Learn how to use
asyncio.gather() to make parallel HTTP requests in a real world application.
Making Parallel HTTP Requests With aiohttp
Learn how to use
00:00 So, instead of just making one request, what if we made a whole bunch of requests, right? So, one request took about 1 second—well, what if we did a whole bunch of these things? So instead of this thing, what I’m going to do is I’m going to say
sums = await, and we’re going to use
asyncio.gather(), and what we’re going to is we’re going to put a whole bunch of coroutines in here and let them run, and then whenever they finish, they’ll bring back their results.
So we’re going to get
2 numbers, then
3 numbers, and then
5 is not inclusive. So we’re going to get
i because we’re using
enumerate()—that’s our index—and then we’re going to get the actual value, let’s call that
02:41 What if I ran more than this? Now obviously you shouldn’t do, like, a million or whatever, because this is a website and you don’t own the website, so you don’t want to hit it with, like, a thousand web requests.
So, it ran from
27—or 28 total workers, and this is the result from each one—and the whole thing took 2.6 seconds. So, if one took one second, how did 28 of them only take two and a half seconds?
03:29 That’s because what’s happening is Python is sending out the request into the ether, is handing the request off to the operating system, and then the operating system just takes over and the operating system sends out all the requests. So once these requests have left Python, then the outside world can handle them however it wants to—if it wants to run them in parallel, then that’s fine.
03:52 Whatever website that I’m sending the request to can potentially run these in parallel. And so that’s how come when you get back these results, this is how it looks like it’s parallel. No, it’s not actually running in parallel in Python—it’s running in parallel external to Python. But from Python’s perspective, everything is just running sequentially because there really only is one process and one thread.
What we did here is we had
theory.py where we were talking about generators, coroutines, and asynchronous generators. And then this is
summer.py, where we sort of put everything together in one nice little package to make it seem as real-world as possible.
Become a Member to join the conversation.