Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

This lesson is for members only. Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Hint: You can adjust the default video playback speed in your account settings.
Sorry! Looks like there’s an issue with video playback 🙁 This might be due to a temporary outage or because of a configuration issue with your browser. Please see our video player troubleshooting guide to resolve the issue.

Making Parallel HTTP Requests With aiohttp

Give Feedback

Learn how to use asyncio.gather() to make parallel HTTP requests in a real world application.

Comments & Discussion

toosto on April 28, 2019

Hi,

I have a couple of questions:

1) What is the state of the python thread when we are ‘awaiting’ on I/O processing? Is the python thread technically sleeping and woken up later by the operating system or is it just waiting/polling? 2) Why do we require the await keyword while attempting to get the text using response.text()? Is it because response becomes a coroutine object rather than a normal python object?

toosto on May 16, 2019

Guys, any updates?

Parijatha Kumar Pasupuleti on May 22, 2019

Where exactly the http GET request gets executed ? Is it in the response = await session.request(method='GET', url=url)statement or in the next statement i.e., value = response.text() statement ? If the request happens in only one of these statements, then why do we need await for both the statements ?

Par Akerstrom on June 1, 2019

Hi there Chyld. Great session, really liked it.

I was just wondering if there was a way to define a generator function like we did in the theory session and use that for the async http call? As opposed to using one line generator expression?

Say that the generator function would look like this:

   def gen_workers(stop):
        for member in range(1, stop + 1):
        yield member, randint(1, 10)

I played around with it but couldn’t figure out how to make the call asynchronous. The below works synchronously, but not in async!

            members = list(gen_workers(1))
            for member in members:
                responses = await asyncio.gather(worker(member[0], member[1], session))
            print(responses)

Thanks again /Par

Par Akerstrom on June 1, 2019

Ah, never mind. I read the tutorial of concurrency and figured out that we can use a different type of syntax for iterating over an already generated list.

Generator function stays the same:

    def gen_workers(stop):
        for member in range(1, stop + 1):
        yield member, randint(1, 10)

We do this for the async function:

    async def alternate_worker_pool():
        async with aiohttp.ClientSession() as session:
            # Synchronous way with external generator
            members = list(gen_workers(10))
            tasks = []
            for member in members:
                task = asyncio.ensure_future(worker(member[0], member[1], session))
                tasks.append(task)
            await asyncio.gather(*tasks, return_exceptions=True)

And we run the event loop with:

    if __name__ == '__main__':
        start = time.perf_counter()
        asyncio.get_event_loop().run_until_complete(
             alternate_worker_pool())
        elapsed = time.perf_counter() - start
        print(f'executed in {elapsed:0.2f} seconds')

Thanks for a great video. Cheers

Pygator on Sept. 14, 2019

So the speed up is from hoping that the website you formed the request to will parallelize the task on their machine? Isn’t Python running the tasks on different cores asynchronously?

jamespeabody on July 17, 2020

The questions above are somewhat recent and deserving of an answer. Requests for I/O are in the domain of the operating system. Those requests are therefore conducted in that process, outside of the process where your python code is being executed. As such, one would need to review the O/S API. It has been some time since the last time I reviewed an O/S API but at that time I recall a set of calls that the calling process would provide a callback to notify when an operation completed and one that return a result when the operation completed.

The first call was non-blocking and the second, blocking.

One might consider that the await is setting up a callback for the O/S to call when a specific operation completes. O/S API calls follow a general pattern as well. Often a process will make a request for data without knowing how much data will be returned. It is a simple matter to know how large a file is in a file system but other data sources can be unbounded streams of data. In the latter cases, a program has to allocate memory to store incoming data and provide a pointer to where that memory is and a counter of how much memory is in the buffer (a counter to provide a count of how many bytes are in the buffer) to the O/S. The O/S then places the incoming data into the buffer and returns a count of how many bytes were required. If the amount of data being retrieved is larger than the buffer, which is most the time, then this is an iterative process. Hence the second await in the second line of code. When python passes a buffer off to the O/S for the O/S to fill it, python also provides a callback for the O/S to call to notify python when the operation is complete. The buffer fills in the O/S thread leaving python free to perform other tasks in that time frame.

When the O/S makes the callback, it is a matter of implementation as to if the callback actually performs work or sets a flag for the event loop to attend to the task during normal processing. In any case, the callback should complete processing as rapidly as possible. For this reason and based on the syntax flow of python in this case, it seems reasonable that the latter is more likely the case.

The async and await key words are extensions to the python language. They facilitate adding housekeeping tasks to an event loop. Conceptually, async tells python that a particular method requires a callback framework and await indicates where those callbacks are implemented within the method. Each await, when encountered, is an opportunity for control to be passed to a different task via the event loop.

Become a Member to join the conversation.