Join us and get access to thousands of tutorials and a community of expert Pythonistas.
This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.
A Real-World asyncio Example
In this lesson you will see how to put everything you have learned about generators, async generators, coroutines, and asyncio to build a real world async application that calls an HTTP API and manipulates data from it.
00:00 What I want to do now is put everything that we’ve learned together—generators, coroutines, asynchronous generators—and sort of build something that is a little more real-world, that actually uses real IO, real input/output, something like calling a database or a website.
00:19 In this case here, we’re going to call an HTTP API. We’re going to call this API and when we get back the results, it’s going to be in JSON format. It stands for JavaScript Object Notation.
00:30 It basically looks like a dictionary in Python. So, we’re going to call a website, we’re going to get back what’s called JSON—which looks like a dictionary—and then we’re just going to process it. Okay.
00:39 So, it might be useful for me to show you what this website looks like and the data that comes from it. This is an API. It basically is a website, or an API, that produces random numbers.
00:52 And they actually have some actual physical equipment that produces random numbers based on vacuums. It has to do with physics. I don’t want to get into how it works, but at the end, the purpose of the website is to produce actual, real random numbers, instead of in a computer they’re what’s called pseudorandom.
01:10
But these are, like, real random numbers—or at least theoretically. Okay, so this is a website. And what happens is I can say, “Give me 10 numbers, 10 random numbers.” And uint16, it means 16-bit, so that means it goes from 0 to 65,000.
01:30
So, “Any number between 0 and 65,000, give me 10 of these.” And I hit Enter, you can see it spinning around, and then it gave me some other numbers. And I can make this into a 20, and now I have 20 numbers here.
01:48
This is JSON right here—these little brackets right here, and then these are the keys, and these are the values. So once again, if you’re used to Python and dictionaries, this should look very similar. And Python has a json package, or a module, that you can bring in that will actually convert JSON into an actual dictionary.
02:06 Okay, so this is what we want to do. I’m going to do this. And because this involves IO, or input/output, the async package is perfect for this. I’m going to come back here and let’s start off and let’s create our imports.
02:20
import asyncio, and then I’m going to import json. json is going to allow us, like I said, to take JSON that comes from a website and then turn it into a dictionary so that Python can convert it.
02:36 If you didn’t do this, it would come in as a string, and it’s kind of hard to deal with strings. Obviously, you could do regular expressions, but that’s just too difficult.
02:47
I’m going to import time, because we want to see how long something takes, and I’m going to import aiohttp. This is an HTTP package, which means I’m going to call a website and I’m going to call it asynchronously.
03:02
So if you want to do this, if you want to call an HTTP endpoint, or a website, and if you want it to be nonblocking, you have to use the aiohttp. Now, there are other packages out here, but this is one built into Python. Actually, I take that back.
03:17
We actually pip installed this one, remember? This is how we started off. All right, so let’s go ahead and create our
03:25
'__main__'. I’m going to say if __name__ == '__main__':. Okay, what I’d like to do is I’d like to time this and see how long this is going to take, so let’s create a start time and we’ll do what we kind of did before time.perf_counter().
03:52
We’re going to create some main() function right here, that’s going to be asynchronous. So let’s go ahead and do that. async def main(): and then pass, just for right now.
04:08
I’m going to use asyncio, and what we need to do is create an event loop and run this main() inside of it, just like we did last time.
04:17
So I’m going to say .run(main()) like that. And then elapsed, the elapsed time, is just time.perf_counter() once again, minus start, the start time.
04:34
And then let’s just see how long that took. We’ll say print(f'')—in that many seconds, just like that. Okay. So, this will be elapsed:0.2f. Okay.
05:02
So, this is our "__main__", and right now our async really is not doing anything, but just to kind of prove this is working, we can say asyncio.sleep(1).
05:17
Oh, wait, we have to do an await on this because it’s in an async. Okay. So let’s just try this, and this will prove that our program is kind of working at least up to this point. So python summer.py,
05:34
and it took about 1.00 second, which is exactly what I had right here. So this shows this is working. Great. What I would like to do is I want main() to create a session, an HTTP session, and then send that off to somewhere.
05:49
So, I don’t want to do the actual work in this main() function. I want to do it in what I want to call a little worker, a little worker task.
05:57 This worker is going to be asynchronous too, which means when you call it, it’ll create a coroutine. And so I could create a whole bunch of these coroutines as little workers, and then have them work—that you might think are running in parallel, but they’re… It is running asynchronously, and really it’s not parallel—at least from Python’s perspective, because once again, Python just has a single thread, so it’s single-threaded. But once it leaves Python and the request is out in the real world, then those requests could happen almost in parallel. Okay?
06:33 So let’s try that. I’m going to create an asynchronous worker,
06:38
def, I’ll call this thing worker(), like this, and this is the name of the worker, n is how many numbers I want to get back from my random number generator on the web. n is how many numbers that I want. And then session is, this is the HTTP session that we’re going to be using. Okay, so I’m going to stop right there, so pass—that makes it correct syntactically. So let’s go back here.
07:09
What I want to do is because I’m opening up a resource, an HTTP resource, normally you would use with, but when you’re doing async stuff, you’re going to async with.
07:23
It’s sort of like for loops—when you’re doing it an asynchronous for, you say for i in range(). When it’s asynchronous, it’s async for i in range().
07:33
But the same thing with with. You have with blah(), with open(file), or async with open(file). In this example here, we’re not doing file IO—we’re doing network IO. Okay.
07:46
So I’m going to say async with aiohttp.ClientSession() as session:.
07:59 This is going to allow us to do HTTP requests.
08:05
Okay. So, what I’d like to do now is maybe let’s do one worker, so let’s make one request for this worker, and let’s just check what the results are. I’m going to say, maybe response = await worker(), and I’m going to pass in a name, let’s say 'bob'—'bob' is the name of the request. And the n, this means how many numbers am I going to get back? Let’s say I want to get back 3. And then I’m going to pass in the session.
08:44
Okay? Great. And let’s print the response out, so I can say print('response:', response).
08:57
Okay, so this looks pretty good. Now I need this worker() to do something. I need it to, like, take in this worker’s name, and then how many numbers I’m going to get from this website, and then this is the HTTP session. Okay. So let’s come back here.
09:13
And the first thing I want to do maybe is print out the worker’s name, So I’m going to say print(f'worker-{name}').
09:31 Okay. So this will print out the name of the worker. Next thing I want to do is I want to create a URI, or a URL, that actually goes to that location. So, the URL is effectively this.
09:43
So I can just copy this, come back here and say url is equal to f''—that. So, that looks pretty good. The only thing I want to do is I’ve hard-coded in 20 here, and I want 20 to just be n. Okay?
10:04
So, “Give me back n results.” Okay, so there’s my url. That looks pretty good. Now let’s go ahead and make a request, so I’m going to say response = await, because what I’m about to do is asynchronous IO.
10:20
So I’m going to do an await on that, and I’m going to say session, which is what we passed in from here, session.
10:26
And I’m going to say .request(),
10:31
and I’m going to do method='GET'. So, if you don’t know much about HTTP, this could be a little confusing here, but when you do this—when you make this request right here—this is called a GET request. There are other ones called POST and PUT and things like that, but let’s not worry too much about that.
10:49
But this is a GET request.
10:52
And I’m going to send in the url=url.
10:59 So just to recap, this line right here is the actual line that sends the request out of the computer and it finds its way to this computer—so it’s going to go from my computer to this computer, okay?
11:12
And then here’s my response. And this thing is asynchronous, so I have to use await, and because I have await here, I have async here.
11:23
Okay. So once you have this back, you need to turn this thing into text. So I’m going to say value = await response.text().
11:38
This response is going to come back and it’s going to be an HTTP response, which could be a little confusing, and this is going to turn that response and just show me just the text from this. Let’s take this text and it will be returned—here’s my value here, and I could just return that.
11:57
So I have my value, now let’s go ahead and return that. So I’m going to say return value like that, and let’s just see what happens. So, this value right here gets returned, and so I’m calling worker() and it should go into response, so value should basically go into response and I can print that out.
12:17
Let’s see what that does. If I run this, it says the worker’s name is bob, and the response is this thing. Now, you’re probably wondering like, “What is this thing? This thing here, it looks like a dictionary.” It’s actually not a dictionary, and I can prove that to you by typing in response and then type() of the response.
12:42
Let’s save that, come back here, run it again. Once again, worker-bob is running and you can see this thing here is not actually a dictionary—it’s a str (string). I want to turn this string, which is JSON, into an actual dictionary.
12:58
And so how I do that is I say value = json.loads()—that means “Load in a string.” Which string am I going to load in? I’m going to load in value.
13:12 So I took the string that came back, which is JSON, and I’m to turn that into an actual dictionary.
13:21
Okay. Let’s try that. So, worker-bob is running, I have this thing. And you may think this looks similar to this, but these have double quotes, these have single quotes.
13:33
That’s a little bit—it’s kind of hard to see that. But the real difference is, look—<class 'dict'> (dictionary), whereas before it says <class 'str'>.
13:44
So now I have something that Python actually knows about, and I want to return this key inside this dictionary. How do I do that? I just say value and I pass in the key, which is—let’s look at it again, it’s called 'data'.
14:04
Okay, let’s sum that up. Let’s come back here, run this again. And you can see now my response is just those three values. Now this whole program is called summer. You’re probably like, “Why is it called summer?” Because I want to return the sum of those three values, so I’m just going to take sum() of those three values, like that.
14:26
Save it, come back here, and let’s see what that is. And yes, 105,000. So yeah, I was able to actually create this—from here, I was able to asynchronously call this website, get back the data, get its text, turn it into JSON—or, turn it into a dictionary, and then take the dictionary, grab its key, and then this key ends up being a list, and so I’m taking the sum() of a list and that ends up being a single value.
14:54 Okay. So, that all is pretty neat and awesome. But how long did this thing take? Let’s look at this thing again. It took about 1 second. But what if I did a whole bunch of these? Let’s say, what if I did 30 of these?
15:06 Would it take 30 * 1 second, or about 30 seconds? Yeah, it’s about 1 second per item, so if I did 30, it would take about 30 seconds; 50, about 50 seconds.
Lee RP Team on Nov. 1, 2019
Link to the API endpoint: qrng.anu.edu.au/API/jsonI.php?length={nums}&type=uint16
Lee RP Team on Nov. 1, 2019
Lee RP Team on Nov. 1, 2019
Great video! I just found out that response in the worker function has a method called .json() which returns the response from the server as json. So you could remove import json and json.loads(value) and just do value = await response.json().
udomsak on Dec. 21, 2020
note: found problem when running in Windows with Proactor eventloop. not sure how to solve that. ( some wil difference from this tutorial ) Python Python 3.8
Code
import asyncio
import json
import time
import aiohttp
async def worker(name, n, session ):
print(f"worker-{name}")
url = f'https://qrng.anu.edu.au/API/jsonI.php?length={n}&type=uint16'
response = await session.request(method='GET', url=url)
value = await response.text()
return value
async def main():
async with aiohttp.ClientSession() as session:
response = await worker('bob', 3, session)
print("response: ", response)
from asyncio import WindowsSelectorEventLoopPolicy
if __name__ == '__main__':
loop = asyncio.new_event_loop()
start = time.perf_counter()
loop.run_until_complete(main())
elapsed = time.perf_counter() - start
print(f"executed in {elapsed:0.2f} seconds")
loop.close()
Exception
executed in 0.67 seconds
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000021C613178B0>
Traceback (most recent call last):
executed in 0.67 seconds
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000021C613178B0>
Traceback (most recent call last):
File "C:\Users\Admin\Documents\SCG\.venv\lib\asyncio\proactor_events.py", line 116, in __del__
self.close()
File "C:\Users\Admin\Documents\SCG\.venv\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Users\Admin\Documents\SCG\.venv\lib\asyncio\base_events.py", line 719, in call_soon
self._check_closed()
File "C:\Users\Admin\Documents\SCG\.venv\lib\asyncio\base_events.py", line 508, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Nelson Igbokwe on March 1, 2021
Hey Guys, in case anyone else can’t communicate to this api, try the following one:
“www.randomnumberapi.com/api/v1.0/random?min=100&max=1000&count=5“
Alexander on June 1, 2021
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x00000177E6437550>
Traceback (most recent call last):
File "C:\Users\Xander\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 116, in __del__
self.close()
File "C:\Users\Xander\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Users\Xander\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 746, in call_soon
self._check_closed()
File "C:\Users\Xander\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Nicholas Feliccia on Aug. 10, 2021
I have the same issue as @Alexander above:
Exception ignored in: <function ProactorBasePipeTransport.del at 0x000001D65C230550>
Traceback (most recent call last):
File "C:\Python\Python39\lib\asyncio\proactorevents.py", line 116, in del
self.close()
File "C:\Python\Python39\lib\asyncio\proactorevents.py", line 108, in close
self.loop.callsoon(self.callconnectionlost, None)
File "C:\Python\Python39\lib\asyncio\baseevents.py", line 746, in callsoon
self.checkclosed()
File "C:\Python\Python39\lib\asyncio\baseevents.py", line 510, in checkclosed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed**
dylanbright on Sept. 25, 2021
I also get this “Event loop closed” error using Python 3.9 on Windows.
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000002526444B700>
Traceback (most recent call last):
File "C:\Users\dcbadmin\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 116, in __del__
self.close()
File "C:\Users\dcbadmin\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Users\dcbadmin\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 746, in call_soon
self._check_closed()
File "C:\Users\dcbadmin\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Did a little searching, adding this line:
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
seems to be a workaround. For example:
if __name__ == '__main__':
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
start = time.perf_counter()
asyncio.run(main())
elapsed = time.perf_counter() - start
print(f'executed in {elapsed:0.2f} seconds')
Take a look at this if you want to read more about it. Seems to be an asyncio on windows thing. github.com/Dinnerbone/mcstatus/issues/133
Sandip Bhattacharya on Dec. 1, 2023
As of now, this API endpoint is rate limited. So concurrent access to the URL will not work. You will get the message:
The QRNG API is limited to 1 requests per minute.
Sandip Bhattacharya on Dec. 1, 2023
You can use this API endpoint instead: www.randomnumberapi.com/api/v1.0/random?min=100&max=1000&count=5
$ curl 'http://www.randomnumberapi.com/api/v1.0/random?min=100&max=1000&count=5'
[915,548,741,545,721]
Become a Member to join the conversation.
Muhun Kim on Aug. 10, 2019
I can’t find
ClientSessionattribute in my computer.I used Python 3.7.1