Managing Demand

00:00 As you start to make more API requests, you need to be aware of how much demand you are placing on their servers.

00:07 So how busy are servers? Popular websites and services manage millions of simultaneous requests, and as you can imagine, that puts a lot of strain on the server.

00:18 It’s like if you worked in a restaurant. A few orders every now and then would be fine, but everyone arriving all at once and ordering at the same time would be difficult.

00:27 It’s for this reason that it’s important to avoid placing too many demands on the API.

00:33 In this lesson, you’re going to learn how APIs can control their usage with rate limiting and how you can control the amount of data you receive with pagination.

00:44 Rate limiting is when APIs restrict the number of requests in a given time frame. Any user who exceeds this number of requests might be blocked from the service, usually for a short period of time.

00:56 For example, GitHub allows about 60 unauthenticated requests per hour. If you exceed that, then your response will contain a message saying that your API rate limit has been exceeded.

01:08 Another way to reduce the demand on the server is to use pagination, where you limit the number of results you want to receive from each request. Then, when you’re ready for more results, you can request the next bunch.

01:19 This is just like on Google, where the results are split up onto different pages so you don’t load them all at once.

01:26 You can see this in action in the REPL. You’re going to import requests and save the API’s URL as usual. This time you’re using the events API, which can be found at https://api.github.com/events.

01:43 Now normally, this would return all the activity that’s happening on GitHub, which you can imagine would be a lot of results. You can reduce the number of results to ask for by including query parameters defining two things.

01:55 Firstly, the number of results you want per page, the parameter for which is called "per_page". In this example, you should ask for one item per page to make it easier to compare two different pages.

02:05 Secondly, which page you want. You should start with page zero, which is the first page.

02:10 Now you can send the request like you’ve done before and save the response in a variable.

02:17 You can investigate the data in the response using the .json() method. You can then repeat the process, but this time adjust your query parameters to ask for page one.

02:31 This time, after receiving a response and using the .json() method, you should see a different event.

02:40 You can confirm the events are different by comparing their IDs, for example. Feel free to keep exploring this and trying different numbers of results per page.

02:51 To demonstrate how you might combine the ideas of rate limiting with pagination, here is a short script that fetches repositories from the Python Foundation on GitHub.

03:01 In this script, I’ve imported requests as usual, but I’ve also imported the sleep function from the time module. This will help you slow down how quickly you are making requests so you don’t place too much demand on the API.

03:13 Next, I save the API URL as a variable: https://api.github.com/org/python/repos I also create a page variable to keep track of which page of results I’m on.

03:29 If you know the exact number of pages you are looping through, then you can use a for loop next, but if you don’t, you can use a while loop.

03:37 I’m using while True here, which means the loop will keep running until I actively break out of it. Each time the while loop runs, I set the query parameters and make the request like you’ve done in previous lessons.

03:49 Notice I’m asking for 25 results per page, and then the "page" parameter is the current value of the page variable. This will increase with every loop.

03:59 I’m extracting the results from the response using the .json() method. When you go beyond the final page of results, the response won’t contain anything.

04:08 That is, if there are four pages of results and you try to request the fifth, the fifth page will be empty. This if statement checks if the response is empty and leaves the while loop if it is.

04:19 It’s worth noting that other APIs can work differently. For instance, some of them will tell you how many pages there are, or will contain links that take you to the next page, so you won’t always have to check if the results are empty.

04:31 As always, check the API’s documentation for guidance. I then print how many results are on the page using the len() function, and then I increment the page number ready to repeat the process.

04:43 To make sure I’m not making requests too quickly, I have the program sleep for one second before fetching the next page.

04:50 I can run it now and confirm that it returns 25 results per page, but on the fourth page, there aren’t enough results left to fill up a full page. That’s why you only get 15.

05:02 And then the program receives no results on the next iteration, so it breaks out of the while loop and finishes.

05:09 You’ve made it to the end of the course! Congratulations! You’ve learned so much about APIs and now have the skills to incorporate their amazing services in your own projects.

05:18 In the next lesson, you’ll see a summary of everything you’ve learned, as well as some ideas for things you can explore next.