Python's Requests Library (Guide)

Getting Started With Python’s Requests Library

Even though the Requests library is a common staple for many Python developers, it’s not included in Python’s standard library. There are good reasons for that decision, primarily that the library can continue to evolve more freely as a self-standing project.

Because Requests is a third-party library, you need to install it before you can use it in your code. As a good practice, you should install external packages into a virtual environment, but you may choose to install requests into your global environment if you’re planning to use it across multiple projects.

Whether you’re working in a virtual environment or not, you’ll need to install requests:

$ python -m pip install requests

Once pip has finished installing requests, you can use it in your application. Importing requests looks like this:

import requests

Now that you’re all set up, it’s time to begin your journey through Requests. Your first goal will be learning how to make a GET request.

The GET Request

HTTP methods, such as GET and POST, determine which action you’re trying to perform when making an HTTP request. Besides GET and POST, there are several other common methods that you’ll use later in this tutorial.

One of the most common HTTP methods is GET. The GET method indicates that you’re trying to get or retrieve data from a specified resource. To make a GET request using Requests, you can invoke requests.get().

To test this out, you can make a GET request to GitHub’s REST API by calling get() with the following URL:

>>> import requests
>>> requests.get("https://api.github.com")
<Response [200]>

Congratulations! You’ve made your first request. Now you’ll dive a little deeper into the response of that request.

The Response

A Response is a powerful object for inspecting the results of the request. Make that same request again, but this time store the return value in a variable so that you can get a closer look at its attributes and behaviors:

>>> import requests
>>> response = requests.get("https://api.github.com")

In this example, you’ve captured the return value of get(), which is an instance of Response, and stored it in a variable called response. You can now use response to see a lot of information about the results of your GET request.

>>> response.status_code
200

if response.status_code == 200:
    print("Success!")
elif response.status_code == 404:
    print("Not Found.")

if response:
    print("Success!")
else:
    raise Exception(f"Non-success status code: {response.status_code}")

import requests
from requests.exceptions import HTTPError

URLS = ["https://api.github.com", "https://api.github.com/invalid"]

for url in URLS:
    try:
        response = requests.get(url)
        response.raise_for_status()
    except HTTPError as http_err:
        print(f"HTTP error occurred: {http_err}")
    except Exception as err:
        print(f"Other error occurred: {err}")
    else:
        print("Success!")

>>> import requests

>>> response = requests.get("https://api.github.com")
>>> response.content
b'{"current_user_url":"https://api.github.com/user", ...}'

>>> type(response.content)
<class 'bytes'>

>>> response.text
'{"current_user_url":"https://api.github.com/user", ...}'

>>> type(response.text)
<class 'str'>

>>> response.encoding = "utf-8"  # Optional: Requests infers this.
>>> response.text
'{"current_user_url":"https://api.github.com/user", ...}'

>>> response.json()
{'current_user_url': 'https://api.github.com/user', ...}

>>> type(response.json())
<class 'dict'>

>>> response_dict = response.json()
>>> response_dict["emojis_url"]
'https://api.github.com/emojis'

>>> import requests

>>> response = requests.get("https://api.github.com")
>>> response.headers
{'Server': 'GitHub.com',
...
'X-GitHub-Request-Id': 'AE83:3F40:2151C46:438A840:65C38178'}

>>> response.headers["Content-Type"]
'application/json; charset=utf-8'

>>> response.headers["content-type"]
'application/json; charset=utf-8'

Query String Parameters

One common way to customize a GET request is to pass values through query string parameters in the URL. To do this using get(), you pass data to params. For example, you can use GitHub’s repository search API to look for popular Python repositories:

import requests

# Search GitHub's repositories for popular Python projects
response = requests.get(
    "https://api.github.com/search/repositories",
    params={"q": "language:python", "sort": "stars", "order": "desc"},
)

# Inspect some attributes of the first three repositories
json_response = response.json()
popular_repositories = json_response["items"]
for repo in popular_repositories[:3]:
    print(f"Name: {repo['name']}")
    print(f"Description: {repo['description']}")
    print(f"Stars: {repo['stargazers_count']}")
    print()

By passing a dictionary to the params parameter of get(), you’re able to modify the results that come back from the search API.

You can pass params to get() in the form of a dictionary, as you’ve just done, or as a list of tuples:

>>> import requests

>>> requests.get(
...     "https://api.github.com/search/repositories",
...     [("q", "language:python"), ("sort", "stars"), ("order", "desc")],
... )
<Response [200]>

You can even pass the values as bytes:

>>> requests.get(
...     "https://api.github.com/search/repositories",
...     params=b"q=language:python&sort=stars&order=desc",
... )
<Response [200]>

Query strings are useful for parameterizing GET requests. Another way to customize your requests is by adding or modifying the headers that you send.

Request Headers

To customize headers, you pass a dictionary of HTTP headers to get() using the headers parameter. For example, you can change your previous search request to highlight matching search terms in the results by specifying the text-match media type in the Accept header:

import requests

response = requests.get(
    "https://api.github.com/search/repositories",
    params={"q": '"real python"'},
    headers={"Accept": "application/vnd.github.text-match+json"},
)

# View the new `text-matches` list which provides information
# about your search term within the results
json_response = response.json()
first_repository = json_response["items"][0]
print(first_repository["text_matches"][0]["matches"])

The Accept header tells the server what content types your application can handle. In this case, since you’re expecting the matching search terms to be highlighted, you’re using the header value application/vnd.github.text-match+json, which is a proprietary GitHub Accept header where the content is a special JSON format.

If you run this code, then you’ll get a result similar to the one shown below:

$ python text_matches.py
[{'text': 'Real Python', 'indices': [23, 34]}]

Before you learn more ways to customize requests, you’ll broaden your horizons by exploring other HTTP methods.

Other HTTP Methods

Aside from GET, other popular HTTP methods include POST, PUT, DELETE, HEAD, PATCH, and OPTIONS. For each of these HTTP methods, Requests provides a function, with a similar signature to get().

You’ll notice that Requests provides an intuitive interface to all the mentioned HTTP methods:

>>> import requests

>>> requests.get("https://httpbin.org/get")
<Response [200]>
>>> requests.post("https://httpbin.org/post", data={"key": "value"})
<Response [200]>
>>> requests.put("https://httpbin.org/put", data={"key": "value"})
<Response [200]>
>>> requests.delete("https://httpbin.org/delete")
<Response [200]>
>>> requests.head("https://httpbin.org/get")
<Response [200]>
>>> requests.patch("https://httpbin.org/patch", data={"key": "value"})
<Response [200]>
>>> requests.options("https://httpbin.org/get")
<Response [200]>

In the example code above, you called each function to make a request to the httpbin service using the corresponding HTTP method.

>>> requests.request("GET", "https://httpbin.org/get")
<Response [200]>

You can inspect the responses in the same way as you did before:

>>> response = requests.head("https://httpbin.org/get")
>>> response.headers["Content-Type"]
'application/json'

>>> response = requests.delete("https://httpbin.org/delete")
>>> json_response = response.json()
>>> json_response["args"]
{}

Headers, response bodies, status codes, and more are returned in the Response for each method.

Next you’ll take a closer look at the POST, PUT, and PATCH methods and learn how they differ from the other request types.

The Message Body

According to the HTTP specification, POST, PUT, and the less common PATCH requests pass their data through the message body rather than through parameters in the query string. Using Requests, you’ll pass the payload to the corresponding function’s data parameter.

data takes a dictionary, a list of tuples, bytes, or a file-like object. You’ll want to adapt the data that send in the body of your request to the specific needs of the service that you’re interacting with.

For example, if your request’s content type is application/x-www-form-urlencoded, then you can send the form data as a dictionary:

>>> import requests

>>> requests.post("https://httpbin.org/post", data={"key": "value"})
<Response [200]>

You can also send that same data as a list of tuples:

>>> requests.post("https://httpbin.org/post", data=[("key", "value")])
<Response [200]>

If, however, you need to send JSON data, then you can use the json parameter. When you pass JSON data via json, Requests will serialize your data and add the correct Content-Type header for you.

Like you learned earlier, the httpbin service accepts test requests and responds with data about the requests. For instance, you can use it to inspect a basic POST request:

>>> response = requests.post("https://httpbin.org/post", json={"key": "value"})
>>> json_response = response.json()
>>> json_response["data"]
'{"key": "value"}'
>>> json_response["headers"]["Content-Type"]
'application/json'

You can see from the response that the server received your request data and headers as you sent them. Requests also provides this information to you in the form of a PreparedRequest that you’ll inspect in more detail in the next section.

Request Inspection

When you make a request, the Requests library prepares the request before actually sending it to the destination server. Request preparation includes things like validating headers and serializing JSON content.

You can view the PreparedRequest object by accessing .request on a Response object:

>>> import requests

>>> response = requests.post("https://httpbin.org/post", json={"key":"value"})

>>> response.request.headers["Content-Type"]
'application/json'
>>> response.request.url
'https://httpbin.org/post'
>>> response.request.body
b'{"key": "value"}'

Inspecting PreparedRequest gives you access to all kinds of information about the request being made, such as payload, URL, headers, authentication, and more.

So far, you’ve made a lot of different kinds of requests, but they’ve all had one thing in common: they’re unauthenticated requests to public APIs. Many services you may come across will want you to authenticate in some way.

Authentication

Authentication helps a service understand who you are. Typically, you provide your credentials to a server by passing data through the Authorization header or a custom header defined by the service. All the functions of Requests that you’ve seen to this point provide a parameter called auth, which allows you to pass your credentials:

>>> import requests

>>> response = requests.get(
...     "https://httpbin.org/basic-auth/user/passwd",
...     auth=("user", "passwd")
... )

>>> response.status_code
200
>>> response.request.headers["Authorization"]
'Basic dXNlcjpwYXNzd2Q='

The request succeeds if the credentials that you pass in the tuple to auth are valid.

When you pass your credentials in a tuple to the auth parameter, Requests applies the credentials using HTTP’s Basic access authentication scheme under the hood.

You could make the same request by passing explicit Basic authentication credentials using HTTPBasicAuth:

>>> from requests.auth import HTTPBasicAuth
>>> requests.get(
...     "https://httpbin.org/basic-auth/user/passwd",
...     auth=HTTPBasicAuth("user", "passwd")
... )
<Response [200]>

Though you don’t need to be explicit for Basic authentication, you may want to authenticate using another method. Requests provides other methods of authentication out of the box, such as HTTPDigestAuth and HTTPProxyAuth.

A real-world example of an API that requires authentication is GitHub’s authenticated user API. This endpoint provides information about the authenticated user’s profile.

If you try to make a request without credentials, then you’ll see that the status code is 401 Unauthorized:

>>> requests.get("https://api.github.com/user")
<Response [401]>

If you don’t provide authentication credentials when accessing a service that requires them, then you’ll get an HTTP error code as a response.

To make a request to GitHub’s authenticated user API, you first need to generate a personal access token with the read:user scope. Then you can pass this token as the second element in a tuple to get():

>>> import requests

>>> token = "<YOUR_GITHUB_PA_TOKEN>"
>>> response = requests.get(
...     "https://api.github.com/user",
...     auth=("", token)
... )
>>> response.status_code
200

Like you learned previously, this approach passes the credentials to HTTPBasicAuth, which expects a username and a password and sends the credentials as a Base64-encoded string with the prefix "Basic ":

>>> response.request.headers["Authorization"]
'Basic OmdocF92dkd...WpremM0SGRuUGY='

This works, but it’s not the right way to authenticate with a Bearer token—and using an empty string input for the superfluous username is awkward.

With Requests, you can supply your own authentication mechanism to fix that. To try this out, create a subclass of AuthBase and implement .__call__():

from requests.auth import AuthBase

class TokenAuth(AuthBase):
    """Implements a token authentication scheme."""

    def __init__(self, token):
        self.token = token

    def __call__(self, request):
        """Attach an API token to the Authorization header."""
        request.headers["Authorization"] = f"Bearer {self.token}"
        return request

Here, your custom TokenAuth mechanism receives a token, then includes that token in the Authorization header of your request, also setting the recommended "Bearer " prefix to the string.

You can now use this custom token authentication to make your call to GitHub’s authenticated user API:

>>> import requests
>>> from custom_token_auth import TokenAuth

>>> token = "<YOUR_GITHUB_PA_TOKEN>"
>>> response = requests.get(
...     "https://api.github.com/user",
...     auth=TokenAuth(token)
... )

>>> response.status_code
200
>>> response.request.headers["Authorization"]
'Bearer ghp_b...Tx'

Your custom TokenAuth created a well-formatted string for the Authorization header. You can now use this more intuitive way of interacting with a token-based authentication scheme such as the one that parts of GitHub’s API require.

Bad authentication mechanisms can lead to security vulnerabilities. Unless a service requires a custom authentication mechanism for some reason, you’ll always want to use a tried-and-true auth scheme like the built-in Basic authentication or OAuth, for example through Requests-OAuthlib.

While you’re thinking about security, consider dealing with SSL certificates using Requests.

SSL Certificate Verification

Anytime the data that you’re trying to send or receive is sensitive, security is important. The way that you communicate with secure sites over HTTP is by establishing an encrypted connection using SSL, which means that verifying the target server’s SSL certificate is critical.

The good news is that Requests does this for you by default. However, there are some cases where you might want to change this behavior.

If you want to disable SSL certificate verification, then you pass False to the verify parameter of the request function:

>>> import requests

>>> requests.get("https://api.github.com", verify=False)
InsecureRequestWarning: Unverified HTTPS request is being made to host
⮑ 'api.github.com'. Adding certificate verification is strongly advised.
⮑ See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
⮑  warnings.warn(
<Response [200]>

Requests even warns you when you’re making an insecure request to help you keep your data safe!

Now that you know how to make all sorts of HTTP requests using Requests, authenticated or not, you may wonder about how you can make sure that your program works as quickly as possible.

In the next section, you’ll learn about a few ways that you can improve performance with the help of Requests.

Performance

When using Requests, especially in a production application environment, it’s important to consider performance implications. Features like timeout control, sessions, and retry limits can help you keep your application running smoothly.

>>> requests.get("https://api.github.com", timeout=1)
<Response [200]>
>>> requests.get("https://api.github.com", timeout=3.05)
<Response [200]>

>>> requests.get("https://api.github.com", timeout=(3.05, 5))
<Response [200]>

import requests
from requests.exceptions import Timeout

try:
    response = requests.get("https://api.github.com", timeout=(3.05, 5))
except Timeout:
    print("The request timed out")
else:
    print("The request did not time out")

 1import requests
 2from custom_token_auth import TokenAuth
 3
 4TOKEN = "<YOUR_GITHUB_PA_TOKEN>"
 5
 6with requests.Session() as session:
 7    session.auth = TokenAuth(TOKEN)
 8
 9    first_response = session.get("https://api.github.com/user")
10    second_response = session.get("https://api.github.com/user")
11
12print(first_response.headers)
13print(second_response.json())

import requests
from requests.adapters import HTTPAdapter
from requests.exceptions import RetryError

github_adapter = HTTPAdapter(max_retries=2)

session = requests.Session()

session.mount("https://api.github.com", github_adapter)

try:
    response = session.get("https://api.github.com/")
except RetryError as err:
    print(f"Error: {err}")
finally:
    session.close()

Conclusion

Nice work, you’ve made it to the end of the tutorial, and you’ve come a long way in increasing your knowledge about Python’s powerful Requests library.

In this tutorial, you’ve learned how to:

• Make requests using a variety of different HTTP methods such as GET, POST, and PUT
• Customize your requests by modifying headers, authentication, query strings, and message bodies
• Inspect the data you send to the server and the data the server sends back to you
• Work with SSL certificate verification
• Use Requests effectively with max_retries, timeout, sessions, and transport adapters

Because you learned how to use Requests, you’re equipped to explore the wide world of web services and build awesome applications using the fascinating data they provide.

Take the Quiz: Test your knowledge with our interactive “HTTP Requests With the "requests" Library” quiz. Upon completion you will receive a score so you can track your learning progress over time:

---

Interactive Quiz

HTTP Requests With the "requests" Library

Test your understanding of the Python "requests" library for making HTTP requests and interacting with web services.