Perform Basic HTTP Requests With urllib.request
00:00
In this lesson, you’ll learn how to perform basic HTTP requests with urllib.request. Before diving deep into what HTTP request is and how it works, you can start by making a basic GET request to a sample URL.
00:13
In this case, you’ll make a request to example.com. You’ll also make a GET request to a mock REST API, such as {JSON} Placeholder, for some JSON data.
00:22 First, make sure you’re using Python 3 or above. Then head over to your favorite code editor and open up a new file.
00:29
Here I am in my code editor, VS Code, and I just created a file named HTTPResponse.py. First, you’ll import urlopen from urllib.request.
00:41
Using the context manager with, you make a request and receive a response with urlopen(). The request will be to an example URL—in this case, www.example.com.
00:53 You’ll be able to find this in the content below the video.
00:56 Then you read the body of the response, and the context opened earlier will automatically close the connection. It is important to close the connection because keeping it open can lead to slow execution and bugs.
01:08 You’ll learn more about closing HTTPResponse objects in a later lesson. You then display the first fifteen positions of the body to check whether the request worked and what got returned.
01:19
To do so, while using the print() function, pass in body. Then you add square brackets with [:15] to slice the content of body at position 15.
01:30
If you want to display the entire body, you can omit the square bracket notation and simply print(body). You’ll know it worked if it looks like an HTML document.
01:41
You can now move over to your terminal and run the Python script you just created. Type python or python3, depending on your operating system, space, HTTPResponse.py, and then press Enter.
01:53
As you can see, the response you got back is an HTML document. Notice the printed output of the body is preceded by b. This indicates a bytes literal, which you may need to decode. Later you’ll learn how to turn bytes into a string, write them to a file, or parse them into a dictionary.
02:11
In the next example, you’ll make a request to {JSON} Placeholder for some fake to-do data. In the previous example, you saw HTML as the response you got back from example.com.
02:21
When you work with APIs, the response is often in JSON format. {JSON} Placeholder offers some API endpoints to play around with when using urllib. Specifically, you’ll be using their /todos endpoint with an id of 1.
02:35 You can break out your favorite code editor and jump into the code now.
02:39
Now remove the content from HTTPResponse.py and start fresh. The content will be similar, but typing it all out again will be good training.
02:49
Okay, your first step is to import urllib.request like so. Next, you can import json. You’ll want to set the URL to the JSON data endpoint.
03:01
The JSON data endpoint is the part of the URL that specifies the desired resource, such as /todos/1. Everything preceding that endpoint is the base URL.
03:11
The URL will be https://jsonplaceholder.typicode.com/todos/1.
03:21
Ensure proper closure by using the with statement and passing url into the urlopen() function. Read the server’s response into the body variable using the .read() method.
03:32
Next, parse the JSON data in the body variable into a Python object using json.loads() function. Assign the result to the variable todo_item.
03:42
This will be todo_item = json.loads(body), and print the value of todo_item using the print() function. In the terminal, you can run your script with py urllib_requests.py and then hit Enter.
04:03
You’ll get back a collection of key-value pairs such as the key userId with the value of 1 and the key id with the value of 1.
04:13
You also see keys like completed and title. So to recap, in this example, you import urllib.request and json, using the json.loads() function with body to decode and parse the returned JSON bytes into a Python dictionary.
04:31
Now that you’ve got your feet wet, next you’ll learn the underlying structure of HTTP messages and learn how urllib.request handles them.
Bartosz Zaczyński RP Team on Jan. 3, 2024
@Logeshkumar The urllib package ships with Python through its standard library, so you don’t need to install anything. What you’ve installed is a separate third-party module called urllib3, which has a slightly different interface and feature set. By installing it with pip, you’ve likely created a conflict, which you can try undoing with the following command:
$ python3 -m pip uninstall urllib3
Dillon O'Leary on Nov. 26, 2024
If the python bytes object doesn’t even know the character set I’m using, how is it able to create a string representation? at time 1:54
Bartosz Zaczyński RP Team on Nov. 27, 2024
@Dillon O’Leary Character encodings are irrelevant to bytes instances because their string representations always consist of ASCII literal characters or hexadecimal representations of non-ASCII byte values:
>>> "This text has only ASCII letters and <special> characters".encode("utf-8")
b'This text has only ASCII letters and <special> characters'
>>> "This text has non-ASCII letters: naïve café".encode("utf-8")
b'This text has non-ASCII letters: na\xc3\xafve caf\xc3\xa9'
In the examples above, the first string contains only ASCII characters, so its encoded byte representation is identical to the original string. In contrast, the second string includes non-ASCII characters like ï and é. These characters are encoded in the byte string using their respective hexadecimal representations using the specified character encoding.
Dillon O'Leary on Nov. 29, 2024
Thank you for the response, that makes sense!
I also found this youtube video that help me understand UTF-8 if anyone else is confused: www.youtube.com/watch?v=MijmeoH9LT4
Become a Member to join the conversation.
Logeshkumar on Jan. 3, 2024
Do I need to install
urllibpackage?Even though I installed
urllib3usingpip, I am getting below error.