Responses: Content and Headers

Making HTTP Requests With Python Christopher Bailey 04:09

When dealing with requests we are mostly interested in the content we receive. In this lesson you’ll learn how to access the response content as well as the response meta information stored in the headers.

You can access the content of a response in three different ways and formats. Here is a quick overview about the different ways and formats:

content: This attribute returns the raw bytes of the response content
text: The text attribute returns the content as a normal UTF-8 encoded Python string
json(): You can use the json() method to get the response content in JSON format

The headers attributes returns a Python dictionary with all of the response headers.

00:00 Next, let’s talk about the data that actually got sent back from the server that’s in the body of the response. I’m going to go ahead and close my script and expand my terminal back up here and clear it.

00:16 So, I’m going to step back into bpython, my REPL. You can use just whatever REPL you have. This is nice because it just shows a lot of extra information. Okay.

00:25 We’re going to import requests again, we’ll set up our url,

00:31 and let’s save our Response from our GET request to the url. Let’s look at the content for the Response—and that’s one of its methods, .content. And it’s pretty verbose, a lot of information.

00:47 And what you might notice here at the very beginning of it is it saved an output in a byte format—the b at the beginning here. What may be a little more human-friendly is to output it as text, so the response.text, and you’ll see it’s just a text string then. But again, it’s still a lot of information and a little hard to look at.

01:14 These are all endpoints that we can go to to get a little more information, so we’re mainly just at the root of github.com and its API. Another way to look at it is as JSON, and that saves it as a JSON dictionary.

01:30 In fact, what I want to do is be able to look at this a little bit easier, so I’m going to make a really simple function to kind of walk through this information and be able to share it with you in a more readable way.

01:41 So let’s make a function. We’ll just call it print_d() for printing the dictionary, and we’ll put our response into it. Let’s actually put our JSON into it.

01:49 And it’s simply going to just go through a for loop. for key, value in json.items(): print(), and we’ll use an f-string here, f'{key} : {value}'.

02:06 All right, so we have a little function now that we can use called print_d(). So, the first thing we need to do is take that response and save it into—let’s just call it the JSON version of the response, json_response.

02:20 And let’s print it out, so print_d() and we’ll say json_response.

02:28 That’s a little more readable. We can see the pairs. So, the current_user_url—and that would be the api.github.com/user—the commit searching, the emails URL, stuff for emojis, and so forth. A little bit easier to walk through. In fact, we could look at any of these pairs directly if we wanted to. Let’s say we want to look at the 'repository_url'.

02:56 We can access any of them directly just by putting in the particular key. If we need a little bit more information, like metadata about the response, we can take a look at the response’s headers.

03:09 I’m going to clear my console. Another method that’s in here. And since it’s a dictionary, we can print this up nicely too.

03:21 So, these are all the headers for this Root API and the response shows the date when we made the request. The content coming back is JSON, the server, here again is our status code.

03:34 Something that’s special about this dictionary object is that since it’s part of the HTTP spec, that means that all these headers are case insensitive. So if we want to access any of them directly, we can access 'Content-Type'. In this case, again, we mentioned earlier that it’s JSON, but let’s look at it again.

03:54 But this time, if we were to do this in lowercase, and we get the same response, which is nice! Okay. We’re going to wrap this section up in the next video.

Abby Jones on June 25, 2019

I am learning SO much about Python, VIM, and now bpython. RealPython is literally the best site I have ever learned from. It’s increasing my workflow productivity.

Pygator on Sept. 15, 2019

how can a dictionary have multiple keys that produce the same value or are they duplicated? Doesn’t that break all universal laws of python dicts?

km on Dec. 6, 2019

Ending with JSONDecodeError: Expecting value, used url=’www.ucla.edu’ - Any help

Chris Bailey RP Team on Dec. 6, 2019

Hi Kishoremarodia, The url you are suggesting is not an API end point, and returns a response of html elements instead of JSON. Is there an API you would like to explore at ucla.edu?

km on Dec. 7, 2019

Thanks a lot, I just took it randomly. Is there any way to find out an END POINT API which can send elements in JSON? The API.github.com is rejecting requests.

Chris Bailey RP Team on Dec. 9, 2019

Hi km, I just tried the example in the lesson, and it responded the same as in the lesson. Maybe its the number of times you have tried it, hopefully that would reset after a while. If you are trying some of the later lessons, you would need an account on github, and then would use your credentials. You could also try some from API’s this reddit thread. Many of them will require you to make an account.

balakumaranrk on Feb. 8, 2020

what is the difference between response.json() and response.content ? Both of them fetch key-value pairs right ?

Ricky White RP Team on Feb. 8, 2020

Not exactly. response.content will give you the raw data which you can parse into json or a dictionary object easily.

yanzaripov on May 3, 2020

Hi that’s my code

url = 'https://instagram.com'

response = requests.get(url)

#output as json
response.json()

Please explain me why do I get keep getting this Error Message:

JSONDecodeError Traceback (most recent call last) <ipython-input-30-7ccffdca51ca> in <module>() 8 9 #output as json —> 10 response.json()

3 frames /usr/local/lib/python3.6/dist-packages/requests/models.py in json(self, kwargs) 896 # used. 897 pass –> 898 return complexjson.loads(self.text, kwargs) 899 900 @property

/usr/lib/python3.6/json/init.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 352 parse_int is None and parse_float is None and 353 parse_constant is None and object_pairs_hook is None and not kw): –> 354 return _default_decoder.decode(s) 355 if cls is None: 356 cls = JSONDecoder

/usr/lib/python3.6/json/decoder.py in decode(self, s, _w) 337 338 “”” –> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 340 end = _w(s, end).end() 341 if end != len(s):

/usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx) 355 obj, end = self.scan_once(s, idx) 356 except StopIteration as err: –> 357 raise JSONDecodeError(“Expecting value”, s, err.value) from None 358 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Ricky White RP Team on May 4, 2020

Hi @yanzaripov. You are getting that error because that url does not return a json response. If you want to play with json responses, you should use a api endpoint that returns json. For example:

url = "https://api.chucknorris.io/jokes/random"
response = requests.get(url)
print(response.json())

techsukenik on Sept. 15, 2021

@yanzaripov ..

If you check the header Content-type it will tell you if the output is in json or not.

import requests
url = 'https://instagram.com'

response = requests.get(url)

#output as json
print(f"{url} Non Json  Content-type {response.headers['Content-Type']}")

url = 'https://api.github.com/api'
response = requests.get(url)
print(f"{url} Json Content-type {response.headers['Content-Type']}")

Output:

https://instagram.com Non Json  Content-type text/html; charset=utf-8
https://api.github.com/api Json Content-type application/json; charset=utf-8

Become a Member to join the conversation.