Query String Parameters

Dear Christopher Bailey,

thank you very much for the great tutorial.

I have a special question.

Using the request library I print out the result of the url = ‘www.dropbox.com/h?preview=ophelp.txt‘

I would like to store the content of this url into it’s name = ‘ophelp’.

How can I use the request library to get the name searching for ‘preview=’ and getting ‘ophelp’?

I thought about using the query string parameters as you shown in your tutorial. But I am not sure how to search for it exactly.

Thank you Anja

Hi @agerbes Anja, I’m not sure I completely understand your question. It looks like you want to search for a file on dropbox, and get a preview of the text. I believe searches would be limited to a particular users files. I haven’t used the dropbox api, but I looked it over a bit and you would need to use a different url to use requests with it. The url would be 'https://api.dropbox.com/'. I did find some of the resources available from their site. I hope they can help you with what you are trying to accomplish. This link has the http documentation for using the dropbox api, including the available end points: Dropbox for HTTP

This link is to their Python package for doing development. It looks pretty well documented also: Dropbox for Python Developers. I hope some of this helps.

The speed of these videos is quite fast, IMHO. Speed and depth of explanations not really beginner-friendly.

Thanks for your feedback there @jc123k. This course is aimed at an intermediate skill level, so it isn’t intended to be a beginner’s tutorial. But thank you for the feedback, maybe there’s room for doing a beginner-focused course or tutorial on this topic in the future :)

The response I got does not relate to the webpage. Where does the data come from?

Response from search.py:

Repository name: grequests Repository description: Requests + Gevent = <3

Response from Url(api.github.com/search/repositories):

{ “message”: “Validation Failed”, “errors”: [ { “resource”: “Search”, “field”: “q”, “code”: “missing” } ], “documentation_url”: “developer.github.com/v3/search” }

Sorry to correct what I said this is the response from the repo:

{ “message”: “Validation Failed”, “errors”: [ { “message”: “The listed users and repositories cannot be searched either because the resources do not exist or you do not have permission to view them.”, “resource”: “Search”, “field”: “q”, “code”: “invalid” } ], “documentation_url”: “developer.github.com/v3/search/” }

Got it. Did some research on string query and worked out what to use in url: api.github.com/search/repositories?q=schedule+user:dbader

DaveyD, I also had the same response from search.py

Repository name: grequests Repository description: Requests + Gevent = <3

if I omitted the ‘+language:python’ part of the query then ‘requests’ is positioned at index 0 of ‘items’. (params={‘q’: ‘requests’}). it seems true for today anyway.

Repository name: requests Repository description: A simple, yet elegant HTTP library.

I’m guessing that just means the repository structure at github has changed a little since the video was made. Since we are looking for index 0 of the items both results are successful queries. Just not what we expected.

Thinking more about why grequests came out at index 0. some questions:

1. When using multiple params are they ‘or’ed together or ‘and’ed together? I was thinkng that you added ‘+language:python’ as an ‘and’ type of filter in case there were other repos named ‘requests’ from another language and you didn’t want them in the response.

2. Since grequests was a match I infer that query ‘requests’ will match any name that contains ‘requests’? i.e. fiddlerequests, frequests lorequests. Or maybe that match was because it is a python language repo? I’m confused as to why grequests matched?

3. Is json.items() an alphabetically ordering?, i.e. if there was a repo name arequests would that bump grequests down.?

Hi @DaveyD and @mikesult, Wow. A lot has changed in the last year, not only for the results of this search but for the “requests” package. It is now under the Python Software Foundation the PSF.

You are both right that existing search now returns ‘grequests’ as the first item. And you are correct, if you remove the ‘+language:python’ part it will now return ‘requests’ as a first item. Not sure why it is not under ‘Python’ for the language, weird.

I was thinking if you want to see more of the search results you could add a simple for loop.

# at the end of the file change it to
json_response = response.json()
for i in range(len(json_response['items')):
    repository = json_response['items'][i]
    print(f'{i}: Respository name: {repository["name"]}')
    print(f'{i}: Respository description: {repository["description"]}')

This would give you list of the returned search items.

To answer your other questions

1: The ‘+’ works like an ‘AND’ statement, so it will reduce your results. In this case narrowing to only the keyword language, with Python. You can try this search out directly from the search bar on the actual site github.com itself. Another keywords as I show later is ‘user:’ and I just tried out another ‘topic:’

2: You are right that it will look within the name of the repository but it also looks at (searches) the description text for the word. It’s truly doing a search.

3: It is a ranking, not alpha. So it depends on how github orders the results.

I hope this helps. Any site will have their own scheme for how this will work. And it will be a moving target for a video to capture results of a search for the moment in time.

Hi Chris and Real Python community! Your tutorial is amazing and I’m learning a lot. Quick question about the query string for the first example. When I go to this url in my browser: api.github.com/search/respositories?q=requests+language:python

I get this:

{ “message”: “Not Found”, “documentation_url”: “developer.github.com/v3” }

I read through the comments here and just want to verify that it won’t work anymore because Github moved it to another repository? Is that correct or am I just misunderstanding?

Hi again, about my previous comment. When I run the code for:

response = requests.get( “api.github.com/search/repositories”, params={‘q’:’requests+language:python’}, )

I’m getting JSON data back, so maybe the way I constructed the URL is incorrect: api.github.com/search/respositories?q=requests+language:python

Hi @Tonya,

As you noticed in my earlier comments, the URL scheme seems to be changed a bit for where to find “requests”. Here is a URL scheme that seems to work now. https://api.github.com/search/repositories?q=requests+org:psf . It looks like the “full name” is now psf/requests. The types of things you can specify in the query string seem to really vary over time. “org” for organizations, language, etc. You can also get an idea if you want to see the types of URL formats, from this URL: https://api.github.com, it will return JSON of the end points to explore. And the documentation is also here.

Have fun exploring!

Hi

I found the root cause of the problem. It is due to percent encoding scheme performed by the requests package on “+” and “:”.

The solution is to pass the string directly.

payload = {‘q’: ‘requests+language:python’, ‘sort’: ‘stars’, ‘order’: ‘desc’} #added two more fields to get the item 0 correctly.

response = requests.get(‘api.github.com/search/repositories’, params=payload)

print(response.url)

Output: (No good) api.github.com/search/repositories?q=requests%2Blanguage%3Apython&sort=stars&order=desc Repository name: httpbin Repository description: HTTP Request & Response Service, written in Python + Flask.

payload_str = “q=requests+language:python&sort=stars&order=desc”

response = requests.get(‘api.github.com/search/repositories’, params=payload_str)

print(response.url)

Output: (This is what we want to see) api.github.com/search/repositories?q=requests+language:python&sort=stars&order=desc Repository name: requests Repository description: A simple, yet elegant HTTP library.

Perfect result!!!

stackoverflow.com/questions/23496750/how-to-prevent-python-requests-from-percent-encoding-my-urls

payload = {‘format’: ‘json’, ‘key’: ‘site:dummy+type:example+group:wheel’}

payload_str = “&”.join(“%s=%s” % (k,v) for k,v in payload.items())

r = requests.get(url, params=payload_str) print(r.url)

The above solution is contributed by a user named furas on Stackoverflow.

Thanks for the solution @mikelisfbay! I have used your proposed solution in the code below to get the requests repository:

payload_str = “q=requests+language:python&sort=stars&order=desc”

response = requests.get( ‘api.github.com/search/repositories’, params=payload_str )

json_response = response.json() repository = json_response[‘items’][0]