Inspect Your Data Source (Recap)
00:00 Let’s do a quick recap on Part 1, inspecting your website. This is actually a very important part of the web scraping process because you really need to understand how is the website structured and how do you interact with it before you know how to scrape it.
00:15 What you learned in here is that you first want to go and explore the website as a normal user, which means just heading to the URL with your browser and clicking around the website, essentially seeing how would you interact with it if you weren’t going to be using it with the script to scrape the information, but just using your hands and your eyes, essentially.
00:37 Next, you learned how to decipher the information that is hidden inside of URLs—the parts that a browser uses to communicate with the site, but that you, as a normal user, usually don’t have to worry about.
00:51 But there’s information in there, and you learned about query parameters and how these are structured and how you can add information in here. Change it even up in the URL and the website is going to change.
01:18 Then, finally, you looked at using developer tools to even deeper inspect the website that you’re working with. You learned how to open them up through a menu item or right-clicking, and then you can hover over different pieces of the website and see the corresponding code in HTML. Or the other way around—click on the HTML, and then see the page structure that relates to it light up. And like this, you can expand, collapse, figure out what are the actual pieces of information in the HTML structure that contain the information that you want, and that you’re going to want to target in your scraping and parsing later on.
Become a Member to join the conversation.