Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

This lesson is for members only. Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Hint: You can adjust the default video playback speed in your account settings.
Hint: You can set the default subtitles language in your account settings.
Sorry! Looks like there’s an issue with video playback 🙁 This might be due to a temporary outage or because of a configuration issue with your browser. Please see our video player troubleshooting guide to resolve the issue.

Scrape HTML Content From a Page (Introduction)

00:00 After inspecting your page in Part 1 you’re now ready for Part 2, which is actually scraping the content information from the website onto your computer using Python, some coding, and the requests library.

00:14 This whole course is focused on static websites—that’s something to keep in mind. There’s other situations on the web, and we will talk a bit about them in this part of the course, but just keep in mind that what you’re learning to do here is scraping static websites, and it is a little bit different and a little bit more complex for other situations. So, I’m going to walk you over how to do this with the Indeed site.

00:36 We’re going to use Jupyter Notebooks for that. And then we are also going to talk a bit about how would you do that with a hidden website, which is just the term I’m using here for websites that are password-protected—so, if you have to log in to access the information that you’re interested in, then your process is a little bit different. We will take a look at that.

00:56 And then finally, there’s also dynamic websites, which means that the content that you’re actually interested in doesn’t get sent back directly by the server, but instead it sends some JavaScript code that your browser executes to generate the content that you’re interested in.

01:12 This is a bit more difficult to scrape because you can’t just ask the server for the information and get the information you want, but you need to run some code first to generate the information.

01:23 We’re not going to go into depth for password-protected sites nor dynamic websites—those are different topics—but you’re going to lay a very great foundation for learning more about those as well if you go over the process of scraping static websites, which is what you’re learning in this course. Okay! So as I mentioned, static websites is our focus and we’re going to start off by scraping the Indeed site that you inspected in the previous part. Then, we will talk a little bit about these other options, these other ways of scraping, and other procedures that you need for those.

01:57 See you in the next lesson!

Become a Member to join the conversation.