A Message From the Future

Web Scraping With Beautiful Soup and Python Martin Breuss 02:57

⚠️ Durabilty Warning ⚠️

Like mentioned in the course, websites frequently change. Unfortunately the job board that you’ll see in the course, indeed.com, has started to block scraping of their site since the recording of the course.

Just like in the associated written tutorial on web scraping with beautiful soup, you can instead use Real Python’s fake jobs site to practice scraping a static website.

All the concepts discussed in the course lessons are still accurate. Translating what you see onto a different website will be a good learning opportunity where you’ll have to synthesize the information and apply it practically.

00:00 Hello from the future. Before I can let you move on, here’s a little warning. As I just mentioned before, durability can be a problem with web scrapers, and indeed it seems that the time for indeed.com has run out unfortunately.

00:16 If you try to scrape the website as I’m showing in this course, you’ll run into this issue of durability.

00:26 If you’d use a URL like the one that you’ll see inside of the course, and you use the same techniques to run the code, then you will see that you only get a 403 error code.

00:41 403 means forbidden, and that means that indeed.com has changed their policy and how they protect themselves against web scraping. They don’t want you to go there, and you’ll have to respect that.

00:56 So instead, there is another option. In the associated written tutorial, I’ve set up a fake Python jobs board that you can use for scraping. This is a static website that is not going to change and that you are encouraged to scrape to learn about web scraping.

01:15 So instead of following along by using the same website that you will see me use in this course, you should focus your attention to the fake Python Jobs board that you can find at this URL.

01:32 Scraping this specific site is also described in more detail in the associated written tutorial. Now, what you’ll need to do when you continue watching this course is just translate what you see me do on indeed.com to the fake-jobs board.

01:49 And this is going to be a great training actually. You will see me work with Indeed, and you will write your code for the fake-jobs board.

01:58 Your focus for the rest of this course is going to be concepts. You want to understand the concepts of web scraping. You want to understand the processes that help you get through web scraping challenges, and you want to learn about the tools.

02:14 Everything that you will learn in the course is still valid. It even gives you a little edge that you will have to do this translation because this is what you encounter all the time when you work with Web Scraping. Every scraper is unique, as you know, and you will always have the issue that something changes and breaks.

02:32 So use it as a chance to synthesize the conceptual knowledge that you can get from this course and apply it practically to the static fake-jobs board that won’t change. And that’s all I wanted to tell you.

02:47 Now I have to say goodbye because the secret door from the future is closing.

Become a Member to join the conversation.