Introduction to Web Scraping
00:00 You’ll start this part off by learning about what is web scraping, then we’re going to learn about why would you even want to scrape the web in the first place, talk about a couple of challenges of web scraping—specifically, variety of websites and how long your code can last—and then finally, talk about APIs, which represent a often useful alternative to web scraping.
00:34 So, just pulling information from the web, whether it’s you doing it manually going to your favorite song lyrics page and just copy-pasting things from there into a local TXT file or whatever—all of this would be web scraping, but generally, when someone talks about web scraping they mean the automated gathering of information from the web.
00:54 So that’s what web scraping is about: writing some code that fetches information from the internet. Now, why would you want to scrape the web? You can think about maybe the song lyrics example that I mentioned before.
01:07 Maybe you want all of the song lyrics from a specific album but you don’t want to keep clicking around and copy-pasting, so a way to do that would be to automate it and just pull all the information with a script. In this course, we’re going to talk about your job search, which is—there’s a bunch of these job search aggregator tools out there that help you to look for a position, but again, it involves a lot of looking at this little card, the information of the job, clicking it, reading over it, finding does this even interest you, et cetera, et cetera. And there’s some ways that you can do to automate this job search.
01:42 Now, as a disclaimer, you’re not going to be able to totally automate this job search with the information that you’re going to learn in this course, even though I’m sure there’s some people who already did things like that as well, but we’re going to use this as an example to learn about web scraping on a more general basis so that you can apply it for whatever task you’re interested in for any gathering of information from the internet. Now, specifically in this course project, we’re going to talk about the web scraping process and tools that you can use for web scraping, and this is the main focus.
02:14 I want to introduce you to the different processes and the tools that are important for this, and we’re going to do it by automating the process of gathering some information from a job board. Specifically, it’s going to be indeed.com.
02:27 And then in the end, I’m going to show you some pointers how you can customize this code for your personal job search and maybe build it out in a way so that it’s actually a useful tool for you to maybe make your job search a little bit easier. Okay!
Become a Member to join the conversation.