Python Web Scraping
Learning Path ⋅ Skills: Web Scraping, HTTP Requests, Data Parsing, Browser Automation
Web scraping with Python lets you extract data from websites and turn unstructured web pages into usable datasets. This learning path teaches you the tools and techniques to build reliable web scrapers.
By completing this path, you’ll be able to:
- Make HTTP requests using urllib and the Requests library
- Parse HTML and extract data with Beautiful Soup
- Build scalable scrapers with Scrapy and store results in MongoDB
- Work with JSON and CSV data from web sources
- Automate browser interactions with Selenium
This path is for Python developers who need to collect data from websites programmatically. You should already be comfortable writing Python scripts.
You’ll start with HTTP fundamentals and HTML structure, then move into dedicated scraping libraries, data handling, and browser automation.
Python Web Scraping
Learning Path ⋅ 11 Resources
Laying the Foundation for Web Scraping
Before you jump into web scraping, it’s important to brush up on some foundational skills, like making HTTP requests and understanding HTML, the markup language behind every web page, along with CSS.
Course
HTTP Requests With Python's urllib.request
In this video course, you'll explore how to make HTTP requests using Python's handy built-in module, urllib.request. You'll try out examples and go over common errors, all while learning more about HTTP requests and Python in general.
Course
Making HTTP Requests With Python
The requests library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application. This course shows you how to work effectively with requests, from start to finish.
Interactive Quiz
Python's Requests Library
Course
HTML and CSS Foundations for Python Developers
There's no way around HTML and CSS when you want to build web apps. Even if you're not aiming to become a web developer, knowing the basics of HTML and CSS will help you understand the Web better. In this video course, you'll get an introduction to HTML and CSS for Python programmers.
Getting Started With Web Scraping
With HTTP requests and page structure behind you, you’ll now pull data out of live pages. You’ll start by parsing HTML with Beautiful Soup, then work through a full scraping project with a set of hands-on exercises to practice on, and finish by scaling up with Scrapy and storing your results in MongoDB. Along the way, a podcast episode covers the tools of the trade and the legal boundaries of scraping.
Course
Web Scraping With Beautiful Soup and Python
In this course, you'll walk through the main steps of the web scraping process. You'll learn how to write a script that uses Python's requests library to scrape data from a website. You'll also use Beautiful Soup to extract the specific pieces of information that you're interested in.
Interactive Quiz
Beautiful Soup: Build a Web Scraper With Python
Course
Introduction to Web Scraping With Python
Learn all about web scraping in Python. You'll see how to parse data from websites and interact with HTML forms using tools such as Beautiful Soup and MechanicalSoup.
Interactive Quiz
A Practical Introduction to Web Scraping in Python
Course
Exercises Course: Introduction to Web Scraping With Python
In this course, you'll practice the main steps of the web scraping process. You'll write a script that uses Python's requests library to scrape and parse data from a website. You'll also interact with HTML forms using tools like Beautiful Soup and Mechanical Soup to extract specific information.
Tutorial
Web Scraping With Scrapy and MongoDB
This tutorial covers how to write a Python web crawler using Scrapy to scrape and parse data, and then store the data in MongoDB.
Interactive Quiz
Web Scraping With Scrapy and MongoDB
Handling Response Data
Course
Working With JSON in Python
Learn how to work with Python's built-in json module to serialize the data in your programs into JSON format. Then, you'll deserialize some JSON from an online API and convert it into Python objects.
Interactive Quiz
Working With JSON Data in Python
Course
Reading and Writing CSV Files
This short course covers how to read and write data to CSV files using Python's built in csv module and the pandas library. You'll learn how to handle standard and non-standard data such as CSV files without headers, or files containing delimeters in the data.
Interactive Quiz
Reading and Writing CSV Files in Python
Automating Your Web Scraping Process
You can now request pages, parse them, and save what you find. Many sites, though, build their content with JavaScript after the page loads, so there is nothing useful in the HTML you get back. In this section, you’ll use Selenium to drive a headless browser, interact with those pages, and scrape what they actually render.
Tutorial
Modern Web Automation With Python and Selenium
Learn advanced Python web automation techniques with Selenium, such as headless browsing, interacting with web elements, and implementing the Page Object Model pattern.
Interactive Quiz
Web Automation With Python and Selenium
Test Your Knowledge
You’ve made it through the entire path! In the wrap-up quiz below, you’ll revisit the most important ideas about making HTTP requests, parsing HTML with Beautiful Soup, extracting data with Scrapy, working with JSON and CSV, and automating browsers with Selenium. Use it to spot any topics that still feel rusty before moving on.
Interactive Quiz
Python Web Scraping
Revisit Requests, Beautiful Soup, Scrapy, and Selenium in this wrap-up quiz covering the Python Web Scraping learning path.
Congratulations on completing this learning path! You’ve learned how to make HTTP requests, scrape web pages with Beautiful Soup and Scrapy, handle JSON and CSV data, and automate browsers with Selenium.
Scraped data is only useful once you can analyze it, so a good next step is the data science core skills path:
Learning Path
Data Science With Python Core Skills
22 Resources ⋅ Skills: Pandas, NumPy, Data Cleaning, Data Visualization, Statistics, CSV, JSON, Jupyter Notebooks
You might also be interested in these related learning paths:
Got feedback on this learning path?
Looking for real-time conversation? Visit the Real Python Community Chat or join the next “Office Hours” Live Q&A Session. Happy Pythoning!
