Episode 194: Automate Tasks With Python & Building a Small Search Engine

The Real Python Podcast

Mar 01, 2024 44m

What are the typical computer tasks you do manually every week? Could you automate those tasks with a Python script? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder’s Weekly articles and projects.

We discuss a recent Hacker News thread about frequently used automation scripts. We share the kinds of tasks we’ve automated with Python in our work and personal lives.

Christopher shares a tutorial about building a micro-search engine from scratch using Python. The post takes you through coding the components of a crawler, index, and ranker. The finished engine is designed to search the posts of the blogs you follow.

We also share several other articles and projects from the Python community, including a news roundup, how a Polars query works under the hood, using Python for data analysis, understanding open-source licensing, summarizing the significant changes between Python versions, a robust TUI hex editor, and a lightweight dataframe library with a universal interface for data wrangling.

This week’s episode is brought to you by Intel.

Topics:

• 00:00:00 – Introduction
• 00:02:23 – uv: Python Packaging in Rust
• 00:02:43 – Rye Grows With uv
• 00:03:20 – Python 3.13.0 Alpha 4 Is Now Available
• 00:03:45 – A Bird’s Eye View of Polars
• 00:07:28 – Polars: Why We Have Rewritten the String Data Type
• 00:09:33 – A Search Engine in 80 Lines of Python
• 00:13:14 – Using Python for Data Analysis
• 00:18:22 – Sponsor: Intel
• 00:18:53 – Understanding Open Source Licensing
• 00:21:54 – Summary of Major Changes Between Python Versions
• 00:23:19 – What Python automation scripts do you reuse frequently at work?
• 00:34:21 – Video Course Spotlight
• 00:35:52 – hexabyte: A modern, modular, and robust TUI hex editor
• 00:39:56 – ibis: The Flexibility of Python With the Scale of Modern SQL
• 00:43:31 – Thanks and goodbye

News:

• uv: Python Packaging in Rust – uv is an extremely fast Python package installer and resolver, designed as a drop-in alternative to pip and pip-tools. This post introduces you to uv and shows some of its performance numbers.
• Rye Grows With uv - Armin Ronacher’s Thoughts and Writings
• Python 3.13.0 Alpha 4 Is Now Available

Show Links:

• A Bird’s Eye View of Polars – This post on the Polars blog introduces you to how Polars works, showing the steps from queries, plans, optimizations, and then the final execution.
• Polars: Why We Have Rewritten the String Data Type – A large refactor on the string data type is underway in Polars. This deep dive explains why and what is changing.
• A Search Engine in 80 Lines of Python – In this post Alex explains how he built a micro-search engine from scratch using Python. The resulting search engine is used to search in the posts of the blogs he follows.
• Using Python for Data Analysis – In this tutorial, you’ll learn the importance of having a structured data analysis workflow, and you’ll get the opportunity to practice using Python for data analysis while following a common workflow process.
• Understanding Open Source Licensing – This article discusses the importance of open-source licensing in software development and its implications for stakeholders.
• Summary of Major Changes Between Python Versions – This article is a quick reference covering the major changes introduced with each new version of Python. Can’t remember when the walrus operator was introduced? This is the place to look that up.

Discussion:

• What Python automation scripts do you reuse frequently at work? - Hacker News

Projects

• hexabyte: A modern, modular, and robust TUI hex editor
• ibis: The Flexibility of Python With the Scale of Modern SQL

Additional Links:

• Episode #193: Wes McKinney on Improving the Data Stack & Composable Systems – The Real Python Podcast
• Open Source Licensing: Software Freedom and Intellectual Property Law
• What’s in which Python - Ned Batchelder
• Automate the Boring Stuff with Python - Al Sweigart
• Working With Files in Python – Real Python
• Build Command-Line Interfaces With Python’s argparse – Real Python