Real Python Podcast Episode #224 Title Artwork

Episode 224: Narwhals: Expanding DataFrame Compatibility Between Libraries

The Real Python Podcast

Oct 18, 2024 1h

Marco Gorelli
Guest

How does a Python tool support all types of DataFrames and their various features? Could a lightweight library be used to add compatibility for newer formats like Polars or PyArrow? This week on the show, we speak with Marco Gorelli about his project, Narwhals.

Episode Sponsor:

Narwhals is a project aimed at library maintainers rather than end users. We discuss how the added compatibility benefits users by supporting modern features like lazy evaluation. We cover several projects Marco has been working with to implement Narwhals, including Altair, scikit-lego, and Ibis.

We also discuss how Marco started contributing to open-source projects. Marco has contributed to both pandas and Polars, which helps explain his interest in growing compatibility between libraries. He also offers advice on making your first contribution.

This episode is sponsored by CodeRabbit.

Course Spotlight: Differences Between Python’s Mutable and Immutable Types

In this video course, you’ll learn how Python’s mutable and immutable data types work internally and how you can take advantage of mutability or immutability to power your code.

Topics:

00:00:00 – Introduction
00:02:02 – Euro SciPy 2024 and sprints
00:04:04 – How did you get involved in open source?
00:07:18 – Finding a good issue to get started
00:09:25 – Discord and open-source projects
00:11:12 – Who would you describe Narwhals?
00:16:47 – Working on Polars
00:19:17 – Apache Arrow and a data interchange protocol
00:22:55 – Sponsor: CodeRabbit
00:23:55 – Digging into eager vs lazy
00:27:04 – Ibis DataFrame library
00:28:57 – What do libraries need from Narwhals?
00:34:57 – The scikit-lego library
00:37:15 – Video Course Spotlight
00:38:45 – Other libraries interested in Narwhals
00:41:56 – Compatibility policy
00:45:18 – What should an end user expect?
00:46:32 – Have other projects that attempted this?
00:47:54 – Keeping the project light and pure Python
00:49:32 – Contributors and how to get involved
00:54:42 – What are you excited about in the world of Python?
00:57:18 – What do you want to learn next?
00:59:05 – How can people follow your work online?
00:59:27 – Thanks and goodbye

Show Links:

Level Up Your Python Skills With These Courses:

The Pandas DataFrame: Make Working With Data Delightful

The pandas DataFrame: Working With Data Efficiently

Pandas GroupBy: Your Guide to Grouping Data in Python

pandas GroupBy: Grouping Real World Data in Python

Python 3.13: Cool New Features for You to Try

What's New in Python 3.13

← Previous All Episodes