Investigating Quasar Data With Polars & Interactive marimo Notebooks (Overview)

00:00 Welcome to Investigating Quasar Data With Polars and Interactive marimo Notebooks. My name is Christopher, and I will be your guide.

00:09 This course is of the how-to variety, and shows you how to create a dashboard in marimo using astronomy data. Along the way, you’ll learn about quasars, spectral lines, and measuring redshift, and well, creating dashboards in Marmo.

00:24 Our dashboard will include a Matplotlib graph, data cleaned and manipulated with Polars, even more data fetched from the web using pandas, and interactive UI components that adjust the graph with switches and sliders in the notebook.

00:40 If you’re a space nerd, there is quasar stuff here, so that’s good. If you’re a data science nerd, then even if you don’t care about the space stuff, you’ll see how to collect data from a couple of different sources and how to get it into a form that can be used. And, well, if you’re a visualization nerd, then you’ll love how powerful marimo and Matplotlib are. From one nerd to another, whatever type you are, there’s something here for you.

01:06 Some of the libraries used to acquire space data can be a little finicky, so I’ve broken the course down into two sets of requirements. The primary focus is on the dashboard, and for that I tested with Python 3.13.5, marimo 0.16.0, Matplotlib 3.10.6, and Polars 1.33.1.

01:27 In the supporting materials drop-down below this video window, you’ll find some CSV files that contain the data used in the dashboard. If you don’t want to connect to the Astro Lab database and fetch your own, you can stick with the provided CSVs.

01:41 If you do want to follow along with the data retrieval side, I used pandas 2.3.2, lxml 6.0.2, and the Astro Data Lab client, sparclclient 1.2.6.

01:55 For your convenience, the supporting materials download also has a directory named requirements, which has two text files you can use to pip install these two sets of packages. Quasars are stellar objects that emit massive amounts of energy that occur when black holes eat things, and the process of pulling the star towards the black hole causes all sorts of funky physics to happen.

02:19 A celestial object’s redshift indicates how far away it is. You find the amount of redshift by comparing known spectral lines to an object’s emissions and calculating how much redshift is needed to match them up.

02:32 Don’t worry if that wasn’t clear. The next lesson covers what you need to know.

02:37 This course was built in conjunction with a colleague of mine who is an astronomy research assistant. One day he demoed a tool his team uses to visualize and interpret all this data, and what you’ll build here is a simplified version of that tool.

02:50 The end goal is an interactive marimo dashboard. Inside of that, you will graph quasar data, superimpose element spectral lines on top of it, then use a slider to change the amount of redshift applied to those lines in an attempt to match up the quasar’s emissions.

03:10 In case you’re not familiar, here’s a quick overview of the key tools that are used in this course.

03:15 marimo is a Python notebook. If you know Jupyter, marimo provides similar capabilities, but I prefer marimo over Jupyter as it stores the notebook data as a Python script, meaning you can use the same coding tools you’re familiar with to manage it.

03:31 Polars and pandas are both DataFrame libraries. These are in-memory representations of row and column data, similar to a spreadsheet. You can do all sorts of slicing and dicing on your data in both of these tools.

03:44 In this course, I’ll use Polars to reformat and clean some data and pandas to acquire some data from the web. Matplotlib is probably the most common of the Python graphing libraries out there.

03:55 It can be a little esoteric, but it’s very, very powerful. To top it all off, marimo speaks Matplotlib, so you can include graphs in your notebooks.

04:06 Next up, I’ll give you a bit of background on the science connected to the data you’ll be playing with.