Working With LazyFrames

It’s a bit unsettling that the 1776 senator query results don’t match mine or the supplied CSV.

shape: (5, 4)
┌───────┬───────┬────────────┬────────────┐
│ state ┆ count ┆ min_birth  ┆ max_birth  │
│ ---   ┆ ---   ┆ ---        ┆ ---        │
│ str   ┆ u32   ┆ date       ┆ date       │
╞═══════╪═══════╪════════════╪════════════╡
│ MA    ┆ 1     ┆ 1776-12-01 ┆ 1776-12-01 │
│ OH    ┆ 2     ┆ 1776-01-03 ┆ 1776-07-04 │
│ CT    ┆ 1     ┆ 1776-09-15 ┆ 1776-09-15 │
│ KY    ┆ 3     ┆ 1776-04-06 ┆ 1776-12-08 │
│ NC    ┆ 1     ┆ 1776-10-31 ┆ 1776-10-31 │
└───────┴───────┴────────────┴────────────┘

@toigopaul I’m not sure where your data came from or whether you followed the lesson closely, but I’m getting exactly the same results on my end. The rows appear in different order, but the data itself matches what’s presented.

@Bartosz Zaczyński I got the data from Supporting Material->Sample Code (.zip)

@toigopau Can you share your code? I’m pretty sure this must have been a pesky human error.

import polars as pl

gov = pl.scan_csv("legislators.csv", try_parse_dates=True)
query = (
    gov
    .select("last_name", "type", "state", "birthday")
    .filter(pl.col("birthday").dt.year() == 1776)
    .filter(pl.col("type") == "sen")
    .group_by("state").agg(
        pl.len().alias("count"),
        pl.min("birthday").alias("min_birth"),
        pl.max("birthday").alias('max_birth')
    )
)
print(query.explain())
result = query.collect()
print(result)

With all due respect, it’s the data not the code as can be readily seen with an old-fashioned Excel filter. I’d like to add a sanity check by mentioning that California was added to the union in 1850. This means that whatever senator was born in 1776 would have to have been at least 74 when they first took office. Not impossible, but certainly improbable.

last_name   first_name  birthday    type    state
Bledsoe Jesse   1776-04-06  sen KY
Locke   Francis 1776-10-31  sen NC
Logan   William 1776-12-08  sen KY
Mills   Elijah  1776-12-01  sen MA
Brown   Ethan   1776-07-04  sen OH
Willey  Calvin  1776-09-15  sen CT
Bibb    George  1776-10-30  sen KY
Morris  Thomas  1776-01-03  sen OH

@toigopaul Thank you for sharing your code. The reason why you’re getting a different result is in this line:

-.select("last_name", "type", pl.col("state").sort(), "birthday")
+.select("last_name", "type", "state", "birthday")

In short, you didn’t call .sort() on the "state" column like shown in the video lesson. However, sorting inside select() reorders only the "state" column, breaking row alignment with the other columns. This looks to me like a mistake, so I’ll pass your comment along to the course author. Thanks again for the helpful feedback!

Wow! Thanks for that priceless insight! There’s still the mystery, to me, of how you don’t have an issue with the varying formats of the birthday column that I commented on in the previous lesson, “Grouping and Aggregation”.

Hey @toigopaul,

Yep, @Bartosz is correct, the sort is mucking it up. Sorry about that. To sort this kind of data, the sort should have been outside of the select, which the group_by would then ruin anyway, so it shouldn’t have been there. We’ll get to work on a correction.

As for your earlier problem with the data file, I have a guess. When I opened the CSV in Excel I noticed that some values are left aligned and some right. That is usually a hint that Excel is treating them differently. When I went to apply a filter, the dates with years after 1900+ show up as little select-able trees, allowing you to select the whole year or months and days inside of them. The dates before that are showing up as full values in the selector, which makes me think Excel is treating them as text.

If I’m right, and you happened to have hit “save”, Excel would have written the things it sees as dates out in its default format for your region, whereas the “string dates” would have been left as is.

Now, why 1900 is the date boundary for Excel would be its own mystery. 1970 is a common boundary in computing, 1880 is another one, but 1900 is new to me.

Hi @toigopaul – I’ve changed this lesson with the update that Chris Trudeau made, removing the sort call. Hopefully now others won’t suffer the same frustration.

@Christopher Trudeau because Excel considers January 1, 1900 Day 1. I don’t know what January 0, 1900 is other than funny. Otherwise, I apologize for muddying the waters and appreciate the help and insight from the RP team. I’m all for letting the record show my misunderstandings and generally oppose censoring anything. That said, given the slight correction in the course I feel my comments and these responses could confuse future consumers of the course. Therefore, if someone makes a motion to delete all germane comments on this tutorial, I second.

-1  ########
0   1/0/1900
1   1/1/1900
2   1/2/1900
3   1/3/1900
4   1/4/1900
5   1/5/1900
6   1/6/1900
7   1/7/1900
8   1/8/1900
9   1/9/1900
10  1/10/1900