How AI Models Draw a Python Reading a Book

How to read the results

Released

The day the model was released. Results are sorted newest release first until you pick another column. In the table view, click a column heading to sort by it, and click again to flip the direction.

Writes Python like it’s

We ask each model to write a small command-line tool from a fixed spec, then look at which idioms it uses: list[str] or typing.List, tomllib or a hand-rolled parser, Path.walk() or os.walk(), and so on. Each idiom arrived in a specific Python version, and each version has a year. One old-fashioned idiom pins the model at the version before that idiom existed, so this reading is deliberately strict.

Modern idioms

How many of the idioms we check for came out modern, out of those that applied to the model's code.

Newest Python it knows

We ask the model which Python release is the newest. Its answer shows roughly where its training data ends.

Lines touched for a tiny edit

We hand the model a 40-line script and ask it to add a --verbose flag, then count how many lines it changed. The minimal answer changes 7. Much more than that means it rewrote things nobody asked about.

Spots a made-up function?

We ask how to use itertools.flatten(), which doesn't exist. Caught it means the model said so. Sidestepped it means it quietly showed a real alternative without mentioning that the function is made up. Fell for it means it invented an answer.

Reasoning effort

The thinking setting we ran with. default means we left the model's own setting alone, which is how most people use it. The count is how many thinking tokens it spent across all the tasks, when the API reports that.

What the run cost

What the API charged for all the tasks on this card, in US dollars. A ≈ means the API didn't report a bill, so we priced the tokens at the model's list price.