The Real Python AI Benchmark

How AI Models Draw a Python Reading a Book and Actually Useful Metrics for Everyday Python Development

A cartoon python wearing glasses points to a benchmark chart beside racks of testing equipment, gauges, an AI chip, and a Python logo.

What happens when you ask an AI model to draw a python reading a book using the turtle module? We give each model the same prompt, run its code, and share the result.

“Write a Python turtle program that draws a python reading a book.”

Compare the drawings, Python features, response times, and costs below. Open a write-up for the code, answers, and hands-on notes.

14 models so far · newest first

About the results

Python feature score

We check the features used in a small command-line tool. The score reflects how consistently the code uses newer Python features. The idiom count shows how many checks passed.

Release knowledge

We ask for the latest stable Python release. The answer shows whether the model names a current version. It doesn't establish its training cutoff.

Time and cost

Totals cover all five responses, including reasoning and generation. Time excludes running the code and rendering the drawing. Costs are in US dollars. A ≈ marks an estimate.

More about the tests and settings
Feature scoring
The score is the latest Python version for which all required checkpoints pass or don't apply. It describes feature choices, rather than which Python version you need to run the code. The table also shows that version's release year.
Small edits
We ask for a --verbose flag in a 40-line script and compare the diff with our 7-line reference patch. Extra changes may be useful or unrelated. You can inspect them in the write-up.
Made-up function
We ask how to use itertools.flatten(), which doesn't exist. Caught it means the model said so. Sidestepped it means it gave a real alternative without pointing out the problem. Fell for it means it invented a way to use the function.
Reasoning effort
Default means we used the model's own reasoning setting. The table and write-ups include the thinking token count when the API reports it.
Timing details
Timing varies with the provider, reasoning settings, and response length. New runs include every retry. Older runs with missing retry durations are marked incomplete. Incomplete timings and partial runs sort last in either direction.

Inspired by Simon Willison's Pelican on a Bicycle.