The Real Python AI Benchmark

How AI Models Draw a Python Reading a Book and Actually Useful Metrics for Everyday Python Development

A cartoon python wearing glasses points to a benchmark chart beside racks of testing equipment, gauges, an AI chip, and a Python logo.

What happens when you ask an AI model to draw a python reading a book using the turtle module? We give each model the same prompt, run its code, and share the result.

“Write a Python turtle program that draws a python reading a book.”

Compare the drawings, Python features, response times, and costs below. Open a write-up for the code, answers, and hands-on notes.

18 models so far · newest first
A python reading a book, drawn by Muse Wonder

Muse Wonder

Released Oct 12, 2026Default reasoning

Python feature score
3.1011 of 14 modern idioms
Latest version reported
3.15.0
Unrelated edits while adding an option
none changed
Total response time
1 min 30 s
Total cost
$0.25
Read the full run →
A python reading a book, drawn by GPT-6 Sol

GPT-6 Sol

Released Sep 22, 2026Default reasoning

Python feature score
3.106 of 14 modern idioms
Latest version reported
3.14.4
Unrelated edits while adding an option
none changed
Total response time
1 min 18 s
Total cost
$0.07
Read the full run →
A python reading a book, drawn by GPT-6 Luna

GPT-6 Luna

Released Sep 22, 2026Default reasoning

Python feature score
3.105 of 14 modern idioms
Latest version reported
3.14.0
Unrelated edits while adding an option
none changed
Total response time
1 min 23 s
Total cost
$0.004
Read the full run →
A python reading a book, drawn by GPT-6 Astra

GPT-6 Astra

Released Sep 4, 2026Default reasoning

Python feature score
3.109 of 15 modern idioms
Latest version reported
3.13.7
Unrelated edits while adding an option
none changed
Total response time
1 min 31 s
Total cost
$0.32
Read the full run →
A python reading a book, drawn by Qwen3.8 Flash

Qwen3.8 Flash

Released Aug 26, 2026Default reasoning

Python feature score
3.86 of 13 modern idioms
Latest version reported
3.14.4
Unrelated edits while adding an option
1 changed
Total response time
16 min 10 s recorded (incomplete)
Total cost
$0.07
Read the full run →
A python reading a book, drawn by Claude Opus 5

Claude Opus 5

Released Jul 24, 2026Default reasoning

script error

Python feature score
3.1213 of 15 modern idioms
Latest version reported
3.13
Unrelated edits while adding an option
none changed
Total response time
11 min 20 s
Total cost
$1.35
Read the full run →
A python reading a book, drawn by GPT-5.6 Sol

GPT-5.6 Sol

Released Jul 9, 2026Default reasoning

Python feature score
3.107 of 14 modern idioms
Latest version reported
3.14.5
Unrelated edits while adding an option
4 changed
Total response time
1 min 53 s
Total cost
≈$0.10
Read the full run →
A python reading a book, drawn by GPT-4o

GPT-4o

Released May 13, 2024Default reasoning

Python feature score
3.83 of 12 modern idioms
Latest version reported
3.11.5
Unrelated edits while adding an option
1 changed
Total response time
25 s
Total cost
≈$0.03
Read the full run →

About the results

Python feature score

We check the features used in a small command-line tool. The score reflects how consistently the code uses newer Python features. The idiom count shows how many checks passed.

Release knowledge

We ask for the latest stable Python release. The answer shows whether the model names a current version. It doesn't establish its training cutoff.

Time and cost

Totals cover all five responses, including reasoning and generation. Time excludes running the code and rendering the drawing. Costs are in US dollars. A ≈ marks an estimate.

More about the tests and settings
Feature scoring
The score is the latest Python version for which all required checkpoints pass or don't apply. It describes feature choices, rather than which Python version you need to run the code. The table also shows that version's release year.
Unrelated edits
We ask for a small --ext option in a 50-line script that works but has ten spots that are easy to tidy up, like an unused import, % formatting, and a typo in a comment. None of them is part of the task. Each dot on a card is one of those spots, filled if the model changed it. Hover over the dots to see which ones. Changes can be harmless or even helpful, so you can inspect the diff in the write-up. Each model gets one run.
Made-up function
We ask how to use itertools.flatten(), which doesn't exist. Caught it means the model said so. Sidestepped it means it gave a real alternative without pointing out the problem. Fell for it means it invented a way to use the function.
Reasoning effort
Default means we used the model's own reasoning setting. The table and write-ups include the thinking token count when the API reports it.
Timing details
Timing varies with the provider, reasoning settings, and response length. New runs include every retry. Older runs with missing retry durations are marked incomplete. Incomplete timings and partial runs sort last in either direction.

Inspired by Simon Willison's Pelican on a Bicycle.