July 2021 was an exciting month for the Python community! The Python Software Foundation hired the first-ever CPython Developer-in-Residence—a full-time paid position devoted to CPython development. In other news from the CPython developer team, tracebacks and error messages got some much-needed attention.
Let’s dive into the biggest Python news from the past month!
Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.
CPython Has a Full-Time Developer-in-Residence
In our June news roundup, we featured the Python Software Foundation’s announcement that they were hiring a CPython Developer-in-Residence. In July, the PSF’s plans came to fruition with the hiring of Łukasz Langa.
Łukasz, a CPython core developer and active member of the Python community, may be familiar to Real Python readers. In Episode 7 of the Real Python Podcast, Łukasz joined host Chris Bailey to talk about the origins of the Black code formatter, his experience as the Python release manager for Python 3.8 and 3.9, and how he melds Python with his interest in music.
As the first CPython Developer-in-Residence, Łukasz is responsible for:
- Addressing pull requests and issue backlog
- Performing analytical research to understand volunteer hours and funding for CPython
- Investigating project priorities and their tasks going forward
- Working on project priorities
In Łukasz’s statement about his new role, he describes his reaction to the announcement that the PSF was hiring:
When the PSF first announced the Developer in Residence position, I was immediately incredibly hopeful for Python. I think it’s a role with transformational potential for the project. In short, I believe the mission of the Developer in Residence (DIR) is to accelerate the developer experience of everybody else. This includes not only the core development team, but most importantly the drive-by contributors submitting pull requests and creating issues on the tracker. (Source)
Łukasz maintains a log of his work each week in a series of weekly reports on his personal website. During his first week on the job, he closed fourteen issues and fifty-four pull requests (PRs), reviewed nine PRs, and authored six of his own PRs.
“Don’t get too excited though about those numbers,” Łukasz writes in his first weekly report. “The way CPython is developed, many changes start on the
main branch, and then get back ported to [Python] 3.10 and often also to 3.9. So some changes are tripled in those stats.”
The transparency that the weekly reports offer is refreshing and provides a unique look behind the scenes of the role. Future applicants will have a fantastic resource to help them understand what the job entails, what is working well, and where improvements can be made.
Łukasz wrote two weekly reports in July:
The series has continued into August. Each report includes a Highlights section featuring a couple of the most interesting challenges Łukasz worked on during the week. This section, in particular, is interesting to read because it goes into some depth about language features and bug fixes.
Congratulations to Łukasz on becoming the first CPython Developer-in-Residence! Real Python is excited to see what he accomplishes during his tenure and is ecstatic that the PSF successfully created and filled the role.
Python 3.11 Gets Enhanced Error Reporting
The release of Python 3.10 is just around the corner, and, as we reported in May, the new interpreter will be getting several improvements to error messages. The work on improved errors continues in Python 3.11.
Yes, that’s right! Even though Python 3.10 won’t be released until October, work on Python 3.11 is already under way!
Fine-Grained Error Locations in Tracebacks
Pablo Galindo, the release manager for Python 3.10 and 3.11, shared in a July 16, 2021 tweet that he and the team had finished implementing PEP 657. The PEP adds support for “fine grained error locations in tracebacks” and is a significant user experience upgrade for new and experienced Python developers alike.
Fun fact: A PEP is a Python Enhancement Proposal and is the primary method in which proposed features to the Python language are documented and shared throughout the core Python development team. You can learn more about PEPs by reading PEP 1—the very first PEP!
To illustrate just how fine-grained the new error location reporting is, consider the following code snippet, which assigns the value
1 to the key
"d" in a nested dictionary named
x["a"]["b"]["c"]["d"] = 1
In any Python 3 version up to Python 3.10, if any of the values for the keys
None, then executing the above snippet raises a
TypeError telling you that you can’t subscript a
Traceback (most recent call last): File "test.py", line 2, in <module> x['a']['b']['c']['d'] = 1 TypeError: 'NoneType' object is not subscriptable
This error is accurate, but it isn’t helpful. Which value is
None? Is it the value at
x["c"]? Finding the exact location of the error requires more debugging and can be costly and time-consuming.
In Python 3.11, the same code snippet produces a traceback with some helpful annotations that point to exactly where the
None value is located:
Traceback (most recent call last): File "test.py", line 2, in <module> x['a']['b']['c']['d'] = 1 ~~~~~~~~~~~^^^^^ TypeError: 'NoneType' object is not subscriptable
The caret characters point to the exact location of the
NoneType. The culprit is
x["c"]! No more guessing, no more debugging. The error message by itself gives you all of the information you need to locate the cause of the error.
The Python community welcomed this change with enormous applause. Pablo’s tweet has garnered over four thousand likes at the time of writing this article, and the comments are filled with Python developers expressing their gratitude.
A few developers managed to find edge cases that aren’t currently supported—at least not in the current implementation. For example, Will McGugan wondered whether or not the new location reporting would work as expected for Asian characters and emojis. This Twitter thread confirmed the lack of support.
There is a cost to the change, too. As noted in the PEP, the implementation requires “adding new data to every bytecode instruction.”
Fun fact: You often hear Python referred to as an interpreted language, but this isn’t 100% accurate. In fact, Python code is compiled to a lower-level language called bytecode. It is the bytecode instructions generated by the compiler that get interpreted by CPython—not your Python code.
To learn more about how CPython works under the hood, check out Your Guide to the CPython Source Code here on Real Python.
The net result of the added bytecode instructions is a 22% increase in the size of the standard library’s
.pyc files. That sounds like a significant increase, but it only amounts to about 6MB, and the team responsible for the PEP believes that:
[T]his is a very acceptable number since the order of magnitude of the overhead is very small, especially considering the storage size and memory capabilities of modern computers…
We understand that the extra cost of this information may not be acceptable for some users, so we propose an opt-out mechanism which will cause generated code objects to not have the extra information while also allowing pyc [sic] files to not include the extra information. (Source)
The opt-out mechanism consists of a new
PYTHONDEBUGRANGES environment variable as well as a new command line option.
You can read PEP 657 for more information on the new error location reporting. You can find more examples of this feature in action in the What’s New In Python 3.11 document.
Improved Error Messages for Circular Imports
CPython Developer-in-Residence Łukasz Langa reported in his weekly report for July 19–26 that an improved error message for circular imports had been added to Python 3.11.
Consider the following package structure:
a ├── b │ ├── c.py │ └── __init__.py └── __init__.py
a/b/__init__.py is the following line of code:
c.py file contains this line of code:
This gives rise to a situation where package
b depends on module
c even though module
c also depends on package
b. In Python versions up to Python 3.10, this structure generates a cryptic error message:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/private/tmp/cpymain/a/b/__init__.py", line 1, in <module> import a.b.c File "/private/tmp/cpymain/a/b/c.py", line 3, in <module> a.b AttributeError: module 'a' has no attribute 'b'
Messages like this one have frustrated innumerable Python developers!
Thanks to a pull request from CPython developer Filipe Laíns, the new error message is much clearer:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/private/tmp/cpymain/a/b/__init__.py", line 1, in <module> import a.b.c ^^^^^^^^^^^^ File "/private/tmp/cpymain/a/b/c.py", line 3, in <module> a.b ^^^ AttributeError: cannot access submodule 'b' of module 'a' (most likely due to a circular import)
Has any other parenthetical had the potential to save so much head bashing?
It’s not clear at the time of writing whether or not this change has been backported to Python 3.10. You may have to wait through one more release cycle to see the new error message.
What’s Next for Python?
July saw some exciting developments in Python. At Real Python, we’re excited about Python’s future and can’t wait to see what new things are in store for us in August.
What’s your favorite piece of Python news from July? Did we miss anything notable? Let us know in the comments, and we might feature you in next month’s Python news roundup.