Python News: What's New From July 2021

Fine-Grained Error Locations in Tracebacks

Pablo Galindo, the release manager for Python 3.10 and 3.11, shared in a July 16, 2021 tweet that he and the team had finished implementing PEP 657. The PEP adds support for “fine grained error locations in tracebacks” and is a significant user experience upgrade for new and experienced Python developers alike.

To illustrate just how fine-grained the new error location reporting is, consider the following code snippet, which assigns the value 1 to the key "d" in a nested dictionary named x:

x["a"]["b"]["c"]["d"] = 1

In any Python 3 version up to Python 3.10, if any of the values for the keys "a", "b", or "c" are None, then executing the above snippet raises a TypeError telling you that you can’t subscript a NoneType:

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    x['a']['b']['c']['d'] = 1
TypeError: 'NoneType' object is not subscriptable

This error is accurate, but it isn’t helpful. Which value is None? Is it the value at x["a"], x["b"], or x["c"]? Finding the exact location of the error requires more debugging and can be costly and time-consuming.

In Python 3.11, the same code snippet produces a traceback with some helpful annotations that point to exactly where the None value is located:

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    x['a']['b']['c']['d'] = 1
    ~~~~~~~~~~~^^^^^
TypeError: 'NoneType' object is not subscriptable

The caret characters point to the exact location of the NoneType. The culprit is x["c"]! No more guessing, no more debugging. The error message by itself gives you all of the information you need to locate the cause of the error.

The Python community welcomed this change with enormous applause. Pablo’s tweet has garnered over four thousand likes at the time of writing this article, and the comments are filled with Python developers expressing their gratitude.

A few developers managed to find edge cases that aren’t currently supported—at least not in the current implementation. For example, Will McGugan wondered whether or not the new location reporting would work as expected for Asian characters and emojis. This Twitter thread confirmed the lack of support.

There is a cost to the change, too. As noted in the PEP, the implementation requires “adding new data to every bytecode instruction.”

The net result of the added bytecode instructions is a 22% increase in the size of the standard library’s .pyc files. That sounds like a significant increase, but it only amounts to about 6MB, and the team responsible for the PEP believes that:

[T]his is a very acceptable number since the order of magnitude of the overhead is very small, especially considering the storage size and memory capabilities of modern computers…

We understand that the extra cost of this information may not be acceptable for some users, so we propose an opt-out mechanism which will cause generated code objects to not have the extra information while also allowing pyc [sic] files to not include the extra information. (Source)

The opt-out mechanism consists of a new PYTHONDEBUGRANGES environment variable as well as a new command line option.

You can read PEP 657 for more information on the new error location reporting. You can find more examples of this feature in action in the What’s New In Python 3.11 document.

Improved Error Messages for Circular Imports

CPython Developer-in-Residence Łukasz Langa reported in his weekly report for July 19–26 that an improved error message for circular imports had been added to Python 3.11.

Consider the following package structure:

a
├── b
│   ├── c.py
│   └── __init__.py
└── __init__.py

Inside a/b/__init__.py is the following line of code:

import a.b.c

The c.py file contains this line of code:

import a.b

This gives rise to a situation where package b depends on module c even though module c also depends on package b. In Python versions up to Python 3.10, this structure generates a cryptic error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/tmp/cpymain/a/b/__init__.py", line 1, in <module>
    import a.b.c
  File "/private/tmp/cpymain/a/b/c.py", line 3, in <module>
    a.b
AttributeError: module 'a' has no attribute 'b'

Messages like this one have frustrated innumerable Python developers!

Thanks to a pull request from CPython developer Filipe Laíns, the new error message is much clearer:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/tmp/cpymain/a/b/__init__.py", line 1, in <module>
    import a.b.c
    ^^^^^^^^^^^^
  File "/private/tmp/cpymain/a/b/c.py", line 3, in <module>
    a.b
    ^^^
AttributeError: cannot access submodule 'b' of module 'a'
(most likely due to a circular import)

Has any other parenthetical had the potential to save so much head bashing?

It’s not clear at the time of writing whether or not this change has been backported to Python 3.10. You may have to wait through one more release cycle to see the new error message.