Python Monthly News

Python News: What's New From April 2021?

by David Amos May 03, 2021 community

If you hang around Python developers long enough, you’ll eventually hear someone talk about how awesome the Python community is. If you want to get up to speed on what happened in the Python community in April 2021, then you’ve come to the right place to get your news!

From better error messages that improve user experience to community-driven efforts to delay a change to CPython, April 2021 was a month full of stories that reminded us that Python is better because of its community.

Let’s dive into the biggest Python news from the past month!

The PSF Is Hiring Thanks to Visionary Sponsors

In February 2021, Google became the first Visionary Sponsor of the Python Software Foundation (PSF). Shortly thereafter, Bloomberg Engineering also stepped up as a Visionary Sponsor.

Visionary Sponsors are the highest tier of sponsorship and provide significant funds to support PSF initiatives. The sponsorship from Google and Bloomberg is helping the PSF hire two new full-time employees.

CPython Developer-in-Residence

Thanks to Google’s sponsorship funds, the PSF has announced its plan to hire a CPython developer-in-residence. According to the PSF’s announcement, the developer-in-residence will “address backlog, perform analytical research to understand the project’s volunteer hours and funding, investigate project priorities and their tasks going forward, and begin working on those priorities.”

The full-time position is funded for one year, and résumé submissions are being accepted until May 16, 2021. However, the position appears to be open only to existing core developers.

Hiring a full-time employee to support CPython development is an enormous step forward for the PSF and for the Python community. The decision was inspired by the Django Fellowship Program, which hires paid contractors to handle administrative and community management tasks.

Python Packaging Project Manager

Bloomberg’s donations are helping to fund a Python Packaging project manager position. According to the PSF’s announcement, the project manager will “oversee improvements and added functionality that will benefit all Python users while leading the development of PyPI into a sustainable service.”

Applications are open until May 18, 2021. For more information about the position, check out the Python Jobs Board.

You can read more about Bloomberg’s decision to support Python and why they’re specifically interested in the Python Packaging ecosystem in their blog post Supporting the Python community by “Shifting Left.”

Python 3.10 Will Have Improved Error Messages

On April 9, 2021, Python core developer Pablo Galindo, who is also the release manager for Python 3.10 and 3.11, tweeted a question directed toward Python educators:

Python educators and users: I have been working on improving SyntaxError messages in CPython lately. What error (only SyntaxErrors for now 😅) messages you or your students have struggle with? Which ones you think we should improve? 🤔 (Pls RT to help reaching more people 🙏). (Source)

The tweet got a lot of engagement, including requests for improved error messages about the assignment (=) and comparison (==) operators, indentation errors, and missing colons. In some cases, Galindo pointed out that he had already improved the errors people mentioned!

For instance, in a pull request titled bpo-42997: Improve error message for missing : before suites, Galindo improved the error messages for missing colons. In Python 3.10, if you forget to type a colon (:) after defining a function, you’ll see this new and improved message:

>>>
>>> # Python 3.10a7
>>> def f()
  File "<stdin>", line 1
    def f()
          ^
SyntaxError: expected ':'

Contrast that to the error message in Python 3.9:

>>>
>>> # Python 3.9.4
>>> def f()
  File "<stdin>", line 1
    def f()
           ^
SyntaxError: invalid syntax

It’s a small change, but pointing out that the interpreter expected a colon rather than just telling the user that their syntax is invalid is much more helpful and not just for beginners. Developers with experience in other languages but are new to Python syntax will appreciate friendlier messaging, too.

Improved error messages for invalid comparisons were introduced in a PR titled bpo-43797: Improve syntax error for invalid comparisons. In Python 3.10, if you accidentally type an assignment operator instead of the comparison operator in an if statement, you’ll see this message:

>>>
>>> # Python 3.10a7
>>> a = 1
>>> b = 2
>>> if a = b:
  File "<stdin>", line 1
    if a = b:
       ^^^^^
SyntaxError: invalid syntax. Maybe you meant '==' or ':=' instead of '='?

This is a significant improvement over the existing message in Python 3.9:

>>>
>>> # Python 3.9.4
>>> a = 1
>>> b = 2
>>> if a = b:
  File "<stdin>", line 1
    if a = b:
         ^
SyntaxError: invalid syntax

Aside from better SyntaxError messages, AttributeError, NameError, and IndentationError messages have also been improved.

An example Galindo shared in an April 14 tweet shows how Python 3.10 will suggest existing attributes if you mistype the name of an attribute:

>>>
>>> # Python 3.10a7
>>> import collections
>>> collections.namedtoplo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'collections' has no attribute 'namedtoplo'. Did you mean: 'namedtuple'?

Some users expressed concern about the cost of searching for existing names. Here’s Galindo’s response:

It only happens when displaying uncaught exceptions, so it happens when the interpreter is going to end so it won’t affect runtime. It also has many limits (like the lengths of the strings or the amount of candidates) to keep the cost minimal even in this case. (Source)

Overall it appears that the improved error messages are a big improvement to Python’s user experience. For a complete list of error message improvements, check out the What’s New in Python 3.10 page in the Python docs.

PEP 563, PEP 649, and the Future of Python Type Annotations

PEP 484 introduced type hints back in 2014. Type hints allow you to specify types on function parameters, class attributes, and variables, which can later be statically type checked using tools like Mypy.

For example, the following code defines the function add(), which adds two integers, x and y:

def add(x: int, y: int) -> int:
    return x + y

The type hints specify that the two parameters x and y should be of type int and that the function returns an int value.

How Type Hints Are Currently Evaluated

Currently, type hints must be valid Python expressions since they are evaluated at function definition time. Once type hints have been evaluated, they are stored as strings in an object’s .__annotations__ attribute:

>>>
>>> def add(x: int, y: int) -> int:
...     return x + y
...
>>> add.__annotations__
{'x': 'int', 'y': 'int', 'return': 'int'}

You can usually retrieve the actual type objects using get_type_hints() from the typing module:

>>>
>>> import typing
>>> typing.get_type_hints(add)
{'x': <class 'int'>, 'y': <class 'int'>, 'return': <class 'int'>}

get_type_hints() provides some basic support for tools to implement runtime type checking. Python, as a dynamically typed language, will likely never support true runtime type checking without third-party tools.

Evaluating type hints at function definition time has some downsides. For example, the name of the type must be defined. Otherwise, Python will raise a NameError:

>>>
>>> def add(x: Number, y: Number) -> Number:
...     return x + y
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'Number' is not defined

You can avoid this by defining a type alias using something like typing.Union to create a Number alias that allows int and float values:

>>>
>>> from typing import Union
>>> Number = Union[int, float]
>>> def add(x: Number, y: Number) -> Number:
...     return x + y
...
>>> # No NameError is raised!

One situation where you can’t define a type alias before you use it, though, occurs when you need to return an instance of a class from a method in the class itself:

class Point:
    def __init__(self, x: float, y: float) -> None:
        self.x = x
        self.y = y

    @classmethod
    def origin(cls) -> Point:  # Point type is still undefined
        return cls(0, 0)

Using a name before it’s defined is known as a forward reference. To get around this situation, PEP 484 requires using a string literal for the type name:

class Point:
    def __init__(self, x: float, y: float) -> None:
        self.x = x
        self.y = y

    @classmethod
    def origin(cls) -> "Point":
        return cls(0, 0)

Evaluation of the string is delayed until the module has completely loaded.

Another downside to evaluating type annotations at function definition time is that it adds computational overhead when importing typed modules.

What PEP 563 Proposed to Improve Type Annotations

In September 2017, PEP 563 proposed changing from evaluating annotations at function definition time to preserving them in string form in the built-in __annotations__ dictionary. This effectively solves the forward reference issue and eliminates the computational overhead from importing typed modules.

Starting with Python 3.7, you can access this new behavior by importing annotations from __future__. This lets you use a Number type hint even if it hasn’t been defined:

>>>
>>> from __future__ import annotations
>>> def add(x: Number, y: Number) -> Number:
...     return x + y
...
>>> # No NameError is raised!
>>> add.__annotations__
{'x': 'Number', 'y': 'Number', 'return': 'Number'}

To evaluate the type hints, you need to call get_type_hints(). In other words, evaluation of type hints is postponed until get_type_hints() or some other function, such as eval(), is called.

Initially, the plan was to gradually introduce this new behavior and eventually adopt it as the default in Python 4. However, the decision was made after the 2020 Python Language Summit to reschedule adoption for Python 3.10, which will be released in October 2021.

With regard to how this new behavior will affect Python users, PEP 563 says the following:

Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation. (Source)

In other words, tools can no longer expect type hints to be evaluated for them and need to update their code to evaluate the annotations explicitly as required.

With PEP 563 accepted and plans in place to change the default behavior, projects like FastAPI and pydantic, which use annotations at runtime, set out to support the PEP.

Why Projects Struggled to Implement PEP 563

On April 15, 2021, pydantic owner and core contributor Samuel Colvin authored an issue on pydantic’s GitHub repository explaining how “trying to evaluate those strings to get the real annotation objects is really hard, perhaps impossible to always do correctly.”

Colvin listed twenty-two pydantic issues highlighting the struggle the maintainers had encountered trying to implement PEP 563. He had this to say about why the implementation was so hard:

The reasons are complicated but basically typing.get_type_hints() doesn’t work all the time and neither do the numerous hacks we’ve introduced to try and get fix it. Even if typing.get_type_hints() was faultless, it would still be massively slower than the current semantics. (Source)

Searching bugs.python.org for “get_type_hints” reveals a number of open issues and seems to confirm Colvin’s claim that typing.get_type_hints() doesn’t work correctly in every scenario.

As a solution, Colvin points to PEP 649, which was authored by Python core developer Larry Hastings in January 2021. PEP 649 proposes a deferred evaluation of annotations instead of PEP 563’s postponed evaluation.

In short, PEP 649 defers the evaluation of type hints until the .__annotations__ attribute is accessed. This solves both of the issues addressed by PEP 563 while simultaneously solving the problems projects like pydantic and FastAPI encountered.

Colvin voiced his support for PEP 649 both in his original issue on pydantic’s GitHub repository as well as in an April 15 message on the python-dev mailing list.

How Python Users and Core Developers Reached an Amicable Solution

After Colvin voiced his concerns on python-dev, core developers began a discourse with him about how to solve the problem.

When one user pleaded for the steering council to accept PEP 649 and avoid breaking pydantic, core developer Paul Ganssle responded by pointing out that those were not the only options and proposed keeping PEP 563 optional until the release of Python 3.11:

I should point out that “accept PEP 649” and “break pydantic” are not the only options here. The thing that will break pydantic is the end of PEP 563’s deprecation period, not a failure to implement PEP 649.

Other viable options include:

  • Leave PEP 563 opt-in until we can agree on a solution to the problem.
  • Leave PEP 563 opt-in forever.
  • Deprecate PEP 563 and go back to status quo ante.

… Assuming this is a real problem (and based in part on how long it took for attrs to get what support it has for PEP 563 … I wouldn’t be surprised if PEP 563 is quietly throwing a spanner in the works in several other places as well), my vote is to leave PEP 563 opt-in until at least 3.11 rather than try to rush through a discussion on and implementation of PEP 649. (Source)

Even Pablo Galindo chimed in on Colvin’s original pydantic issue, expressing his wish that Colvin had notified the core team earlier while also confirming that the team was taking Colvin’s feedback seriously:

As a release manager of Python 3.10, it makes me sad that the first issues that are mentioned here … go back to 2018 but we are hearing about all these problems and how they impact pydantic critically dangerously close to beta freeze. …

In any case, for us, making sure that all our user base is taken into account is a very serious matter so you can be sure that we will take this into account when discussing the overall issue. (Source)

Python core developer and steering council member Carol Willing also posted to Colvin’s issue in order to validate his concerns and assure everyone that a solution could be reached:

Let me state up front that I’m a very satisfied user of pydantic and FastAPI, and I’m very thankful for the work and contributions the maintainers and community around them have put in. …

I’m optimistic that we can find a win-win for pydantic / FastAPI and Python. I believe this is possible if we try not to polarize the solution prematurely to “all or nothing” or “accept or reject 649”. To accomplish that we need to look at this through the lens of “what is possible”, balance the tradeoffs, and work toward a “good but perhaps not ideal” solution. (Source)

Finally, on April 20, just five days after Colvin alerted the Python core developers to the issues pydantic faced, the steering council announced that it would be postponing the adoption of PEP 563 until Python 3.11:

The Steering Council has considered the issue carefully, along with many of the proposed alternatives and solutions, and we’ve decided that at this point, we simply can’t risk the compatibility breakage of PEP 563. We need to roll back the change that made stringified annotations the default, at least for 3.10. (Pablo is already working on this.)

To be clear, we are not reverting PEP 563 itself. The future import will keep working like it did since Python 3.7. We’re delaying making PEP 563 string-based annotations the default until Python 3.11. This will give us time to find a solution that works for everyone. (Source)

The steering council’s decision was welcomed by Colvin and celebrated by pydantic and FastAPI users. The decision also attracted praise from Python creator Guido van Rossum, who applauded the steering council:

You have the wisdom of Solomon. Rolling back the code that made PEP 563 the default behavior is the only sensible solution for 3.10. (Source)

In the end, the pydantic maintainers avoided a serious headache, as did the Python core developers. As pointed out by Galindo, though, it’s important that maintainers who experience issues while implementing a PEP contact the core developers as soon as possible in order to avoid a chaotic situation and ensure needs can be met in a timely manner.

It looks like Colvin has taken that feedback to heart. In his response to Galindo after hearing about the steering council’s decision, he stated:

This has been a really positive experience for me and I feel much more positive about talking to the python-dev community.

In future I will engage with the release process and treat it like something I’m a (very small) part of, rather than something that happens to me. (Source)

The idea that project maintainers are a part of the Python release process, rather than helpless bystanders subject to the whims of faceless developers, is an important takeaway for everyone.

As Python users, our feedback is instrumental in helping Python core developers and the steering council make decisions. And even though we may not agree with every decision that gets made, Colvin’s experience is proof that the steering council listens to us.

What’s Next for Python?

April saw some exciting new developments in Python. At Real Python, we’re excited about Python’s future and can’t wait to see what new things are in store for us in May.

What’s your favorite piece of Python news from April? Did we miss anything notable? Let us know in the comments, and we might feature you in next month’s Python news roundup.

Happy Pythoning!

🐍 Python Tricks 💌

Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

Python Tricks Dictionary Merge

About David Amos

David Amos David Amos

David is a writer, programmer, and mathematician passionate about exploring mathematics through code.

» More about David

Each tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are:

Master Real-World Python Skills With Unlimited Access to Real Python

Join us and get access to hundreds of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

Master Real-World Python Skills
With Unlimited Access to Real Python

Join us and get access to hundreds of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

What Do You Think?

Real Python Comment Policy: The most useful comments are those written with the goal of learning from or helping out other readers—after reading the whole article and all the earlier comments. Complaints and insults generally won’t make the cut here.

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Keep Learning

Related Tutorial Categories: community