Python News: What's New From April 2021

CPython Developer-in-Residence

Thanks to Google’s sponsorship funds, the PSF has announced its plan to hire a CPython developer-in-residence. According to the PSF’s announcement, the developer-in-residence will “address backlog, perform analytical research to understand the project’s volunteer hours and funding, investigate project priorities and their tasks going forward, and begin working on those priorities.”

The full-time position is funded for one year, and résumé submissions are being accepted until May 16, 2021. However, the position appears to be open only to existing core developers.

Hiring a full-time employee to support CPython development is an enormous step forward for the PSF and for the Python community. The decision was inspired by the Django Fellowship Program, which hires paid contractors to handle administrative and community management tasks.

Python Packaging Project Manager

Bloomberg’s donations are helping to fund a Python Packaging project manager position. According to the PSF’s announcement, the project manager will “oversee improvements and added functionality that will benefit all Python users while leading the development of PyPI into a sustainable service.”

Applications are open until May 18, 2021. For more information about the position, check out the Python Jobs Board.

You can read more about Bloomberg’s decision to support Python and why they’re specifically interested in the Python Packaging ecosystem in their blog post Supporting the Python community by “Shifting Left.”

How Type Hints Are Currently Evaluated

Currently, type hints must be valid Python expressions since they are evaluated at function definition time. Once type hints have been evaluated, they are stored as strings in an object’s .__annotations__ attribute:

>>> def add(x: int, y: int) -> int:
...     return x + y
...
>>> add.__annotations__
{'x': 'int', 'y': 'int', 'return': 'int'}

You can usually retrieve the actual type objects using get_type_hints() from the typing module:

>>> import typing
>>> typing.get_type_hints(add)
{'x': <class 'int'>, 'y': <class 'int'>, 'return': <class 'int'>}

get_type_hints() provides some basic support for tools to implement runtime type checking. Python, as a dynamically typed language, will likely never support true runtime type checking without third-party tools.

Evaluating type hints at function definition time has some downsides. For example, the name of the type must be defined. Otherwise, Python will raise a NameError:

>>> def add(x: Number, y: Number) -> Number:
...     return x + y
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'Number' is not defined

You can avoid this by defining a type alias using something like typing.Union to create a Number alias that allows int and float values:

>>> from typing import Union
>>> Number = Union[int, float]
>>> def add(x: Number, y: Number) -> Number:
...     return x + y
...
>>> # No NameError is raised!

One situation where you can’t define a type alias before you use it, though, occurs when you need to return an instance of a class from a method in the class itself:

class Point:
    def __init__(self, x: float, y: float) -> None:
        self.x = x
        self.y = y

    @classmethod
    def origin(cls) -> Point:  # Point type is still undefined
        return cls(0, 0)

Using a name before it’s defined is known as a forward reference. To get around this situation, PEP 484 requires using a string literal for the type name:

class Point:
    def __init__(self, x: float, y: float) -> None:
        self.x = x
        self.y = y

    @classmethod
    def origin(cls) -> "Point":
        return cls(0, 0)

Evaluation of the string is delayed until the module has completely loaded.

Another downside to evaluating type annotations at function definition time is that it adds computational overhead when importing typed modules.

What PEP 563 Proposed to Improve Type Annotations

In September 2017, PEP 563 proposed changing from evaluating annotations at function definition time to preserving them in string form in the built-in __annotations__ dictionary. This effectively solves the forward reference issue and eliminates the computational overhead from importing typed modules.

Starting with Python 3.7, you can access this new behavior by importing annotations from __future__. This lets you use a Number type hint even if it hasn’t been defined:

>>> from __future__ import annotations
>>> def add(x: Number, y: Number) -> Number:
...     return x + y
...
>>> # No NameError is raised!
>>> add.__annotations__
{'x': 'Number', 'y': 'Number', 'return': 'Number'}

To evaluate the type hints, you need to call get_type_hints(). In other words, evaluation of type hints is postponed until get_type_hints() or some other function, such as eval(), is called.

Initially, the plan was to gradually introduce this new behavior and eventually adopt it as the default in Python 4. However, the decision was made after the 2020 Python Language Summit to reschedule adoption for Python 3.10, which will be released in October 2021.

With regard to how this new behavior will affect Python users, PEP 563 says the following:

Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation. (Source)

In other words, tools can no longer expect type hints to be evaluated for them and need to update their code to evaluate the annotations explicitly as required.

With PEP 563 accepted and plans in place to change the default behavior, projects like FastAPI and pydantic, which use annotations at runtime, set out to support the PEP.

Why Projects Struggled to Implement PEP 563

On April 15, 2021, pydantic owner and core contributor Samuel Colvin authored an issue on pydantic’s GitHub repository explaining how “trying to evaluate those strings to get the real annotation objects is really hard, perhaps impossible to always do correctly.”

Colvin listed twenty-two pydantic issues highlighting the struggle the maintainers had encountered trying to implement PEP 563. He had this to say about why the implementation was so hard:

The reasons are complicated but basically typing.get_type_hints() doesn’t work all the time and neither do the numerous hacks we’ve introduced to try and get fix it. Even if typing.get_type_hints() was faultless, it would still be massively slower than the current semantics. (Source)

Searching bugs.python.org for “get_type_hints” reveals a number of open issues and seems to confirm Colvin’s claim that typing.get_type_hints() doesn’t work correctly in every scenario.

As a solution, Colvin points to PEP 649, which was authored by Python core developer Larry Hastings in January 2021. PEP 649 proposes a deferred evaluation of annotations instead of PEP 563’s postponed evaluation.

In short, PEP 649 defers the evaluation of type hints until the .__annotations__ attribute is accessed. This solves both of the issues addressed by PEP 563 while simultaneously solving the problems projects like pydantic and FastAPI encountered.

Colvin voiced his support for PEP 649 both in his original issue on pydantic’s GitHub repository as well as in an April 15 message on the python-dev mailing list.

How Python Users and Core Developers Reached an Amicable Solution

After Colvin voiced his concerns on python-dev, core developers began a discourse with him about how to solve the problem.

When one user pleaded for the steering council to accept PEP 649 and avoid breaking pydantic, core developer Paul Ganssle responded by pointing out that those were not the only options and proposed keeping PEP 563 optional until the release of Python 3.11:

I should point out that “accept PEP 649” and “break pydantic” are not the only options here. The thing that will break pydantic is the end of PEP 563’s deprecation period, not a failure to implement PEP 649.

Other viable options include:

• Leave PEP 563 opt-in until we can agree on a solution to the problem.
• Leave PEP 563 opt-in forever.
• Deprecate PEP 563 and go back to status quo ante.

… Assuming this is a real problem (and based in part on how long it took for attrs to get what support it has for PEP 563 … I wouldn’t be surprised if PEP 563 is quietly throwing a spanner in the works in several other places as well), my vote is to leave PEP 563 opt-in until at least 3.11 rather than try to rush through a discussion on and implementation of PEP 649. (Source)

Even Pablo Galindo chimed in on Colvin’s original pydantic issue, expressing his wish that Colvin had notified the core team earlier while also confirming that the team was taking Colvin’s feedback seriously:

As a release manager of Python 3.10, it makes me sad that the first issues that are mentioned here … go back to 2018 but we are hearing about all these problems and how they impact pydantic critically dangerously close to beta freeze. …

In any case, for us, making sure that all our user base is taken into account is a very serious matter so you can be sure that we will take this into account when discussing the overall issue. (Source)

Python core developer and steering council member Carol Willing also posted to Colvin’s issue in order to validate his concerns and assure everyone that a solution could be reached:

Let me state up front that I’m a very satisfied user of pydantic and FastAPI, and I’m very thankful for the work and contributions the maintainers and community around them have put in. …

I’m optimistic that we can find a win-win for pydantic / FastAPI and Python. I believe this is possible if we try not to polarize the solution prematurely to “all or nothing” or “accept or reject 649”. To accomplish that we need to look at this through the lens of “what is possible”, balance the tradeoffs, and work toward a “good but perhaps not ideal” solution. (Source)

Finally, on April 20, just five days after Colvin alerted the Python core developers to the issues pydantic faced, the steering council announced that it would be postponing the adoption of PEP 563 until Python 3.11:

The Steering Council has considered the issue carefully, along with many of the proposed alternatives and solutions, and we’ve decided that at this point, we simply can’t risk the compatibility breakage of PEP 563. We need to roll back the change that made stringified annotations the default, at least for 3.10. (Pablo is already working on this.)

To be clear, we are not reverting PEP 563 itself. The future import will keep working like it did since Python 3.7. We’re delaying making PEP 563 string-based annotations the default until Python 3.11. This will give us time to find a solution that works for everyone. (Source)

The steering council’s decision was welcomed by Colvin and celebrated by pydantic and FastAPI users. The decision also attracted praise from Python creator Guido van Rossum, who applauded the steering council:

You have the wisdom of Solomon. Rolling back the code that made PEP 563 the default behavior is the only sensible solution for 3.10. (Source)

In the end, the pydantic maintainers avoided a serious headache, as did the Python core developers. As pointed out by Galindo, though, it’s important that maintainers who experience issues while implementing a PEP contact the core developers as soon as possible in order to avoid a chaotic situation and ensure needs can be met in a timely manner.

It looks like Colvin has taken that feedback to heart. In his response to Galindo after hearing about the steering council’s decision, he stated:

This has been a really positive experience for me and I feel much more positive about talking to the python-dev community.

In future I will engage with the release process and treat it like something I’m a (very small) part of, rather than something that happens to me. (Source)

The idea that project maintainers are a part of the Python release process, rather than helpless bystanders subject to the whims of faceless developers, is an important takeaway for everyone.

As Python users, our feedback is instrumental in helping Python core developers and the steering council make decisions. And even though we may not agree with every decision that gets made, Colvin’s experience is proof that the steering council listens to us.