Python 3.12 Preview: Static Typing Improvements

Generic Classes and Functions

You’ve seen how you use TypeVar to define type variables. In Python 3.12, you don’t need to explicitly declare type variables when you’re defining generic classes. Instead, you can use the following syntax:

# generic_queue.py

from collections import deque

class Queue[T]:
    def __init__(self) -> None:
        self.elements: deque[T] = deque()

    def push(self, element: T) -> None:
        self.elements.append(element)

    def pop(self) -> T:
        return self.elements.popleft()

The syntax Queue[T] implies that you’re defining a generic class parametrized by T. Note that this implicitly defines T as a type variable that you can refer to within the class definition.

In addition to the cleaner syntax, the fact that you don’t need to import Generic or TypeVar or explicitly define T makes your code cleaner and more readable.

Pyright already supports the new syntax. However, when you’re using it, you need to explicitly tell Pyright that you’re using Python 3.12 syntax:

$ pyright --pythonversion 3.12 generic_queue.py
0 errors, 0 warnings, 0 informations

You can provoke the type checker similarly to earlier if you want to see the kind of errors that it can report.

So far, you’ve looked at generic classes. You can also define generic functions. These are functions that can handle several different types in their parameters.

As an example, you’ll implement push_and_pop() which can push an element to the end of a generic list and pop one off the front of it. First, you’ll implement it with the old syntax for type variables:

# list_helpers.py

from typing import TypeVar

T = TypeVar("T")

def push_and_pop(elements: list[T], element: T) -> T:
    elements.append(element)
    return elements.pop(0)

As earlier, you declare that T is a type variable. Then you use T as part of your annotations of push_and_pop(). In particular, you make sure that element matches the type of the elements already in the list.

You can experiment with the function to see that it allows you to use a list as a dynamic queue where an element pops off each time you insert a new one:

>>> from list_helpers import push_and_pop

>>> elements = [28, 1]
>>> push_and_pop(elements, 23)
28
>>> push_and_pop(elements, 10)
1

>>> elements
[23, 10]

You start with a list with two elements. When you insert 23 at the back of the list, then you’re also removing the first element, 28.

In Python 3.12, you can use the new syntax for such generic functions as well. Like you would for the generic class syntax, you declare the type variable inside square brackets:

# list_helpers.py

def push_and_pop[T](elements: list[T], element: T) -> T:
    elements.append(element)
    return elements.pop(0)

Again, note that you don’t need to declare the type variable any longer, apart from listing it within the square brackets.

This code is equivalent to the one above. There’s one significant difference, though. In the earlier example, T is defined in the global scope and is available outside the definition of push_and_pop(). In the latter code block, T is only available in annotations within the function.

In the examples that you’ve explored so far, the type variables could take on any type. This is often the case with collections that can contain any type of object. Type variables can also be useful in cases where the type is more constrained. You’ll look more closely at this in the next section.

Constrained and Bounded Type Variables

Sometimes you have a function that can take arguments of several—but not all—different types. Type variables support this through constraints and bounds:

• A constrained type variable takes on one of a fixed set of possible types.
• A bounded type variable takes on the type of the boundary type or one of its subtypes.

You’ll investigate both constrained and bounded type variables in this section and learn how the new syntax supports them.

As an example, say that you’re wrapping the concatenation operator (+) in a function to make combining strings more explicit:

# concatenation.py

def concatenate(first: str, second: str) -> str:
    return first + second

This function adds two strings together to form a new string. Your motivation for adding concatenate() is that it names the operation properly. You can use it to combine two strings as follows:

>>> from concatenation import concatenate

>>> concatenate("twenty", "twentythree")
'twentytwentythree'

In the type hints, you’ve noted that concatenate() is meant to work on strings. Technically, there are many objects in Python that you could add together. However, you wouldn’t necessarily call the operation concatenation. Therefore, your function doesn’t semantically support numbers.

However, there’s another type where + represents concatenation: bytes objects. You can also use concatenate() on them. For example, you can combine the following bytes literals:

>>> from concatenation import concatenate

>>> concatenate(b"twenty", b"twentythree")
b'twentytwentythree'

The only visible difference from strings is the b prefix indicating that each quoted element is a bytes object. You’d like the type hints to include bytes in addition to str.

You can do this by defining a constrained type variable. In the old syntax, this looked as follows:

# concatenation.py

from typing import TypeVar

T = TypeVar("T", str, bytes)

def concatenate(first: T, second: T) -> T:
    return first + second

In the declaration of T, you constrain T such that only str and bytes are valid types. If you try to concatenate two numbers, then the type checker will warn you, but the code will run. This T can only be str or bytes, and no other types are valid.

The new syntax also supports constraints. You can add type constraints in a literal tuple:

# concatenation.py

def concatenate[T: (str, bytes)](first: T, second: T) -> T:
    return first + second

As earlier, you declare T by adding it to the function definition inside square brackets. Additionally, you note that T must be either str or bytes by specifying those types in a tuple literal separated from T by a colon.

A related use case concerns bounded type variables. A bounded type variable is also limited by a specific type—its boundary type. In contrast to constrained type variables, bounded type variables can materialize as subtypes of their boundary type. For example, a type variable bounded by str can be represented by str or any subclass of str.

In the following example, you implement inspect(), which provides a simple description of a string. This function is meant to help with debugging, so it returns the original string, such that you can insert inspect() in the middle of the original processing.

You also define Words as a subclass of str. This is the start of a word-oriented string type where you only override how to calculate the length of a string:

# inspect_string.py

class Words(str):
    def __len__(self):
        return len(self.split())

def inspect[S: str](text: S) -> S:
    print(f"'{text.upper()}' has length {len(text)}")
    return text

The annotation [S: str] says that inspect() is a generic function parametrized by the type variable S. Furthermore, S is bound by str. In practice, this means that the text argument must be a string or a subclass of str. For example, both of the following uses are valid:

>>> from inspect_string import inspect, Words

>>> inspect("Hello, World!")
'HELLO, WORLD!' has length 13
'Hello, World!'

>>> inspect(Words("Hello, World!"))
'HELLO, WORLD!' has length 2
'Hello, World!'

Both the string and the Words subclass are valid arguments to inspect() since you’ve annotated text with a type variable bound to str.

You could’ve defined inspect() as a regular function and used a plain text: str type hint to say the same. You need to use the type variable in order to connect the type of the text argument to the return type of inspect().

You can use constrained and bounded type variables when you define generic classes as well. To do so, you use the same syntax, with the type variable separated by a colon from the constraint or bound. To sum up, compare the syntax of a free, constrained, and bounded type variable:

>>> def free[T](argument: T) -> T: ...

>>> def constrained[T: (str, bytes)](argument: T) -> T: ...

>>> def bounded[T: str](argument: T) -> T: ...

In these examples, T is a type variable:

• In free(), T can be any type.
• In constrained(), T can only be either str or bytes.
• In bounded(), T can be str or any subclass of str.

In addition to generic classes and functions, there’s a third way to use type variables. Namely, you can use them in generic type aliases. You’ll learn more about those in the next section.

Type Aliases

Sometimes, you want to use a different name or alias for a type. This could be because the type is cumbersome to describe. For example, you could be working with data represented as a list of tuples, with each tuple consisting of a string and an integer. In that case, you could define an alias:

ListOfTuples = list[tuple[str, int]]

Another reason for using an alias is to be more descriptive about your data structures. Maybe your list of tuples actually represents a deck of cards. In that case, you can use the type alias to say so:

CardDeck = list[tuple[str, int]]

Originally, type aliases were defined as simple assignments. PEP 613 introduced explicit type aliases to make them less ambigous.

Before Python 3.12, you needed to import TypeAlias from typing to use explicit type aliases:

from typing import TypeAlias

CardDeck: TypeAlias = list[tuple[str, int]]

Using TypeAlias allows the type checker to infer that CardDeck is indeed a type alias. Without this annotation, that wasn’t always clear.

In Python 3.12, you can use a powerful new syntax to define type aliases:

type CardDeck = list[tuple[str, int]]

One advantage to using type like this is that you don’t need to make any imports. You use type aliases as drop-in replacements for the more complex type descriptions that they alias:

# deck.py

import random

type CardDeck = list[tuple[str, int]]

def shuffle(deck: CardDeck) -> CardDeck:
    return random.sample(deck, k=len(deck))

Here, shuffle() returns a new deck with the original cards shuffled into a random order. You use the CardDeck alias to annotate both the deck argument and the return value of shuffle().

You can also define generic type aliases. These are type aliases that are parametrized by some other type. With the traditional syntax, you’ll do something like this:

from typing import TypeAlias, TypeVar

T = TypeVar("T")

Ordered: TypeAlias = list[T] | tuple[T, ...]

In this case, Ordered is a generic type that you can represent as either a list or a tuple. In Python 3.12, you can simplify this definition as follows:

type Ordered[T] = list[T] | tuple[T, ...]

This combines the type syntax for defining type aliases with the square brackets that you used earlier to define generics. Together, they give you generic type aliases.

There are two points to note about the new type syntax. First of all, it uses type as a soft keyword. Keywords are identifiers that are reserved by the language and can’t be used variables. Python has more than thirty such reserved words, including def, class, and return.

A soft keyword is a keyword that’s only reserved in certain contexts. Normally, type refers to the built-in type() function. However, in type alias assignments, it’s recognized as a keyword.

Second, type aliases defined with type are only usable as type hints. Since the old-style type aliases are simple assignments, you can sometimes use them for runtime checks. For example, you can use isinstance() as follows:

>>> from typing import TypeAlias
>>> number: TypeAlias = int | float

>>> isinstance(3.12, number)
True
>>> isinstance("Python", number)
False

You define number as a type alias representing either an integer or a floating point number. You can then use number directly when using isinstance() to check the type of different objects.

You can’t do the same with type aliases defined with type:

>>> type number = int | float
>>> isinstance(3.12, number)
Traceback (most recent call last):
  ...
TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union

Using this new type alias in isinstance() causes a TypeError. This shouldn’t be a big issue. Type aliases are mainly meant for type annotations, which you want to keep separate from any runtime type enforcement that you need to do.

As you’ve seen, Python 3.12 simplifies the syntax for working with generic classes, functions, and type aliases. It makes working with generics less cumbersome and more intuitive. Unfortunately, this is new syntax, so it won’t be backported to older versions of Python.

The development, discussion, and implementation of PEP 695 has been a massive undertaking. Core developer Jelle Zijlstra has been instrumental in this effort. He’s also written an interesting report with detail and insight about the implementation.

In the rest of this tutorial, you’ll investigate a couple of new static typing features that also join the language in Python 3.12.

Inheritance in Python

To see an example of @override in action, you’ll implement a basic object-oriented quiz program. Along the way, you’ll practice using inheritance in Python. In the first iteration, you won’t use @override. Consider the following Question class:

# quiz.py

from dataclasses import dataclass

@dataclass
class Question:
    question: str
    answer: str

    def ask_question(self) -> bool:
        answer = input(f"\n{self.question} ")
        return answer == self.answer

For simplicity, you define Question as a data class. It’ll have two string attributes, .question and .answer. Additionally, .ask_question() will ask the question and compare a user’s answer to the correct one. The method returns True if the user answers correctly and False if they don’t.

To run a quiz, you’ll add a couple of questions and have the program ask them in a loop:

# quiz.py

import random
from dataclasses import dataclass

# ...

questions = [
    Question("Who created Python?", "Guido van Rossum"),
    Question("What's a PEP?", "A Python Enhancement Proposal"),
]

score = 0
for question in random.sample(questions, k=len(questions)):
    if question.ask_question():
        score += 1
        print("Yes, that's correct!")
    else:
        print(f"No, the answer is '{question.answer}'")
print(f"\nYou got {score} out of {len(questions)} correct")

You use random.sample() to shuffle the questions so that your program asks them in a random order. Since .ask_question() returns True or False depending on whether the answer is correct, you can use an if block to give some feedback to your user and keep track of their score. Running the quiz will look something like this:

$ python quiz.py
What's a PEP? A Python Enhancement Proposal
Yes, that's correct!

Who created Python? Uncle Barry
No, the answer is 'Guido van Rossum'

You got 1 out of 2 correct

The example code is intentionally basic, as you’ll soon use it to explore the new @override decorator. If you’re interested in creating a more solid quiz program, then give Build a Quiz Application With Python a spin.

You’ll introduce one improvement to this quiz program, though. To make the game less of a spelling bee, you’ll add support for multiple-choice questions. One way to do this is to add a new class that inherits from Question:

# quiz.py

import random
from dataclasses import dataclass
from string import ascii_lowercase

# ...

@dataclass
class MultipleChoiceQuestion(Question):
    distractors: list[str]

    def ask_question(self) -> bool:
        print(f"\n{self.question}")

        alternatives = random.sample(
            self.distractors + [self.answer], k=len(self.distractors) + 1
        )
        labeled_alternatives = dict(zip(ascii_lowercase, alternatives))
        for label, alternative in labeled_alternatives.items():
            print(f"   {label}) {alternative}", end="")

        answer = input("\n\nChoice? ")
        return labeled_alternatives.get(answer) == self.answer

# ...

The new class, MultipleChoiceQuestion, inherits from Question and overrides .ask_question() to handle showing answer alternatives to the user. To create a multiple-choice question, you need to provide a list of distractors, or wrong answers, in addition to the question and answer. Your code shuffles all the answer alternatives and labels them each with a letter that the user can use when answering.

To use your new class, you update one of your questions:

# quiz.py

# ...

questions = [
    Question("Who created Python?", "Guido van Rossum"),
    MultipleChoiceQuestion(
        "What's a PEP?",
        "A Python Enhancement Proposal",
        distractors=[
            "A Pretty Exciting Policy",
            "A Preciously Evolved Python",
            "A Potentially Epic Prize",
        ],
    ),
]

# ...

Here, you’ve changed the PEP question to a MultipleChoiceQuestion by adding three distractors. You can see your new code in action:

$ python quiz.py
Who created Python? Guido van Rossum
Yes, that's correct!

What's a PEP?
   a) A Potentially Epic Prize   b) A Python Enhancement Proposal
   c) A Preciously Evolved Python   d) A Pretty Exciting Policy

Choice? b
Yes, that's correct!

You got 2 out of 2 correct

Now, change gears to see how @override can improve your code. First, there’s nothing wrong with your code so far. However, if you’re not careful, there are a few potential bugs that could sneak into your code.

For example, if there’s a typo in the name when you define .ask_question() in MultipleChoiceQuestion, then your code will call .ask_question() in the Question class instead. Try to rename MultipleChoiceQuestion.ask_question() to something like .ask_questoin() to see what happens! The game still runs, but you lose the special handling of multiple-choice questions.

Similarly, you may realize that you can simplify the method name by changing Question.ask_question() to Question.ask(). Your editor may help you with this refactoring. However, if you miss renaming .any_question() in the subclass, then the game will break again. As above, you won’t get any error messages. You’ll only see that multiple-choice questions stop behaving properly.

If you use @override, then you’ll have some protection against these kinds of issues.

Safe Refactoring With @override

You’ll now add the @override decorator to your quiz program and experiment with how it can help keep you safe from certain bugs. Python 3.12 adds @override to the typing standard library module. As with many typing features, you may also import it from typing_extensions on earlier Python versions.

Start by marking MultipleChoiceQuestion.ask_question() as an override:

# quiz.py

import random
from dataclasses import dataclass
from string import ascii_lowercase
from typing import override

# ...

@dataclass
class MultipleChoiceQuestion(Question):
    distractors: list[str]

    @override
    def ask_question(self) -> bool:
        # ...

# ...

You need to make two updates to your code. First, you import @override from typing. Next, you mark .ask_question() as an override by applying the decorator to the method. This won’t have any effect on what your program does when you run it.

You can use Pyright to check that you’ve properly typed your file:

$ pyright --pythonversion 3.12 quiz.py
0 errors, 0 warnings, 0 informations

Currently, quiz.py is in tip-top shape! Next, you’ll introduce some bugs in your code to see how Pyright will warn you. First, you’d want to refactor the method name by changing .ask_question() to .ask(). Do the update, but only in the parent Question class and in the supporting code:

# quiz.py

# ...

@dataclass
class Question:
    question: str
    answer: str

    def ask(self) -> bool:
        answer = input(f"\n{self.question} ")
        return answer == self.answer

# ...

score = 0
for question in random.sample(questions, k=len(questions)):
    if question.ask():
        score += 1
        print("Yes, that's correct!")
    else:
        print(f"No, the answer is '{question.answer}'")
print(f"\nYou got {score} out of {len(questions)} correct")

Notably, you don’t update MultipleChoiceQuestion.ask_question(). Many editors can help you with renaming methods like this, but in big codebases, it’s still possible to miss updating a method name. However, Pyright can now point out what’s gone wrong:

$ pyright --pythonversion 3.12 quiz.py
quiz.py
  quiz.py:21:9 - error: Method "ask_question" is marked as override, but no
      base method of same name is present (reportGeneralTypeIssues)
1 error, 0 warnings, 0 informations

The message says that you’ve marked .ask_question() as an override, but the method isn’t overriding any method in a parent class. Because you changed the name to .ask() in Question, you’d need to change the name to .ask() in MultipleChoiceQuestion as well.

You’ll get similar protection against misspelling a method name. Say that when you go back to your code to update the method name, you end up with the following:

# quiz.py

# ...

@dataclass
class MultipleChoiceQuestion(Question):
    distractors: list[str]

    @override
    def askk(self) -> bool:
        # ...

# ...

Note that .askk() is misspelled. For your program, this will have the same effect as not renaming the method. Your multiple-choice questions will not behave properly. If you run Pyright, then you’ll also get a similar message:

$ pyright --pythonversion 3.12 quiz.py
quiz.py
  quiz.py:39:9 - error: Method "askk" is marked as override, but no base
      method of same name is present (reportGeneralTypeIssues)
1 error, 0 warnings, 0 informations

Hopefully, these messages will alert you to the correct issue. If you’re working on a project that uses inheritance, then applying @override can give you more confidence that your code works correctly.

Using @override is optional. Your type checker shouldn’t enforce these hints by default. Consider what happens if you fix MultipleChoiceQuestion but remove @override:

# quiz.py

# ...

@dataclass
class MultipleChoiceQuestion(Question):
    distractors: list[str]

    def ask(self) -> bool:
        # ...

# ...

By removing @override, you’re leaving the type checker in the dark about whether .ask() is meant to override its parent method or not. However, Pyright won’t complain:

$ pyright --pythonversion 3.12 quiz.py
0 errors, 0 warnings, 0 informations

If you want to make sure that you’re using @override consistently, then you can add a diagnostic flag to Pyright’s configuration. One way to do this is to specify the following in a file named pyproject.toml:

[tool.pyright]
reportImplicitOverride = true

You can now rerun Pyright:

$ pyright --pythonversion 3.12 quiz.py
quiz.py
  quiz.py:38:9 - error: Method "ask" is not marked as override but is
      overriding a method in class "Question" (reportImplicitOverride)
1 error, 0 warnings, 0 informations

By enabling reportImplicitOverride, you’re telling Pyright that you intend to use @override everywhere that it’s applicable. You’re therefore getting a message saying that you should annotate .ask() since it’s overriding Question.ask().

This offers you the choice to gradually introduce @override in projects where that makes sense and to enforce the use of @override when you find that convenient.