Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: What's New in Python 3.12
Python 3.12 was published on October 2, 2023. As usual, the new version comes out in October after lots of effort by volunteers worldwide.
The new version comes with several new features and improvements that you’ll explore in this tutorial. You can also dive into the documentation to see a complete list of all changes.
In this tutorial, you’ll learn about new features and improvements, like:
- Better error messages with helpful suggestions and guidance
- More expressive f-strings that are backed by Python’s PEG parser
- Optimizations, including inlined comprehensions, for a faster Python
- A new syntax for type variables that you use to annotate generics
- Support for the powerful
perf
profiler on Linux
If you want to try any of the examples in this tutorial, then you’ll need to use Python 3.12. The tutorials Python 3 Installation & Setup Guide and How Can You Install a Pre-Release Version of Python? walk you through several options for adding a new version of Python to your system.
In addition to learning more about the new features coming to the language, you’ll also get some advice about what to consider before upgrading to the new version. Click the link below to download code examples demonstrating the new capabilities of Python 3.12:
Free Bonus: Click here to download your sample code for a sneak peek at Python 3.12, coming in October 2023.
Improved Error Messages
Python is usually recognized as a good beginner language, and it’s lauded for its readable syntax. One area where it’s become even more user-friendly lately is error messages.
In Python 3.10, many error messages—especially for syntax errors—got more informative and precise. Similarly, Python 3.11 added more information in tracebacks, making it more convenient to pinpoint offending code.
The latest version of Python continues the work of improving your developer experience by providing better error messages. In particular, several common error messages now come with helpful suggestions. In the rest of this section, you’ll explore the new and improved messages.
Several of the improvements relate to importing modules. In the next three examples, you try to work with π by importing pi
from math
. In each example, you’ll see one of the new suggestions in Python 3.12. Here’s the first one:
>>> math.pi
Traceback (most recent call last):
...
NameError: name 'math' is not defined. Did you forget to import 'math'?
When you use math
without importing it first, you’ll get a traditional NameError
. Additionally, the parser helpfully reminds you that you need to import math
before accessing it.
The reminder about remembering to import modules only triggers for standard library modules. For these error messages, Python 3.12 doesn’t track third-party packages that you’ve installed.
It’s possible to import specific names from a module using a from
-import
statement. If you happen to switch the order of the keywords, you’ll now get a friendly nudge towards the correct syntax:
>>> import pi from math
...
import pi from math
^^^^^^^^^^^^^^^^^^^
SyntaxError: Did you mean to use 'from ... import ...' instead?
Here, you tried to import pi
from math
, but Python needs you to reorder the statement and put from
before import
.
To see a third new error message, check out what happens if you import py
and not pi
from math
:
>>> from math import py
Traceback (most recent call last):
...
ImportError: cannot import name 'py' from 'math'. Did you mean: 'pi'?
There’s no py
name in math
, so you get an ImportError
. The parser suggests that you probably meant pi
instead of py
. Python 3.10 introduced a similar suggestion feature, where Python looks for similar names. What’s new in Python 3.12 is the ability to do this while importing.
In addition to these three improvements related to imports, there’s a final improvement concerning methods defined inside classes. Have a look at the following implementation of a Circle
class:
1# shapes.py
2
3from math import pi
4
5class Circle:
6 def __init__(self, radius):
7 self.radius = radius
8
9 def area(self):
10 return pi * radius**2
Note that you wrongly refer to radius
instead of self.radius
inside .area()
. This will cause an error when you call the method:
>>> from shapes import Circle
>>> Circle(5).area()
Traceback (most recent call last):
...
File "/home/realpython/shapes.py", line 10, in area
return pi * radius**2
^^^^^^
NameError: name 'radius' is not defined. Did you mean: 'self.radius'?
Instead of raising a plain NameError
, Python recognizes that .radius
is an attribute available on self
. It then suggests that you use the instance attribute self.radius
instead of the local name radius
.
The suggestions that you’ve seen in these examples are all new in Python 3.12. Together, they make Python a little more user-friendly. You can learn more about these error message improvements and how they’ve been implemented in Python 3.12 Preview: Ever Better Error Messages.
More Powerful F-Strings
Formatted strings, or f-strings for short, were introduced in PEP 498 and Python 3.6. With f-strings, Python added string interpolation to the language. You can recognize f-strings by the leading f
in examples like the following:
>>> from datetime import date
>>> major = 3
>>> minor = 11
>>> release = date(2023, 10, 2)
>>> f"Python {major}.{minor + 1} is released on {release:%B %-d}"
'Python 3.12 is released on October 2'
This f-string contains three pairs of curly braces ({}
). Each pair contains an expression that’s interpolated into the final string. The first expression only refers to major
directly, while the second one applies a small operation on minor
.
The third expression shows that you can add certain format specifiers to control the interpolation of an expression. In this case release
is a date, so %B
and %-d
are interpreted as date format specifiers, formatting the date as month day.
F-strings were originally implemented with a dedicated parser. That means that even though expressions inside f-strings are regular Python expressions, the expressions weren’t parsed with the regular Python parser. Instead, the developers implemented a separate parser, which they needed to maintain.
The primary reason for this was the inabilty of Python’s old LL(1) parser to support f-strings. After the introduction of the PEG parser in Python 3.9, this is no longer true. The current parser could parse expressions within f-strings.
Based on PEP 701, f-strings are now formalized as additions to Python’s grammar in Python 3.12. In practice, this means that the PEG parser will parse f-strings, just like regular Python code.
For the most part, you won’t notice this change to f-strings. It mainly benefits the core developers maintaining Python’s source code. However, there are a few changes visible to anyone using f-strings.
In general, the new implementation of f-strings lifts some restrictions that were added originally. Many of these restrictions were put in place to make f-strings easier to handle for external tools like IDEs and code highlighters. Below, you’ll explore some examples that weren’t possible before.
You may now reuse the string quote character inside of the f-string. For example, if you’ve delimited your f-string with double-quotes ("
), you can still use "
inside expressions:
>>> version = {"major": 3, "minor": 12}
>>> f"Python {version["major"]}.{version["minor"]}"
'Python 3.12'
Even though double-quotes delimit the f-string, you can still use "
to specify the keys inside the f-string.
Until now, you haven’t been able to use a backslash character (\
) inside an f-string expression. Going forward, you can use backslashes in f-string expressions as in any other expression:
>>> names = ["Brett", "Emily", "Gregory", "Pablo", "Thomas"]
>>> print(f"Steering Council:\n {"\n ".join(names)}")
Steering Council:
Brett
Emily
Gregory
Pablo
Thomas
Here, you use \n
, which represents a newline character, in both the string and expression parts of the f-string. Previously, the latter wasn’t allowed.
As with other types of braces and parentheses, you can now add newlines inside the curly braces delimiting expressions in f-strings. As an added bonus, you can also add comments to expressions. Since a comment extends to the end of the line, you need to close the expression on the next line or later.
To see how this works, continue the example from above:
>>> f"Steering council: {
... ", ".join(names) # Steering council members
... }"
'Steering council: Brett, Emily, Gregory, Pablo, Thomas'
The highlighted line shows a commented f-string expression on a line by itself.
While the important changes to f-strings have happened mostly under the hood, you’ve seen that some dusty corners have been improved, and f-string expressions are now more consistent with the rest of Python.
If you want to dive deeper into the changes in f-strings, have a look at Python 3.12 Preview: More Intuitive and Consistent F-Strings.
Faster Python: More Specializations and Inlined Comprehensions
When Python 3.11 came out in 2022, there was a lot of buzz about optimizations to the interpreter that made Python faster. That work was part of an ongoing effort named faster-cpython, and it’s continued into Python 3.12.
Before a Python script starts running, the code is translated into bytecode. The bytecode is the code that the Python interpreter runs. Python 3.11 uses a specialized adaptive interpreter that can change and adapt the bytecode during execution in order to optimize operations that happen often. This depends on two steps:
- Quickening is the process of noticing that certain bytecode is executed several times, making it a candidate for specialization.
- Specialization means that the interpreter replaces a general bytecode with a specialized one. For example, an operation that adds two floating-point numbers can replace a general addition operation.
In Python 3.12, quickening happens faster than in Python 3.11, and the interpreter can now specialize many new bytecodes. To see the quickening and specialization in action, define the following function:
>>> def feet_to_meters(feet):
... return 0.3048 * feet
...
You can use feet_to_meters()
to convert from feet to meters. To look behind the curtain of the interpreter, you’ll use dis
, which lets you disassemble your Python code and look at the bytecode instead:
>>> import dis
>>> dis.dis(feet_to_meters, adaptive=True)
1 0 RESUME 0
2 2 LOAD_CONST__LOAD_FAST 1 (0.3048)
4 LOAD_FAST 0 (feet)
6 BINARY_OP 5 (*)
10 RETURN_VALUE
Each line shows information about one bytecode instruction. The five columns are the line number, the byte address, the operation code name, the operation parameters, and an interpretation of the parameters in parentheses.
You don’t need to understand the details of this bytecode listing. Still, note that one line of Python code often compiles to several bytecode instructions. In this example, return 0.3048 * feet
translates to four bytecode instructions.
Actually, there’s no separate quickening step any longer. In principle, all bytecode instructions are immediately ready for specialization. In Python 3.11, the specialization kicked in after a bytecode had executed with the same types eight times. Now, this happens already after two calls:
>>> feet_to_meters(1.1)
0.33528
>>> feet_to_meters(2.2)
0.67056
You call feet_to_meters()
twice, each time with a float
argument. The interpreter will then specialize and assume that the multiplication will continue to be between floating point numbers:
>>> dis.dis(feet_to_meters, adaptive=True)
1 0 RESUME 0
2 2 LOAD_CONST__LOAD_FAST 1 (0.3048)
4 LOAD_FAST 0 (feet)
6 BINARY_OP_MULTIPLY_FLOAT 5 (*)
10 RETURN_VALUE
The interpreter has adapted the original BINARY_OP
instruction, replacing it with BINARY_OP_MULTIPLY_FLOAT
, which is faster when both operands are float
.
Even though the interpreter is adapting certain bytecodes, this doesn’t harm Python’s dynamic nature. You can still use feet_to_meters()
with integer arguments. The only bonus to using the same data types is that your program may run faster.
To learn more, check out core developer Brandt Bucher’s presentation at PyCon 2023: Inside Python’s new specializing, adaptive interpreter.
PEP 709 describes a new optimization in Python 3.12: inlined comprehensions. Python supports list comprehensions, dictionary comprehensions, and set comprehensions that you use to transform iterables. For example:
>>> names = ["Brett", "Emily", "Gregory", "Pablo", "Thomas"]
>>> [name[::-1].title() for name in names]
['Tterb', 'Ylime', 'Yrogerg', 'Olbap', 'Samoht']
Here, you use a list comprehension to reverse each of the names. Such a comprehension is currently compiled as a nested function. To explore this, first wrap the comprehension in a function:
>>> def reverse_names(names):
... return [name[::-1].title() for name in names]
...
Similarly to the example above, you reverse each name and make sure that it starts with a capital letter. Now, use dis
to disassemble the function on Python 3.11:
>>> import dis
>>> dis.dis(reverse_names)
1 0 RESUME 0
2 2 LOAD_CONST 1 (<code object <listcomp> at 0x7f2e61f42d30)
4 MAKE_FUNCTION 0
6 LOAD_FAST 0 (names)
8 GET_ITER
10 PRECALL 0
14 CALL 0
24 RETURN_VALUE
Disassembly of <code object <listcomp> at 0x7f2e61f42d30>:
2 0 RESUME 0
2 BUILD_LIST 0
4 LOAD_FAST 0 (.0)
>> 6 FOR_ITER 31 (to 70)
8 STORE_FAST 1 (name)
10 LOAD_FAST 1 (name)
12 LOAD_CONST 0 (None)
14 LOAD_CONST 0 (None)
16 LOAD_CONST 1 (-1)
18 BUILD_SLICE 3
20 BINARY_SUBSCR
30 LOAD_METHOD 0 (title)
52 PRECALL 0
56 CALL 0
66 LIST_APPEND 2
68 JUMP_BACKWARD 32 (to 6)
>> 70 RETURN_VALUE
There are many details in this listing that you can ignore. The important thing to notice is that a new listcomp
code object has been created. In the top part of the bytecode listing, you can see that this internal function has to be loaded and then called.
Compiling a comprehension into a nested function like this is convenient, as the function call isolates the comprehension such that it doesn’t leak variables. However, it’s not necessarily the most efficient implementation. Especially if the comprehension runs over a small iterable, then the overhead of calling the nested function is noticeable.
In Python 3.12, comprehensions are inlined into the bytecode. Have a look at the disassembly of reverse_names()
in the new version:
>>> dis.dis(reverse_names)
1 0 RESUME 0
2 2 LOAD_FAST 0 (names)
4 GET_ITER
6 LOAD_FAST_AND_CLEAR 1 (name)
8 SWAP 2
10 BUILD_LIST 0
12 SWAP 2
>> 14 FOR_ITER 24 (to 66)
18 STORE_FAST 1 (name)
20 LOAD_FAST 1 (name)
22 LOAD_CONST 0 (None)
24 LOAD_CONST 0 (None)
26 LOAD_CONST 1 (-1)
28 BUILD_SLICE 3
30 BINARY_SUBSCR
34 LOAD_ATTR 1 (NULL|self + title)
54 CALL 0
62 LIST_APPEND 2
64 JUMP_BACKWARD 26 (to 14)
>> 66 END_FOR
68 SWAP 2
70 STORE_FAST 1 (name)
72 RETURN_VALUE
Again, the details of the bytecode aren’t important. Instead, note that there’s no extra code object, and no extra function call happens.
In general, the inlined comprehensions are faster than the previous implementation. Comprehensions running over small iterables can be as much as twice as fast as before. If the comprehension runs over a larger iterable with thousands of elements, then you may notice that the new implementation is on par with or even slightly slower than in Python 3.11. However, real-world benchmarks suggest that you may expect a speedup in your code.
You can find code that benchmarks different comprehensions in the accompanying materials that you can download for this tutorial. Use this to check the performance of comprehensions on your computer:
Free Bonus: Click here to download your sample code for a sneak peek at Python 3.12, coming in October 2023.
The effort to make Python faster continues, and there are already many ideas slated for Python 3.13.
Dedicated Type Variable Syntax
Python added support for annotations in version 3.0. While type hinting was one of the motivations for annotations, Python’s support for static typing wasn’t in place until Python 3.5, several years later.
Type variables constitute an important and powerful part of Python’s typing system. A type variable can stand in for a concrete type during static type checking. You use type variables to parametrize generic classes and generic functions. Consider the following example, which returns the first element in a given list:
def first(elements):
return elements[0]
The type of the return value of first()
depends on the kind of list that you pass in. For example, if elements
is a list of integers, then first()
returns int
, while if it’s a list of strings, then the return type is str
. You’ll use type variables to express this relationship.
Python 3.12 introduces a new syntax for type variables. With the new syntax, you can write the following:
def first[T](elements: list[T]) -> T:
return elements[0]
By adding T
inside square brackets to the function definition, you declare that first()
is a generic function parametrized by the type variable T
.
Note: It often makes sense to annotate parameters with Sequence
instead of list
. A sequence is an iterable that supports element access with integer indices. If you want to use Sequence
, then you can rewrite first()
:
from collections.abc import Sequence
def first[T](elements: Sequence[T]) -> T:
return elements[0]
Lists, tuples, and strings are all examples of sequences in Python.
Declaring first()
as a generic function has no effect at runtime. Instead, it helps your editor or static type checker in tracking the types that you use. Look at the following two examples:
>>> def first[T](elements: list[T]) -> T:
... return elements[0]
...
>>> first([2, 3, 1, 0])
2
>>> first(["Brett", "Emily", "Gregory", "Pablo", "Thomas"])
'Brett'
In the first invocation of first()
, you pass in a list of integers. In this case, a type checker will see that T
can be int
and will deduce that first()
returns an int
. In the second example, you pass in a list of string names. In this case, elements
is list[str]
, so T
will be treated as str
.
As noted, type variables have been available for a long time. What Python 3.12 brings to the table is the new and powerful syntax for using them. Previously, you’d implement the same example by importing TypeVar
from typing
:
from typing import TypeVar
T = TypeVar("T")
def first(elements: list[T]) -> T:
return elements[0]
There are two main advantages to the new syntax. First of all, it’s part of Python’s regular grammar, so you don’t need to import the TypeVar
class. Additionally, T
is declared in the function definition, instead of outside the function. This makes the role of the type variable more clear.
You’ve seen the most straightforward use of type variables. The new syntax supports other uses as well. These include multiple type variables, constrained type variables, and bounded type variables, in addition to generic classes and generic type aliases. You’ll explore these use cases in the following examples.
Note: When you’re using TypeVar
, you need to specify whether a type variable is covariant, contravariant, or invariant. This relates to how subtypes interact inside composite types. For example, bool
is a subtype of int
. What does that mean for list[bool]
compared to list[int]
?
These are technical questions. The good news is that with the new syntax, you don’t need to be explicit about variance. Instead, the type checkers will be able to deduce the correct categorization when needed.
You often use tuples to represent a heterogeneous sequence with a predetermined number of elements. A basic example would be a tuple representing information about a person as a name-age pair. In terms of types, you could describe these tuples as tuple[str, int]
.
Now, say that you have a function that flips the order of such tuple pairs. In general, the types of the two elements will be different, so you’d need two type variables to represent them. You can declare two or more type variables inside the square brackets, separated by commas:
def flip[T0, T1](pair: tuple[T0, T1]) -> tuple[T1, T0]:
first, second = pair
return (second, first)
Here, T0
and T1
are two independent type variables. They can take on different types, but they may also be the same. For example, maybe you pass in a pair of Booleans.
By default, type variables can be materialized by any type. However, sometimes you want to express type relationships that are constrained to one of just a few types or bounded as a subtype of some type. You can do so by adding a condition after the type variable, separated by a colon. You’ll use the following syntax:
>>> def free[T](argument: T) -> T: ...
>>> def constrained[T: (int, float, complex)](argument: T) -> T: ...
>>> def bounded[T: str](argument: T) -> T: ...
In the examples so far, you’ve used the free syntax. This implies that T
can be any type. In constrained()
, T
is a type variable that can only be int
, float
, or complex
. You express this by using a literal tuple of types. In bounded()
, T
can be str
or any subclass of str
.
You can also define generic classes. As for generic functions, you declare any type variables in square brackets. The following example implements a simple stack based on list
:
# generic_stack.py
class Stack[T]:
def __init__(self) -> None:
self.stack: list[T] = []
def push(self, element: T) -> None:
self.stack.append(element)
def pop(self) -> T:
return self.stack.pop()
Here you’ve added [T]
to the class name. Then, you can use the type variable when you’re annotating the method parameters and return types inside the class. In practice, you can instantiate stacks containing specific types. Next, you’ll create a stack of integers:
>>> from generic_stack import Stack
>>> numbers = Stack[int]()
>>> numbers.push(3)
>>> numbers.push(12)
>>> numbers.stack
[3, 12]
>>> numbers.pop()
12
By adding [int]
when you instantiate Stack
, you’re telling the type checker that your data structure will be composed of integers. It can then warn you if other types may end up inside your stack.
Observe that when you pop a number off your stack, you get the last number that was pushed. This is often referred to as last-in, first-out (LIFO). The idea is reminiscent of a stack of plates that you may have in your kitchen cupboard. Stacks are useful in many different computer algorithms as well.
You can also use a new syntax for type aliases. A type alias is a name that you give to a specific type, either to simplify working with a nested data type, or to more explicitly describe your data type.
You can now use type
to declare type aliases:
>>> type Person = tuple[str, int]
>>> type ListOrSet[T] = list[T] | set[T]
Here Person
is a type represented by a tuple consisting of a string and an integer. ListOrSet
is a generic type alias, which a list or a set will represent. You can later annotate an argument with something like ListOrSet[int]
which would require the argument to be either a list of integers or a set of integers.
You can learn more about the new syntax for type variables and see more practical examples in Python 3.12 Preview: Static Typing Improvements and PEP 695.
Support for the Linux perf
Profiler
A profiler is a tool for monitoring and diagnosing the performance of your scripts and programs. By profiling your code, you’ll get accurate measurements that you can use to tune your implementation.
Python has long supported profiling with tools like timeit
and cProfile
in the standard library. Additionally, third-party tools like line-profiler
, Pyinstrument, and Fil provide other capabilities.
The perf
profiler is a profiler built into the Linux kernel. While it’s only available on Linux, it’s a popular and powerful tool that can provide information about everything from hardware events and system calls to running library code.
Until now, running perf
hasn’t worked well with Python. The CPython interpreter is the program that usually runs your Python code. Python code is evaluated with a C function named _PyEval_EvalFrameDefault()
, and a typical profile of a Python program will only show that it spent most of the time in _PyEval_EvalFrameDefault()
.
Python 3.12 adds proper support for perf
and gives it the ability to monitor Python functions through a technique called trampoline instrumentation. This allows individual Python functions to show up in profiling reports that perf
produces:
Samples: 10K of event 'cycles', Event count (approx.): 34217160648
- 100.00% 10432
- 99.16% /home/realpython/python-custom-build/bin/python3.12
__libc_start_main
- Py_BytesMain
- 99.89% pymain_run_python.constprop.0
_PyRun_AnyFileObject
_PyRun_SimpleFileObject
run_mod
run_eval_code_obj
PyEval_EvalCode
py::<module>:/home/realpython/project/script.py
_PyEval_EvalFrameDefault
PyObject_Vectorcall
py::main:/home/realpython/project/script.py
_PyEval_EvalFrameDefault
- PyObject_Vectorcall
+ 66.61% py::slow_function:/home/realpython/project/script.py
+ 33.39% py::fast_function:/home/realpython/project/script.py
+ 0.84% /proc/kcore
If you’re running Linux and are interested in profiling your code, then you should give perf
a try. For more information, including how to set up perf
on your system and profile your code, check out Python 3.12 Preview: Support for the Linux Perf Profiler.
Other Pretty Cool Features
Until now, you’ve seen the biggest changes and improvements in Python 3.12. However, there’s much more to explore. In this section, you’ll take a look at some of the new features that may be sneaking under the headlines. They include more internal changes to the interpreter, a new typing feature, and new functions for grouping iterables and listing files.
One GIL Per Subinterpreter
Python has a global interpreter lock (GIL) that simplifies a lot of internal code in the interpreter. At the same time, the GIL imposes some restrictions on running concurrent Python code. In particular, only one thread is usually allowed to run at a time, which makes parallel processing cumbersome.
Over time, there’ve been several initiatives to remove the GIL from the language. Recently, PEP 703 and the nogil project have caused a lot of buzz, and there’s a roadmap for removing the GIL from Python.
A related initiative is seeing the light in Python 3.12. PEP 684 describes a per-interpreter GIL. The Python interpreter is the program that executes your Python scripts and programs. It’s possible to spawn new interpreters, so-called subinterpreters, but you can only do so in extension modules through the C API.
Note: There’s ongoing work on a new standard library module named interpreters
that will expose subinterpreters to Python code. This is described in PEP 554 and in Real Python’s guide to subinterpreters.
Having a per-interpreter GIL means that there’s a separate interpreter lock for each subinterpreter. This opens up the possibility of new and efficient ways of doing parallelism in Python that take better advantage of the multiple cores found in modern computers. One such interesting model is communicating sequential processes (CSP) which has inspired concurrency in several languages, including Erlang and Go.
To achieve a per-interpreter GIL, the core developers have refactored several parts of the CPython internals. Python has both global state storage and per-interpreter storage. In this initiative, much of what was previously stored as global state is now stored for each interpreter.
Probably, you won’t notice this change when running Python 3.12. None of the changes are exposed to regular users of Python. Instead, they’re laying the groundwork for new ways to implement parallelism in the future.
You can learn more about subinterpreters, including the plans for making them more accessible in Python 3.13, in Python 3.12 Preview: Subinterpreters.
Immortal Objects
The introduction of immortal objects into Python is another internal feature that improves the CPython interpreter and prepares the way for new developments in the future.
For efficiency, several objects in Python are singletons. For example, there’s only one None
object during program execution, independent of how many times you refer to None
in your code. This optimization is possible because None
is immutable. It’ll never change its value.
It turns out that while None
is immutable as seen from Python’s perspective, the None
object handled by the CPython interpreter does change. The interpreter represents every object in a struct that includes the object’s reference count in addition to the object’s data. The reference count changes every time your code references an object.
You can check an object’s reference count for yourself:
>>> import sys
>>> a = 3.12
>>> sys.getrefcount(a)
2
>>> b = a
>>> sys.getrefcount(a)
3
>>> del b
>>> sys.getrefcount(a)
2
You use sys.getrefcount()
to inspect the reference count of an object. Here a
refers to the float
object 3.12
. You can see that the reference count increases when b
refers to the same object. Likewise, the reference count decreases when the name b
is deleted.
Note: The first call to sys.getrefcount()
returns 2
, while you’ve only created one reference to a
. However, passing a
as an argument to getrefcount()
creates the second reference. In general, the count returned will often be one higher than what you expect.
The reference count is important for Python’s memory management. CPython uses a garbage collector that removes objects from memory once their reference count hits zero.
Immortal objects are objects which are truly immutable, including inside the interpreter. This means that their reference count doesn’t change. Immortal objects are identified by having their reference count set to a special flag. This is done to keep backwards compatibility of the C-API and to mostly treat immortal objects the same as regular objects.
You can see this mechanism if you look at the reference count of an immortal object:
>>> import sys
>>> sys.getrefcount(None)
4294967295
>>> a = None
>>> sys.getrefcount(None)
4294967295
At first, you get the impression that None
is referenced more than four billion times. However, 4,294,967,295 is a special value indicating that None
is an immortal object. Note that the number doesn’t change when you create a new reference to None
.
The special flag isn’t chosen at random. It corresponds to the hexadecimal number FFFFFFFF, which is the largest integer that a 32-bit system can natively represent:
>>> hex(sys.getrefcount(None))
'0xffffffff'
In other words, the immortal flag value is big enough so that the interpreter won’t reach it as a reference count in normal use. And it has a representation that’s efficient to test against.
While immortal objects are an implementation detail in the interpreter, they bring a couple of advantages:
- Truly immutable objects have better cache behavior. In certain code, it’ll be more efficient to work with singletons like
None
,True
,False
, and so on. - Immortal objects don’t need the GIL’s protection. Therefore, they’ll work well with the per-interpreter GIL that you learned about earlier. They also simplify some of the work towards removing the GIL from Python.
You can learn more about immortal objects in PEP 683. Furthermore, Introducing Immortal Objects for Python provides some context and background for the implementation, while Understanding Immortal Objects in Python 3.12 discusses the implementation itself.
Override Decorator for Explicit Inheritance
Python is an object-oriented language with good support for working with classes and inheritance. There’s a close connection between Python’s classes and its static typing system, as each class also defines a type that you can use in type hints.
One of the new typing features in Python 3.12 is @override
. You can use this decorator to mark methods in subclasses that override methods in the parent class. This feature is partly inspired by similar mechanisms in other languages like Java and C++.
Using @override
can help you avoid certain kinds of bugs, especially when you refactor your code. A static type checker can warn you in the following cases:
- When you rename a method but forget to rename the corresponding methods in subclasses
- When you misspell the name of a subclass method that’s supposed to override a parent method
- When you add a new method to a class that accidentally overrides an existing method in a subclass
Until now, type checkers had no way of knowing whether a method was meant to override another or not. In Python 3.12, @override
is added to typing
. On earlier versions of Python, you can import @override
from the third-party typing-extensions
library.
For an example of how you can use the new decorator, you’ll implement two classes representing bank accounts. BankAccount
will represent a basic bank account, while SavingsAccount
will subclass BankAccount
with some special features.
For simplicity, you’ll use data classes to define your bank account classes. Start with the general bank account class:
# accounts.py
import random
from dataclasses import dataclass
from typing import Self
def generate_account_number() -> str:
"""Generate a random eleven-digit account number"""
account_number = str(random.randrange(10_000_000_000, 100_000_000_000))
return f"{account_number[:4]}.{account_number[4:6]}.{account_number[6:]}"
@dataclass
class BankAccount:
account_number: str
balance: float
@classmethod
def from_balance(cls, balance: float) -> Self:
return cls(generate_account_number(), balance)
def deposit(self, amount: float) -> None:
self.balance += amount
def withdraw(self, amount: float) -> None:
self.balance -= amount
Your bank account has two regular metods: .deposit()
and .withdraw()
. Additionally, you add an alternative constructor, .from_balance()
, that can create a new bank account given an initial balance. When you do this, the generate_account_number()
utility function generates a random account number.
You can use this bank account as follows:
>>> from accounts import BankAccount
>>> account = BankAccount.from_balance(1000)
>>> account.withdraw(123.45)
>>> account
BankAccount(account_number='2801.83.60525', balance=876.55)
Here, you start a new bank account with an initial balance of $1000. After withdrawing $123.45, you have $876.55 left in the account.
Next, you’ll add a savings account. Compared to the regular bank account, the savings account can accrue interest. Additionally, the bank provide a small bonus to the account holder by rounding withdrawals bigger than $100 down to the nearest dollar amount.
You implement SavingsAccount
as a subclass of BankAccount
and mark the methods that override the parent methods. Make sure to add override
to your imports:
# accounts.py
import random
from dataclasses import dataclass
from typing import Self, override
# ...
@dataclass
class SavingsAccount(BankAccount):
interest: float
def add_interest(self) -> None:
self.balance *= (1 + self.interest / 100)
@classmethod
@override
def from_balance(cls, balance: float, interest: float = 1.0) -> Self:
return cls(generate_account_number(), balance, interest)
@override
def withdraw(self, amount: float) -> None:
self.balance -= int(amount) if amount > 100 else amount
First, you add .interest
as a new attribute and .add_interest()
as a new method for adding interest to the savings account. Next, you update .from_balance()
to support specifying the interest. Since this constructor overrides the corresponding method in BankAccount
, you mark it with @override
.
Note: @override
doesn’t have any runtime effects, except that it adds an .__override__
attribute on the method. When combining @override
and @classmethod
, you should specify @override
last.
This doesn’t matter to the static type checker, but it allows the consistent runtime semantics setting .__override__
.
You also override .withdraw()
to add a small bonus for account holders. If a customer withdraws more than $100, then you’ll only subtract the amount rounded down to the nearest dollar from the balance:
>>> from accounts import SavingsAccount
>>> savings = SavingsAccount.from_balance(1000, interest=2)
>>> savings.withdraw(123.45)
>>> savings
SavingsAccount(account_number='5283.78.04835', balance=877, interest=2)
Here, you’ve only subtracted $123 from the balance even though you withdrew $123.45. This shows that SavingsAccount
uses the overridden method. You can continue the example to test the new method:
>>> savings.add_interest()
>>> savings
SavingsAccount(account_number='5283.78.04835', balance=894.54, interest=2)
>>> savings.withdraw.__override__
True
After adding interest to the account, you also confirm that you’ve tagged .withdraw()
as an override. Even though you’ve added the .__override__
attribute, this won’t have any effect. Instead, you should use a static type checker to help you catch errors in overriding methods. Have a look at Python 3.12: Static Typing Improvements for more details on how you can activate your type checker.
If you’re interested in catching similar issues at runtime, then check out the third-party library overrides
.
Calendar Constants for Days and Months
Python’s calendar
module is one of the batteries that’s been included in the language for a long time. Usually, when you’re working with dates, you’ll use the datetime
module, which provides date
and datetime
classes that represent dates and dates with timestamps, respectively.
With calendar
, you can bridge the gap between the technical use of datetime
and the often more user-friendly representations of a calendar. For example, you can use calendar
to quickly show a calendar in your terminal:
$ python -m calendar 2023 10
October 2023
Mo Tu We Th Fr Sa Su
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
By running the module with -m
, you display a calendar. In the example above, you show October 2023 and observe that October 2 is a Monday.
You can also use calendar
in your own code if you need to build custom calendars. In the new Python version, you have more support for working with weekdays and months. First, look at what’s been available for a long time, like lists of weekday and month names:
>>> import calendar
>>> ", ".join(calendar.month_abbr)
', Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec'
>>> calendar.weekday(2023, 10, 2)
0
>>> calendar.day_name[0]
'Monday'
You first list all months, abbreviated. Note that the list of months has thirteen elements, where the first one is an empty string. This is done so that indices match the usual month numbers, where January corresponds to 1 and so on.
In the second example, you use calendar.weekday()
to calculate which weekday October 2 falls on in 2023. You then look up that answer—0
—in the list of day names to confirm that October 2 is a Monday.
Everything that you’ve seen so far is possible on older versions of Python. In Python 3.12, you’ll find two minor additions, namely the enumerations Day
and Month
. An enumeration is a collection of constant values in a common namespace. These make it more convenient to handle weekdays and months.
Enumeration members are constants. You can access them in a few different ways:
>>> calendar.Month(10)
calendar.OCTOBER
>>> calendar.Month.OCTOBER
calendar.OCTOBER
>>> calendar.Month["OCTOBER"]
calendar.OCTOBER
First, you ask for the tenth month by calling the enumeration. Then, you ask specifically for the .OCTOBER
constant by using dot notation. Finally, you look up "OCTOBER"
as if it were a key on Month
.
All three of these return calendar.OCTOBER
, which is a constant with the value 10
. You can use the enumeration and its value interchangably:
>>> october = calendar.Month(10)
>>> october
calendar.OCTOBER
>>> october.name
'OCTOBER'
>>> october.value
10
>>> october + 1
11
>>> calendar.month_abbr[october]
'Oct'
You can explicitly look up the name and value of the enumeration by using .name
and .value
, respectively. In addition, you can make calculations or use the enumeration in expressions as if it were a regular value.
Day
works similarly to Month
, but it contains weekday constants. Monday is represented by 0
, Tuesday by 1
, and so on up to Sunday
, which corresponds to 6
. In Python 3.12, the weekday()
function returns a Day
enumeration:
>>> release_day = calendar.weekday(2023, 10, 2)
>>> release_day
calendar.MONDAY
>>> release_day.name
'MONDAY'
>>> release_day.value
0
Here, you represent release_day
with a Day
enumeration. As above, you can peek at its name and value.
Note: Month
starts numbering the months at 1
to be consistent with normal use, where January is month number one. Day
is different and starts numbering at 0
. Weekdays aren’t numbered in everyday use, so the numbering scheme aims to be consistent with the datetime
module and the usual zero-indexed nature of Python.
One final feature of enumerations is that you can iterate over them. In the next example, you loop over Month
to create a quick sales report:
>>> sales = {1: 5, 2: 9, 3: 6, 4: 14, 5: 9, 6: 8, 7: 15, 8: 22, 9: 20, 10: 23}
>>> for month in calendar.Month:
... if month in sales:
... print(f"{month.value:2d} {month.name:<10} {sales[month]:2d}")
...
1 JANUARY 5
2 FEBRUARY 9
3 MARCH 6
4 APRIL 14
5 MAY 9
6 JUNE 8
7 JULY 15
8 AUGUST 22
9 SEPTEMBER 20
10 OCTOBER 23
Each month
is an enumeration constant. Note how you can treat the constant like a regular integer when you check if month
is a key in sales
and when you use month
as an index to sales
.
Note: You could’ve also used the enumeration constants as keys in your sales
dictionary. For convenience, each month constant is available as an attribute directly on calendar
:
>>> sales = {
... calendar.JANUARY: 5,
... calendar.FEBRUARY: 9,
... calendar.MARCH: 6,
... # ...
... }
This can be helpful in making literals more readable.
Similarly to months, you can iterate over weekdays:
>>> ", ".join(day.name for day in calendar.Day)
'MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY'
In this example, you combine the names of all weekdays into a string. However, if you need to work with the weekday names, using calendar.day_name
or calendar.day_abbr
might still be the better option. These give you localized names, which means that they’ll be translated into your local language.
itertools.batched()
: Group Items in an Iterable
The itertools
standard library module includes many powerful functions for working with and manipulating iterables. For example, you can calculate all the combinations of two weekdays:
>>> import itertools
>>> list(itertools.combinations(["Mo", "Tu", "We", "Th", "Fr", "Sa", "Su"], 2))
[('Mo', 'Tu'), ('Mo', 'We'), ('Mo', 'Th'), ('Mo', 'Fr'), ('Mo', 'Sa'),
('Mo', 'Su'), ('Tu', 'We'), ('Tu', 'Th'), ('Tu', 'Fr'), ('Tu', 'Sa'),
('Tu', 'Su'), ('We', 'Th'), ('We', 'Fr'), ('We', 'Sa'), ('We', 'Su'),
('Th', 'Fr'), ('Th', 'Sa'), ('Th', 'Su'), ('Fr', 'Sa'), ('Fr', 'Su'),
('Sa', 'Su')]
This lists the twenty-one ways to pair up two weekdays. There are many other functions available for you to play with to work efficiently with iterables.
A frequently requested function has been batched()
, which can group the elements of an iterator into batches of a given size. Until now, you could implement batched()
yourself based on the official itertools
recipes, or you could rely on the third-party more-itertools
library.
In Python 3.12 batched()
has been included in itertools
, making it more readily available. The function returns an iterator that yields tuples of the given length. The final batch may be smaller than the others. In the following example, you group the first ten integers into batches of four numbers:
>>> import itertools
>>> for batch in itertools.batched(range(10), n=4):
... print(batch)
...
(0, 1, 2, 3)
(4, 5, 6, 7)
(8, 9)
The first two batches consist of four numbers each. However, the last batch contains only the two numbers left over in the range.
In many cases, you’ll work with iterables that you know divide cleanly into batches. Picking up from the previous section, you may want to group the months into quarters. You can use batched()
as follows:
>>> import calendar
>>> import itertools
>>> for quarter in itertools.batched(calendar.Month, n=3):
... print([month.name for month in quarter])
...
['JANUARY', 'FEBRUARY', 'MARCH']
['APRIL', 'MAY', 'JUNE']
['JULY', 'AUGUST', 'SEPTEMBER']
['OCTOBER', 'NOVEMBER', 'DECEMBER']
You iterate over the new Month
enumeration and group the months into batches of three months. You use a list comprehension to print out the name of each month. Each list in your output represents the months in a quarter.
The new batched()
function adds another powerful tool to the itertools
library. If you work with iterables, you should make sure that you’re familiar with the possibilities in itertools
.
Path.walk()
: List Files and Subdirectories
Another useful module in the standard library is pathlib
. You use pathlib
to work with paths in your file system, and you can use it read and write files efficiently:
>>> from pathlib import Path
>>> names = ["Brett", "Emily", "Gregory", "Pablo", "Thomas"]
>>> council_path = Path("council.txt")
>>> council_path.write_text("\n".join(names), encoding="utf-8")
32
Here, you create a path to a file named council.txt
in your working directory. Then you write a list of names separated by newlines to that file.
While a Path
object has many methods for manipulating paths and creating new paths, it only has limited support for listing files and directories. You can use .glob()
, but this method really shines when you’re looking for files and directories with names matching a specific pattern:
>>> for path in Path.cwd().glob("*.txt"):
... print(path)
...
/home/realpython/council.txt
You use .glob()
to list all the files ending with the .txt
suffix in the current working directory (cwd
). You can use the recursive counterpart, .rglob()
, to list and filter all the files in subdirectories as well.
In Python 3.12, you can use the new .walk()
method to work with files and directories. Assume that you have some information about musicians stored in the following file hierarchy:
musicians/
│
├── trumpet/
│ ├── armstrong.txt
│ └── davis.txt
│
├── vocal/
│ └── fitzgerald.txt
│
└── readme.txt
First, use .rglob()
to list all files and directories recursively:
>>> for path in sorted(Path.cwd().rglob("*")):
... print(path)
...
/home/realpython/musicians
/home/realpython/musicians/readme.txt
/home/realpython/musicians/trumpet
/home/realpython/musicians/trumpet/armstrong.txt
/home/realpython/musicians/trumpet/davis.txt
/home/realpython/musicians/vocal
/home/realpython/musicians/vocal/fitzgerald.txt
This gives you one path for each file or directory in your hierarchy. In general, the order of paths isn’t deterministic when you use .glob()
and .rglob()
. One way to keep your file listings reproducible is to sort them with sorted()
.
The new .walk()
method works slightly differently, as it focuses on directories:
>>> for path, directories, files in Path.cwd().walk():
... print(path, directories, files)
...
/home/realpython ['musicians'] []
/home/realpython/musicians ['trumpet', 'vocal'] ['readme.txt']
/home/realpython/musicians/trumpet [] ['armstrong.txt', 'davis.txt']
/home/realpython/musicians/vocal [] ['fitzgerald.txt']
Note that .walk()
yields tuples of three elements. The path will always refer to a directory. The last two elements are lists of subdirectories and files directly inside that directory, respectively. You can use the top_down
parameter to control the order in which directories are listed.
The new method is based on os.walk()
. The main difference is that the new .walk()
yields Path
objects.
So, Should You Upgrade to Python 3.12?
You’ve seen the coolest new features and improvements in Python 3.12. The next question might be whether you should upgrade to the new version, and if so, when should you upgrade?
As always, the not-so-helpful answer is it depends!
A small start is to install Python 3.12 alongside your current system. This allows you to start playing with the new features when doing local development. This comes with minimal risk because any bugs that you encounter should have limited effect. At the same time, you get to go ahead and take advantage of the improved error messages and optimizations to the interpreter.
You should be more careful with updating any production environment that you control, as the consequences of bugs and errors are more severe in that context. All new Python releases are well tested in the beta phase. Still, it might be a good idea to wait for the first few bugfix releases before switching.
One possible problem with updating to Python 3.12 is that you depend on a third-party library that’s not ready for the new version. In particular, packages that use C-extensions must be compiled specially for version 3.12, and this may take some time. Luckily, this is less of a problem than it used to be, as more and more package maintainers update their packages ahead of the release.
So far, you’ve considered when you can start using the new interpreter. Another important question is when you can start to take advantage of the updated syntax and new features in the language. If you’re maintaining a library that needs to support older versions of Python, then you can’t use the new type variable syntax or the improvements to f-strings. You need to stick with code compatible with the older versions.
The situation is different if you’re developing an application where you control the environment that it’s running in. In that case, you can upgrade the environment to Python 3.12 as soon as your dependencies are available, and then start using the new syntax.
Conclusion
A new version of Python is always a great occasion to celebrate your favorite language and all the volunteers that put time and effort into its development. Thanks to the work of so many developers, Python 3.12 brings several improvements to the table.
In this tutorial, you’ve seen new features and improvements, like:
- Better error messages with helpful suggestions and guidance
- More expressive f-strings that are backed by Python’s PEG parser
- Optimizations, including inlined comprehensions, to help Python run faster
- A new syntax for type variables that you use to annotate generics
- Support for the powerful
perf
profiler on Linux
While you may not be able to take advantage of all these features right away, you can install Python 3.12 and play with them. If you’re thirsty for more information about the new release, then check out the comprehensive podcast episode and these tutorials, which focus on one new feature each:
- Python 3.12 Preview: Ever Better Error Messages
- Python 3.12 Preview: Support For the Linux perf Profiler
- Python 3.12 Preview: More Intuitive and Consistent F-Strings
- Python 3.12 Preview: Subinterpreters
- Python 3.12 Preview: Static Typing Improvements
It’s also a good time to start testing your existing code on Python 3.12 to make sure that it’s ready for the future.
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: What's New in Python 3.12