Python 3.12 Preview: More Intuitive and Consistent F-Strings

F-Strings Had Some Limitations Before Python 3.12

You can use Python’s f-strings for string formatting and interpolation. An f-string is a string literal prefixed with the letter F, either in uppercase or lowercase. This kind of literal lets you interpolate variables and expressions, which Python evaluates to produce the final string.

F-strings have gained a lot of popularity in the Python community since their introduction in Python 3.6. People have embraced them with enthusiasm, turning them into a standard in modern Python programming. The reasons? They provide a concise and readable syntax that allows you to format strings and interpolate variables and expressions without needing the .format() method or the old-style string formatting operator (%).

However, to introduce f-strings, the CPython core development team had to decide how to implement them, especially how to parse them. As a result, f-strings came with their own parsing code. In other words, CPython has a dedicated parser for f-strings. Because of this, the f-string grammar isn’t part of the official Python grammar.

From the core developers’ point of view, this implementation decision implies considerable maintenance costs because they have to manually maintain a separate parser. On the other hand, not being part of the official grammar means that other Python implementations, such as PyPy, can’t know if they’ve implemented f-strings correctly.

However, the most important burden is on the user’s side. From the user’s perspective, the current f-string implementation imposes some limitations:

• Reusing quotes or string delimiters isn’t possible.
• Embedding backslashes isn’t possible, which means you can’t use escape characters.
• Adding inline comments is forbidden.
• Nesting of f-strings is limited to the available quoting variations in Python.

PEP 536 lists these limitations. However, exploring them with a few small examples will help you understand how they can affect your use of f-strings in your Python code.

First, say that you need to interpolate a dictionary key in an f-string. If you try the following code, then you’ll get an error:

>>> employee = {
...     "name": "John Doe",
...     "age": 35,
...     "job": "Python Developer",
... }

>>> f"Employee: {employee["name"]}"
  File "<stdin>", line 1
    f"Employee: {employee["name"]}"
                           ^^^^
SyntaxError: f-string: unmatched '['

In this example, you try to interpolate the employee name in your f-strings. However, you get an error because the double quotes around the "name" key break the string literal. To work around this, you need to use a different type of quotation mark to delimit the key:

>>> f"Employee: {employee['name']}"
'Employee: John Doe'

Now you use double quotes for the f-string and single quotes for the dictionary key. Your code works now, but having to switch quotes can get annoying at times.

The second limitation of f-strings is that you can’t use backslash characters in embedded expressions. Consider the following example, where you try to concatenate strings using the newline (\n) escape sequence:

>>> words = ["Hello", "World!", "I", "am", "a", "Pythonista!"]

>>> f"{'\n'.join(words)}"
  File "<stdin>", line 1
    f"{'\n'.join(words)}"
                         ^
SyntaxError: f-string expression part cannot include a backslash

In this example, you get a SyntaxError because f-strings don’t allow backslash characters inside expressions delimited by curly brackets. Again, you can implement a work-around, but it’s not exactly pretty:

>>> word_lines = "\n".join(words)

>>> f"{word_lines}"
'Hello\nWorld!\nI\nam\na\nPythonista!'

>>> print(f"{word_lines}")
Hello
World!
I
am
a
Pythonista!

In this example, you run the string concatenation, store the result in a variable, and finally have the f-string interpolate that variable’s content. This approach avoids the backslash issue, but it feels like something is wrong with f-strings. Why don’t they allow all valid Python expressions?

Another limitation of f-strings is that they don’t allow you to insert comments in embedded expressions. This limitation may seem superfluous, but in some cases, a good comment can help other developers better understand your code:

>>> employee = {
...     "name": "John Doe",
...     "age": 35,
...     "job": "Python Developer",
... }

>>> f"""Storing employee's data: {
...     employee['name'].upper()  # Always uppercase name before storing
... }"""
  File "<stdin>", line 3
    }"""
        ^
SyntaxError: f-string expression part cannot include '#'

In this example, you use triple quotes to build a string that spans multiple lines. When you try to add an inline comment beside the interpolated expression, you get a SyntaxError. This behavior seems weird because you can add comments in a normal Python expression wrapped in brackets. So, embedded expressions in f-strings don’t work as normal Python expressions do.

Finally, f-strings have another limitation. The number of nesting levels in an f-string is limited by the available string delimiters in Python, which are ", ', """, and '''. In the following example, you run into the issue:

>>> f"""{
...     f'''{
...         f"{f'{42}'}"
...     }'''
... }"""
'42'

>>> f"""{
...     f'''{
...         f"{f'{f"{42}"}'}"
...     }'''
... }"""
  File "<stdin>", line 1
    (f"{f'{f"{42}"}'}")
             ^
SyntaxError: f-string: f-string: unterminated string

Even though nesting f-strings may not have many use cases, you’ll probably find some interesting ones. If you need an additional level of nesting in a specific use case, then you’re out of luck because you can’t reuse quotes.

It’s important to note that only triple-quoted f-strings can span multiple lines. However, this isn’t a big issue because that’s the expected behavior of Python strings, where only triple-quoted strings can occupy multiple lines.

While f-strings are pretty cool, and most Python developers love them, all these limitations make them feel incomplete and inconsistent with the general behavior of Python itself. Fortunately, Python is constantly improving, and the next version, 3.12, is lifting these limitations to make f-strings even better.

Python 3.12 Brings Syntactic Formalization of F-Strings

You can thank PEP 701 for lifting the restrictions and limitations of Python’s f-strings. One of the goals of this PEP is to guarantee that f-strings can include all valid Python expressions, including those with:

• Backslashes
• Unicode escape sequences
• Multiline expressions
• Inline comments
• The same type of quote as the containing f-string

The PEP proposed a syntactic formalization of f-strings along with a new implementation that takes advantage of the PEG parser that joined the language in Python 3.9. With Python 3.12, the f-string grammar will become part of the official Python grammar to remove the current inconsistency.

The new implementation provides the following general benefits, among others:

• Allows taking advantage of the PEG parser’s power for present and future improvements
• Lifts the current f-string limitations and turns them into features
• Reduces the learning burden for f-string literals
• Reduces the maintenance cost by removing the dedicated parser

Lifting the restrictions and limitations makes f-strings more intuitive, straightforward, flexible, and consistent with the general Python syntax and behavior. This will have a direct impact on the user experience of Python developers.

Additionally, PEP 701 suggests that the tokenize module from the standard library will be adapted to work with the new f-string syntax formalization and its implementation. So, tools like linters, editors, and IDEs can take advantage of this formalization, avoiding the need to implement their own f-string parser.

PEP 701 also aims to redefine f-strings with an emphasis on clearly separating the pure string and expression components. The latter is the part of an f-string that you embed in a pair of curly brackets ({...}).

Finally, it’s important to note that PEP 701 doesn’t introduce semantic changes into f-strings. Therefore, existing code that uses f-strings will continue to work. The new f-string implementation will be fully backward-compatible.

All these benefits and advantages sound exciting. Are you ready to give them a try? Yes? Then keep reading. But hold on, before jumping to action, you need to install a pre-release version of Python 3.12. The 3.12.0b4 version has just arrived, so go for it!

Embedded Expressions Can Reuse Quotes

In the new f-string implementation, the embedded expression component can contain any Python expression, including a string literal that uses the same type of quotation marks as the containing f-string.

In Python 3.12, you can now do the following:

>>> employee = {
...     "name": "John Doe",
...     "age": 35,
...     "job": "Python Developer",
... }

>>> f"Employee: {employee["name"]}"
'Employee: John Doe'

In this example, you use double quotes to define the f-string and delimit the employee dictionary key. Now you don’t have to switch to a different type of quote when you use string literals inside embedded expressions.

Even though this new behavior seems cool and consistent, some people think that reusing quotes within the same f-string is confusing and hard to read. They might be right. Reusing quotes violates the Python rule that a matching pair of quotes delimits a string. However, focusing on separating the pure string part from the embedded expression part can help with readability.

In this regard, the authors of PEP 701 say that:

We believe that forbidding quote reuse should be done in linters and code style tools and not in the parser, the same way other confusing or hard-to-read constructs in the language are handled today. (Source)

This statement makes sense. In Python, you’ll find many things that you can do. However, best practices and style recommendations might suggest avoiding them in your code. So, if you find reusing quotation marks unreadable or confusing, then stick to the old practice of using different quotation marks in f-string literals. They’ll work the same.

Backslashes Now Allowed in F-Strings

The lack of support for backslashes inside expressions in f-strings is another issue in Python 3.11 and lower. In Python 3.12, this issue is resolved. Now your f-strings can contain backslashes as needed:

>>> words = ["Hello", "World!", "I", "am", "a", "Pythonista!"]

>>> f"{'\n'.join(words)}"
'Hello\nWorld!\nI\nam\na\nPythonista!'

>>> print(f"{'\n'.join(words)}")
Hello
World!
I
am
a
Pythonista!

Being able to include backslashes in expressions embedded in your f-string literals is a great forward step. It saves you the effort of finding alternative work-arounds that may make your code less concise, direct, and consistent.

In the example above, you don’t have to define a new variable to store the intermediate step of concatenating your list of words. You can do this directly in the f-string.

Comments Accepted in Multiline Expressions

The new implementation of f-strings also allows multiline expressions and inline comments in those expressions. This feature opens the possibility for you to comment on your code when you need to clarify some aspects that may not be obvious to other developers:

>>> employee = {
...     "name": "John Doe",
...     "age": 35,
...     "job": "Python Developer",
... }

>>> f"""Storing employee's data: {
...     employee['name'].upper()  # Always uppercase name before storing
... }"""
"Storing employee's data: JOHN DOE"

In Python 3.12’s f-strings, you can define expressions that span multiple physical lines. Every line can include an inline comment if needed. The hash character (#) won’t break your f-strings anymore. This new behavior is consistent with the behavior of expressions that you enclose in brackets outside f-strings.

Arbitrary Levels of F-String Nesting Possible

As a result of allowing quote reuse, the new f-string implementation allows arbitrary levels of nesting. The usefulness of this feature is limited because many levels of nesting may make your code hard to read and understand. However, you can get it done:

>>> f"{
...     f"{
...         f"{
...             f"{
...                 f"{
...                     f"Deeply nested f-string!"
...                 }"
...             }"
...         }"
...     }"
... }"
'Deeply nested f-string!'

Wow! That was a lot of nesting. Before the new f-string implementation, there was no formal limit on how many levels of nesting you could have. However, the fact that you couldn’t reuse string quotes imposed a natural limit on the allowed levels of nesting in f-string literals.

In the above example, you can also note that the new f-string implementation allows newlines in literals within the curly brackets of embedded expressions. This behavior is consistent with the regular Python behavior that lets you wrap an expression in a pair of brackets—typically parentheses—to make it span multiple lines.

Better Error Messages Available for F-Strings Too

As you’ve learned, Python 3.12’s new f-string implementation removes several limitations on how you can use f-strings in real-life code. That’s cool! Now those limitations have turned into new features. But there’s even more to it.

Because Python now uses a PEG parser to parse the new f-string grammar, you get an additional, remarkable benefit. Yes, you get better error messages for f-strings too.

The Python development team has put a lot of work into improving Python error messages after introducing the PEG parser. Previously, these enhanced error messages weren’t available for f-strings because they didn’t use the PEG parser.

So, before Python 3.12, the error messages related to f-strings were less specific and clear. As an example, compare the error messages that the following f-string produces in 3.11 vs 3.12:

>>> # Python 3.11
>>> f"{42 + }"
  File "<stdin>", line 1
    (42 + )
          ^
SyntaxError: f-string: invalid syntax

>>> # Python 3.12
>>> f"{42 + }"
  File "<stdin>", line 1
    f"{42 + }"
          ^
SyntaxError: f-string: expecting '=', or '!', or ':', or '}'

The error message in the first example is generic and doesn’t point to the exact location of the error within the offending line. Additionally, the expression is in parentheses, which add noise to the problem because the original code doesn’t include parentheses.

In Python 3.12, the error message is more precise. It signals the exact location of the problem in the affected line. Additionally, the exception message provides some suggestions that might help you fix the issue.

Python 3.12’s F-Strings Still Have Some Limitations

The new f-string implementation doesn’t remove some current limitations of f-string literals. For example, the rules around using colons (:), exclamation points (!), and escaping curly braces with a backslash are still in place.

To use colons (:) and exclamation points (!) for a purpose other than string formatting, you need to surround the expression containing either of these symbols with a pair of parentheses. Otherwise, the f-string won’t work.

According to the authors of PEP 701, here’s why they didn’t remove the restriction:

The reason is that this [removing the restriction] will introduce a considerable amount of complexity [in the f-string parsing code] for no real benefit. (Source)

Apart from using these characters for string formatting, you’ll hardly find a suitable use case for them. Even the related examples in PEP 701 are useless:

>>> # Python 3.11
>>> f"Useless use of lambdas: { lambda x: x*2 }"
  File "<stdin>", line 1
    ( lambda x)
              ^
SyntaxError: f-string: invalid syntax

>>> # Python 3.12
>>> f"Useless use of lambdas: { lambda x: x*2 }"
  File "<stdin>", line 1
    f'Useless use of lambdas: { lambda x: x*2 }'
                                ^^^^^^^^^
SyntaxError: f-string: lambda expressions are not allowed without parentheses

In these examples, you use a colon as part of the syntax of a lambda function. This function is useless because there’s no way to call it. This code fails on both Python 3.11 and 3.12. However, note how the error message in 3.12 is way more precise and includes a suggestion for fixing the issue.

Following the suggested fix, to call a lambda function like the one above, you’ll have to enclose it in a pair of parentheses and then call the function with an appropriate argument:

>>> f"Useless use of lambdas: { (lambda x: x*2) }"
'Useless use of lambdas: <function <lambda> at 0x1010747c0>'

In this example, the pair of parentheses surrounding the lambda function works around the restriction of using a colon in an f-string. Now your f-string interpolates the resulting function object in the final string. Again, this may not be particularly useful. If you want to actually call the function, then you need to add a second pair of parentheses with a proper argument.

Finally, even though the new f-string implementation allows you to use backslashes for escaping characters, using backslashes to escape the curly brackets isn’t allowed:

>>> f"\{ 42 \}"
  File "<stdin>", line 1
    f"\{ 42 \}"
             ^
SyntaxError: unexpected character after line continuation character

In this example, you try to use a backslash to escape the curly brackets. The code doesn’t work, though. Here’s what the authors of PEP 701 say about this restriction:

We have decided to disallow (for the time being) using escaped braces (\{ and \}) in addition to the {{ and }} syntax. Although the authors of the PEP believe that allowing escaped braces is a good idea, we have decided to not include it in this PEP, as it is not strictly necessary for the formalization of f-strings proposed here, and it can be added independently in a regular CPython issue. (Source)

Reading between the lines, you can infer that this restriction may be lifted in upcoming patch releases of Python. For now, if you want to escape the curly brackets in an f-string literal, then you need to double them:

>>> f"{{ 42 }}"
'{ 42 }'

Doubling the curly brackets is the way to escape these characters in an f-string literal for now. However, this may change in the future.

Conclusion

Python 3.12 is available in beta versions for you to experiment with new features. This version brings a new f-string implementation that removes a few restrictions that have affected f-strings up to Python 3.11. With this tutorial, you’ve gotten up-to-date with the most relevant features of this new f-string implementation and learned how they can help you write better Python code.

In this tutorial, you’ve learned about:

• Limitations of f-strings in Python less than 3.12
• Advantages of formalizing the f-string syntax
• New capabilities of f-strings in Python 3.12
• Better error messages for the new f-string implementation

You’re now all caught up with the latest developments around Python’s f-strings. That’s cool!