A New Parser and Type Hints for Generics

00:00 In the previous lesson, I showed you what annotated type hints looked like. In this lesson, I’m going to talk about the new parser and the changes to generic hints.

00:10 Since what now feels like the dawn of time, Python has been based on what’s called an LL(1) parser. This means the parser reads a single character at a time and doesn’t backtrack.

00:22 This is one of the simpler types of parsers to build and maintain. For the most part, that’s a good thing for the kind folks who volunteer their time building Python.

00:32 That being said, it also means there are some corner cases. There are some situations where to get the syntax that you have and that you’re used to right now, some hairy stuff has to be done to work around the LL(1) parser. Work recently has been done on changing over the main parser to something called a PEG parser, or a Parsing Expression Grammar. The work’s now complete enough that it’s been included in Python 3.9. Moving to the PEG parser means some of those corner cases that I mentioned before can be worked around more easily. For Python 3.9, both the PEG parser and the LL(1) parser will be built-in. The PEG parser is used by default and constructs the exact same abstract syntax tree.

01:16 During testing of the 3.9 release, everything was parsed twice—once with PEG and once with LL(1)—to make sure that there were no bugs in the parser.

01:26 You can still use the LL(1) parser if you want, by using a -X parameter on the command line. -X oldparser will run the old way.

01:37 The LL(1) will be fully replaced in Python 3.10, so 3.9 is a transition. Make sure your code works, and if you happen to be one of those unlucky sorts who finds a bug in the parser—well, you can still use LL(1) and then report the bug. As of 3.10, once the new parser’s the only parser there, the maintainers can take advantage of this and start to introduce features that would not have been possible with the LL(1).

02:02 PEP 622, Structural Pattern Matching—which is kind of like a fancy version of a switch statement in other languages—will be one of those things that could be implemented.

02:14 In a previous lesson, I talked about the Annotated class being introduced to type hints. That’s not the only change. There’s also been one other simplification for generic type hints.

02:24 Recall that a type hint is a way of indicating the type a variable is using the basic types of the system: str, int, bool, float—whatever. That kind of stuff. Up until Python 3.9, you weren’t able to do this with generics, like lists and dictionaries.

02:42 You had to use a special class imported from the typing module. Python 3.9 removes this need. You can now use list directly as a type.

02:54 Here’s an example prior to Python 3.9. Importing capital-L List, I have a radius with a type of float and assigned a value of 3.9, and I have a radii, which is a List of floats.

03:07 Notice the use of the capital-L List class.

03:11 Now in Python 3.9, list can be used directly as a type hint. The radius example doesn’t change, but the radii example no longer needs the custom capital-L List.

03:22 It uses a small-L list directly.

03:25 The intention of this change is to reduce some confusion. You no longer run into the problem of “What kind of list is this?” and “Why is there a capital-L List?” and “Why would it work in this situation or that situation?” It should make any code with generics as type hints easier to read.

03:42 The __future__ module is a module available in Python that allows you to use features that are going to become part of the language in the future.

03:51 One of the features that was put in there was the delayed evaluation of type annotations. What this means is the evaluation doesn’t happen at load time, which means you can get away with doing some things that the parser would otherwise balk at.

04:04 This delayed evaluation was put into __future__ inside of Python 3.7 and 3.8. This means you can take advantage of the small-L list use of annotations by importing from __future__ in versions prior to Python 3.9.

04:23 Next up, I’ll be talking about the new prefix and suffix removal tool for strings, and graph sorting.