Exceptions and Task Groups

00:00 In the previous lesson, I showed you Tom’s Obvious Minimal Language. In this lesson, I’ll be covering two things: new language features for exceptions and an addition to the asyncio library.

00:12 There are two additions to exceptions in Python 3.11. The first is based on PEP 678—that just sounds made up—called Enriching Exceptions With Notes. A note in this case is a little message that you can add to an exception.

00:28 It’s particularly helpful if you are catching an exception to do some handling and then re-raising it. You can add a note to the exception before you raise it, giving more info to your user. Let me show you what this looks like.

00:44 Let me try something here. Dad joke for the win.

01:04 In this code, I’m raising a ValueError exception and then catching it. Once caught, I’m calling the new .add_note() method to add some information to the exception. When re-raise it, you can see the note at the bottom of the traceback.

01:21 The .add_note() method works on any Exception object, whether it has been raised or not.

01:32 You can add more than one note to an exception … and you can see all the notes in the .__notes__ attribute. The other new exceptional feature—somebody really should stop me—is exception groups.

01:50 Sometimes your error-handling code itself causes an exception. The output from this can be a bit confusing to the user. You may have seen errors that say *when handling this exception, that exception happened*, giving you multiple stack traces. Personally, this always slows me down trying to figure out what caused which in what order.

02:09 Python 3.11 has a new class of exceptions called the ExceptionGroup. It’s a way of grouping two or more exceptions together. In addition to grouping them, a new bit of syntax has been added to the language, allowing you to catch exceptions that are contained within groups.

02:25 Let’s go look at some examples. Okay, I’m in the REPL. Let’s raise an exception … and another one. Nothing new here. Now let me do the new thing. I’ll group them together.

02:55 You raise an ExceptionGroup just like an exception. To construct a group, you give it a group name—I creatively called mine "group"—and a list of the exceptions in the group, in this case, a ValueError and TypeError.

03:09 The traceback output has some nice little ASCII art showing you the contents of the group along with the stack information normally found in a traceback.

03:17 Let’s try out something else. Really, somebody should stop me.

03:32 Inside the block, I’m raising an ExceptionGroup like above.

03:40 Now to catch it, I’m using the new syntax. The except* syntax says I can catch exceptions that are inside of the group.

03:55 Let me do that again for the TypeError.

04:09 When the code runs, both of these exception conditions get triggered, and my two statements get output. Note that with the except*, you’re getting an exception group filtered to contain what you’re catching.

04:21 Let’s look at that more closely.

04:35 I’ve created a new group with three errors inside of it.

04:44 I can call the .split() method on the group to filter the contents. The filter returns two groups. The first contains the exceptions I split on, the ValueError, and the second contains everything else.

05:00 Thetry block above that used the except* syntax is essentially doing this split for you. That’s why you get an exception group in the catch portion rather than a single exception.

05:10 You might be catching two or more exceptions of the same type.

05:18 The asyncio library has a new way of creating parallel tasks. It’s called TaskGroup(), and it can be used as a context manager. And in case you’re wondering why my lesson on exceptions has asyncio stuff in it, the task group uses the new exception groups when something goes wrong.

05:38 Before looking at the changes, I thought I’d do a quick review of how coroutines work in Python. The asyncio library is an abstraction for parallel execution of code.

05:48 It uses the async and await keywords in Python. You create a task, run those tasks in parallel, and then wait for them to complete.

05:58 In the example I’m about to show you, I’ve written a function called write_letters(), which will be writing letters to the screen. I’ve defined it as a coroutine, and I’m going to use asyncio to run multiple versions of this as a task. Each of the two versions in parallel will write some output to the screen.

06:18 They’re going to fight. One will run a little bit, and then the other will run a little bit …

06:28 and through this you can see the parallel execution going on. Let’s go run the actual code. Before demonstrating the new feature, I’m going to start with the old way of doing things.

06:41 Here’s my write_letters() routine, which is going to run twice as independent tasks. One of the tasks will be printing out small letters, and the other will be printing out capital letters. Inside of the coroutine function, I iterate through the letters that are passed in and print out each one individually.

07:00 I’m setting the end argument in print() to an empty string so that no newline character is printed. I’m also using flush=True to make sure Python doesn’t wait for the print buffer to be full before printing the content.

07:15 Once I’ve printed a letter, I’m sleeping a random amount of time. This is to make sure that both coroutines get to run, and the code switches back and forth. For small tasks, without this kind of sleep, one coroutine might finish before the other one even started.

07:30 And because I want to demonstrate how exceptions work in this situation, I’m raising one at the end of the coroutine. Let me just scroll down. This function, which I’ve called bootstrap(), is responsible for creating the parallel execution. In here, I’m going to use the old way of defining tasks first.

07:51 This function is the one that’s going change when I show you the 3.11 version shortly. In here, I’ve made a list of task objects which are created using asyncio’s create_task() factory.

08:03 The factory gets a reference to the async write_letters() function. The first task is being given the lowercase alphabet while the second one is using the uppercase version. Once I’ve defined the tasks, they have to go through the asyncio.gather() function. Remember this: this is the key line that changes and will be different in the 3.11 version.

08:24 Finally, I kick it all off by calling async bootstrap() with the run() function. All right, let’s run this.

08:36 Doesn’t take long to print twenty-six letters twice. Let me scroll back up here. you can see the intermixing of the lower and upper cases. The randomness of the sleep means that you get different lengths of letter groupings.

08:50 Sometimes, a coroutine can print two letters before the other one wakes up. In this run, the uppercase letters win, and the capital Z gets printed out.

08:59 That’s Z for my non-American friends. I’m bilingual. The upper coroutine goes to sleep. The lower coroutine prints w and x, and then the upper routine wakes up and raises the exception. Let me just scroll back down here so you can see the stack trace, and at the bottom you noticed that the exception got raised.

09:23 Now let’s do this over again using the new feature, task groups, in Python 3.11.

09:33 Nothing in the write_letters() coroutine has changed. It’s the same function. Scrolling down … The bootstrap() function has changed, though.

09:44 Instead of creating a list of tasks and then using gather(), the new way creates a TaskGroup instance as a context manager. Inside of the context block, I use the task group’s create_task() factory to create my tasks.

09:58 This isn’t startlingly different from before, but it is more succinct, and the context manager makes it look cleaner, or at least I think it does. Just like before, the run() function is used on my bootstrap() and bootstraps it.

10:14 Let’s run this …

10:20 Still rabbit-quick. Bit of a spoiler here with those pretty ASCII lines. I’ll come back to those in a second. Let me scroll up. This time my lowercase coroutine finished first. Randomness is, well, random.

10:34 Once it was done, it threw an exception just like before. Scrolling down … and like I said, the output is now an exception group. This also feels cleaner.

10:46 It neatly distinguishes between the exception in the coroutine itself and the exception that the task group manages when something goes wrong. A positive side effect of this is that because the task group is handling the error first, it can force the other tasks to stop before re-raising the underlying case.

11:06 So as a summary, this is the old way of doing things, and this is the new 3.11. To achieve the same thing, you need a little less code, and it seems a little easier for me to read.

11:17 The use of the context manager TaskGroup() also means you don’t have to remember to do the gather(). It’s taken care of by the context manager block automatically.

11:28 What would a new Python release be without some additions to the typing library? Next up, new additions to the typing library.