Context Managers and the "with" Statement

00:00 Hey there! This is Dan. Today I’m going to talk about the with statement in Python. The with statement relates to something that is called context managers, and it is sometimes regarded as a bit of an obscure feature by some people,

00:16 but actually, when you peek behind the scenes there’s relatively little magic involved and it’s actually a highly useful feature that can help you write cleaner and easier-to-understand, easier-to-maintain code. So, the first question that I want to talk about is, “Okay, so what is the with statement actually good for?”

00:36 The with statement is great for simplifying some of the common resource management patterns that you’ll see in your code. So, by resources in this case, I mean system resources, like files, locks, or some kind of network connection—stuff like that.

00:53 And so with the with statement, you can abstract away some of the functionality there that is commonly required, like dealing with the acquisition and releasing these resources again.

01:05 A good example here would be dealing with files. An open file is a system resource and you need to acquire it in some way, and then also release it to give that resource back to the system so it doesn’t have to keep track of the file you’re working with.

01:21 I want to take a look at this simple example here, where we’re using the with statement to open a file and then write to it, and to have the file automatically closed. So, the big advantage here is what I just said, where if we use the with statement to open the file, Python is going to make sure that this file is going to be closed once we’re done with it. So basically, when you do this, you create this context where we’re working with the file object, right? We’re giving it the name f, which is maybe not an ideal name, right?

01:56 Maybe you should call this outfile or something like that. Then, in the block that follows—in this indented block here—we can work with f, and as soon as the execution of your program leaves the context, Python will automatically close the file.

02:12 So it’s going to call f.close() and that will return the resource, the file descriptor, back to the operating system and it doesn’t have to keep track of that resource anymore.

02:24 So, to demystify how the with statement works, let’s take a look at what actually happens behind the scenes. So, if you were to write this manually, you would probably do something like that, right? So, you would open the file, just assign it to f here on this line, and then I’m using a try / finally statement here to actually write to the file. And then, to make sure to actually close the file, so even if something goes wrong here—you know, we have some kind of exception, some error where we can’t write to the file—this would make sure that we’re actually calling .close() on the file, which in a longer-running program—that’s super important, because you always want to give up these resources.

03:00 So the try / finally here is really significant, right? It wouldn’t be enough to just say, “Okay, I’m going to do an open() and then a .write() and then a .close(),” because you could potentially leak that resource by not releasing it again if some kind of exception would happen. Using the with statement really simplifies the code we have to write here. Because what you see in the second code example—that is super common. You’re going to use a pattern like this very, very frequently if you’re dealing with any kind of resource like open files, open network connections—stuff like that.

03:30 You always want to make sure you’re using this try / finally pattern. So, the with statement is a way to abstract that away, to factor out that functionality so that you don’t have to write that every single time you open a file or you create a file. Now, how does this actually work behind the scenes, right?

03:49 Because this seems like a really nice feature, and in my opinion, it’s a highly useful feature in Python. But how do we go from this to that, behind the scenes?

04:00 This is what I’m going to explain to you now. So, how can you support the with statement in your own objects? Because there’s really nothing magical or special about the way open() works—there’s not some magic sauce that’s only available to Python built-in objects.

04:14 You can support the same functionality in your own programs or in your own objects by implementing so-called context managers.

04:24 So, what’s a context manager? Really what it boils down to is that it’s a simple protocol or some interface or contract that your objects follow so that they can be used with the with statement. I’m going to show you a simple example here.

04:40 So, this ManagedFile class basically emulates what open() did there and how we were able to use it with the with statement. So, there’s two methods that an object needs to support in order to be used with the with statement, and that is the .__enter__() and the .__exit__() method. It’s a very, very simple contract and a very, very simple interface, and with this ManagedFile class here, we can go ahead and say, okay, with ManagedFile() as f, and that looks exactly the same as the open() function call looked earlier.

05:14 And now we can actually go ahead and write something to that file, and we’re going to get the same result. The file’s automatically going to be closed.

05:23 All right, so I want to talk a little bit more about the steps that Python takes behind the scenes for this example to actually work. You can see here, we defined this ManagedFile class, and we’ve got this constructor here that just assigns the name and remembers the name from the file that we want to create.

05:39 It doesn’t actually open the file until the .__enter__() method is called. And that’s kind of how you want to structure your context managers, right?

05:47 So that, really, the resource gets acquired when the .__enter__() method is called and it gets released when the .__exit__() method is called. And then the .__exit__() method also takes additional parameters that will tell you about some exception that might’ve happened, in case you want to inspect that, log that, or do something with that.

06:04 I’m not doing that here—I’m just checking like, “Hey, did we actually open anything?” If so, then we’re going to close the file. And now, obviously, with this example you need to remember that this is sort of a useless wrapper around the open() function here, because the open() function already pretty much does that when it functions as a context manager, right?

06:24 So this is merely an example to illustrate that, but you could imagine this would be some other kind of resource, like a database connection. So now, what’s going on here in this example?

06:34 Because when we create an instance of ManagedFile, we’re not actually immediately calling the .__enter__() method. So I could also do something like this—I’m going to call this mf, which is not an ideal name, but for this example it’ll work. So, I’m creating this ManagedFile object. And up until now, we didn’t actually call the .__enter__() method, right?

07:02 You can see that here with this exception, because the object doesn’t have a .file attribute yet—we didn’t assign that yet, because we didn’t call .__enter__() yet—or, Python didn’t call .__enter__() yet. Now, when I go ahead and do with mf, and then I can say something like with mf as the_file,

07:23 then I’m going to enter this context where the resource is acquired, and now I would have to go the_file.write('hello.txt'),

07:35 and we can talk about what’s going on here. So, what you can see here with this statement, with mf as the_file—behind the scenes this is going to call .__enter__(), and then it’s going to assign the return value—so, this is going to return the actual file—

07:50 it’s going to assign that to this name here: the_file. And then within the context of this ManagedFile I can use the_file to actually write to the file, right?

08:01 Because this was the actual file object that I needed to write to the file. Then, when we leave the context—like, right after I’m making this call here—we’re going back one indentation level, we’re leaving this context. Then immediately Python is going to call the .__exit__() method, or the dunder exit method. All right.

08:19 So, I hope that gives you a better idea of how these context managers actually work behind the scenes. Now, I just explained to you how class-based context managers work, but this isn’t the only way to support the with statement in Python, and this is not the only way to implement a context manager.

08:39 So, there is the contextlib module in the standard library, and it provides a couple of abstractions on top of the basic context manager protocol.

08:49 And this can make your life a little easier if you’re trying to implement a use case that matches well with what the contextlib module offers.

08:58 So, here’s a quick example of what you can do with this. I’m going to re-implement the same ManagedFile functionality using the contextlib library. So, there’s a decorator in there called contextmanager and this thing is highly useful.

09:14 What that allows you to do is you can define objects that follow the context manager protocol that you can use with the with statement simply by writing a generator. So, when looking at this code, again, you can see our familiar try and finally pattern here, where I’m acquiring the resource and then I’m yielding it, and then later, I’m closing the resource.

09:39 And what happens is that this @contextmanager decorator will turn this generator function—or, this generator that I just defined here—it will turn that into a full-blown context manager that I can use with the with statement. So again, I can do something like this: I can say with managed_file('hello.txt')—

10:06 I’m just going to call it f again—I can say, “Write something to the file,” and of course we can also write more stuff to the same file. And now, when I run this, we’re actually writing to the file.

10:18 If you’re wondering what this is—so, this is just the number of characters written, because .write() returns how many characters were written to the file, right?

10:26 It’s just kind of leaking back into my interpreter session here. So, with this technique, using contextlib and using the contextmanager decorator, you can write some of these context managers a lot quicker.

10:40 I guess the downside is that for someone to understand this piece of code, they would have to have some basic knowledge about decorators and they would have to have some basic knowledge about how generators work in Python, right? So, when you compare that with the class-based implementation, you could actually say, “Hey, the class-based implementation communicates what’s going on much more clearly for someone who’s not an expert at Python.” So depending, you know, how your team feels about this, and what everyone’s expertise level is, it’s kind of a fine balance with what concepts you want to pull into this. Personally, I think this is pretty beautiful, right? This is awesome.

11:20 This is very Pythonic. But on the other hand, it requires someone to understand many, many more concepts in order to actually understand what’s going on looking at this. I mean, they’re going to still use it the same way, so they might not care, but I just wanted to mention that because, in my mind, code always needs to communicate what’s going on. Code is communication, and this is a great example where you can see where the trade-offs are, right?

11:46 And I can’t make those decisions for you, but it will be something that you need to decide within the context of your team, the project you’re working on, and so on, right?

11:55 I just wanted to mention that. All right. So, I hope that gave you a good overview of how these context managers work, how the with statement works behind the scenes, and what you can do with them.