In this lesson, you’ll learn how to create generator functions, and how to use the
yield statement. Generator functions are a special kind of function that return a lazy iterator. These are objects that you can loop over like a list. However, unlike lists, lazy iterators do not store their contents in memory.
You’ll also compare list comprehensions and generator expressions. Once you’ve learned the difference in syntax, you’ll compare the memory footprint of both, and profile their performance using
So here’s my code editor. I’ve written a function here. It’s called
infinite_sequence() and it initializes a variable called
num, and then
while True:, there’s a loop here which returns
num and then increments
1 in each iteration.
So, there is a problem with this function, and you can see that my IDE is already highlighting the
num variable on line 5, but I’ve chosen to ignore that for now. Instead, what I’m hoping will happen is that since
True is always
while loop will run forever, and so it will return an infinite sequence like the name suggests. Let’s try this out and see what happens.
It evaporates. And we never progress past the initial value of
0. Now a generator function would allow us to get around this problem and I’ll show you how. I’ll turn this function, this very normal function, into a generator function with one quick change.
Since this is a generator function, I won’t use it in quite the same way as a normal function. Instead, what I’m going to do is I’m going to create a variable for it and I will store my function there. Let’s call this variable
Okay, there we go, so now I have this variable,
infinite, and in that variable, I’m storing this generator function. Let’s just check that this is an actual generator, and I’m going to do that with the
type() function. Okay, so you can see that this object is of the
I’m going to call
next() on it right now. Okay, so it gave us the value
0. So far it hasn’t done anything very extraordinary, it just gave us back the same value the previous function was giving us, but watch what happens if I call
next() on it again.
What we’re seeing here is quite different from the behavior we see from normal functions. In this case, we started the generator, and then every time we’ve been calling
next() on it, it’s running this loop once and it’s stopping.
It’s stopping here, where the
yield keyword is. So, the
yield keyword is causing it to stop and pause there and return this value, or yield this value. Let’s try running this again and you see, it keeps going, and rather than just stopping and evaporating, the way a normal function would, the generator function is remembering where it left off. This is called state.
We say that it’s keeping state between calls, so it’s keeping track of its own condition. And this is quite nifty because we can keep on calling
next(), potentially forever. And yet, as we’ll see later, the memory footprint is quite modest because we don’t have to store an infinite sequence in memory. Okay.
Let’s see how this plays out when we call
next() on it. So the first time I run this, the same thing happens as last time. It runs all the way to the first
yield statement, it yields
num, whose value is
0 because it was just initialized up here and it stopped here, the way it had the last time we called this.
So, what happens if you have several
yield statements in a generator is that every time the generator is called, it runs until the next
yield statement, and then it stops, and that’s where it will wait. Let’s call this one more time.
1. If I call it again, I get the second
yield statement to run. Okay. There’s one more special case I’d like to show you before we move on to something else. And that’s what happens when we exhaust our generator.
So, as you can see, I got a traceback and the error type is a
StopIteration. Now, something interesting is that whenever you’re using an iterator in Python on any structure which has baked-in double-under
__iter__() methods, this is what’s happening behind the scenes.
next() function is being called to move the iterator along and when it reaches the end, it actually raises a
StopIteration, but you tend not to see it because those
StopIterations are being caught by
breaks. That’s sort of an aside. If you want to go a bit deeper into that topic, you can read up on iterators.
But the main point I want to make here is that when generators are exhausted, that is to say, when they’ve reached the end of the values which they can yield, they will raise this
You’ve seen how to create a generator function using the
yield keyword, basically writing a normal function and replacing
yield, but there’s another way to create generators, and that’s generator expressions.
07:24 You might remember list comprehensions. As a reminder, here on line 1, there’s a list comprehension which returns a list of all squared numbers in the range from zero to five. We can try running this,
And what you have here is a generator comprehension. So, let’s see what happens when we call that. And you see, in this case, a generator object is being returned. And if we had saved this in a variable, then we could operate on it using the
08:15 An easy way to remember this is that when you’re creating a list in Python, you’re using square brackets, right? So you open and close square brackets, and in between, you have the different list elements. A generator comprehension, on the other hand, uses parentheses, as you can see here on line 2. Okay.
So now that we’ve seen the syntax to create generators using functions, or how to create generator comprehensions, let’s spend a minute or two looking at how they perform in terms of both memory and speed. The way I’m going to do that is I’m going to import
sys, and this will allow us to see how much memory is being taken up by different objects. So now I will first create a list comprehension with squared numbers,
09:03 So just as a quick reminder, in the list comprehension, I used square brackets, and for the generator comprehension, I used parentheses. Now let’s see how much memory these different objects are taking up.
Basically, you can take any function and turn it into a generator by replacing
return with a
yield. You just have to remember that every time that this generator is called, it will run until it hits the
yield statement, and that’s where it will stop.
10:43 The syntax for this is very similar to a list comprehension, except that instead of using square brackets, you’re using parentheses. And the key points here are that generators are iterable objects, they keep state between calls, that is to say that they remember where they left off.
It’s a bit like having a bookmark in a book. You don’t have to start all the way from the beginning each time. Whereas a normal function would exit when it hit
return and it would start from the very beginning each time. They save memory, they have a very modest memory footprint, but you do pay a price for it in terms of speed.
11:14 So there is a tradeoff there between memory and speed, and what is the right solution for you really depends on what kind of application you’re working on, and what your use case is. In the next video, we’ll go a bit deeper into generators by looking at some advanced generator methods you can use. I’ll see you there!
Become a Member to join the conversation.