Join us and get access to thousands of tutorials and a community of expert Pythonistas.
This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.
Generators
In this lesson, you’ll learn about generators. Generators are a useful way to iterate through a sequence using constant memory. Here’s an example:
>>> g = (x for x in [1, 2, 3])
>>> g
<generator object>
>>> next(g)
1
>>> next(g)
2
>>> next(g)
3
>>> next(g)
StopIteration
You can also define generators using a function:
>>> def f():
... yield 1
... yield 2
... yield 3
>>> f()
<generator object>
>>> g = f()
>>> next(g)
1
>>> next(g)
2
>>> next(g)
3
>>> next(g)
StopIteration
If you want to learn more, check out What Are Python Generators?
00:00 Generators are a special type of iterator that you can use to iterate over a sequence of values. They’re special because they’re lazily evaluated—that means that you only evaluate values when you need them.
00:11
Let’s create a generator to iterate over the values 0 through 5. So, g = and then it’s like a list comprehension, except you use parentheses so that this is a generator expression. (i for i in range(6))—because we want to include 5, and range() is exclusive. g is a generator.
00:30
To get the next value, you type next(g). 0, 1, 2, 3, 4, 5. If you call next() again, it will raise a StopIteration because there are no more values to iterate through. What’s special about a generator is that once it’s done—as in, once we’ve exhausted all the values, you can’t get the values again—calling next() doesn’t give you anything. It will always raise an error.
00:54
So to get the values again, you have to instantiate it again, and then call next() again. Let’s compare this with a list. Let’s say we want to sum all the values between 0 and 1000, including 1000. Using a list, you would do sum(), list comprehension—[i for i in range(1, 1001)].
01:19 Let’s do the same thing with a generator. We’ll just replace these brackets with parentheses.
01:27
This is because sum(), under the hood, calls __iter__() on our generator, which actually just returns the generator, but I’ll save it into a variable called iterator. Then, it calls the next() over and over on our generator.
01:44 Same thing goes for the list.
01:51
So, why even use generators? Well, let’s look at how much memory is taken with a list comprehension. Let’s save this in a variable lst, import sys, and sys.getsizeof(lst).
02:04
So, this is 9,000 bytes. Let’s do this for our generator. I don’t remember if we made a g variable yet, but it’s fine, we’ll just redefine it.
02:15
Size of g is only 128 bytes. So you can imagine, as our list grows really large—millions and millions of values—this is going to take up a lot more memory. Generators, on the other hand, will always be this constant number of bytes.
02:32 So in an interview, if you’re asked to loop through something that might contain lots and lots of values, you should use a generator. You’d also want to use a generator if you don’t need to evaluate all the values at once—you just need to evaluate them one at a time.
02:46
So, another example might be—with our lst,
02:51
if you had, like, some slow_method(), or something, that you call with i on each time, this will evaluate the slow_method() for all the values at once and then proceed with the code, while a generator, like this, would only evaluate the slow_method() when you need to, like this.
03:08
There’s also something called a generator function, which is a function that when called returns a generator. Any function can be a generator function if it contains the word yield.
03:18
So, yield is like return, except when you call next() on this generator that is returned, it will resume where it left off. So, yield 1, yield 2, and yield 3.
03:28
So calling f(), you would think according to most rules would go inside the function, but this actually returns a generator object. To go inside, you have to call next() on that generator object, and you have to actually save this into a variable, and then call next(). 1, 2, 3.
03:46
Calling next() on f() like this is going to create a new generator each time. There are other examples of generator functions, and I will link a Real Python video on generators down below. In the next video, you’ll learn about dictionaries and defaultdicts, which are a very useful data structure to store default values.
Fabio Alvarez on Oct. 28, 2022
Great! I have learned a lot with this series
Angel Diaz on April 5, 2023
was tinkering a bit with timing the execution of generators as opposed to simple array and it seems that the lazy execution means its strictly slower when traversing the array and performing a sum due to not having the next element in cache even though you pay the upfront cost of creating the entire array:
import timeit def addd(): sum([i for i in range(1,100001)])
def addd_gen(): sum((i for i in range(1,100001)))
timeit.timeit(addd, number=100) 0.5602305000011256 timeit.timeit(addd_gen, number=100) 0.7002893999997468
so generators are more appropriate when speaking about memory savings but is range() special? because I thought it too return a generator
Become a Member to join the conversation.
James Uejio RP Team on April 27, 2020
If you want to learn more, here is a Real Python walkthrough video on generators: What Are Python Generators?