What Are Python Generators?

Hands-On Python 3 Concurrency With the asyncio Module Chyld Medford 09:31

Learn the basics of creating and using generators. In order to understand how asyncio works, it is important to have a basic understanding of how generators work. In this lesson, you’ll learn how to turn a regular function into a generator using the yield keyword.

You’ll also see a couple of examples like the one below that show how to create and use a generator to make a sequence of values.

Python
      
    
def odds(start, stop):
    for odd in range(start, stop + 1):
        yield odd

00:00 I have theory.py opened and I’m going to start off with talking about generators, and then coroutines, and then finally asynchronous generators. That’s part of the theory section.

00:11 After that, we’ll actually produce a small application. Okay. Generators—what are they? Generators are basically something in Python that produces a sequence of values.

00:21 That sequence could be a sequence of integers, floats, strings, Booleans—a sequence of any type of value. What I’d like to do is I would like to produce a sequence of odd values starting at some starting point and ending at some stopping point.

00:39 That’s what I’d like to do. So I’m going to create a function

00:44 called odds(),

00:50 and this function will have two parameters, one called start and one called stop. And so let’s say the start is 3 and the stop is 15—that means it will produce a sequence of odd values from that starting point to the stopping point. And in this case here, it’s going to be inclusive of the stopping point. Okay, so I need a way of creating a bunch of numbers from a starting point to a stopping point. And of course, range() is a good way of doing that. So I’m going to say for i in range()—how about we call this odd in range() and we’re going to start at the starting point and we’re going to stop at the stopping point, but we’re going to add 1 because range(), of course, doesn’t—the stopping point is actually exclusive, it doesn’t include it.

01:37 So I’m going to do + 1 so it does include the stop point. And I’m going to go every 2 values, so this is going to be odd. So if I start at 3, it’s going to go 3, 5, 7, 9, et cetera—up until the stop point.

01:50 Okay. Now, this looks like a normal function so far, with a normal for loop. What makes it different is I’m going to use this keyword yield.

02:02 And what yield is going to do is it’s basically going to return out the first value and then pause, and then whoever’s calling this function next() will ask this function to resume.

02:15 So, a generator is basically a function that can pause its execution and then resume at any time later, okay? And I think the best way to show this is through IPython, so what I’m going to do is I’m going to switch to my terminal. I’m going to go into IPython and load up this function.

02:38 Here’s IPython, it’s interactive, and I’m going to load up theory.py. And if I do whos here, whos basically lets me see all the variables that are loaded in the session, and odds is my function that I created. Now, remember, I said this odds() thing is actually a generator. If you call it, it’s a generator, so if I say odds()—or, returns a generator. So if I say odds(), and I say 3 to 15, let’s say—that was my example.

03:09 And I hit Enter. It actually doesn’t run this function. It actually produces, like I said, a generator. So this is your generator. Now, there’s something that you can do from this generator.

03:19 Let’s go ahead and put this into a variable. We’ll call it g for generator, and there it is. How this thing works is if you say next()—now, next() is a built-in function in Python, and if I say next(g) and run it, this next() function actually causes the generator to run up until the yield point, and then it stops. So it gets up to yield, it yields out the value, returns out the value, and then it stops execution.

03:50 To start the function, re-executing again, you just say next(g) again, and you have a 5, 7, 9, 11, 13, 15.

04:00 Now we’re at the end of execution. What’s going to happen now is if I do another next(), I actually get this StopIteration exception from Python, and that lets me know—and it lets also Python know—that this generator has been exhausted.

04:15 It’s now quote-unquote “empty,” okay? Because if I call this generator again, next(), you’ll continue getting a StopIteration.

04:22 Now you could actually produce another generator, right? I could say g1

04:29 and I can go from, let’s say, 7 to 11.

04:34 And so what’s g1? It’s a generator as well too. And if I say next(g1)? 7, 9, 11. Now, there’s something else we could possibly do.

04:45 Let’s create another generator. Let’s call this g2 and let’s go from 7 to 21. Instead of just saying next(), next(), next(), I could actually just produce a list of these things.

04:59 I could say list(g2).

05:05 And when you say list() of a generator, what actually happens internally is it calls next(), next(), next(), next(), next(), and it exhausts the generator until it’s empty. So, where normally you don’t call next(), next(), next() on a generator, there’s these built-in functions—for example like for loops, or these lists constructors where I type in list() and the generator and it exhausts the generator and it puts every value from the generator into this list. If I were to do one more, let’s say g3.

05:40 Let’s go 3 to 21. So, g3 is a generator. I can say for x in g3, which is a generator, and I can print(x), like that. And yeah, so now there was a couple of different ways of causing this generator to run.

05:57 One is I can just do next(), next(), next() on it. The other one is I can call, like, some constructor function, like a list() or a tuple() for example, and that’ll cause the generator to run.

06:07 Or I can just use this for in loop, and that also causes the generator to run. So once again, this thing produces a generator, and the generator will run up until the yield point. It stops executing, and then once you call next() again, it’s like cranking the wheel. It cranks until the next iteration, cranks again, the next iteration, cranks again, the next iteration, until it’s done.

06:32 And once it’s done, you get a StopIteration exception. Okay, so this looks pretty good. Let’s actually use this in a real-life—use this inside of theory.py, instead of—like, a main function.

06:44 So I’m going to say def main():

06:48 and I’m going to create some odd_values, okay? Now, how I’m going to do that is I’m going to create a list. I’m going to say [odd for odd in odds()].

07:06 I’m going to pass in, say, 3 to 15. Okay. [odd for odd in odds(3, 15)]. So this is going to produce some numbers from 3 up to 15.

07:19 It’s inclusive because I do a + 1 here. And this is a list comprehension, this is going to produce a new list for me, but I could have done the list() thing that I just mentioned a minute ago. Now, if I print out odd_values, that should work. And then finally I’m going to say if the __name__ of this module is equal to "__main__", then let’s run our actual main() program, like that.

07:54 Okay, so let’s try that. If I exit out of my IPython terminal, and now I’m going to say python theory.py. And yeah, so here it actually produced a list from 3 up to 15.

08:11 Right here, from 3 to 15. Now, I could have done just another way—and I had mentioned this before in the IPython terminal—I could say odds2 is equal to, let’s say, a tuple of them.

08:27 odds(), let’s say from 21 to 29. And now this will produce a tuple of odds from 21 to 29, so here’s my original odd_values, and then I can print out

08:43 odds2.

08:48 So, these are multiple ways of causing this generator to run. One is I can do this for in loop, which causes the generator to run.

08:56 The other thing I can do is just pass the generator into this tuple() and it will also cause it to run. So let’s try that. And yeah, so this was the original list, this was from the list comprehension, and this is from the generator, basically turned into a tuple until it ends.

09:17 This was a little demo about generators and how they work, and then we will be adding onto this and showing you how generators actually apply to asynchronous IO.

Eddie on April 23, 2019

Not very important I guess, but I guess odds will give us even numbers if we use an even number as start and/or as end, won’t it? Something like this could help avoid that:

def odds_2(start, stop):
    if start % 2 == 0:
        start =+ 1
    for odd in range(start, stop+1, 2):
        yield odd

UBBA on May 15, 2019

Just an observation that instead of jumping to different applications -note that the creation of the directory and files can easily be done inside of Visual Studio Code and that interactive running of the code can be done inside of Python interactive (Jupyter) inside of Visual Studio Code.

Pygator on Sept. 14, 2019

Interesting, can there be multiple yield keywords in our generator or should we only use one per generator function?

Sam Ezebunandu on Dec. 6, 2019

@pygator. I have tried multiple yield statements in the same generator and it seems to work.

J on Jan. 21, 2020

Why didnt calling odds yield or return a value? Why did you have to call the next function to return a value?

J on Jan. 21, 2020

what was the purpose and what is the logic behine the if statement name == main ?

Become a Member to join the conversation.