Python Generators 101 (Overview)
Have you ever had to work with a dataset so large that it overwhelmed your machine’s memory? Or maybe you have a complex function that needs to maintain an internal state every time it’s called, but the function is too small to justify creating its own class. In these cases and more, generators and the Python
yield statement are here to help.
By the end of this course, you’ll know:
- What generators are and how to use them
- How to create generator functions and expressions
- How the Python
- How to use multiple Python
yieldstatements in a generator function
- How to use advanced generator methods
- How to build data pipelines with multiple generators
If you’re a beginner or intermediate Pythonista and you’re interested in learning how to work with large datasets in a more Pythonic way, then this is the course for you.
Hello, and welcome to this course on generators and the
yield keyword in Python. Before I show you any code, I’d like to give you a general intuition of what generators are and why you would want to use them. So, first, consider a staircase.
00:14 You have a sequence of steps, one by one, right? And the way stairs are designed to be used, or intended to be used, are that you put one foot in front of the other and you go one step at a time, one by one, and then you go up the stairs. But there is an alternative, and I’m sure you’ve tried this as well, and that’s to take several steps at a time.
00:36 So depending on how long your legs are, you might be able to go two steps at a time, three steps at a time, maybe even as many as four steps at a time. But there’s a limit to this, and that limit is really the length of your legs.
00:47 You might be able to go up a staircase with two, three, four, maybe even five steps in one go, but there are staircases out there who are just too long, who completely exceed the length of your legs. And then you have no choice but to take them in smaller chunks. That might not be a single step at a time, but it’ll be a few steps at a time, then. You can’t just do this in one go.
01:09 But because you’re able to break down the task of going up the stairs into smaller, manageable chunks, then you’re able to go up taller stairs, and so a high number of steps isn’t an obstacle which you can’t overcome.
Consider the example of lists. So, a list takes up a bit of memory but it’s an iterable structure, so once it’s in memory, you can iterate over the items, one by one. In this case, you would return
3. Here’s a slightly longer list, and so, since it’s longer, it takes up more memory.
But still, this is something which most computers can handle and you could iterate over the items on this list:
4, and so on, all the way up to
14. In the case of an even-longer list, even more memory is taken up. So, I think you can see where I’m going with this.
02:06 Eventually, you will reach a point where your computer’s memory just can’t handle this. Lists just become too big. And if you’re working with big data or maybe you’re working on an application that uses infinite sequences, then since your computer’s memory is, by definition, limited, and not infinite, then you need a different way to approach this problem.
02:26 And that’s where generators come in. They allow us to attack these very large problems one small manageable chunk at a time. In this course, you are going to learn what generators are and how to use them.
02:39 We’re going to be looking at the syntax and, sort of, the rules that you have to keep in mind when you’re using generators. We’ll also keep in mind what tradeoffs you have to consider. So, as I hinted at a moment ago, they can save a lot of memory, but you do pay a price for that in terms of speed.
You’ll learn how to use multiple
yield statements in a single generator function. And then to wrap up, we’ll look at some advanced generator methods and I’ll show you how to build data pipelines using multiple generators.
03:14 This content is broken up into five videos. Right now, we’re on the first one, the introduction. In the next video, I’ll tell you about generators, how to implement them, what syntax rules you have to obey, and so on.
Become a Member to join the conversation.