Iterating With Generators

00:00 In the previous lesson, I covered a couple of example iterators. In this lesson, I’ll show you how to use generator functions as iterators.

00:09 A good old-fashioned regular function performs some work and optionally returns a value. It does this as a logical block. You don’t get anything out until the function is complete.

00:19 By contrast, a generator function is a special kind of function that returns a generator iterator. You create these kinds of functions using the yield keyword.

00:30 Python translates your yield statement to act like a next() or more accurately the .__next__() in the iterator. When you call a generator function, it returns immediately giving a reference to a generator iterator.

00:44 Each call to next() works in conjunction with the yield keyword, whatever you yield is what comes back from next(), and the function acts like it is suspended until the subsequent call to next().

00:56 Writing generator functions takes a lot less code than your typical iterator class. You write the function more or less like the body of your .__next__() call, yielding the next value in the iterator. Python then takes care of translating your generator function into a generator iterator.

01:12 In fact, most programmers won’t bother making the distinction between the two and will typically just call the whole thing a generator. Let’s go look at one in the REPL.

01:23 At risk of being unoriginal, I’m going to revisit the square a sequence use case, but this time I’m gonna do it with a generator.

01:34 To declare your generator function like any other function,

01:42 and then you write the logic in a similar fashion as well. Here I’m looping through the sequence passed in, and then the yield is where I’m doing my data transformation.

01:52 By calling yield on the squared item, this is declaring the response for the iterator. When the iterator that wraps this function gets called the first invocation of .__next__() will start the loop, do the math, then yield the first square.

02:07 A subsequent call to .__next__() picks up on the line after the yield. For this very simple function that’s the bottom of the loop, so then the code goes back to the top of the loop.

02:18 The for continues to iterate on the sequence giving the second value. Yes, there’s an iterator in our generator function, iterating on the sequence interval.

02:28 The second value then gets squared and sent back from yield. This keeps happening until the loop is finished. When it is finished and the .__next__() gets called, there’s no code left, so the end of the function gets reached and the generator iterator raises a stop iteration for us.

02:45 Cool, huh? I’ve now done in three lines what took 15 as an iterator class. If I call the function directly,

02:57 I get back the automatically created generator iterator object. This is kind of like calling iter on a list. You just get back the iterator. Instead let’s use this in a for loop.

03:15 And once more, you’ve got some squares.

03:18 Python loves generators so much that it’s even added a quick way to create them. A generator expression is a dynamically created generator typically in a single line of code.

03:30 A syntax is similar to a list comprehension, but you use parentheses instead of square brackets.

03:37 So why might you want to use a generator expression instead? Well, a list comprehension creates a list object, whereas a generator expression returns a generator iterator.

03:48 This means a generator expression typically takes way less memory than the equivalent list comprehension.

03:55 One caveat though, list comprehensions have been massively optimized and often far outperform their looping equivalents. Depending on the size of the list, comprehension might outperform the equivalent generator expression.

04:08 This is a common trade off in computing. You can often go faster by using more memory. As with all optimization, write your code and then measure whether or not you should optimize it.

04:18 Let’s go write a generator expression.

04:22 I’m gonna start with a list comprehension.

04:29 If you haven’t seen one of these before, it dynamically creates a list using syntax similar to a for loop. I find the best way to read these is to start in the middle with the for item part that tells me what I’m getting in my loop.

04:43 To the right of that tells me what I am iterating on. In this case, that’s a tuple and to the left of it, at the beginning, it tells me what I’m doing with that item.

04:53 The equivalent of the loop’s body. In this case, to nobody’s surprise, I hope you see I’m squaring the value. The return of a list comprehension is a list.

05:03 Since I’m in the REPL, you see the resulting list printed out with our oh, so familiar squares. The equivalent generator expression only requires switching out the brackets,

05:19 and there you go. Unlike the list comprehension, which returns a list, this generator expression returns a generator iterator object with an anonymous name.

05:29 Replacing square brackets with parentheses, that’s round brackets for my fellow Canadian programmers, is all it takes to go from comprehension to expression.

05:38 Like with any iterator object, I can use it directly in a loop or store it in a value to be used in the loop.

05:56 And there you go. Another way to make squares.

06:01 A few final words on generators before moving on. First, I want to highlight that you can pretty much do anything with a generator that you can do with an iterator class.

06:09 You can yield the data being iterated over. You can transform the data before returning it, like you’ve seen with the squares, and you can write functions that calculate data as well.

06:19 The Fibonacci example in an earlier lesson could be done in a generator just as easily. In fact, I’d say more easily as it takes less code. The example generator expression that I showed you before was inline, which means it can only really be used once, but you can get around that restriction by writing a function that returns a generator expression.

06:40 This allows you to reuse your expressions, and it means you can use the function’s arguments in the expression as well. The two lines on the screen here, take our squaring generator expression and wrap it in a function.

06:52 Technically, I’m not reusing the expression as a new one is getting created each time the function gets called, but it is code reuse you don’t have to write it twice, even if each call creates a different object.

07:06 In the next lesson, I’ll add a few things to our story about iterators.