Saving Memory With Lazy Iterators

00:00 In this lesson, you’ll solve one problem twice, once with lists and once with iterators, and compare how much memory each version needs. Here’s the problem.

00:11 Given a list called inputs and a positive integer n, split the list into groups of length n. For now, assume that the length of the list divides evenly by n.

00:21 So with six numbers and an n of two, you’d expect three pairs. A first attempt might work out how many groups there are, then slice the list once per group.

00:33 naive_grouper() divides the length of inputs by n, then builds a list of tuples, one slice at a time.

00:42 Try it on the numbers from one to ten.

00:45 With n set to two, you get five pairs. It works. Now compare that with a version built from iterators. better_grouper() makes a list with n references to the same iterator, then unpacks that list into zip().

01:01 To see what that list looks like, build one with two references. Both entries are the very same object. There’s only one iterator here. And better_grouper() gives you the same five pairs as before.

01:15 Here’s why that works. Both entries in the list point to one shared iterator. When zip() asks the first entry for a value, it gets one. When it asks the second entry, the shared iterator has already moved on, so it gets two.

01:30 That’s the first pair. Then the same thing happens again, and zip() builds three and four, then five and six, and so on, until the iterator is exhausted.

01:40 To see the difference, you can run each function on a range of a hundred million numbers and measure it with the time command. The naive version needs about four and a half gigabytes, because it builds every tuple and the whole list up front.

01:54 The lazy version needs around eleven megabytes, because it only ever holds one group at a time. That’s about four hundred times less memory. The lazy version also accepts any iterable, even an infinite one, because it never calls len().

02:09 In this lesson, you replaced a list-building function with a two-line iterator pipeline and saw it use a tiny fraction of the memory. But better_grouper() has a problem.

02:19 Next up, you’ll see what happens when the length of your data isn’t a multiple of n, and how itertools fixes it.