Saving Memory With Lazy Iterators

00:00 In this lesson, you’ll solve one problem twice, once with lists and once with iterators, and compare how much memory each version needs. Here’s a problem. Given a list called inputs and a positive integer n, split the list into groups of length n.

00:16 For now, assume that the length of the list divides evenly by n. So with six numbers and an n of 2, you’d expect three pairs. A first attempt might work out how many groups there are, then slice the list once per group.

00:32 naive_grouper() divides the length of inputs by n, then builds a list of tuples, one slice at a time.

00:41 Try it on the numbers from 1 to 10. With n set to 2, you get five pairs. It works. Now compare that with a version built from iterators. better_grouper() makes a list with n references to the same iterator, then unpacks that list into zip().

01:00 To see what that list looks like, build one with two references. Both entries are the very same object. There’s only one iterator here. And better_grouper() gives you the same five pairs as before.

01:13 Here’s why that works. Both entries in the list point to one shared iterator. When zip() asks the first entry for a value, it gets 1. When it asks the second entry, the shared iterator has already moved on, so it gets 2.

01:29 That’s the first pair. Then the same thing happens again, and zip() builds 3 and 4, then 5 and 6, and so on, until the iterator is exhausted. To see the difference, you can run each function on a range of 100 million numbers and measure it with the time command.

01:45 The naive version needs about 4.5 gigabytes because it builds every tuple and the whole list up front. The lazy version needs around 11 megabytes because it only ever holds one group at a time.

01:58 That’s about 400 times less memory. The lazy version also accepts any iterable, even an infinite one, because it never calls len().

02:07 In this lesson, you replaced a list-building function with a two-line iterator pipeline and saw it use a tiny fraction of the memory. But better_grouper() has a problem.

02:19 Next up, you’ll see what happens when the length of your data isn’t a multiple of n and how itertools fixes it.