You must own this product to watch this lesson.
You must own this product to watch this lesson.
List Comprehensions
00:00 In this tutorial, I’m going to talk about list comprehensions. List comprehensions are actually one of my favorite features in Python. I know they can seem a little bit arcane at first or a little bit strange, but they are actually super helpful. They can help make your programs easier to read and easier to maintain, once you’ve actually wrapped your head around how they work.
00:19
And when you break them down to their first principles, they are relatively easy to understand. Behind the scenes, it’s pretty much all just for loops, and that’s what I’m going to teach you in this tutorial.
00:31
I’m going to show you exactly how you can take an existing for loop in Python and then transform it into a list comprehension, and also do the same thing the other way around. All right, so let’s jump right in. So, here’s a simple list comprehension that I just defined. You can see here, it looks kind of similar to defining a list constant, right, where we would just put in values between these square brackets.
00:54
So, I defined this list comprehension here, and then I assigned the result of it to the squares variable. The reason I did that is so you can see what the output of that list comprehension is, and I also wanted to give it a name because it’s—well, it’s computing all the integer square numbers from 0 to 9.
01:15 And of course, I could have just taken the same list comprehension and run that immediately, right? Now, when you look at this list comprehension, the syntax is fairly human-readable, I want to say.
01:26
And really what this syntax is—it’s kind of a shorthand for a regular for loop. I’m going to clear my screen here, and I’m going to bring back the list comprehension.
01:35
Now, I’m going to write a for loop that runs the exact same calculation. We’re going to start out with an empty squares list, and we’re going to populate that list as we go along. To start the for loop, I’m going to take this part here from the list comprehension.
01:48 So, I’m just copying and pasting it over.
01:51
And now what I need to do is I need to update squares, so I need to append something to it. For that, I’m going to take this part of the calculation here. All right, so you can see where I copied these parts from.
02:04
The squares one—so that one, and then the for loop is that one here, that part here in the list comprehension, and then the actual calculation that I’m running here, or the expression that I’m evaluating, I took that from here to calculate the squares.
02:21
All right, so when I take a look at how this squares list was populated by the for loop solution, you can see it’s exactly the same.
02:28
So when I rerun the original list comprehension, you can see it has the exact same output. So, that’s how you transform a very simple list comprehension into a for loop.
02:41 Now, if you try and generalize this structure—so, the transformation that we just applied—if you try and generalize that for any other kind of list comprehension, you might end up with a pattern somewhat like this.
02:53
So, what I’ve done here is I’ve taken the list comprehension up here that I used in the example, and I’ve replaced parts of it—kind of, the parts that you would swap out that are not part of the skeleton for the list comprehension—and I replaced them with these markers here. So in this case, the (values)—which is our output values—would correspond to what was called squares in the example, and then the (expression) would be the x * x operation that we’re doing here.
03:24
And then, of course, this for part would be exactly the same. Then, we’re taking a (value)—in this case, it would be called x—from a (collection). In this case, it would be the range of integers from 0 to 9.
03:39
This is sort of the template behind this list comprehension. And as you’ve seen, we can take this template and transform it into another template. So, this list comprehension, if you wanted to transform it into a for loop, it would look like this.
03:59
This is pretty much exactly what we’ve done in the example. So again, our (values) here, that’s just squares, and we’re creating an empty list and then we’re iterating over each (value) in the (collection)—and you can see how these correspond here—and then we’re updating the output (values) that we previously initialized.
04:19
We’re just appending stuff to that list, and we’re not just taking the value from the collection and adding it, but we’re evaluating some kind of expression based on that value to calculate the actual output value. So, this is how this transformation goes down, and the cool part is that it works in both directions, so you could apply the same transformation to actually take a for loop and turn it into a list comprehension again.
04:47 And that’s kind of the powerful thing about these list comprehensions, because once you’re getting tired of writing loops that look exactly like this—well, now you have a tool in your toolbox that you can apply to make these parts in your program a little bit more readable, and basically turn these three lines into a one-liner that does exactly the same thing, which can be pretty powerful. It can be great for improving readability.
05:12 It can also be horrible for improving readability, but I’m going to talk about that in a little bit, at the end. All right, so we just looked at this fairly simple pattern here, and what I want to talk about now is how you can expand this pattern to actually include filtering. And filtering is going to make the list comprehensions that you can write a lot more powerful and a lot more flexible, so let’s take a look at that.
05:34 I’ve updated my example here by adding filtering, and you can see here that this looks very similar to the previous example. All of the parts that you’re already familiar with are still there.
05:43
We’ve got our output list of values, we’ve got the expression we calculate, and we’ve got this for <value> in <collection> part that we run.
05:53
This new stuff here at the end, that’s where the filtering happens. So in this case, we’re calculating the same set of values, but we’re only keeping the values where this condition here is True.
06:08
And in case you haven’t seen the modulo (%) operator before—it divides two numbers and then gives you the remainder. So, if I go 31 % 2, the remainder of that is 1, because it’s an odd number. We can’t cleanly divide it by 2, so the remainder of that is 1. And if I go 30 % 2, then the remainder is 0.
06:32
I’m taking advantage of that—here in this filtering expression—to only include values in this even_squares list that are actually even. So, this is a great way to find out if a number is even or odd.
06:46
You can just use the modulo operator for that. All right, so let’s take a look at the output here that we’ve generated. As you can see, this filtered down the previous list of squares so that it only includes the even squares. So remember, this is what we had previously, before the filtering, and that also included odd numbers—and now, those are all gone because we filtered them out with this if part in the list comprehension.
07:12
Now, let’s take a look at how that affects our template behind the scenes—or how can we transform this into a standard for loop template behind the scenes.
07:23
So, if you wanted to generate the same list with a for loop, you would probably do something like this. Again, you would start out by creating an empty output list, and then we would iterate over this range again.
07:36
And now what we need to add is the filtering. So in this case, I would just add the filtering condition here, inside the for loop,
07:47
and then if the filter condition is True, only then am I going to append the calculated value to my output list. This is exactly what I’ve done here, so hopefully, we’ll get the same result. Yup—that looks pretty good.
08:02
You can see here how I took the initial list comprehension with filtering and transformed it into an equivalent for loop. All right, so I’ve got the updated template here, and pretty much all that I’ve added is this if condition part.
08:17
We can take a look at how this would transform to a for loop. All right, and this is how it would transform into a for loop. So again, this is really similar to the previous template, but now we’ve added this if condition line that applies the filtering.
08:33
So you can see, if you build your understanding of list comprehensions step-by-step, then I think it becomes much easier to actually see what’s going on. Because the jump from the previous list comprehension and then adding this if condition here is relatively small.
08:52
It’s relatively easy to wrap your head around that. But if you’ve seen that for the first time, if you had seen a more complex list comprehension—you know, like this one—it would seem a little bit overwhelming, to be honest, because it just helps to break things down into these little steps, and hopefully that example showed you how these list comprehensions break down into these relatively simple for loops.
09:17 Now, there’s a lot more to say about these list comprehensions—or comprehensions in general, because there are also dictionary and set comprehensions, and I’m going to put a link into the description of this video where you can learn about this stuff if you want to—but before you run off and jump into refactoring your programs to add list comprehensions to them, I want to talk a little bit about two things.
09:38
So, number one is formatting: How should you format your list comprehension so they look nice and clean? Because one downside here is that they can get super long and can kind of go past the line length limit that you might have set for yourself, or if you’re using pep8, you know, it’s really easy to go over that limit.
09:55
So, I want to talk a little bit about that, how I like to format my list comprehensions. And then also I want to give a bit of a caveat here, where list comprehensions seem like a really great tool—and they are—but there’s also a danger of overusing them, so I want to give just my opinion on that, and when it makes sense to use or not use a list comprehension. All right, so in terms of formatting, what I like to do is if I have a slightly more complex list comprehension and it doesn’t fit into one line, often what I’ll do is I’ll align my comprehension that way so that I’ve got the expression on the first line, and then I’ll put the for part below that and I’ll put the filter part on another line. I feel like that reads pretty well and looks pretty clean, and it’s a good way to format your list comprehensions if they run the danger of spilling across the line boundary, right? So, that’s a way you could do this.
10:52
To show you an example of what this would look like with the even_squares example that we had previously, I might do something like this—or actually in this case, because I have enough space, I would probably do something like this, and actually leave the filtering down there, or even put it on a single line because that’s only 56 or 55 characters here, so this is not going to hurt too much.
11:15 But I hope this gives you an idea of what you can do if you have, for example, if you have a longer function call here then it might make sense to format this differently and actually move this down or do something like this.
11:26 Like basically, what I wanted to say here is that I like using these parts here in the template as my breakpoints if I need them. That can help you break it down if you’re getting close to the line boundary. All right.
11:40 So, the other thing I wanted to talk about is that you want to be careful with how often and under what circumstances you use these list comprehensions.
11:51
So, one downside of list comprehensions is that, well, they are more terse than for loops. They can feel—or they can get a little bit overwhelming if someone is not familiar with them.
12:00 So with this feature, I think it makes sense to balance it with, for example, who you’re working with. If you want to introduce Python to a new team, maybe blasting them with a bunch of list comprehensions isn’t the greatest way to do that.
12:14
Because a for loop is probably going to seem more familiar to them than hitting them with a list comprehension right away. So depending on the developer audience you’re working with, these list comprehensions might be a good feature to use, or they might not be.
12:28 This is something to keep in mind. The other thing is that these list comprehensions can also be nested. You can have list comprehensions inside list comprehensions.
12:37 And this is something that I would encourage to stay away from that. It’s much better to turn that into a loop or actually even move part of it into a separate function call, because it’s going to be a lot easier to read and a lot easier to maintain. So, you can write some pretty gnarly list comprehensions, and usually—you know, it’s tempting, I know it’s tempting—but usually it’s not a great idea to do that, because it gets super confusing really quickly and it gets extremely hard to maintain.
13:05 So it makes sense to really spend some time to think about it, whether it’s better to use a list comprehension in a specific use case or not. So, that was a word of warning here at the end, but I think it’s definitely something you want to keep in mind.
13:18 Like, all code is communication and you always want to make sure you’re communicating your intent clearly, and that you’re not writing code just for the sake of making it extremely terse or concise or overly concise, right? Because—I know it’s fun, I’ve done it in the past and I’ve done it to other people—and really, it just kind of sucks, so I would recommend that you don’t do that.
13:41
I mean, definitely use list comprehensions—they are awesome. I think for something like that, in my mind, that’s much cleaner than using the equivalent for loop, but be careful with deep nesting and some of the crazier list comprehensions you might want to write.
parsarmin on April 18, 2020
Hello Dan, I’ve the same question like Bob. I’m searching for the same link that You mentioned in the Video with the list comprehension to put into the description below.
Are the tutorials respectively the course ‘Python Tricks’ not supported anymore. I’m just wondering because I’ve bought the pdf via the Real Python page right one day ago.
Please give us a hint where we could find a summary of the links if collected somewhere else safer :)
Thank You for the great Python Tricks
Dan Bader RP Team on April 19, 2020
Check out this tutorial on Python dictionary comprehensions. I’d also recommend our “When to Use a List Comprehension in Python” as a followup.
radikyl07 on June 5, 2020
Hi Dan,
I have started to work on improving my Python and the Python Tricks books has been a real gem. One question with regards to the Interactive Shell in your videos supporting syntax higlights, intellisense and more. Is this publically available and is there a tutorial on setting it up?
Thanks,
Dan Bader RP Team on June 5, 2020
I’m using an alternative Python REPL called bpython in my videos. You can learn more about it here: bpython-interpreter.org. If you find bpython difficult to install I can also recommend ptpython as an alternative.
Dan B on Nov. 10, 2020
Thanks, came with the same question bpython ftw!
joerg7 on Jan. 28, 2021
I just got here after buying the tricks. Have no time to watch the videos rigth now. Are they always available using the same link above, as there does not seem to be a download opportunity?
Bartosz Zaczyński RP Team on Jan. 28, 2021
@joerg7 Yes, the videos are always going to be available here 🙂
kedmist on July 20, 2021
Hi Dan…thank you for the video. I’ve recently studied Big O notation and wonder how List Comprehensions fit in to that. It seems to me that it likely does not change the Big O of a given FOR loop since all that seems to be happening here is a “rearrangement” of the FOR loop. But I believe List Comprehensions are supposed to be “faster under the hood”, and as such I wondered if Big O might be affected (positively especially). ??
Bartosz Zaczyński RP Team on July 21, 2021
@kedmist You’re right in both respects. Using list comprehensions is a bit faster than Python for-loops because of their optimized implementation. At the same time, they don’t magically change the time-space complexity of the underlying “algorithm.”
The Big-O notation isn’t a direct measure of speed. You can exercise the same algorithm on two different machines against different data volumes, and the outcome will likely consume a varying amount of time and memory. The Big-O notation is intentionally stripped away from any absolute components to make comparisons between algorithms meaningful. It’s a relative measure of the number of elementary operations with respect to the size of input data. How you define an elementary operation is somewhat arbitrary, but it’s usually the one that runs most often. In a loop, that could be incrementing the counter, for instance.
Mikhail Chmir on Sept. 11, 2023
@Bartosz Zaczyński Thank you very much for the clarification. I’ve explained this concept to many people: that code performance at the technical level of implementation and the time-space complexity of the “algorithm” don’t always solve the same problem in the same way. And that problem is sometimes a very bad one. Some programmers try to optimize the code and spend time on it, but the complexity at the algorithm level doesn’t change and sometimes even increases. And sometimes, it is tough to explain it to a programmer.
Please advise me to read some articles or documentation on this topic. I’ve always understood this problem intuitively, purely at the mathematical level.
Thanks again.
Bartosz Zaczyński RP Team on Sept. 12, 2023
@Михаил Чмир You’re welcome, always happy to help 🙂
Our tutorial on the binary search algorithm has a section devoted to analyzing the time-space complexity of the algorithm step by step, using an analytical approach, as well as graphical and “manual” one. It’s a nice practical example that illustrates how to work with time complexities.
Mikhail Chmir on Sept. 12, 2023
Greetings!
@Bartosz Zaczyński Thank you so much! This is just the coolest article ever!
Petar Petrenko on Oct. 14, 2023
Hi, I can’t download the videos. When I click the radio button (to select it, I hope) it starts the video. I can’t see any DOWNLOAD button.
Bartosz Zaczyński RP Team on Oct. 14, 2023
@Petar Petrenko Unfortunately, you cannot download the individual videos because they’re copyrighted. You can only watch them online on the Real Python website with an active membership. However, you can download the supporting materials for offline use, including slides and sample code.
Petar Petrenko on Oct. 14, 2023
Sorry, but I can’t find where to download the supporting materials from, either.
Bartosz Zaczyński RP Team on Oct. 16, 2023
@Petar Petrenko There’s a Download & Resources drop-down button just below the video. It seems that this Python Tricks video course doesn’t come with the slides, though.
You must own this product to join the conversation.
Wolfswain on March 9, 2020
You mention in the video you’d add links to dictionary and set comprehension videos, are those available yet?