Locked learning resources

Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Locked learning resources

This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Why Use the reduce() Function?

Now that you know the basics of how the reduce() function works, you’re going to look into some of the interesting ways in which you can use this building block of functional programming. You’ll see how the reduce() function allows you to group your data set into arbitrary categories.

You’ll also learn about Python’s defaultdict class, which is defined in the collections module. Next, you’ll familiarize yourself with some useful helpers in the itertools module, such as itertools.groupby.

00:00 At this point, you might be wondering, “Okay, why do I even need to use the reduce() function in this case at all?” And the reason is that the reduce() function—it can actually go far beyond what you’ve seen here.

00:13 We can talk about some pretty crazy examples here. Why don’t we look at some more interesting uses of the reduce() function? All right, so one interesting thing that I thought we could do with the reduce() function is grouping scientists by field.

00:28 Basically, I want to have some output like this. I want to fill this up—this dictionary here—and I want to populate that with the scientists grouped by their respective field.

00:41 We can do that by creatively using the accumulator field. All right, so what we’re going to do here is we’re going to define a function that we can pass to the reduce() function that will take our existing list of scientists and

00:57 assign it to these fields and group people that way. The way we’re going to do it—there is actually a way you can do this in a single lambda expression, but I just started recording that example and I was like, “Man, this actually really detracts from what I want to show you here.” So here, I’m just going to define my reducer() function and it’s going to take the

01:23 accumulator and a value. I’m just going to say, okay, in the accumulator, well, you want to look at the .field that the scientist belongs to and then we’re just going to append the .name of that particular scientist.

01:41 And then, of course, we need to return the accumulator. You might be wondering, “What is that? What does that do?” You’re going to see that in a minute. So here, I’m now typing out the reduce() function call, so I need to pass my reducer() function, and I need to pass my scientists, and then I need to initialize the accumulator. And so here, this is where this thing comes in because we need to make sure that this dictionary here actually has slots, or has these keys in here for all the different fields, so that the reducer() can go over it and it can update the accumulator according to the individual field that each of these scientists belongs to. All right. So now, once I’ve run this…

02:28 let’s pretty-print it out. And yeah, you can see here it worked! So, for 'astronomy', you’ve got 'Vera Rubin', 'chemistry', 'math', and 'physics'they’re all kind of sorted that way and assigned to the right categories.

02:45 And I hope you can see how this worked here with this reducer() function, and the main thing you need to figure out is how—what I was struggling with the most was how these names like accumulator and value, how they correspond to actual items and what the accumulator looks like as it gets updated. Right?

03:05 That’s like, the biggest thing you need to wrap your head around. It makes sense to play through this, maybe even on a piece of paper or certainly here in the Python REPL.

03:13 One thing that I really don’t like about this is that I have to give it a list of categories upfront that then gets populated. That’s kind of stupid. I mean, if I make a mistake, you know, I have a typo here, then it’s just all going to blow up. But there is a better way to do it, and that is the defaultdict class in the collections module.

03:37 So, I’m going to kind of have to import collections and what we’re going to do now—I’m going to override this scientist_by_field object here, again.

03:47 I’m going to pass it my reducer() function, I’m going to pass it the scientists, and instead of initializing the accumulator, I’m going to go collections.defaultdict(list).

04:02 Then, what that will do—I’ll show you that in a minute. What that will do is it will lead to the same result because this collections.defaultlist() thing—or, .defaultdict() thing is pretty magical.

04:19 So, let’s just create an instance of that class, so, this is a defaultdict. And now this thing, every time you access a key that doesn’t exist,

04:31 it will be created and it will be populated with whatever you pass in here, whatever factory function you pass in here. Right? So now, I can go

04:43 and I can put all these crazy keys here and the dictionary will be updated. So, now I can do stuff like

04:54 dd['xyz'].append() and this will work

04:59 because it’s going to create that slot for me just in time. And I can keep doing that and it will keep updating because now from the second

05:13 call on, it knows that it has this field in there, in the defaultdict, and it doesn’t need to recreate it. So, this is a little trick you can use to get around having to manually define the accumulator here. But of course, it also adds more complication to this function call and I’ll bet you this would actually take many people a while to figure out and to read that, and so that’s actually a pretty good reason to not use that in production code.

05:41 However, I think it makes for a really interesting thought exercise in Python and I think it’s pretty cool to work with these different kinds of programming paradigms and programming styles, like functional programming, object-oriented programming, and more, like procedural programming.

05:59 And it makes sense to be comfortable with these different styles, because even if you don’t always stick to any one style, you know, 100% of the time, you’re going to learn a lot about when these things have their strength and when you should apply them. I think that is really valuable and that’s going to put you above and beyond what most people do when they learn programming, right? So, I think there’s a lot of value in exploring these things.

Avatar image for senatoduro8

senatoduro8 on July 24, 2019

Dan, What is the difference between functional and procedural programming?

Avatar image for angelojwillems

angelojwillems on March 28, 2020

Waw, I use the Counter function from collections as a means to the same end. Didn’t realize there was a defaultdict. From now on I’ll just use that instead.

Avatar image for darth88vader88

darth88vader88 on April 1, 2020

I get this error with bpython console:

>>> total_age = reduce(lambda acc,val: acc + val['age'],names_and_ages,0)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    total_age = reduce(lambda acc,val: acc + val['age'],names_and_ages,0)
NameError: name 'reduce' is not defined
Avatar image for cellist

cellist on April 1, 2020

If you are on Python 3 you have to do a

from functools import reduce # only in Python 3

first

Avatar image for darth88vader88

darth88vader88 on April 2, 2020

Thanks!

Avatar image for darth88vader88

darth88vader88 on April 2, 2020

Hahaha! I missed that in the first video on reduce(). Dumb!

Avatar image for jignashreddy

jignashreddy on April 3, 2020

There was an error when I used the same in lambda function. I’m not sure why?

lambda x,y:x[y.field].append(y.name) 

The error is

TypeError: 'NoneType' object is not subscriptable
Avatar image for Zarata

Zarata on May 7, 2020

In production, how do people remember that the argument to a function such as “reducer” takes an argument “map” and that the map values are list? Since Python isn’t strongly typed (is that a correct statement?) such functions will break if they aren’t called with arguments of correct type. Just something I realized / wondered as I thought about what’s going on in “reducer”, trying to wrap my head around it (your words, video midpoint).

Avatar image for Varun Vaddiparty

Varun Vaddiparty on May 10, 2020

@jignash you get that error when x is None. Most likely you are not passing in the optional argument of reduce function.

Avatar image for Ivan Smalzer

Ivan Smalzer on June 8, 2020

defaultdict is one of the things that makes Python cool.

Another way to handle expanding accumulator would be explicitly check val.fields in reducer for each object and create new key if it doesn’t exist.

Avatar image for Peter

Peter on July 3, 2020

Couldn’t the same thing be achieved with just specifying an empty dictionary as the starting value.

Something like:

def group_scientists(acc, val):
    if val.field not in acc:
        acc[val.field] = {}
    acc[val.field].append(val.name)
    return acc

Granted defaultdict is somewhat simpler.

Avatar image for Dan Bader

Dan Bader RP Team on Aug. 12, 2020

Couldn’t the same thing be achieved with just specifying an empty dictionary as the starting value.

Yep…

Granted defaultdict is somewhat simpler.

Exactly :)

Avatar image for Dan Bader

Dan Bader RP Team on Aug. 12, 2020

I wanted to address the 'NoneType' object is not subscriptable error that some of you were seeing when trying to re-implement the reducer() function as a lambda function.

Let’s assume the following setup code:

>>> import collections

>>> Scientist = collections.namedtuple('Scientist', [
...     'name',
...     'field',
...     'born',
...     'nobel',
... ])

>>> scientists = (
...     Scientist(name='Ada Lovelace', field='math', born=1815, nobel=False),
...     Scientist(name='Emmy Noether', field='math', born=1882, nobel=False),
...     Scientist(name='Marie Curie', field='math', born=1867, nobel=True),
...     Scientist(name='Tu Youyou', field='physics', born=1930, nobel=True),
...     Scientist(name='Ada Yonath', field='chemistry', born=1939, nobel=True),
...     Scientist(name='Vera Rubin', field='chemistry', born=1928, nobel=False),
...     Scientist(name='Sally Ride', field='physics', born=1951, nobel=False),
... )

>>> from functools import reduce
>>> from collections import defaultdict

Now, when I try to replace the reducer() function in the video with a lambda it’s easy to get a TypeError: 'NoneType' object is not subscriptable error:

>>> reduce(lambda acc, val: acc[val.field].append(val.name), scientists, defaultdict(list))
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    reduce(lambda acc, val: acc[val.field].append(val.name), scientists, defaultdict(list))
  File "<input>", line 1, in <lambda>
    reduce(lambda acc, val: acc[val.field].append(val.name), scientists, defaultdict(list))
TypeError: 'NoneType' object is not subscriptable

Let’s take a look at reducer() again:

def reducer(acc, val):
    acc[val.field].append(val.name)
    return acc

Notice the return acc statement at the end. That’s the problem: the lambda version has the wrong a return value, meaning it implicitly returns None, leading to the TypeError.

So, how do we fix that?

The challenge with Python lambdas is that they’re single-expression functions, meaning they can’t include return statements:

>>> reduce(lambda acc, val: acc[val.field].append(val.name); acc, scientists, defaultdict(list))
  File "<input>", line 1
    reduce(lambda acc, val: acc[val.field].append(val.name); acc, scientists, defaultdict(list))
                                                           ^
SyntaxError: invalid syntax
>>> reduce(lambda acc, val: acc[val.field].append(val.name) return acc, scientists, defaultdict(list))
  File "<input>", line 1
    reduce(lambda acc, val: acc[val.field].append(val.name) return acc, scientists, defaultdict(list))
                                                            ^
SyntaxError: invalid syntax

(For more info about lambdas in Python, check out this course: realpython.com/courses/python-lambda-functions/)

However, you can kind of “cheat” your way around this and take advantage of how Python evaluates or expressions:

>>> reduce(lambda acc, val: acc[val.field].append(val.name) or acc, scientists, defaultdict(list))
defaultdict(<class 'list'>, {'math': ['Ada Lovelace', 'Emmy Noether', 'Marie Curie'], 'physics': ['Tu Youyou', 'Sally Ride'], 'chemistry': ['Ada Yonath', 'Vera Rubin']})

That worked!

Here’s the lambda version of reducer() by itself:

lambda acc, val: acc[val.field].append(val.name) or acc

Notice the or acc at the end.

Since acc will be truthy, the expression acc[val.field].append(val.name) or acc evaluates to the value of acc.

And because lambda functions consist of a single-expression and return the value of that expression at the end…we get the desired result:

A lambda function that updates the accumulator variable (acc) and then returns the updated accumulator (via or acc).

At this point you can probably think of a few good reasons why using a standalone function (with def) is usually the better way to go…

Imagine having to explain all of this stuff every time a new colleague starts working on this piece of code ;-)

Anyway, I hope this cleared up the confusion around the 'NoneType' object is not subscriptable TypeError that some of you were getting!

Happy Pythoning!

Avatar image for Brennan Barker

Brennan Barker on Nov. 30, 2020

In keeping with this course’s emphasis on immutability, I would suggest an implementation of reducer that doesn’t rely on the list.append mutator, or indeed modify acc at all:

def reducer(acc, val): 
    return defaultdict(list, acc, **{val.field: acc[val.field] + [val.name]})

Alternatively, starting in Python 3.9 you could use the dictionary merge operator.

In addition to the benefits of testability and composability that functional programming touts, focusing on pure functions like the above instead of relying on side-effects also helps reduce a source of type of ‘NoneType’ object is not subscriptable errors Dan deals with in his last comment, because pure functions always have return values!

Become a Member to join the conversation.