Why Use the reduce() Function?

Dan, What is the difference between functional and procedural programming?

Waw, I use the Counter function from collections as a means to the same end. Didn’t realize there was a defaultdict. From now on I’ll just use that instead.

I get this error with bpython console:

>>> total_age = reduce(lambda acc,val: acc + val['age'],names_and_ages,0)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    total_age = reduce(lambda acc,val: acc + val['age'],names_and_ages,0)
NameError: name 'reduce' is not defined

If you are on Python 3 you have to do a

from functools import reduce # only in Python 3

first

Thanks!

Hahaha! I missed that in the first video on reduce(). Dumb!

There was an error when I used the same in lambda function. I’m not sure why?

lambda x,y:x[y.field].append(y.name) 

The error is

TypeError: 'NoneType' object is not subscriptable

In production, how do people remember that the argument to a function such as “reducer” takes an argument “map” and that the map values are list? Since Python isn’t strongly typed (is that a correct statement?) such functions will break if they aren’t called with arguments of correct type. Just something I realized / wondered as I thought about what’s going on in “reducer”, trying to wrap my head around it (your words, video midpoint).

@jignash you get that error when x is None. Most likely you are not passing in the optional argument of reduce function.

defaultdict is one of the things that makes Python cool.

Another way to handle expanding accumulator would be explicitly check val.fields in reducer for each object and create new key if it doesn’t exist.

Couldn’t the same thing be achieved with just specifying an empty dictionary as the starting value.

Something like:

def group_scientists(acc, val):
    if val.field not in acc:
        acc[val.field] = {}
    acc[val.field].append(val.name)
    return acc

Granted defaultdict is somewhat simpler.

Couldn’t the same thing be achieved with just specifying an empty dictionary as the starting value.

Yep…

Granted defaultdict is somewhat simpler.

Exactly :)

I wanted to address the 'NoneType' object is not subscriptable error that some of you were seeing when trying to re-implement the reducer() function as a lambda function.

Let’s assume the following setup code:

>>> import collections

>>> Scientist = collections.namedtuple('Scientist', [
...     'name',
...     'field',
...     'born',
...     'nobel',
... ])

>>> scientists = (
...     Scientist(name='Ada Lovelace', field='math', born=1815, nobel=False),
...     Scientist(name='Emmy Noether', field='math', born=1882, nobel=False),
...     Scientist(name='Marie Curie', field='math', born=1867, nobel=True),
...     Scientist(name='Tu Youyou', field='physics', born=1930, nobel=True),
...     Scientist(name='Ada Yonath', field='chemistry', born=1939, nobel=True),
...     Scientist(name='Vera Rubin', field='chemistry', born=1928, nobel=False),
...     Scientist(name='Sally Ride', field='physics', born=1951, nobel=False),
... )

>>> from functools import reduce
>>> from collections import defaultdict

Now, when I try to replace the reducer() function in the video with a lambda it’s easy to get a TypeError: 'NoneType' object is not subscriptable error:

>>> reduce(lambda acc, val: acc[val.field].append(val.name), scientists, defaultdict(list))
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    reduce(lambda acc, val: acc[val.field].append(val.name), scientists, defaultdict(list))
  File "<input>", line 1, in <lambda>
    reduce(lambda acc, val: acc[val.field].append(val.name), scientists, defaultdict(list))
TypeError: 'NoneType' object is not subscriptable

Let’s take a look at reducer() again:

def reducer(acc, val):
    acc[val.field].append(val.name)
    return acc

Notice the return acc statement at the end. That’s the problem: the lambda version has the wrong a return value, meaning it implicitly returns None, leading to the TypeError.

So, how do we fix that?

The challenge with Python lambdas is that they’re single-expression functions, meaning they can’t include return statements:

>>> reduce(lambda acc, val: acc[val.field].append(val.name); acc, scientists, defaultdict(list))
  File "<input>", line 1
    reduce(lambda acc, val: acc[val.field].append(val.name); acc, scientists, defaultdict(list))
                                                           ^
SyntaxError: invalid syntax
>>> reduce(lambda acc, val: acc[val.field].append(val.name) return acc, scientists, defaultdict(list))
  File "<input>", line 1
    reduce(lambda acc, val: acc[val.field].append(val.name) return acc, scientists, defaultdict(list))
                                                            ^
SyntaxError: invalid syntax

(For more info about lambdas in Python, check out this course: realpython.com/courses/python-lambda-functions/)

However, you can kind of “cheat” your way around this and take advantage of how Python evaluates or expressions:

>>> reduce(lambda acc, val: acc[val.field].append(val.name) or acc, scientists, defaultdict(list))
defaultdict(<class 'list'>, {'math': ['Ada Lovelace', 'Emmy Noether', 'Marie Curie'], 'physics': ['Tu Youyou', 'Sally Ride'], 'chemistry': ['Ada Yonath', 'Vera Rubin']})

That worked!

Here’s the lambda version of reducer() by itself:

lambda acc, val: acc[val.field].append(val.name) or acc

Notice the or acc at the end.

Since acc will be truthy, the expression acc[val.field].append(val.name) or acc evaluates to the value of acc.

And because lambda functions consist of a single-expression and return the value of that expression at the end…we get the desired result:

A lambda function that updates the accumulator variable (acc) and then returns the updated accumulator (via or acc).

At this point you can probably think of a few good reasons why using a standalone function (with def) is usually the better way to go…

Imagine having to explain all of this stuff every time a new colleague starts working on this piece of code ;-)

Anyway, I hope this cleared up the confusion around the 'NoneType' object is not subscriptable TypeError that some of you were getting!

Happy Pythoning!

In keeping with this course’s emphasis on immutability, I would suggest an implementation of reducer that doesn’t rely on the list.append mutator, or indeed modify acc at all:

def reducer(acc, val): 
    return defaultdict(list, acc, **{val.field: acc[val.field] + [val.name]})

Alternatively, starting in Python 3.9 you could use the dictionary merge operator.

In addition to the benefits of testability and composability that functional programming touts, focusing on pure functions like the above instead of relying on side-effects also helps reduce a source of type of ‘NoneType’ object is not subscriptable errors Dan deals with in his last comment, because pure functions always have return values!

Hi Dan,

Can you please add some explanation why you need to explicitly return acc in your reducer function? Since acc is a mutable dictionary in your example, it would be updated/mutated even without the return statement. I think I know the answer but I think this is an interesting bit that would deserve some attention.

Great course btw! Thank you.

@Andras In general, relying on side effects is less explicit than passing values around. More importantly, however, the functools.reduce() function calls your binary reducer, expecting that it returns the reduced value to be accumulated in each iteration. Had the reducer not returned anything, then it would return None implicitly, which would then be passed into it later as the accumulator, causing an error since you can’t subscript None.