Using the Python defaultdict Type for Handling Missing Keys

Grouping Items

A typical use of the Python defaultdict type is to set .default_factory to list and then build a dictionary that maps keys to lists of values. With this defaultdict, if you try to get access to any missing key, then the dictionary runs the following steps:

1. Call list() to create a new empty list
2. Insert the empty list into the dictionary using the missing key as key
3. Return a reference to that list

This allows you to write code like this:

>>> from collections import defaultdict
>>> dd = defaultdict(list)
>>> dd['key'].append(1)
>>> dd
defaultdict(<class 'list'>, {'key': [1]})
>>> dd['key'].append(2)
>>> dd
defaultdict(<class 'list'>, {'key': [1, 2]})
>>> dd['key'].append(3)
>>> dd
defaultdict(<class 'list'>, {'key': [1, 2, 3]})

Here, you create a Python defaultdict called dd and pass list to .default_factory. Notice that even when key isn’t defined, you can append values to it without getting a KeyError. That’s because dd automatically calls .default_factory to generate a default value for the missing key.

You can use defaultdict along with list to group the items in a sequence or a collection. Suppose that you’ve retrieved the following data from your company’s database:

With this data, you create an initial list of tuple objects like the following:

dep = [('Sales', 'John Doe'),
       ('Sales', 'Martin Smith'),
       ('Accounting', 'Jane Doe'),
       ('Marketing', 'Elizabeth Smith'),
       ('Marketing', 'Adam Doe')]

Now, you need to create a dictionary that groups the employees by department. To do this, you can use a defaultdict as follows:

from collections import defaultdict

dep_dd = defaultdict(list)
for department, employee in dep:
    dep_dd[department].append(employee)

Here, you create a defaultdict called dep_dd and use a for loop to iterate through your dep list. The statement dep_dd[department].append(employee) creates the keys for the departments, initializes them to an empty list, and then appends the employees to each department. Once you run this code, your dep_dd will look something like this:

defaultdict(<class 'list'>, {'Sales': ['John Doe', 'Martin Smith'],
                             'Accounting' : ['Jane Doe'],
                             'Marketing': ['Elizabeth Smith', 'Adam Doe']})

In this example, you group the employees by their department using a defaultdict with .default_factory set to list. To do this with a regular dictionary, you can use dict.setdefault() as follows:

dep_d = dict()
for department, employee in dep:
    dep_d.setdefault(department, []).append(employee)

This code is straightforward, and you’ll find similar code quite often in your work as a Python coder. However, the defaultdict version is arguably more readable, and for large datasets, it can also be a lot faster and more efficient. So, if speed is a concern for you, then you should consider using a defaultdict instead of a standard dict.

Grouping Unique Items

Continue working with the data of departments and employees from the previous section. After some processing, you realize that a few employees have been duplicated in the database by mistake. You need to clean up the data and remove the duplicated employees from your dep_dd dictionary. To do this, you can use a set as the .default_factory and rewrite your code as follows:

dep = [('Sales', 'John Doe'),
       ('Sales', 'Martin Smith'),
       ('Accounting', 'Jane Doe'),
       ('Marketing', 'Elizabeth Smith'),
       ('Marketing', 'Elizabeth Smith'),
       ('Marketing', 'Adam Doe'),
       ('Marketing', 'Adam Doe'),
       ('Marketing', 'Adam Doe')]

dep_dd = defaultdict(set)
for department, employee in dep:
    dep_dd[department].add(employee)

In this example, you set .default_factory to set. Sets are collections of unique objects, which means that you can’t create a set with repeated items. This is a really interesting feature of sets, which guarantees that you won’t have repeated items in your final dictionary.

Counting Items

If you set .default_factory to int, then your defaultdict will be useful for counting the items in a sequence or collection. When you call int() with no arguments, the function returns 0, which is the typical value you’d use to initialize a counter.

To continue with the example of the company database, suppose you want to build a dictionary that counts the number of employees per department. In this case, you can code something like this:

>>> from collections import defaultdict
>>> dep = [('Sales', 'John Doe'),
...        ('Sales', 'Martin Smith'),
...        ('Accounting', 'Jane Doe'),
...        ('Marketing', 'Elizabeth Smith'),
...        ('Marketing', 'Adam Doe')]
>>> dd = defaultdict(int)
>>> for department, _ in dep:
...     dd[department] += 1
>>> dd
defaultdict(<class 'int'>, {'Sales': 2, 'Accounting': 1, 'Marketing': 2})

Here, you set .default_factory to int. When you call int() with no argument, the returned value is 0. You can use this default value to start counting the employees that work in each department. For this code to work correctly, you need a clean dataset. There must be no repeated data. Otherwise, you’ll need to filter out the repeated employees.

Another example of counting items is the mississippi example, where you count the number of times each letter in a word is repeated. Take a look at the following code:

>>> from collections import defaultdict
>>> s = 'mississippi'
>>> dd = defaultdict(int)
>>> for letter in s:
...     dd[letter] += 1
...
>>> dd
defaultdict(<class 'int'>, {'m': 1, 'i': 4, 's': 4, 'p': 2})

In the above code, you create a defaultdict with .default_factory set to int. This sets the default value for any given key to 0. Then, you use a for loop to traverse the string s and use an augmented assignment operation to add 1 to the counter in every iteration. The keys of dd will be the letters in mississippi.

As counting is a relatively common task in programming, the Python dictionary-like class collections.Counter is specially designed for counting items in a sequence. With Counter, you can write the mississippi example as follows:

>>> from collections import Counter
>>> counter = Counter('mississippi')
>>> counter
Counter({'i': 4, 's': 4, 'p': 2, 'm': 1})

In this case, Counter does all the work for you! You only need to pass in a sequence, and the dictionary will count its items, storing them as keys and the counts as values. Note that this example works because Python strings are also a sequence type.

Accumulating Values

Sometimes you’ll need to calculate the total sum of the values in a sequence or collection. Let’s say you have the following Excel sheet with data about the sales of your Python website:

Next, you process the data using Python and get the following list of tuple objects:

incomes = [('Books', 1250.00),
           ('Books', 1300.00),
           ('Books', 1420.00),
           ('Tutorials', 560.00),
           ('Tutorials', 630.00),
           ('Tutorials', 750.00),
           ('Courses', 2500.00),
           ('Courses', 2430.00),
           ('Courses', 2750.00),]

With this data, you want to calculate the total income per product. To do that, you can use a Python defaultdict with float as .default_factory and then code something like this:

 1from collections import defaultdict
 2
 3dd = defaultdict(float)
 4for product, income in incomes:
 5    dd[product] += income
 6
 7for product, income in dd.items():
 8    print(f'Total income for {product}: ${income:,.2f}')

Here’s what this code does:

• In line 1, you import the Python defaultdict type.
• In line 3, you create a defaultdict object with .default_factory set to float.
• In line 4, you define a for loop to iterate through the items of incomes.
• In line 5, you use an augmented assignment operation (+=) to accumulate the incomes per product in the dictionary.

The second loop iterates through the items of dd and prints the incomes to your screen.

If you put all this code into a file called incomes.py and run it from your command line, then you’ll get the following output:

$ python3 incomes.py
Total income for Books: $3,970.00
Total income for Tutorials: $1,940.00
Total income for Courses: $7,680.00

You now have a summary of incomes per product, so you can make decisions on which strategy to follow for increasing the total income of your site.

defaultdict vs dict

For you to better understand the Python defaultdict type, a good exercise would be to compare it with its superclass, dict. If you want to know the methods and attributes that are specific to the Python defaultdict type, then you can run the following line of code:

>>> set(dir(defaultdict)) - set(dir(dict))
{'__copy__', 'default_factory', '__missing__'}

In the above code, you use dir() to get the list of valid attributes for dict and defaultdict. Then, you use a set difference to get the set of methods and attributes that you can only find in defaultdict. As you can see, the differences between these two classes are. You have two methods and one instance attribute. The following table shows what the methods and the attribute are for:

In the above table, you can see the methods and the attribute that make a defaultdict different from a regular dict. The rest of the methods are the same in both classes.

Additionally, you might notice that a defaultdict is equal to a dict with the same items:

>>> std_dict = dict(numbers=[1, 2, 3], letters=['a', 'b', 'c'])
>>> std_dict
{'numbers': [1, 2, 3], 'letters': ['a', 'b', 'c']}
>>> def_dict = defaultdict(list, numbers=[1, 2, 3], letters=['a', 'b', 'c'])
>>> def_dict
defaultdict(<class 'list'>, {'numbers': [1, 2, 3], 'letters': ['a', 'b', 'c']})
>>> std_dict == def_dict
True

Here, you create a regular dictionary std_dict with some arbitrary items. Then, you create a defaultdict with the same items. If you test both dictionaries for content equality, then you’ll see that they’re equal.

defaultdict.default_factory

The first argument to the Python defaultdict type must be a callable that takes no arguments and returns a value. This argument is assigned to the instance attribute, .default_factory. For this, you can use any callable, including functions, methods, classes, type objects, or any other valid callable. The default value of .default_factory is None.

If you instantiate defaultdict without passing a value to .default_factory, then the dictionary will behave like a regular dict and the usual KeyError will be raised for missing key lookup or modification attempts:

>>> from collections import defaultdict
>>> dd = defaultdict()
>>> dd['missing_key']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    dd['missing_key']
KeyError: 'missing_key'

Here, you instantiate the Python defaultdict type with no arguments. In this case, the instance behaves like a standard dictionary. So, if you try to access or modify a missing key, then you’ll get the usual KeyError. From this point on, you can use dd as a normal Python dictionary and, unless you assign a new callable to .default_factory, you won’t be able to use the ability of defaultdict to handle missing keys automatically.

If you pass None to the first argument of defaultdict, then the instance will behave the same way you saw in the above example. That’s because .default_factory defaults to None, so both initializations are equivalent. On the other hand, if you pass a valid callable object to .default_factory, then you can use it to handle missing keys in a user-friendly way. Here’s an example where you pass list to .default_factory:

>>> dd = defaultdict(list, letters=['a', 'b', 'c'])
>>> dd.default_factory
<class 'list'>
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c']})
>>> dd['numbers']
[]
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c'], 'numbers': []})
>>> dd['numbers'].append(1)
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c'], 'numbers': [1]})
>>> dd['numbers'] += [2, 3]
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c'], 'numbers': [1, 2, 3]})

In this example, you create a Python defaultdict called dd, then you use list for its first argument. The second argument is called letters and holds a list of letters. You see that .default_factory now holds a list object that will be called when you need to supply a default value for any missing key.

Notice that when you try to access numbers, dd tests if numbers is in the dictionary. If it’s not, then it calls .default_factory(). Since .default_factory holds a list object, the returned value is an empty list ([]).

Now that dd['numbers'] is initialized with an empty list, you can use .append() to add elements to the list. You can also use an augmented assignment operator (+=) to concatenate the lists [1] and [2, 3]. This way, you can handle missing keys in a more Pythonic and more efficient way.

On the other hand, if you pass a non-callable object to the initializer of the Python defaultdict type, then you’ll get a TypeError like in the following code:

>>> defaultdict(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    defaultdict(0)
TypeError: first argument must be callable or None

Here, you pass 0 to .default_factory. Since 0 is not a callable object, you get a TypeError telling you that the first argument must be callable or None. Otherwise, defaultdict doesn’t work.

Keep in mind that .default_factory is only called from .__getitem__() and not from other methods. This means that if dd is a defaultdict and key is a missing key, then dd[key] will call .default_factory to provide a default value, but dd.get(key) still returns None instead of the value that .default_factory would provide. That’s because .get() doesn’t call .__getitem__() to retrieve the key.

Take a look at the following code:

>>> dd = defaultdict(list)
>>> # Calls dd.__getitem__('missing')
>>> dd['missing']
[]
>>> # Don't call dd.__getitem__('another_missing')
>>> print(dd.get('another_missing'))
None
>>> dd
defaultdict(<class 'list'>, {'missing': []})

In this code fragment, you can see that dd.get() returns None rather than the default value that .default_factory would provide. That’s because .default_factory is only called from .__missing__(), which is not called by .get().

Notice that you can also add arbitrary values to a Python defaultdict. This means that you’re not limited to values with the same type as the values generated by .default_factory. Here’s an example:

>>> dd = defaultdict(list)
>>> dd
defaultdict(<class 'list'>, {})
>>> dd['string'] = 'some string'
>>> dd
defaultdict(<class 'list'>, {'string': 'some string'})
>>> dd['list']
[]
>>> dd
defaultdict(<class 'list'>, {'string': 'some string', 'list': []})

Here, you create a defaultdict and pass in a list object to .default_factory. This sets your default values to be empty lists. However, you can freely add a new key that holds values of a different type. That’s the case with the key string, which holds a str object instead of a list object.

Finally, you can always change or update the callable you initially assign to .default_factory in the same way you would do with any instance attribute:

>>> dd.default_factory = str
>>> dd['missing_key']
''

In the above code, you change .default_factory from list to str. Now, whenever you try to get access to a missing key, your default value will be an empty string ('').

Depending on your use cases for the Python defaultdict type, you might need to freeze the dictionary once you finish creating it and make it read-only. To do this, you can set .default_factory to None after you finish populating the dictionary. This way, your dictionary will behave like a standard dict, which means you won’t have more automatically generated default values.

defaultdict vs dict.setdefault()

As you saw before, dict provides .setdefault(), which will allow you to assign values to missing keys on the fly. In contrast, with a defaultdict you can specify the default value up front when you initialize the container. You can use .setdefault() to assign default values as follows:

>>> d = dict()
>>> d.setdefault('missing_key', [])
[]
>>> d
{'missing_key': []}

In this code, you create a regular dictionary and then use .setdefault() to assign a value ([]) to the key missing_key, which wasn’t defined yet.

On the other hand, if you use a defaultdict to accomplish the same task, then the default value is generated on demand whenever you try to access or modify a missing key. Notice that, with defaultdict, the default value is generated by the callable you pass up front to the initializer of the class. Here’s how it works:

>>> from collections import defaultdict
>>> dd = defaultdict(list)
>>> dd['missing_key']
[]
>>> dd
defaultdict(<class 'list'>, {'missing_key': []})

Here, you first import the Python defaultdict type from collections. Then, you create a defaultdict and pass list to .default_factory. When you try to get access to a missing key, defaultdict internally calls .default_factory(), which holds a reference to list, and assigns the resulting value (an empty list) to missing_key.

The code in the above two examples does the same work, but the defaultdict version is arguably more readable, user-friendly, Pythonic, and straightforward.

>>> list()
[]
>>> set()
set([])
>>> dict()
{}
>>> str()
''
>>> float()
0.0
>>> int()
0

Finally, using a defaultdict to handle missing keys can be faster than using dict.setdefault(). Take a look a the following example:

# Filename: exec_time.py

from collections import defaultdict
from timeit import timeit

animals = [('cat', 1), ('rabbit', 2), ('cat', 3), ('dog', 4), ('dog', 1)]
std_dict = dict()
def_dict = defaultdict(list)

def group_with_dict():
    for animal, count in animals:
        std_dict.setdefault(animal, []).append(count)
    return std_dict

def group_with_defaultdict():
    for animal, count in animals:
        def_dict[animal].append(count)
    return def_dict

print(f'dict.setdefault() takes {timeit(group_with_dict)} seconds.')
print(f'defaultdict takes {timeit(group_with_defaultdict)} seconds.')

If you run the script from your system’s command line, then you’ll get something like this:

$ python3 exec_time.py
dict.setdefault() takes 1.0281260240008123 seconds.
defaultdict takes 0.6704721650003194 seconds.

Here, you use timeit.timeit() to measure the execution time of group_with_dict() and group_with_defaultdict(). These functions perform equivalent actions, but the first uses dict.setdefault(), and the second uses a defaultdict. The time measure will depend on your current hardware, but you can see here that defaultdict is faster than dict.setdefault(). This difference can become more important as the dataset gets larger.

Additionally, you need to consider that creating a regular dict can be faster than creating a defaultdict. Take a look at this code:

>>> from timeit import timeit
>>> from collections import defaultdict
>>> print(f'dict() takes {timeit(dict)} seconds.')
dict() takes 0.08921320698573254 seconds.
>>> print(f'defaultdict() takes {timeit(defaultdict)} seconds.')
defaultdict() takes 0.14101867799763568 seconds.

This time, you use timeit.timeit() to measure the execution time of dict and defaultdict instantiation. Notice that creating a dict takes almost half the time of creating a defaultdict. This might not be a problem if you consider that, in real-world code, you normally instantiate defaultdict only once.

Also notice that, by default, timeit.timeit() will run your code a million times. That’s the reason for defining std_dict and def_dict out of the scope of group_with_dict() and group_with_defaultdict() in exec_time.py. Otherwise, the time measure will be affected by the instantiation time of dict and defaultdict.

At this point, you may have an idea of when to use a defaultdict rather than a regular dict. Here are three things to take into account:

1. If your code is heavily base on dictionaries and you’re dealing with missing keys all the time, then you should consider using a defaultdict rather than a regular dict.

2. If your dictionary items need to be initialized with a constant default value, then you should consider using a defaultdict instead of a dict.

3. If your code relies on dictionaries for aggregating, accumulating, counting, or grouping values, and performance is a concern, then you should consider using a defaultdict.

You can consider the above guidelines when deciding whether to use a dict or a defaultdict.

defaultdict.__missing__()

Behind the scenes, the Python defaultdict type works by calling .default_factory to supply default values to missing keys. The mechanism that makes this possible is .__missing__(), a special method supported by all the standard mapping types, including dict and defaultdict.

So, how does .__missing__() work? If you set .default_factory to None, then .__missing__() raises a KeyError with the key as an argument. Otherwise, .default_factory is called without arguments to provide a default value for the given key. This value is inserted into the dictionary and finally returned. If calling .default_factory raises an exception, then the exception is propagated unchanged.

The following code shows a viable Python implementation for .__missing__():

 1def __missing__(self, key):
 2    if self.default_factory is None:
 3        raise KeyError(key)
 4    if key not in self:
 5        self[key] = self.default_factory()
 6    return self[key]

Here’s what this code does:

• In line 1, you define the method and its signature.
• In lines 2 and 3, you test to see if .default_factory is None. If so, then you raise a KeyError with the key as an argument.
• In lines 4 and 5, you check if the key is not in the dictionary. If it’s not, then you call .default_factory and assign its return value to the key.
• In line 6, you return the key as expected.

Keep in mind that the presence of .__missing__() in a mapping has no effect on the behavior of other methods that look up keys, such as .get() or .__contains__(), which implements the in operator. That’s because .__missing__() is only called by .__getitem__() when the requested key is not found in the dictionary. Whatever .__missing__() returns or raises is then returned or raised by .__getitem__().

Now that you’ve covered an alternative Python implementation for .__missing__(), it would be a good exercise to try to emulate defaultdict with some Python code. That’s what you’ll be doing in the next section.

Using lambda

A flexible way to pass arguments to .default_factory is to use lambda. Suppose you want to create a function to generate default values in a defaultdict. The function does some processing and returns a value, but you need to pass an argument for the function to work correctly. Here’s an example:

>>> def factory(arg):
...     # Do some processing here...
...     result = arg.upper()
...     return result
...
>>> def_dict = defaultdict(lambda: factory('default value'))
>>> def_dict['missing']
'DEFAULT VALUE'

In the above code, you create a function called factory(). The function takes an argument, does some processing, and returns the final result. Then, you create a defaultdict and use lambda to pass the string 'default value' to factory(). When you try to get access to a missing key, the following steps are run:

1. The dictionary def_dict calls its .default_factory, which holds a reference to a lambda function.
2. The lambda function gets called and returns the value that results from calling factory() with 'default value' as an argument.

If you’re working with def_dict and suddenly need to change the argument to factory(), then you can do something like this:

>>> def_dict.default_factory = lambda: factory('another default value')
>>> def_dict['another_missing']
'ANOTHER DEFAULT VALUE'

This time, factory() takes a new string argument ('another default value'). From now on, if you try to access or modify a missing key, then you’ll get a new default value, which is the string 'ANOTHER DEFAULT VALUE'.

Finally, you can possibly face a situation where you need a default value that’s different from 0 or []. In this case, you can also use lambda to generate a different default value. For example, suppose you have a list of integer numbers, and you need to calculate the cumulative product of each number. Then, you can use a defaultdict along with lambda as follows:

>>> from collections import defaultdict
>>> lst = [1, 1, 2, 1, 2, 2, 3, 4, 3, 3, 4, 4]
>>> def_dict = defaultdict(lambda: 1)
>>> for number in lst:
...     def_dict[number] *= number
...
>>> def_dict
defaultdict(<function <lambda> at 0x...70>, {1: 1, 2: 8, 3: 27, 4: 64})

Here, you use lambda to supply a default value of 1. With this initial value, you can calculate the cumulative product of each number in lst. Notice that you can’t get the same result using int because the default value returned by int is always 0, which is not a good initial value for the multiplication operations you need to perform here.

Using functools.partial()

functools.partial(func, *args, **keywords) is a function that returns a partial object. When you call this object with the positional arguments (args) and keyword arguments (keywords), it behaves similar to when you call func(*args, **keywords). You can take advantage of this behavior of partial() and use it to pass arguments to .default_factory in a Python defaultdict. Here’s an example:

>>> def factory(arg):
...     # Do some processing here...
...     result = arg.upper()
...     return result
...
>>> from functools import partial
>>> def_dict = defaultdict(partial(factory, 'default value'))
>>> def_dict['missing']
'DEFAULT VALUE'
>>> def_dict.default_factory = partial(factory, 'another default value')
>>> def_dict['another_missing']
'ANOTHER DEFAULT VALUE'

Here, you create a Python defaultdict and use partial() to supply an argument to .default_factory. Notice that you can also update .default_factory to use another argument for the callable factory(). This kind of behavior can add a lot of flexibility to your defaultdict objects.