Metaclass Usage in Enum

00:00 In this lesson, I’ll cover Python’s Enum, a metaclass usage in the standard library.

00:07 In the standard library, you’ll find the enum (small e) module containing the Enum (capital E) class. Enum was kind of a late addition to Python, and unlike most languages, where it’s a fundamental data type, Python has used metaclass magic to implement it.

00:24 The metaclass in question is called EnumType. I showed you the .__new__() hook in a metaclass, but that’s not the only hook. The changes to how metaclasses work that were introduced in Python 3 include the addition of the .__prepare__() method.

00:40 This method is called before .__new__() and is used as a hook for the attribute dictionary passed to .__new__().

00:49 It gets called first with the attribute dictionary passed in, and what it returns is what gets passed to .__new__(). This allows you to make changes to the arguments before they’re processed by .__new__().

01:01 The EnumType metaclass uses .__prepare__() to create the EnumDict class. This is a hand-crafted ordered dictionary.

01:11 Remember, in Python 3.0, dictionaries weren’t ordered by default. This has some special features to deal with the naming enforced for Enum member attributes and also gets used internally to store the member attributes themselves.

01:28 Consider an Enum describing some fruit. You define this by inheriting the Enum class. The Enum base class uses the EnumType metaclass,

01:41 which of course inherits from type. EnumType uses .__prepare__() to modify the argument dictionary. The modification it makes is to convert it to a dictionary subclass called EnumDict.

01:55 Fruit is an Enum, Enum is an EnumType, and EnumType has an EnumDict. The sample code includes a copy of this module renamed as py_enum. That way if you want to dig into it yourself, you can do that without having to go find it on your own machine.

02:15 I renamed it to avoid namespace conflicts in case you want to actually run it yourself. When I first dug into this, I stuck a breakpoint in pretty much every dunder method in EnumType to better understand what was going on. All right, let’s go take a look at the actual code.

02:35 This is part of the EnumType metaclass from the standard library’s enum module. The .__prepare__() method is called as a hook for the creation of the class attributes dictionary.

02:46 Because a class that inherits from Enum, can be inherited from, the first thing done here is to check all the base classes for attributes. It then creates the special EnumDict class, which is a subclass of dictionary, and it then examines the base classes for information.

03:05 The Enum class supports something called sunder values. These are like dunder values, but with a single underscore (_). A sunder value exposes special information about parts of the enum.

03:17 Think back to my fruit example. I can do Fruit.apple._name_, and it returns apple as a string, or Fruit.pear._value_, and it returns 2, the value of pear.

03:33 Theses values get used for a bunch of internal mechanisms as well, and if this isn’t the first enum to be created, the _generate_next_value_ sunder value is copied from the previous enum.

03:44 This sunder is responsible for creating values when the auto feature is used, and you only want one of these, or you’ll get two attributes with the same value.

03:54 .__prepare__() is responsible for returning the attribute dictionary. In this case, it’s created a new kind of dictionary, enum_dict, and returns that instead of the original.

04:06 Are you ready for this? Hold your breath. It’s a deep dive.

04:22 That’s a lot of code, huh? You know what’s scary? That’s only the .__new__() method. It’s just a few lines shy of being 200 long. Let me scroll back up at the top here. Still scrolling.

04:38 Ah, there’s the top. Remember, this is just a single method inside a single metaclass inside the enum module. After .__prepare__() has prepared the argument dictionary, it calls .__new__() on the metaclass.

04:53 The arguments to .__new__() here start with class inheritance and the arguments dictionary as before, but it also adds a few more for special cases of enums themselves.

05:05 One of those special cases is an additional argument called _simple. This flag is set to True for a subset of the Enum case called SimpleEnum.

05:15 Simple enums are created when a utility decorator wraps a plain class, turning the class into an Enum. This case requires special handling, but most of the handling is done in that decorator itself, so .__new__() only needs to create the class. See?

05:33 I’m already dealing with corner conditions, and I’m only on the first line of the method. Enum supports a special sunder value called _ignore.

05:43 This allows you to define attributes in the class that don’t participate in the actual Enum. This can be handy if you’re generating attributes on the fly and you need some values to do your computation, but don’t want them in the end result.

05:57 This chunk of code removes the _ignore items from the attributes dictionary.

06:03 For readability, the ._member_names attribute of the EnumDict is re-referenced as member_names here. Enums have rules about what you’re allowed to call the attributes.

06:15 This line defines what you’re not allowed to call them and checks for any intersection with the names you’ve defined in your Enum. Empty string isn’t allowed, and mro is a pseudo-keyword that gets used in class hierarchies, and using it as an attribute would break lots of things, so it’s disallowed.

06:34 If any invalid names are detected, an error is raised. Let me scroll down here.

06:41 Remember when I spoke about those special sunder values? Well, the next bit of code does some cleanup on them. This same method was called in .__prepare__(), and it gets information about the inheritance hierarchy.

06:54 Our class might be inherited from a class that inherits from Enum. With the base class info in hand, this method tries to figure out which .__new__() method to use for the instance.

07:06 There are multiple choices because of inheritance, and it can get even messier if you’re pickling things. I’m not going to go into details here. Just know that the code has to figure out which method is the correct one to assign. Once it’s picked one, it assigns it to a sunder value for later use.

07:25 The next chunk is yet another special case. The Flag class is a kind of Enum where all the attribute values are treated as binary flags.

07:33 This allows you to do binary operations on flags like bitwise OR. This code does some work with the attributes to make sure they can be used in a flag.

07:44 Scrolling down again …

07:47 The next few lines define a bunch of defaults for some of the sunder values.

07:53 And like the need to go searching for .__new__(), this goes searching for the right .__repr__() method.

08:00 More flag-handling special code … and finally (well, not so finally, actually), this is where the class is actually created. Phew, it’s a lot, huh? And there’s 120 more lines of code just for special case handling after this.

08:18 This is the core of it though. Process the attributes passed in to make sure they’re unique and have allowed names, set up some sunder values, make sure to use the right .__new__() and .__repr__() methods if you’ve got multi-inheritance going on, and then you’re actually creating the class.

08:36 Let’s go see how this metaclass gets used.

08:44 This is still from the enum module. It’s the actual Enum class. Note how it uses the EnumType in its inheritance declaration.

08:54 The Enum class also overrides its .__new__() method. Remember, this is the object’s .__new__(), not the class’s.

09:03 So why does it do that? Well, you can get an Enum attribute by instantiating an object of the class, passing in its value. It’s kind of like overloading the constructor.

09:13 The normal Enum-y things are handled by the metaclass, but the .__new__() of the object handles this way of constructing a member value.

09:22 The example in the comments here is constructing the Color Enum, passing in the integer 3. What gets returned in that case is the actual Color.RED Enum attribute.

09:33 If what is being passed in is a class member itself, then a class member gets returned. It’s sort of the short-circuit response. If what gets passed in is a value like 3, then the dictionary where the values are stored is checked for the corresponding attribute.

09:50 As the things in the Enum don’t have to be hashable, there’s a fallback situation here where all the attributes are walked through. I find this code interesting.

09:59 It makes me wonder whether it’s a premature optimization. The vast majority of enums I’ve come across have so few attributes that sticking with the order-n search probably would be good enough.

10:10 Be interesting to hack this up and test out how much difference is actually made efficiency-wise. There are still a couple of more corner cases to go, but you get the idea.

10:20 The Enum class uses the EnumType as a metaclass to do all the attribute magic, and the Enum class also overrides the object .__new__() to handle the attribute instantiation case. Quite the ride, huh?

10:38 If you’ve made it this far, congratulations. It’s kind of simpler from here on in. Next up, I’ll summarize the course and point you at some further investigation.