Python's Magic Methods: Leverage Their Power in Your Classes

Python's Magic Methods: Leverage Their Power in Your Classes

by Leodanis Pozo Ramos Jan 03, 2024 advanced python

As a Python developer who wants to harness the power of object-oriented programming, you’ll love to learn how to customize your classes using special methods, also known as magic methods or dunder methods. A special method is a method whose name starts and ends with a double underscore. These methods have special meanings for Python.

Python automatically calls magic methods as a response to certain operations, such as instantiation, sequence indexing, attribute managing, and much more. Magic methods support core object-oriented features in Python, so learning about them is fundamental for you as a Python programmer.

In this tutorial, you’ll:

  • Learn what Python’s special or magic methods are
  • Understand the magic behind magic methods in Python
  • Customize different behaviors of your custom classes with special methods

To get the most out of this tutorial, you should be familiar with general Python programming. More importantly, you should know the basics of object-oriented programming and classes in Python.

Getting to Know Python’s Magic or Special Methods

In Python, special methods are also called magic methods, or dunder methods. This latter terminology, dunder, refers to a particular naming convention that Python uses to name its special methods and attributes. The convention is to use double leading and trailing underscores in the name at hand, so it looks like .__method__().

The double underscores flag these methods as core to some Python features. They help avoid name collisions with your own methods and attributes. Some popular and well-known magic methods include the following:

Special Method Description
.__init__() Provides an initializer in Python classes
.__str__() and .__repr__() Provide string representations for objects
.__call__() Makes the instances of a class callable
.__len__() Supports the len() function

This is just a tiny sample of all the special methods that Python has. All these methods support specific features that are core to Python and its object-oriented infrastructure.

Here’s how the Python documentation defines the term special methods:

A method that is called implicitly by Python to execute a certain operation on a type, such as addition. Such methods have names starting and ending with double underscores. (Source)

There’s an important detail to highlight in this definition. Python implicitly calls special methods to execute certain operations in your code. For example, when you run the addition 5 + 2 in a REPL session, Python internally runs the following code under the hood:

Python
>>> (5).__add__(2)
7

The .__add__() special method of integer numbers supports the addition that you typically run as 5 + 2.

Reading between the lines, you’ll realize that even though you can directly call special methods, they’re not intended for direct use. You shouldn’t call them directly in your code. Instead, you should rely on Python to call them automatically in response to a given operation.

Magic methods exist for many purposes. All the available magic methods support built-in features and play specific roles in the language. For example, built-in types such as lists, strings, and dictionaries implement most of their core functionality using magic methods. In your custom classes, you can use magic methods to make callable objects, define how objects are compared, tweak how you create objects, and more.

Note that because magic methods have special meaning for Python itself, you should avoid naming custom methods using leading and trailing double underscores. Your custom method won’t trigger any Python action if its name doesn’t match any official special method names, but it’ll certainly confuse other programmers. New dunder names may also be introduced in future versions of Python.

Magic methods are core to Python’s data model and are a fundamental part of object-oriented programming in Python. In the following sections, you’ll learn about some of the most commonly used special methods. They’ll help you write better object-oriented code in your day-to-day programming adventure.

Controlling the Object Creation Process

When creating custom classes in Python, probably the first and most common method that you implement is .__init__(). This method works as an initializer because it allows you to provide initial values to any instance attributes that you define in your classes.

The .__new__() special method also has a role in the class instantiation process. This method takes care of creating a new instance of a given class when you call the class constructor. The .__new__() method is less commonly implemented in practice, though.

In the following sections, you’ll learn the basics of how these two methods work and how you can use them to customize the instantiation of your classes.

Initializing Objects With .__init__()

Python calls the .__init__() method whenever you call the constructor of a given class. The goal of .__init__() is to initialize any instance attribute that you have in your class. Consider the following Point class:

Python
>>> class Point:
...     def __init__(self, x, y):
...         self.x = x
...         self.y = y
...

>>> point = Point(21, 42)
>>> point.x
21
>>> point.y
42

When you create the point instance by calling the class constructor, Point(), Python automatically calls .__init__() under the hood using the same arguments that you’ve passed to the constructor. You don’t have to call .__init__() by yourself. You just rely on Python’s implicit behavior. Note how the .x and .y attributes hold the values that you pass in when you call the constructor.

Creating Objects With .__new__()

When you call a class constructor to create a new instance of a class, Python implicitly calls the .__new__() method as the first step in the instantiation process. This method is responsible for creating and returning a new empty object of the underlying class. Python then passes the just-created object to .__init__() for initialization.

The default implementation of .__new__() is enough for most practical use cases. So, you probably won’t need to write a custom implementation of .__new__() in most cases.

However, you’ll find that this method is useful in some advanced use cases. For example, you can use .__new__() to create subclasses of immutable types, such as int, float, tuple, and str. Consider the following example of a class that inherits from the built-in float data type:

Python
>>> class Storage(float):
...     def __new__(cls, value, unit):
...         instance = super().__new__(cls, value)
...         instance.unit = unit
...         return instance
...

In this example, you’ll note that .__new__() is a class method because it gets the current class (cls) rather than the current instance (self) as an argument.

Then, you run three steps. First, you create a new instance of the current class, cls, by calling .__new__() on the float class through the built-in super() function. This call creates a new instance of float and initializes it using value as an argument.

Then, you customize the new instance by dynamically attaching a .unit attribute to it. Finally, you return the new instance to meet the default behavior of .__new__().

Here’s how this class works in practice:

Python
>>> disk = Storage(1024, "GB")

>>> disk
1024.0
>>> disk.unit
'GB'

>>> isinstance(disk, float)
True

Your Storage class works as expected, allowing you to use an instance attribute to store the unit in which you’re measuring the storage space. The .unit attribute is mutable, so you can change its value anytime you like. However, you can’t change the numeric value of disk because it’s immutable like its parent type, float.

Representing Objects as Strings

A common task that you’ll perform in your Python code is to display data or produce output. In some cases, your programs may need to display useful information to the user. In other cases, you may need to show information to other programmers using your code.

These two types of output can be pretty different because their respective target audiences can have different needs. The information for the user may not need to highlight implementation details. It may just need to be user-friendly and clear.

In contrast, the information for an audience of programmers may need to be technically sound, providing details of your code’s implementation.

Python has you covered in both scenarios. If you want to provide user-friendly output, then you can use the .__str__() method. On the other hand, when you need to provide developer-friendly output, then you can use the .__repr__() method. These methods support two different string representations for Python objects.

Along with .__init__(), the .__str__() and .__repr__() methods are arguably the most commonly used special methods in custom classes. In the following sections, you’ll learn about these two useful methods.

User-Friendly String Representations With .__str__()

The .__str__() special method returns a human-readable string representation of the object at hand. Python calls this method when you call the built-in str() function, passing an instance of the class as an argument.

Python also calls this method when you use the instance as an argument to the print() and format() functions. The method is meant to provide a string that’s understandable by the end user of the application.

To illustrate how to use the .__str__() method, consider the following Person class:

Python person.py
class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __str__(self):
        return f"I'm {self.name}, and I'm {self.age} years old."

This class has two attributes, the person’s name and age. In the .__str__() method, you return a user-friendly string representation of a person. This representation includes the name and the age in a descriptive message.

Here’s an example of how your class works:

Python
>>> from person import Person

>>> jane = Person("Jane Doe", 25)

>>> str(jane)
"I'm Jane Doe, and I'm 25 years old."

>>> print(jane)
I'm Jane Doe, and I'm 25 years old.

When you use an instance of Person as an argument to str() or print(), you get a string representation of the object on your screen. This representation is specially crafted for the user of your application or code.

Developer-Friendly String Representations With .__repr__()

The .__repr__() method returns a string representation of an object that’s targeted at the developer. Ideally, the string returned should be crafted in such a way that you can construct an identical instance of the object from the string.

To dive into how .__repr__() works, go back to the Person class and update it as in the code snippet below:

Python person.py
class Person:
    # ...

    def __repr__(self):
        return f"{type(self).__name__}(name='{self.name}', age={self.age})"

In .__repr__(), you use an f-string to build the object’s string representation. The first part of this representation is the class’s name. Then, you use a pair of parentheses to wrap the constructor’s arguments and their corresponding values.

Now restart your REPL session and run the following code:

Python
>>> from person import Person

>>> john = Person("John Doe", 35)

>>> john
Person(name='John Doe', age=35)

>>> repr(john)
"Person(name='John Doe', age=35)"

In this example, you create an instance of Person. Then, you access the instance directly through the REPL. The standard REPL automatically uses the .__repr__() method to display immediate feedback about an object.

In the last example, you use the built-in repr() function, which falls down to calling .__repr__() on the object that you pass in as an argument.

Note that you can copy and paste the resulting string representation, without the quotations, to reproduce the current object. This string representation is pretty handy for other developers using your code. From this representation, they can obtain critical information about how the current object is constructed.

Supporting Operator Overloading in Custom Classes

Python has multiple types of operators, which are special symbols, combinations of symbols, or keywords that designate some type of computation. Internally, Python supports operators with special methods. For example, the .__add__() special method supports the plus operator (+) as you already saw.

In practice, you’ll take advantage of these methods that are behind operators for something that’s known as operator overloading.

Operator overloading means providing additional functionality to the operators. You can do this with most built-in types and their specific supported operators. However, that’s not all you can do with the special methods that support Python operators. You can also use these methods to support some operators in your custom classes.

In the following sections, you’ll learn about the specific special methods that support Python operators, including arithmetic, comparison, membership, bitwise, and augmented operators. To kick things off, you’ll start with arithmetic operators, which are arguably the most commonly used operators.

Arithmetic Operators

Arithmetic operators are those that you use to perform arithmetic operations on numeric values. In most cases, they come from math, and therefore, Python represents them with the usual math signs.

In the following table, you’ll find a list of the arithmetic operators in Python and their supporting magic methods:

Note that all these methods take a second argument called other. In most cases, this argument should be the same type as self or a compatible type. If that’s not the case, then you can get an error.

To illustrate how you can overload some of these operators, get back to the Storage class. Say that you want to make sure that when you add or subtract two instances of this class, both operands have the same unit:

Python storage.py
class Storage(float):
    def __new__(cls, value, unit):
        instance = super().__new__(cls, value)
        instance.unit = unit
        return instance

    def __add__(self, other):
        if not isinstance(other, type(self)):
            raise TypeError(
                "unsupported operand for +: "
                f"'{type(self).__name__}' and '{type(other).__name__}'"
            )
        if not self.unit == other.unit:
            raise TypeError(
                f"incompatible units: '{self.unit}' and '{other.unit}'"
            )

        return type(self)(super().__add__(other), self.unit)

In the .__add__() method, you first check if other is also an instance of Storage. If not, then you raise a TypeError exception with an appropriate error message. Next, you check if both objects have the same unit. If not, then you raise a TypeError again. If both checks pass, then you return a new instance of Storage that you create by adding the two values and attaching the unit.

Here’s how this class works:

Python
>>> from storage import Storage

>>> disk_1 = Storage(500, "GB")
>>> disk_2 = Storage(1000, "GB")
>>> disk_3 = Storage(1, "TB")

>>> disk_1 + disk_2
1500.0

>>> disk_2 + disk_3
Traceback (most recent call last):
    ...
TypeError: incompatible units: 'GB' and 'TB'

>>> disk_1 + 100
Traceback (most recent call last):
    ...
TypeError: unsupported operand for +: 'Storage' and 'int'

In this example, when you add two objects that have the same unit, you get the correct value. If you add two objects with different units, then you get a TypeError telling you that the units are incompatible. Finally, when you try to add a Storage instance with a built-in numeric value, you get an error as well because the built-in type isn’t a Storage instance.

As an exercise, would you like to implement the .__sub__() method in your Storage class? Check the collapsible section below for a possible implementation:

Here’s a possible implementation of .__sub__() in the Storage class:

Python storage.py
class Storage(float):
    def __new__(cls, value, unit):
        instance = super().__new__(cls, value)
        instance.unit = unit
        return instance

    def __add__(self, other):
        if not isinstance(other, type(self)):
            raise TypeError(
                "unsupported operand for +: "
                f"'{type(self).__name__}' and '{type(other).__name__}'"
            )
        if not self.unit == other.unit:
            raise TypeError(
                f"incompatible units: '{self.unit}' and '{other.unit}'"
            )

        return type(self)(super().__add__(other), self.unit)

    def __sub__(self, other):
        if not isinstance(other, type(self)):
            raise TypeError(
                "unsupported operand for -: "
                f"'{type(self).__name__}' and '{type(other).__name__}'"
            )
        if not self.unit == other.unit:
            raise TypeError(
                f"incompatible units: '{self.unit}' and '{other.unit}'"
            )

        return type(self)(super().__sub__(other), self.unit)

Note that .__add__() and .__sub__() have several lines in common, which makes for repetitive code. You can fix this by using a helper method to extract the common logic.

In the above example, you overloaded the addition operator (+) on the built-in float type. You can also use the .__add__() method and any other operator method to support the corresponding operator in a custom class.

Consider the following example of a Distance class:

Python distance.py
class Distance:
    _multiples = {
        "mm": 0.001,
        "cm": 0.01,
        "m": 1,
        "km": 1_000,
    }

    def __init__(self, value, unit="m"):
        self.value = value
        self.unit = unit.lower()

    def to_meter(self):
        return self.value * type(self)._multiples[self.unit]

    def __add__(self, other):
        return self._compute(other, "+")

    def __sub__(self, other):
        return self._compute(other, "-")

    def _compute(self, other, operator):
        operation = eval(f"{self.to_meter()} {operator} {other.to_meter()}")
        cls = type(self)
        return cls(operation / cls._multiples[self.unit], self.unit)

In Distance, you have a non-public class attribute called ._multiples. This attribute contains a dictionary of length units and their corresponding conversion factors. In the .__init__() method, you initialize the .value and .unit attributes according to the user’s definition. Note that you’ve used the str.lower() method to normalize the unit string and enforce the use of lowercase letters.

In .to_meter(), you use ._multiples to express the current distance in meters. You’ll use this method as a helper in the .__add__() and .__sub__() methods. This way, your class will be able to add or subtract distances of different units correctly.

The .__add__() and .__sub__() methods support the addition and subtraction operators, respectively. In both methods, you use the ._compute() helper method to run the computation.

In ._compute(), you take other and operator as arguments. Then, you use the built-in eval() function to evaluate the expression and obtain the intended result for the current operation. Next, you create an instance of Distance using the built-in type() function. Finally, you return a new instance of the class with the computed value converted into the unit of the current instance.

Here’s how your Distance class works in practice:

Python
>>> from distance import Distance

>>> distance_1 = Distance(200, "m")
>>> distance_2 = Distance(1, "km")

>>> total = distance_1 + distance_2
>>> total.value
1200.0
>>> total.unit
'm'

>>> displacement = distance_2 - distance_1
>>> displacement.value
0.8
>>> displacement.unit
'km'

Wow! This is really cool! Your class works as expected. It supports the addition and subtraction operators.

More on Arithmetic Operators

The magic methods that support operators are affected by the relative position of each object in the containing expression. That’s why displacement in the above section is in kilometers, while total is in meters.

Take .__add__(), for example. Python calls this method on the left-hand operand. If that operand doesn’t implement the method, then the operation will fail. If the left-hand operand implements the method, but its implementation doesn’t behave as you want, then you can face issues. Thankfully, Python has the right-hand version of operator methods (.__r*__()) to tackle these issues.

For example, say that you want to write a Number class that supports addition between its object and integer or floating-point numbers. In that situation, you can do something like the following:

Python number.py
class Number:
    def __init__(self, value):
        self.value = value

    def __add__(self, other):
        print("__add__ called")
        if isinstance(other, Number):
            return Number(self.value + other.value)
        elif isinstance(other, int | float):
            return Number(self.value + other)
        else:
            raise TypeError("unsupported operand type for +")

    def __radd__(self, other):
        print("__radd__ called")
        return self.__add__(other)

The .__add__() method works when you use an instance of Number as the left-hand operator in an addition expression. In contrast, .__radd__() works when you use an instance of Number as the right-hand operand. Note that the r at the beginning of the method’s name stands for right.

Here’s how your Number class works:

Python
>>> from number import Number

>>> five = Number(5)
>>> ten = Number(10)

>>> fifteen = five + ten
__add__ called
>>> fifteen.value
15

>>> six = five + 1
__add__ called
>>> six.value
6

>>> twelve = 2 + ten
__radd__ called
__add__ called
>>> twelve.value
12

In this code snippet, you create two instances of Number. Then you use them in an addition where Python calls .__add__() as expected. Next up, you use five as the left-hand operator in an expression that mixes your class with the int type. In this case, Python calls Number.__add__() again.

Finally, you use ten as the right-hand operand in an addition. This time, Python implicitly calls Number.__radd__(), which falls back to calling Number.__add__().

Here’s a summary of the .__r*__() methods:

Python calls these methods when the left-hand operand doesn’t support the corresponding operation and the operands are of different types.

Python also has some unary operators. It calls them unary because they operate on a single operand:

Operator Supporting Method Description
- .__neg__(self) Returns the target value with the opposite sign
+ .__pos__(self) Provides a complement to the negation without performing any transformation

You can also overload these operators in Python’s built-in numeric types and support them in your own classes.

Comparison Operator Methods

You’ll also find that special methods are behind comparison operators. For example, when you execute something like 5 < 2, Python calls the .__lt__() magic method. Here’s a summary of all the comparison operators and their supporting methods:

To exemplify how you can use some of these methods in your custom classes, say that you’re writing a Rectangle class to create multiple different rectangles. You want to be able to compare these rectangles.

Here’s a Rectangle class that supports the equality (==), less than (<), and greater than (>) operators:

Python rectangle.py
class Rectangle:
    def __init__(self, height, width):
        self.height = height
        self.width = width

    def area(self):
        return self.height * self.width

    def __eq__(self, other):
        return self.area() == other.area()

    def __lt__(self, other):
        return self.area() < other.area()

    def __gt__(self, other):
        return self.area() > other.area()

Your class has an .area() method that computes the rectangle’s area using its height and width. Then you have the three required special methods to support the intended comparison operators. Note that all of them use the rectangle’s area to determine the final result.

Here’s how your class works in practice:

Python
>>> from rectangle import Rectangle

>>> basketball_court = Rectangle(15, 28)
>>> soccer_field = Rectangle(75, 110)

>>> basketball_court < soccer_field
True
>>> basketball_court > soccer_field
False
>>> basketball_court == soccer_field
False

That’s great! Your Rectangle class now supports equality (==), less than (<), and greater than (>) operations. As an exercise, you can go ahead and implement the rest of the operators using the appropriate special method.

Membership Operators

Python has two operators that allow you to determine whether a given value is in a collection of values. The operators are in and not in. They support a check that’s known as a membership test.

For example, say that you want to determine whether a number appears in a list of numbers. You can do something like this:

Python
>>> 2 in [2, 3, 5, 9, 7]
True

>>> 10 in [2, 3, 5, 9, 7]
False

In the first example, the number 2 is in the list of numbers, so you get True as a result. In the second example, the number 10 is not on the list, and you get False. The not in operator works similarly to in but as a negation. With this operator, you find out if a given value is not in a collection.

The special method that supports the membership operators in and not in is .__contains__(). By implementing this method in a custom class, you can control how its instances will behave in a membership test.

The .__contains__() method must take an argument that represents the value that you want to check for.

To illustrate how to support membership tests in a custom class, say that you want to implement a basic stack data structure that supports the usual push and pop operations as well as the in and not in operators.

Here’s the code for your Stack class:

Python stack.py
class Stack:
    def __init__(self, items=None):
        self.items = list(items) if items is not None else []

    def push(self, item):
        self.items.append(item)

    def pop(self):
        return self.items.pop()

    def __contains__(self, item):
        for current_item in self.items:
            if item == current_item:
                return True
        return False

In this example, you define Stack and provide the .push() and .pop() methods. The former method appends a new item to the top of the stack, while the latter removes and returns the item at the top of the stack.

Then, you implement the .__contains__() method, which takes the target item as an argument and uses a for loop to determine whether the item is in the list that stores the stack’s data.

Here’s how you can use an instance of Stack in a membership test:

Python
>>> from stack import Stack

>>> stack = Stack([2, 3, 5, 9, 7])

>>> 2 in stack
True
>>> 10 in stack
False

>>> 2 not in stack
False
>>> 10 not in stack
True

By implementing the .__contains__() method in your custom classes, you can customize how objects of those classes respond to membership checks using the in and not in operators.

Bitwise Operators

Python’s bitwise operators let you manipulate individual bits of data at the most granular level. With these operators, you can perform bitwise AND, OR, XOR, NOT, and bit shift operations. These operators are also implemented through special methods.

Here’s a summary of Python’s bitwise operators and their supporting methods:

With these methods, you can make your custom classes support bitwise operators. Consider the following toy example:

Python bitwise_number.py
class BitwiseNumber:
    def __init__(self, value):
        self.value = value

    def __and__(self, other):
        return type(self)(self.value & other.value)

    def __or__(self, other):
        return type(self)(self.value | other.value)

    def __xor__(self, other):
        return type(self)(self.value ^ other.value)

    def __invert__(self):
        return type(self)(~self.value)

    def __lshift__(self, places):
        return type(self)(self.value << places)

    def __rshift__(self, places):
        return type(self)(self.value >> places)

    def __repr__(self):
        return bin(self.value)

To implement the required methods in your custom class, BitwiseNumber, you use the bitwise operators that correspond to the method at hand. The .__and__(), .__or__(), and .__xor__() methods work on the value of your current BitwiseNumber instance and on other. This second operand should be another instance of BitwiseNumber.

The .__invert__() method supports the bitwise NOT operator, which is a unary operator because it acts on a single operand. Therefore, this method doesn’t take a second argument.

Finally, the two methods that support the bitwise shift operators take an argument called places. This argument represents the number of places that you want to shift the bit in either direction.

Here’s how your class works in practice:

Python
>>> from bitwise_number import BitwiseNumber

>>> five = BitwiseNumber(5)
>>> ten = BitwiseNumber(10)

>>> # Bitwise AND
>>> #    0b101
>>> # & 0b1010
>>> # --------
>>> #      0b0
>>> five & ten
0b0

>>> # Bitwise OR
>>> #    0b101
>>> # | 0b1010
>>> # --------
>>> #   0b1111
>>> five | ten
0b1111

>>> five ^ ten
0b1111
>>> ~five
-0b110
>>> five << 2
0b10100
>>> ten >> 1
0b101

Great! Your BitwiseNumber class works as expected. It supports all the bitwise operators, and they return the expected bits every time.

Augmented Assignments

When it comes to arithmetic operations, you’ll find a common expression that uses the current value of a variable to update the variable itself. A classic example of this operation is when you need to update a counter or an accumulator:

Python
>>> counter = 0
>>> counter = counter + 1
>>> counter = counter + 1
>>> counter
2

The second and third lines in this code snippet update the counter’s value using the previous value. This type of operation is so common in programming that Python has a shortcut for it, known as augmented assignment operators.

For example, you can shorten the above code using the augmented assignment operator for addition:

Python
>>> counter = 0
>>> counter += 1
>>> counter += 1
>>> counter
2

This code looks more concise, and it’s more Pythonic. It’s also more elegant. This operator (+=) isn’t the only one that Python supports. Here’s a summary of the supported operators in the math context:

When you use these methods in your custom classes, you should attempt to do the underlying operation in place, which is possible when you’re working with mutable types. If the data type is immutable, then you’ll return a new object with the updated value. If you don’t define a specific augmented method, then the augmented assignment falls back to the normal methods.

As an example, get back to the Stack class:

Python stack.py
class Stack:
    # ...

    def __add__(self, other):
        return type(self)(self.items + other.items)

    def __iadd__(self, other):
        self.items.extend(other.items)
        return self

    def __repr__(self):
        return f"{type(self).__name__}({self.items!r})"

In this code snippet, you first add a quick implementation of .__add__() that allows you to add two Stack objects using the regular addition operator, +. Then, you implement the .__iadd__() method, which relies on .extend() to append the items of other to your current stack.

Note that the .__add__() method returns a new instance of the class instead of updating the current instance in place. This is the most important difference between these two methods.

Here’s how the class works:

Python
>>> from stack import Stack

>>> stack = Stack([1, 2, 3])

>>> stack += Stack([4, 5, 6])
>>> stack
Stack([1, 2, 3, 4, 5, 6])

Your Stack class now supports the augmented assignment operator operator (+=). In this example, the .__iadd__() method appends the new items to the end of the current object.

In the above example, you implemented .__iadd__() in a mutable data type. In an immutable data type, you probably won’t need to implement the .__iadd__() method. The .__add__() method will be sufficient because you can’t update an immutable data type in place.

Last but not least, you should know that bitwise operators also have an augmented variations:

Again, you can use these methods to support the corresponding augmented bitwise operations in your own classes.

Introspecting Your Objects

You can also use some magic methods to support introspection in your custom classes. For example, you can control how an object behaves when you inspect it using built-in functions such as dir(), isinstance(), and others.

Here are some special methods that are useful for introspection:

Method Description
.__dir__() Returns a list of attributes and methods of an object
.__hasattr__() Checks whether an object has a specific attribute
.__instancecheck__() Checks whether an object is an instance of a certain class
.__subclasscheck__() Checks whether a class is a subclass of a certain class

By implementing these special methods in your classes, you can provide custom behaviors and control the introspection process when interacting with your objects.

As an example, you can add a .__dir__() method to your Rectangle class from before:

Python rectangle.py
class Rectangle:
    # ...

    def __dir__(self):
        print("__dir__ called")
        return sorted(self.__dict__.keys())

In this custom implementation of .__dir__(), you only return the names of the current instance attributes of your Rectangle class. Here’s how your class behaves when you use one of its instances as an argument to the built-in dir() function:

Python
>>> from rectangle import Rectangle

>>> dir(Rectangle(12, 24))
__dir__ called
['height', 'width']

As you can see, when you pass an instance of Rectangle to the dir() function, you get a list containing the names of all the instance attributes of this class. This behavior differs from the default behavior that returns all the attributes and methods.

Controlling Attribute Access

Behind every attribute access, assignment, and deletion, you’ll find special methods that support the specific operation. The following table provides a summary of the involved methods:

Method Description
.__getattribute__(self, name) Runs when you access an attribute called name
.__getattr__(self, name) Runs when you access an attribute that doesn’t exist in the current object
.__setattr__(self, name, value) Runs when you assign value to the attribute called name
.__delattr__(self, name) Runs when you delete the attribute called name

Among other things, you could use these methods to inspect and customize how you access, set, and delete attributes in your custom classes. In the following sections, you’ll learn how to use these methods in your classes.

Retrieving Attributes

Python has two methods to handle attribute access. The .__getattribute__() method runs on every attribute access, while the .__getattr__() method only runs if the target attribute doesn’t exist in the current object.

Here’s a Circle class that illustrates both methods:

Python circle.py
import math

class Circle:
    def __init__(self, radius):
        self.radius = radius

    def __getattribute__(self, name):
        print(f"__getattribute__ called for {name}")
        return super().__getattribute__(name)

    def __getattr__(self, name):
        print(f"__getattr__ called for {name}")
        if name == "diameter":
            return self.radius * 2
        return super().__getattr__(name)

In the .__getattribute__() method, you only print a message to identify when Python has called the method. Then, you delegate the attribute access to the parent class using the built-in super() function. Remember that Python calls this method in response to every attribute access.

Next, you have the .__getattr__() method. There, you print a message to identify the method call. In the conditional statement, you check if the accessed attribute is called "diameter". If that’s the case, then you compute the diameter and return it. Otherwise, you delegate the attribute access to the parent class.

Consider how your Circle class works in practice:

Python
>>> from circle import Circle

>>> circle = Circle(10)

>>> circle.radius
__getattribute__ called for 'radius'
10

>>> circle.diameter
__getattribute__ called for 'diameter'
__getattr__ called for 'diameter'
__getattribute__ called for 'radius'
20

When you access .radius using dot notation, Python implicitly calls .__getattribute__(). Therefore, you get the corresponding message and the attribute’s value. Note that Python calls this method in every attribute access.

Next, you access .diameter. This attribute doesn’t exist in Circle. Python calls .__getattribute__() first. Because this method doesn’t find the .diameter attribute, Python proceeds to call .__getattr__(). To compute the diameter value, you access .radius again. That’s why the final message appears on your screen.

Setting Attributes

Setting a value to an attribute is the complementary operation of accessing the value. You’ll typically use the assignment operator to set a new value to a given attribute. When Python detects the assignment, it calls the .__setattr__() magic method.

With this method, you can customize certain aspects of the assignment process. Say that you want to validate the radius as a positive number. In this situation, you can do something like the following:

Python circle.py
import math

class Circle:
    def __init__(self, radius):
        self.radius = radius

    # ...

    def __setattr__(self, name, value):
        if name == "radius":
            if not isinstance(value, int | float):
                raise TypeError("radius must be a number")
            if value <= 0:
                raise ValueError("radius must be positive")
        super().__setattr__(name, value)

The .__setattr__() method allows you to customize the behavior of your class in the context of an assignment.

In this example, you check if the target attribute is named "radius". If that’s the case, then you check whether the provided value is an integer or a floating-point number. Next, you check whether the number is greater than 0. In both cases, you raise an appropriate exception. Finally, you delegate the assignment to the parent class.

Go ahead and restart your REPL session. Then, run the following code:

Python
>>> from circle import Circle

>>> circle = Circle(10)
__setattr__ called for 'radius'

>>> circle.radius = 20
__setattr__ called for 'radius'

>>> circle.radius = "42"
__setattr__ called for 'radius'
Traceback (most recent call last):
    ...
TypeError: radius must be a number

>>> circle.diameter = 42
__setattr__ called for 'diameter'
>>> circle.diameter
__getattribute__ called for 'diameter'
42

When you call the class constructor, Circle(), you get a message indicating that .__setattr__() was called. This is because the .__init__() method runs an assignment.

Then, you assign a new value to .radius, and Python calls .__setattr__() again. When you try to assign a string to .radius, you get a TypeError because the input value isn’t a valid number.

Finally, you can attach a new attribute like .diameter to your instance of Circle, just like you’d do with an instance of a class that doesn’t implement a custom .__setattr__() method. This is possible because you’ve delegated the assignment to the parent’s .__setattr__() method at the end of your implementation.

Deleting Attributes

Sometimes, you need to delete a given attribute from an object after you use it or when something happens. If you want to have fine control over how your classes respond to attribute deletion with the del statement, then you can use the .__delattr__() magic method.

For example, say that you want to create a class that forbids attribute deletion:

Python non_deletable.py
class NonDeletable:
    def __init__(self, value):
        self.value = value

    def __delattr__(self, name):
        raise AttributeError(
            f"{type(self).__name__} doesn't support attribute deletion"
        )

This toy class has an attribute called .value. You override the .__delattr__() method and raise an exception whenever someone tries to remove an attribute in your class.

Check out how the class works in practice:

Python
>>> from non_deletable import NonDeletable

>>> one = NonDeletable(1)
>>> one.value
1

>>> del one.value
Traceback (most recent call last):
    ...
AttributeError: NonDeletable doesn't support attribute deletion

When you try to use the del statement to remove the .value attribute, you get an AttributeError exception. Note that the .__delattr__() method will catch all the attempts to delete instance attributes in your class.

Managing Attributes Through Descriptors

Python descriptors are classes that allow you to have fine control over the attributes of a given custom class. You can use descriptors to add function-like behavior on top of the attributes of a given class.

A descriptor class needs at least the .__get__() special method. However, the full descriptor protocol consists of the following methods:

Method Description
.__get__(self, instance, type=None) Getter method that allows you to retrieve the current value of the managed attribute
.__set__(self, instance, value) Setter method that allows you to set a new value to the managed attribute
.__delete__(self, instance) Deleter method that allows you to remove the managed attribute from the containing class
.__set_name__(self, owner, name) Name setter method that allows you to define a name for the managed attribute

To illustrate what descriptors are good for, say that you have a shapes.py module. You’ve defined the Circle, Square, and Rectangle classes. You’ve realized that you need to validate parameters like the circle’s radius, the square’s side, and so on. You’ve also noted that the validation logic is common to all these parameters.

In this situation, you can use a descriptor to manage the validation logic, which consists of checking if the provided value is a positive number. Here’s a possible implementation of this descriptor:

Python shapes.py
class PositiveNumber:
    def __set_name__(self, owner, name):
        self._name = name

    def __get__(self, instance, owner):
        return instance.__dict__[self._name]

    def __set__(self, instance, value):
        if not isinstance(value, int | float) or value <= 0:
            raise ValueError("positive number expected")
        instance.__dict__[self._name] = value

In this class, you first define the .__set_name__() special method. The owner argument represents the containing class, while the name argument represents the attribute name.

Then, you define the .__get__() method. This method takes an instance and a reference to the containing class. Inside this method, you use the .__dict__ special attribute to access the instance’s namespace and retrieve the value of the managed attribute.

Finally, you define the .__set__() method, which takes an instance of the containing class and a new value for the managed attribute. In this method, you check if the provided value is not a positive number, in which case you raise a ValueError exception. Otherwise, you assign the input value to the managed attribute using the .__dict__ namespace again.

Now you can use this descriptor to define the parameters of your shapes:

Python shapes.py
import math

# ...

class Circle:
    radius = PositiveNumber()

    def __init__(self, radius):
        self.radius = radius

    def area(self):
        return round(math.pi * self.radius**2, 2)

class Square:
    side = PositiveNumber()

    def __init__(self, side):
        self.side = side

    def area(self):
        return round(self.side**2, 2)

class Rectangle:
    height = PositiveNumber()
    width = PositiveNumber()

    def __init__(self, height, width):
        self.height = height
        self.width = width

    def area(self):
        return self.height * self.width

In this code, the highlighted lines show how you create the managed attributes using the PositiveNumber descriptor. Note that the managed attributes must be class attributes with an instance attribute counterpart.

Here’s how your shapes work:

Python
>>> from shapes import Circle, Square, Rectangle

>>> circle = Circle(-10)
Traceback (most recent call last):
    ...
ValueError: positive number expected

>>> square = Square(-20)
Traceback (most recent call last):
    ...
ValueError: positive number expected

>>> rectangle = Rectangle(-20, 30)
Traceback (most recent call last):
    ...
ValueError: positive number expected

Great! All your shapes validate the input parameter on instantiation. This is because all of them use your PositiveNumber to manage their parameters. This way, you’ve reused the validation code in all your classes.

Supporting Iteration With Iterators and Iterables

In Python, iterators and iterables are two different but related tools that come in handy when you need to iterate over a data stream or container. Iterators power and control the iteration process, while iterables typically hold data that you can iterate through.

Iterators and iterables are fundamental components in Python programming. You’ll use them directly or indirectly in almost all your programs. In the following sections, you’ll learn the basics of how to use special methods to turn your custom classes into iterators and iterables.

Creating Iterators

To create an iterator in Python, you need two special methods. By implementing these methods, you’ll take control of the iteration process. You define how Python retrieves items when you use the class in a for loop or another iteration construct.

The table below shows the methods that make up an iterator. They’re known as the iterator protocol:

Method Description
.__iter__() Called to initialize the iterator. It must return an iterator object.
.__next__() Called to iterate over the iterator. It must return the next value in the data stream.

As you can see, both methods have their own responsibility. In .__iter__(), you typically return self, the current object. In .__next__(), you need to return the next value from the data stream in a sequence. This method must raise a StopIteration when the stream of data is exhausted. This way, Python knows that the iteration has reached its end.

By implementing these two methods in your custom classes, you’ll turn them into iterators. For example, say you want to create a class that provides an iterator over the numbers in the Fibonacci sequence. In that situation, you can write the following solution:

Python fibonacci.py
class FibonacciIterator:
    def __init__(self, stop=10):
        self._stop = stop
        self._index = 0
        self._current = 0
        self._next = 1

    def __iter__(self):
        return self

    def __next__(self):
        if self._index < self._stop:
            self._index += 1
            fib_number = self._current
            self._current, self._next = (
                self._next,
                self._current + self._next,
            )
            return fib_number
        else:
            raise StopIteration

In this FibonacciIterator class, you first initialize a few attributes. The ._stop attribute defines the number of values that your class will return. It defaults to 10. The ._index attribute holds the index of the current Fibonacci value in the sequence. The ._current and ._next attributes represent the current and next values in the Fibonacci sequence, respectively.

In .__iter__(), you just return the current object, self. This practice is common when you create iterators in Python.

The .__next__() method is a bit more elaborate. In this method, you have a conditional that checks if the current sequence index hasn’t reached the ._stop value, in which case you increment the current index to control the iteration process. Then, you compute the Fibonacci number that corresponds to the current index, returning the result to the caller of .__next__().

When ._index grows to the value of ._stop, you raise a StopIteration exception, which terminates the iteration process.

Your class is now ready for iteration. Here’s how it works:

Python
>>> from fib_iter import FibonacciIterator

>>> for fib_number in FibonacciIterator():
...     print(fib_number)
...
0
1
1
2
3
5
8
13
21
34

Your FibonacciIterator class computes new values in real time, yielding values on demand when you use the class in a loop. When the number of Fibonacci values grows up to 10, the class raises the StopIteration exception, and Python terminates the loop. That’s great! Your class is now a working iterator.

Building Iterables

Iterables are a bit different from iterators. Typically, a collection or container is iterable when it implements the .__iter__() special method.

For example, Python built-in container types—such as lists, tuples, dictionaries, and sets—are iterable objects. They provide a stream of data that you can iterate over. However, they don’t provide the .__next__() method, which drives the iteration process. So, to iterate over an iterable using a for loop, Python implicitly creates an iterator.

As an example, get back to your Stack class. Update the class with the code below to make it iterable:

Python stack.py
class Stack:
    # ...

    def __iter__(self):
        return iter(self.items[::-1])

Note that this implementation of .__iter__() doesn’t return the current object, self. Instead, the method returns an iterator over the items in the stack using the built-in iter() function. The iter() function provides the input iterable with a default .__next__() method, enabling it to be a full-fledged iterator.

You reverse the list of items in the stack before iteration to be consistent with the order elements are popped from the stack. Run the following code in your Python interactive session to try out this new feature of Stack:

Python
>>> from stack import Stack

>>> stack = Stack([1, 2, 3, 4])

>>> for value in stack:
...     print(value)
...
4
3
2
1

In this snippet, you first create a new instance of Stack with four numbers. Then, you start a for loop over the instance. The loop prints the current value in each iteration. This shows the order elements would be popped if you call .pop() repeatedly. Your Stack class now supports iteration.

Making Your Objects Callable

In Python, a callable is any object that you can call using a pair of parentheses and, optionally, a series of arguments. For example, functions, classes, and methods are all callables in Python. Callables are fundamental in many programming languages because they allow you to perform actions.

Python allows you to create your own callable. To do this, you can add the .__call__() special method to your class. Any instance of a class with a .__call__() method behaves like a function.

A common use case of callable instances is when you need a stateful callable that caches computed data between calls. This will be handy when you need to optimize some algorithms. For example, say that you want to code a Factorial class that caches already-computed values for efficiency. In this situation, you can do something like the following:

Python factorial.py
class Factorial:
    def __init__(self):
        self._cache = {0: 1, 1: 1}

    def __call__(self, number):
        if number not in self._cache:
            self._cache[number] = number * self(number - 1)
        return self._cache[number]

In this example, you define a Factorial with a .__call__() method. In the .__init__() method, you create and initialize a ._cache attribute to hold already-computed values.

Next, you define .__call__(). Python automatically calls this method when you use an instance of the containing class as a function. In this example, the method computes the factorial of a given number while caching the computed values in ._cache.

Go ahead and give your class a try by running the following code:

Python
>>> from factorial import Factorial

>>> factorial_of = Factorial()

>>> factorial_of(4)
24
>>> factorial_of(5)
120
>>> factorial_of(6)
720

In this code example, you create an instance of Factorial. Then, you use that instance as a function to compute the factorial of different numbers.

By implementing the .__call__() magic method in a class, you make them behave like callable objects that behave like functions when you call them with a pair of parentheses. This allows you to add flexibility and functionality to your object-oriented code.

Implementing Custom Sequences and Mappings

Python’s sequences and mappings are fundamental built-in data types. Lists, tuples, and strings are examples of sequences, while dictionaries are examples of mapping types.

You can create your own sequence-like and mapping-like classes by implementing the required special methods. With this purpose, Python defines the sequence and mapping protocols. These protocols are collections of special methods.

For example, the sequence protocol consists of the following methods:

Method Description
.__getitem__() Called when you access an item using indexing like in sequence[index]
.__len__() Called when you invoke the built-in len() function to get the number of items in the underlying sequence
.__contains__() Called when you use the sequence in a membership test with the in or not in operator
.__reversed__() Called when you use the built-in reversed() function to get a reversed version of the underlying sequence

By implementing these special methods, you can build custom sequence-like classes in Python. To illustrate with an example, get back to your good friend the Stack class. You’ve already implemented the .__contains__() method to support membership. So, you only need the other three methods:

Python stack.py
class Stack:
    # ...

    def __contains__(self, item):
        for current_item in self.items:
            if item == current_item:
                return True
        return False

    # ...

    def __getitem__(self, index):
        return self.items[index]

    def __len__(self):
        return len(self.items)

    def __reversed__(self):
        return type(self)(reversed(self.items))

In the .__getitem__() method, you return the target item from the internal data container, which is a regular list as you already know. With this method, you’re supporting indexing operations.

Next, you define .__len__() to support the built-in len() function. This method returns the number of items in the internal list of data. Finally, you define .__reversed__(). This method allows you to get a reversed version of your original data using the built-in reversed() function.

Here’s how these new features work in practice. Remember that you need to restart your interactive session so that you can import the updated class:

Python
>>> from stack import Stack

>>> stack = Stack([1, 2, 3, 4])

>>> stack[1]
2
>>> stack[-1]
4

>>> len(stack)
4

>>> reversed(stack)
Stack([4, 3, 2, 1])

Now, your stack object supports indexing with the [index] operator. It also supports the built-in len() and reversed() functions.

Note that when you call reversed() with a Stack instance as an argument, you get a new Stack instance instead of a list. This behavior differs from the default behavior of reversed(), which typically returns a reversed iterator. In this specific example, you experience the flexibility and power behind magic methods.

Similarly to what you did in the example above, you can create your own mapping-like classes. To do that, you need to implement the special methods defined in the collections.abc.Mapping or collections.abc.MutableMapping abstract base class, depending on whether your class will be immutable or mutable.

Handling Setup and Teardown With Context Managers

Python’s with statement is quite useful for managing external resources in your programs. It allows you to take advantage of context managers to automatically handle the setup and teardown phases when you’re using external resources.

For example, the built-in open() function returns a file object that’s also a context manager. You’ll typically use this function in a with statement to hand control over to the underlying context manager, which is a runtime context that takes care of opening the file and closing it when you finish your work. Those processes comprise the setup and teardown, respectively.

Here’s an example of using open() to work with a text file:

Python
>>> with open("hello.txt", mode="w", encoding="utf-8") as file:
...    file.write("Hello, World!")
...
13

>>> with open("hello.txt", mode="r", encoding="utf-8") as file:
...     print(file.read())
...
Hello, World!

The first with statement opens the hello.txt file for writing. In the indented block, you write some text to the file. Once the block runs, Python closes the hello.txt file to release the acquired resource.

The second statement opens the file for reading and prints its content on your terminal window.

If you want to create a context manager or add context manager functionality to an existing class, then you need to code two special methods:

Method Description
.__enter__() Sets up the runtime context, acquires resources, and may return an object that you can bind to a variable with the as specifier on the with header
.__exit__() Cleans up the runtime context, releases resources, handles exceptions, and returns a Boolean value indicating whether to propagate any exceptions that may occur in the context

To illustrate how you can support the context manager protocol in one of your classes, get back to the example at the beginning of this section and say that you want a context manager that allows you to open a file for reading. In this situation, you can do something like the following:

Python reader.py
class TextFileReader:
    def __init__(self, file_path, encoding="utf-8"):
        self.file_path = file_path
        self.encoding = encoding

    def __enter__(self):
        self.file_obj = open(self.file_path, mode="r", encoding=self.encoding)
        return self.file_obj

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.file_obj.close()

This class takes a file path as an argument. The .__enter__() method opens the file for reading and returns the resulting file object. The .__exit__() method takes three mandatory arguments representing an exception type, value, and traceback. Then, you close the file object by calling the .close() method on that object.

Here’s how your TextFileReader class works in practice:

Python
>>> from reader import TextFileReader

>>> with TextFileReader("hello.txt") as file:
...     print(file.read())
...
Hello, World!

Your class works just like when you use open() to open a file in read mode. The TextFileReader class is a sample of the power that you can leverage in a custom class when you make it support the .__enter__() and .__exit__() methods.

Conclusion

You now know how to customize your classes using special methods, also known as magic methods or dunder methods in Python. These methods support core object-oriented features in Python, so learning about them is a fundamental skill for Python programmers who want to create powerful classes in their code.

In this tutorial, you’ve learned:

  • What Python’s special or magic methods are
  • What the magic behind magic methods in Python is
  • How to use special methods to customize different class’s behaviors

With this knowledge, you’re now ready to deeply customize your own Python classes by leveraging the power of special methods in your code.

🐍 Python Tricks 💌

Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

Python Tricks Dictionary Merge

About Leodanis Pozo Ramos

Leodanis is an industrial engineer who loves Python and software development. He's a self-taught Python developer with 6+ years of experience. He's an avid technical writer with a growing number of articles published on Real Python and other sites.

» More about Leodanis

Each tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are:

Master Real-World Python Skills With Unlimited Access to Real Python

Locked learning resources

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

Master Real-World Python Skills
With Unlimited Access to Real Python

Locked learning resources

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

What Do You Think?

Rate this article:

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal.


Looking for a real-time conversation? Visit the Real Python Community Chat or join the next “Office Hours” Live Q&A Session. Happy Pythoning!

Keep Learning

Related Tutorial Categories: advanced python