Python Protocols: Leveraging Structural Subtyping

The Meaning of “Protocol” in Python

During Python’s evolution, the term protocol became overloaded with two subtly different meanings. The first meaning refers to internal protocols, such as the iterator, context manager, and descriptor protocols.

These protocols are widely understood in the community and consist of special methods that make up a given protocol. For example, the .__iter__() and .__next__() methods define the iterator protocol.

Python 3.8 introduced a second, slightly different type of protocol. These protocols specify the methods and attributes that a class must implement to be considered of a given type. So, these protocols also have to do with a class’s internal structure.

With this kind of protocol, you can define interchangeable classes as long as they share a common internal structure. This feature allows you to enforce a relationship between types or classes without the burden of inheritance. This relationship is known as structural subtyping or static duck typing.

In this tutorial, you’ll focus on this second meaning of the term protocol. First, you’ll have a look at how Python manages types.

Dynamic and Static Typing in Python

Python is a dynamically typed language, which means that the Python interpreter checks an object’s type when the code runs. It also means that while a variable can only reference one object at a time, the type of that object can change during the variable’s lifetime.

For example, you can have a variable that starts as a string and changes into an integer number:

>>> value = "One hundred"
>>> value
'One hundred'

>>> value = 100
>>> value
100

In this example, you have a variable that starts as a string. Later in your code, you change the variable’s value to an integer.

Because of its dynamic nature, Python has embraced a flexible typing system that’s known as duck typing.

>>> numbers = [1, 2, 3]
>>> person = ("Jane", 25, "Python Dev")
>>> letters = "abc"
>>> ordinals = {"one": "first", "two": "second", "three": "third"}
>>> even_digits = {2, 4, 6, 8}

>>> containers = [numbers, person, letters, ordinals, even_digits]

>>> for container in containers:
...     for element in container:
...         print(element, end=" ")
...     print()
...
1 2 3 
Jane 25 Python Dev 
a b c 
one two three 
8 2 4 6 

 1from typing import Union
 2
 3def add(a: Union[float, int], b: Union[float, int]) -> float:
 4    return float(a + b)
 5
 6print(add(2, 4))
 7
 8print(add("2", "4"))

$ python calculations.py
6.0
24.0

$ python -m pip install mypy

$ mypy calculations.py
calculations.py:8: error: Argument 1 to "add" has incompatible type "str";
    expected "float | int"  [arg-type]
calculations.py:8: error: Argument 2 to "add" has incompatible type "str";
    expected "float | int"  [arg-type]
Found 2 errors in 1 file (checked 1 source file)

 1class Duck:
 2    def quack(self):
 3        return "The duck is quacking!"
 4
 5def make_it_quack(duck: Duck) -> str:
 6    return duck.quack()
 7
 8class Person:
 9    def quack(self):
10        return "The person is imitating a duck quacking!"
11
12print(make_it_quack(Duck()))
13
14print(make_it_quack(Person()))

$ mypy birds_v1.py
birds_v1.py:14: error: Argument 1 to "make_it_quack" has incompatible type
    "Person"; expected "Duck"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

class QuackingThing:
    def quack(self):
        raise NotImplementedError(
            "Subclasses must implement this method"
        )

class Duck(QuackingThing):
    def quack(self):
        return "The duck is quacking!"

class Person(QuackingThing):
    def quack(self):
        return "The person is imitating a duck quacking!"

def make_it_quack(duck: QuackingThing) -> str:
    return duck.quack()

print(make_it_quack(Duck()))

print(make_it_quack(Person()))

$ mypy birds_v2.py
Success: no issues found in 1 source file

Structural Subtyping and Protocols

In Python’s type system, you’ll find two ways to decide whether two objects are compatible as types:

1. Nominal subtyping is strictly based on inheritance. A class that inherits from a parent class is a subtype of its parent.
2. Structural subtyping is based on the internal structure of classes. Two classes with the same methods and attributes are structural subtypes of one another.

To understand the first subtyping strategy, say you have a class hierarchy to represent animals:

class Animal:
    def __init__(self, name):
        self.name = name

    def eat(self):
        print(f"{self.name} is eating.")

    def drink(self):
        print(f"{self.name} is drinking.")

class Dog(Animal):
    def bark(self):
        print(f"{self.name} is barking.")

class Cat(Animal):
    def meow(self):
        print(f"{self.name} is meowing.")

In this code, Dog and Cat are nominal subtypes of Animal. This means that you can use instances of Dog or Cat when Animal instances are expected. This form of subtyping is quick to understand and matches how the built-in isinstance() works:

>>> from animals_v1 import Animal, Cat, Dog

>>> pluto = Dog("Pluto")
>>> isinstance(pluto, Animal)
True

>>> tom = Cat("Tom")
>>> isinstance(tom, Animal)
True

The built-in isinstance() function allows you to check whether a given object is an instance of a class. This function considers subtypes when running the check. That’s why you get True in both examples above.

Checking types explicitly isn’t a popular practice in Python. Instead, the language favors duck typing, where you use the objects without considering their types but only the expected operations. Here’s where structural subtyping comes in handy.

For example, you can reimplement the animal classes without setting a formal relationship between them:

class Dog:
    def __init__(self, name):
        self.name = name

    def eat(self):
        print(f"{self.name} is eating.")

    def drink(self):
        print(f"{self.name} is drinking.")

    def make_sound(self):
        print(f"{self.name} is barking.")

class Cat:
    def __init__(self, name):
        self.name = name

    def eat(self):
        print(f"{self.name} is eating.")

    def drink(self):
        print(f"{self.name} is drinking.")

    def make_sound(self):
        print(f"{self.name} is meowing.")

Now Dog and Cat don’t have a strict inheritance relationship. They’re completely decoupled and independent classes.

However, they implement the same public interface. In other words, they have the same internal structure, including methods and attributes. Because of this characteristic, they’re structural subtypes, and you can use them in a duck typing context:

>>> from animals_v2 import Cat, Dog

>>> for animal in [Cat("Tom"), Dog("Pluto")]:
...     animal.eat()
...     animal.drink()
...     animal.make_sound()
...
Tom is eating.
Tom is drinking.
Tom is meowing.
Pluto is eating.
Pluto is drinking.
Pluto is barking.

These classes work okay in a duck typing context. However, up to this point, you don’t have a formal way to indicate that Cat and Dog are subtypes. You only know they are because the code tells you they have the same internal structure. To formalize this subtype relationship, you can use a protocol.

As you’ve already learned, protocols allow you to specify the expected methods and attributes that a class should have to support a given feature without requiring explicit inheritance. So, protocols are explicit sets of methods and attributes.

In practice, a class can support multiple protocols. For example, you can informally say that your Dog and Cat classes have a living protocol consisting of the .eat() and .drink() methods, which are the operations required to support life. You can also say that the classes have a sounding protocol comprised of the .make_sound() method.

With this in mind, you can create other classes that support any of these protocols:

class Person:
    def __init__(self, name):
        self.name = name

    def eat(self):
        print(f"{self.name} is eating.")

    def drink(self):
        print(f"{self.name} is drinking.")

    def talk(self):
        print(f"{self.name} is talking.")

This Person class supports the living protocol because it implements the required behaviors, .eat() and .drink(). However, the class doesn’t implement the .make_sound() method, so it doesn’t support the sounding protocol.

Protocols are particularly useful when it’s impractical to modify the inheritance structure of classes. With protocols, you can focus on defining the desired behavior and characteristics without having to design complex inheritance relationships.

Static Duck Typing With Protocols

As you’ve already learned, duck typing and type hints collide, imposing restrictions on Python coders who want to use both techniques. Python 3.8 introduced a way to create formal protocols in the typing system.

From this version on, the typing module defines a base class called Protocol that you can use to create custom protocols. This new class provides a formal way to support structural subtyping in the type hint system.

Since Python 3.8, the static type checkers started to support structural subtyping, which means that they check whether objects are of a specific nominal type and also whether they meet the requirement to be of a given structural type.

Consider the following toy example:

from typing import Protocol

class Adder(Protocol):
    def add(self, x, y): ...

class IntAdder:
    def add(self, x, y):
        return x + y

class FloatAdder:
    def add(self, x, y):
        return x + y

def add(adder: Adder) -> None:
    print(adder.add(2, 3))

add(IntAdder())
add(FloatAdder())

In this code, you define a class called Adder by inheriting from typing.Protocol. Adder implements an .add() method, which defines the Adder protocol itself. Note that protocol methods don’t have a body, which you typically indicate with the ellipsis (...) syntax.

Then, you define two classes, IntAdder and FloatAdder. These classes implement the Adder protocol because they have an .add() method. Therefore, you can use objects of either class as arguments to the add() function, which takes an Adder object as an argument.

The static type checker will be happy with your type hints on the add() function:

$ mypy adder_v1.py
Success: no issues found in 1 source file

The static type check passes because both IntAdder and FloatAdder support the Adder protocol. In this case, mypy considers the object’s internal structure rather than its nominal type.

Note that you haven’t made these classes inherit from Adder. They’re completely independent and decoupled classes that you can use in a duck typing context, such as the add() function.

Custom Protocols in Python

There’s much more to learn about creating custom protocols in Python. For example, you can create protocols through single or multiple inheritance. You can have protocols with different types of members. You can even create generic protocols.

In the following sections, you’ll learn about these topics and how they can help you improve your custom protocols and make them meet your typing requirements.

from typing import Protocol

class Adder(Protocol):
    def add(self, x: float, y: float) -> float:
        ...

class IntAdder:
    def add(self, x, y):
        return x + y

def add(adder: Adder) -> None:
    print(adder.add(2, 3))

add(IntAdder()) # mypy: Success: no issues found in 1 source file

# ...

class IntAdder:
    def add(self, x: int, y: int) -> int:
        return x + y

def add(adder: Adder) -> None:
    print(adder.add(2, 3))

add(IntAdder())

$ mypy adder_v2.py
adder_v2.py:17: error: Argument 1 to "add" has incompatible type "IntAdder";
    expected "Adder"  [arg-type]
adder_v2.py:17: note: Following member(s) of "IntAdder" have conflicts:
adder_v2.py:17: note:     Expected:
adder_v2.py:17: note:         def add(self, x: float, y: float) -> float
adder_v2.py:17: note:     Got:
adder_v2.py:17: note:         def add(self, x: int, y: int) -> int
Found 1 error in 1 file (checked 1 source file)

from typing import Protocol

class Adder(Protocol):
    def add(self, x: int | float, y: int | float) -> int | float:
        ...

# ...

$ mypy adder_v3.py
adder_v2.py:17: error: Argument 1 to "add" has incompatible type "IntAdder";
    expected "Adder"  [arg-type]
adder_v2.py:17: note: Following member(s) of "IntAdder" have conflicts:
adder_v2.py:17: note:     Expected:
adder_v2.py:17: note:         def add(self, x: int | float, y: int | float)
    -> int | float
adder_v2.py:17: note:     Got:
adder_v2.py:17: note:         def add(self, x: int, y: int) -> int
Found 1 error in 1 file (checked 1 source file)

from typing import Protocol, TypeVar

T = TypeVar("T")

class GenericProtocol(Protocol[T]):
    def method(self, arg: T) -> T:
        ...

from typing import Protocol, TypeVar

T = TypeVar("T", bound=int | float)

class Adder(Protocol[T]):
    def add(self, x: T, y: T) -> T:
        ...

class IntAdder:
    def add(self, x: int, y: int) -> int:
        return x + y

class FloatAdder:
    def add(self, x: float, y: float) -> float:
        return x + y

def add(adder: Adder) -> None:
    print(adder.add(2, 3))

add(IntAdder())
add(FloatAdder())

from typing import Protocol

class Adder(Protocol):
    def add[T: int | float](self, x: T, y: T) -> T:
        ...

# ...

from abc import abstractmethod
from typing import ClassVar, Protocol

class ProtocolMembersDemo(Protocol):
    class_attribute: ClassVar[int]
    instance_attribute: str = ""

    def instance_method(self, arg: int) -> str:
        ...

    @classmethod
    def class_method(cls) -> str:
        ...

    @staticmethod
    def static_method(arg: int) -> str:
        ...

    @property
    def property_name(self) -> str:
        ...

    @property_name.setter
    def property_name(self, value: str) -> None:
        ...

    @abstractmethod
    def abstract_method(self) -> str:
        ...

from typing import Protocol

class ContentCreator(Protocol):
    def create_content(self) -> str:
        ...

class Blogger(ContentCreator, Protocol):
    posts: list[str]

    def add_post(self, title: str, content: str) -> None:
        ...

class Vlogger(ContentCreator, Protocol):
    videos: list[str]

    def add_video(self, title: str, path: str) -> None:
        ...

# ...

class Blog:
    def __init__(self):
        self.blog_posts = []

    def create_content(self) -> str:
        return "Creating a post."

    def add_post(self, title: str, content: str) -> None:
        self.blog_posts.append(f"{title}: {content}")
        print(f"Post added: {title}")

class Vlog:
    def __init__(self):
        self.videos = []

    def create_content(self) -> str:
        return "Recording a video."

    def add_video(self, title: str, path: str) -> None:
        self.videos.append(f"{title}: {path}")
        print(f"Video added: {title}")

# ...

def produce_content(creator: ContentCreator):
    print(creator.create_content())

def add_post(blogger: Blogger, title: str, content: str):
    blogger.add_post(title, content)

def add_video(vlogger: Vlogger, title: str, path: str):
    vlogger.add_video(title, path)

$ mypy contents.py
Success: no issues found in 1 source file

from typing import Optional, Protocol

class LinkedListNode(Protocol):
    value: int
    next_node: Optional["LinkedListNode"]

    def __str__(self) -> str:
        return f"{self.value} -> {self.next_node}"

from __future__ import annotations
from typing import Optional, Protocol

class LinkedListNode(Protocol):
    value: int
    next_node: Optional[LinkedListNode]

    def __str__(self) -> str:
        return f"{self.value} -> {self.next_node}"

# ...

class Node:
    def __init__(
        self,
        value: int,
        next_node: Optional["LinkedListNode"] = None,
    ):
        self.value = value
        self.next_node = next_node

    def __str__(self) -> str:
        return f"{self.value} -> {self.next_node}"

def print_linked_list(start_node: LinkedListNode):
    print(start_node)

node3 = Node(3)
node2 = Node(2, node3)
node1 = Node(1, node2)

print_linked_list(node1)

$ mypy linked_list.py
Success: no issues found in 1 source file

Predefined Protocols in Python

Python has several predefined protocols. Iterable and Iterator are good examples. A few of them are in the typing module under the Protocols section. However, most of the currently available protocols live in the collections.abc module because they’re implemented as abstract base classes.

Some of the most common predefined protocols include the following:

Even though these classes are abstract base classes (ABC) rather than formal protocols, you can use them as protocols in your type hints. The static type checkers should be able to process them as expected.

In some situations, you probably won’t need to use these classes because you can just use the concrete built-in type. For example, instead of using the Set ABC to type hint an argument or a return value, you can use the built-in set class.

However, there are situations where using the concrete type won’t meet your needs. For example, say that you want to code a function that takes some integer values and filters the even numbers, returning a list. Here’s the function without type hints:

def filter_even_numbers(numbers):
    return [number for number in numbers if number % 2 == 0]

print(filter_even_numbers([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))
print(filter_even_numbers((1, 2, 3, 4, 5, 6, 7, 8, 9, 10)))
print(filter_even_numbers({1, 2, 3, 4, 5, 6, 7, 8, 9, 10}))

This function works with any iterable object. In other words, you can use a list, tuple, set, or any other iterable object as an argument to filter_even_numbers().

How would you type hint the numbers argument in this function? You can think of doing something like the following:

def filter_even_numbers(numbers: list[int]) -> list[int]:
    return [number for number in numbers if number % 2 == 0]

print(filter_even_numbers([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))
print(filter_even_numbers((1, 2, 3, 4, 5, 6, 7, 8, 9, 10)))
print(filter_even_numbers({1, 2, 3, 4, 5, 6, 7, 8, 9, 10}))

The list[int] type hint for numbers doesn’t work well. It causes numbers to only accept list objects, which makes the function less generic. Your static type checker will fail with tuples, sets, or any other iterable:

$ mypy even_v2.py
even_v2.py:5: error: Argument 1 to "filter_even_numbers" has incompatible
    type "tuple[int, int, int, int, int, int, int, int, int, int]";
    expected "list[int]"  [arg-type]
even_v2.py:6: error: Argument 1 to "filter_even_numbers" has incompatible
    type "set[int]"; expected "list[int]"  [arg-type]
Found 2 errors in 1 file (checked 1 source file)

This output shows that the two final calls to print() cause issues related to the data types of the input values for the numbers argument.

To fix the issues and provide a generic type hint that allows your function to take different iterables of numbers, you can use the Iterable ABC as in the code below:

from collections.abc import Iterable

def filter_even_numbers(numbers: Iterable[int]) -> list[int]:
    return [number for number in numbers if number % 2 == 0]

print(filter_even_numbers([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))
print(filter_even_numbers((1, 2, 3, 4, 5, 6, 7, 8, 9, 10)))
print(filter_even_numbers({1, 2, 3, 4, 5, 6, 7, 8, 9, 10}))

In this update, you import the Iterable class from collections.abc. This class implements the .__iter__() method, satisfying the iterable protocol. Being an iterable of integer values, this method is exactly what your filter_even_numbers() function needs to work properly.

Now, go ahead and run mypy again:

$ mypy even_v3.py
Success: no issues found in 1 source file

The output is now clean. This means that the type hints for the function work correctly. You can use this function with any input iterable of integer values.

Protocols vs Abstract Base Classes

An abstract base class (ABC) is designed to be subclassed but not instantiated. This type of class defines a specific public interface (API) and enforces that interface in its subclasses. To create an abstract base class, you can use the ABC class from the abc module.

To illustrate how you’ll typically use ABCs, consider the following example:

from abc import ABC, abstractmethod
from math import pi

class Shape(ABC):
    @abstractmethod
    def get_area(self) -> float:
        pass

    @abstractmethod
    def get_perimeter(self) -> float:
        pass

class Circle(Shape):
    def __init__(self, radius) -> None:
        self.radius = radius

    def get_area(self) -> float:
        return pi * self.radius**2

    def get_perimeter(self) -> float:
        return 2 * pi * self.radius

class Square(Shape):
    def __init__(self, side) -> None:
        self.side = side

    def get_area(self) -> float:
        return self.side**2

    def get_perimeter(self) -> float:
        return 4 * self.side

In this example, the Shape class is an abstract base class. With this class, you’re stating that any class that you create as a Shape subclass must have the .get_area() and .get_perimeter() methods in their public interface. That’s what the Circle and Square classes do. If you fail to implement one of the required methods, then you’ll get an error.

You can use the Shape class to provide type hints to your client code. Go ahead and add the following code to your shapes_v1.py file:

# ...

def print_shape_info(shape: Shape):
    print(f"Area: {shape.get_area()}")
    print(f"Perimeter: {shape.get_perimeter()}")

circle = Circle(10)
square = Square(5)
print_shape_info(circle)
print_shape_info(square)

The type hint for the shape argument will work correctly because Circle and Square are subclasses of Shape.

In this example, you’ve used an ABC and inheritance to enforce a specific interface in a group of classes. However, sometimes it’s desirable to have a similar result without inheritance. That’s when you can use a protocol:

from math import pi
from typing import Protocol

class Shape(Protocol):
    def get_area(self) -> float:
        ...

    def get_perimeter(self) -> float:
        ...

class Circle:
    def __init__(self, radius) -> None:
        self.radius = radius

    def get_area(self) -> float:
        return pi * self.radius**2

    def get_perimeter(self) -> float:
        return 2 * pi * self.radius

class Square:
    def __init__(self, side) -> None:
        self.side = side

    def get_area(self) -> float:
        return self.side**2

    def get_perimeter(self) -> float:
        return 4 * self.side

def print_shape_info(shape: Shape):
    print(f"Area: {shape.get_area()}")
    print(f"Perimeter: {shape.get_perimeter()}")

circle = Circle(10)
square = Square(5)
print_shape_info(circle)
print_shape_info(square)

In this new implementation, you use a protocol instead of an ABC. Now, you don’t rely on inheritance for your type hints to work correctly. Your classes are decoupled from each other. Their only point of coincidence is that they have a piece of interface in common.

In short, the main difference between an abstract base class and a protocol is that the former works through a formal inheritance relationship, while the latter doesn’t need this relationship. Note that this difference doesn’t make ABCs better than protocols or vice versa. They have different use cases and purposes.

ABCs are suitable when you have control over the class hierarchy and want to define a consistent interface across subclasses. Meanwhile, protocols are useful in scenarios where modifying class hierarchies is impractical or when there’s no clear inheritance relationship between classes.

Considering Potential Downsides of Protocols

It’s worthwhile to mention a downside of protocols that ABCs don’t exhibit. For example, protocols can make the type checker accept a completely unrelated type, which happens to satisfy the protocol completely by accident but is otherwise inappropriate.

Here’s a relevant example:

from typing import Protocol

class Message(Protocol):
    def encode(self) -> bytes:
        ...

def send(message: Message) -> None:
    ...

send("Hello, World!")  # Passes the type checker

In this example, strings happen to have the .encode() method, which returns bytes and satisfies the protocol. However, you may want the send() function to only accept instances of custom classes that implement the Message protocol. This behavior creates a loophole in the type checking system.

Another potential downside of protocols is that isinstance() will raise an exception when used with them. Consider the following code that takes advantage of the shapes example from the previous section:

>>> from shapes_v2 import circle, Shape

>>> isinstance(circle, Shape)
Traceback (most recent call last):
    ...
TypeError: Instance and class checks can only be used with
    @runtime_checkable protocols

This check could be unreliable because protocols don’t set an inheritance relationship. If you need your classes to work with isinstance() even if they’re not subclasses, then you can use the @runtime_checkable decorator in the class definition:

from typing import Protocol, runtime_checkable

@runtime_checkable
class Shape(Protocol):
    ...

The @runtime_checkable decorator marks a protocol class as a runtime protocol so that you can use it with isinstance() and issubclass().

Here’s how the previous example works now:

>>> from shapes_v3 import circle, Shape

>>> isinstance(circle, Shape)
True

After decorating your Shape protocol class with @runtime_checkable, the built-in isinstance() function works as you wanted.

Conclusion

Now you know how to use protocols in Python. Protocols let you define a type relationship between objects without the burden of inheritance. This relationship is based on the internal structure of classes.

With protocols, you can perform structural subtyping or static duck typing using Python’s type hint system and external static type checkers, like mypy, Pyright, and Pyre.

In this tutorial, you’ve:

• Gained clarity about the different uses of the term protocol in Python
• Learned how type hints facilitate static type checking
• Learned how protocols provide support for static duck typing
• Created custom protocols with the Protocol class
• Understood the differences between protocols and ABCs

With this knowledge, you’re ready to get the most out of using Python’s type hint system and static type checkers.

Take the Quiz: Test your knowledge with our interactive “Python Protocols: Leveraging Structural Subtyping” quiz. You’ll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

Python Protocols: Leveraging Structural Subtyping

Take this quiz to test your understanding of how to create and use Python protocols while providing type hints for your functions, variables, classes, and methods.