Python's doctest: Document and Test Your Code at Once

Python's doctest: Document and Test Your Code at Once

by Leodanis Pozo Ramos intermediate python testing

Are you interested in writing usage examples for your code that work as documentation and test cases simultaneously? If your answer is yes, then Python’s doctest module is for you. This module provides a testing framework that doesn’t have too steep a learning curve. It’ll allow you to use code examples for two purposes: documenting and testing your code.

Apart from allowing you to use your code’s documentation for testing the code itself, doctest will help you keep your code and its documentation in perfect sync at any moment.

In this tutorial, you’ll:

  • Write doctest tests in your code’s documentation and docstrings
  • Understand how doctest works internally
  • Explore the limitations and security implications of doctest
  • Use doctest for test-driven development
  • Run your doctest tests using different strategies and tools

You won’t have to install any third-party libraries or learn complex APIs to follow this tutorial. You only need to know the basics of Python programming and how to use the Python REPL or interactive shell.

Documenting Your Code With Examples and Tests

Almost all experienced programmers will tell you that documenting your code is a best practice. Some will say that code and its documentation are equally important and necessary. Others will tell you that documentation is even more important than the code itself.

In Python, you’ll find many ways to document a project, app, or even modules and scripts. Larger projects generally require dedicated external documentation. But in small projects, using explicit names, comments, and docstrings might be sufficient:

Python
 1"""This module implements functions to process iterables."""
 2
 3def find_value(value, iterable):
 4    """Return True if value is in iterable, False otherwise."""
 5    # Be explicit by using iteration instead of membership
 6    for item in iterable:
 7        if value == item:  # Find the target value by equality
 8            return True
 9    return False

Explicit names like find_value() help you clearly express the content and aim of a given object. Such names improve your code’s readability and maintainability.

Comments, like the ones on lines 5 and 7, are pieces of text that you insert at different places in your code to clarify what the code does and why. Note that a Python comment starts with a # symbol and can occupy its own line or be part of an existing line.

Comments have a couple of drawbacks:

  • They’re ignored by the interpreter or compiler, which makes them unreachable at runtime.
  • They often get outdated when the code evolves and the comments remain untouched.

Documentation strings, or simply docstrings, are a neat Python feature that can help you document your code as you go. The advantage of docstrings compared to comments is that the interpreter doesn’t ignore them. They’re a living part of your code.

Because docstrings are active parts of your code, you can access them at runtime. To do this, you can use the .__doc__ special attributes on your packages, modules, classes, methods, and functions.

Tools like MkDocs and Sphinx can take advantage of docstrings for generating project documentation automatically.

You can add docstrings to your packages, modules, classes, methods, and functions in Python. If you want to learn how to write good docstrings, then PEP 257 proposes a series of conventions and recommendations that you can follow.

When you write docstrings, a common practice is to embed usage examples for your code. Those examples typically simulate REPL sessions.

Embedding code examples in your docstrings provides an effective way to document the code and a quick way to test the code as you write it. Yes, your code examples can work as test cases if you write them in a proper manner and use the right tool to run them.

Embedding REPL-like code examples in your code helps you:

  • Keep the documentation in sync with the current state of your code
  • Express your code’s intended usage
  • Test your code as you write it

Those benefits sound neat! Now, how can you run the code examples that you’ve embedded in your documentation and docstrings? You can use Python’s doctest module from the standard library.

Getting to Know Python’s doctest Module

In this section, you’ll get to know Python’s doctest module. This module is part of the standard library, so you don’t have to install any third-party library to be able to use it in your day-to-day coding. Among other things, you’ll learn what doctest is and when to use this neat Python tool. To kick things off, you’ll start by diving into what doctest is.

What doctest Is and How It Works

The doctest module is a lightweight testing framework that provides quick and straightforward test automation. It can read the test cases from your project’s documentation and your code’s docstrings. This framework is shipped with the Python interpreter and adheres to the batteries-included philosophy.

You can use doctest from either your code or your command line. To find and run your test cases, doctest follows a few steps:

  1. Searches for text that looks like Python interactive sessions in your documentation and docstrings
  2. Parses those pieces of text to distinguish between executable code and expected results
  3. Runs the executable code like regular Python code
  4. Compares the execution result with the expected result

The doctest framework searches for test cases in your documentation and the docstrings of packages, modules, functions, classes, and methods. It doesn’t search for test cases in any objects that you import.

In general, doctest interprets as executable Python code all those lines of text that start with the primary (>>>) or secondary (...) REPL prompts. The lines immediately after either prompt are understood as the code’s expected output or result.

What doctest Is Useful For

The doctest framework is well suited for the quick automation of acceptance tests at the integration and system testing levels. Acceptance tests are those tests that you run to determine if the specifications of a given project are met, while integration tests are intended to guarantee that different components of a project work correctly as a group.

Your doctest tests can live in your project’s documentation and your code’s docstrings. For example, a package-level docstring containing doctest tests is a great and fast way to do integration tests. At this level, you can test the entire package and the integration of its modules, classes, functions, and so on.

A set of high-level doctest tests is an excellent way to define a program’s specification up front. At the same time, lower-level unit tests let you design the individual building blocks of your program. Then it’s just a matter of letting your computer check the code against the tests any time you want.

At the class, method, and function level, doctest tests are a powerful tool for testing your code as you write it. You can gradually add test cases to your docstrings while you’re writing the code itself. This practice will allow you to generate more reliable and robust code, especially if you stick to the test-driven development principles.

In summary, you can use doctest for the following purposes:

  • Writing quick and effective test cases to check your code as you write it
  • Running acceptance, regression, and integration test cases on your projects, packages, and modules
  • Checking if your docstrings are up-to-date and in sync with the target code
  • Verifying if your projects’ documentation is up-to-date
  • Writing hands-on tutorials for your projects, packages, and modules
  • Illustrating how to use your projects’ APIs and what the expected input and output must be

Having doctest tests in your documentation and docstrings is an excellent way for your clients or teammates to run those tests when evaluating the features, specifications, and quality of your code.

Writing Your Own doctest Tests in Python

Now that you know what doctest is and what you can use it for, you’ll learn how to use doctest to test your code. No particular setup is required because doctest is part of the Python standard library.

In the following sections, you’ll learn how to check the return value of functions, methods, and other callables. Similarly, you’ll understand how to check the printed output of a given piece of code.

You’ll also learn how to create test cases for code that must raise exceptions and how to run preparation steps before executing your test cases. Finally, you’ll review a few details about the doctest test syntax.

Creating doctest Tests for Checking Returned and Printed Values

The first and probably most common use case of code testing is checking the return value of functions, methods, and other callables. You can do this with doctest tests. For example, say you have a function called add() that takes two numbers as arguments and returns their arithmetic sum:

Python
# calculations.py

def add(a, b):
    return float(a + b)

This function adds two numbers together. Documenting your code is a good practice, so you can add a docstring to this function. Your docstring can look something like this:

Python
# calculations.py

def add(a, b):
    """Compute and return the sum of two numbers.

    Usage examples:
    >>> add(4.0, 2.0)
    6.0
    >>> add(4, 2)
    6.0
    """
    return float(a + b)

This docstring includes two examples of how to use add(). Each example consists of an initial line that starts with Python’s primary interactive prompt, >>>. This line includes a call to add() with two numeric arguments. Then the example has a second line that contains the expected output, which matches the function’s expected return value.

In both examples, the expected output is a floating-point number, which is required because the function always returns this type of number.

You can run these tests with doctest. Go ahead and run the following command:

Shell
$ python -m doctest calculations.py

This command won’t issue any output to your screen. Displaying no output means that doctest ran all your test cases and didn’t find any failing tests.

If you want doctest to be verbose about the process of running your test, then use the -v switch:

Shell
$ python -m doctest -v calculations.py
Trying:
    add(4.0, 2.0)
Expecting:
    6.0
ok
Trying:
    add(4, 2)
Expecting:
    6.0
ok
1 items had no tests:
    calculations
1 items passed all tests:
   2 tests in calculations.add
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

Running doctest with the -v option produces detailed output that describes the test-running process. The first two highlighted lines show the actual tests and their corresponding expected output. The line immediately after the expected output of each test displays the word ok, meaning that the target test passed successfully. In this example, the two tests passed, as you can confirm in the last highlighted line.

Another common use case of doctest tests is to check the printed output of a given piece of code. Go ahead and create a new file called printed_output.py and add the following code to it:

Python
# printed_output.py

def greet(name="World"):
    """Print a greeting to the screen.

    Usage examples:
    >>> greet("Pythonista")
    Hello, Pythonista!
    >>> greet()
    Hello, World!
    """
    print(f"Hello, {name}!")

This function takes a name as an argument and prints a greeting to the screen. You can run the test in this function’s docstring using doctest from your command line as usual:

Shell
$ python -m doctest -v printed_output.py
Trying:
    greet("Pythonista")
Expecting:
    Hello, Pythonista!
ok
Trying:
    greet()
Expecting:
    Hello, World!
ok
1 items had no tests:
    printed_output
1 items passed all tests:
   2 tests in printed_output.greet
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

These tests work as expected because the Python REPL displays returned and printed values on the screen. This behavior allows doctest to match both returned and printed values in your test cases.

Testing what a function prints on the screen is pretty straightforward using doctest. With other testing frameworks, doing this kind of test may be a bit more complicated. You would need to deal with the standard output stream, which may require advanced Python knowledge.

Understanding How doctest Matches Expected and Actual Test Output

In practice, doctest is very strict when matching expected output with actual results. For example, using integers instead of floating-point numbers will break the test cases for your add() function.

Other tiny details like using spaces or tabs, wrapping returned strings in double quotes, or inserting blank lines can also cause tests to break. Consider the following toy test cases as a sampling of the above issues:

Python
# failing_tests.py

"""Sample failing tests.

The output must be an integer
>>> 5 + 7
12.0

The output must not contain quotes
>>> print("Hello, World!")
'Hello, World!'

The output must not use double quotes
>>> "Hello," + " World!"
"Hello, World!"

The output must not contain leading or trailing spaces
>>> print("Hello, World!")
  Hello, World!

The output must not be a blank line
>>> print()

"""

When you run these tests using doctest from your command line, you get lengthy output. Here’s a breakdown:

Shell
$ python -m doctest -v failing_tests.py
Trying:
    5 + 7
Expecting:
    12.0
**********************************************************************
File ".../failing_tests.py", line 6, in broken_tests
Failed example:
    5 + 7
Expected:
    12.0
Got:
    12

In this first piece of output, the expected result is a floating-point number. However, 5 + 7 returns an integer value of 12. Therefore, doctest flags the test as a failing one. The Expected: and Got: headings give you hints on the detected problem.

The next piece of output looks something like this:

Shell
Trying:
    print("Hello, World!")
Expecting:
    'Hello, World!'
**********************************************************************
File ".../failing_tests.py", line 10, in broken_tests
Failed example:
    print("Hello, World!")
Expected:
    'Hello, World!'
Got:
    Hello, World!

In this example, the expected output uses single quotes. However, the print() function issues its output without quotes, making the test fail.

The command’s output continues with the following:

Shell
Trying:
    "Hello," + "World!"
Expecting:
    "Hello, World!"
**********************************************************************
File ".../failing_tests.py", line 14, in broken_tests
Failed example:
    "Hello," + " World!"
Expected:
    "Hello, World!"
Got:
    'Hello, World!'

This piece of output shows another failing test. In this example, the problem is that Python uses single quotes rather than double quotes when displaying strings in an interactive section. Again, this tiny difference makes your test fail.

Next up, you get the following piece of output:

Shell
Trying:
    print("Hello, World!")
Expecting:
      Hello, World!
**********************************************************************
File ".../failing_tests.py", line 18, in broken_tests
Failed example:
    print("Hello, World!")
Expected:
      Hello, World!
Got:
    Hello, World!

In this example, the test fails because the expected output contains leading whitespaces. However, the actual output doesn’t have leading spaces.

The final piece of output goes as follows:

Shell
Trying:
    print()
Expecting nothing
**********************************************************************
File ".../failing_tests.py", line 22, in broken_tests
Failed example:
    print()
Expected nothing
Got:
    <BLANKLINE>
**********************************************************************
1 items had failures:
   5 of   5 in broken_tests
5 tests in 1 items.
0 passed and 5 failed.
***Test Failed*** 5 failures.

In a regular REPL session, calling print() without arguments displays a blank line. In a doctest test, a blank line means that the code you just executed doesn’t issue any output. That’s why the output of doctest says that nothing was expected, but <BLANKLINE> was obtained. You’ll learn more about this <BLANKLINE> placeholder tag in the section about the limitations of doctest.

To summarize, you must guarantee a perfect match between the actual test output and the expected output. So, make sure that the line immediately after every test case perfectly matches what you need your code to return or print.

Writing doctest Tests for Catching Exceptions

Besides testing for successful return values, you’ll often need to test code that’s expected to raise exceptions in response to errors or other issues.

The doctest module follows mostly the same rules when catching return values and exceptions. It searches for text that looks like a Python exception report or traceback and checks it against any exception that your code raises.

For example, say that you’ve added the following divide() function to your calculations.py file:

Python
# calculations.py
# ...

def divide(a, b):
    return float(a / b)

This function takes two numbers as arguments and returns their quotient as a floating-point number. The function works as expected when the value of b isn’t 0, but it raises an exception for b == 0:

Python
>>> from calculations import divide

>>> divide(84, 2)
42.0

>>> divide(15, 3)
5.0

>>> divide(42, -2)
-21.0

>>> divide(42, 0)
Traceback (most recent call last):
    ...
ZeroDivisionError: division by zero

The first three examples show that divide() works well when the divisor, b, is different from 0. However, when b is 0, the function breaks with a ZeroDivisionError. This exception signals that the operation isn’t allowed.

How can you test for this exception using a doctest test? Check out the docstring in the below code, especially the last test case:

Python
# calculations.py
# ...

def divide(a, b):
    """Compute and return the quotient of two numbers.

    Usage examples:
    >>> divide(84, 2)
    42.0
    >>> divide(15, 3)
    5.0
    >>> divide(42, -2)
    -21.0

    >>> divide(42, 0)
    Traceback (most recent call last):
    ZeroDivisionError: division by zero
    """
    return float(a / b)

The first three tests work as expected. So, focus on the last test, especially on the highlighted lines. The first highlighted line holds a header that’s common to all exception tracebacks. The second highlighted line contains the actual exception and its specific message. These two lines are the only requirement for doctest to successfully check for expected exceptions.

When dealing with exception tracebacks, doctest completely ignores the traceback body because it can change unexpectedly. In practice, doctest is only concerned about the first line, which reads Traceback (most recent call last):, and the last line. As you already know, the first line is common to all exception tracebacks, while the last line shows information about the raised exception.

Because doctest completely ignores the traceback body, you can do whatever you want with it in your docstrings. Typically, you’ll only include the traceback body if it adds significant value to your documentation. In terms of your options, you can:

  1. Completely remove the traceback body
  2. Replace parts of the traceback body with an ellipsis (...)
  3. Completely replace the traceback body with an ellipsis
  4. Replace the traceback body with any custom text or explanation
  5. Include the complete traceback body

In any case, the traceback body only has meaning for humans reading your documentation. The second, fourth, and last options in this list will be useful only if the traceback adds value to your code’s documentation.

Here’s how the docstring of divide() would look if you included the complete traceback in the last test case:

Python
# calculations.py
# ...

def divide(a, b):
    """Compute and return the quotient of two numbers.

    Usage examples:
    >>> divide(84, 2)
    42.0
    >>> divide(15, 3)
    5.0
    >>> divide(42, -2)
    -21.0

    >>> divide(42, 0)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
        divide(42, 0)
      File "<stdin>", line 2, in divide
        return float(a / b)
    ZeroDivisionError: division by zero
    """
    return float(a / b)

The traceback body shows information about the file and line that caused the exception. It also shows the entire stack trace down to the failing code line. Sometimes, this information can be useful when documenting your code.

In the example above, notice that if you include the complete traceback body, then you must keep the body’s original indentation. Otherwise, the test will fail. Now go ahead and run your test with doctest on your command line. Remember to use the -v switch to get verbose output.

Building More Elaborate doctest Tests

Often, you need to test functionality that depends on other objects in your code. For example, you may need to test the methods of a given class. To do this, you’ll need to instantiate the class first.

The doctest module is able to run code that creates and imports objects, calls functions, assigns variables, evaluates expressions, and more. You can take advantage of this capability to perform all kinds of preparation steps before running your actual test cases.

For example, say you’re writing a queue data structure and decide to use the deque data type from the collections module to implement it efficiently. After a few minutes of coding, you end up with the following code:

Python
# queue.py

from collections import deque

class Queue:
    def __init__(self):
        self._elements = deque()

    def enqueue(self, element):
        self._elements.append(element)

    def dequeue(self):
        return self._elements.popleft()

    def __repr__(self):
        return f"{type(self).__name__}({list(self._elements)})"

Your Queue class only implements two basic queue operations, enqueue and dequeue. Enqueue allows you to add items or elements to the end of the queue, while dequeue lets you remove and return items from the beginning of the queue.

Queue also implements a .__repr__() method that provides the class’s string representation. This method will play an important role in writing and running your doctest tests, as you’ll explore in a moment.

Now say that you want to write doctest tests to guarantee that the .enqueue() and .dequeue() methods work fine. To do this, you first need to create an instance of Queue and populate it with some sample data:

Python
 1# queue.py
 2
 3from collections import deque
 4
 5class Queue:
 6    def __init__(self):
 7        self._elements = deque()
 8
 9    def enqueue(self, element):
10        """Add items to the right end of the queue.
11
12        >>> numbers = Queue()
13        >>> numbers
14        Queue([])
15
16        >>> for number in range(1, 4):
17        ...     numbers.enqueue(number)
18
19        >>> numbers
20        Queue([1, 2, 3])
21        """
22        self._elements.append(element)
23
24    def dequeue(self):
25        """Remove and return an item from the left end of the queue.
26
27        >>> numbers = Queue()
28        >>> for number in range(1, 4):
29        ...     numbers.enqueue(number)
30        >>> numbers
31        Queue([1, 2, 3])
32
33        >>> numbers.dequeue()
34        1
35        >>> numbers.dequeue()
36        2
37        >>> numbers.dequeue()
38        3
39        >>> numbers
40        Queue([])
41        """
42        return self._elements.popleft()
43
44    def __repr__(self):
45        return f"{type(self).__name__}({list(self._elements)})"

Inside the docstring of enqueue(), you first run some setup steps. Line 12 creates an instance of Queue, while lines 13 and 14 check that the instance has been successfully created and is currently empty. Notice how you’ve used the custom string representation of Queue to express the output of this preparation step.

Lines 16 and 17 run a for loop that uses .enqueue() to populate the Queue instance with some sample data. In this case, .enqueue() doesn’t return anything, so you don’t have to check any return value. Finally, lines 19 and 20 run the actual test by confirming that the Queue instance now contains the sample data in the expected order.

In .dequeue(), lines 27 to 31 create a new instance of Queue, populate it with some sample data, and check that the data was successfully added. Again, these are setup steps that you need to run before testing the .dequeue() method itself.

The real tests appear on lines 33 to 41. In those lines, you call .dequeue() three times. Each call has its own output line. Finally, lines 39 and 40 verify that your instance of Queue is completely empty as a result of calling .dequeue().

An important point to highlight in the above example is that doctest runs individual docstrings in a dedicated context or scope. Therefore, names declared in one docstring can’t be used in another docstring. So, the numbers object defined in .enqueue() isn’t accessible in .dequeue(). You need to create a new Queue instance in .dequeue() before you can test this latter method.

You’ll dive deeper into how doctest manages the execution scope of your test cases in the Understanding the doctest Scoping Mechanism section.

Dealing With Whitespaces and Other Characters

Regarding characters such as whitespaces and backslashes, the rules are a bit complex. Expected outputs can’t consist of blank lines or lines containing only whitespace characters. Such lines are interpreted as the end of the expected output.

If your expected output includes blank lines, then you must use the <BLANKLINE> placeholder tag to replace them:

Python
# greet.py

def greet(name="World"):
    """Print a greeting.

    Usage examples:
    >>> greet("Pythonista")
    Hello, Pythonista!
    <BLANKLINE>
    How have you been?
    """
    print(f"Hello, {name}!")
    print()
    print("How have you been?")

The expected output of greet() contains a blank line. To make your doctest test pass, you must use the <BLANKLINE> tag on every expected blank line, just like you did in the highlighted line above.

Tab characters are also complex to match when they appear in test outputs. Tabs in the expected output are automatically converted into spaces. In contrast, tabs in the actual output aren’t modified.

This behavior will make your tests fail because the expected and actual output won’t match. If your code’s output includes tabs, then you can make the doctest tests pass with the NORMALIZE_WHITESPACE option or directive. For an example of how to deal with tabs in your outputs, check out the Embedding Directives in Your doctest Tests section.

Backslashes also require special attention in your doctest tests. Tests that use backslashes for explicit line joining or other reasons must use a raw string, also known as an r-string, which will preserve your backslashes exactly as you type them:

Python
# greet.py

def greet(name="World"):
    r"""Print a greeting.

    Usage examples:
    >>> greet("Pythonista")
    /== Hello, Pythonista! ==\
    \== How have you been? ==/
    """
    print(f"/== Hello, {name}! ==\\")
    print("\\== How have you been? ==/")

In this example, you use a raw string to write the docstring of this new version of greet(). Note the leading r in the docstring. Note that in the actual code, you double the backslash (\\) to escape it, but in the docstring you don’t need to double it.

If you don’t want to use a raw string as a way to escape backslashes, then you can use regular strings that escape the backslash by doubling it. Following this advice, you can also write the above test case like in the example below:

Python
# greet.py

def greet(name="World"):
    """Print a greeting.

    Usage examples:
    >>> greet("Pythonista")
    /== Hello, Pythonista! ==\\
    \\== How have you been? ==/
    """
    print(f"/== Hello, {name}! ==\\")
    print("\\== How have you been? ==/")

In this new version of the test cases, you double the backslash characters to escape them in the expected output of your doctest tests.

Summarizing the doctest Test Syntax

As you already know, doctest recognizes tests by looking for pieces of text that mimic Python interactive sessions. According to this rule, lines starting with the >>> prompt are interpreted as simple statements, compound statement headers, or expressions. Similarly, lines beginning with the ... prompt are interpreted as continuation lines in compound statements.

Any lines that don’t start with >>> or ..., up to the next >>> prompt or blank line, represent the output that you expect from the code. The output must appear as it would in a Python interactive session, including both return values and printed outputs. Blank lines and the >>> prompt work as test separators or terminators.

If you don’t have any output lines between lines that start with >>> or ..., then doctest assumes that the statement is expected to have no output, which is the case when you call functions that return None or when you have assignment statements.

The doctest module ignores anything that doesn’t follow the doctest test syntax. This behavior allows you to include explanatory text, diagrams, or whatever you need in between your tests. You take advantage of this feature in the example below:

Python
# calculations.py
# ...

def divide(a, b):
    """Compute and return the quotient of two numbers.

    Usage examples:
    >>> divide(84, 2)
    42.0
    >>> divide(15, 3)
    5.0
    >>> divide(42, -2)
    -21.0

    The test below checks if the function catches zero divisions:
    >>> divide(42, 0)
    Traceback (most recent call last):
    ZeroDivisionError: division by zero
    """
    return float(a / b)

In this update of divide(), you add explanatory text above the final test. Note that if the explanatory text is between two tests, then you need a blank line before the explanation itself. This blank line will tell doctest that the output of the previous test has finished.

Here’s a summary of doctest test syntax:

  • Tests start after the >>> prompt and continue with the ... prompt, just like in a Python interactive session.
  • Expected outputs must occupy the line or lines immediately after the test.
  • Outputs sent to the standard output stream are captured.
  • Outputs sent to the standard error stream aren’t captured.
  • The column at which a test starts doesn’t matter as long as the expected output is at the same level of indentation.

The concepts of standard input and output streams are beyond the scope of this tutorial. To dive deeper into these concepts, check out the The Standard I/O Streams section in The subprocess Module: Wrapping Programs With Python.

Understanding the Output of Failing Tests

So far, you’ve mostly run successful doctest tests. However, in the real world, you’ll probably face many failing tests before you get your code working. In this section, you’ll learn how to interpret and understand the output that a failing doctest test produces.

When you have a failing test, doctest displays the failing test and the failure’s cause. You’ll have a line at the end of the test report that summarizes the successful and failing tests. For example, consider the following sample failing tests in a slightly modified version of your original failing_tests.py file:

Python
# failing_tests.py

"""Sample failing tests.

The output must be an integer
>>> 5 + 7
12.0

The output must not contain quotes
>>> print("Hello, World!")
'Hello, World!'

The output must not use double quotes
>>> "Hello," + "World!"
"Hello, World!"

The output must not contain leading or trailing spaces
>>> print("Hello, World!")
  Hello, World!

The traceback doesn't include the correct exception message
>>> raise ValueError("incorrect value")
Traceback (most recent call last):
ValueError: invalid value
"""

This file contains a series of failing tests. The tests fail for different reasons. The comment before each test highlights the underlying cause of failure. If you run these tests using doctest from the command line, then you’ll get lengthy output. To better understand the output, you can break it into small chunks:

Shell
$ python -m doctest failing_tests.py
**********************************************************************
File "failing_tests.py", line 2, in failing_tests.py
Failed example:
    5 + 7
Expected:
    12.0
Got:
    12

In this first piece of output, the test fails because you’re using a floating-point number as the expected output. However, the actual output is an integer number. You can quickly spot the failing test by checking the line immediately after the Failed example: heading.

Similarly, you’ll find the expected output by checking the line below the Expected: heading, and the actual output is displayed below the Got: heading. By comparing the expected vs actual output, you’ll likely find the failure’s cause.

All failing tests will have a similar output format. You’ll find Expected: and Got: headings that’ll guide you to the problem that’s making the test fail:

Shell
**********************************************************************
File "failing_tests.py", line 6, in failing_tests.py
Failed example:
    print("Hello, World!")
Expected:
    'Hello, World!'
Got:
    Hello, World!
**********************************************************************
File "failing_tests.py", line 10, in failing_tests.py
Failed example:
    "Hello," + " World!"
Expected:
    "Hello, World!"
Got:
    'Hello, World!'
**********************************************************************
File "failing_tests.py", line 14, in failing_tests.py
Failed example:
    print("Hello, World!")
Expected:
      Hello, World!
Got:
    Hello, World!

The differences between the expected and the actual outputs can be pretty subtle, such as having no quotes, using double quotes instead of single quotes, or even accidentally inserting leading or trailing whitespaces.

When it comes to tests that check for raised exceptions, the output can get cluttered because of the exception traceback. However, a careful inspection will often guide you to the failure’s cause:

Shell
**********************************************************************
File "failing_tests.py", line 18, in failing_tests.py
Failed example:
    raise ValueError("incorrect value")
Expected:
    Traceback (most recent call last):
    ValueError: invalid value
Got:
    Traceback (most recent call last):
      ...
        raise ValueError("incorrect value")
    ValueError: incorrect value

In this example, the expected exception displays a message that’s slightly different from the message in the actual exception. Something like this can happen when you update the code but forget to update the corresponding doctest tests.

The final part of the output shows a summary of the failing tests:

Shell
**********************************************************************
1 items had failures:
   5 of   5 in broken_tests.txt
***Test Failed*** 5 failures.

In this example, all five tests have failed, as you can conclude from reading the last line. This last line has the following general format: ***Test Failed*** N failures. Here, N represents the number of failing tests in your code.

Providing doctest Tests in Your Projects

With doctest, you can execute test cases from your documentation, your dedicated test files, and the docstrings in your code files.

In this section, you’ll use a module called calculations.py as a sample project. Then you’ll learn how to use doctest to run tests from the following parts of this small project:

  • The README.md file
  • A dedicated test file
  • Docstrings

In the following collapsible section, you’ll find the complete source code for the calculations.py file:

Python
# calculations.py

"""Provide several sample math calculations.

This module allows the user to make mathematical calculations.

Module-level tests:
>>> add(2, 4)
6.0
>>> subtract(5, 3)
2.0
>>> multiply(2.0, 4.0)
8.0
>>> divide(4.0, 2)
2.0
"""

def add(a, b):
    """Compute and return the sum of two numbers.

    Tests for add():
    >>> add(4.0, 2.0)
    6.0
    >>> add(4, 2)
    6.0
    """
    return float(a + b)

def subtract(a, b):
    """Calculate the difference of two numbers.

    Tests for subtract():
    >>> subtract(4.0, 2.0)
    2.0
    >>> subtract(4, 2)
    2.0
    """
    return float(a - b)

def multiply(a, b):
    """Compute and return the product of two numbers.

    Tests for multiply():
    >>> multiply(4.0, 2.0)
    8.0
    >>> multiply(4, 2)
    8.0
    """
    return float(a * b)

def divide(a, b):
    """Compute and return the quotient of two numbers.

    Tests for divide():
    >>> divide(4.0, 2.0)
    2.0
    >>> divide(4, 2)
    2.0
    >>> divide(4, 0)
    Traceback (most recent call last):
    ZeroDivisionError: division by zero
    """
    return float(a / b)

Save the above code in a file called calculations.py. Put this file in a directory with a proper name.

Including doctest Tests in Your Project’s Documentation

To kick things off with the doctest tests for this small project, you’ll start by creating README.md in the same directory that contains calculations.py. This README.md file will provide minimal documentation for your calculations.py file using the Markdown language:

Markdown Text
<!-- README.md -->

# Functions to Perform Arithmetic Calculations

The `calculations.py` Python module provides basic arithmetic
operations, including addition, subtraction, multiplication, and division.

Here are a few examples of how to use the functions in `calculations.py`:

```python
>>> import calculations

>>> calculations.add(2, 2)
4.0

>>> calculations.subtract(2, 2)
0.0

>>> calculations.multiply(2, 2)
4.0

>>> calculations.divide(2, 2)
1.0

```

These examples show how to use the `calculations.py` module in your code.

This Markdown file includes a minimal description of your calculations.py file and a few usage examples wrapped in a Python code block. Note that the first line of code imports the module itself.

Another important detail is that you’ve included a blank line after the final test and right before the closing triple backticks (```). You need this blank line to signal that your doctest tests have finished, or else the triple backticks are taken to be expected output.

You can run the test in the above Markdown file using the doctest module as usual:

Shell
$ python -m doctest -v README.md
Trying:
    import calculations
Expecting nothing
ok
Trying:
    calculations.add(2, 2)
Expecting:
    4.0
ok
Trying:
    calculations.subtract(2, 2)
Expecting:
    0.0
ok
Trying:
    calculations.multiply(2, 2)
Expecting:
    4.0
ok
Trying:
    calculations.divide(2, 2)
Expecting:
    1.0
ok
1 items passed all tests:
   5 tests in README.md
5 tests in 1 items.
5 passed and 0 failed.
Test passed.

As you can confirm from the above output, all the doctest tests that live in your README.md file successfully ran, and all of them passed.

Adding Dedicated Test Files to Your Project

Another way to provide doctest tests in your project is by using a dedicated test file. To do this, you can use a plain text file. For example, you can use a file named test_calculations.txt with the following contents:

Text
>>> import calculations

>>> calculations.add(2, 2)
4.0

>>> calculations.subtract(2, 2)
0.0

>>> calculations.multiply(2, 2)
4.0

>>> calculations.divide(2, 2)
1.0

This TXT file is a dedicated test file with a few doctest tests. Again, you can run these sample test cases using doctest from your command line:

Shell
$ python -m doctest -v test_calculations.txt
Trying:
    import calculations
Expecting nothing
ok
Trying:
    calculations.add(2, 2)
Expecting:
    4.0
ok
Trying:
    calculations.subtract(2, 2)
Expecting:
    0.0
ok
Trying:
    calculations.multiply(2, 2)
Expecting:
    4.0
ok
Trying:
    calculations.divide(2, 2)
Expecting:
    1.0
ok
1 items passed all tests:
   5 tests in test_calculations.txt
5 tests in 1 items.
5 passed and 0 failed.
Test passed.

All your doctest tests ran and passed successfully. A dedicated test file that you can run with doctest is a good option if you don’t want to clutter your documentation with too many doctest tests.

Embedding doctest Tests in Your Code’s Docstrings

The final, and probably the most common, way to provide doctest tests is through your project’s docstrings. With docstrings, you can have tests at different levels:

  • Package
  • Module
  • Class and methods
  • Functions

You can write package-level doctest tests inside the docstring of your package’s __init__.py file. The other tests will live in the docstrings of their respective container objects. For example, your calculations.py file has a module-level docstring containing doctest tests:

Python
# calculations.py

"""Provide several sample math calculations.

This module allows the user to make mathematical calculations.

Module-level tests:
>>> add(2, 4)
6.0
>>> subtract(5, 3)
2.0
>>> multiply(2.0, 4.0)
8.0
>>> divide(4.0, 2)
2.0
"""

# ...

Likewise, you have function-level docstrings that contain doctest tests in all the functions you defined in calculations.py. Check them out!

If you turn back to the queue.py file where you defined the Queue class, then you can add class-level doctest tests like in the following code snippet:

Python
# queue.py

from collections import deque

class Queue:
    """Implement a Queue data type.

    >>> Queue()
    Queue([])

    >>> numbers = Queue()
    >>> numbers
    Queue([])

    >>> for number in range(1, 4):
    ...     numbers.enqueue(number)
    >>> numbers
    Queue([1, 2, 3])
    """

    # ...

The doctest tests in the above docstring check how the Queue class works. This example only added tests for the .enqueue() method. Could you add tests for the .dequeue() method as well? That would be great exercise!

You can run all the doctest tests in your project’s docstrings from your command line, as you’ve done so far. But in the following sections, you’ll dive deeper into different ways to run your doctest tests.

Understanding the doctest Scoping Mechanism

An essential aspect of doctest is that it runs individual docstrings in a dedicated context or scope. When you run doctest against a given module, doctest creates a shallow copy of the module’s global scope. Then doctest creates a local scope with the variables defined in whichever docstring will be executed first.

Once the tests run, doctest cleans up its local scope, throwing away any local names. Therefore, local names declared in one docstring can’t be used in the next docstring. Every docstring will run in a custom local scope, but the doctest global scope is common for all the docstrings in the module.

Consider the following examples:

Python
# context.py

total = 100

def decrement_by(number):
    """Decrement the global total variable by a given number.

    >>> local_total = decrement_by(50)
    >>> local_total
    50

    Changes to total don't affect the code's global scope
    >>> total
    100
    """
    global total
    total -= number
    return total

def increment_by(number):
    """Increment the global total variable by a given number.

    The initial value of total's shallow copy is 50
    >>> increment_by(10)
    60

    The local_total variable is not defined in this test
    >>> local_total
    Traceback (most recent call last):
    NameError: name 'local_total' is not defined
    """
    global total
    total += number
    return total

If you run this file with doctest, then all the tests will pass. In decrement_by(), the first test defines a local variable, local_total, which ends with a value of 50. This value results from subtracting number from the global shallow copy of total. The second test shows that total keeps its original value of 100, confirming that doctest tests don’t affect the code’s global scope, only its shallow copy.

By creating a shallow copy of the module’s global scope, doctest ensures that running the tests doesn’t change the actual module’s global scope. However, changes to variables in the shallow copy of your global scope propagate to other doctest tests. That’s why the first test in increment_by() returns 60 instead of 110.

The second test in increment_by() confirms that the local scope is cleaned after the tests run. So, local variables defined in a docstring aren’t available to other docstrings. Cleaning up the local scope prevents inter-test dependencies so that traces of a given test case won’t cause other test cases to pass or fail.

When you use a dedicated test file to provide doctest tests, all the tests from this file run in the same execution scope. This way, the execution of a given test can affect the result of a later test. This behavior isn’t beneficial. Tests need to be independent of each other. Otherwise, knowing which test failed doesn’t give you clear clues about what’s wrong in your code.

In this case, you can give individual tests their own execution scopes by placing each test in its own file. This practice will solve the scope issue but will add extra effort to the test-running task.

Another way to give each test its own execution scope is to define each test within a function, as follows:

Text
>>> def test_add():
...     import calculations
...     return calculations.add(2, 4)
>>> test_add()
6.0

In this example, the only object in the shared scope is the test_add() function. The calculations module won’t be available.

The doctest scoping mechanism is mainly intended to guarantee the secure and independent execution of your doctest tests.

Exploring Some Limitations of doctest

Probably the most significant limitation of doctest compared to other testing frameworks is the lack of features equivalent to fixtures in pytest or the setup and teardown mechanisms in unittest. If you ever need setup and teardown code, then you’ll have to write it in every affected docstring. Alternatively, you can use the unittest API, which provides some setup and teardown options.

Another limitation of doctest is that it strictly compares the test’s expected output with the test’s actual output. The doctest module requires exact matches. If only a single character doesn’t match, then the test fails. This behavior makes it hard to test some Python objects correctly.

As an example of how this strict matching might get in the way, imagine that you’re testing a function that returns a set. In Python, sets don’t store their elements in any particular order, so your tests will fail most of the time because of the random order of elements.

Consider the following example that implements a User class:

Python
# user.py

class User:
    def __init__(self, name, favorite_colors):
        self.name = name
        self._favorite_colors = set(favorite_colors)

    @property
    def favorite_colors(self):
        """Return the user's favorite colors.

        Usage examples:
        >>> john = User("John", {"#797EF6", "#4ADEDE", "#1AA7EC"})
        >>> john.favorite_colors
        {'#797EF6', '#4ADEDE', '#1AA7EC'}
        """
        return self._favorite_colors

This User class takes name and a series of favorite colors. The class initializer converts the input colors into a set object. The favorite_colors() property returns the user’s favorite colors. Because sets store their elements in random order, your doctest tests will fail most of the time:

Shell
$ python -m doctest -v user.py
Trying:
    john = User("John", {"#797EF6", "#4ADEDE", "#1AA7EC"})
Expecting nothing
ok
Trying:
    john.favorite_colors
Expecting:
    {'#797EF6', '#4ADEDE', '#1AA7EC'}
**********************************************************************
File ".../user.py", line ?, in user.User.favorite_colors
Failed example:
    john.favorite_colors
Expected:
    {'#797EF6', '#4ADEDE', '#1AA7EC'}
Got:
    {'#797EF6', '#1AA7EC', '#4ADEDE'}
3 items had no tests:
    user
    user.User
    user.User.__init__
**********************************************************************
1 items had failures:
   1 of   2 in user.User.favorite_colors
2 tests in 4 items.
1 passed and 1 failed.
***Test Failed*** 1 failures.

The first test is the User instantiation, which doesn’t have any expected output because the result is assigned to a variable. The second test checks the expected output against the function’s actual output. The outputs are different because sets are unordered collections, which makes the test fail.

To work around this issue, you can use the built-in sorted() function in your doctest test:

Python
# user.py

class User:
    def __init__(self, name, favorite_colors):
        self.name = name
        self._favorite_colors = set(favorite_colors)

    @property
    def favorite_colors(self):
        """Return the user's favorite colors.

        Usage examples:
        >>> john = User("John", {"#797EF6", "#4ADEDE", "#1AA7EC"})
        >>> sorted(john.favorite_colors)
        ['#1AA7EC', '#4ADEDE', '#797EF6']
        """
        return self._favorite_colors

Now the second doctest test is wrapped in a call to sorted(), which returns a list of sorted colors. Note that you must also update the expected output to contain a list of sorted colors. Now the test will successfully pass. Go ahead and try it out!

The lack of parametrization capabilities is another limitation of doctest. Parametrization consists of providing a given test with multiple combinations of input arguments and expected outputs. The testing framework must take care of running the target test with every combination and checking if all of the combinations pass.

Parametrization allows you to quickly create multiple test cases with a single test function, which will increase your test coverage and boost your productivity. Even though doctest doesn’t support parametrization directly, you can simulate the feature with some handy techniques:

Python
# even_numbers.py

def get_even_numbers(numbers):
    """Return the even numbers in a list.

    >>> args = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
    >>> expected = [[2, 4], [6, 8], [10, 12]]

    >>> for arg, expected in zip(args, expected):
    ...     get_even_numbers(arg) == expected
    True
    True
    True
    """
    return [number for number in numbers if number % 2 == 0]

In this example, you first create two lists containing the input arguments and expected outputs for get_even_numbers(). The for loop iterates over the two lists in parallel using the built-in zip() function. Inside the loop, you run a test that compares the actual output of get_even_numbers() with the corresponding expected output.

Another challenging use case of doctest is to test an object’s creation when the object relies on the default string representation, object.__repr__(). The default string representation of Python objects will typically include the object’s memory address, which will vary from run to run, making your tests fail.

To continue with the User example, say that you want to add the following test to the class initializer:

Python
# user.py

class User:
    def __init__(self, name, favorite_colors):
        """Initialize instances of User.

        Usage examples:
        >>> User("John", {"#797EF6", "#4ADEDE", "#1AA7EC"})
        <user.User object at 0x103283970>
        """
        self.name = name
        self._favorite_colors = set(favorite_colors)

# ...

When you instantiate User, the default string representation is displayed. In this case, the output includes a memory address that varies from execution to execution. This variation will make your doctest test fail because the memory address will never match:

Shell
$ python -m doctest -v user.py
Trying:
    User("John", {"#797EF6", "#4ADEDE", "#1AA7EC"})
Expecting:
    <user.User object at 0x103283970>
**********************************************************************
File ".../user.py", line 40, in user.User.__init__
Failed example:
    User("John", {"#797EF6", "#4ADEDE", "#1AA7EC"})
Expected:
    <user.User object at 0x103283970>
Got:
    <user.User object at 0x10534b070>
2 items had no tests:
    user
    user.User
**********************************************************************
1 items had failures:
   1 of   1 in user.User.__init__
1 tests in 3 items.
0 passed and 1 failed.
***Test Failed*** 1 failures.

This kind of test always fails because every time you run the code, the User instance will occupy a different memory address. As a work-around for this issue, you can use the ELLIPSIS directive of doctest:

Python
# user.py

class User:
    def __init__(self, name, favorite_colors):
        """Initialize instances of User.

        Usage examples:
        >>> User("John", {"#797EF6", "#4ADEDE", "#1AA7EC"}) # doctest: +ELLIPSIS
        <user.User object at 0x...>
        """
        self.name = name
        self._favorite_colors = set(favorite_colors)

# ...

You’ve added a comment at the end of the highlighted line. This comment enables the ELLIPSIS directive on the test. Now you can replace the memory address with an ellipsis in the expected output. If you run the test now, then it’ll pass because doctest understands the ellipsis as a replacement for the varying portion of the test’s output.

A similar issue will appear when you have expected outputs that include an object’s identity, like in the example below:

Python
>>> id(1.0)
4402192272

You won’t be able to test code like this using doctest. In this example, you can’t use the ELLIPSIS directive because you would have to replace the complete output with an ellipsis, and doctest will interpret the three dots as a continuation prompt. Therefore it will look like the test doesn’t have output.

Consider the following demonstrative example:

Python
# identity.py

def get_id(obj):
    """Return the identity of an object.

    >>> get_id(1)  # doctest: +ELLIPSIS
    ...
    """
    return id(obj)

This function is just an example of how an object’s identity will make your test fail even if you use the ELLIPSIS directive. If you run this test with doctest, then you’ll get a failure:

Shell
$ python -m doctest -v identity.py
Trying:
    get_id(1)  # doctest: +ELLIPSIS
Expecting nothing
**********************************************************************
File ".../identity.py", line 4, in identity.get_id
Failed example:
    get_id(1)  # doctest: +ELLIPSIS
Expected nothing
Got:
    4340007152
1 items had no tests:
    identity
**********************************************************************
1 items had failures:
   1 of   1 in identity.get_id
1 tests in 2 items.
0 passed and 1 failed.
***Test Failed*** 1 failures.

As you can confirm from the highlighted lines in this output, doctest expects the output to be nothing, but an actual object identity was received. Therefore, the test fails.

A final topic to remember when you’re using doctest is that having many tests in a docstring can make the code hard to read and follow because a long docstring increases the distance between the function’s signature and its body.

Fortunately, this isn’t a big problem nowadays because most code editors allow you to fold the docstrings and focus on the code. Alternatively, you can move the tests to the module-level docstring or a dedicated test file.

Considering Security While Using doctest

Security is a common and crucial requirement in today’s information technology industry. Running code from external sources, including code that comes as strings or docstrings, always implies a security risk.

The doctest module internally uses exec() to execute any tests embedded in docstrings and documentation files, as you can confirm from the module’s source code:

Python
# doctest.py

class DocTestRunner:
    # ...

    def __run(self, test, compileflags, out):
        # ...
        try:
            # Don't blink!  This is where the user's code gets run.
            exec(
                compile(example.source, filename, "single", compileflags, True),
                test.globs
            )
            self.debugger.set_continue() # ==== Example Finished ====
            exception = None
        except KeyboardInterrupt:
        # ...

As the highlighted line points out, the user’s code runs in a call to exec(). This built-in function is well-known in the Python community for being a fairly risky tool that allows the execution of arbitrary code.

The doctest module isn’t immune to the potential security issues associated with exec(). So, if you ever get into an external codebase with doctest tests, then avoid running the tests until you carefully read through them and make sure that they’re safe to run on your computer.

Using doctest for Test-Driven Development

In practice, you can use two different approaches for writing and running your tests with doctest. The first approach consists of the following steps:

  1. Write your code.
  2. Run your code in a Python REPL.
  3. Copy the relevant REPL fragments into your docstrings or documentation.
  4. Run the tests using doctest.

The main drawback of this approach is that the implementation you write in step 1 may be faulty. It also goes against the test-driven development (TDD) philosophy because you’re writing the tests after you’ve written the code.

In contrast, the second approach involves writing the doctest tests before writing the code that passes those tests. The steps, in this case, are:

  1. Write the tests in your docstrings or documentation using doctest syntax.
  2. Write the code to pass the tests.
  3. Run the tests using doctest.

This approach preserves the spirit of TDD, which holds that you should write the test before the code itself.

As an example, say that you’re in a job interview, and the interviewer asks you to implement the FizzBuzz algorithm. You need to write a function that takes a list of numbers and replaces any number divisible by three with the word "fizz", and any number divisible by five with the word "buzz". If a number is divisible by three and five, then you must replace it with the "fizz buzz" string.

You want to write this function using the TDD technique to ensure reliability. So, you decide to use doctest tests as a quick solution. First, you write a test to check numbers that are divisible by three:

Python
# fizzbuzz.py

# Replace numbers that are divisible by 3 with "fizz"
def fizzbuzz(numbers):
    """Implement the Fizz buzz game.

    >>> fizzbuzz([3, 6, 9, 12])
    ['fizz', 'fizz', 'fizz', 'fizz']
    """

The function has no implementation yet. It only has a doctest test that checks if the function works as expected when the input numbers are divisible by three. Now you can run the test to check if it passes or not:

Shell
$ python -m doctest -v fizzbuzz.py
Trying:
    fizzbuzz([3, 6, 9, 12])
Expecting:
    ['fizz', 'fizz', 'fizz', 'fizz']
**********************************************************************
File ".../fizzbuzz.py", line 5, in fizzbuzz.fizzbuzz
Failed example:
    fizzbuzz([3, 6, 9, 12])
Expected:
    ['fizz', 'fizz', 'fizz', 'fizz']
Got nothing
1 items had no tests:
    fizzbuzz
**********************************************************************
1 items had failures:
   1 of   1 in fizzbuzz.fizzbuzz
1 tests in 2 items.
0 passed and 1 failed.
***Test Failed*** 1 failures.

This output tells you that you have one failing test, which fits with the fact that your function doesn’t have any code yet. Now you need to write code to pass the test:

Python
# fizzbuzz.py

# Replace numbers that are divisible by 3 with "fizz"
def fizzbuzz(numbers):
    """Implement the Fizz buzz game.

    >>> fizzbuzz([3, 6, 9, 12])
    ['fizz', 'fizz', 'fizz', 'fizz']
    """
    result = []
    for number in numbers:
        if number % 3 == 0:
            result.append("fizz")
        else:
            result.append(number)
    return result

Now your function iterates over the input numbers. In the loop, you use the modulo operator (%) in a conditional statement to check if the current number is divisible by three. If the check succeeds, then you append the "fizz" string to result, which initially holds an empty list object. Otherwise, you append the number itself.

If you run the test with doctest now, then you’ll get the following output:

Shell
python -m doctest -v fizzbuzz.py
Trying:
    fizzbuzz([3, 6, 9, 12])
Expecting:
    ['fizz', 'fizz', 'fizz', 'fizz']
ok
1 items had no tests:
    fizz
1 items passed all tests:
   1 tests in fizz.fizzbuzz
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

Cool! You’ve made the test pass. Now you need to test for numbers that are divisible by five. Here are the updated doctest tests along with the code to pass them:

Python
# fizzbuzz.py

# Replace numbers that are divisible by 5 with "buzz"
def fizzbuzz(numbers):
    """Implement the Fizz buzz game.

    >>> fizzbuzz([3, 6, 9, 12])
    ['fizz', 'fizz', 'fizz', 'fizz']

    >>> fizzbuzz([5, 10, 20, 25])
    ['buzz', 'buzz', 'buzz', 'buzz']
    """
    result = []
    for number in numbers:
        if number % 3 == 0:
            result.append("fizz")
        elif number % 5 == 0:
            result.append("buzz")
        else:
            result.append(number)
    return result

The first two highlighted lines provide the test and expected output for numbers divisible by five. The second pair of highlighted lines implements the code that runs the check and replaces the number with the desired string, "buzz". Go ahead and run the test to make sure that the code passes.

The final step is to check for numbers divisible by three and five. You can do this in a single step by checking for numbers divisible by fifteen. Here are the doctest tests and the required code updates:

Python
# fizzbuzz.py

# Replace numbers that are divisible by 3 and 5 with "fizz buzz"
def fizzbuzz(numbers):
    """Implement the Fizz buzz game.

    >>> fizzbuzz([3, 6, 9, 12])
    ['fizz', 'fizz', 'fizz', 'fizz']

    >>> fizzbuzz([5, 10, 20, 25])
    ['buzz', 'buzz', 'buzz', 'buzz']

    >>> fizzbuzz([15, 30, 45])
    ['fizz buzz', 'fizz buzz', 'fizz buzz']

    >>> fizzbuzz([3, 6, 5, 2, 15, 30])
    ['fizz', 'fizz', 'buzz', 2, 'fizz buzz', 'fizz buzz']
    """
    result = []
    for number in numbers:
        if number % 15 == 0:
            result.append("fizz buzz")
        elif number % 3 == 0:
            result.append("fizz")
        elif number % 5 == 0:
            result.append("buzz")
        else:
            result.append(number)
    return result

In this final update to your fizzbuzz() function, you add doctest tests for checking numbers that are divisible by three and five. You also add a final test to check the function with various numbers.

In the function’s body, you add a new branch at the beginning of your chained ifelif statement. This new branch checks for numbers that are divisible by three and five, replacing them with the "fizz buzz" string. Note that you need to place this check at the beginning of the chain because otherwise, the function won’t work well.

Running Your Python doctest Tests

Up to this point, you’ve run many doctest tests. To run them, you’ve used your command line and the doctest command with the -v option to generate verbose outputs. However, this isn’t the only way to run your doctest tests.

In the following sections, you’ll learn how to run doctest tests from inside your Python code. You’ll also learn additional details about running doctest from your command line or terminal.

Running doctest From Your Code

Python’s doctest module exports two functions that come in handy when you need to run doctest tests from your Python code rather than from your command line. These functions are the following:

Function Description
testfile() Runs doctest tests from a dedicated test file
testmod() Runs doctest tests from a Python module

Taking your test_calculations.txt as the starting point, you can use testfile() from your Python code to run the test in this file. To do this, you only need two lines of code:

Python
# run_file_tests.py

import doctest

doctest.testfile("test_calculations.txt", verbose=True)

The first line imports doctest, while the second line calls testfile() with your test file as an argument. In the above example, you use the verbose argument, which makes the function produce detailed output, just like the -v option does when you run doctest from the command line. If you don’t set verbose to True, then testfile() won’t display any output unless you have a failing test.

The test file’s content is treated as a single docstring containing your doctest tests. The file doesn’t need to be a Python program or module.

The testfile() function takes several other optional arguments that allow you to customize further details in the process of running your tests. You must use keyword arguments to provide any of the function’s optional arguments. Check out the function’s documentation for more information about its arguments and their respective meanings.

If you need to run doctest tests that live in a regular Python module from your codebase, then you can use the testmod() function. You can use this function in two different ways. The first way consists of appending the following code snippet in the target module:

Python
if __name__ == "__main__":
    import doctest
    doctest.testmod(verbose=True)

The name-main idiom allows you to execute code when the file runs as a script, but not when it’s imported as a module. Inside this conditional, you first import doctest and then call testmod() with verbose set to True. If you run the module as a script, then doctest will run all the tests that it finds in the module itself.

All the arguments to testmod() are optional. To provide them, you need to use keyword arguments for all except the first argument, which will optionally hold a module object.

The second way to run your doctest tests with testmod() is to create a dedicated test runner file. For example, if you want to run the tests in calculation.py without modifying the module itself, then you can create a run_module_tests.py file with the following content:

Python
# run_module_tests.py

import doctest

import calculations

doctest.testmod(calculations, verbose=True)

This time, you need to import the target module, calculations, and pass the module object as the first argument to testmod(). This call will make doctest run all the tests defined in calculations.py. Go ahead and give it a try with the following command:

Shell
$ python run_module_tests.py

After running this command, you’ll get typical doctest output with all the details about the tests in your calculations module. Regarding the command’s output, it’s essential to keep in mind that if you don’t set verbose to True, then you won’t get any output unless you have failing tests. You’ll learn more about the output of failing tests in the following section.

Apart from the target module and the verbose flag, testmod() takes several other arguments that allow you to tweak different aspects of your test execution. Check out the function’s documentation for more details about the current arguments.

Finally, the functions in this section have the goal of making doctest straightforward to use. However, they give you limited customization capabilities. If you require more fine-grained control over the process of testing code with doctest, then you can use the module’s advanced API.

Executing doctest From Your Command Line

You already know the basics of running your doctest tests from the command line with the doctest command. The most bare-bones way to use this command is with the target file or module as an argument. For example, you can run all the tests in your calculations.py file by executing the following command:

Shell
$ python -m doctest calculations.py

This command runs the test but doesn’t issue any output unless you have some failing tests. That’s why you’ve used the -v switch in almost all the examples that you’ve run so far.

As you already learned, the -v or --verbose switch makes doctest issue a detailed report of all the tests it has run, along with a summary at the end of the report. Apart from this command-line option, doctest also accepts the following options:

Option Description
-h, --help Shows command-line help for doctest
-o, --option Specifies one or more doctest option flags or directives to use while running your tests
-f, --fail-fast Stops running your doctest tests after the first failure

You’ll probably be running doctest from your command line on most occasions. In the above table, you find that the most complex option is -o or --option because there’s a long list of flags that you can use with this option. You’ll learn more about these flags in the Using Flags at the Command Line section.

Controlling the Behavior of doctest: Flags and Directives

The doctest module provides a series of named constants that you can use as flags when you run doctest from your command line with the -o or --option switch. You can also use these constants when you add directives to your doctest tests.

Using this set of constants either as command-line flags or as directives will allow you to control various behaviors of doctest, including:

  • Accepting True for 1
  • Rejecting blank lines
  • Normalizing whitespaces
  • Abbreviating outputs with an ellipsis (...)
  • Ignoring exception details like the exception message
  • Skipping a given test
  • Finishing after the first failing test

This list doesn’t include all the current options. You can check the documentation for the complete list of constants and their meanings.

In the following section, you’ll start by learning how to use this neat doctest feature from the command line.

Using Flags at the Command Line

You can use flag constants when you run doctest from the command line using the -o or --option switch. For example, say that you have a test file called options.txt with the following content:

Text
>>> 5 < 7
1

In this test, you use 1 as the expected output instead of using True. This test will pass because doctest allows True and False to be replaced with 1 and 0, respectively. This feature ties in with the fact that Python Boolean values can be expressed as integer numbers. So, if you run this file with doctest, then the test will pass.

Historically, doctest let Boolean values be replaced by 1 and 0 to ease the transition to Python 2.3, which introduced a dedicated Boolean type. However, this behavior may not be entirely correct in some situations. Fortunately, the DONT_ACCEPT_TRUE_FOR_1 flag will make this test fail:

Shell
$ python -m doctest -o DONT_ACCEPT_TRUE_FOR_1 options.txt
**********************************************************************
File "options.txt", line 3, in options.txt
Failed example:
    5 < 7
Expected:
    1
Got:
    True
**********************************************************************
1 items had failures:
   1 of   1 in options.txt
***Test Failed*** 1 failures.

By running the doctest command with the DONT_ACCEPT_TRUE_FOR_1 flag, you’re making the test strictly check for Boolean values, True or False, and fail with integer numbers. To fix the test, you must update the expected output from 1 to True. After that, you can run the test again, and it’ll pass.

Now say that you have a test with extensive output, and you need a way to abbreviate the expected output. In this situation, doctest allows you to use an ellipsis. Go ahead and add the following test to the end of your options.txt tile:

Text
>>> print("Hello, Pythonista! Welcome to Real Python!")
Hello, ... Python!

If you run this file with doctest, then the second test will fail because the expected output doesn’t match the actual output. To avoid this failure, you can run doctest with the ELLIPSIS flag:

Windows PowerShell
PS> python -m doctest `
> -o DONT_ACCEPT_TRUE_FOR_1 `
> -o ELLIPSIS options.txt
Shell
$ python -m doctest \
    -o DONT_ACCEPT_TRUE_FOR_1 \
    -o ELLIPSIS options.txt

This command won’t issue any output for your second test because you used the ELLIPSIS flag. This flag makes doctest understand that the ... character replaces part of the expected output.

Note that to pass multiple flags in a single run of doctest, you need to use the -o switch every time. Following this pattern, you can use as many flags as you need to make your tests more robust, strict, or flexible.

Dealing with whitespace characters like tabs is quite a challenging task because doctest automatically replaces them with regular spaces, making your test fail. Consider adding a new test to your options.txt file:

Text
>>> print("\tHello, World!")
    Hello, World!

Even if you use tab characters in the test and its expected output, this test will fail because doctest internally replaces tabs with spaces in the expected output:

Windows PowerShell
PS> python -m doctest `
> -o DONT_ACCEPT_TRUE_FOR_1 `
> -o ELLIPSIS options.txt
**********************************************************************
File "options.txt", line 9, in options.txt
Failed example:
    print("\tHello, World!")
Expected:
            Hello, World!
Got:
        Hello, World!
**********************************************************************
1 items had failures:
   1 of   3 in options.txt
***Test Failed*** 1 failures.
Shell
$ python -m doctest \
    -o DONT_ACCEPT_TRUE_FOR_1 \
    -o ELLIPSIS options.txt
**********************************************************************
File "options.txt", line 9, in options.txt
Failed example:
    print("\tHello, World!")
Expected:
            Hello, World!
Got:
        Hello, World!
**********************************************************************
1 items had failures:
   1 of   3 in options.txt
***Test Failed*** 1 failures.

If you ever have a test that issues a special whitespace character like this one, then you can use the NORMALIZE_WHITESPACE flag like in the following command:

Windows PowerShell
PS> python -m doctest `
> -o DONT_ACCEPT_TRUE_FOR_1 `
> -o ELLIPSIS `
> -o NORMALIZE_WHITESPACE options.txt
Shell
$ python -m doctest \
    -o DONT_ACCEPT_TRUE_FOR_1 \
    -o ELLIPSIS \
    -o NORMALIZE_WHITESPACE options.txt

Now your output will be clean because doctest has normalized the tab characters for you.

Embedding Directives in Your doctest Tests

A doctest directive consists of an inline comment that begins with # doctest: and then includes a comma-separated list of flag constants. Directives either enable or disable a given doctest feature. To enable the feature, write a plus sign (+) before the flag name. To disable a feature, write a minus sign () instead.

Directives work similarly to flags when you use doctest from the command line. However, directives allow you more fine-grained control because they work on specific lines in your doctest tests.

For example, you can add some directives to your options.txt so that you don’t need to pass multiple command-line flags when running doctest:

Text
>>> 5 < 7  # doctest: +DONT_ACCEPT_TRUE_FOR_1
True

>>> print(
...    "Hello, Pythonista! Welcome to Real Python!"
... )  # doctest: +ELLIPSIS
Hello, ... Python!

>>> print("\tHello, World!")  # doctest: +NORMALIZE_WHITESPACE
    Hello, World!

In this code, the highlighted lines insert inline directives alongside your tests. The first directive enables the obligatory use of Boolean values. The second directive allows you to use an ellipsis to abbreviate your test’s expected output. The final directive normalizes whitespace characters in the expected and actual outputs.

Now you can run the options.txt file without passing any flags to the doctest command:

Shell
$ python -m doctest options.txt

This command won’t issue any output, because the doctest directives have addressed all the requirements of your tests.

Flags and directives are quite similar in doctest. The main difference is that flags are intended to be used from the command line, and directives must be used in the tests themselves. In a sense, flags are more dynamic than directives. You can always find a good balance when using flags, directives, or both in your testing strategy.

Running doctest Tests With unittest and pytest

The doctest module provides an incredibly convenient way to add test cases to a project’s documentation. However, doctest isn’t a substitute for a full-fledged testing framework, like the standard-library unittest or the third-party pytest. This is especially true in large projects with an extensive and complex codebase. For this kind of project, doctest may not be sufficient.

To illustrate, say that you’re starting a new project to provide an innovative web service for a small number of clients. At this point, you think that using doctest for automating your testing process is okay because the project is small in size and scope. So, you embed a bunch of doctest tests in your documentation and docstrings, and everybody is happy.

Without warning, your project starts to grow in size and complexity. You’re now serving a more significant number of users, and they constantly ask for new features and bug fixes. Now your project needs to provide a more reliable service.

Because of this new condition, you’ve noticed that doctest tests aren’t flexible and powerful enough to ensure reliability. You need a full-featured testing framework with fixtures, setup and teardown mechanisms, parametrization, and more.

In this situation, you think that if you decide to use unittest or pytest, then you’ll have to rewrite all your old doctest tests. The good news is that you don’t have to. Both unittest and pytest can run doctest tests. This way, your old tests will automatically join your arsenal of test cases.

Using unittest to Run doctest Tests

If you ever want to run doctest tests with unittest, then you can use the doctest API. The API allows you to convert doctest tests into unittest test suites. To do this, you’ll have two main functions:

Function Description
DocFileSuite() Converts doctest tests from one or more text files into a unittest test suite
DocTestSuite() Converts doctest tests from a module into a unittest test suite

To integrate your doctest tests with the unittest discovery mechanism, you must add a load_tests() function to your unittest boilerplate code. As an example, get back to your test_calculations.txt file:

Text
>>> import calculations

>>> calculations.add(2, 2)
4.0

>>> calculations.subtract(2, 2)
0.0

>>> calculations.multiply(2, 2)
4.0

>>> calculations.divide(2, 2)
1.0

As you already know, this file contains doctest tests for your calculations.py file. Now say that you need to integrate the doctest tests within test_calculations.txt into your unittest infrastructure. In that case, you can do something like the following:

Python
# test_calculations.py

import doctest
import unittest

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocFileSuite("test_calculations.txt"))
    return tests

# Your unittest tests goes here...

if __name__ == "__main__":
    unittest.main()

The load_tests() function is automatically called by unittest, and the framework discovers tests in your code. The highlighted line does the magic. It loads the doctest tests defined in test_calculations.txt and converts them into a unittest test suite.

Once you’ve added this function to your unittest infrastructure, then you can run the suite with the following command:

Shell
$ python test_calculations.py
.
---------------------------------------------------------------
Ran 1 test in 0.004s

OK

Cool! Your doctest tests ran successfully. From this output, you can conclude that unittest interprets the content of your test file as a single test, which is coherent with the fact that doctest interprets test files as a single docstring.

In the above example, all your tests passed. If you ever have a failing test, then you’ll get output that mimics the regular doctest output for failing tests.

If your doctest tests live in your code’s docstrings, then you can integrate them into your unittest suites with the following variation of load_tests():

Python
# test_calculations.py

import doctest
import unittest

import calculations

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocTestSuite(calculations))
    return tests

# Your unittest goes here...

if __name__ == "__main__":
    unittest.main()

Instead of loading the doctest tests from a dedicated test file, you’re reading them from the calculations.py module using the DocTestSuite() function. If you run the above file now, then you’ll get the following output:

Shell
$ python test_calculations.py
.....
---------------------------------------------------------------
Ran 5 tests in 0.004s

OK

This time, the output reflects five tests. The reason is that your calculations.py file contains one module-level docstring and four function-level docstrings with doctest tests. Each independent docstring is interpreted as a separate test.

Finally, you can also combine tests from one or more text files and from a module inside your load_tests() function:

Python
import doctest
import unittest

import calculations

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocFileSuite("test_calculations.txt"))
    tests.addTests(doctest.DocTestSuite(calculations))
    return tests

if __name__ == "__main__":
    unittest.main()

This version of load_tests() runs doctest tests from test_calculations.txt and the calculations.py module. Go ahead and run the above script from your command line. Your output will reflect six passing tests, including five tests from calculations.py and one test from test_calculations.txt. Remember that dedicated test files like test_calculations.txt are interpreted as a single test.

Using pytest to Run doctest Tests

If you decide to use the pytest third-party library to automate your project’s tests, then you can also integrate your doctest tests. In this case, you can use pytest’s --doctest-glob command-line option like in the example below:

Shell
$ pytest --doctest-glob="test_calculations.txt"

When you run this command, you get output like the following:

pytest Output
===================== test session starts =====================
platform darwin -- Python 3.10.3, pytest-7.1.1, pluggy-1.0.0
rootdir: .../python-doctest/examples
collected 1 item

test_calculations.txt .                                  [100%]

===================== 1 passed in 0.02s =======================

Just like unittest, pytest interprets your dedicated test file as a single test. The --doctest-glob option accepts and matches patterns that’ll allow you to run multiple files. A helpful pattern could be "test*.txt".

You can also execute doctest tests directly from your code’s docstrings. To do this, you can use the --doctest-modules command-line option. This command-line option will scan all the modules under your working directory, loading and running any doctest tests it finds.

If you want to make this integration permanent, then you can add the following parameter to pytest’s configuration file in your project’s root directory:

Configuration File
; pytest.ini

[pytest]
addopts = --doctest-modules

From now on, whenever you run pytest on your project’s directory, all the doctest tests will be found and executed.

Conclusion

Now you know how to write code examples that work as documentation and test cases at the same time. To run your examples as test cases, you used the doctest module from the Python standard library. This module armed you with a quick testing framework with a low learning curve, allowing you to start automating your testing process immediately.

In this tutorial, you learned how to:

  • Add doctest tests to your documentation and docstrings
  • Work with Python’s doctest module
  • Work around the limitations and security implications of doctest
  • Use doctest with a test-driven development approach
  • Execute doctest tests using different strategies and tools

With doctest tests, you’ll be able to quickly automate your tests. You’ll also guarantee that your code and its documentation are in sync all the time.

🐍 Python Tricks 💌

Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

Python Tricks Dictionary Merge

About Leodanis Pozo Ramos

Leodanis is an industrial engineer who loves Python and software development. He's a self-taught Python developer with 6+ years of experience. He's an avid technical writer with a growing number of articles published on Real Python and other sites.

» More about Leodanis

Each tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are:

Master Real-World Python Skills With Unlimited Access to Real Python

Locked learning resources

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

Master Real-World Python Skills
With Unlimited Access to Real Python

Locked learning resources

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

What Do You Think?

Rate this article:

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal.


Looking for a real-time conversation? Visit the Real Python Community Chat or join the next “Office Hours” Live Q&A Session. Happy Pythoning!