Python's doctest: Document and Test Your Code at Once

What doctest Is and How It Works

The doctest module is a lightweight testing framework that provides quick and straightforward test automation. It can read the test cases from your project’s documentation and your code’s docstrings. This framework is shipped with the Python interpreter and adheres to the batteries-included philosophy.

You can use doctest from either your code or your command line. To find and run your test cases, doctest follows a few steps:

1. Searches for text that looks like Python interactive sessions in your documentation and docstrings
2. Parses those pieces of text to distinguish between executable code and expected results
3. Runs the executable code like regular Python code
4. Compares the execution result with the expected result

The doctest framework searches for test cases in your documentation and the docstrings of packages, modules, functions, classes, and methods. It doesn’t search for test cases in any objects that you import.

In general, doctest interprets as executable Python code all those lines of text that start with the primary (>>>) or secondary (...) REPL prompts. The lines immediately after either prompt are understood as the code’s expected output or result.

What doctest Is Useful For

The doctest framework is well suited for the quick automation of acceptance tests at the integration and system testing levels. Acceptance tests are those tests that you run to determine if the specifications of a given project are met, while integration tests are intended to guarantee that different components of a project work correctly as a group.

Your doctest tests can live in your project’s documentation and your code’s docstrings. For example, a package-level docstring containing doctest tests is a great and fast way to do integration tests. At this level, you can test the entire package and the integration of its modules, classes, functions, and so on.

A set of high-level doctest tests is an excellent way to define a program’s specification up front. At the same time, lower-level unit tests let you design the individual building blocks of your program. Then it’s just a matter of letting your computer check the code against the tests any time you want.

At the class, method, and function level, doctest tests are a powerful tool for testing your code as you write it. You can gradually add test cases to your docstrings while you’re writing the code itself. This practice will allow you to generate more reliable and robust code, especially if you stick to the test-driven development principles.

In summary, you can use doctest for the following purposes:

• Writing quick and effective test cases to check your code as you write it
• Running acceptance, regression, and integration test cases on your projects, packages, and modules
• Checking if your docstrings are up-to-date and in sync with the target code
• Verifying if your projects’ documentation is up-to-date
• Writing hands-on tutorials for your projects, packages, and modules
• Illustrating how to use your projects’ APIs and what the expected input and output must be

Having doctest tests in your documentation and docstrings is an excellent way for your clients or teammates to run those tests when evaluating the features, specifications, and quality of your code.

Creating doctest Tests for Checking Returned and Printed Values

The first and probably most common use case of code testing is checking the return value of functions, methods, and other callables. You can do this with doctest tests. For example, say you have a function called add() that takes two numbers as arguments and returns their arithmetic sum:

# calculations.py

def add(a, b):
    return float(a + b)

This function adds two numbers together. Documenting your code is a good practice, so you can add a docstring to this function. Your docstring can look something like this:

# calculations.py

def add(a, b):
    """Compute and return the sum of two numbers.

    Usage examples:
    >>> add(4.0, 2.0)
    6.0
    >>> add(4, 2)
    6.0
    """
    return float(a + b)

This docstring includes two examples of how to use add(). Each example consists of an initial line that starts with Python’s primary interactive prompt, >>>. This line includes a call to add() with two numeric arguments. Then the example has a second line that contains the expected output, which matches the function’s expected return value.

In both examples, the expected output is a floating-point number, which is required because the function always returns this type of number.

You can run these tests with doctest. Go ahead and run the following command:

$ python -m doctest calculations.py

This command won’t issue any output to your screen. Displaying no output means that doctest ran all your test cases and didn’t find any failing tests.

If you want doctest to be verbose about the process of running your test, then use the -v switch:

$ python -m doctest -v calculations.py
Trying:
    add(4.0, 2.0)
Expecting:
    6.0
ok
Trying:
    add(4, 2)
Expecting:
    6.0
ok
1 items had no tests:
    calculations
1 items passed all tests:
   2 tests in calculations.add
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

Running doctest with the -v option produces detailed output that describes the test-running process. The first two highlighted lines show the actual tests and their corresponding expected output. The line immediately after the expected output of each test displays the word ok, meaning that the target test passed successfully. In this example, the two tests passed, as you can confirm in the last highlighted line.

Another common use case of doctest tests is to check the printed output of a given piece of code. Go ahead and create a new file called printed_output.py and add the following code to it:

# printed_output.py

def greet(name="World"):
    """Print a greeting to the screen.

    Usage examples:
    >>> greet("Pythonista")
    Hello, Pythonista!
    >>> greet()
    Hello, World!
    """
    print(f"Hello, {name}!")

This function takes a name as an argument and prints a greeting to the screen. You can run the test in this function’s docstring using doctest from your command line as usual:

$ python -m doctest -v printed_output.py
Trying:
    greet("Pythonista")
Expecting:
    Hello, Pythonista!
ok
Trying:
    greet()
Expecting:
    Hello, World!
ok
1 items had no tests:
    printed_output
1 items passed all tests:
   2 tests in printed_output.greet
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

These tests work as expected because the Python REPL displays returned and printed values on the screen. This behavior allows doctest to match both returned and printed values in your test cases.

Testing what a function prints on the screen is pretty straightforward using doctest. With other testing frameworks, doing this kind of test may be a bit more complicated. You would need to deal with the standard output stream, which may require advanced Python knowledge.

Understanding How doctest Matches Expected and Actual Test Output

In practice, doctest is very strict when matching expected output with actual results. For example, using integers instead of floating-point numbers will break the test cases for your add() function.

Other tiny details like using spaces or tabs, wrapping returned strings in double quotes, or inserting blank lines can also cause tests to break. Consider the following toy test cases as a sampling of the above issues:

# failing_tests.py

"""Sample failing tests.

The output must be an integer
>>> 5 + 7
12.0

The output must not contain quotes
>>> print("Hello, World!")
'Hello, World!'

The output must not use double quotes
>>> "Hello," + " World!"
"Hello, World!"

The output must not contain leading or trailing spaces
>>> print("Hello, World!")
  Hello, World!

The output must not be a blank line
>>> print()

"""

When you run these tests using doctest from your command line, you get lengthy output. Here’s a breakdown:

$ python -m doctest -v failing_tests.py
Trying:
    5 + 7
Expecting:
    12.0
**********************************************************************
File ".../failing_tests.py", line 6, in broken_tests
Failed example:
    5 + 7
Expected:
    12.0
Got:
    12

In this first piece of output, the expected result is a floating-point number. However, 5 + 7 returns an integer value of 12. Therefore, doctest flags the test as a failing one. The Expected: and Got: headings give you hints on the detected problem.

The next piece of output looks something like this:

Trying:
    print("Hello, World!")
Expecting:
    'Hello, World!'
**********************************************************************
File ".../failing_tests.py", line 10, in broken_tests
Failed example:
    print("Hello, World!")
Expected:
    'Hello, World!'
Got:
    Hello, World!

In this example, the expected output uses single quotes. However, the print() function issues its output without quotes, making the test fail.

The command’s output continues with the following:

Trying:
    "Hello," + "World!"
Expecting:
    "Hello, World!"
**********************************************************************
File ".../failing_tests.py", line 14, in broken_tests
Failed example:
    "Hello," + " World!"
Expected:
    "Hello, World!"
Got:
    'Hello, World!'

This piece of output shows another failing test. In this example, the problem is that Python uses single quotes rather than double quotes when displaying strings in an interactive section. Again, this tiny difference makes your test fail.

Next up, you get the following piece of output:

Trying:
    print("Hello, World!")
Expecting:
      Hello, World!
**********************************************************************
File ".../failing_tests.py", line 18, in broken_tests
Failed example:
    print("Hello, World!")
Expected:
      Hello, World!
Got:
    Hello, World!

In this example, the test fails because the expected output contains leading whitespaces. However, the actual output doesn’t have leading spaces.

The final piece of output goes as follows:

Trying:
    print()
Expecting nothing
**********************************************************************
File ".../failing_tests.py", line 22, in broken_tests
Failed example:
    print()
Expected nothing
Got:
    <BLANKLINE>
**********************************************************************
1 items had failures:
   5 of   5 in broken_tests
5 tests in 1 items.
0 passed and 5 failed.
***Test Failed*** 5 failures.

In a regular REPL session, calling print() without arguments displays a blank line. In a doctest test, a blank line means that the code you just executed doesn’t issue any output. That’s why the output of doctest says that nothing was expected, but <BLANKLINE> was obtained. You’ll learn more about this <BLANKLINE> placeholder tag in the section about the limitations of doctest.

To summarize, you must guarantee a perfect match between the actual test output and the expected output. So, make sure that the line immediately after every test case perfectly matches what you need your code to return or print.

Writing doctest Tests for Catching Exceptions

Besides testing for successful return values, you’ll often need to test code that’s expected to raise exceptions in response to errors or other issues.

The doctest module follows mostly the same rules when catching return values and exceptions. It searches for text that looks like a Python exception report or traceback and checks it against any exception that your code raises.

For example, say that you’ve added the following divide() function to your calculations.py file:

# calculations.py
# ...

def divide(a, b):
    return float(a / b)

This function takes two numbers as arguments and returns their quotient as a floating-point number. The function works as expected when the value of b isn’t 0, but it raises an exception for b == 0:

>>> from calculations import divide

>>> divide(84, 2)
42.0

>>> divide(15, 3)
5.0

>>> divide(42, -2)
-21.0

>>> divide(42, 0)
Traceback (most recent call last):
    ...
ZeroDivisionError: division by zero

The first three examples show that divide() works well when the divisor, b, is different from 0. However, when b is 0, the function breaks with a ZeroDivisionError. This exception signals that the operation isn’t allowed.

How can you test for this exception using a doctest test? Check out the docstring in the below code, especially the last test case:

# calculations.py
# ...

def divide(a, b):
    """Compute and return the quotient of two numbers.

    Usage examples:
    >>> divide(84, 2)
    42.0
    >>> divide(15, 3)
    5.0
    >>> divide(42, -2)
    -21.0

    >>> divide(42, 0)
    Traceback (most recent call last):
    ZeroDivisionError: division by zero
    """
    return float(a / b)

The first three tests work as expected. So, focus on the last test, especially on the highlighted lines. The first highlighted line holds a header that’s common to all exception tracebacks. The second highlighted line contains the actual exception and its specific message. These two lines are the only requirement for doctest to successfully check for expected exceptions.

When dealing with exception tracebacks, doctest completely ignores the traceback body because it can change unexpectedly. In practice, doctest is only concerned about the first line, which reads Traceback (most recent call last):, and the last line. As you already know, the first line is common to all exception tracebacks, while the last line shows information about the raised exception.

Because doctest completely ignores the traceback body, you can do whatever you want with it in your docstrings. Typically, you’ll only include the traceback body if it adds significant value to your documentation. In terms of your options, you can:

1. Completely remove the traceback body
2. Replace parts of the traceback body with an ellipsis (...)
3. Completely replace the traceback body with an ellipsis
4. Replace the traceback body with any custom text or explanation
5. Include the complete traceback body

In any case, the traceback body only has meaning for humans reading your documentation. The second, fourth, and last options in this list will be useful only if the traceback adds value to your code’s documentation.

Here’s how the docstring of divide() would look if you included the complete traceback in the last test case:

# calculations.py
# ...

def divide(a, b):
    """Compute and return the quotient of two numbers.

    Usage examples:
    >>> divide(84, 2)
    42.0
    >>> divide(15, 3)
    5.0
    >>> divide(42, -2)
    -21.0

    >>> divide(42, 0)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
        divide(42, 0)
      File "<stdin>", line 2, in divide
        return float(a / b)
    ZeroDivisionError: division by zero
    """
    return float(a / b)

The traceback body shows information about the file and line that caused the exception. It also shows the entire stack trace down to the failing code line. Sometimes, this information can be useful when documenting your code.

In the example above, notice that if you include the complete traceback body, then you must keep the body’s original indentation. Otherwise, the test will fail. Now go ahead and run your test with doctest on your command line. Remember to use the -v switch to get verbose output.

Building More Elaborate doctest Tests

Often, you need to test functionality that depends on other objects in your code. For example, you may need to test the methods of a given class. To do this, you’ll need to instantiate the class first.

The doctest module is able to run code that creates and imports objects, calls functions, assigns variables, evaluates expressions, and more. You can take advantage of this capability to perform all kinds of preparation steps before running your actual test cases.

For example, say you’re writing a queue data structure and decide to use the deque data type from the collections module to implement it efficiently. After a few minutes of coding, you end up with the following code:

# queue.py

from collections import deque

class Queue:
    def __init__(self):
        self._elements = deque()

    def enqueue(self, element):
        self._elements.append(element)

    def dequeue(self):
        return self._elements.popleft()

    def __repr__(self):
        return f"{type(self).__name__}({list(self._elements)})"

Your Queue class only implements two basic queue operations, enqueue and dequeue. Enqueue allows you to add items or elements to the end of the queue, while dequeue lets you remove and return items from the beginning of the queue.

Queue also implements a .__repr__() method that provides the class’s string representation. This method will play an important role in writing and running your doctest tests, as you’ll explore in a moment.

Now say that you want to write doctest tests to guarantee that the .enqueue() and .dequeue() methods work fine. To do this, you first need to create an instance of Queue and populate it with some sample data:

 1# queue.py
 2
 3from collections import deque
 4
 5class Queue:
 6    def __init__(self):
 7        self._elements = deque()
 8
 9    def enqueue(self, element):
10        """Add items to the right end of the queue.
11
12        >>> numbers = Queue()
13        >>> numbers
14        Queue([])
15
16        >>> for number in range(1, 4):
17        ...     numbers.enqueue(number)
18
19        >>> numbers
20        Queue([1, 2, 3])
21        """
22        self._elements.append(element)
23
24    def dequeue(self):
25        """Remove and return an item from the left end of the queue.
26
27        >>> numbers = Queue()
28        >>> for number in range(1, 4):
29        ...     numbers.enqueue(number)
30        >>> numbers
31        Queue([1, 2, 3])
32
33        >>> numbers.dequeue()
34        1
35        >>> numbers.dequeue()
36        2
37        >>> numbers.dequeue()
38        3
39        >>> numbers
40        Queue([])
41        """
42        return self._elements.popleft()
43
44    def __repr__(self):
45        return f"{type(self).__name__}({list(self._elements)})"

Inside the docstring of enqueue(), you first run some setup steps. Line 12 creates an instance of Queue, while lines 13 and 14 check that the instance has been successfully created and is currently empty. Notice how you’ve used the custom string representation of Queue to express the output of this preparation step.

Lines 16 and 17 run a for loop that uses .enqueue() to populate the Queue instance with some sample data. In this case, .enqueue() doesn’t return anything, so you don’t have to check any return value. Finally, lines 19 and 20 run the actual test by confirming that the Queue instance now contains the sample data in the expected order.

In .dequeue(), lines 27 to 31 create a new instance of Queue, populate it with some sample data, and check that the data was successfully added. Again, these are setup steps that you need to run before testing the .dequeue() method itself.

The real tests appear on lines 33 to 41. In those lines, you call .dequeue() three times. Each call has its own output line. Finally, lines 39 and 40 verify that your instance of Queue is completely empty as a result of calling .dequeue().

An important point to highlight in the above example is that doctest runs individual docstrings in a dedicated context or scope. Therefore, names declared in one docstring can’t be used in another docstring. So, the numbers object defined in .enqueue() isn’t accessible in .dequeue(). You need to create a new Queue instance in .dequeue() before you can test this latter method.

You’ll dive deeper into how doctest manages the execution scope of your test cases in the Understanding the doctest Scoping Mechanism section.

Dealing With Whitespaces and Other Characters

Regarding characters such as whitespaces and backslashes, the rules are a bit complex. Expected outputs can’t consist of blank lines or lines containing only whitespace characters. Such lines are interpreted as the end of the expected output.

If your expected output includes blank lines, then you must use the <BLANKLINE> placeholder tag to replace them:

# greet.py

def greet(name="World"):
    """Print a greeting.

    Usage examples:
    >>> greet("Pythonista")
    Hello, Pythonista!
    <BLANKLINE>
    How have you been?
    """
    print(f"Hello, {name}!")
    print()
    print("How have you been?")

The expected output of greet() contains a blank line. To make your doctest test pass, you must use the <BLANKLINE> tag on every expected blank line, just like you did in the highlighted line above.

Tab characters are also complex to match when they appear in test outputs. Tabs in the expected output are automatically converted into spaces. In contrast, tabs in the actual output aren’t modified.

This behavior will make your tests fail because the expected and actual output won’t match. If your code’s output includes tabs, then you can make the doctest tests pass with the NORMALIZE_WHITESPACE option or directive. For an example of how to deal with tabs in your outputs, check out the Embedding Directives in Your doctest Tests section.

Backslashes also require special attention in your doctest tests. Tests that use backslashes for explicit line joining or other reasons must use a raw string, also known as an r-string, which will preserve your backslashes exactly as you type them:

# greet.py

def greet(name="World"):
    r"""Print a greeting.

    Usage examples:
    >>> greet("Pythonista")
    /== Hello, Pythonista! ==\
    \== How have you been? ==/
    """
    print(f"/== Hello, {name}! ==\\")
    print("\\== How have you been? ==/")

In this example, you use a raw string to write the docstring of this new version of greet(). Note the leading r in the docstring. Note that in the actual code, you double the backslash (\\) to escape it, but in the docstring you don’t need to double it.

If you don’t want to use a raw string as a way to escape backslashes, then you can use regular strings that escape the backslash by doubling it. Following this advice, you can also write the above test case like in the example below:

# greet.py

def greet(name="World"):
    """Print a greeting.

    Usage examples:
    >>> greet("Pythonista")
    /== Hello, Pythonista! ==\\
    \\== How have you been? ==/
    """
    print(f"/== Hello, {name}! ==\\")
    print("\\== How have you been? ==/")

In this new version of the test cases, you double the backslash characters to escape them in the expected output of your doctest tests.

Summarizing the doctest Test Syntax

As you already know, doctest recognizes tests by looking for pieces of text that mimic Python interactive sessions. According to this rule, lines starting with the >>> prompt are interpreted as simple statements, compound statement headers, or expressions. Similarly, lines beginning with the ... prompt are interpreted as continuation lines in compound statements.

Any lines that don’t start with >>> or ..., up to the next >>> prompt or blank line, represent the output that you expect from the code. The output must appear as it would in a Python interactive session, including both return values and printed outputs. Blank lines and the >>> prompt work as test separators or terminators.

If you don’t have any output lines between lines that start with >>> or ..., then doctest assumes that the statement is expected to have no output, which is the case when you call functions that return None or when you have assignment statements.

The doctest module ignores anything that doesn’t follow the doctest test syntax. This behavior allows you to include explanatory text, diagrams, or whatever you need in between your tests. You take advantage of this feature in the example below:

# calculations.py
# ...

def divide(a, b):
    """Compute and return the quotient of two numbers.

    Usage examples:
    >>> divide(84, 2)
    42.0
    >>> divide(15, 3)
    5.0
    >>> divide(42, -2)
    -21.0

    The test below checks if the function catches zero divisions:
    >>> divide(42, 0)
    Traceback (most recent call last):
    ZeroDivisionError: division by zero
    """
    return float(a / b)

In this update of divide(), you add explanatory text above the final test. Note that if the explanatory text is between two tests, then you need a blank line before the explanation itself. This blank line will tell doctest that the output of the previous test has finished.

Here’s a summary of doctest test syntax:

• Tests start after the >>> prompt and continue with the ... prompt, just like in a Python interactive session.
• Expected outputs must occupy the line or lines immediately after the test.
• Outputs sent to the standard output stream are captured.
• Outputs sent to the standard error stream aren’t captured.
• The column at which a test starts doesn’t matter as long as the expected output is at the same level of indentation.

The concepts of standard input and output streams are beyond the scope of this tutorial. To dive deeper into these concepts, check out the The Standard I/O Streams section in The subprocess Module: Wrapping Programs With Python.

Understanding the Output of Failing Tests

So far, you’ve mostly run successful doctest tests. However, in the real world, you’ll probably face many failing tests before you get your code working. In this section, you’ll learn how to interpret and understand the output that a failing doctest test produces.

When you have a failing test, doctest displays the failing test and the failure’s cause. You’ll have a line at the end of the test report that summarizes the successful and failing tests. For example, consider the following sample failing tests in a slightly modified version of your original failing_tests.py file:

# failing_tests.py

"""Sample failing tests.

The output must be an integer
>>> 5 + 7
12.0

The output must not contain quotes
>>> print("Hello, World!")
'Hello, World!'

The output must not use double quotes
>>> "Hello," + "World!"
"Hello, World!"

The output must not contain leading or trailing spaces
>>> print("Hello, World!")
  Hello, World!

The traceback doesn't include the correct exception message
>>> raise ValueError("incorrect value")
Traceback (most recent call last):
ValueError: invalid value
"""

This file contains a series of failing tests. The tests fail for different reasons. The comment before each test highlights the underlying cause of failure. If you run these tests using doctest from the command line, then you’ll get lengthy output. To better understand the output, you can break it into small chunks:

$ python -m doctest failing_tests.py
**********************************************************************
File "failing_tests.py", line 2, in failing_tests.py
Failed example:
    5 + 7
Expected:
    12.0
Got:
    12

In this first piece of output, the test fails because you’re using a floating-point number as the expected output. However, the actual output is an integer number. You can quickly spot the failing test by checking the line immediately after the Failed example: heading.

Similarly, you’ll find the expected output by checking the line below the Expected: heading, and the actual output is displayed below the Got: heading. By comparing the expected vs actual output, you’ll likely find the failure’s cause.

All failing tests will have a similar output format. You’ll find Expected: and Got: headings that’ll guide you to the problem that’s making the test fail:

**********************************************************************
File "failing_tests.py", line 6, in failing_tests.py
Failed example:
    print("Hello, World!")
Expected:
    'Hello, World!'
Got:
    Hello, World!
**********************************************************************
File "failing_tests.py", line 10, in failing_tests.py
Failed example:
    "Hello," + " World!"
Expected:
    "Hello, World!"
Got:
    'Hello, World!'
**********************************************************************
File "failing_tests.py", line 14, in failing_tests.py
Failed example:
    print("Hello, World!")
Expected:
      Hello, World!
Got:
    Hello, World!

The differences between the expected and the actual outputs can be pretty subtle, such as having no quotes, using double quotes instead of single quotes, or even accidentally inserting leading or trailing whitespaces.

When it comes to tests that check for raised exceptions, the output can get cluttered because of the exception traceback. However, a careful inspection will often guide you to the failure’s cause:

**********************************************************************
File "failing_tests.py", line 18, in failing_tests.py
Failed example:
    raise ValueError("incorrect value")
Expected:
    Traceback (most recent call last):
    ValueError: invalid value
Got:
    Traceback (most recent call last):
      ...
        raise ValueError("incorrect value")
    ValueError: incorrect value

In this example, the expected exception displays a message that’s slightly different from the message in the actual exception. Something like this can happen when you update the code but forget to update the corresponding doctest tests.

The final part of the output shows a summary of the failing tests:

**********************************************************************
1 items had failures:
   5 of   5 in broken_tests.txt
***Test Failed*** 5 failures.

In this example, all five tests have failed, as you can conclude from reading the last line. This last line has the following general format: ***Test Failed*** N failures. Here, N represents the number of failing tests in your code.

Including doctest Tests in Your Project’s Documentation

To kick things off with the doctest tests for this small project, you’ll start by creating README.md in the same directory that contains calculations.py. This README.md file will provide minimal documentation for your calculations.py file using the Markdown language:

<!-- README.md -->

# Functions to Perform Arithmetic Calculations

The `calculations.py` Python module provides basic arithmetic
operations, including addition, subtraction, multiplication, and division.

Here are a few examples of how to use the functions in `calculations.py`:

```python
>>> import calculations

>>> calculations.add(2, 2)
4.0

>>> calculations.subtract(2, 2)
0.0

>>> calculations.multiply(2, 2)
4.0

>>> calculations.divide(2, 2)
1.0

```

These examples show how to use the `calculations.py` module in your code.

This Markdown file includes a minimal description of your calculations.py file and a few usage examples wrapped in a Python code block. Note that the first line of code imports the module itself.

Another important detail is that you’ve included a blank line after the final test and right before the closing triple backticks (```). You need this blank line to signal that your doctest tests have finished, or else the triple backticks are taken to be expected output.

You can run the test in the above Markdown file using the doctest module as usual:

$ python -m doctest -v README.md
Trying:
    import calculations
Expecting nothing
ok
Trying:
    calculations.add(2, 2)
Expecting:
    4.0
ok
Trying:
    calculations.subtract(2, 2)
Expecting:
    0.0
ok
Trying:
    calculations.multiply(2, 2)
Expecting:
    4.0
ok
Trying:
    calculations.divide(2, 2)
Expecting:
    1.0
ok
1 items passed all tests:
   5 tests in README.md
5 tests in 1 items.
5 passed and 0 failed.
Test passed.

As you can confirm from the above output, all the doctest tests that live in your README.md file successfully ran, and all of them passed.

Adding Dedicated Test Files to Your Project

Another way to provide doctest tests in your project is by using a dedicated test file. To do this, you can use a plain text file. For example, you can use a file named test_calculations.txt with the following contents:

>>> import calculations

>>> calculations.add(2, 2)
4.0

>>> calculations.subtract(2, 2)
0.0

>>> calculations.multiply(2, 2)
4.0

>>> calculations.divide(2, 2)
1.0

This TXT file is a dedicated test file with a few doctest tests. Again, you can run these sample test cases using doctest from your command line:

$ python -m doctest -v test_calculations.txt
Trying:
    import calculations
Expecting nothing
ok
Trying:
    calculations.add(2, 2)
Expecting:
    4.0
ok
Trying:
    calculations.subtract(2, 2)
Expecting:
    0.0
ok
Trying:
    calculations.multiply(2, 2)
Expecting:
    4.0
ok
Trying:
    calculations.divide(2, 2)
Expecting:
    1.0
ok
1 items passed all tests:
   5 tests in test_calculations.txt
5 tests in 1 items.
5 passed and 0 failed.
Test passed.

All your doctest tests ran and passed successfully. A dedicated test file that you can run with doctest is a good option if you don’t want to clutter your documentation with too many doctest tests.

Embedding doctest Tests in Your Code’s Docstrings

The final, and probably the most common, way to provide doctest tests is through your project’s docstrings. With docstrings, you can have tests at different levels:

• Package
• Module
• Class and methods
• Functions

You can write package-level doctest tests inside the docstring of your package’s __init__.py file. The other tests will live in the docstrings of their respective container objects. For example, your calculations.py file has a module-level docstring containing doctest tests:

# calculations.py

"""Provide several sample math calculations.

This module allows the user to make mathematical calculations.

Module-level tests:
>>> add(2, 4)
6.0
>>> subtract(5, 3)
2.0
>>> multiply(2.0, 4.0)
8.0
>>> divide(4.0, 2)
2.0
"""

# ...

Likewise, you have function-level docstrings that contain doctest tests in all the functions you defined in calculations.py. Check them out!

If you turn back to the queue.py file where you defined the Queue class, then you can add class-level doctest tests like in the following code snippet:

# queue.py

from collections import deque

class Queue:
    """Implement a Queue data type.

    >>> Queue()
    Queue([])

    >>> numbers = Queue()
    >>> numbers
    Queue([])

    >>> for number in range(1, 4):
    ...     numbers.enqueue(number)
    >>> numbers
    Queue([1, 2, 3])
    """

    # ...

The doctest tests in the above docstring check how the Queue class works. This example only added tests for the .enqueue() method. Could you add tests for the .dequeue() method as well? That would be great exercise!

You can run all the doctest tests in your project’s docstrings from your command line, as you’ve done so far. But in the following sections, you’ll dive deeper into different ways to run your doctest tests.

Running doctest From Your Code

Python’s doctest module exports two functions that come in handy when you need to run doctest tests from your Python code rather than from your command line. These functions are the following:

Taking your test_calculations.txt as the starting point, you can use testfile() from your Python code to run the test in this file. To do this, you only need two lines of code:

# run_file_tests.py

import doctest

doctest.testfile("test_calculations.txt", verbose=True)

The first line imports doctest, while the second line calls testfile() with your test file as an argument. In the above example, you use the verbose argument, which makes the function produce detailed output, just like the -v option does when you run doctest from the command line. If you don’t set verbose to True, then testfile() won’t display any output unless you have a failing test.

The test file’s content is treated as a single docstring containing your doctest tests. The file doesn’t need to be a Python program or module.

The testfile() function takes several other optional arguments that allow you to customize further details in the process of running your tests. You must use keyword arguments to provide any of the function’s optional arguments. Check out the function’s documentation for more information about its arguments and their respective meanings.

If you need to run doctest tests that live in a regular Python module from your codebase, then you can use the testmod() function. You can use this function in two different ways. The first way consists of appending the following code snippet in the target module:

if __name__ == "__main__":
    import doctest
    doctest.testmod(verbose=True)

The name-main idiom allows you to execute code when the file runs as a script, but not when it’s imported as a module. Inside this conditional, you first import doctest and then call testmod() with verbose set to True. If you run the module as a script, then doctest will run all the tests that it finds in the module itself.

All the arguments to testmod() are optional. To provide them, you need to use keyword arguments for all except the first argument, which will optionally hold a module object.

The second way to run your doctest tests with testmod() is to create a dedicated test runner file. For example, if you want to run the tests in calculation.py without modifying the module itself, then you can create a run_module_tests.py file with the following content:

# run_module_tests.py

import doctest

import calculations

doctest.testmod(calculations, verbose=True)

This time, you need to import the target module, calculations, and pass the module object as the first argument to testmod(). This call will make doctest run all the tests defined in calculations.py. Go ahead and give it a try with the following command:

$ python run_module_tests.py

After running this command, you’ll get typical doctest output with all the details about the tests in your calculations module. Regarding the command’s output, it’s essential to keep in mind that if you don’t set verbose to True, then you won’t get any output unless you have failing tests. You’ll learn more about the output of failing tests in the following section.

Apart from the target module and the verbose flag, testmod() takes several other arguments that allow you to tweak different aspects of your test execution. Check out the function’s documentation for more details about the current arguments.

Finally, the functions in this section have the goal of making doctest straightforward to use. However, they give you limited customization capabilities. If you require more fine-grained control over the process of testing code with doctest, then you can use the module’s advanced API.

Executing doctest From Your Command Line

You already know the basics of running your doctest tests from the command line with the doctest command. The most bare-bones way to use this command is with the target file or module as an argument. For example, you can run all the tests in your calculations.py file by executing the following command:

$ python -m doctest calculations.py

This command runs the test but doesn’t issue any output unless you have some failing tests. That’s why you’ve used the -v switch in almost all the examples that you’ve run so far.

As you already learned, the -v or --verbose switch makes doctest issue a detailed report of all the tests it has run, along with a summary at the end of the report. Apart from this command-line option, doctest also accepts the following options:

You’ll probably be running doctest from your command line on most occasions. In the above table, you find that the most complex option is -o or --option because there’s a long list of flags that you can use with this option. You’ll learn more about these flags in the Using Flags at the Command Line section.

Using Flags at the Command Line

You can use flag constants when you run doctest from the command line using the -o or --option switch. For example, say that you have a test file called options.txt with the following content:

>>> 5 < 7
1

In this test, you use 1 as the expected output instead of using True. This test will pass because doctest allows True and False to be replaced with 1 and 0, respectively. This feature ties in with the fact that Python Boolean values can be expressed as integer numbers. So, if you run this file with doctest, then the test will pass.

Historically, doctest let Boolean values be replaced by 1 and 0 to ease the transition to Python 2.3, which introduced a dedicated Boolean type. However, this behavior may not be entirely correct in some situations. Fortunately, the DONT_ACCEPT_TRUE_FOR_1 flag will make this test fail:

$ python -m doctest -o DONT_ACCEPT_TRUE_FOR_1 options.txt
**********************************************************************
File "options.txt", line 3, in options.txt
Failed example:
    5 < 7
Expected:
    1
Got:
    True
**********************************************************************
1 items had failures:
   1 of   1 in options.txt
***Test Failed*** 1 failures.

By running the doctest command with the DONT_ACCEPT_TRUE_FOR_1 flag, you’re making the test strictly check for Boolean values, True or False, and fail with integer numbers. To fix the test, you must update the expected output from 1 to True. After that, you can run the test again, and it’ll pass.

Now say that you have a test with extensive output, and you need a way to abbreviate the expected output. In this situation, doctest allows you to use an ellipsis. Go ahead and add the following test to the end of your options.txt tile:

>>> print("Hello, Pythonista! Welcome to Real Python!")
Hello, ... Python!

If you run this file with doctest, then the second test will fail because the expected output doesn’t match the actual output. To avoid this failure, you can run doctest with the ELLIPSIS flag:

• Windows
• Linux + macOS

PS> python -m doctest `
> -o DONT_ACCEPT_TRUE_FOR_1 `
> -o ELLIPSIS options.txt

$ python -m doctest \
    -o DONT_ACCEPT_TRUE_FOR_1 \
    -o ELLIPSIS options.txt

This command won’t issue any output for your second test because you used the ELLIPSIS flag. This flag makes doctest understand that the ... character replaces part of the expected output.

Note that to pass multiple flags in a single run of doctest, you need to use the -o switch every time. Following this pattern, you can use as many flags as you need to make your tests more robust, strict, or flexible.

Dealing with whitespace characters like tabs is quite a challenging task because doctest automatically replaces them with regular spaces, making your test fail. Consider adding a new test to your options.txt file:

>>> print("\tHello, World!")
    Hello, World!

Even if you use tab characters in the test and its expected output, this test will fail because doctest internally replaces tabs with spaces in the expected output:

• Windows
• Linux + macOS

PS> python -m doctest `
> -o DONT_ACCEPT_TRUE_FOR_1 `
> -o ELLIPSIS options.txt
**********************************************************************
File "options.txt", line 9, in options.txt
Failed example:
    print("\tHello, World!")
Expected:
            Hello, World!
Got:
        Hello, World!
**********************************************************************
1 items had failures:
   1 of   3 in options.txt
***Test Failed*** 1 failures.

$ python -m doctest \
    -o DONT_ACCEPT_TRUE_FOR_1 \
    -o ELLIPSIS options.txt
**********************************************************************
File "options.txt", line 9, in options.txt
Failed example:
    print("\tHello, World!")
Expected:
            Hello, World!
Got:
        Hello, World!
**********************************************************************
1 items had failures:
   1 of   3 in options.txt
***Test Failed*** 1 failures.

If you ever have a test that issues a special whitespace character like this one, then you can use the NORMALIZE_WHITESPACE flag like in the following command:

• Windows
• Linux + macOS

PS> python -m doctest `
> -o DONT_ACCEPT_TRUE_FOR_1 `
> -o ELLIPSIS `
> -o NORMALIZE_WHITESPACE options.txt

$ python -m doctest \
    -o DONT_ACCEPT_TRUE_FOR_1 \
    -o ELLIPSIS \
    -o NORMALIZE_WHITESPACE options.txt

Now your output will be clean because doctest has normalized the tab characters for you.

Embedding Directives in Your doctest Tests

A doctest directive consists of an inline comment that begins with # doctest: and then includes a comma-separated list of flag constants. Directives either enable or disable a given doctest feature. To enable the feature, write a plus sign (+) before the flag name. To disable a feature, write a minus sign (–) instead.

Directives work similarly to flags when you use doctest from the command line. However, directives allow you more fine-grained control because they work on specific lines in your doctest tests.

For example, you can add some directives to your options.txt so that you don’t need to pass multiple command-line flags when running doctest:

>>> 5 < 7  # doctest: +DONT_ACCEPT_TRUE_FOR_1
True

>>> print(
...    "Hello, Pythonista! Welcome to Real Python!"
... )  # doctest: +ELLIPSIS
Hello, ... Python!

>>> print("\tHello, World!")  # doctest: +NORMALIZE_WHITESPACE
    Hello, World!

In this code, the highlighted lines insert inline directives alongside your tests. The first directive enables the obligatory use of Boolean values. The second directive allows you to use an ellipsis to abbreviate your test’s expected output. The final directive normalizes whitespace characters in the expected and actual outputs.

Now you can run the options.txt file without passing any flags to the doctest command:

$ python -m doctest options.txt

This command won’t issue any output, because the doctest directives have addressed all the requirements of your tests.

Flags and directives are quite similar in doctest. The main difference is that flags are intended to be used from the command line, and directives must be used in the tests themselves. In a sense, flags are more dynamic than directives. You can always find a good balance when using flags, directives, or both in your testing strategy.

Using unittest to Run doctest Tests

If you ever want to run doctest tests with unittest, then you can use the doctest API. The API allows you to convert doctest tests into unittest test suites. To do this, you’ll have two main functions:

To integrate your doctest tests with the unittest discovery mechanism, you must add a load_tests() function to your unittest boilerplate code. As an example, get back to your test_calculations.txt file:

>>> import calculations

>>> calculations.add(2, 2)
4.0

>>> calculations.subtract(2, 2)
0.0

>>> calculations.multiply(2, 2)
4.0

>>> calculations.divide(2, 2)
1.0

As you already know, this file contains doctest tests for your calculations.py file. Now say that you need to integrate the doctest tests within test_calculations.txt into your unittest infrastructure. In that case, you can do something like the following:

# test_calculations.py

import doctest
import unittest

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocFileSuite("test_calculations.txt"))
    return tests

# Your unittest tests goes here...

if __name__ == "__main__":
    unittest.main()

The load_tests() function is automatically called by unittest, and the framework discovers tests in your code. The highlighted line does the magic. It loads the doctest tests defined in test_calculations.txt and converts them into a unittest test suite.

Once you’ve added this function to your unittest infrastructure, then you can run the suite with the following command:

$ python test_calculations.py
.
---------------------------------------------------------------
Ran 1 test in 0.004s

OK

Cool! Your doctest tests ran successfully. From this output, you can conclude that unittest interprets the content of your test file as a single test, which is coherent with the fact that doctest interprets test files as a single docstring.

In the above example, all your tests passed. If you ever have a failing test, then you’ll get output that mimics the regular doctest output for failing tests.

If your doctest tests live in your code’s docstrings, then you can integrate them into your unittest suites with the following variation of load_tests():

# test_calculations.py

import doctest
import unittest

import calculations

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocTestSuite(calculations))
    return tests

# Your unittest goes here...

if __name__ == "__main__":
    unittest.main()

Instead of loading the doctest tests from a dedicated test file, you’re reading them from the calculations.py module using the DocTestSuite() function. If you run the above file now, then you’ll get the following output:

$ python test_calculations.py
.....
---------------------------------------------------------------
Ran 5 tests in 0.004s

OK

This time, the output reflects five tests. The reason is that your calculations.py file contains one module-level docstring and four function-level docstrings with doctest tests. Each independent docstring is interpreted as a separate test.

Finally, you can also combine tests from one or more text files and from a module inside your load_tests() function:

import doctest
import unittest

import calculations

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocFileSuite("test_calculations.txt"))
    tests.addTests(doctest.DocTestSuite(calculations))
    return tests

if __name__ == "__main__":
    unittest.main()

This version of load_tests() runs doctest tests from test_calculations.txt and the calculations.py module. Go ahead and run the above script from your command line. Your output will reflect six passing tests, including five tests from calculations.py and one test from test_calculations.txt. Remember that dedicated test files like test_calculations.txt are interpreted as a single test.

Using pytest to Run doctest Tests

If you decide to use the pytest third-party library to automate your project’s tests, then you can also integrate your doctest tests. In this case, you can use pytest’s --doctest-glob command-line option like in the example below:

$ pytest --doctest-glob="test_calculations.txt"

When you run this command, you get output like the following:

===================== test session starts =====================
platform darwin -- Python 3.10.3, pytest-7.1.1, pluggy-1.0.0
rootdir: .../python-doctest/examples
collected 1 item

test_calculations.txt .                                  [100%]

===================== 1 passed in 0.02s =======================

Just like unittest, pytest interprets your dedicated test file as a single test. The --doctest-glob option accepts and matches patterns that’ll allow you to run multiple files. A helpful pattern could be "test*.txt".

You can also execute doctest tests directly from your code’s docstrings. To do this, you can use the --doctest-modules command-line option. This command-line option will scan all the modules under your working directory, loading and running any doctest tests it finds.

If you want to make this integration permanent, then you can add the following parameter to pytest’s configuration file in your project’s root directory:

; pytest.ini

[pytest]
addopts = --doctest-modules

From now on, whenever you run pytest on your project’s directory, all the doctest tests will be found and executed.