Using pytest

00:00 In the previous lesson, I gave an overview of the course. In this lesson, I’ll introduce you to writing tests with pytest. Want a little challenge? Make an estimate of the number of times I say the word test in this course.

00:13 Write it down. At the end, I’ll tell you how close you were. If it helps, the course has five lessons, twenty-four slides, and eleven example code files.

00:23 Take a guess. See how you’ll do. Testing, testing! Is this thing on? A little Joan Rivers call-out for the old folks in the room. pytest is a third-party library, and as such, you should be using a virtual environment and using pip to install it.

00:39 Nothing tricky here, simply run pip install with the pytest library, and you’ll be all set. To get started, let me first show you what a typical test case looks like if you’re using the standard library’s unittest module.

00:54 A test case in unittest is inside of a class that inherits from the TestCase base class. Each method in the class that begins with .test_ is considered a test.

01:06 I’ve defined two here, one on line 5 and one on line 8. The first one will pass, and the second one will fail. Most tests contain some sort of assertion.

01:18 You have some code and then assert an expected value, with the test passing if the assertion passes. The TestCase base class has a dozen different assertion methods in it, which you get by inheriting. In this sample code, I’m using just one of them, self.assertTrue().

01:35 If the value passed into the assertion method resolves to True, then the assertion is valid, and in this case, the test passes. Of course, in the real world, you’re likely to be passing in an actual value or doing a comparison operation rather than just the True or False keywords. But you get the idea.

01:55 You’ll notice that there is a fair amount of boilerplate here. These are the shortest possible tests, and in order to use them, you have to import the module and declare a class. Let’s run these in the bottom window.

02:09 I’ve gone into a director here where the tests live. Throughout this course, I’m going to be doing this. I’m going to be picky about where I run things.

02:16 That’s because the testing tools look in subdirectories for files beginning with the name test. So I’ve got sample code divided up by directory so that I’m only running one collection at a time.

02:27 I can run all the tests found in this directory by invoking python using the -m argument, calling the unittest module.

02:38 The first thing you’ll see here is an F and a dot (.). There will be some sort of character for each test run. The F indicates the test run failed. The dot means it passed.

02:50 This tells me two tests were found: one ran, and one failed. The rest of the output is the details of the failure. It tells me what line of code was the source of the assertion failure and why it failed.

03:03 It is right, of course. False is not true. Let’s contrast this with the pytest way instead.

03:13 This is the pytest equivalent of the same test suite. A couple of things are different here. First, nothing to import. pytest is both a library name and a command.

03:23 When you run the command, it looks for files starting with the name test and runs any function inside the file that starts with the name test. Second, no need for a class to inherit from. Third—and this is the biggest difference—rather than using a parent class’s assorted assert methods, pytest just uses the assert keyword.

03:45 Anything that evaluates to False in an assertion is considered a failed test. In case you haven’t used the assert keyword before, one quick word of warning: it isn’t a function. Don’t put parentheses like you’re calling a function. That causes problems. As assert is just a keyword, any parentheses you use would get treated as a tuple, and non-empty tuples are considered true.

04:12 So you can end up asserting something completely different than what you were trying to assert. More recent versions of Python will warn you if you do this, but older versions don’t, so be very careful. All right, let’s run this.

04:29 Like before, I want to be in the right directory, so the command will only find the tests I’m interested in right now. And now I’ll run the pytest command.

04:40 That’s a lot. Let me just scroll up here. pytest starts out by telling you what it is running. You’ll see here that I’m on a Mac—that’s that darwin thing—running Python 3.10.0, pytest-7.1.1, with the default plugins installed. It then tells you what directory it is using to search for tests, how many tests it found, and what file they were found in. Like unittest, it has an indicator for each test, but in this case, it’s divided up by the file. Also like unittest, the F means failed, and the dot means success. After that, you get a list of all the failures that happened in your test suite.

05:21 Each failure tells you what function had the problem, what line it was, and what the error was. In this case, it’s assert False. At the bottom here, there is a quick summary. It feels redundant because everything more or less fits on the screen, but if you’ve got lots of failed tests, these lines at the bottom can be rather helpful. Because I like failure, let’s do that again. This time I’m going to use the -q argument to pytest.

05:50 -q means quiet. Because my test failed, it isn’t too quiet. You still get all the failure information, but the header about the versions and all the files is gone.

06:00 Just the dot and F indicators. If the tests had all passed, this would be significantly shorter.

06:09 Pretty much anything that has truthiness can be used in an insertion inside of a pytest function. This first example is a comparison, as is the second one, albeit a more complicated one,

06:24 and this third example uses the in operator to check if 37 is in the set that is created by the set comprehension in the rest of this blurb.

06:34 This example is also a good representation of the just because you can doesn’t mean you should school of coding.

06:45 Here is another example of a test suite. This one uses a class to organize the tests. Unlike unittest, this doesn’t have to inherit from anything specific.

06:54 It is just a plain old class. In addition to looking for functions, pytest also looks for classes that start with the word test and the methods inside of them that start with test_.

07:07 If you want to group your tests together, you can do it this way.

07:14 There is another way of grouping your tests together, which is you can label them with marks. When you call pytest on the command line, you can pass in an argument so that only the tests with a certain mark get run.

07:27 There are also a bunch of marks built into pytest that you can take advantage of. Let’s go look at some marks.

07:36 Marking a function requires two things. First, you use a decorator to indicate the mark on the test function, and second, you have to register the mark. I’ll show you that second part in a minute.

07:48 I have two tests in here, each of which have some marks. You can apply multiple decorator marks to a function so that it can be a member of multiple groups. When testing large suites of software, it is often common to have a smoke test, a set of tests that can be run quickly but provide a decent amount of coverage, and then a full regression test that might take a lot longer.

08:10 Here I’ve marked the test_smoke() function with both the smoke mark and the regression mark, so this function gets run with either of those marks.

08:20 The test_more() function only gets run in the regression case. This is probably overkill, as in all likelihood, the regression suite would just be all of the tests, and so you wouldn’t need to mark it explicitly. I’m doing it just to illustrate the point.

08:38 pytest requires you to register your marks to prevent you from running nothing by specifying a nonexistent mark on the command line. This registration is done in a file named pytest.ini.

08:50 Other configuration can go in this file as well, but here I’m showing the creation of the two marks I used in the test functions before. If you fail to do this registration, you’ll get a warning message if you try to call pytest with marks.

09:04 There are two marks declared in this file: one for smoke, one for regression. Anything to the right of a colon in a mark declaration is considered a comment.

09:14 All right, let’s go run these.

09:22 The -m argument specifies a mark. Here, I’ve run the smoke test. Note that only one test was run.

09:36 And there is the full regression, with both of the tests getting run.

09:43 pytest provides some marks for you built in. The skip mark indicates that a test should be skipped. The skipif mark takes an expression, and if it evaluates to True, skips the test.

09:57 The xfail mark indicates the test is expected to fail, and the suite will still be considered to pass if that happens.

10:06 You can see a full list of all the marks by running pytest with the --markers argument. In the next lesson, I’ll add some complication and feed some data to tests using fixtures.