Extracting pydoc Comments With autodoc

00:00 In the previous lesson, I showed you RST markup and how to use it in a Sphinx document. In this lesson, I’ll be talking about the autodoc extension that allows you to incorporate the comments in your code into your documentation.

00:14 Sphinx has a mechanism for writing extensions, and in fact, it ships with some. The one I’m going to be showing you is called autodoc, and it adds directives for extracting pydoc comments from your code and including them in your documentation.

00:28 autodoc comes with the Sphinx package, so you don’t have to install anything additional. You do have to configure it though. Open up your conf.py file and add sphinx.ext.autodoc as a string to the extensions list to enable it.

00:46 The autodoc extension directive that I use most frequently is automodule. This takes a module filename as an argument and causes autodoc to parse that file for pydoc comments. The names of classes and functions with pydoc comments get pulled into your documentation.

01:04 There are a bunch of different options available for the directive. You can explicitly list those members of the module you want to include, or you can leave it blank and have them all included.

01:16 You can specify whether to include private members in your docs, those being things that start with single or double underscores. The special-members option will include dunder methods from a class if present.

01:29 You can optionally list those dunder methods you are interested in explicitly as well. By default, autodoc only looks for things that have pydoc comments.

01:39 If you add the undoc-members option, it will also include members without pydoc comments. exclude-members allows you to specify anything you wish to exclude.

01:51 And member-order allows you to change what order they appear in. I typically have a method to my code-ordering madness when I write my code, so I like to use the bysource argument to member-order so that the docs show the same order as my code.

02:09 Rather than include a whole module, you can explicitly include a code thing. There are directives for classes, functions, decorators, data items, methods, and attributes.

02:23 To demonstrate all of this, I have added some source to my Serenity project. The serenity/ directory now has two directories inside of it: the docs/ one I created before and a new one for my source.

02:35 I’m going to add a new RST file to the docs that uses the automodule directive to pull in the contents of my source code. The source is split into two files, one file containing a class and the other a function.

02:51 This is what the whole project tree looks like. In the project directory, I added README.rst, and in the docs/ directory, I’ve added serenity.rst.

03:00 The src/ directory has the Serenity code in it. This module has a ship and actions and __init__ Python file. Inside __init__, I’ve put a variable containing the version number for my release.

03:16 I’m going to update the conf.py file to use this value so I don’t have to update the release number in two places.

03:25 Inside of your pydoc comments, you can use special notations to cross-reference to other pieces of code. All of these are variations on references for classes.

03:36 Relative referencing can be a bit finicky, and as you might move your classes into new files, I find the best thing to do is always just use the second one, the fully named module. Prefacing that with a tilde (~) will mean the resulting link will only show the class name if you prefer it to be shorter.

03:55 There are similar notations for functions, attributes, and modules themselves. In HTML, all of these will end up as links to the place in your document where that specific thing has been incorporated.

04:10 In addition to cross-referencing code, there are some notations you can use to help describe part of a function or method inside of your pydoc. If you use the param notation, Sphinx will build a little table of your arguments to the function.

04:25 The returns notation is to document what your function or method responds with.

04:32 If you want to bring a bit of style to your documentation, you can choose a different theme. Themes are available through PyPI and are pip installed.

04:42 Once you’ve got it installed, you need to tell the output that it is using the theme you want to do that. This is done in the conf.py file. I’ll be using the Read the Docs theme in this demo.

04:54 You can get that by pip installing sphinx-rtd-theme. Note that the package name uses hyphens, but the module uses underscores.

05:03 Remember that for when I show you the config. Okay, you’re all set to incorporate pydoc comments into your documentation. Let’s go take a look.

05:14 This is the conf.py file in the docs directory. I’ve messed with it a little bit. First off, I’ve changed the copyright variable from a simple string to a bit of code.

05:25 I grab the datetime class, use it to calculate the current year, and then check if the year is later than 2023. That’s the year of the recording. If it is, I make the copyright value have a range. If it’s still 2023, I just use the year and author like before. All this is possible because conf.py is, well, .py. This file gets run every time you build, so you can make your logic as convoluted as you like.

05:55 If you can do it in Python, you can do it in your config. Recall that I added a version variable to the __init__ file in the serenity module. It’s named __version__.

06:07 These three lines dynamically load that file and set the value of release in conf.py to the value of __version__.

06:16 This means I only have to bump my version number in one place, the __init__ file, in my code. My docs will stay in sync because conf.py reads it.

06:27 Scrolling down, the next change was to add the extensions configuration. As I mentioned before, that’s just a matter of adding a string with the full module name of the extension into this list. As autodoc comes with Sphinx, this is all you need to do to use the automodule directive and its friends.

06:47 Finally, I’m changing the look and feel of the HTML build by setting the html_theme value to the Read the Docs theme as promised. All right, with all that configuration in place, let’s go look at what I’ve done to my index.rst file.

07:08 Earlier I mentioned this little trick. Often the contents of your doc home page is going to be the same as the README for your GitHub repo. As GitHub supports the RST format, you can write a README.rst file then use the include directive to suck that into your index.rst file. This way, you don’t have to write the content twice.

07:32 Although I did change how release gets populated, this line doesn’t change. It’ll still pull it in, referencing the newly dynamically loaded variable from conf.py.

07:43 I’m going to put my pydoc stuff into a file called serenity.rst, so like with movie.rst, I’ve added it to my table of contents.

07:54 Let’s go look at the README file …

07:59 and here it is. This time I remembered to use the title style heading the way I should have before. Nothing fancy in this file, just a few sentences, but the magic is that it can be both my GitHub landing page for the project as well as the document homepage.

08:15 Now on to serenity.rst.

08:21 This is an RST file like any other. You can write as much or as little as you like here. If all your info is in your pydoc comments, this doc might be short, like in this example.

08:32 If you need to write something more or include images or whatnot, you can RST to your heart’s content. The first automodule directive here references the ship module.

08:43 Note that the name of the module is fully qualified. It needs both serenity and ship to find the right one. The members option means all the contents of this file will be parsed.

08:54 The special-members option means dunder methods will also be included in the output. My second automodule directive is similar, but this one is for the actions file. And that’s it.

09:07 With that in place, the docs are ready to go. Let’s look at the code that this file is going to be parsing.

09:15 This is my __init__ file. As promised, I have declared a __version__ variable to hold my release number. This is what gets read in by conf.py, and it can also be used by my module itself.

09:32 This is ship.py. Up at the top here, I have my first pydoc comment, the pydoc for the class itself. It’s actually good practice to put this here rather than in, say, the __init__, as __init__ doesn’t always get included. Speaking of __init__, this part of the pydoc is using the param notation to describe the passengers argument for the .__init__() method.

09:59 And similarly, here inside of the .crazy_ivan() method, I’m using the returns notation to indicate that .crazy_ivan() returns the new direction. On to actions.py.

10:14 The class marker here will result in a link to the Firefly class documentation. This second reference uses the angle bracket (<>) style to change how the name is displayed. And this is a reference to the .pilot attribute inside the class. Attributes don’t get linked, but they do get styled appropriately.

10:36 Note that although I’m using these references in the pydoc itself, you can use them anywhere in your documentation. If one of your RST files mentions a class or function, you can use the same notation there to deep link into the docs for that class or function. I’ll skip the boring make html part.

10:55 Let’s go see the result.

11:00 The new look and feel here is because of the Read the Docs theme being applied. The content next to my pointer is the README file that was included. Notice how the nav on the left and the table of contents now have the links to the Serenity Module.

11:18 Let me click that. And here is the autodoc content. A couple things to note: because I used the special-members option to the automodule directive but didn’t explicitly list which dunder methods, I’ve got all of them: .__init__() and .__weakref__().

11:37 If you only wanted .__init__(), you could specify that in the option to the automodule directive. Note how the parameters and returns markers are being presented, showing the description.

11:52 This is even clearer on the change_pilot() function where you have multiple arguments. A nice little listing is created.

11:59 The cross-reference here isn’t fantastic, as it points to the same page, but it is a fully qualified link with a hash actor. Clicking it does send me to the top of the class. Seeing as that’s all on the same page, there’s not much to see by doing that though.

12:17 One little caveat: autodoc actually imports your code. That means your code has to be importable. In the case of frameworks like Django, things can go wrong if you haven’t got the environment set up correctly. As conf.py is a script, you can do whatever setup you need though. This little snippet works for Django.

12:38 It adds the parent directory to the sys.path, imports django, and runs Django’s required setup. Depending on your framework and where you’ve put your code, this snippet would need to change. In fact, if I recall correctly, the code I pulled this out of didn’t use an src/ directory, so that path statement would likely need to change if Serenity was a Django thing.

12:59 You might have to fiddle a little bit based on what framework you’re using, but you get the general idea: whatever setup you need can be done in conf.py.

13:10 And there you go. You’ve got some docs. Next up, I’ll show you how to host them at Read the Docs.