Packaging Details

00:00 In the previous lesson, I showed you what happens when you have local Python files with the same name as installed libraries. In this lesson, I’ll show you what an installable package looks like.

00:11 Remember how I said I often use the word package to mean the installable thing, whereas technically it’s the directory structure? Well, this is why. You can bundle a package directory in a fashion that is installable and then for whatever reason, that thing still gets called a package.

00:27 See, Python likes shadowing so much, it uses it in its terminology.

00:31 The installable kind of package contains the directory kind of package, as well as some metadata about it. The result is the same kind of bundle that you get from PyPI when you pip install or however you get your packages.

00:45 You don’t have to upload an installable package to PyPI though. You can make it locally and install it using pip. It only requires an extra command line flag.

00:55 For better or for worse, the core development team at Python has decided that packaging should not be part of the core product. The idea is that people will create better tools and they’ll compete.

01:06 The result is the standards have sort of been suggestions. There is a blessed packaging team called the Python Packaging Authority, but there are lots of tools out there that aren’t by them, and as there were a lot of problems in the packaging space, many of the alternative tools raced ahead of recommendations to solve certain problems.

01:26 This is starting to settle down as the specs have gotten better, but it is still a bit of a wild west. See on the slide here where I wrote the word complicated?

01:36 Yeah, that’s me being nice. Hot mess would be more accurate. To be truly accurate, I’d have to use vocabulary my mother wouldn’t approve of.

01:45 This isn’t a course on packaging, so I’m not going to go over the different competitors or pros and cons, and I’ll be keeping it relatively simple. I’m going to be following the current best practices for tools that ship from the Python Packaging Authority as of the year of this recording, 2024.

02:00 That means using a pyproject.toml file to describe the package and putting your package directory into a directory named src. This technically is configurable and honestly, it’s something I personally don’t like, but I’ve found that most tools expect it to by default, and sometimes it’s just easier to go with expectations rather than fight them.

02:21 Not that you’ll need them for what you’ll be doing in this course, but these are the latest versions of the tools from the packaging authority and would be one way of creating an installable package and putting it up on PyPI.

02:30 Let’s go look at some code.

02:35 This is the directory structure for the installable package that I’m going to bundle up. Note that the package directory is inside of a src directory and that’s where the actual Python modules exist.

02:46 Alongside the src directory are two things: the pyproject.toml file that contains metadata about our installable package and a standalone Python script.

02:56 The standalone script isn’t necessary for the package to be installable, but it allows me to run the code inside of the package directly, and it’s quite common to see test scripts at this level.

03:07 Note that I’ve named the installable bundle the same thing as the module inside of it, but you don’t have to. For example, it’s quite common for third-party Django libraries to have their installable packages named beginning with django_, but the actual module you import into the code typically doesn’t include that.

03:24 I’m gonna start out by looking at gift.py inside the src directory.

03:31 Not a lot in here. I’m declaring a constant and printing out a message. Remember, this is inside the package directory, inside the src directory, and it’s a module that you’d import.

03:42 Now, stepping out of the src directory,

03:46 this is getgift.py, a script in the installable directory, but not in the module that would get installed. The purpose of this is to show you how to run the contents of the package.

03:56 Using the src directory means that Python isn’t going to know where to find the module. You have to do a little extra work. In an earlier lesson, I briefly mentioned the idea of a module path.

04:06 This is where Python looks to find things to load. As the current directory is in the module path, you don’t often have to be aware of the path because you can just run the modules where you are, except in this case, the package module isn’t in the current directory.

04:21 It’s hidden away in the src directory, so to use it, you have to add that directory to the search path. Module search path is inside the sys module and pathlib gets used to manipulate file names.

04:35 The first thing I need to do is find out where this script is as I want a fully qualified path of the src directory. The __file__ value is set by the interpreter and contains the path of the currently running script.

04:48 I use that value to create a path object. The .parent attribute contains a path for the directory that getgift.py is inside. The path object supports appending paths using the slash operator.

05:01 It’s not quite division, but it does look like a path. By sticking src on the end here, I have in the folder variable, a new path object pointing to the src directory.

05:12 Next, I insert that into the module search path. The module search path is in sys and it’s called path. It’s just a list, so I’ve used the .insert() method on the list to stick the string value of the src directory location into the start of the search path.

05:30 Note, I’ve cast the path object as a string because the path list expects strings, not path objects. Unfortunately, both of those things are called path, so it’s a little confusing.

05:42 Once I’ve updated the search path, I can import things like normal. These two lines import the constant from our gift module and then print out the result.

05:52 Let’s try this out, and there you go, you’ve got a red gift. Now let’s actually create that installable Python bundle I was talking about.

06:05 This is the pyproject.toml file. TOML is a text-based specification for storing dictionary and list-like data. It’s somewhat similar to the old INI file spec.

06:16 Python build tools expect to find a file named pyproject.toml and inside it expect a section called build-system. The square brackets here define a section, which is kind of like the name of a dictionary with the values underneath it being the key-value pairs in the dict.

06:33 The build-system section defines what tools are supposed to be used to build the installable bundle.

06:40 The first entry in this section specifies what library to use to build the bundle. The current version of setuptools is 72, so asking for something over 40.9 hopefully isn’t a big deal.

06:52 Before 40.9, setuptools expected extra configuration files on top of pyproject.toml, so by saying I want 40.9 or more, I’m ensuring I don’t need them.

07:04 There are actually different ways of bundling things together. You may have come across “egg” packages before. Well, the most recent format is “wheel” and it’s the recommended one, and that’s what’s being asked for at the end of this list.

07:17 Some program must be responsible for doing bundling. This value says that it is the setuptools program’s build_meta call that’s going to do the work.

07:27 If you were using third-party tools like Poetry, you might specify something else. The [project] section is where you put meta information about your installable package.

07:37 Normally, this section is significantly longer, containing things like author info and details about what’s inside the package. I’m keeping it small here just at bare minimum. The name of our package is, well, “package”, which might have been confusing. Bad teacher.

07:53 And each bundle needs a version number, which you would bump up if you release a new copy. There are ways to import this value from elsewhere, so it is only defined once in one place, but I’ve gone with hard coding for now just to keep the file simple.

08:07 With the pyproject.toml file defined, you’re ready to actually create the bundle.

08:12 I’m inside the parent package directory. From here, I use pip to install our bundle. Normally, pip goes off to PyPI, but if you add the -e argument, it installs from a local directory instead.

08:26 Remember, whenever playing with installable packages, whether from PyPI or elsewhere, you should always use a virtual environment.

08:39 Wow, that’s a lot. The important part is the bit on the end successfully installed package 0.1. Note, that’s the name and version numbers specified in the pyproject.toml file using the name and version number values.

08:53 Now, if I run pip freeze, I can see everything that’s installed and there’s our package. The @ symbol tells you that it’s local and points to the fully qualified path of the package.

09:05 The cool thing about this is you can change your module in place and Python still finds it. Local copies don’t have to be reinstalled. They use a link to the actual code.

09:16 Remember all that stuff I had to go through in getgift? Well, if I’m using the virtual environment that I installed this into, I no longer have to do that.

09:23 The module has been installed. Several lessons ago, I kind of glossed over relative imports, grabbing modules from the local directory. In the next lesson, you’ll see why that’s not recommended practice.