Reviewing Packages and Virtual Environments
00:00 In the previous lesson, I gave an overview of the course. In this lesson, I’ll be showing you what a Python package is, how it fits into your coding landscape, and what all that has to do with virtual environments.
00:13 All the complexity I’ve been referring to comes about because of Python’s age. It actually predates the Web and was originally a local scripting solution.
00:22
There wasn’t much thought about shipping anything except maybe the scripts themselves, and packages really weren’t a consideration. It wasn’t until almost ten years into its existence that Python started thinking about package support, and thus, distils was born. Three years later, the Python Cheese Shop was created. Python The language is named after Monty Python the comedy troupe, not the slithering, apple-tempting things.
00:48 The Cheese Shop is a famous Monty Python sketch, so when it came time to think about a site, it got named for the sketch. In fact, the original URL still exists and redirects to PyPI.
00:59 Not only was it named in a funny way, but it didn’t do much either. The original PyPI was only a listing of the existing packages out there. As package management advanced and multiple versions of Python and libraries started kicking around the same machine, a problem occurred: library and version conflicts.
01:18 Thus was born virtual environments. I’ll come back to that in a bit. And like all history, it continues to be made. For those who listen to the Real Python Podcast, you’ll have heard Mr. Bailey and I discuss this several times: things in the packaging world are adapting quickly.
01:35 I suspect I’ll be having to update this course in a short period of time.
01:41 All right, let’s get some terminology out of the way. And like all terminology, there’s the right way, and there’s the way everybody uses it. The script is more or less agreed upon. That’s a single file.
01:53 A module technically is a single Python file that can be imported, but colloquially, most developers tend to mean the directory the file is in and encapsulates the thing being imported.
02:05 A package is a collection of modules, but again, the real use of this differs a little bit. Typically, you’ll hear module to mean the code that is installed and being imported, while package typically gets used to mean the thing you’re downloading and installing, resulting in one or more of those modules that can be used.
02:23 And with all this kind of vocabulary stuff, if you look on the Internet, you’ll see arguments about who’s using it correctly or not. I could care less, but it would take a lot of effort. And why is this stuff so confusing?
02:34
Well, take the following example that doesn’t help. The -m argument to the Python interpreter lets you you directly execute some code. This particular snippet is a handy little tool that pretty-prints a JSON file.
02:47
You might think Hey, -m must mean module, and you’d kind of be right, but what -m is actually doing is looking for a file called __main__ inside the json.tool package. It then executes that file, so technically -m is referencing the package. -m, m for package. Yeah. Anyhow, don’t worry about it.
03:11 Just understand that depending on who you’re talking to, they may or may not be using the terminology the same way you are. I am going to stick with the colloquial.
03:20 It’ll be module for the thing you import and package for the structure of the thing you are downloading or attempting to upload.
03:29
Python uses the sys.path variable to store the list of directories that are searched for importing a module. The REPL session here shows me importing and displaying sys.path.
03:41
There’s a lot of stuff in here, most of it being the system library, but tacked on the end is a folder called site-packages/. When you install a third-party package, this is where it gets put. Note that the names of the directories might be different on your machine. This is the structure used on a mac.
04:01
The pip tool downloads packages from the PyPI Cheese Shop and installs them in the very same site-packages/ directory. This is me pip installing the requests library. In fact, funny story: pip itself is a Python module, and so you’ll also see it written like this.
04:21 A lot of articles on the Web use this style, as it means the only thing that has to be on your executable path is Python itself. It’s a little safer.
04:30
And there’s that crazy -m thing again, this time pointing at the pip module. Inhale. Like I said, either variation on this command does the same thing.
04:40
It fetches and installs the most-excellent requests library.
04:45
The requests library depends on other libraries. Thankfully, pip knows this and installs the others for you. One of those dependencies is the urllib3 library, and requests needs a version between 1.21.1 and 1.27,
05:01
and that’s where the problem arises. What if you need urllib3 in one of your scripts, and it’s got to be 1.20 to work? Well, now you have a problem.
05:13
Enter virtual environments. A virtual environment is a miniature copy of your Python setup inside a dedicated directory. The end result is that it gets its own site-packages/ folder, meaning whatever is installed in one environment won’t interfere with another environment. If you need the urllib3 library version 1.20 for one of your projects and requests for another, you just give them separate virtual environments.
05:41 There are several tools out there for creating and managing these environments. This course doesn’t get into them in details. I mention them as they come about because of the nature of packages and their dependencies. To save some disk space, the virtual environment isn’t a complete copy of Python, but links to the important bits.
05:58 The added benefit of this is if you go and install an update, say from Python 3.11.1 to Python 3.11.2, you don’t have to re-create your 3.11 virtual environments.
06:08 They all just point to the 3.11 interpreter and get the upgrade for free.
06:14 Your mission, if you choose to accept it, is to create an installable package and upload it to PyPI. As always, should you be caught, the secretary will disavow any knowledge of your actions.
06:27 We can’t afford the music rights, so … [singing Mission:Impossible theme]
06:40 All right, enough dated TV show references. Yep, I’m old. It was a TV show first. Sorry, Tom Cruise. Next up, what does a package actually look like?
Christopher Trudeau RP Team on April 1, 2024
Hi Ori,
Packaging is one of those things that is still very much in flux in the Python world. The good news is: the concepts covered in this course still apply.
The “Exploring Other Build Systems” section could now be longer, as there are some exciting new entries out there, but personally, I keep it simple using a pyproject.toml file, build and twine.
What packaging system works for you is a tricky question and it depends a lot on the kind of work you’re doing. For me, maintaining small libraries where I’m the most likely consumer, keeping it simple has worked. If you’re working in large teams, or maintaining platforms with lots of dependencies, then you may see a productivity gain by using other tools.
My recommendation is to start with the basics that fulfill your needs (whatever those are), and then if you’re finding you’re running into trouble or doing manual tasks a lot, then investigate which of the many tools scratch your particular itch.
Become a Member to join the conversation.
ori yungrais on March 31, 2024
Hi, are there any news about this topic? What packaging method you’ll recommend nowadays?