Using Pipenv

00:00 In the previous lesson, I showed you how to use pip and where it puts any installed packages. In this lesson, I’ll introduce you to using pipenv to manage both your package installation and virtual environments.

00:13 In the previous lesson, you saw some of the problems of using pip on its own without a virtual env. If you have multiple projects with conflicting dependencies, keeping everything in the same site-packages directory is problematic.

00:25 A common way of managing your package versions is by including their information in a requirements.txt file. This also has some problems. If you only include your dependencies and not your dependents’ dependents inside of your requirements.txt file, your build won’t be deterministic.

00:43 Think back to the example of installing requests in the previous lesson. requests is dependent on certifi, amongst other things.

00:51 If certifi releases something new, your next installation will end up using a different set of libraries than your first installation. This is a good way for dev and production to be mismatched and can result in a “worked on my machine” kind of problem.

01:05 So instead of specifying only requests in your requirements.txt, you could redirect pip freeze and grab all the dependencies like I showed you in the previous lesson. This solves the problem above, but also means you’re now responsible for all of the packages.

01:21 You might want to get that new version of certifi because of a bug fix. Keeping everything in requirements.txt means you’re managing all of it.

01:30 Lastly, requirements.txt is one flat file, meaning it can only specify a single environment. What if your dev environment needs testing packages that your production environment doesn’t require?

01:41 I have seen some package maintainers have requirements and dev-requirements files, which works. But now you’ve got twice the things you have to maintain. As you might’ve guessed, like a late-night TV commercial talking about all the problems of dull knives, I’ve got a sharper knife to sell you. pipenv solves most of these problems.

01:59 If you hadn’t heard of it before, pipenv is a third-party tool maintained by the same people who maintain the core Python libraries. it uses pip and virtualenv under the hood to give you that sharper knife.

02:13 You may recall from the previous lesson that I did some of the work in a directory called without_pipenv. That is because pipenv does some magic things if it finds a requirements.txt file. So to avoid that, I’m going to create a new directory called using_pipenv and stay inside of that one to avoid any side effects.

02:34 pipenv itself is a third-party package, so the first step is to install it.

02:43 It is okay to have this in your site-packages directory, as you’re always going to want it. Once this is in place, it will be virtual envs from here on in. Let me just scroll up a bit.

02:55 Nothing here you haven’t seen before. Package is installed, along with all of its dependencies.

03:02 I’m going to create a new project called hola.

03:09 To create a new virtual environment, you use the pipenv shell command.

03:17 The shell command is used for entering a virtual environment, but if it doesn’t find one, it creates it, like in this case. There are a few things going on here. First off, it tells you that it is creating a virtual environment and a Pipfile.

03:31 The Pipfile is one of two files used to specify your packages. I’ll get into details with those in a second. Next, it tells you what version of Python it’s going to use. In my case, it was C-Python 3.10.

03:43 It also adds some useful base packages to the virtual environment—those for doing package management, pip, setuptools, and wheel. Four lines from the bottom, it tells you where the virtual env files will be stored.

03:55 pipenv uses a centrally located place for all of its files. Your location is OS specific. Mine is my home directory .local/share/virtualenvs. Note that it names the virtual env with the name of my directory and a hash.

04:10 This naming is how it knows which environment to activate on subsequent calls to the shell command. Finally, pipenv activates my virtual environment.

04:19 You can see that it modifies the prompt to tell me what environment and directory I am in. Depending on your settings, this may also include your host and username information.

04:30 The virtual environment is created in a central location, but creating it creates a local file, Pipfile. This, combined with another file called Pipfile.lock, is the replacement for requirements.txt.

04:43 These files specify information about the packages you’re using in your project. Let’s take a look at one.

04:52 Here’s the default Pipfile before you install anything. It uses the TOML format, which is similar to Windows INI files. Sections in the file are denoted by double square brackets, and the sections themselves contain key/value pairs.

05:07 The source section specifies where to get packages from. This defaults to the PyPI repository. The packages and dev-packages section specifies packages that are installed. Right now, these are empty, and the requires section specifies the version of Python being used.

05:25 Now that you’ve seen the file, let’s install something and look at the difference. Let’s repeat what was done in the previous lesson, but this time in a virtual environment.

05:39 The command is similar, but this time it is pipenv install. The output is a bit different, but the result is the same. The package is installed.

05:49 pipenv doesn’t bother telling you about what requests depends on. It just updates the files. Looking in the directory, you’ve now got both a Pipfile and a Pipfile.lock.

06:01 Let’s look inside of them. Here’s the state of the Pipfile after I did the installation. The only change here is to the packages section. It has an entry for the requests package.

06:14 The asterisk there indicates that any version of requests is allowed—i.e., it should grab the most recent. It did that because I didn’t ask for a specific version when I did the installation. After you’ve done the installation, you can always edit the Pipfile to change the behavior, so if you wanted to pin requests, you could do that by editing this file.

06:36 That being said, you don’t normally do that because the other file, the lock file, is for pinning.

06:44 Pipfile specifies what packages you install. Pipfile.lock contains a fully pinned listing of all the dependencies. If you want a production environment that exactly mimics your dev environment, you can do an installation with the lock file instead. On the other hand, as you’re developing, if all you care about is the libraries you installed, you can continue to use the Pipfile itself and then create a new lock when you’re ready to go. The format of the lock file is JSON.

07:14 It is made up of a series of nested dictionaries. The first part here specifies some metainformation and indicates what version of the Pipfile spec is being used and which Python version. The third section states where the libraries were sourced from, in this case the PyPI repository.

07:33 The next section of the lock file contains an entry for each of the libraries installed in the virtual environment. Each library is named, has a hash describing its contents, and fully pins a version. On the screen right now are the pinned versions of certifi, charset-normalizer, and idna.

07:52 Wow, lots of screenfuls of lock files. This is the last one. The default section continues on this screen here, pinning the requests and urllib3 libraries.

08:04 Finally, at the bottom, there’s a section for libraries only to be used in the dev environment, which at the moment is empty. Let’s play around with pipenv some more.

08:16 Before showing you the contents of Pipfile and Pipfile.lock, I was inside of the hola virtual environment. Back in the terminal now, I use the exit command to get out of that shell.

08:28 Let’s create another project.

08:38 Running pipenv shell inside of the new salve directory creates a new virtual environment and activates it. Just like pip, pipenv install supports the -e argument.

08:49 Remember the hello package? Let’s try it again.

08:58 So far, so good. And now the bonjour package.

09:07 Well, that didn’t go well. Unlike pip, pipenv has a very strong opinion about conflicts. It won’t let you do it by default. Let me scroll up just a bit here.

09:26 pipenv attempted to create the lock file, and the locking failed because of the conflict. And it explicitly tells you that. About halfway down here, you’ll see the keyword critical, and it tells you exactly what conflict happened: colorama-0.3.9 and 0.4.4 were specified by two different libraries. If I scroll back down, you’ll see it even gives you some suggestions as to what you can do about it.

09:54 Now, if you want to shoot yourself in the foot, you’re going to have to do it explicitly. pipenv puts the safety on the gun and gives you steel-toe boots.

10:03 You can still pull the trigger, but it shouldn’t happen by accident.

10:08 You’ve seen the basics, but wait, there’s more! It slices, it dices, it even juliennes! Next up, more pipenv goodness.