Join us and get access to thousands of tutorials and a community of expert Pythonistas.
This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.
Capturing Dependencies
00:00 Now how do you create one of those requirements files? One way to do it would be to just create a text file, and type your dependencies into it. And that might be a pretty good way to do it.
00:12 On the other hand, it becomes very difficult to actually capture all of the dependencies, including the secondary or transitive dependencies for you application.
00:22
Luckily, pip can help you with that. So what I am going to do now is I am going to show you the pip freeze command you can use to create requirements files very easily.
00:32
I am in my terminal here and I’ve got a virtual environment activated. So to show you that this is a completely new virtual environment where I haven’t installed any third-party dependencies, I am going to run the pip list command.
00:46
You can see here that this just contains the internals for pip and there are no third-party packages here. pip freeze is the magic incantation for creating these requirements files.
00:59
Now watch what happens when I run pip freeze on a virtual environment without third-party packages. That is right, nothing happens. Because there is nothing for pip to freeze.
01:10
So now I am going to install a couple of third-party packages so we can actually start creating a requirements file. Alright, I just installed the latest version of Requests and now, when I run pip list again, you can see here that Requests shows up in this list.
01:30
Now we could just take that and type it out into a separate requirements.txt file but really this is what the pip freeze command is made for.
01:39
So when I run the pip freeze command now, I get a different result, it actually captured the Requests library as a third-party dependency, so the output of the pip freeze command is all we need to create our requirements.txt file.
01:53
We can actually just take that output and pipe it (>) into a requirements.txt file. And we don’t even need to copy paste it and go through a separate editor. Let me show you how that works.
02:07
So this is the command I would run here, and once that finished, it created a new requirements.txt file. So when I take a look at this file you can see that it contains exactly the output of the pip freeze command, and that is a very quick way to capture the dependencies that are installed in your virtual environment or in any Python environment for that matter.
02:28
Now before we move on, I want to show you how pip freeze, not only captures the top level dependencies, but it actually is smart enough to go and capture all of the secondary or so called transitive dependencies, to include them in its output which we could then put into requirements.txt file.
02:48
So Requests doesn’t actually have any third-party dependencies, so it doesn’t really make a great example here, so now I am going to install the Flask module, as another third-party dependency because Flask actually contains a bunch of secondary dependencies that I can then use to demo how pip freeze deals with them.
03:10
Okay, so I just installed Flask here and you can already see that it came with a bunch of third-party dependencies, and now these should actually show up if I run pip freeze again, as secondary or transitive dependencies.
03:27
Alright, so there we go, when I run pip freeze again, with Flask installed, it’s also listing the other dependencies or the secondary dependencies that Flask brought with it. Because really all I did here initially was run pip install flask and then Flask itself brought in all of these other secondary dependencies.
03:47
Capturing these secondary dependencies is super important to make an environment reproducible. As you can see here, pip freeze does exactly that, so if I regenerate my requirements file, you can see that this captures all of the dependencies, so with this requirements file, we’re in a pretty good shape to completely reproduce this exact environment on another machine including the exact version specifiers and including all of the dependencies and secondary dependencies that our application might need.
04:25
Here is a quick recap on capturing dependencies using the pip freeze command, basically you’ll run the pip freeze command and then take its output to create your requirements.txt file, and then you would include the requirements file with your project so that another developer can recreate the exact same environment that you were using at the time when you created the pip freeze file.
04:47 Please note that the requirements file only specifies third-party packages and their version numbers, so this is not going to specify exactly which version of the Python interpreter to use for example.
05:01
pip freeze is the most important ingredient for achieving repeatability, so with pip freeze you can capture all dependencies of your programs including secondary dependencies and their exact version numbers.
05:14 So it’s really important to capture the secondary dependencies also because if you only specify the first level or top level dependencies, what could happen is there might be a silent update for a secondary dependency and this could then cause trouble down the line when someone tries to install your program at a later point in time.
05:35 To avoid surprises, it’s a good idea to always provide a requirements file that includes all of the secondary dependencies and their exact version numbers with your program.
Dan Bader RP Team on March 22, 2020
Hi Raj, that’s right, using requirements.txt alone you can’t pin a specific version of the Python interpreter. For working with different versions of the Python interpreter, I can recommend the following options:
The links above will point you to the related Real Python tutorials. Personally I like to use pyenv for setting up project specific versions of the Python interpreter because it has the best performance for local development.
Docker is a great option too and if used across the board you can guarantee that your local development environment, your testing/CI environment, and production all use the exact same version of Python and the underlying operating system. So depending on your use case, that’s something to look into as well.
Hope this helped you out :)
Raj Dutt on March 23, 2020
Dan - Thanks for your quick response! This certainly helps me. I am also getting ready to start using VSCODE for my Python development. I will also appreciate your suggestions on working with different versions of the Python interpreter. Thanks again ~Raj
DoubleA on Jan. 17, 2021
Hi Dan,
Touching upon the question re the Python interpreter version used in a specific venv wouldn’t it be an option to use:
python --version > requirements.txt
pip list >> requirements.txt
to capture all the third party dependencies PLUS the exact version of the Python interpreter in a single requirements.txt file?
PS: for those working on Windows machines, to see the content of a text file, instead of the cat command in terminal pass type which does exact the same thing: type requirements.txt
Dan Bader RP Team on Jan. 17, 2021
@DoubleA: Yep, that would write out a line of text like Python 3.8.2 to requirements.txt.
But the requirements files are meant to be parseable by pip install and adding a line that Pip doesn’t understand would break the “restoring captured dependencies” workflow explained in the next lesson.
Pip cannot be used to manage Python interpreter versions. It only serves as a tool for installing/removing Python packages inside of an already configured Python environment.
For managing interpreter versions you’d use a separate tool, like pyenv.
Marcus Åberg on May 7, 2022
Hi,
A tricky question :)
What happens if 2 third party dependencies have the same secondary dependency, but with different versions?
Will pip handle this and what happens in that case to the requirements file? Or do you get an error?
Bartosz Zaczyński RP Team on May 9, 2022
@Marcus Åberg It’s a tricky question indeed! I can’t recall running into such a problem in Python myself, but my assumption would be that a dependency manager tool, such as poetry, would take care of resolving such version conflicts for you as long as it was possible. In particular, the required versions must be defined loosely using a range of version numbers instead of a specific version. Poetry would then generate a lock file to pin those transitive dependency versions for future installs.
If you’re using a recent version of pip, then you might try specifying a constraints file, which is kind of like a requirements.txt file, but it restricts the problem space for the version resolver implemented in pip.
I wish there was a more elegant way of handling dependencies in Python. For example, in the Java world, Maven’s <exclude> directive does exactly what you were asking for in a much more explicit way.
Become a Member to join the conversation.
Raj Dutt on March 22, 2020
Dan, I noted that requirements file is not going to specify exactly which version of the Python interpreter to use. This is another major issue that we are running into. The solutions that we work on are often based on earlier versions of Python (v.2.7). What are the best ways to develop solutions which integrate different versions of Python interpreters? I would appreciate any details that you could share.
~Raj