Why Can't Python Find My Modules?
It’s not uncommon for new Pythonistas to have trouble installing packages and using their modules. Frustrating errors like this often arise, even if you think you’ve installed a package properly:
ImportError: No module named <package_name>
This is caused by the fact that the version of Python you’re running your script with is not configured to search for modules where you’ve installed them. This happens when you use the wrong installation of pip to install packages.
In general, each Python installation comes bundled with its own pip executable, used for installing packages. By default, that pip executable will install packages in a location where that specific Python installation can find them.
The problem is that it’s very common to have multiple Python interpreters installed (and by extension, multiple pip executables.) Depending on your shell’s PATH, running pip may invoke the pip executable linked to the version of Python you’re using, or to a different one. If the wrong pip is invoked, then the packages it installs will likely not be visible to the Python interpreter you’re using, causing the ImportError.
To use the version of pip specific to your desired Python version, you can use python -m pip. Here, python is the path to the desired Python interpreter, so something like /usr/local/bin/python3.7 -m pip will use the pip executable for /usr/local/bin/python3.7. However, this still has its limitations.
There are also other ways to get around this issue. You can modify your shell’s PATH so it uses the correct pip executable, or change the PYTHONPATH so that your desired version of Python can find the packages located in a different directory. But these can all get messy fast.
Instead, virtual environments are often used to isolate Python installations from one another. A virtual environment contains, among other things, a Python interpreter, a pip executable, and a site-packages directory, which is the standard location for most packages downloaded with pip.
By activating a virtual environment within your shell, you expose it to only the pip and Python executables installed within your virtual environments, ensuring that the right versions of both applications are invoked and that packages are always installed to the correct location. Virtual environments also allow you to run different versions of the same package with different projects, something not possible if you are using a global Python installation.
There are many different virtual environments to choose from. This course uses Conda, bundled with Anaconda. You can learn more about virtual environments in Working With Python Virtual Environments.
00:00
A common error that new Pythonistas will come across is that the packages they think they’ve installed are not actually being recognized by Python. This will present itself as an ImportError, meaning that the module you’ve tried to import cannot be located. To learn why this is, we have to take a little tour around our operating system.
00:25
When you run a Python program, what you’re really doing is running the Python interpreter and passing it your Python script to interpret and run. The Python interpreter is simply an executable file located somewhere within your system, meaning that it’s literally a program whose job it is to run your Python program. It’s kind of trippy if you think about it. On Windows, this is called python.exe, and on Mac and other Unix environments, it’s just an executable called python.
01:01 There’s a good chance that you’ve got multiple Python interpreters installed. In fact, macOS and most Linux distributions come bundled with an older version of Python, Python 2, which is used for internal system functionality.
01:17 That’s why—if you’ve ever set up a brand new macOS installation and tried to run Python in the terminal—you’ll see a prompt for Python 2. Aside from this built-in Python, you can install other Python versions, like Python 3—or more specifically, 3.7 or 3.8—and they will all live in different locations on your system.
01:42
This is important because new Python versions are not entirely backwards compatible with old versions. Just look at Python 2 versus 3 and how they differ regarding printing to stdout (standard out).
01:56 If you code a program with Python 3 and try to run it with 2, all of your print statements—among other things—will break. The same will happen if you code for Python 2 and try to run it in 3. Apple can’t replace Python 2 with 3 in macOS because the inner workings of macOS still rely on functionality that’s present in Python 2, and that functionality might not exist—or it’ll just be different—in Python 3.
02:28 It’s not worth their time to re-engineer the OS to use Python 3, so they just leave it alone.
02:37
Having the ability to install different Python interpreters on one system is great for compatibility with older software, but it can complicate things when trying to develop new software. That’s because almost every Python interpreter comes bundled with its own version of pip, the package manager you’ll learn about in the rest of this course.
03:00
In that pip application, we’ll install its packages to a specific folder on disk often called site-packages, which sometimes corresponds to a single pip executable.
03:14
This means that invoking the wrong pip executable within your terminal will result in packages being installed in the wrong location. If I accidentally run the pip executable that came with my Python 3.7 installation, the packages it installed may not be searched for by Python 3 because it may install to a different site-packages/ folder.
03:40
So then, how do we fix this? There are many ways around this—for example, by modifying your shell’s path so that it uses the correct pip executable, or changing the Python path so that your desired version of Python can find the packages located somewhere else.
03:58 But these can all get messy pretty quickly, so instead, I almost always recommend using what’s called a virtual environment.
04:09
A virtual environment is a folder containing—among many things—a Python interpreter for a specific version of Python, a pip executable for installing and managing packages, and a site-packages/ folder that corresponds to that exact Python interpreter and pip executable.
04:30 Everything you need to develop your application is self-contained within this folder, isolated from any other Python installations you may have on your system.
04:41
In fact, virtual environments often allow you to activate them from within the shell, temporarily blinding the shell from any other versions of pip and Python installed on your system.
04:55 And, you can have many different virtual environments using different Python versions and different sets of packages. A common practice is to create a different virtual environment for each Python project you work on. This is especially helpful when two different projects require different versions of the same package.
05:17 Having a separate virtual environment for each project will allow each one to use a different version of the package.
05:26
There are many tools you can use to create and manage virtual environments. My personal favorite is called Conda, which is often used in data science and scientific computing. Conda comes as a part of Anaconda, which gives you access to not only Python packages that can be installed with pip, but also its own package manager and package repository.
05:52
This is a course on pip, so I’m not going to go any further in-depth on virtual environments. If you’re interested in learning more, we’ve got a great article on realpython.com that will get you up and running with virtual environments.
06:07 I’ll have a link for that down in the video notes below. You don’t have to use a virtual environment to follow along, but I recommend doing so. It’s also important to note that while Anaconda does come with its own package manager, I won’t be using it in this course.
06:26
I’ll just use Anaconda to create a virtual environment with Python and pip.
06:32
Then, I’ll use pip as the package manager for our new virtual environment.
Become a Member to join the conversation.
Gerold Csendes on Jan. 20, 2021
I guess it should be pointed out that using
pipin acondaenvironment is not necessarily a good practice: anaconda docs