The Module Search Path

Python Modules and Packages: An Introduction Christopher Bailey 07:00

In this lesson, you’ll learn about the module search path. Continuing with the example from the previous lesson, take a look at what happens when Python executes the following statement:

Python
      
        
      
    
>>> import mod
>>> mod.a
[100, 200, 300]
>>> mod.s
'Computers are useless. They can only give you answers.'

When the interpreter executes the above import statement, it searches for mod.py in a list of directories assembled from the following sources:

The directory from which the input script was run, or the current directory if the interpreter is being run interactively
The list of directories contained in the PYTHONPATH environment variable, if it is set. (The format for PYTHONPATH is OS-dependent but should mimic the PATH environment variable.)
An installation-dependent list of directories configured at the time Python is installed

The resulting search path is accessible in the Python variable sys.path, which is obtained from a module named sys:

Python
      
>>> import sys
>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.7/bin', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python37.zip', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages']

Note: The exact contents of sys.path are installation-dependent. The above code block will almost certainly look slightly different on your computer. The operating system used in this lesson is macOS. If you would like to see what the path structure looks like in a Windows environment, check out the original article that this course is based on.

So, to ensure that your module is found, you need to do one of the following:

Put mod.py in the directory where the input script is located, or the current directory if interactive
Modify the PYTHONPATH environment variable to contain the directory where mod.py is located before starting the interpreter. Or put mod.py in one of the directories already contained in the PYTHONPATH variable.
Put mod.py in one of the installation-dependent directories, which you may or may not have write-access to, depending on the OS.

There is also one additional option: You can put the module file in any directory of your choice and then modify sys.path at run-time so that it contains that directory. For example, in this case, you could put mod.py in directory /Users/chris/ModulesAndPackages and then issue the following statements:

Python
      
        
      
    
>>> sys.path.append(r'/Users/chris/ModulesAndPackages')
>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.7/bin', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python37.zip', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages', '/Users/chris/ModulesAndPackages']
>>> import mod
>>> mod.s
'Computers are useless. They can only give you answers.'

Once you’ve imported a module, you can determine the location where it was found with the module’s __file__ attribute:

Python
      
        
      
    
>>> import mod
>>> mod.__file__
'/Users/chris/ModulesAndPackages/mod.py'
>>> import re
>>> re.__file__
'/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py'

The directory portion of __file__ should be one of the directories in sys.path.

00:00 In this video, you’re going to explore the module search path. So, where can you import a module from?

00:08 When the interpreter executes the import statement, it searches for that file. In your example here, it was mod.py. It’s going to search for that in a list of directories. The possibilities are the current directory, meaning where the input script was run from, and it also searches in the PYTHONPATH environment variable, which has a list of directories.

00:29 There are links below this video that give you a little more detail about the PYTHONPATH if you choose to dive in a little deeper. When you install Python, there’s a set of directories that are configured as part of that Python installation also. When you import from built-in modules, that’s where that code is coming from.

00:47 And there’s a way to see all of these, to see this list of directories. Let me show you that in code. Right now, you can see that in the terminal that I’m in, I’m currently in a directory named ModulesAndPackages.

01:02 And inside that directory, there’s the mod.py file. If I were to start a Python interpreter, it will start that interpreter from inside this current directory.

01:13 If you saved your mod.py file in that same directory as the current directory, you can type import and mod. The mod.py file is right there.

01:24 Great. So now mod.a is available, mod.s is available. Okay. If you wanted to see all the places that can be imported from currently, there is a tool called the sys module that’s part of the built-in library that you can import.

01:39 And that can show you the current PYTHONPATH and the directories that were set up when Python was installed. import sys, which is a module that includes system-specific parameters and functions that you can access. And one of those is sys.path.

01:54 So sys.path—that will show you a list. Hopefully you can see that here with the square brackets. And these are all the locations your interpreter’s going to search for when you type the word import and then search for a particular module that you’re importing.

02:10 This is the default directory that you’re in—the current directory—so it would search right there. Then from there, it goes into the version of Python that you installed.

02:19 This is what it looks like on my particular computer here, on a Mac, where it puts it in a Library/ file, but that may vary depending on what the installation was.

02:27 It might be in, say, an Anaconda installation or something different on a Windows installation. But you can see here the variety of places that it’s going to look for—those modules.

02:37 I’m going to exit out of here.

02:40 So, those are all locations that you could save the file—either in the current directory or in any of these other directories that you see after running sys.path.

02:50 What if you want to add locations for it to be able to import? That’s a possibility too. To prove this point, I’m going to make this not be the current directory.

03:00 I’m going to use the command change directory, cd .. , and that takes me to the root of this directory. And after you’ve done that, and you start your REPL from that directory, if you were to type import and then mod, it’s going to say that module is not found—it’s not currently in that directory or in any of the other locations that if we import sys and use sys.path, it’s not in any of these locations that import is searching within.

03:33 So, where should you put your module file? To make sure that your script—that you’re using the import command from—can find it, you have a few choices.

03:43 You could put it in that same directory the input script is written in, which is considered usually the current directory. You can modify the PYTHONPATH environment variable, or you could put it in one of the directories already in that PYTHONPATH.

03:56 You could also put it in one of the directories that are configured as part of your Python installation. Now, there is one other solution that can be done at run-time. In that case, you could actually modify the sys.path list. To do that you’re going to use an .append() statement, just like you would depend to any other list. In this case, I’m using an example of what it may look like in a Windows environment.

04:18 Let me have you check this out in code.

04:21 So, in this particular case, from sys.path you can use .append(). And as an argument, you’re going to enter in a raw string, so you start with r and the single quote (').

04:31 Now you need the directory path, which you could just simply type in. But I like to use this trick. I know it works on Windows and on a Mac. If you open up a file browser, or Explorer on Windows, you can drag the folder icon, or if you will, the directory, and drop that in to the terminal window.

04:51 It actually will then print out the entire path. Closing my quotation mark there. So, what did that do? Well, if you look at sys.path now, at the very end of it it has appended this particular location, this directory, into the path.

05:05 If you were to type import mod,

05:10 it now can import it in. And here are all those objects that are part of mod.py module. Great! One thing you need to remember though, if you were to append those locations or those directories for you to import them in, that that only lasts for that particular session.

05:26 So if you were to exit and start again and try to import mod,

05:33 it’s not available. Because if you import sys and use sys.path again,

05:40 you can see it’s no longer part of that list. So that would need to be part of your input script, then, that you would add the sys.path.append() and the directory.

05:49 So that leaves you a few choices: to either add it to one of these directories, or, when you’re importing, make sure that it’s available in the current directory.

06:00 Let me have you modify sys.path one more time. In this example, I’ll simply type it out, but you could do either way.

06:09 Now the interpreter can search that directory. And I’ll have you import mod.

06:15 Great. Another tool that you can use is a dunder method for your module, which is __file__, to see the location of that module’s file. So that file came from here.

06:53 In the next video, you’re going to take a deeper dive into that import statement.

Pygator on Feb. 15, 2020

Is it possible to make the path.append() call permanent on the python path? Very interesting!

Chris Bailey RP Team on Feb. 15, 2020

Hi @Pygator, I guess is depends on what you are looking for. It is possible to make an addition to the PYTHONPATH environment variable. I just did an experiment, where I added a directory on my desktop to the PYTHONPATH and was able to import from it. That would be done inside of the .bash_profile on a mac, and probably Linux. It would probably look a bit different on windows. Here are some of the details. stackoverflow.com/questions/3387695/add-to-python-path-mac-os-x

But I would probably stick to the items mentioned in the original article, first most placing the module into the site-packages directory. Which would be based on your installation. I use virtual environments, and each one should have a directory, venv/lib/<python_ver#>/site-packages. Which is good spot to use.

I’m not sure of a way to have it always call path.append() outside of writing it into particular scripts you want to run.

drewmullen on April 17, 2020

thanks for putting this series together, chris! you should consider updating this video to include some qualifiers on how pythonistas might (and might not) use the pythonpath. i can tell you from personal experience that i spent wayyy to long “installing” my modules by weird frankenstein scripts + sys.path appends.

its important for people to understand what the pythonpath is, so im glad you made this video! but i think its also important for people to know that you shouldnt spend much time attempting to manipulate the pythonpath, short of debugging / development. you can even whet their appetite and elude to setuptools / pyproject.toml, etc

Anurag Gupta on Nov. 26, 2021

I tried the .__file__ technique on the math module, but it shows an error (module ‘math’ has no attribute '__file__'). However, when I used it with the collections module, it worked as shown in the video. So, what is the difference here?

Bartosz Zaczyński RP Team on Nov. 26, 2021

@Anurag Gupta That’s surprising as it should just work:

>>> import math
>>> math.__file__
'/home/realpython/.pyenv/versions/3.10.0/lib/python3.10/lib-dynload/math.cpython-310-x86_64-linux-gnu.so'

>>> import collections
>>> collections.__file__
'/home/realpython/.pyenv/versions/3.10.0/lib/python3.10/collections/__init__.py'

Are you sure you’re running exactly the same code as above? What environment do you use?

Loris on May 16, 2023

I am a Windows 11 user and found an alternative to PYTHONPATH that did not work for me: https://python-forum.io/thread-32255.html.

The solution consists in:

identifying the user site-packages directory:

>>> import site
>>> site.getusersitepackages()
"C:\\Users\\user_name\\AppData\\Roaming\\Python\\Python39\\site-packages"

and, in that directory, create (or enhance) the file usercustomize.py that will contain the list of the relevant paths as in the example below:

# usercustomize.py
import sys

path_list = [
    "C:/Users/user_name/OneDrive/Coding/python/packages",
    "C:/Users/user_name/OneDrive/Coding/modules"
]

sys.path.extend(path_list)

Loris on May 16, 2023

Sorry 😫😫😫 about the formatting on my last post but the rendering on this site does not at all correspond to what is rendered in VS Code for the same text that I pasted above.

Hopefully readers will be able to figure out by referring to the URL mentioned in my post.

@Bartosz Zaczyński: I am wondering how you managed to get your code blocks so nicely displayed.

Bartosz Zaczyński RP Team on May 17, 2023

@Loris No worries, I fixed your formatting. It looks like our Markdown renderer sometimes doesn’t like indentation in front of a code block, so I just removed them.

Loris on May 17, 2023

@Bartosz Zaczyński: Brilliant, thanks a lot for your feedback and prompt action!

Become a Member to join the conversation.