Python Zip Imports: Distribute Modules and Packages Quickly

Bundle Python Modules Into ZIP Files

In this section, you’ll use PyZipFile.writepy() to compile a .py file down to bytecode and add the resulting .pyc file to a ZIP archive. To try .writepy() out, say that you have a hello.py module:

"""Print a greeting message."""
# hello.py

def greet(name="World"):
    print(f"Hello, {name}! Welcome to Real Python!")

This module defines a function called greet() that takes name as an argument and prints a friendly greeting message to the screen.

Now say that you want to package this module into a ZIP file that you can import later. To do this, you can run the following code:

>>> import zipfile

>>> with zipfile.PyZipFile("hello.zip", mode="w") as zip_module:
...     zip_module.writepy("hello.py")
...

>>> with zipfile.PyZipFile("hello.zip", mode="r") as zip_module:
...     zip_module.printdir()
...
File Name                                             Modified             Size
hello.pyc                                      2021-10-18 05:40:04          313

After running this code, you’ll have a hello.zip file in your current working directory. The call to .writepy() on zip_module automatically compiles hello.py to hello.pyc and stores it in the underlying ZIP file, hello.zip. That’s why .printdir() displays hello.pyc instead of your original hello.py file. This automatic compilation ensures an efficient import process.

You can also manually package .py and .pyc files into a ZIP file by using any regular file archiver. If the resulting archive contains .py files without the corresponding .pyc files, then Python will compile them the first time you import from that specific ZIP file.

Python won’t modify the underlying ZIP file to add the newly compiled .pyc files. So the next time you run the import, Python will compile the code again. This behavior will make the import process slower.

You can also pass a directory as the first argument to .writepy(). If the input directory isn’t a Python package, then the method scans it for .py files, compiles them to .pyc files, and adds those .pyc files at the top level of the target ZIP file. The scanning step isn’t recursive, which means that subdirectories aren’t scanned for source files.

You can tweak the compilation process further by setting the optimize argument of PyZipFile to one of the following values:

With these values, you can fine-tune the level of optimization you want to use when .writepy() compiles your .py files to .pyc files before archiving them.

So far, you’ve learned how to bundle one or more modules into a ZIP file. In your day-to-day coding, you might also need to zip a complete Python package. You’ll learn how to do that in the following section.

Bundle Python Packages Into ZIP Files

You can also bundle Python packages into ZIP files by using PyZipFile and its .writepy() method. As you already learned, if you pass a regular directory as the first argument to .writepy(), then the method scans the directory for .py files, compiles them, and adds the corresponding .pyc files to the resulting ZIP file.

On the other hand, if the input directory is a Python package, then .writepy() compiles all the .py files and adds them to the ZIP file, keeping the package’s internal structure.

To try .writepy() with a Python package, create a new hello/ directory and copy your hello.py file into it. Then add an empty __init__.py module to turn the directory into a package. You should end up with the following structure:

hello/
|
├── __init__.py
└── hello.py

Now suppose that you want to bundle this package into a ZIP file for distribution purposes. If that’s the case, then you can run the following code:

>>> import zipfile

>>> with zipfile.PyZipFile("hello_pkg.zip", mode="w") as zip_pkg:
...     zip_pkg.writepy("hello")
...

>>> with zipfile.PyZipFile("hello_pkg.zip", mode="r") as zip_pkg:
...     zip_pkg.printdir()
...
File Name                                             Modified             Size
hello/__init__.pyc                             2021-10-18 05:56:00          110
hello/hello.pyc                                2021-10-18 05:56:00          319

The call to .writepy() takes the hello package as an argument, searches for .py files inside it, compiles them to .pyc files, and finally adds them to the target ZIP file, keeping the same package structure.

Understand the Limitations of Zip Imports

When you use ZIP files to distribute Python code, you need to consider a few limitations of Zip imports:

• Loading dynamic files, such as .pyd, .dll, and .so, isn’t possible.
• Importing code from .py files implies a performance compromise.
• Importing code from compressed files will fail if the decompression library isn’t available.

You can include any type of file in your ZIP archives. However, when your users import code from these archives, only .py, .pyw, .pyc, and .pyo files are read. Importing code from dynamic files, such as .pyd, .dll, and .so, isn’t possible if they live in a ZIP file. For example, you can’t load shared libraries and extension modules written in C from ZIP archives.

You can work around this limitation by extracting dynamic modules from your ZIP files, writing them to the file system, and then loading their code. However, that means you need to create temporary files and deal with possible errors and security risks, which can complicate things.

Zip imports can also imply a performance compromise, as you learned earlier in this tutorial. If your archive contains .py modules, then Python will compile them to satisfy the imports. However, it won’t save the corresponding .pyc files. This behavior may reduce the performance of import operations.

Finally, if you need to import code from a compressed ZIP file, then zlib must be available in your working environment for decompression purposes. Importing code from compressed archives fails with a missing zlib message if this library isn’t available. Additionally, the decompression step adds extra performance overhead to the import process. For these reasons, you’ll use uncompressed ZIP files in this tutorial.

Use sys.path Dynamically for Zip Imports

Because sys.path is a list object, you can manipulate it from your Python code by using regular list methods. In general, to add new items to a list object, you can use .insert(), .append(), or .extend().

Typically, you’ll use .insert(0, item) to add new items to sys.path from your Python code. Calling .insert() this way inserts item at the beginning of the list, ensuring that your newly added item has precedence over the existing ones. Having item at the beginning enables you to shadow existing modules and packages when name collisions are possible.

Now say that you need to add the hello.zip file containing your hello.py module to your current Python’s sys.path. In this case, you can run the code in the example below. Note that for this example to work on your machine, you need to provide the correct path to hello.zip:

>>> import sys

>>> # Insert the hello.zip into sys.path
>>> sys.path.insert(0, "/path/to/hello.zip")

>>> sys.path[0]
'/path/to/hello.zip'

>>> # Import and use the code
>>> import hello

>>> hello.greet("Pythonista")
Hello, Pythonista! Welcome to Real Python!

Once you’ve added the path to hello.zip to your sys.path, then you can import objects from hello.py as you would with any regular module.

If, like hello_pkg.zip, your ZIP file contains a Python package, then you can add it to sys.path too. In this case, the imports should be package-relative:

>>> import sys

>>> sys.path.insert(0, "/path/to/hello_pkg.zip")

>>> from hello import hello

>>> hello.greet("Pythonista")
Hello, Pythonista! Welcome to Real Python!

Because your code is in a package now, you need to import the hello module from the hello package. Then you can access the greet() function as usual.

Another option for adding items to sys.path is to use .append(). This method takes a single object as an argument and adds it to the end of the underlying list. Restart your Python interactive session and run the code that provides the path to hello.zip:

>>> import sys

>>> sys.path.append("/path/to/hello.zip")

>>> # The hello.zip file is at the end of sys.path
>>> sys.path[-1]
'/path/to/hello.zip'

>>> from hello import greet
>>> greet("Pythonista")
Hello, Pythonista! Welcome to Real Python!

This technique works similarly to using .insert(). However, the path to your ZIP file is now at the end of sys.path. If any preceding item in the list contains a module called hello.py, then Python will import from that module instead of from your newly added hello.py module.

You can also use .append() in a loop to add several files to sys.path, or you can just use .extend(). This method takes an iterable of items and adds its content to the end of the underlying list. As with .append(), keep in mind that .extend() will add your files to the end of sys.path, so existing names can shadow modules and packages in your ZIP files.

Use PYTHONPATH for System-Wide Zip Imports

In some situations, you may need a given ZIP file to be available for importing its content from any script or program that you run on your computer. In these situations, you can use the PYTHONPATH environment variable to make Python automatically load your archive into sys.path whenever you run the interpreter.

PYTHONPATH uses the same format as the PATH environment variable, a list of directory paths separated by os.pathsep. On Unix systems, such as Linux and macOS, this function returns a colon (:), while on Windows, it returns a semicolon (;).

For example, if you’re on Linux or macOS, then you can add your hello.zip file to PYTHONPATH by running the following command:

$ export PYTHONPATH="$PYTHONPATH:/path/to/hello.zip"

This command adds /path/to/hello.zip to your current PYTHONPATH and exports it so that it’s available in the current terminal session.

# .bashrc

if [ -f /path/to/hello.zip ]; then
    export PYTHONPATH="$PYTHONPATH:/path/to/hello.zip"
fi

Now you can issue the python command to run the interpreter. Once you’re there, check the content of sys.path as usual:

>>> import sys

>>> sys.path
[..., '/path/to/hello.zip', ...]

Cool! Your hello.zip file is in the list. From this point on, you’ll be able to import objects from hello.py as you did in the above section. Go ahead and give it a try!

An important point to note in the above output is that your hello.zip file isn’t at the beginning of sys.path, which implies that a same-named module that appears earlier will take precedence over your hello module, according to how Python handles its module seach path.

To add an item to PYTHONPATH on a Windows system, you can execute a command in your cmd.exe window:

C:\> set PYTHONPATH=%PYTHONPATH%;C:\path\to\hello.zip

This command adds C:\path\to\hello.zip to the current content of the PYTHONPATH variable on your Windows machine. To check it out, run the Python interpreter in the same command prompt session and look at the content of sys.path, as you did before.

Adding directories and ZIP files to the PYTHONPATH environment variable makes those entries available for whatever Python interpreter you run under the terminal session at hand. Finally, it’s important to note that Python will silently ignore nonexistent directories and ZIP files listed in PYTHONPATH, so keep an eye on that.

Use a .pth File for Interpreter-Wide Zip Imports

Sometimes you may want to import code from a given ZIP file only when you’re running a specific Python interpreter. This is useful when you have a project that uses code from that ZIP file, and you don’t want the code to be available for the rest of your projects.

Python’s path configuration files allow you to extend the sys.path of a given interpreter with your custom directories and ZIP files.

A path configuration file uses the .pth file extension and can hold a list of paths to directories and ZIP files, one per line. This list of paths is added to sys.path every time you run the Python interpreter that provides the .pth file.

Python’s .pth files have a straightforward format:

• Each line must contain a single path entry.
• Blank lines and lines beginning with a number sign (#) are skipped.
• Lines starting with import are executed.

Once you have a suitable .pth file, you need to copy it to one of the site directories so that Python can find it and load its content. To get the site directories of your current Python environment, you can call getusersitepackages() from the site module. If you don’t have admin privileges on your current machine, then you can use the user site directory at site.USER_SITE.

For example, the following command creates a hello.pth path configuration file for the system-wide Python 3 interpreter on Ubuntu:

$ sudo nano /usr/lib/python3/dist-packages/hello.pth

This command creates hello.pth, using the GNU nano text editor as root. Once there, type in the path to your hello.zip file. Save the file by pressing Ctrl+X, then Y, and finally Enter. Now this ZIP file will be available in sys.path when you launch the system Python interpreter again:

>>> import sys

>>> sys.path
[..., '/path/to/hello.zip', ...]

That’s it! From this point on, you can import objects from hello.py as long as you use the system-wide Python interpreter.

Again, nonexistent directories and ZIP files won’t be added to sys.path when Python reads and loads the content of a given .pth file. Finally, repeated entries in a .pth file are added only once to sys.path.

Understand the Basics of zipimport

The main component of zipimport is zipimporter. This class takes the path to a ZIP file as an argument and creates an importer instance. Here’s an example of how to use zipimporter and some of its attributes and methods:

>>> from zipimport import zipimporter

>>> importer = zipimporter("/path/to/hello.zip")

>>> importer.is_package("hello")
False

>>> importer.get_filename("hello")
'/path/to/hello.zip/hello.pyc'

>>> hello = importer.load_module("hello")
>>> hello.__file__
'/path/to/hello.zip/hello.pyc'

>>> hello.greet("Pythonista")
Hello, Pythonista! Welcome to Real Python!

In this example, you first import zipimporter from zipimport. Then you create a zipimporter instance with the path to your hello.zip file.

The zipimporter class provides several useful attributes and methods. For example, .is_package() returns True if the input name is a package and False otherwise. The .get_filename() method returns the path (.__file__) to a given module inside the archive.

If you want to bring the module’s name into your current namespace, then you can use .load_module(), which returns a reference to the input module. With that reference, you can access any code object from the module as usual.

Build a Plugin System With zipimport

As you learned above, Python internally uses zipimport to load code from ZIP files. You also learned that this module provides tools that you can use in some real-life coding situations. For example, say that you want to implement a custom plugin system in which each plugin lives in its own ZIP file. Your code should search for ZIP files in a given folder and automatically import the plugin’s functionality.

To experience this example in action, you’ll implement two toy plugins that take a message and a title and show them in both your default web browser and a Tkinter message box. Each plugin should live in its own directory, in a module called plugin.py. This module should implement the plugin’s functionality and provide a main() function as the plugin’s entry point.

Go ahead and create a folder called web_message/ with a plugin.py file in it. Open the file in your favorite code editor or IDE and type in the following code for the web browser plugin:

"""A plugin that displays a message using webbrowser."""
# web_message/plugin.py

import tempfile
import webbrowser

def main(text, title="Alert"):
    with tempfile.NamedTemporaryFile(
        mode="w", suffix=".html", delete=False
    ) as home:
        html = f"""
            <html>
                <head>
                    <title>{title}</title>
                </head>
                <body>
                    <h1>
                        {text}
                    </h1>
                </body>
            </html>
        """
        home.write(html)
        path = "file://" + home.name
    webbrowser.open(path)

The main() function in this code takes a text message and a window title. Then it creates a NamedTemporaryFile in a with statement. The file will contain a minimal HTML document showing title and text on the page. To open this file in your default web browser, you use webbrowser.open().

The next plugin provides similar functionality but uses the Tkinter toolkit. The code for this plugin should also live in a module called plugin.py. You can place the module in a directory called tk_message/ in your file system:

"""A plugin that displays a message using Tkinter."""
# tk_message/plugin.py

import tkinter
from tkinter import messagebox

def main(text, title="Alert"):
    root = tkinter.Tk()
    root.withdraw()
    messagebox.showinfo(title, text)

Following the same pattern as the web browser plugin, main() takes text and title. In this case, the function creates a Tk instance to hold the plugin’s top-level window. However, you don’t need to show that window, only a message box. So, you use .withdraw() to hide the root windows and then call .showinfo() on messagebox to show a dialog with the input text and title.

Now you need to pack each plugin into its own ZIP file. To do so, start a Python interactive session in the directory containing the web_message/ and tk_message/ folders and run the following code:

>>> import zipfile

>>> plugins = ("web_message", "tk_message")

>>> for plugin in plugins:
...     with zipfile.PyZipFile(f"{plugin}.zip", mode="w") as zip_plugin:
...         zip_plugin.writepy(plugin)
...

The next step is to create a root folder for your plugin system. This folder must contain a plugins/ directory with the newly created ZIP files in it. Here’s how your directory should look:

rp_plugins/
│
├── plugins/
│   │
│   ├── tk_message.zip
│   └── web_message.zip
│
└── main.py

In main.py, you’ll place the client code for your plugin system. Go ahead and populate main.py with the following code:

 1# main.py
 2
 3import zipimport
 4from pathlib import Path
 5
 6def load_plugins(path):
 7    plugins = []
 8    for zip_plugin in path.glob("*.zip"):
 9        importer = zipimport.zipimporter(zip_plugin)
10        plugin_module = importer.load_module("plugin")
11        plugins.append(getattr(plugin_module, "main"))
12    return plugins
13
14if __name__ == "__main__":
15    path = Path("plugins/")
16    plugins = load_plugins(path)
17    for plugin in plugins:
18        plugin("Hello, World!", "Greeting!")

Here’s how this code works line by line:

• Line 3 imports zipimport to dynamically load your plugins from the corresponding ZIP files.
• Line 4 imports pathlib to manage system paths.
• Line 6 defines load_plugins(), which takes the path to the directory containing your plugin archives.
• Line 7 creates an empty list to hold the current plugins.
• Line 8 defines a for loop that iterates over the .zip files in the plugins directory.
• Line 9 creates a zipimporter instance for every plugin in the system.
• Line 10 loads the plugin module from each plugin ZIP file.
• Line 11 appends each plugin’s main() function to the plugins list.
• Line 12 returns the plugins list to the caller.

Lines 14 to 18 call load_plugins() to generate the current list of available plugins and execute them in a loop.

If you run the main.py script from your command line, then you first get a Tkinter message box displaying the Hello, World! message and the Greeting! title. After closing that window, your web browser will display the same message and title on a new page. Go ahead and give it a try!