Dependency Management With Python Poetry

Learn the Relevant Terminology

If you’ve ever used an import statement in one of your Python scripts, then you’ve worked with modules and packages. Some of them might have been Python files you wrote on your own. Others could’ve been standard library modules that ship with Python, like datetime. However, sometimes, what Python provides isn’t enough. That’s when you might turn to external modules and packages maintained by third parties.

When your Python code relies on such external modules and packages, they become the requirements or dependencies of your project.

To find packages contributed by the Python community that aren’t part of the Python standard library, you can browse PyPI. Once you’ve found a package you’re interested in, you can use Poetry to manage and install that package in your project. Before seeing how this works, you need to install Poetry on your system.

Install Poetry on Your Computer

Poetry is distributed as a Python package itself, which means that you can install it into a virtual environment using pip, just like any other external package:

• Windows
• Linux + macOS

(venv) PS> python -m pip install poetry

(venv) $ python3 -m pip install poetry

This is fine if you just want to quickly try it out. However, the official documentation strongly advises against installing Poetry into your project’s virtual environment, which the tool must manage. Because Poetry depends on several external packages itself, you’d run the risk of a dependency conflict between one of your project’s dependencies and those required by Poetry. In turn, this could cause Poetry or your code to malfunction.

In practice, you always want to keep Poetry separate from any virtual environment that you create for your Python projects. You also want to install Poetry system-wide to access it as a stand-alone application regardless of the specific virtual environment or Python version that you’re currently working in.

There are several ways to get Poetry running on your computer, including:

1. A tool called pipx
2. The official installer
3. Manual installation
4. Pre-built system packages

In most cases, the recommended way to install Poetry is with the help of pipx, which takes care of creating and maintaining isolated virtual environments for command-line Python applications. After installing pipx, you can install Poetry by issuing the following command in your terminal window:

• Windows
• Linux + macOS

PS> pipx install poetry

$ pipx install poetry

While this command looks very similar to the one you saw previously, it’ll install Poetry into a dedicated virtual environment that won’t be shared with other Python packages.

Crucially, pipx will also set an alias to the poetry executable in your shell so that you can invoke Poetry from any directory without manually activating the associated virtual environment:

• Windows
• Linux + macOS

PS> poetry --version
Poetry (version 1.7.1)

$ poetry --version
Poetry (version 1.7.1)

When you type poetry in your terminal or PowerShell console, you’ll always refer to the executable script installed in its isolated virtual environment. In fact, you’ll often run the poetry command from within an active virtual environment of your project, and Poetry will correctly pick it up!

If, for some reason, you can’t or don’t want to use pipx, then the next best thing you can do is take advantage of the official installer, which you can download and execute with a command like this:

• Windows
• Linux + macOS

PS> (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | python -

$ curl -sSL https://install.python-poetry.org | python3 -

$ open "/Applications/Python 3.12/Install Certificates.command"

Underneath the install.python-poetry.org URL is a cross-platform Python script that, more or less, replicates what pipx does, but in a slightly different way.

If you’re interested in the technical details, then take a peek at the underlying source code of the installation script. By following its steps manually, you’ll have a chance to tweak the installation process if you ever need something specific for your setup. This can be particularly useful when you prepare a continuous integration environment.

Finally, some operating systems may bundle Poetry as a native package. For example, if you’re using a Debian-based system like Ubuntu, then you might be able to install Poetry with apt, like so:

$ sudo apt install python3-poetry

Bear in mind that the Poetry version packaged like that might not be the latest one. Moreover, the native package may bring hundreds of megabytes of extra dependencies, such as another Python interpreter, which would be completely unnecessary if you installed Poetry using one of the earlier methods.

Once the Poetry installation is complete, you can confirm that it’s working correctly by typing poetry --version at your command prompt. This should print out your current version of Poetry.

Now that you’ve verified that Poetry was installed correctly, it’s time to see how it works.

Create a New Poetry Project

To create a new Poetry project from scratch, use the poetry new command followed by the desired project name. In this tutorial, you’ll work on a project named rp-poetry. Go ahead, create that project now, and then navigate into the newly created directory by entering the following commands in your terminal:

$ poetry new rp-poetry
$ cd rp-poetry/

Running poetry new rp-poetry creates a new folder named rp-poetry/. When you look inside that folder, you’ll see the following structure:

rp-poetry/
│
├── rp_poetry/
│   └── __init__.py
│
├── tests/
│   └── __init__.py
│
├── README.md
└── pyproject.toml

Did you spot how Poetry translated the dash (-) in the provided name into an underscore (_) in the corresponding rp_poetry/ subfolder? The tool automatically normalizes Python package names for you. This ensures you’ll be able to import them as Python packages. Otherwise, rp-poetry wouldn’t be a valid identifier because a dash represents the minus operator in the Python syntax.

To have more control over the resulting package name, you can optionally pass the --name argument, which lets you specify a different name for the Python package than your project folder:

$ poetry new rp-poetry --name realpoetry

Note that Poetry will also normalize the package name provided through this argument should it contain a dash.

When creating the folder structure for a new project, Poetry follows the flat layout by default. In this layout, your Python package resides at the root level of the project folder. Another popular choice in Python is the src layout, which keeps the code in an additional src/ parent folder. If you prefer this layout instead, then pass the --src flag to Poetry when creating a new project:

$ poetry new rp-poetry --src

Adding this flag will result in a slightly different folder structure:

rp-poetry/
│
├── src/
│   │
│   └── rp_poetry/
│       └── __init__.py
│
├── tests/
│   └── __init__.py
│
├── README.md
└── pyproject.toml

Now, your rp_poetry package is nested in the src/ folder. Both the flat and src layouts have their pros and cons, which you can compare by reading the Python Packaging Guide if you’re interested. However, you’ll stick to the default flat layout for the rest of this tutorial.

As you can see, creating a new project with Poetry is quite straightforward and convenient. You get a basic folder structure for free without having to think too much about how to organize your Python files.

Inspect the Project Structure

The rp_poetry/ subfolder itself isn’t very spectacular yet. Inside, you’ll only find an empty __init__.py file, which turns it into an importable Python package.

Similarly, the tests/ subfolder is yet another Python package meant to contain the unit tests and possibly other kinds of tests for your project, promoting best programming practices.

Poetry also generates an empty README file with an .md extension by default, suggesting the use of Markdown for formatting. If you’re not fond of Markdown, then your other options are either plain text or the reStructuredText format. These are the three formats that PyPI can render at the moment.

Lastly, Poetry creates a configuration file named pyproject.toml with the minimum required metadata for your project, as described in PEP 518. You’ll take a closer look at it now.

Understand the pyproject.toml File

One of the most important files for working with Poetry is the pyproject.toml file. This file isn’t an invention of Poetry but a configuration file standard defined in PEP 518:

This PEP specifies how Python software packages should specify what build dependencies they have in order to execute their chosen build system. As part of this specification, a new configuration file is introduced for software packages to use to specify their build dependencies (with the expectation that the same configuration file will be used for future configuration details). (Source)

The authors considered a few file formats for the new configuration file mentioned in the quote above. Ultimately, they’ve decided on the TOML format, which stands for Tom’s Obvious Minimal Language. In their opinion, TOML is flexible enough while having better readability and less complexity than other options like YAML, JSON, CFG, or INI.

To see what a TOML document looks like, open the pyproject.toml file that Poetry created in your project’s folder:

 1[tool.poetry]
 2name = "rp-poetry"
 3version = "0.1.0"
 4description = ""
 5authors = ["Philipp Acsany <philipp@realpython.com>"]
 6readme = "README.md"
 7
 8[tool.poetry.dependencies]
 9python = "^3.12"
10
11[build-system]
12requires = ["poetry-core"]
13build-backend = "poetry.core.masonry.api"

Currently, you can see three sections denoted with square brackets in the pyproject.toml file above. These sections are known as tables in the TOML terminology. They contain declarative instructions, which tools like Poetry can recognize and use for managing dependencies, building the project, or performing other tasks.

While PEP 518 defines a specific table name, [build-system], which contains the obligatory build requirements for your project, the standard leaves the door open for custom extensions. Third-party tools can group their configuration options under unique namespaces prefixed with the word tool.

For example, Poetry generates the [tool.poetry] and [tool.poetry.dependencies] tables. The first one is where you define your project’s metadata, such as the name and version, while the second table lets you specify external libraries managed by Poetry for your project.

"tool": {
    "poetry": {
        "name": "rp-poetry",
        "version": "0.1.0",
        ⋮
        "dependencies": {
            "python": "^3.12"
        }
    }
}

In this case, the two subtables belong to only one external tool—Poetry. But you’ll often find examples like [tool.black], [tool.isort], [tool.mypy], or [tool.pytest.ini_options] for their corresponding tools. While many Python tools are moving their configuration to pyproject.toml files, there are still notable exceptions. For example, flake8 didn’t support them at the time of writing, so you might need a few separate configuration files.

The pyproject.toml file starts with the [tool.poetry] subtable, where you can store general information about your project. Poetry defines a few table keys that are valid in this subtable. While some of them are optional, there are four that you must always specify:

1. name: The name of your distribution package that will appear on PyPI
2. version: The version of your package, ideally following semantic versioning
3. description: A short description of your package
4. authors: A list of authors, in the format name <email>

The subtable [tool.poetry.dependencies] on line 8 is essential for your dependency management. You’ll learn more about it and its variants in the next section when you add external dependencies. As of now, it only states the intended Python version for your project.

The last table, [build-system] on line 11 in the pyproject.toml file, defines metadata that Poetry and other build tools can work with. As mentioned earlier, this table isn’t tool-specific, as it doesn’t have a prefix. It has two keys:

1. requires: A list of dependencies required to build your package, making this key mandatory
2. build-backend: The Python object used to perform the build process

If you want to learn about this section of the pyproject.toml file, then you can find out more by reading about source trees in PEP 517.

When you create a new project with Poetry, this is the pyproject.toml file that you start with. Over time, you’ll add more configuration details about your package and the tools you use. As your project grows, your pyproject.toml file will grow with it. That’s particularly true for the [tool.poetry.dependencies] subtable.

In the next section, you’ll find out how to expand your project by adding third-party dependencies using Poetry.

Activate a Custom Virtual Environment

When you create a new Python project by hand, it’s a good practice to also create an associated virtual environment. Otherwise, you may confuse different dependencies from different projects. Poetry comes with built-in support for virtual environments to ensure that it never interferes with your global Python installation. The tool can do the bulk of virtual environment management for you.

However, Poetry doesn’t create a virtual environment right away when you start a new project. That’s by design to let you decide whether you want to manage your virtual environments yourself or let Poetry handle them for you automatically.

Poetry will detect a manually activated virtual environment when you run one of the Poetry commands in your project’s folder:

(venv) $ cd rp-poetry/
(venv) $ poetry env info --path
/Users/Philipp/.virtualenvs/venv

Here, after changing your working directory to rp-poetry/, you display the environment information using the poetry env info command. Because the current shell session has an activated virtual environment—as indicated by the prompt prefix (venv)—Poetry confirms it’ll use that environment for all subsequent commands within your project scope.

In other words, if you now tried adding dependencies to your project through Poetry, you’d install them into the activated environment as if with the regular pip install command. Poetry would also update the necessary metadata in pyproject.toml, as you’ll soon discover.

On the other hand, Poetry automatically creates a virtual environment—or reuses one it had created before—when you run certain commands without an activated environment in the shell. For example, it does so when you add or remove a dependency using Poetry’s command-line interface. This prevents projects from messing with your system-wide Python installation, ensuring that project dependencies remain isolated at all times.

Letting Poetry create virtual environments on its own is the preferred way of isolating dependencies in your projects. You’ll see how that works now.

Use Poetry’s Virtual Environments

You can list all virtual environments that Poetry manages within your project by running the following command in your project’s directory:

$ poetry env list

At this point, you won’t see any output because you haven’t executed any commands that would trigger Poetry to create a virtual environment yet. Later, you may define more than one virtual environment to test your project against several configurations or different Python versions. Note that the above command remains unaffected by the state of your shell, so it doesn’t matter if you currently have an activated virtual environment or not.

To see Poetry’s support for virtual environments in action, make sure to deactivate any virtual environment that you might have activated before:

(venv) $ deactivate

From now on, Poetry will take care of the creation and management of virtual environments in your projects when you execute some of its commands.

If you want to have better control over the creation of a virtual environment, then you may tell Poetry explicitly which Python version you want to use up front:

$ poetry env use python3

With this command, you specify the path to a desired Python interpreter on your disk. Using bare python3 will work as long as you have the corresponding Python executable in your PATH. You can also use the minor Python version like 3.12 or just 3.

When Poetry creates a new virtual environment, for example, after you’ve run the poetry env use <python> command for the first time, you’ll see a message similar to this one:

$ poetry env use python3
Creating virtualenv rp-poetry-Dod5cRxq-py3.12 in
⮑ /Users/Philipp/Library/Caches/pypoetry/virtualenvs
Using virtualenv:
⮑ /Users/Philipp/Library/Caches/pypoetry/virtualenvs/rp-poetry-Dod5cRxq-py3.12

As you can see, Poetry constructed a unique name for your project’s virtual environment. It contains your package name from pyproject.toml and the corresponding Python version. The seemingly random string in the middle, Dod5cRxq, is a Base64-encoded hash value of the path leading up to your project’s parent directory. It ties the name of a virtual environment to your project’s location on disk.

When you move the project to another folder, Poetry will detect that and create a brand-new virtual environment behind the scenes if necessary. Thanks to the unique string in the middle, Poetry can handle multiple projects with identical names and the same Python version while keeping all virtual environments in one folder by default.

Unless you tell it otherwise, Poetry creates virtual environments in the virtualenvs/ subfolder of its cache directory, which is specific to the operating system:

If you want to change the default cache directory, then you can edit Poetry’s configuration. It can be useful when you already use virtualenvwrapper or another third-party tool for managing your virtual environments and don’t want to clutter your disk or mix a few cache locations.

To reveal your current Poetry configuration, which includes the cache-dir and virtualenvs.path settings, run this command:

$ poetry config --list

Usually, you don’t have to change these paths. If you want to learn more about interacting with Poetry’s virtual environments, then the Poetry documentation contains a chapter about managing environments.

As long as you’re inside your project folder, Poetry will use the virtual environment associated with it. If you’re ever in doubt, you can check whether the virtual environment has been created by running the poetry env list command again:

$ poetry env list
rp-poetry-Dod5cRxq-py3.10
rp-poetry-Dod5cRxq-py3.11
rp-poetry-Dod5cRxq-py3.12 (Activated)
rp-poetry-Dod5cRxq-py3.13

Poetry indicates which of the virtual environments connected to your project is currently selected as the default one. Although it says activated, the corresponding virtual environment isn’t actually activated in your shell in the traditional sense. Instead, Poetry will temporarily activate that virtual environment in a subprocess when you run one of the Poetry commands.

With that skill under your belt, you’re ready to add some dependencies to your project and see Poetry shine.

Declare Runtime Dependencies

When you created your rp-poetry project using Poetry, it scaffolded the minimal pyproject.toml file with the [tool.poetry.dependencies] subtable. So far, it only contains a declaration of the Python interpreter version, but it’s also where you can specify the external Python packages that your project requires.

Editing this file by hand can become tedious, though, and it doesn’t actually install anything into the project’s virtual environment. That’s where Poetry’s CLI comes into play again.

You may have used pip before to install packages that aren’t part of the Python standard library. If you run pip install with the package name as an argument, then pip looks for packages on the Python Package Index (PyPI). You can use Poetry in a similar way.

Running the poetry add command will automatically update your pyproject.toml file with the new dependency and install the package at the same time. In fact, you can even specify multiple packages in one go:

$ poetry add requests beautifulsoup4

This will find the latest versions of both dependencies on PyPI, install them in the corresponding virtual environment, and insert the following two declarations in your pyproject.toml file:

[tool.poetry]
name = "rp-poetry"
version = "0.1.0"
description = ""
authors = ["Philipp Acsany <philipp@realpython.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
requests = "^2.31.0"
beautifulsoup4 = "^4.12.3"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

The order of these declarations reflects the order in which you specified those packages in the command line.

Note the caret symbol (^) before the version specifiers, which indicates that Poetry is free to install any version matching the leftmost non-zero digit of the version string. For example, if the Requests library releases a new version 2.99.99, then Poetry will consider it an acceptable candidate for your project. However, version 3.0 wouldn’t be allowed.

If you’d like to add a particular version of an external package or define custom version constraints, then Poetry lets you do that on the command line:

$ poetry add requests==2.25.1 "beautifulsoup4<4.10"

When you run this command, Poetry will first remove any previously installed versions of these packages and downgrade their indirect or transitive dependencies as needed. It’ll then determine the most suitable versions of these packages, taking into account other existing constraints to resolve potential conflicts.

If you want to remove one or more dependencies from your project, then Poetry provides the related poetry remove command:

$ poetry remove requests
Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 0 installs, 0 updates, 5 removals

  • Removing certifi (2023.11.17)
  • Removing chardet (4.0.0)
  • Removing idna (2.10)
  • Removing requests (2.25.1)
  • Removing urllib3 (1.26.18)

Writing lock file

As you can see, it’ll remove the given dependency along with its transitive dependencies, so you don’t need to worry about leftover packages that are no longer needed by your project. This is an advantage of Poetry over plain pip, which can only uninstall the individual packages.

Notice that Poetry informs you about writing to the lock file whenever you add or remove a dependency. You’ll learn about that file later, but in a nutshell, you can think of it as a snapshot of your project’s dependencies at a given point in time. It serves the same purpose as the constraints file leveraged by pip.

So far, you’ve been adding runtime dependencies that were necessary to make your program work correctly. However, you might only need some dependencies at specific stages of development, such as during testing. Next up, you’ll see what tools Poetry gives you to manage those kinds of dependencies.

Group Dependencies and Add Extras

Another neat feature in Poetry, which is missing from pip, is the ability to manage groups of dependencies, allowing you to keep logically related dependencies separate from your runtime dependencies. For example, during development, you’ll often want additional packages, such as linters, type checkers, or testing frameworks, which would only bloat the final distribution package. Your users don’t need those, after all.

With Poetry, you can group dependencies under arbitrary names so that you can selectively install those groups later on when needed. Here’s how to add a few dependencies to a group called dev and some dependencies to another group called test:

$ poetry add --group dev black flake8 isort mypy pylint
$ poetry add --group test pytest faker

Running these commands will cause Poetry to install the listed packages into the project’s virtual environment and also add two additional subtables in your pyproject.toml file, which look as follows:

[tool.poetry]
name = "rp-poetry"
version = "0.1.0"
description = ""
authors = ["Philipp Acsany <philipp@realpython.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
beautifulsoup4 = "<4.10"

[tool.poetry.group.dev.dependencies]
black = "^24.1.1"
flake8 = "^7.0.0"
isort = "^5.13.2"
mypy = "^1.8.0"
pylint = "^3.0.3"

[tool.poetry.group.test.dependencies]
pytest = "^8.0.0"
faker = "^22.6.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

The new subtables start with the name tool.poetry.group and end with the word dependencies. The part in the middle must be a unique group name.

Normally, when you don’t provide any group name, Poetry puts the specified packages in an implicit main group, which corresponds to the [tool.poetry.dependencies] subtable. Therefore, you can’t use the name main for one of your groups because it’s been reserved.

You can define optional groups by setting the corresponding attribute in the pyproject.toml file. For example, this declaration will turn your test group into an optional one:

# ...

[tool.poetry.group.test]
optional = true

[tool.poetry.group.test.dependencies]
pytest = "^8.0.0"
faker = "^22.6.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry won’t install dependencies belonging to such a group unless you explicitly instruct it to by using the --with option. Note that you must declare another TOML subtable to mark a group as optional because the group’s configuration is kept separately from its dependency list.

In addition to this, you can add the individual packages as optional to let the user choose whether to install them:

$ poetry add --optional mysqlclient psycopg2-binary

Optional dependencies are meant to be available at runtime when explicitly requested by the user during installation. It’s common to mark packages as optional when they’re platform-specific or when they provide features, such as a particular database adapter, that only some users will need.

In pyproject.toml, optional dependencies look slightly more verbose:

[tool.poetry]
name = "rp-poetry"
version = "0.1.0"
description = ""
authors = ["Philipp Acsany <philipp@realpython.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
beautifulsoup4 = "^4.12.3"
mysqlclient = {version = "^2.2.1", optional = true}
psycopg2-binary = {version = "^2.9.9", optional = true}

# ...

The mysqlclient and psycopg2-binary dependencies have their optional flag set to true, while their version string is kept in another attribute.

However, this isn’t enough to expose such optional dependencies to the user. You must also define extras in your pyproject.toml file, which are sets of optional dependencies that your users can install together:

# ...

[tool.poetry.extras]
databases = ["mysqlclient", "psycopg2-binary"]
mysql = ["mysqlclient"]
pgsql = ["psycopg2-binary"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Depending on your particular needs, you can opt in to the desired features by selecting one or more extras during installation. As you can see, it’s also possible to combine a few extras, which might be tailored to a specific use case, such as testing.

You’ve barely scratched the surface when it comes to dependency management with Poetry. There’s so much more to explore! That said, you’re already well-equipped to implement Poetry in your projects. In the next section, you’ll learn how to work with a Poetry project from a user’s perspective.

Install Your Package With Poetry

Imagine that you’ve just cloned a Git repository with the rp-poetry project from GitHub and are starting fresh with no virtual environment. To simulate that, you can remove some of Poetry’s metadata and any virtual environments associated with the project:

$ rm poetry.lock
$ poetry env remove --all

If you’re a developer who wants to contribute to this project, then you can execute the poetry install command inside the rp-poetry/ folder to get the ball rolling:

$ poetry install

If other contributors have committed the poetry.lock file to the remote repository, which they generally should, then Poetry will read that file. It’ll reproduce the exact same environment on your machine with identical versions of all the dependencies listed in the most recent snapshot of poetry.lock.

Otherwise, Poetry will fall back to reading the top-level dependencies outlined in the pyproject.toml file and will resolve the set of packages satisfying the version constraints. As a result, you’ll have a new local poetry.lock file. This may potentially lead to a different state of dependencies in your virtual environment than those of other contributors who’ve installed the project at a different time.

The dependency resolution becomes essential when you have many dependencies that require several third-party packages with different versions of their own. Before installing any packages, Poetry figures out which version of a package fulfills the version constraints that other packages set as their requirements. That’s not a trivial task. In rare cases, a solution may not even exist!

By default, Poetry installs dependencies from the implicit main group as well as all dependency groups, unless you marked them as optional. It also installs your own rp-poetry distribution package in editable mode to allow for changes in your source code to be immediately reflected in the environment without the need for reinstallation.

In contrast, Poetry won’t automatically install extra sets of dependencies and optional groups of dependencies. To get those, you must use some of the following parameters:

• --all-extras
• --extras {extra}
• --with {optional groups}

You can install a few extras at the same time by repeating the --extras parameter for each desired set of optional dependencies. However, Poetry treats the individual extras as mutually exclusive until you say otherwise. So, it’ll only install those specified on the command line while removing all other extras from your virtual environment if needed. In particular, it’ll remove all extras when you don’t select at least one during installation.

Apart from that, you have a few more options, allowing you to cherry-pick exactly which dependencies to install, including:

When you specify the --only {groups} option, Poetry will ignore the --with {optional groups} and --without {groups} options.

Resolving dependencies results in updating or producing a new poetry.lock file. It’s where Poetry keeps track of all packages and their exact versions that your project uses. You’re going to have a closer look at that file now.

Manually Lock Dependencies

Poetry generates and refreshes the poetry.lock file when needed as long as you stick to its command-line interface. While the lock file is not meant to be changed by hand, you can edit the related pyproject.toml file at will. Unfortunately, this may sometimes cause both files to become out of sync.

Suppose you wanted to bring back the Requests library that you removed from the rp-poetry project earlier in this tutorial. You can open the pyproject.toml file in your text editor and type the necessary declaration in the main group of dependencies:

[tool.poetry]
name = "rp-poetry"
version = "0.1.0"
description = ""
authors = ["Philipp Acsany <philipp@realpython.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
requests = "*"
beautifulsoup4 = "<4.10"
mysqlclient = {version = "^2.2.1", optional = true}
psycopg2-binary = {version = "^2.9.9", optional = true}

# ...

By using the asterisk (*) as the version constraint, you indicate that you’re not specifying any particular version of the Requests library, and that any version will be acceptable. But this library isn’t installed yet.

If you now open your terminal and navigate to the project’s parent directory, then you can tell Poetry to install the manually added dependencies into the associated virtual environment and update the lock file:

$ poetry install
Installing dependencies from lock file
Warning: poetry.lock is not consistent with pyproject.toml.
⮑ You may be getting improper dependencies.
⮑ Run `poetry lock [--no-update]` to fix it.

Because rp-poetry depends on requests (*)
⮑ which doesn't match any versions,
⮑ version solving failed.

In this case, Poetry refuses to install the dependencies because your poetry.lock file doesn’t currently mention the Requests library present in the companion pyproject.toml file. Conversely, if you removed a declaration of a resolved and installed dependency from pyproject.toml, then you’d face a similar complaint. Why’s this happening?

Remember that Poetry always installs the resolved dependencies from the poetry.lock file, where it pinned down the exact package versions. It’ll only consider the dependencies that you’ve listed in the pyproject.toml file when it needs to update or regenerate a missing lock file.

Therefore, to fix such a discrepancy, you could delete the lock file and run poetry install again to let Poetry resolve all dependencies from scratch. That’s not the best approach, though. It’s potentially time-consuming. But even worse, it disregards the specific versions of previously resolved dependencies, removing the guarantee of reproducible builds.

A far better approach to align the two files is by manually locking the new dependencies with the poetry lock command:

$ poetry lock
Updating dependencies
Resolving dependencies... (1.0s)

Writing lock file

This will update your poetry.lock file to match the current pyproject.toml file without installing any dependencies.

Poetry processes all dependencies in your pyproject.toml file, finds packages that satisfy the declared constraints, and pins their exact versions in the lock file. But Poetry doesn’t stop there. When you run poetry lock, it also recursively traverses and locks all dependencies of your direct dependencies.

$ poetry lock --no-update
Resolving dependencies... (0.1s)

It’s important to note that dependency locking is only about two things:

1. Resolving: Finding packages that satisfy all version constraints
2. Pinning: Taking a snapshot of the resolved versions in the poetry.lock file

Poetry doesn’t actually install the resolved and pinned dependencies for you after running poetry lock. To confirm this, try importing the locked Requests library from the associated virtual environment, which Poetry manages for you:

$ poetry run python -c "import requests"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'

Here, you specify a one-liner program by providing the -c option to the Python interpreter in your virtual environment. The program tries to import the Requests library but fails due to a missing module, which wasn’t installed.

Now that you’ve pinned all dependencies, it’s time to install them so that you can use them in your project.

Synchronize Your Environment

When the poetry.lock file agrees with its pyproject.toml counterpart, then you can finally install dependencies that Poetry locked for you:

$ poetry install
Installing dependencies from lock file

Package operations: 5 installs, 0 updates, 0 removals

  • Installing certifi (2023.11.17)
  • Installing charset-normalizer (3.3.2)
  • Installing idna (3.6)
  • Installing urllib3 (2.2.0)
  • Installing requests (2.31.0)

Installing the current project: rp-poetry (0.1.0)

By running poetry install, you make Poetry read the poetry.lock file and install all dependencies listed there. In this case, your virtual environment already had most of the required dependencies in place, so Poetry only installed the missing ones.

When you run the same command again, Poetry won’t have much left to do anymore:

$ poetry install
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: rp-poetry (0.1.0)

As a result, the Requests library should already be available for grabs when you import it within the interactive Python REPL session started through Poetry:

$ poetry run python -q
>>> import requests
>>> requests.__version__
'2.31.0'

This time, you can import requests without any trouble, which is great! You may now exit the interpreter by typing exit() and hitting Enter.

What if your virtual environment contains leftover packages that you previously installed but no longer need? Poetry doesn’t mind such packages, even if they’re not formally declared in pyproject.toml or poetry.lock. You could’ve installed them as optional dependency groups some time ago or completely outside of Poetry’s control by, for instance, using pip directly:

$ poetry run python -m pip install httpie

The httpie package indirectly brings ten additional dependencies, which take up space and could potentially interfere with your project’s actual dependencies. Besides, external packages might sometimes create security holes if you don’t keep them up-to-date.

To synchronize your virtual environment with the locked packages pinned in poetry.lock, you can pass the optional --sync flag to the poetry install command:

$ poetry install --sync
Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 10 removals

  • Removing defusedxml (0.7.1)
  • Removing httpie (3.2.2)
  • Removing markdown-it-py (3.0.0)
  • Removing mdurl (0.1.2)
  • Removing multidict (6.0.4)
  • Removing pygments (2.17.2)
  • Removing pysocks (1.7.1)
  • Removing requests-toolbelt (1.0.0)
  • Removing rich (13.7.0)
  • Removing setuptools (69.0.3)

Installing the current project: rp-poetry (0.1.0)

This ensures that your virtual environment only contains the packages specified in your pyproject.toml and poetry.lock files, preventing potential conflicts caused by unnecessary or outdated dependencies.

In the next section, you’ll learn how to update your project’s dependencies using Poetry.

Update and Upgrade Dependencies

Suppose you added the Requests library to your project in December 2020, when the latest version of this library was the 2.25.1 release. By default, Poetry configured a permissive version constraint in your pyproject.toml file, which involves a caret (^2.25.1) to allow future non-breaking updates to pass through.

Over the years, you’ve been adding many other dependencies to your project with poetry add, and Poetry automatically picked up more recent releases of Requests that still satisfied the original version constraint. Poetry then updated the lock file accordingly and installed new versions of dependencies into your virtual environment. Now, you have the 2.29.0 release of Requests pinned in the poetry.lock file and installed in your environment.

Fast forward to the time of writing, when the library’s 2.31.0 release has become a cutting-edge version. To verify this, you can ask Poetry to compare your locked dependencies against their latest releases on PyPI by running this command:

$ poetry show --latest --top-level
beautifulsoup4     4.9.3      4.12.3     Screen-scraping library
requests           2.29.0     2.31.0     Python HTTP for Humans.

This new release continues to satisfy your version constraint for Requests, so Poetry should accept it. However, when you try to run poetry install again, it doesn’t do anything:

$ poetry install
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: rp-poetry (0.1.0)

Recall that the poetry install command prioritizes the poetry.lock file over your version constraints declared in pyproject.toml to ensure reproducible environments. If you want to update dependencies with compatible versions, then you have the following choices:

• Remove poetry.lock and run poetry install
• Run poetry lock followed by poetry install
• Run poetry update

As mentioned earlier, the first option has its drawbacks because it forces Poetry to recalculate all dependencies from scratch. The other two options are essentially equivalent, so you might choose the latter to handle both steps at once.

Updating dependencies always carries some risk. To better understand what’s about to happen, you can ask Poetry to perform a dry run before taking the dive:

$ poetry update --dry-run
Updating dependencies
Resolving dependencies... (0.7s)

Package operations: 0 installs, 2 updates, 0 removals, 3 skipped

  • Updating urllib3 (1.26.18 -> 2.2.0)
  • Installing certifi (2023.11.17): Skipped (...) Already installed
  • Installing charset-normalizer (3.3.2): Skipped (...) Already installed
  • Installing idna (3.6): Skipped (...) Already installed
  • Updating requests (2.29.0 -> 2.31.0)

The highlighted lines indicate which dependencies will be updated and in which direction.

The poetry update command will lock and update all packages along with their dependencies to their latest compatible versions. If you want to update one or more specific packages, then you can list them as arguments:

$ poetry update requests beautifulsoup4

With this command, Poetry will search for a new version of requests and a new version of beautifulsoup4 that fulfill the version constraints listed in your pyproject.toml file. Then, it’ll resolve all dependencies of your project and pin their versions in your poetry.lock file. At the same time, your pyproject.toml file won’t change because the listed constraints remain valid.

Normally, to upgrade a dependency to a version that’s outside of the version constraints declared in your pyproject.toml file, you must adjust that file beforehand. Alternatively, you can forcefully upgrade a dependency to its latest version by running the poetry add command with the at operator (@) and a special keyword, latest:

$ poetry add requests@latest

When you run the poetry add command with the latest keyword, Poetry will ignore your current version constraint in pyproject.toml and replace it with a new constraint based on the latest version found. It’s as if you’ve never added that dependency to your project before. Use this option with caution, as an incompatible version of one of your dependencies could break the project.

If you now open your pyproject.toml file, then the version constraint for the Requests library will look as follows:

[tool.poetry]
name = "rp-poetry"
version = "0.1.0"
description = ""
authors = ["Philipp Acsany <philipp@realpython.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
requests = "^2.31.0"
beautifulsoup4 = "<4.10"
mysqlclient = {version = "^2.2.1", optional = true}
psycopg2-binary = {version = "^2.9.9", optional = true}

# ...

Without using the latest keyword or an explicit version constraint in the poetry add command, Poetry would conclude that the requested package is already present in your project and would do nothing.

Now you’ve gotten a handle on how Poetry uses pyproject.toml and poetry.lock. Next up, you’ll have a final look at both files.

Compare pyproject.toml and poetry.lock

The version constraints of the dependencies declared in your pyproject.toml file can be fairly loose. This allows for some level of flexibility when incorporating bug fixes or resolving version conflicts. By having more package versions to choose from, Poetry is more likely to find a combination of compatible dependencies.

On the other hand, Poetry tracks the exact versions of the dependencies that you’re actually using in the poetry.lock file. This improves Poetry’s performance by caching the resolved package versions so that it doesn’t have to resolve them again every time you install or update your dependencies.

To ensure reproducible environments across your team, you should consider committing the poetry.lock file to your version control system like Git. By keeping this file tracked in a version control system, you ensure that all developers will use identical versions of required packages. However, there’s one notable exception.

When you develop a library rather than an application, it’s common practice not to commit the poetry.lock file. Libraries typically need to remain compatible with multiple versions of their dependencies rather than with a single, locked-down set.

When you come across a repository that contains a poetry.lock file, it’s a good idea to use Poetry to manage its dependencies. On the other hand, if other developers on your team aren’t using Poetry yet, then you should coordinate with them about the potential adoption of Poetry across the board before making any decisions. You must always maintain compatibility with other dependency management tools that the team might be using.

Convert a Folder Into a Poetry Project

Say you have an rp-hello/ folder with a hello.py script inside:

print("Hello, World!")

It’s the classic Hello, World! program, which prints the famous string on the screen. But maybe this is just the beginning of a grand project, so you decide to add Poetry to it. Instead of using the poetry new command from before, you’ll use the poetry init command inside your project folder:

$ cd rp-hello/
$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [rp-hello]:
Version [0.1.0]:
Description []: My Grand Project
Author [Philipp Acsany <philipp@realpython.com>, n to skip]:
License []:
Compatible Python versions [^3.12]:

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no

(...)

The poetry init command collects the necessary information to generate a pyproject.toml file by asking you questions interactively. It gives you recommendations with sensible defaults for most of the configurations that you need to set up, and you can press Enter to accept them. When you don’t declare any dependencies, the pyproject.toml file that Poetry creates looks something like this:

[tool.poetry]
name = "rp-hello"
version = "0.1.0"
description = "My Grand Project"
authors = ["Philipp Acsany <philipp@realpython.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

This content resembles the examples that you went through in the previous sections.

Now you can use all the commands that Poetry offers. With a pyproject.toml file present, you can now run your script from an isolated virtual environment:

$ poetry run python hello.py
Creating virtualenv rp-hello-aVGSp4kH-py3.12 in
⮑ /Users/Philipp/Library/Caches/pypoetry/virtualenvs
Hello, World!

Because Poetry didn’t find any virtual environments to use, it created a new one before executing your script. After doing this, it displays your Hello, World! message without any errors. That means you now have a working Poetry project.

Import Dependencies to Poetry

Sometimes you have a project that already comes with its own requirements file, which lists all the external dependencies. Take a look at the requirements.txt file of this Python web scraper project:

beautifulsoup4==4.9.3
certifi==2020.12.5
chardet==4.0.0
idna==2.10
requests==2.25.1
soupsieve==2.2.1
urllib3==1.26.4

This project requires seven third-party packages to run properly. Granted, only the Requests and Beautiful Soup libraries are directly referenced in the source code, while the rest are pulled in as their transitive dependencies.

Poetry doesn’t deal with such requirements files out of the box, as it keeps your project dependencies elsewhere. However, once you’ve created a Poetry project with poetry init, you can quickly import an existing requirements.txt file into the project by using this command:

$ poetry add $(cat requirements.txt)
Creating virtualenv web-scraping-bs4-MsO9mbrq-py3.12 in
⮑ /Users/Philipp/Library/Caches/pypoetry/virtualenvs

Updating dependencies
Resolving dependencies... (0.9s)

Package operations: 7 installs, 0 updates, 0 removals

  • Installing certifi (2020.12.5)
  • Installing chardet (4.0.0)
  • Installing idna (2.10)
  • Installing soupsieve (2.2.1)
  • Installing urllib3 (1.26.4)
  • Installing beautifulsoup4 (4.9.3)
  • Installing requests (2.25.1)

Writing lock file

The cat utility reads the specified file and writes its content to the standard output stream. In this case, you pass that content to the poetry add command with the help of the shell’s command substitution syntax. That, in turn, installs each dependency listed in the requirements.txt file into your Poetry project.

When a requirements file is straightforward like this, using poetry add and cat can save you some manual work. However, this isn’t ideal because all dependencies end up being listed in your pyproject.toml file:

[tool.poetry]
name = "web-scraping-bs4"
version = "0.1.0"
description = ""
authors = ["Philipp Acsany <philipp@realpython.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
beautifulsoup4 = "4.9.3"
certifi = "2020.12.5"
chardet = "4.0.0"
idna = "2.10"
requests = "2.25.1"
soupsieve = "2.2.1"
urllib3 = "1.26.4"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Normally, you only want to list those packages that your project directly depends on. Apart from that, you should generally avoid using exact version numbers, as it can lead to conflicts with existing packages or prevent you from getting the latest features and security updates. Instead, you should specify a version range that allows for some flexibility while still adhering to semantic versioning principles.

In a perfect world, your pyproject.toml file should contain only these two dependencies with slightly more permissive version specifiers:

[tool.poetry]
name = "web-scraping-bs4"
version = "0.1.0"
description = ""
authors = ["Philipp Acsany <philipp@realpython.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
beautifulsoup4 = "^4.9.3"
requests = "^2.25.1"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry will resolve the remaining dependencies and lock them in the corresponding poetry.lock file.

Now, when you show your project’s dependencies as a tree, you’ll know exactly which of them are used by your project directly and which are their transitive dependencies:

$ poetry show --tree
beautifulsoup4 4.12.3 Screen-scraping library
└── soupsieve >1.2
requests 2.31.0 Python HTTP for Humans.
├── certifi >=2017.4.17
├── charset-normalizer >=2,<4
├── idna >=2.5,<4
└── urllib3 >=1.21.1,<3

Okay. You know how to import dependencies into a Poetry project. What about exporting them the other way around?

Export Dependencies From Poetry

In some situations, you must have a requirements.txt file. For example, you may want to host your Django project on Heroku, which expects the requirements.txt file to determine what dependencies to install. Poetry gives you a few options to go about creating this file.

As long as your project’s virtual environment is up-to-date, you can pin your dependency versions using the traditional pip freeze command:

$ poetry run python -m pip freeze > requirements.txt

To execute the command within the context of your project’s virtual environment managed by Poetry, you must use poetry run. Alternatively, you could’ve activated the virtual environment in an interactive shell and then run the same command from there. Either way, you must first ensure that all the required dependencies are installed in their expected versions by, for example, issuing the poetry install command.

If that sounds like too much of a hassle to you, then you can install Poetry’s export plugin, which replaces the legacy poetry export command. It essentially lets you export dependencies from poetry.lock to various file formats, including requirements.txt, which is the default.

Depending on how you installed Poetry itself, these are the commands that you can choose from to install the mentioned plugin:

• poetry self add poetry-plugin-export
• pipx inject poetry poetry-plugin-export
• python -m pip install poetry-plugin-export

Once the plugin is installed, you can use the following command to export dependencies from your Poetry-managed project to a requirements.txt file:

$ poetry export --output requirements.txt

The resulting file includes hashes and environment markers by default, meaning that you can work with very strict requirements that resemble the content of your poetry.lock file. As a matter of fact, the export plugin needs the lock file in order to generate the requirements file. If that file doesn’t exist, then Poetry will create it after resolving and locking your dependencies.

Even though the plugin looks at the lock file to get a big picture of your project’s dependencies, it’ll only export dependencies from the implicit main group in your pyproject.toml file. If you’d like to include additional dependencies from dependency groups, including the optional and non-optional groups, then use the --with parameter followed by comma-separated names of those groups:

$ poetry export --output requirements.txt --with dev,test

Conversely, to generate a requirements file with dependencies belonging to only a few of your dependency groups while excluding the implicit main group, use the --only option:

$ poetry export --output requirements-dev.txt --only development

In all cases, the extras with optional dependencies won’t be exported. To include them as well, you must either list them explicitly by repeating the --extras parameter or enable them all at once with the --all-extras flag.

You might have noticed that the export plugin shares a few options with the poetry install command that you saw before. That’s no coincidence. To reveal the plugin’s available options and their descriptions, run poetry export --help in your terminal.