Continuous Integration and Deployment for Python With GitHub Actions

Anatomy of a Workflow File

Workflow files are declaratively written YAML files with a predefined structure that must be adhered to for a workflow to run successfully. Your YAML workflow files are stored and defined in a .github/workflows/ folder in your project’s root directory.

Your workflow folder can have multiple workflow files, each of which will perform a certain task. You can name these workflow files anything you’d like. However, for the sake of simplicity and readability, it’s common practice to name them after the tasks they achieve, such as test.yml.

Each file has a few elements that are required, but many, many more that are optional. The GitHub Actions documentation is thorough and well-written, so be sure to check it out after you’ve finished reading this tutorial.

There are three main parts that make up the bulk of a workflow file: triggers, jobs, and steps. You’ll cover these in the next sections.

Workflow Triggers

A trigger is an event that causes a workflow to run. There are many kinds of triggers. The most common ones are those that occur on a:

• Pull request
• Pushed commit to the default branch
• Tagged commit
• Manual trigger
• Request by another workflow
• New issue being opened

You might also want to restrict triggers further by limiting it to a specific branch or set of files. Here’s a simple example of a trigger that runs a workflow on any push to the main branch:

on:
  push:
    branches:
      - main

For detailed information about the triggers not covered in this tutorial, you can check out the official documentation.

Now that you know how events trigger workflows, it’s time to explore the next component of a workflow file: jobs.

Workflow Jobs

Each workflow has a single jobs section, which is the container for the meat and potatoes of the workflow. A workflow can include one or more jobs that it will run, and each job can contain one or more steps.

Here’s an example of what this section would look like without any steps:

# ...

jobs:
  my_first_job:
    name: My first job
  my_second_job:
    name: My second job

When you’re creating a job, the first thing to do is define the runner you want to use to run your job. A runner is a GitHub-hosted virtual machine (VM) that executes your jobs for you. GitHub will provision and de-provision the VM so you don’t have to worry about maintaining any infrastructure for your CI/CD.

There are multiple supported operating systems available. You can find the full list of GitHub-hosted runners in the documentation.

Defining a runner takes as little as a single line of YAML:

# ...

jobs:
  my_first_job:
    name: My first job
    runs-on: ubuntu-latest
    # ...
  my_second_job:
    name: My second job
    runs-on: windows-latest
    # ...

In the above example, my_first_job will run inside an Ubuntu VM, and my_second_job will run inside a Windows VM. Both use the -latest suffix in this case, but you could also specify the exact version of the operating system—for example, ubuntu-20.24, as long as it’s a supported version.

Workflow Steps

Steps are the main part of a job. As you’ve probably guessed, the steps declare the actions that need to be performed when executing the workflow. This can include tasks such as installing Python, running tests, linting your code, or using another GitHub action.

Just like your Python code, common and repeatable tasks can be abstracted away into separate workflows and reused. This means you can and should use other people’s GitHub Actions in your own workflows, similar to how you would when importing a Python library, to save you time reimplementing that functionality.

In the next section, you’ll see how you can use other GitHub Actions and how to find them.

Exploring the GitHub Marketplace

The GitHub Marketplace is an online repository of all the actions people can use in their own workflows. GitHub, third-party vendors, and individuals build and maintain these GitHub Actions. Anyone can use the GitHub Action template to create their own action and host it in the marketplace.

This has led to a vast array of GitHub Actions available for nearly every type of task automation imaginable. All actions in the GitHub Marketplace are open source and free to use.

In the next section, you’ll look at two GitHub Actions that you’ll use for every Python project.

Including Actions in Workflows

Every Python-based workflow you create needs to not only check out your current repository into the workflow environment but also install and set up Python. Fortunately, GitHub has official GitHub Actions to help with both tasks:

# ...

jobs:
  my_first_job:
    name: My first job
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - run: python -m pip install -r requirements.txt

In the example above, you can see that the first step in steps is to use the official checkout action. This action checks out the code from your repository into the current GitHub workspace, allowing your workflow to access it. The @4 following checkout is a version specifier, indicating which version of the action to use. As of now, the latest version is v4.2.2, so you can refer to it using this syntax to specify the latest major version.

The second step of this example sets up Python in the environment. Again, this example uses the official GitHub Action to do this because of its ongoing support and development. Most actions, if not all, have extra configurations you can add to the step.

The Setup Python action documentation contains the complete list of configurations. For now, the minimum you need to install Python into your workflow environment is to declare which version of Python you wish to install.

In the final step of the example, you use the run command. This command allows you to execute any bash or powershell command, depending on which runner you’re using for the step. In this case, you’re installing the project’s dependencies from the requirements file.

Hopefully, you can see how powerful GitHub Actions can be. With very little code and effort, you have a reproducible way to set up an environment that’s ready for building, testing, and deploying your Python project.

You now have a basic understanding of the structure of a workflow file and how you can create your first workflow for a project. In the next section, you’ll do just that with a real-world example.

Testing on Multiple Versions of Python

In the linting workflow, you used the setup-python action in your steps to set up Python 3.13 in the Ubuntu instance, which looked like this:

# ...

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
          cache: "pip"
      # ...
# ...

Unfortunately, you can’t just add a list of Python versions to python-version and be done. What you need is a strategy matrix to test on multiple versions of Python.

To quote the official documentation:

A matrix strategy lets you use variables in a single job definition to automatically create multiple job runs that are based on the combinations of the variables. For example, you can use a matrix strategy to test your code in multiple versions of a language or on multiple operating systems. (Source)

In short, whatever variables you define in your matrix will run the same steps in the job, but using those variables. Here, you want to run on different versions of Python, but you could also use this to run or build your code on different operating systems.

Declaring a strategy is relatively straightforward. Before defining your steps but as part of your job, you can define your required strategy:

 1name: Run Tests
 2
 3on:
 4  push:
 5    branches:
 6      - master
 7  pull_request:
 8    branches:
 9      - master
10  workflow_call:
11  workflow_dispatch:
12
13jobs:
14  testing:
15    runs-on: ubuntu-latest
16    strategy:
17      matrix:
18        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]

As you can see, you’re declaring a variable python-version, which is an array of version numbers. Great, this is the first part done! The second part is to tell the setup-python action that you want to use these versions using a special variable syntax:

 1name: Run Tests
 2
 3on:
 4  push:
 5    branches:
 6      - master
 7  pull_request:
 8    branches:
 9      - master
10  workflow_call:
11  workflow_dispatch:
12
13jobs:
14  testing:
15    runs-on: ubuntu-latest
16    strategy:
17      matrix:
18        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
19
20    steps:
21      - uses: actions/checkout@v4
22      - name: Set up Python ${{ matrix.python-version }}
23        uses: actions/setup-python@v5
24        with:
25          python-version: ${{ matrix.python-version }}
26          cache: "pip"

The Python setup step of the workflow now has two changes. The first is the added name to the step. As you learned earlier, this isn’t required but it will help you identify which Python version failed by referencing the Python version in the step’s name. This is helpful, given that this step will run for five different versions of Python.

The second change is that instead of hard coding the version number into the with: python-version part of setup-python, you can now refer to the python-version defined in the matrix.

GitHub has a few special contexts that you can access as part of your workflows. Matrix is one of these. By defining the matrix as part of the strategy, python-version has now become a property of the matrix context. This means that you can access any variable defined as part of the matrix with the dot (.) syntax, for example, matrix.python-version.

Although this isn’t something that needs to be done for Real Python Reader, you could do the same with different OS versions. For example:

strategy:
  matrix:
    os: [ubuntu-latest, windows-latest]

You could then use the same dot notation to access the os variable you defined in the matrix with matrix.os.

Now that you know how to use a matrix to run your steps declaratively using a different version of Python, it’s time to complete the testing workflow in full.

Finalizing the Testing Workflow

There are just a few more steps needed in order to finish the workflow. Now that Python is installed, the workflow will need to install the developer dependencies and then finally run pytest.

The Real Python Reader package uses a pyproject.toml configuration file for declaring its dependencies. It also has optional developer dependencies, which include pytest. You can install them the same way you installed Ruff earlier, using the run command:

 1name: Run Tests
 2
 3on:
 4  push:
 5    branches:
 6      - master
 7  pull_request:
 8    branches:
 9      - master
10  workflow_call:
11  workflow_dispatch:
12
13jobs:
14  testing:
15    runs-on: ubuntu-latest
16    strategy:
17      matrix:
18        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
19
20    steps:
21      - uses: actions/checkout@v4
22      - name: Set up Python ${{ matrix.python-version }}
23        uses: actions/setup-python@v5
24        with:
25          python-version: ${{ matrix.python-version }}
26          cache: "pip"
27
28      - name: Install dependencies
29        run: |
30          python -m pip install --upgrade pip
31          python -m pip install .[dev]

This step is all you need to install the required dependencies. The only remaining step is to run pytest:

 1name: Run Tests
 2
 3on:
 4  push:
 5    branches:
 6      - master
 7  pull_request:
 8    branches:
 9      - master
10  workflow_call:
11  workflow_dispatch:
12
13jobs:
14  testing:
15    runs-on: ubuntu-latest
16    strategy:
17      matrix:
18        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
19
20    steps:
21      - uses: actions/checkout@v4
22      - name: Set up Python ${{ matrix.python-version }}
23        uses: actions/setup-python@v5
24        with:
25          python-version: ${{ matrix.python-version }}
26          cache: "pip"
27
28      - name: Install dependencies
29        run: |
30          python -m pip install --upgrade pip
31          python -m pip install .[dev]
32
33      - name: Run Pytest
34        run: pytest

At this point, you have both a linting and testing workflow that are triggered whenever a PR or push event happens on master. Next, you’ll turn your attention to the CD part of CI/CD, and learn how you can automatically publish a package to PyPI.

Setting Up and Building the Package

As with the past two workflows, the first step is to define the triggers for the workflow. You’ve seen some common triggers that revolve around typical developer workflows, but automatically releasing with every new PR or push to the main branch isn’t ideal for Real Python Reader.

It makes more sense to bump the version of the package after several pull requests, bug fixes, or after adding new features. The modern way of triggering such a release after a version bump is to use the developer’s best friend, Git.

Git allows you to tag a commit to denote a notable point in time in the software’s development. This is often the tool of choice to define a new release. GitHub Actions have built-in support for using Git tags as triggers through the tags keyword:

 1name: Publish to PyPI
 2on:
 3  push:
 4    tags:
 5      - "*.*.*"

As you can see here, triggers also support glob patterns. So an asterisk (*) can match any character in a sequence. The pattern outlined above will match any character followed by a decimal point (.), another character, another decimal point, and finally, another character.

This means that 1.0.0 is a valid match, as is 2.5.60. This matches the semantic versioning used by Real Python Reader. You could also use v*.*.* instead if you prefer. So, your Git tags should start with a v, which stands for version. For example, v1.0.0 would be a valid tag.

In order to trigger this workflow, you’d tag a commit with the version name:

$ git tag -a "1.0.0" -m "1.0.0"
$ git push --tags

Pushing your new tag to GitHub will then trigger this workflow. Next, you’ll set up the environment and install the dependencies:

 1name: Publish to PyPI
 2on:
 3  push:
 4    tags:
 5      - "*.*.*"
 6
 7jobs:
 8  publish:
 9    runs-on: ubuntu-latest
10    steps:
11    - uses: actions/checkout@v4
12    - name: Set up Python
13      uses: actions/setup-python@v5
14      with:
15        python-version: "3.13"
16
17    - name: Install dependencies
18      run: |
19        python -m pip install --upgrade pip
20        python -m pip install .[build]
21
22    - name: Build package
23      run: python -m build

First, you define the publish job and install Python 3.13 into an Ubuntu VM. The next step installs the build dependencies of Real Python Reader. In the last step, you use the same run command you’ve used before, but this time, instead of running Ruff or pytest, you’ll build the Real Python Reader package. By default, build will place the distribution files in a folder called dist.

Excellent! You’ve implemented the first two main parts of the workflow plan. Before you can deploy to PyPI, you should know how to keep your PyPI API token secure.

Keeping Your Secrets Secure

As you learned earlier, workflows get access to special contexts like matrix. Another context that all workflows have access to is the secrets context. By storing sensitive data as a repository secret, you can ensure you never accidentally leak API keys, passwords, or other credentials. Your workflow can access those sensitive credentials using the secrets context.

You can add secrets to your repository on the GitHub website. Once you’ve added them, you can’t view or edit them. You can only replace them with a new value. It’s a good idea to review the GitHub documentation to see how to add secrets on the GitHub website. The official docs are continually updated with any UI changes, making them the best source for learning how to use this GitHub feature.

Deploying Your Package

After securing your API key as a GitHub secret, you can access it in the workflow:

 1name: Publish to PyPI
 2on:
 3  push:
 4    tags:
 5      - "*.*.*"
 6
 7jobs:
 8  publish:
 9    runs-on: ubuntu-latest
10    steps:
11    - uses: actions/checkout@v4
12    - name: Set up Python
13      uses: actions/setup-python@v5
14      with:
15        python-version: "3.13"
16
17    - name: Install dependencies
18      run: |
19        python -m pip install --upgrade pip
20        python -m pip install .[build]
21
22    - name: Build package
23      run: python -m build
24
25    - name: Test publish package
26      uses: pypa/gh-action-pypi-publish@release/v1
27      with:
28        user: __token__
29        password: ${{ secrets.PYPI_API_TOKEN }}
30        repository-url: https://test.pypi.org/legacy/
31
32    - name: Publish package
33      uses: pypa/gh-action-pypi-publish@release/v1
34      with:
35        user: __token__
36        password: ${{ secrets.PYPI_API_TOKEN }}

In this step, you get to use the official GitHub Action from the Python Packaging Authority (PyPA), which manages PyPI. This GitHub Action does most of the work and only needs a reference to your PyPI API token. Again, by default, it will look in your dist folder for any new version of a package to upload.

Rather than using a traditional username and password to authenticate to PyPI, it’s best practice to use a scoped API token instead for automatic releases.

Since you’re using an API token and there’s no username, using __token__ as the username tells the GitHub Action that token authentication is being used. Just like with the previous matrix strategy, you can use dot notation to access the secret context, as in secrets.PYPI_API_TOKEN.

The name of the secret when stored in GitHub doesn’t matter, as long as it makes sense to you. The GitHub secret is named PYPI_API_TOKEN, so you reference it inside the workflow using that name.

You may have noticed that the workflow includes a test step prior to publishing the package to PyPI. This step is almost identical to the publishing step, with one key difference: you’ll need to provide a repository-url to override the default URL and push the package to test.pypi.org.

Using TestPyPI is an excellent way to ensure that your package is built and versioned correctly. It allows you to identify and address any potential issues that might cause problems when publishing to the main PyPI repository.

If you’re following along with your own fork of the repository and intend to push your version to PyPI, then you’ll need to update the name of the project to a unique name. If you don’t update the project name, you’ll recieve an HTTP 403 error when trying to upload it. This is because you don’t have permission to publish the realpython-reader package to PyPI. Updating the project name will allow you to publish your own version.

As an example, you could add your username as a prefix to the project name:

[build-system]
requires      = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "username-realpython-reader"
# ...

There’s just one more step of the workflow to complete—creating a GitHub release to promote and then sharing the release directly. Before you can do this, you’ll learn about GitHub environment variables.

Accessing GitHub Environment Variables

In order to publish a release to a GitHub repo, a GitHub token is required. You may have used these before if you’ve ever used the GitHub API. Given the security risk of using personal GitHub tokens in workflows, GitHub creates a read-only token in the secrets context by default. This means that you always have access to it if you need it.

In addition, every GitHub runner includes the handy GitHub CLI by default. This makes performing certain tasks, like creating a release, so much simpler. The GitHub CLI has many ways to authenticate the user, one of which is by setting an environment variable called GITHUB_TOKEN.

You may see where this is going. The provided GitHub token can be used to access the CLI and ultimately create a seamless way to create the GitHub release. Here’s what that would look like in the workflow:

 1name: Publish to PyPI
 2on:
 3  push:
 4    tags:
 5      - "*.*.*"
 6
 7jobs:
 8  publish:
 9    runs-on: ubuntu-latest
10    steps:
11    - uses: actions/checkout@v4
12    - name: Set up Python
13      uses: actions/setup-python@v5
14      with:
15        python-version: "3.13"
16
17    - name: Install dependencies
18      run: |
19        python -m pip install --upgrade pip
20        python -m pip install .[build]
21
22    - name: Build package
23      run: python -m build
24
25    - name: Test publish package
26      uses: pypa/gh-action-pypi-publish@release/v1
27      with:
28        user: __token__
29        password: ${{ secrets.PYPI_API_TOKEN }}
30        repository-url: https://test.pypi.org/legacy/
31
32    - name: Publish package
33      uses: pypa/gh-action-pypi-publish@release/v1
34      with:
35        user: __token__
36        password: ${{ secrets.PYPI_API_TOKEN }}
37
38    - name: Create GitHub Release
39      env:
40        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
41      run: |
42        gh release create ${{ github.ref_name }} ./dist/* --generate-notes

You’ll see that on lines 39 and 40, the workflow specifically assigns the GitHub token from the secrets context to an environment variable called GITHUB_TOKEN. Any key values set in env will be set as environment variables for the current step. This means that when you run the GitHub CLI (gh), it will have access to the token through the assigned environment variable. The GitHub CLI can’t directly access the secrets context itself.

GitHub also lets you access a special context called github. The workflow references the ref_name attribute in the github context. This is defined in the GitHub docs as follows:

The short ref name of the branch or tag that triggered the workflow run. (Source)

So, github.ref_name will be replaced with the attribute that triggered the workflow, which in this case is the Git tag’s name.

The gh command above will create a release with the same name as the tag used to trigger the release, upload all files from ./dist, and auto-generate release notes. These release notes include any PRs that developers have merged since they created the last release, giving proper credit to the authors with links and usernames for their contributions.

You may want to add any missing details to the release notes. Remember that releases can be edited after creation if you need to include additional information, such as deprecation notices.

Congratulations! You now have automated linting, testing, and deployment in place. You can tag your latest commit, and the final deployment workflow should run successfully:

Now that the Real Python Reader has a CI/CD pipeline to ensure that any future codebase changes are robust and use readable and consistent code, you can add one more workflow to Real Python Reader. The cherry on the top of our CI/CD cake, so to speak.

In the next section, you’ll learn how to configure Dependabot to automate security and dependency updates.