How to Get a List of All Files in a Directory With Python

Conditional Listing Using .glob()

The .glob() method of a Path object behaves in much the same way as .rglob(). If you pass the "*" argument, then you’ll get a list of items in the directory, but without recursion:

>>> import pathlib
>>> desktop = pathlib.Path("Desktop")

>>> # .glob() produces a generator too
>>> desktop.glob("*")
<generator object Path.glob at 0x000001A8A50E2F00>

>>> # Which you can wrap in a list() constructor to materialize
>>> list(desktop.glob("*"))
[WindowsPath('Desktop/Notes'),
 WindowsPath('Desktop/realpython'),
 WindowsPath('Desktop/scripts'),
 WindowsPath('Desktop/todo.txt')]

Using the .glob() method with the "*" glob pattern on a Path object produces a generator that yields all the items in the directory that’s represented by the Path object, without going into the subdirectories. In this way, it produces the same result as .iterdir(), and you can use the resulting generator in a for loop or a comprehension, just as you would with iterdir().

But as you already learned, what really sets the glob methods apart are the different patterns that you can use to match only certain paths. If you only wanted paths that ended with .txt, for example, then you could do the following:

>>> desktop = pathlib.Path("Desktop")
>>> list(desktop.glob("*.txt"))
[WindowsPath('Desktop/todo.txt')]

Since this directory only has one text file, you get a list with just one item. If you wanted to get only items that start with real, for example, then you could use the following glob pattern:

>>> list(desktop.glob("real*"))
[WindowsPath('Desktop/realpython')]

This example also only produces one item, because only one item’s name starts with the characters real. Remember that on Unix-like systems, glob patterns are case-sensitive.

You can also get the contents of a subdirectory by including its name, a forward slash (/), and an asterisk. This type of pattern will yield everything inside the target directory:

>>> list(desktop.glob("realpython/*"))
[WindowsPath('Desktop/realpython/iterate-dict.md'),
 WindowsPath('Desktop/realpython/tictactoe.md')]

In this example, using the "realpython/*" pattern yields all the files within the realpython directory. It’ll give you the same result as creating a path object representing the Desktop/realpython path and calling .glob("*") on it.

Next up, you’ll look a bit further into filtering with .rglob() and learn how it differs from .glob().

Conditional Listing Using .rglob()

Just the same as with the .glob() method, you can adjust the glob pattern of .rglob() to give you only a certain file extension, except that .rglob() will always search recursively:

>>> list(desktop.rglob("*.md"))
[WindowsPath('Desktop/Notes/hash-tables.md'),
 WindowsPath('Desktop/realpython/iterate-dict.md'),
 WindowsPath('Desktop/realpython/tictactoe.md')]

By adding .md to the glob pattern, now .rglob() produces only .md files across different directories and subdirectories.

You can actually use .glob() and get it to behave in the same way as .rglob() by adjusting the glob pattern passed as an argument:

>>> list(desktop.glob("**/*.md"))
[WindowsPath('Desktop/Notes/hash-tables.md'),
 WindowsPath('Desktop/realpython/iterate-dict.md'),
 WindowsPath('Desktop/realpython/tictactoe.md')]

In this example, you can see that the call to .glob("**/*.md") is equivalent to .rglob(*.md). Likewise, a call to .glob("**/*") is equivalent to .rglob("*").

The .rglob() method is a slightly more explicit version of calling .glob() with a recursive pattern, so it’s probably better practice to use the more explicit version instead of using recursive patterns with the normal .glob().

Advanced Matching With the Glob Methods

One of the potential drawbacks with the glob methods is that you can only select files based on glob patterns. If you want to do more advanced matching or filter on the attributes of the item, then you need to reach for something extra.

To run more complex matching and filtering, you can follow at least three strategies. You can use:

1. A for loop with a conditional check
2. A comprehension with a conditional expression
3. The built-in filter() function

Here’s how:

>>> import pathlib
>>> desktop = pathlib.Path("Desktop")

>>> # Using a for loop
>>> for item in desktop.rglob("*"):
...     if item.is_file():
...         print(item)
...
Desktop\todo.txt
Desktop\Notes\hash-tables.md
Desktop\realpython\iterate-dict.md
Desktop\realpython\tictactoe.md
Desktop\scripts\rename_files.py
Desktop\scripts\request.py

>>> # Using a comprehension
>>> [item for item in desktop.rglob("*") if item.is_file()]
[WindowsPath('Desktop/todo.txt'),
 WindowsPath('Desktop/Notes/hash-tables.md'),
 WindowsPath('Desktop/realpython/iterate-dict.md'),
 WindowsPath('Desktop/realpython/tictactoe.md'),
 WindowsPath('Desktop/scripts/rename_files.py'),
 WindowsPath('Desktop/scripts/request.py')]

>>> # Using the filter() function
>>> list(filter(lambda item: item.is_file(), desktop.rglob("*")))
[WindowsPath('Desktop/todo.txt'),
 WindowsPath('Desktop/Notes/hash-tables.md'),
 WindowsPath('Desktop/realpython/iterate-dict.md'),
 WindowsPath('Desktop/realpython/tictactoe.md'),
 WindowsPath('Desktop/scripts/rename_files.py'),
 WindowsPath('Desktop/scripts/request.py')]

In these examples, you’ve first called the .rglob() method with the "*" pattern to get all the items recursively. This produces all the items in the directory and its subdirectories. Then you use the three different approaches listed above to filter out the items that aren’t files. Note that in the case of filter(), you’ve used a lambda function.

The glob methods are extremely versatile, but for large directory trees, they can be a bit slow. In the next section, you’ll be examining an example in which reaching for more controlled iteration with .iterdir() may be a better choice.

Using .rglob() to Filter Whole Directories

If you use .rglob(), you can just filter out the items once they’re produced by .rglob(). To properly discard paths that are in a junk directory, you can check if any of the elements in the path match with any of the elements in a list of directories to skip:

>>> SKIP_DIRS = ["temp", "temporary_files", "logs"]

Here, you’re defining SKIP_DIRS as a list that contains the strings of the paths that you want to exclude.

A call to .rglob() with a bare asterisk as an argument will produce all the items, even those in the directories that you aren’t interested in. Because you have to go through all the items, there’s a potential issue if you only look at the name of a path:

large_dir/documents/notes/temp/2/0.txt

Since the name is just 0.txt, it wouldn’t match any items in SKIP_DIRS. You’d need to check the whole path for the blocked name.

You can get all the elements in the path with the .parts attribute, which contains a tuple of all the elements in the path:

>>> import pathlib
>>> temp_file = pathlib.Path("large_dir/documents/notes/temp/2/0.txt")
>>> temp_file.parts
('large_dir', 'documents', 'notes', 'temp', '2', '0.txt')

Then, all you need to do is to check if any element in the .parts tuple is in the list of directories to skip.

You can check if any two iterables have an item in common by taking advantage of sets. If you cast one of the iterables to a set, then you can use the .isdisjoint() method to determine whether they have any elements in common:

>>> {"documents", "notes", "find_me.txt"}.isdisjoint({"temp", "temporary"})
True

>>> {"documents", "temp", "find_me.txt"}.isdisjoint({"temp", "temporary"})
False

If the two sets have no elements in common, then .isdisjoint() returns True. If the two sets have at least one element in common, then .isdisjoint() returns False. You can incorporate this check into a for loop that goes over all the items returned by .rglob("*"):

>>> SKIP_DIRS = ["temp", "temporary_files", "logs"]
>>> large_dir = pathlib.Path("large_dir")

>>> # With a for loop
>>> for item in large_dir.rglob("*"):
...     if set(item.parts).isdisjoint(SKIP_DIRS):
...         print(item)
...
large_dir\documents
large_dir\documents\0.txt
large_dir\documents\1.txt
large_dir\documents\2.txt
large_dir\documents\3.txt
large_dir\documents\4.txt
large_dir\documents\notes
large_dir\documents\tools
large_dir\documents\notes\0.txt
large_dir\documents\notes\find_me.txt
large_dir\documents\tools\33.txt
large_dir\documents\tools\34.txt
large_dir\documents\tools\36.txt
large_dir\documents\tools\37.txt
large_dir\documents\tools\real_python.txt

In this example, you print all the items in large_dir that aren’t in any of the junk directories.

To check if the path is in one of the unwanted folders, you cast item.parts to a set and use .isdisjoint() to check if SKIP_DIRS and .parts don’t have any items in common. If that’s the case, then the item gets printed.

You can also accomplish the same effect with filter() and comprehensions, as below:

>>> # With a comprehension
>>> [
...     item
...     for item in large_dir.rglob("*")
...     if set(item.parts).isdisjoint(SKIP_DIRS)
... ]

>>> # With filter()
>>> list(
...     filter(
...         lambda item: set(item.parts).isdisjoint(SKIP_DIRS),
...         large_dir.rglob("*")
...     )
... )

These methods are already getting a bit cryptic and hard to follow, though. Not only that, but they aren’t very efficient, because the .rglob() generator has to produce all the items so that the matching operation can discard that result.

You can definitely filter out whole folders with .rglob(), but you can’t get away from the fact that the resulting generator will yield all the items and then filter out the undesirable ones, one by one. This can make the glob methods very slow, depending on your use case. That’s why you might opt for a recursive .iterdir() function, which you’ll explore next.

Creating a Recursive .iterdir() Function

In the example of junk directories, you ideally want the ability to opt out of iterating over all the files in a given subdirectory if they match one of the names in SKIP_DIRS:

# skip_dirs.py

import pathlib

SKIP_DIRS = ["temp", "temporary_files", "logs"]

def get_all_items(root: pathlib.Path, exclude=SKIP_DIRS):
    for item in root.iterdir():
        if item.name in exclude:
            continue
        yield item
        if item.is_dir():
            yield from get_all_items(item)

In this module, you define a list of strings, SKIP_DIRS, that contains the names of directories that you’d like to ignore. Then you define a generator function that uses .iterdir() to go over each item.

The generator function uses the type annotation : pathlib.Path after the first argument to indicate that you can’t just pass in a string that represents a path. The argument needs to be a Path object.

If the item name is in the exclude list, then you just move on to the next item, skipping the whole subdirectory tree in one go.

If the item isn’t in the list, then you yield the item, and if it’s a directory, you invoke the function again on that directory. That is, within the function body, the function conditionally invokes the same function again. This is a hallmark of a recursive function.

This recursive function efficiently yields all the files and directories that you want, excluding all that you aren’t interested in:

>>> import pathlib
>>> import skip_dirs
>>> large_dir = pathlib.Path("large_dir")

>>> list(skip_dirs.get_all_items(large_dir))
[WindowsPath('large_dir/documents'),
 WindowsPath('large_dir/documents/0.txt'),
 WindowsPath('large_dir/documents/1.txt'),
 WindowsPath('large_dir/documents/2.txt'),
 WindowsPath('large_dir/documents/3.txt'),
 WindowsPath('large_dir/documents/4.txt'),
 WindowsPath('large_dir/documents/notes'),
 WindowsPath('large_dir/documents/notes/0.txt'),
 WindowsPath('large_dir/documents/notes/find_me.txt'),
 WindowsPath('large_dir/documents/tools'),
 WindowsPath('large_dir/documents/tools/33.txt'),
 WindowsPath('large_dir/documents/tools/34.txt'),
 WindowsPath('large_dir/documents/tools/36.txt'),
 WindowsPath('large_dir/documents/tools/37.txt'),
 WindowsPath('large_dir/documents/tools/real_python.txt')]

Crucially, you’ve managed to opt out of having to examine all the files in the undesired directories. Once your generator identifies that the directory is in the SKIP_DIRS list, it just skips the whole thing.

So, in this case, using .iterdir() is going to be far more efficient than the equivalent glob methods.

In fact, you’ll find that .iterdir() is generally more efficient than the glob methods if you need to filter on anything more complex than can be achieved with a glob pattern. However, if all you need to do is to get a list of all the .txt files recursively, then the glob methods will be faster.

Check out the downloadable materials for some tests that demonstrate the relative speed of different ways to list files in Python:

With that information under your belt, you’ll be ready to select the best way to list the files and folders that you need!