Python Performance Optimization: A Practical Guide

The Goal of Optimization

Here’s the part that trips people up: optimization isn’t the same as fixing bugs. Your code already works, so the question isn’t whether it produces the right answer but how efficiently it gets there. You pursue that efficiency deliberately, not by tinkering at random.

The Doctor Analogy

Think about how a good doctor works. You walk in feeling unwell, and they don’t immediately hand you a prescription. They run tests first, then treat the specific problem the results reveal. Prescribing medication before a diagnosis would be reckless.

When you optimize without measuring, you’re doing exactly that: prescribing a cure for a disease you haven’t diagnosed. The bottleneck—the specific part of your program that consumes the most time or memory—is the disease.

Keep this picture in mind because it anchors everything that follows: diagnose, confirm, then treat. Profilers do the diagnosing, and Real Python has tutorials devoted to them. This tutorial picks up at the confirmation step and focuses on treatment.

Now that you have a clear definition and the right mental model, you’re ready to put them to work. In the next section, you’ll first confirm your suspicion, then choose the fix that matches what you find.

Choosing Better Algorithms and Data Structures

Choosing the right algorithm or data structure often yields the biggest performance gains. Even on a fast machine, a slow algorithm still produces a slow program. A common example is membership checking: searching for an item in a list is O(n), meaning Python may have to scan every element. With a set, Python performs the same check in constant time on average, O(1).

numbers_list holds 100,000 numbers, and the target is the very last one. That’s the worst case for a list because Python has to scan every element before it finds a match:

import timeit

numbers_list = list(range(100_000))

target = 99_999

list_time = timeit.timeit(lambda: target in numbers_list, number=1000)

print(target in numbers_list)
print(f"list membership: {list_time:.8f} seconds")

$ python step-2-slow.py
True
list membership: 0.39360604 seconds

numbers_set holds the same 100,000 numbers, stored in a hash table. Python jumps directly to the right location regardless of how many items the set holds:

import timeit

numbers_list = list(range(100_000))
target = 99_999

numbers_set = set(numbers_list)

set_time = timeit.timeit(lambda: target in numbers_set, number=1000)

print(target in numbers_set)
print(f"set membership:  {set_time:.8f} seconds")

$ python step-2-fast.py
True
set membership:  0.00003825 seconds

The same caveat applies to every comparison that follows. What matters is the ratio between the times rather than the raw measurements.

Building the set isn’t free, though. set(numbers_list) walks the whole list once, but hashing and inserting each item is several times as expensive as a single comparison, so building the set takes roughly as long as two to three full scans. That’s a bargain when you’re testing membership over and over, which is the usual reason to reach for a set. Converting a list you’ll only search once will cost you more than it saves.

Real Python’s guide to Common Python Data Structures compares lists, sets, dictionaries, and the specialized containers in collections, so you can match the container to the access pattern your code actually needs.

Reaching for Built-Ins and the Standard Library

Python’s built-in functions and many standard library tools are already highly optimized, whether through an underlying C implementation or carefully tuned Python code. collections.Counter tallies items in a single call and beats a manual counting loop, not because it’s magic, but because someone has already done the optimization work for you.

manual_count() loops through the list and updates a dictionary one item at a time, with a conditional check on every iteration:

import timeit

words = ["apple", "pear", "apple", "cherry", "pear", "apple"] * 1000

def manual_count(items):
    counts = {}
    for item in items:
        if item in counts:
            counts[item] += 1
        else:
            counts[item] = 1
    return counts

print(manual_count(words))

manual_time = timeit.timeit(lambda: manual_count(words), number=1000)
print(f"manual loop: {manual_time:.4f} seconds")

$ python step-3-slow.py
{'apple': 3000, 'pear': 2000, 'cherry': 1000}
manual loop: 0.2081 seconds

counter_count() hands the entire list to Counter, which does the same job without running a loop in Python. Counter is a dict subclass, so you get the same counts wrapped in a richer type.

Counter itself is written in Python, but when you give it an iterable, it delegates the counting to a helper that CPython implements in C, so the per-item work happens at C speed. The word list contains 6,000 items (6 entries multiplied by 1,000), and timeit runs each function 1,000 times, making the speed difference visible in the output:

from collections import Counter
import timeit

words = ["apple", "pear", "apple", "cherry", "pear", "apple"] * 1000

def counter_count(items):
    return Counter(items)

print(counter_count(words))

counter_time = timeit.timeit(
    lambda: counter_count(words), number=1000
)
print(f"collections.Counter: {counter_time:.4f} seconds")

$ python step-3-fast.py
Counter({'apple': 3000, 'pear': 2000, 'cherry': 1000})
collections.Counter: 0.0831 seconds

Reaching for a built-in isn’t automatically a win, though. Wrapping the same comparison in any() still runs one Python-level step per item, so it can land slower than the plain in it replaced. What makes Counter fast is that it pushes the per-item work down into C, not that it’s part of the standard library. When you swap in a built-in, measure it like any other change.

For more ready-made tools like this one, browse Python’s Built-in Functions and the collections module. If you’d rather work through the standard library systematically, Real Python’s Important Standard Library Modules learning path has tutorials and video courses on the modules that come up most often in everyday Python code.

Caching Results With functools.cache

If your code computes the same result repeatedly, then store it and reuse it. That’s caching, and functools.cache is the most direct way to use it. It arrived in Python 3.9, so on older versions you’ll want @lru_cache(maxsize=None) instead.

Caching pays off most when the repeated work is expensive, and sometimes a set isn’t an option. If you need to know an item’s position rather than just whether it exists, then list.index() is usually the right call, and a dict mapping values to positions is faster still. But when you wrap the lookup in your own logic—a filter, a transformation, or a comparison that .index() can’t express—you’re left scanning the list yourself.

find_index() does exactly that, checking each element in turn until it finds a match:

import timeit

numbers = list(range(2_000_000))

def find_index(target):
    for i, value in enumerate(numbers):
        if value == target:
            return i
    return -1

target = 1_999_999

print(find_index(target))

slow_time = timeit.timeit(lambda: find_index(target), number=20)
print(f"without cache: {slow_time:.4f} seconds")

$ python step-4-slow.py
1999999
without cache: 1.1048 seconds

find_index_cached() adds @cache so the scan happens only once. On every subsequent call with the same target, Python skips the loop entirely and returns the stored result:

from functools import cache
import timeit

numbers = list(range(2_000_000))
target = 1_999_999

@cache
def find_index_cached(target):
    for i, value in enumerate(numbers):
        if value == target:
            return i
    return -1

print(find_index_cached(target))

fast_time = timeit.timeit(
    lambda: find_index_cached(target), number=20
)
print(f"with cache: {fast_time:.8f} seconds")

$ python step-4-fast.py
1999999
with cache: 0.00000379 seconds

The two timings aren’t measuring the same work. The 1.1 seconds represents twenty full scans. The microseconds represent twenty cache hits because the first call ran and filled the cache before the timer started.

Adding @cache doesn’t make that first scan any faster. It still takes as long as the uncached version. Subsequent calls with the same argument are nearly free, which is the whole point. But you realize those savings only when you call the function again with the same argument.

One trap to keep in mind is that @cache stores every unique set of arguments indefinitely. If you call your function with many different inputs over time, the cache grows without bound and can become a memory problem. In that case, switch to @lru_cache(maxsize=128) or whichever size fits your use case. This caps the cache size and evicts the least recently used entries when it fills up, balancing speed and memory efficiency.

There’s a second trap to watch for. find_index_cached() reads numbers from the global scope, but the cache key is built from target alone. If numbers changes after the first call—say, you append a new item—the cache still returns the old answer, and the code won’t warn you.

Caching is safe only when a function’s result depends solely on its arguments, so be careful when applying it to a function that reads external data. To understand what’s happening behind the decorator, read Caching in Python Using the LRU Cache Strategy or watch Caching in Python With lru_cache.

That’s all three fixes, each one measured against a baseline rather than assumed. The widget below hands you the same loop: pick a suspect function, measure it before you change anything, then try each fix and watch whether the number actually moves:

Whichever suspect you start from, take a baseline first, and then change one thing at a time.

Going Further With Memory, Concurrency, and C Extensions

These next fixes require more effort, so reach for them only after the first three come up short. Each one links to a full tutorial rather than a quick snippet:

1. Memory Efficiency: Speed isn’t the only cost. Swap lists for generators to produce values one at a time rather than all at once, and add .__slots__ to classes to shrink each instance. Read How to Use Generators and yield in Python to go deeper.
2. Concurrency: When one core isn’t keeping up, reach for threading on I/O-bound tasks, multiprocessing on CPU-bound ones, and asyncio for high-concurrency I/O. See Speed Up Your Python Program With Concurrency for the full picture.
3. C Extensions: When nothing else works, drop down to C with Cython or a handwritten extension. It’s a last resort that adds complexity. Building a Python C Extension Module walks you through it.

With three fixes covered and links to full tutorials on three more, you have a complete map for approaching any performance problem. Next, you’ll see the costs of slow code and of speeding up the wrong part of it.

The Cost of Not Optimizing

The cost of slow code compounds. A data pipeline that takes ten minutes and runs every day quietly burns about sixty hours a year. A web endpoint that feels sluggish under light traffic can tip over into a full outage once real users arrive and pile requests on top of each other.

The Cost of Optimizing the Wrong Thing

Guessing wrong wastes time you can’t get back. A common rule of thumb is that roughly 80 percent of runtime tends to hide in about 20 percent of the code. Optimize the rest of the code, and you’ll get almost nothing back, no matter how clever your rewrite is.

The Common Hotspots

You’ll encounter performance problems in more places than you might expect:

• Data pipelines and extract, transform, load (ETL) scripts that chew through large files or datasets
• Web APIs that slow to a crawl when serving concurrent users
• Scheduled automation scripts that take a little longer every month
• Machine learning pipelines that preprocess data and engineer features
• Command-line interface (CLI) tools where the wait is something a human actually feels

Wherever you recognize your own code in that list, the same diagnostic process applies.

The Mindset Shift

Performance work isn’t a heroic weekend rewrite. It’s a calm, repeatable diagnostic process, and the earlier fixes give you an order to follow. The next section shows you the habits that work against that process.

Pitfall 1: Optimizing Before Measuring

This is the big one. You decide where the slowdown must be, rewrite that spot, and see no improvement because your guess was wrong. It’s the doctor writing a prescription before running any tests. The fix is to reach for a profiler when you don’t know where the time goes, then use timeit to confirm a specific suspect. The real bottleneck is often somewhere you never thought to look.

Pitfall 2: Skipping Straight to Advanced Techniques

It’s tempting to reach for concurrency or C extensions right away because they feel powerful and impressive. Often, a better data structure would have solved the problem in five minutes. Swapping a list for a set in a membership check removes an O(n) scan and can make threading completely unnecessary. That’s exactly why you start at the top of the sequence and not the bottom.

Pitfall 3: Treating Optimization as a One-Time Event

Performance decays quietly as a codebase grows. A script that ran in two seconds last year now takes twenty because fifty new features landed on top of it with nobody watching the clock. Treat profiling as a regular habit rather than an emergency response, and you’ll catch a slowdown while it’s still small.