Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

This lesson is for members only. Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Hint: You can adjust the default video playback speed in your account settings.
Hint: You can set the default subtitles language in your account settings.
Sorry! Looks like there’s an issue with video playback 🙁 This might be due to a temporary outage or because of a configuration issue with your browser. Please see our video player troubleshooting guide to resolve the issue.

Race Conditions

00:00 In the previous lesson, I introduced you to the threading library in Python. In this lesson, I’m going to take a little side journey and show you what happens when race conditions complicate your concurrent program.

00:12 I have a little bit of a confession to make. I actually went to school for a lot of this work. My research in grad school was on concurrency and parallelism and distributed computing. And it’s been many, many years since I did that, and the one thing I still remember is concurrency is hard.

00:30 You should always choose carefully whether or not it’s the right thing to do. It introduces all sorts of complications. In an earlier lesson, I went through list including things like deadlocks and resource starvation that can be quite complicated.

00:44 Race conditions are related to deadlock, in that the behavior of the program changes depending on the order of execution. This is particularly noticeable when two threads share the same memory and objects.

00:56 The nature of threaded computation means you can’t predict what order the code is going to run in, and depending on how those shared objects are used, it can affect the outcome.

01:06 Locking mechanisms and threading.local() in Python can help you, but you have to remember when to use them, when things are local and when they don’t need to be local. A surprising number of things that are common tools in the programmer’s tool belt are not thread-safe.

01:21 You’ve already seen the example of a requests.Session(). The requests library itself is not thread-safe. If you’re doing multi-threaded programming with the requests library, you need to make sure that you’ve got locks in place.

01:34 Even something as fundamental as print() isn’t thread-safe. It doesn’t happen often, but it is possible for two different threads to print at the same time, and you end up with a print buffer partially populated from the first thread, populated by the second thread, and then populated by the first thread, which ends up being a mess on the screen.

01:54 If you’ve ever done any server-side web programming, you may notice that logging is the way to go. Python’s logger is actually thread-safe, and so multiple threads in a web server don’t cause this problem.

02:05 Unfortunately, it does this through locking, which slows things down. And this is the compromise you always have to do with concurrency. The speed-up is there, but it’s often done at the expense of thread safety.

02:18 The safest thing to do is assume that the code you are using is not thread-safe, unless it explicitly says that it is.

02:28 The program I’m about to show you has a race condition in it. It’s also what we in the business call “a convoluted example.” You’ll never see code that actually looks like this, but it does show off the fact that the race condition can happen.

02:42 Race conditions are heisenbugs, the kinds of bugs that often disappear when you’re looking for them. Things are timed correctly, the bug doesn’t happen. Things are timed incorrectly, and the bug does happen.

02:53 And the timing is all related to how the operating system and how Python schedules the threads and how long each thread is asked to execute. Race conditions can be brutal to try and track down.

03:07 The heart of this program is this function at line 10. The function named race() takes a number of threads, and I’m going to call it starting with one thread and then a few more, and you’ll see how it changes based on that.

03:21 The key part of this function is messy line number 13. This list comprehension generates a list of numbers that alternate between -1 and 1. There’ll be 1000 numbers in it, 500 of which are -1 and 500 of which are 1.

03:39 The function that will be run in parallel on the threads starts at line 5, and this is the change_counter(). This uses a global variable called counter, and all it does is 10000 times add to the global counter the number passed in. What’s going to be passed in will either be a -1 or a 1. If this code is run synchronously, the sum total of -1 and 1 should be 0, no matter how many times you do it. Line 15 defines the thread pool,

04:11 and the mapping happens between the data, which is the list that alternates between -1 and 1, and the change_counter() that adds that amount to the global.

04:22 The max_workers parameter of the thread pool indicates how many threads to run. If this is called with one thread, the program is synchronous.

04:31 If this is called with more than one thread, the program will be concurrent.

04:37 Let’s see this in practice. Import the function,

04:42 and I’m going to start by calling the function with 1 thread.

04:47 Since this is synchronous, the race condition doesn’t happen yet. A -1 is added to the counter 10,000 times, then 1 is added to the counter 10,000 times, and it does that for 998 more times, the end result being 0. Now let me try this again with a few more threads.

05:09 Mmm. That’s not 0. And this is the problem. From a mathematics standpoint, the reason this is failing is line 7. You can think of that for loop like a multiplier. In the synchronous case, -1 is multiplied by 10,000 and added, and then 1 is multiplied by 10,000 and added, and you result in 0. In the threaded case, you get partway through that multiplication.

05:36 So instead of it being 10,000 times -1, it might be 500 times -1, and then the other case comes in. That might be 600 times 1, giving you a delta.

05:49 The sum total of the multiplication factors is not the same as the synchronous addition, so you get a race condition. You get the wrong number. Not only is it the wrong number, it’s not even consistently the wrong number.

06:04 If I run this again, I get a different result. And this is because the scheduling changes how things are happening. The effect of the multiplier in line 7 is different this time through because the amount of time each thread is run is different, and you get a different number. Third times the charm.

06:26 Well, still not working. This is an exaggerated example. I’ve done it on purpose to show you what happens. Race conditions generally are far more subtle than this, and it would be far harder to find the problem if one in a hundred executions actually caused the bug to happen, and that’s a common result in race conditions. It makes them nasty to find.

06:50 I’d like to tell you that asyncio solves the race condition problem. Unfortunately, it doesn’t, but it is another way of programming concurrency in Python and is of interest.

hauntarl on Jan. 5, 2021

from pprint import pprint
from concurrent.futures import ThreadPoolExecutor
import threading

counter = 0
thread_iter = dict()

def change_counter(amount):
    global counter
    # to keep track of all the threads and total iterations each one is doing
    # along with the amount value that thread has been assigned
    if (threading.get_ident(), amount) not in thread_iter:
        thread_iter[(threading.get_ident(), amount)] = 0

    for _ in range(10000):
        thread_iter[(threading.get_ident(), amount)] += 1
        counter += amount

def race(num_threads):
    global counter
    counter = 0
    data = [-1 if x % 2 else 1 for x in range(1000)]

    # when the num_threads value is 1, the program is synchronous
    with ThreadPoolExecutor(max_workers=num_threads) as executor:
        executor.map(change_counter, data)

    pprint(thread_iter)
    print(f'counter value(ideally should be 0): {counter}')
    print(f'total iterations (1,000 x 10,000): {sum(thread_iter.values())}')

    delta = 0
    for k, v in thread_iter.items():
        delta += v if k[1] > 0 else -v
    print(f'delta (positive iterations - negative iterations): {delta}')

if __name__ == '__main__':
    # race(1)
    race(5)  # increase num_threads to consistently achieve race condition

"""
OUTPUT: 
{(1272, -1): 960000,
 (1272, 1): 1020000,
 (5704, -1): 940000,
 (5704, 1): 930000,
 (7960, -1): 1000000,
 (7960, 1): 1050000,
 (12428, -1): 1000000,
 (12428, 1): 900000,
 (12800, -1): 1100000,
 (12800, 1): 1100000}
counter value(ideally should be 0): 111490
total iterations (1,000 x 10,000): 10000000
delta (positive iterations - negative iterations): 0
"""

I tried to keep track of each thread, what amount it is using and how many iterations it is doing with that amount.

From the above output, it is clearly visible that each thread is doing different number of iterations for different amount value, but if we take the delta for iterations done for amount = 1 and amount = -1, it comes out as zero. Yet the counter value comes out as some arbitrary number instead of 0.

I am not able to make sense of this output :(

Also I am finding it difficult to walk through the program and understand exactly how the race condition is occurring.

Christopher Trudeau RP Team on Jan. 5, 2021

Hi hauntarl,

The race condition is a caused by a combination of two things: the variable counter being global and shared across all threads, and the fact that the change_counter() method can be interrupted at any time.

The crux of the matter is the assumption that “counter += amount” can’t be interrupted. If you put a semaphore lock around “counter += amount” the code would likely work fine.

I suspect what is happening is that “counter += amount” breaks down into several different instructions in the underlying virtual machine and occasionally it is being interrupted midway through those instructions.

The code you added to try to show the problem happening is only recording the state outside of the problem area: after the increment/decrement is done. To truly instrument the problem you would have to be able to inspect the value of “counter” during the “counter += amount” statement.

The reason I used two large ranges (once in the “data” variable, the second in the for-loop in change_counter() ) was to increase the likelihood of just such an interruption. If you drop those numbers down to lower values you might have to run the program a large number of times before you run into a problem – this is what makes race conditions so brutal to catch in real life.

hauntarl on Jan. 6, 2021

Thanks for the wonderful explanation, I think I finally get it.

If we try to breakdown the statement counter += amount in a 2 threaded system:

  • evaluation of expression counter + amount: (Read operation)

    thread1: counter = 0, amount = 1, evaluation results in 1

    thread2: counter = 0, amount = -1, evaluation results in -1

As there is no mutual exclusion, both the threads simultaneously read the same “counter” and are trying to add “amount” to it.

This here is a race condition as no thread is willing to wait for the other one to update the counter value after its evaluation, which results in one of them using the old counter value depending on which thread is able to perform write operation first.

  • assignment operation counter = evaluated value: (Write operation)

    thread1: counter = 1

    thread2: counter = -1

Let us assume that “thread1” is able to perform the write operation before “thread2”, the new “counter” value becomes 1, soon after “thread2” performs its own write operation and writes -1 to the “counter” Finally the counter value results in -1, instead of 0 :)

Correct me if I am wrong and feel free to provide more input.

Christopher Trudeau RP Team on Jan. 6, 2021

Hi hauntarl,

Yep, you’ve got the right idea. The specifics are slightly different, but conceptually you’ve got it.

Remember that Python is an interpreted language. Statements in Python get translated into a series of steps called byte code, it is the byte code that actually gets run by the interpreter. Why do I bring this up? Because even things that look like single statements in Python aren’t necessarily atomic.

You can see the bytes that are run by the interpreter using the “compile()” built-in and the “dis” library:

>>> counter = 0
>>> amount = 1
>>> code = compile("counter += amount", "<string>", "exec")
>>> code
<code object <module> at 0x7fe81635ec90, file "<string>", line 1>
>>> import dis
>>> dis.disco(code)
  1           0 LOAD_NAME                0 (counter)
              2 LOAD_NAME                1 (amount)
              4 INPLACE_ADD
              6 STORE_NAME               0 (counter)
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

Notice here that “counter += amount” turns into 6 separate byte code operations, and the thread interruption can happen during any of these steps.

The problem is the same as you described, just worse – there are five different places that this thing that looks like a single statement can get interrupted. This only gets worse for any larger chunk of code.

Become a Member to join the conversation.