Avoiding Deadlocks With RLock

00:00 In the previous lesson, I showed you how to make a block of code atomic by using Python’s lock object. In this lesson, I’ll cover some of the complexities resulting from multiple locks and what to do about it.

00:13 A deadlock happens when two or more entities are waiting for each other. This causes your program to freeze up or at least the threads in the deadlock to block.

00:22 It’s like if you and I are standing in front of a door, both saying after you and both insisting, we’ll stand there forever until the other person goes first.

00:31 That’s a deadlock. Deadlocks can be tricky beasts. Sometimes, the deadlock only happens as a result of a race condition, which means they can be hard to debug.

00:40 There are a few things that make it more likely you’ll get a deadlock. The first is nested locks. That’s if your thread tries to acquire the same lock twice.

00:50 On the second call to .acquire(), you block waiting for yourself to call release, which of course if you’re blocked, you can never do. Deadlock can also happen when dealing with multiple locks.

01:01 If you have to have multiple locks, you need to keep the acquisition order consistent to avoid this problem. This can be hard to suss out sometimes, as you could have A waiting for B, waiting for C, waiting for A, and of course that alphabet chain could be really long.

01:19 A variation on the lock is the reentrant lock. A reentrant lock allows the same thread to acquire it multiple times, which solves the nesting problem. Internally, this kind of lock keeps a counter.

01:31 Each time you acquire the lock, the counter goes up and each time you release the lock, the counter goes down. Other threads acquiring the lock stay blocked until the counter is zero.

01:41 The threading library in Python has a reentrant lock called RLock. Let’s modify our banking service fee example a little to see the differences between a lock and an RLock.

01:54 In this version of the problem, I’m going to construct either a Lock or an RLock based on the argument passed in. Here, I’m grabbing the command-line argument.

02:03 Inside the constructor, I look for a command-line argument of "R". If so, I construct an RLock; otherwise, the regular kind.

02:14 To demonstrate the difference between the locks, I need to change the code a little so that the lock gets acquired multiple times. To see this happen, I’ve added some debug information.

02:24 The threading library’s current_thread() function returns a thread object for the thread being executed. The .name attribute of the thread is an automatically generated value unique to the thread.

02:37 Once I’ve acquired the lock using the context manager like before, the first thing I do is call some more debug to say that the lock got acquired.

02:46 This is the reentrant part. Instead of updating the balance in a single line here, now I’m calling a method instead, which I’ll show you in just a second.

02:56 Then down at the end of withdraw(), I’ve got some more debug. Let me go down to that update method now.

03:08 Then this is the update method. It also starts with some debug. Then it acquires the lock, and once locked, it updates the balance. That’s pretty much all I need in order to test this out.

03:19 But before moving on, I’ve made one more change and it’s a safety thing. Let me go down to the bottom.

03:27 Inside the ThreadPoolExecutor, you create a thread by calling submit(). This spawns the thread. If you don’t care when the thread finishes and don’t need to interact with that thread, you’re fine.

03:38 Well, sort of fine. There’s one problem. If there’s an exception on your thread, it doesn’t get reported, it just gets swallowed. When I wrote this example, I had a copy and paste error in the update call.

03:50 Because it was being called on a thread, the syntax error that got raised was swallowed by the thread. The code appeared to run fine, it just didn’t do what it was supposed to.

03:59 After pulling some hair out, I realized what had gone wrong and fixed it. The fix involves using the return from submit(). I’ve called it future.

04:10 This is a bit of concurrency terminology. It means that at some time in the future, a value will be available here. If your thread was doing a computation and returning a result, you’d need this object in order to get the result. In the examples so far, I haven’t done that, but by using this I can deal with the exception problem.

04:30 You get the result from the thread by calling result() on the future. This is a blocking call, so it is a transition boundary between your asynchronous code and the synchronous part of the main thread.

04:41 Although I don’t need a result in our case, calling result() has another benefit. If the thread finished due to an exception, that same exception gets raised during the result() call.

04:52 If I’d had these two extra lines, I wouldn’t have needed to pull out my hair because I would’ve seen the syntax error immediately.

04:58 Got it? Good. Best practice is always use the result() call, even if you don’t care what the result is. It will save you loads of pain down the road, and possibly stop you from going bald.

05:09 Alright, there’s our code. Let’s try this out.

05:14 First, I’ll run using the normal lock.

05:19 If you’re waiting for something specific, you’re going to be waiting for a long time. While I’m sitting here, let’s talk about the four lines of debug. The first line of debug says that the thread named 0_0 is acquiring the lock.

05:33 The second line says thread 0_0 has got the lock. The third line says thread 0_1 is waiting on the lock. That’s fine. That’s what it’s supposed to do, as the code is currently in the atomic block.

05:46 Atomic block sounds like a good name for a punk band. The fourth line is where the problem is. Thread 0_0 is attempting to acquire the lock.

05:54 Again, it’s going to wait here until the lock is released, which can’t happen because the thread it’s waiting for is the thread that’s responsible for releasing it.

06:04 Let me just kill this. As a quick side note, you may have to hit Ctrl-C multiple times with your code. The kill signal doesn’t necessarily kill the whole program.

06:13 It may only interfere with the blocked thread first. Let’s try this code again, this time using the reentrant lock.

06:26 Wow, that’s a lot of debug. Since there are 50 accounts, and then each gets the fee charged and then reimbursed, each lock is acquired and released four times: twice in withdraw(), including the nested call, and twice in deposit() for a nested call there as well.

06:42 Definitely too much debug, but if you look at the last part of it, you can see that reentrant lock is doing its job.

06:50 A lock, reentrant or otherwise, is either on or off.

06:55 Sometimes you want looser grain control than that. In the next lesson, I’ll cover semaphores.