Join us and get access to thousands of tutorials and a community of expert Pythonistas.
This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.
In this lesson, you’ll learn what a semaphore is and see an example program that uses semaphores to protect a limited resource. If you download the sample code, you can get your own copy of 14-semaphore.py:
To learn more, you can also check out the documentation.
00:00
Congratulations on creating a full-blown message queueing system that is thread-safe and efficient. And congrats on making it this far in the course! Before you leave, I just want to share a few more things that you might come across when you’re working with threads and that I think are important to at least have a general understanding of. One of them is a Semaphore.
00:22
The Semaphore is created from the threading module. Once you import threading, you can create a Semaphore from threading.Semaphore().
00:33
It’s going to give you a Semaphore object.
00:37
Let’s go ahead and print the dir() representation of Semaphore and see what this offers us.
00:44 Open up a terminal and execute our program,
00:49
and we’ll see the dunder methods related to this Semaphore object. You see that it has a 'release' and an 'acquire', which looks very familiar, similar to a Lock.
00:58
And it also has this '_value' and '_cond' attribute. We know that Semaphore can be acquired and released, but one of the greatest use cases of a Semaphore is as an atomic counter—atomic meaning that it’s guaranteed the CPU won’t execute any code in between the time it takes to increment the counter or decrement the counter.
01:23
Let’s give an example to demonstrate what that means. We’ll create a new sem and this will be a threading.Semaphore and this time we’ll specify the value.
01:34
We’ll say value=50. I’m going to delete the first sem. And then let’s just print(sem._value).
01:48
Clear the screen, and we’ll run the program and we see that it’s 50. So now, we know that the sem also has an .acquire() and .release() method.
01:58
Let’s do sem.acquire(). Let’s do that three times and then we’ll print the ._value after those three acquires.
02:09
You probably guessed that the ._value was decremented by 3, and it’s now 47. And if we do a sem.release(),
02:20
then you can probably guess that the value becomes 48. The value here with Semaphore is that it’s atomic, and this is really important when you’re dealing with limited resources—for example, like a connection pool in a database. With limited resources, you want to be sure that only a certain number of machines can access those resources.
02:46
I wrote a little program that uses a Semaphore that I want to share with you as an example of how you might use this. I’m going to go ahead and drop down and paste some code in here.
02:59
A lot of it is going to be very familiar to you. We have an entry point down here with a max number of users of 50. Imagine we have a website and we have a little welcome() function that welcomes the visitor, and it keeps track of the visitor_number.
03:17
We’re defining our max number of users to be 50, so our website can only handle 50 users. Our Semaphore starts with a value of 50 and then we fire off two threads using a ThreadPoolExecutor.
03:34
We have a monitor thread which keeps track of the number of users that are on the website. And it prints out the current ._value of the semaphore.
03:44
If the max number of users is reached, then the monitor() method will kick a user out, and what that means is it will release one of the semaphores and then it will allow one more user in.
04:00 I’m going to go ahead and open up the terminal and run this program.
04:07
And we’ll see it’s welcoming visitor #0. And then the semaphore is actually 49, so once we got the first visitor, the semaphore was decremented. And visitors are coming in faster than the monitor() function is printing this display. We get a bunch of visitors, we see the semaphore starts to decrease and decrease. As these visitors keep rolling in, the semaphore is becoming smaller and smaller. We’re closely approaching our capacity of 50 here. So, we’ve reached 50, and then the monitor says we reached the max users, kicking a user out..., so semaphore is 0.
04:45 Another visitor comes, and then we have our max again. We’re at our max capacity, so at this point in our program only one visitor is coming in at a time. We’re kicking one out so we have space for another one.
04:58 You can use your imagination for other use cases of semaphores, but the key value of a semaphore, rather than just a regular plus/minus is that it’s atomic, so it’s guaranteed that nothing in between the time it takes to increment or decrement is going to be executed by the CPU.
Lee RP Team on Dec. 17, 2019
Hey philipwong37,
You may want to look at threading.Barrier which can let you set a barrier so the threads return in the order you want.
philipwong37 on Dec. 17, 2019
Thanks. I was able to trace down to lowest level of the dictionary and do
mydict[key1][key2] = [futures.result() for futures in concurrent.futures.as_completed(mydict)]
the mydict stored all result from the for-loop in the multi-thread.
and put back to the correct keys of the dictionary. However, I have another issue. I have noticed and cannot use multi-threads along different part of the process. The memory will not release after doing multi-thread, and technically I have to find the part of the process that is essential to use multi-thread and I can only use it once?
Become a Member to join the conversation.
philipwong37 on Dec. 13, 2019
I am using the concurrent.future to run multi-threads within a for loop and i am able to get the result back (pandas dataframes using future.result()); however, the result return was not in order of of the for-loop above I will need to put the dataframes back into a Python dictionary (two-level keys) the same order as the for-loop above. How do I achieve that? using queue and event ..etc? Thanks. the for-loop could be 1 x 3 or 4 x 3 …etc