Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

This lesson is for members only. Join us and get access to hundreds of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Thread Pool

Give Feedback

In this lesson, you’ll refactor your previous code by using a thread pool executor from the concurrent.futures module. If you download the sample code, you can get your own copy of 07-thread_pool.py:

Download

Sample Code (.zip)

12.9 KB

To learn more, you can also check out the documentation for concurrent.futures.ThreadPoolExecutor and concurrent.futures.Executor.map.

Comments & Discussion

nightfury on Dec. 26, 2019

Hi,

Would this be a cool way to find the number of threads supported by an OS ? Out of my curiosity I modified the code a bit to see if it crashes beyond certain point

import threading
import time
import concurrent.futures

def my_func(name):
    print(f'my_func started with {name}')
    time.sleep(5)
    print(f'my_func ended with {name}')

if __name__ == '__main__':
    max_workers = 5000
    print('Main started')

    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as e:
        arg_list = ['func_' + str(i+1) for i in range(0,max_workers)]
        e.map(my_func,arg_list)

    print('Main ended')

On my system (running mac OS) , the code crashes beyond max_workers=2048 with the following error

RuntimeError: can't start new thread

Any comments on whats happening here ?

Lee RP Team on Jan. 12, 2020

Hey @nightfury, that’s cool :) I imagine there is a limit of number of threads per process the OS can create. Looks like its 2048 on yours. I just ran it on my machine and it stopped at 4096.

Become a Member to join the conversation.