Join us and get access to thousands of tutorials and a community of expert Pythonistas.
This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.
Choosing a Queue
A queue is similar to a stack in that is an ordered collection of objects. The order is different though, it is First-In/First-Out (FIFO). This is like a line-up at a bank. For single threaded Python programs you can use either the built-in list type or collections.deque for a FIFO queue. For concurrent code, use either the queue.Queue or multiprocessing.Queue objects for multi-threaded or multi-processing, respectively.
Here are resources and additional documentation about queue, multiprocessing, concurrency, and pickle:
- collections – Container Datatypes: deque objects | Python Documentation
- queue – A synchronized queue class: Queue | Python Documentation
- multiprocessing – Process-based parallelism: Queue | Python Documentation
- What Is the Python Global Interpreter Lock (GIL) | Real Python Article
- Speed Up Python With Concurrency | Real Python Course
- The Python pickle Module: How to Persist Objects in Python | Real Python Article
00:00 In the previous lesson, I talked about queues in threaded and multiprocessor environments. In this lesson, I’ll wrap up the section on queues and talk about how to choose a queue library.
00:12
The first factor in choosing an implementation of a queue is to determine whether or not you are doing concurrent code. If not, use the deque object in the collections library, unless you’re just doing very small amounts of data—then you can get away with a list.
00:27
If you are being brave and going down the whole concurrency road, your choice is bound by what kind of parallelism you are implementing. With threading, use the queue library. With multiprocessing, use the multiprocessing library.
00:41 That’s a pretty straightforward choice.
00:44
As always, a good source of info is the Python documentation. Here’s the collections library for the deque object, the thread-safe version of a queue, and the multiprocessing.Queue.
00:58 If you’re new to concurrency or coming from another programming language, understanding the GIL and its impact on parallel code in Python is important. This article can point you in the right direction.
01:09
If you’re using the multiprocessing.Queue, you’ll need to make sure whatever you’re sending between processes is pickleable. This article introduces you to Python’s pickle.
01:19 And if you’re after more understanding of threads, async I/O, multiprocessing, and how to use them to get faster, more responsive code, this course introduces all those concepts.
Become a Member to join the conversation.