Creating Pipes With Popen

00:00 Earlier in the course, you learned about pipes and how to simulate a pipe using run(). Now, we did cover back then that this was not a real pipe, but the simulation of a pipe.

00:12 To create a real pipe, we will need to use Popen(). So let’s move on to the Popen class.

00:20 Now, Popen firstly stands for pipe open. Earlier in the course, you also learned that Popen is the underlying class, for the whole subprocess module.

00:31 So what that means is that all the functions are convenience wrappers around the Popen() constructor or the Popen class. And you should use run() for all cases it can handle, that’s what the documentation says.

00:45 But there might be situations that are a bit more complex, a bit more edgy for which you will use Popen. And creating real pipes is one of those situations.

00:55 Now run() is based on Popen. run() again is a convenience wrapper around the Popen() constructor or the Popen class.

01:03 So using run() and using Popen, that’s very similar, but there are some key differences that you will discover in this video. Now firstly, let’s start by refreshing your memory on how to simulate a pipe using run().

01:20 This is the code that we went through before and you’ll remember the simulation of the pipe by setting the input of the grep process to be the output or the stdout of the ls process.

01:35 Now let’s have a look at the code using Popen and what the key differences are. So in the second screen here, this is the code using Popen() and just for convenience, I’m leaving the code using run() visible at the top.

01:50 So instead of run(), we will use Popen(). Everything else here is the same, but for the grep process, again, we’re going to use Popen().

02:00 And here is a key difference. Instead of saying input and assigning the stdout of the ls process to input, we are assigning it to stdin.

02:13 Now that is quite a difference. The rest of the line is the same. Now why is that a difference? Because the stdin here expects a stream as an input and not a bytes object.

02:25 This touches on a few key differences here and let’s take a step back.

02:30 So the first key difference we spotted is that Popen() actually is written with a capital whereas run() is not.

02:37 The second piece is that Popen() is not a blocking call. Now, earlier in the course when we talked about run(), it was said that run() is a blocking call.

02:48 So what that means is that run() waits for the process to be finished before the program continues. That is not the case with Popen().

02:58 With Popen(), the two processes are running in parallel. So here we have the ls process and the grep process. They are running in parallel. So that is a key difference.

03:09 The other key difference is that the run() function returned a completed process, whereas the Popen() constructor returns a Popen object and those are very different objects.

03:22 Now, the Popen object has as its stdout a stream, an actual stream, whereas run(), which returns a CompletedProcess object, the stdout there is a bytes object as you have seen earlier in the course.

03:39 So now because the output, the stdout, is not a bytes object but a stream, you’ll need a loop to read the stdout while the process is running.

03:50 And that is what we are doing here. This is quite a crucial difference because the fact that your stdout is an actual stream, and the fact that the processes are running in parallel, that means that you can communicate between the processes while they are running and that is very different from run().

04:10 And that is also why you can create an actual pipe as opposed to simulate a pipe.

04:17 So all that’s left to do now is to run the code and see what it does. Press Enter, press Enter again, and I get these four lines. Remember, these are filenames that have the word python in it, and just to prove that that’s what was happening here as well.

04:33 There you go. That is the simulation of the pipe giving us the same output.

04:40 Okay, let’s just summarize the differences between run() and Popen(). Firstly, run() is a blocking call, whereas Popen() is a non-blocking call.

04:49 So run() will wait for the process to finish before the program continues, whereas Popen() will have processes run in parallel.

04:58 run() returns a CompletedProcess object of which stdout is a bytes object, whereas Popen() returns a Popen object of which the stdout and the standard stream attributes are actual streams and therefore with run(), you can only simulate a pipe.

05:17 But with Popen you can create actual pipes.

05:23 As a final point, the use of subprocess with actual pipes and parallel processes can become quite complex. So if you are considering complex use of subprocess that goes beyond run(), there is a module called asyncio and I will include a link to an excellent tutorial and that is a module that will help you along your journey.

05:47 That’s it for introduction to Popen(). Now it’s time to gather your thoughts and think about all you have learned during this course. The course summary is up next.