Run External Program with Subprocess in Python

There are many scenarios where you want to write a small programs that are designed to automate simple tasks. For instance, you want to resize thousands of images in your computer to a smaller sizes, or send email to multiple user whenever a row of data added into a csv file.

This is what people usually called scripting. And Python is a powerful programming language for this kind of job. Because, there are a bunch of built-in modules and third-party libraries that helps you to do some cool automation stuff with Python.

One of the problems you might encounter when you want to automate stuff is running other software that written in other programming languages. It could be the system software that built-in to your operating system. Or, it could be a third-party software that you have installed from the internet.

In effect, Python offers a built-in module called subprocess. It’s a Python Standard Library that used to spawn new processes (programs), read the input/output/error, and obtain their return codes.

Getting Started

One of the very basic usage of subprocess module is by using the run method. This method allows you to run external program by passing the command as a string inside the first argument.

In this example, we’re going to execute the ls command, which is a command that available in Unix operating systems (MacOS, Linux, etc.) to print all files and folders inside current directory.

import subprocess

subprocess.run('ls')

The output will be:

$ python demo.py
demo.py		filename.txt

Running shell command

If you are on windows, and you are trying to run commands like dir, you will get an error. It’s because that command is built-in to the shell. So, to fix it, you can add the shell=True argument to the run method like below:

subprocess.run('ls', shell=True)

Adding the shell=True argument also allows you to run your entire command as a string. Because, if you dont enable the shell argument, you actually can’t pass the entire command as a string like we did. Instead, we need to pass our command as a list of arguments.

subprocess.run(['ls', '-la'])

Capturing output

If you pay attention to the console, you will see that the command result is automatically printed out to the console. Even if we don’t use print function by ourself. That’s because the subprocess.run method doesn’t capture our command output by default.

To prove this, we can try to capture our command by putting it into a variable.

result = subprocess.run('ls')
print(result)

You will see something like this:

demo.py		filename.txt
CompletedProcess(args='ls', returncode=0)

First of all, the interpreter execute the subprocess.run method, run the ls command, and print the output to the terminal. Then, we try to print the value inside the result variable and get this weird CompletedProcess object. What’s going on here?

As I said, the subprocess.run method doesn’t return the process ouput. It just print the result to the console. One way to get the process output is by adding the capture_output=True named argument, and access the stdout property of the result.

result = subprocess.run('ls', capture_output=True)
print(result.stdout)

Output:

demo.py		filename.txt

Now, instead of printing out the output directly to the terminal, the run method return the output to us, so we can grab it and print it out the the terminal by ourself.

Decode Bytes to String

There is still a problem though. If you check the type of our result.stdout, you can see that the type is not string, it’s bytes. So what’s the difference?

result = subprocess.run('ls', capture_output=True)
print(type(result.stdout))

Output:

<class 'bytes'>

Well, the big difference of bytes and string in Python is that bytes is immutable / cannot be modified. Bytes object are machine readable, in other hand, strings are human readable.

To convert bytes object into strings, we can use the decode method from the bytes object.

result = subprocess.run('ls', capture_output=True)
print(result.stdout.decode())

Or, we can add text=True parameter to our run method. This makes it automatically decode our output into a string.

result = subprocess.run('ls', capture_output=True, text=True)
print(result.stdout)

Checking Errors

To make sure that your command runs properly without any error, you can check the return code of your process. Usually, if the return code is 0, your code run successfully. Otherwise, if the return code is 1, something went wrong.

For example, if we want pass the a wrong argument to the program we want to run, it will throw an error like below.

# notexist folder is not exist!
result = subprocess.run(['ls', 'notexist'])

The output:

ls: notexist: No such file or directory

To make sure if the program doesn’t throws an error, we can check the return code like below.

if result.returncode == 0:
    # Do something
else:
    print("Something went wrong!")

Also, the result comes with the stderr property. Use it if you want to see the error message.

if result.returncode == 0:
    # Do something
else:
    print(result.stderr)

Tags: Python, Subprocess