How to Execute Shell Commands in Python Using the Subprocess Run Method

Subprocess is a built-in Python module that can be used to create new processes and interact with their input and output data streams. In simpler terms, you can use it to run shell commands and run executable binaries usually scattered in various “bin” folders across a Linux file system. You can also supply a full path to an executable binary and use any command-line switches associated with the binary. This article will explain how to use the subprocess module and its run method in Python apps. All code samples in the article are tested with Python 3.8.2 on Ubuntu 20.04.

The Subprocess.run Method

The Subprocess.run method takes a list of arguments. When the method is called, it executes the command and waits for the process to finish, returning a “CompletedProcess” object in the end. The “CompletedProcess” object returns stdout, stderr, original arguments used while calling the method, and a return code. Stdout refers to the data stream produced by the command, while stderr refers to any errors raised during execution of the program. Any non-zero return code (exit code) would mean error with the command executed in the subprocess.run method.

Example 1: Output Contents of A Text File Using the Subprocess.run Method

The command below will output the contents of a “data.txt” file, assuming that it contains a “name=John” string.

import subprocess
subprocess.run(["cat", "data.txt"])

Running the code above will return the following output:

name=John
CompletedProcess(args=['cat', 'data.txt'], returncode=0)

The first element of the list argument is the name of the command to be executed. Any element in the list that follows the first element are considered command-line options or switches. You can use single dash and double dashes, as well, to define the options. For example, to list files and folders in a directory, the code would be “subprocess.run([“ls”, “-l”]”. In most of these cases, you can consider any space-separated argument in a shell command as an individual element in the list supplied to the subprocess.run method.

Example 2: Suppress Output of Subprocess.run Method

To suppress the output of the subprocess.run method, you will have to supply “stdout=subprocess.DEVNULL” and “stderr=subprocess.DEVNULL” as additional arguments.

import subprocess
subprocess.run(["cat", "data.txt"], stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL)

Running the code above will produce the following output:

CompletedProcess(args=['cat', 'data.txt'], returncode=0)

Example 3: Capture Output of Subprocess.run Method

To capture the output of the subprocess.run method, use an additional argument named “capture_output=True”.

import subprocess
output = subprocess.run(["cat", "data.txt"], capture_output=True)
print (output)

Running the code above will produce the following output:

CompletedProcess(args=['cat', 'data.txt'], returncode=0,
stdout=b'name=John\n', stderr=b")

You can individually access stdout and stderr values by using “output.stdout” and “output.stderr” methods. The output is produced as a byte sequence. To get a string as output, use “output.stdout.decode(“utf-8”)” method. You can also supply “text=True” as an extra argument to the subprocess.run call to get the output in string format. To get exit status code, you can use the “output.returncode” method.

Example 4: Raise Exception on Failure of Command Executed by Subprocess.run Method

To raise an exception when the command exits with a non-zero status, use the “check=True” argument.

import subprocess
subprocess.run(["cat", "data.tx"], capture_output=True, text=True, check=True)

Running the code above will produce the following output:

raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cat', 'data.tx']'
returned non-zero exit status 1.

Example 5: Pass a String to Command Executed by the Subprocess.run Method

You can pass a string to the command to be executed by subprocess.run method by using “input='string'” argument.

import subprocess
output = subprocess.run(["cat"], input="data.txt", capture_output=True,
text=True, check=True)
print (output)

Running the code above will produce the following output:

CompletedProcess(args=['cat'], returncode=0, stdout="data.txt", stderr=")

As you can see, the code above passes “data.txt” as a string and not as a file object. To pass “data.txt” as a file, use the “stdin” argument.

with open("data.txt") as f:
output = subprocess.run(["cat"], stdin=f, capture_output=True,
text=True, check=True)
print (output)

Running the code above will produce the following output:

CompletedProcess(args=['cat'], returncode=0, stdout="name=John\n", stderr=")

Example 6: Execute Command Directly in Shell Using the Subprocess.run Method

It is possible to run a command directly into a shell “as is,” instead of using a string split in the main command and the options that follow it. To do this, you must pass “shell=True” as an additional argument. This is however, discouraged by python developers as using “shell=True” can lead to security issues. You can read more about security implications from here.

import subprocess
subprocess.run("cat 'data.txt'", shell=True)

Running the code above will produce the following output:

name=John

Conclusion

The subprocess.run method in Python is pretty powerful, as it allows you to run shell commands within python itself. This helps in limiting all code to python itself without the need to have additional shell script code in separate files. It can be, however, quite tricky to correctly tokenize shell commands in a python list. You can use the “shlex.split()” method to tokenize simple shell commands, but in long, complex commands - especially those with pipe symbols - shlex fails to correctly split the command. In such cases, debugging can be a tricky issue. You can use the “shell=True” argument to avoid this, but there are certain security concerns associated with this action.