This article will show you how to use the Python subprocess module to run external programs and scripts from Python.
It is often necessary to call external applications from Python. Usually these are command-line applications which you can use to perform tasks outside of the Python environment, like manipulating files, or interacting with third party services. For example, you might call the wget command to retrieve a remote file. Any application which is accessible on the command line is available to subprocess, and can be executed from within Python.
You can also call other Python scripts, and even Bash and other shell scripts using subprocess.
Python 3 Only!
The subprocess.run function is only available in Python 3 – and is designed to replace the os.system and os.spawn modules from Python 2.
Run External Program Using subprocess.run
The subprocess.run function is used to execute external programs from within Python. It is part of the subprocess library which is included in all Python 3 installations. Any program available on the command line can be executed using subprocess.
The subprocess.run function accepts a list of arguments. These are the commands which will be executed on the command line. Each argument in the list will be joined to form the final command line command which will be executed.
Below, the uname command is called which prints information about the system. The -a flag is passed as an additional argument, this tells uname to print additional information about the system.
import subprocess subprocess.run(["uname", "-a"])
The output of the executed program will be printed to the console, but nothing will be done with the output (yet!).
Handling Output
The output from an executed program can be captured by assigning it to variable.
Then, the results of the standard pipes of the output and error (STDOUT and STDERR) can be accessed.
import subprocess
output = subprocess.run(["uname", "-a"], capture_output=True, text=True)
print("STDOUT: ", output.stdout)
print("STDERR: ", output.stderr)
The return value of subprocess.run is a CompletedProcess object that can be assigned to a Python variable.
By setting capture_output, the output is returned as part of this object. The output is usually sent as a byte sequence, but can be returned as a string by enabling the text argument.
Output from a successful execution will be stored in the stdout attribute of the resulting CompletedProcess object. If the external program or script encounters an error, stderr will store the error response. If there is no error, the stderr attribute will have a value of None.
import subprocess
output = subprocess.run(["uname", "-xyz"], capture_output=True, text=True)
print("STDOUT: ", output.stdout)
print("STDERR: ", output.stderr)
Above, the -xyz argument is an invalid argument for the uname command so an error is generated, the output of which is stored in the stderr attribute of the subprocess.run output.
Note that when output is captured, it is not printed to the console!
Handling Errors & Exit Codes
If an error has occurred in the program called by subprocess.run, even if the output is captured, it will not cause an error to be thrown within Python, and the Python code following it will continue executing. This is fine for some use cases, but often you will want to deal with the error – for example by retrying or alerting the user.
If you want Python to stop and throw an error on a failure of the external program, the check attribute can be set.
import subprocess output = subprocess.run(["uname", "-xyz"], capture_output=True, text=True, check=True)
As before, the -xyz flag is not a valid flag for the uname command. This time, however, the script will stop and an error will be thrown as the check attribute of subprocess.run is set.
import subprocess
try:
output = subprocess.run(["uname", "-xyz"], capture_output=True, text=True, check=True)
except:
print("An exception occurred when running external program")
Once an exception is generated, it can be caught using a try/except statement and handled appropriately.
Timing Out
If you want to limit the amount of time an external command can be run for, you can set a timeout. This is useful if you are concerned that the external application may freeze. Some programs take time to execute.
Set the timeout attribute with an integer number of seconds which is the maximum time the external program has to complete. If it does not complete in time, the external process will be killed and no result will be returned.
import subprocess subprocess.run(["sleep", "5"], timeout=3)
Note that even if check is not set to enable throwing errors, an error will be thrown if the timeout is exceeded. This is because the error generated when the external application times out does not come from the external application, but is an error thrown by Python.
Handling Input
You can pass data to STDIN using the input attribute. This data is sent to the standard input when the external application is executed by subprocess.
import subprocess subprocess.run(["cat", "/dev/stdin"], input=b"Hello world")
Above, the cat command is used to print the contents of STDIN. This input is provided by the input attribute. Note that the value of input is b”Hello world” – the b means that this isn’t a string variable, it’s a sequence of bytes. If the text attribute is enabled, both the input as well as the output will be treated as text, and the input must be sent as a string value.
import subprocess subprocess.run(["cat", "/dev/stdin"], text=True, input="Hello world")
Executing Python Using subprocess
Python code can also be executed using subprocess.run. External scripts can also be called. This is useful if you have code in an existing Python script you wish to execute, or have multiple scripts which all need to call the same script. You can call the script directly rather than having duplicate code in multiple files.
Executing Python Code
Python code can be executed directly by executing the Python executable externally. sys.executable provides the path to the Python executable currently in use. By passing the -c flag, Python commands can be passed directly to the Python executable.
Below, the Python code to print “hello world” is executed as a subprocess.
import subprocess
import sys
subprocess.run([sys.executable, "-c", "print('Hello world')"])
Executing an External Python Script
Similarly, entire Python scripts can be called by providing the path to the Python executable, and the path to the script to be executed.
import subprocess import sys subprocess.run([sys.executable, "script.py"])
Any text printed by the Python script will be available as part of STDOUT, the same as any other external program executed from subprocess.run, and can be read by enabling capture_output.
Executing a Bash/Shell script from Python
Bash, Zsh, and other shells are executables like any other, so you can call shell scripts by executing the shell executable followed by the path to the script.
import subprocess subprocess.run(["bash", "https://cd.linuxscrew.com/path/to/script.sh"])
If your shell script includes the correct shebang, you can use the sh command instead, and the correct shell will be used:
import subprocess subprocess.run(["sh", "https://cd.linuxscrew.com/path/to/script.sh"])
Any text printed by the shell script will be available as part of STDOUT, the same as any other external program executed from subprocess.run, and can be read by enabling capture_output.
Invoking the Shell
By default, the programs are executed directly by Python. It is possible to invoke the external commands through the system’s Shell, making available the features of that shell when constructing commands, like file globbing, additional features and additional environment variables. This is done using the shell attribute.
import subprocess subprocess.run(["uname", "-a"], shell=True)
Generally, unless you have a specific requirement and are aware of the security implications of the code you are running, it is best to avoid invoking the shell when using subprocess.run to execute external programs. Invoking the shell increases the risk that the user can run arbitrary commands on your system.
Warning: Handle Input With Care
Never take input from the user and pass it directly to subprocess.run (or any other function that executes commands). This leaves you with no control over what users can do on your system, and is a security nightmare.
Make sure that you sanitize all user input, and pass it to subprocess in a way that it cannot be mis-interpreted as commands, lest you give bad actors a way to execute arbitrary commands on your system command line.
This is especially important for web applications and other user-facing applications.
Conclusion
Python is a flexible and capable programming language, but you will run into situations where Python lacks some required functionality which is available through another command-line applications. In this situation, being able to call an external application is a big time-saver rather than having to code that functionality yourself. There’s no need to re-invent the wheel!
