How to run a program or system command from Python

Sooner or later a Python script has to call something else — git, ffmpeg, pg_dump. This article shows how to do that with subprocess.run(), and demonstrates what the warnings about shell=True are actually about.

The one to use

import subprocess

result = subprocess.run(["echo", "hello"], capture_output=True, text=True)
returncode : 0
stdout     : 'hello\n'
stderr     : ''
type       : CompletedProcess

subprocess.run() has been the recommended interface since Python 3.5. Three things about that call are worth knowing before you go further.

Pass a list, not a string. One list element per argument. More on why in a moment.

capture_output=True gives you the output. Without it the output goes straight to your terminal and result.stdout is None:

the next line is written by echo itself, not by python:
  ...straight through
and r.stdout is        : None

Which is fine when you just want the program to run, and confusing when you expected a string.

text=True gives you str instead of bytes. Without it you get b'hello\n', and you will be calling .decode() on it two lines later anyway.

A failure is not an exception

This surprises people coming from other languages:

subprocess.run(['false']).returncode : 1
-> no exception was raised, the script just carried on

The command failed. Your script did not notice. If a failing command should stop your program, ask for it:

with check=True : CalledProcessError: Command '['false']' returned non-zero exit status 1.

check=True raises CalledProcessError, which carries returncode, stdout and stderr so you can log something useful.

Why the list form, part one: typos

Here is a difference that is easy to miss. Call a program that does not exist:

list form  : FileNotFoundError: [Errno 2] No such file or directory: 'definitely-not-a-program'
shell=True : no exception, returncode=127
             stderr='/bin/sh: 1: definitely-not-a-program: not found'

The list form raises immediately. With shell=True there is no exception at all — the shell starts, fails to find the program, and exits with 127. Unless you check the return code, your script carries on as though everything worked.

That is how a typo in a program name survives all the way into production.

Why the list form, part two: the security bit

Now the reason everyone tells you to avoid shell=True. Suppose a filename comes from somewhere you do not control — an upload form, a config file, a queue message:

filename from a user : 'nonexistent; echo INJECTED >&2'

Run it through a shell:

subprocess.run(f"ls {filename}", shell=True, ...)
with shell=True, stderr contains : 'INJECTED'
-> the part after the semicolon ran as a second command

The semicolon ended the ls command and started a new one. Whatever came after it ran with your script’s privileges. Replace echo INJECTED with something less friendly and you have the whole problem.

The list form:

with the list form, stderr : "ls: cannot access 'nonexistent; echo INJECTED >&2': No such file or directory"
-> treated as one (missing) filename, nothing was executed

The string was handed to the operating system as a single argument. There is no shell involved, so there is nothing to interpret the semicolon. That is not “escaping done well” — it is a different mechanism, with nothing to escape.

What the shell was doing for you

To be fair to shell=True, it does earn its keep sometimes:

list form does not glob : '*\n'
shell=True does glob    : '/etc/hostname\n'

Wildcards, pipes, &&, environment variable expansion and redirection are all shell features. Without a shell you get none of them.

The answer is usually not to reach for shell=True but to do the job in Python. Wildcards are glob, and pipes can be built directly:

p1 = subprocess.Popen(["printf", "b\\na\\nc\\n"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["sort"], stdin=p1.stdout, stdout=subprocess.PIPE, text=True)
p1.stdout.close()
out = p2.communicate()[0]
printf | sort -> 'a\nb\nc\n'

The p1.stdout.close() is not decoration — it lets the first process receive SIGPIPE if the second one exits early.

If you genuinely need a shell, at least run the arguments through shlex.quote() first.

When you only have a command as a string

Sometimes the command arrives as one string and you need the list. shlex.split() does it properly:

shlex.split("echo 'one argument' two") -> ['echo', 'one argument', 'two']
naive .split()                         -> ['echo', "'one", "argument'", 'two']

Splitting on whitespace breaks the moment an argument contains a space, which on most systems means the moment a path does.

The older ways, and what they give you

You will meet these in older code:

os.system returned    : 0  (an exit status, not the output)
os.popen().read()     : 'popen-output\n'

os.system() cannot give you the output at all. It returns an exit status, prints to the terminal, and runs everything through a shell — so it carries the injection problem by construction. os.popen() gives you the output but nothing else, no return code and no stderr.

subprocess.call() and subprocess.check_output() still work and are not deprecated, but run() does everything they do:

check_output(['echo','x']) : 'x\n'
on failure                 : CalledProcessError (returncode 1)

Two arguments worth remembering

Feeding a program something on stdin:

subprocess.run(["cat"], input="fed in through stdin\n", capture_output=True, text=True)
input= : 'fed in through stdin\n'

And not waiting forever:

timeout=0.3 on 'sleep 5' : TimeoutExpired after 0.3s

A timeout belongs on anything that talks to a network or to hardware. Without it, a hung command hangs your script for as long as it takes somebody to notice.

About Netcup (advertisement)

The German host Netcup offers, among other things, affordable and powerful web hosting packages, KVM-based root servers and dedicated servers. With our voucher codes you can save even more (6€ off your first order, 30% off all KVM-based root servers, ...).