How to run a program or system command from Python
Sooner or later a Python script has to call something else — git, ffmpeg,
pg_dump. This article shows how to do that with
subprocess.run(),
and demonstrates what the warnings about shell=True are actually about.
The one to use
import subprocess
result = subprocess.run(["echo", "hello"], capture_output=True, text=True)
returncode : 0
stdout : 'hello\n'
stderr : ''
type : CompletedProcess
subprocess.run() has been the recommended interface since Python 3.5. Three
things about that call are worth knowing before you go further.
Pass a list, not a string. One list element per argument. More on why in a moment.
capture_output=True gives you the output. Without it the output goes
straight to your terminal and result.stdout is None:
the next line is written by echo itself, not by python:
...straight through
and r.stdout is : None
Which is fine when you just want the program to run, and confusing when you expected a string.
text=True gives you str instead of bytes. Without it you get
b'hello\n', and you will be calling .decode() on it two lines later
anyway.
A failure is not an exception
This surprises people coming from other languages:
subprocess.run(['false']).returncode : 1
-> no exception was raised, the script just carried on
The command failed. Your script did not notice. If a failing command should stop your program, ask for it:
with check=True : CalledProcessError: Command '['false']' returned non-zero exit status 1.
check=True raises
CalledProcessError,
which carries returncode, stdout and stderr so you can log something
useful.
Why the list form, part one: typos
Here is a difference that is easy to miss. Call a program that does not exist:
list form : FileNotFoundError: [Errno 2] No such file or directory: 'definitely-not-a-program'
shell=True : no exception, returncode=127
stderr='/bin/sh: 1: definitely-not-a-program: not found'
The list form raises immediately. With shell=True there is no exception at
all — the shell starts, fails to find the program, and exits with 127. Unless
you check the return code, your script carries on as though everything worked.
That is how a typo in a program name survives all the way into production.
Why the list form, part two: the security bit
Now the reason everyone tells you to avoid shell=True. Suppose a filename
comes from somewhere you do not control — an upload form, a config file, a
queue message:
filename from a user : 'nonexistent; echo INJECTED >&2'
Run it through a shell:
subprocess.run(f"ls {filename}", shell=True, ...)
with shell=True, stderr contains : 'INJECTED'
-> the part after the semicolon ran as a second command
The semicolon ended the ls command and started a new one. Whatever came after
it ran with your script’s privileges. Replace echo INJECTED with something
less friendly and you have the whole problem.
The list form:
with the list form, stderr : "ls: cannot access 'nonexistent; echo INJECTED >&2': No such file or directory"
-> treated as one (missing) filename, nothing was executed
The string was handed to the operating system as a single argument. There is no shell involved, so there is nothing to interpret the semicolon. That is not “escaping done well” — it is a different mechanism, with nothing to escape.
What the shell was doing for you
To be fair to shell=True, it does earn its keep sometimes:
list form does not glob : '*\n'
shell=True does glob : '/etc/hostname\n'
Wildcards, pipes, &&, environment variable expansion and redirection are all
shell features. Without a shell you get none of them.
The answer is usually not to reach for shell=True but to do the job in
Python. Wildcards are glob,
and pipes can be built directly:
p1 = subprocess.Popen(["printf", "b\\na\\nc\\n"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["sort"], stdin=p1.stdout, stdout=subprocess.PIPE, text=True)
p1.stdout.close()
out = p2.communicate()[0]
printf | sort -> 'a\nb\nc\n'
The p1.stdout.close() is not decoration — it lets the first process receive
SIGPIPE if the second one exits early.
If you genuinely need a shell, at least run the arguments through
shlex.quote()
first.
When you only have a command as a string
Sometimes the command arrives as one string and you need the list.
shlex.split()
does it properly:
shlex.split("echo 'one argument' two") -> ['echo', 'one argument', 'two']
naive .split() -> ['echo', "'one", "argument'", 'two']
Splitting on whitespace breaks the moment an argument contains a space, which on most systems means the moment a path does.
The older ways, and what they give you
You will meet these in older code:
os.system returned : 0 (an exit status, not the output)
os.popen().read() : 'popen-output\n'
os.system() cannot
give you the output at all. It returns an exit status, prints to the terminal,
and runs everything through a shell — so it carries the injection problem by
construction. os.popen() gives you the output but nothing else, no return
code and no stderr.
subprocess.call() and subprocess.check_output() still work and are not
deprecated, but run() does everything they do:
check_output(['echo','x']) : 'x\n'
on failure : CalledProcessError (returncode 1)
Two arguments worth remembering
Feeding a program something on stdin:
subprocess.run(["cat"], input="fed in through stdin\n", capture_output=True, text=True)
input= : 'fed in through stdin\n'
And not waiting forever:
timeout=0.3 on 'sleep 5' : TimeoutExpired after 0.3s
A timeout belongs on anything that talks to a network or to hardware.
Without it, a hung command hangs your script for as long as it takes somebody
to notice.
About Netcup (advertisement)
The German host Netcup offers, among other things, affordable and powerful web hosting packages, KVM-based root servers and dedicated servers. With our voucher codes you can save even more (6€ off your first order, 30% off all KVM-based root servers, ...).