Fix: Python subprocess Not Working — Output Empty, Command Not Found, or Permission Denied

The Problem

subprocess.run() runs but the output is empty or None:

import subprocess

result = subprocess.run(['ls', '-la'])
print(result.stdout)  # None

Or the command isn’t found even though it works in the terminal:

result = subprocess.run(['python', 'script.py'])
# FileNotFoundError: [Errno 2] No such file or directory: 'python'
# Works fine in the terminal — why not in subprocess?

Or the command succeeds but stdout and stderr are mixed or lost:

result = subprocess.run(['npm', 'install'], capture_output=True, text=True)
print(result.stdout)  # Empty — output went to stderr

Or a long-running command hangs the Python process indefinitely:

result = subprocess.run(['ffmpeg', '-i', 'large_video.mp4', 'out.mp4'])
# Hangs forever — no timeout

Why This Happens

subprocess.run() has safe defaults that don’t match most expectations:

• Output not captured by default — without capture_output=True or stdout=PIPE, output goes directly to the terminal and result.stdout is None.
• text=False by default — without text=True (or encoding=), captured output is bytes, not str.
• shell=False by default — commands are executed directly, not through a shell. Shell builtins like cd, source, and pipe operators (|, &&) don’t work without shell=True.
• PATH differences — the Python process may have a different PATH than your interactive shell, especially inside virtual environments, Docker, or cron jobs.
• stderr is separate — stdout and stderr are captured independently. If the command writes to stderr, result.stdout is still empty.

The other reason this confuses people is the long lineage of overlapping APIs inside the subprocess module. Before Python 3.5, calling external commands meant choosing between subprocess.call(), subprocess.check_call(), subprocess.check_output(), and subprocess.Popen() — each with different signatures, return types, and ways of capturing output. Tutorials, Stack Overflow answers, and even internal codebases still mix all four, which is why you constantly see incompatible snippets that all claim to “run a command.”

Python 3.5 (Sept 2015) introduced subprocess.run() as the unified entry point and returned a CompletedProcess object with .returncode, .stdout, and .stderr attributes. Python 3.7 then added the capture_output=True shortcut and renamed universal_newlines=True to text=True (the old name still works as an alias). Python 3.8 added the pipesize argument to Popen for tuning OS pipe buffers, and Python 3.11 added process_cpu_count() plus tightened up shell= security on Windows. If you read code written across that span — or even mix versions inside your own project — you will inevitably hit subtle behaviour differences, especially around how output is decoded and how None versus b'' versus '' is returned for unset streams.

A third source of trouble is the runtime environment. Subprocesses inherit the parent’s environment variables, working directory, file descriptors, and signal mask. Inside a systemd unit, a cron job, a Docker container, or a CI runner, $PATH typically only contains /usr/bin:/bin, locale variables are often unset (which breaks text=True decoding), and there is no controlling TTY (which makes commands like git, npm, or apt-get switch to non-interactive modes that produce different output). The Python script works on your laptop, then fails in production with FileNotFoundError or empty stdout — same code, completely different surface.

Fix 1: Capture Output Correctly

Always pass capture_output=True, text=True when you need the output as a string:

import subprocess

# WRONG — stdout goes to terminal, result.stdout is None
result = subprocess.run(['ls', '-la'])
print(result.stdout)  # None

# CORRECT — capture stdout and stderr as text
result = subprocess.run(
    ['ls', '-la'],
    capture_output=True,
    text=True
)
print(result.stdout)   # Directory listing as string
print(result.stderr)   # Any errors
print(result.returncode)  # 0 for success

# Equivalent explicit form (pre-Python 3.7):
result = subprocess.run(
    ['ls', '-la'],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    encoding='utf-8'
)

Capture both stdout and stderr together:

# Merge stderr into stdout
result = subprocess.run(
    ['npm', 'install'],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,  # Redirect stderr to stdout
    text=True
)
print(result.stdout)  # Contains both stdout and stderr

Check return code and raise on failure:

# check=True raises CalledProcessError if return code != 0
try:
    result = subprocess.run(
        ['git', 'pull'],
        capture_output=True,
        text=True,
        check=True  # Raises if command fails
    )
    print(result.stdout)
except subprocess.CalledProcessError as e:
    print(f"Command failed with exit code {e.returncode}")
    print(f"stderr: {e.stderr}")

Fix 2: Handle PATH and Environment Issues

The subprocess inherits the parent process’s environment, which may differ from your interactive shell:

import subprocess
import os
import sys

# WRONG — 'python' may not be in PATH or points to wrong version
result = subprocess.run(['python', 'script.py'], capture_output=True, text=True)

# CORRECT — use sys.executable to run the same Python interpreter
result = subprocess.run(
    [sys.executable, 'script.py'],
    capture_output=True,
    text=True
)

# CORRECT — use full path when you know where the binary is
result = subprocess.run(
    ['/usr/local/bin/node', 'server.js'],
    capture_output=True,
    text=True
)

# Debug PATH issues — print what subprocess sees
result = subprocess.run(
    ['env'],  # or ['printenv'] on Linux
    capture_output=True,
    text=True
)
print(result.stdout)

Pass a custom environment:

import os

# Extend the current environment with extra variables
env = os.environ.copy()
env['MY_VAR'] = 'my_value'
env['PATH'] = f"/custom/bin:{env['PATH']}"

result = subprocess.run(
    ['my_command'],
    capture_output=True,
    text=True,
    env=env
)

Set the working directory:

result = subprocess.run(
    ['npm', 'run', 'build'],
    capture_output=True,
    text=True,
    cwd='/path/to/project'  # Run in this directory
)

Fix 3: Use shell=True Correctly (and Safely)

shell=True runs the command through /bin/sh, enabling shell features like pipes, redirection, and glob expansion — but it introduces security risks:

# shell=True — enables shell features
result = subprocess.run(
    'ls -la | grep ".py" | wc -l',
    shell=True,
    capture_output=True,
    text=True
)

# Same as above but with a list (preferred on Unix)
result = subprocess.run(
    ['bash', '-c', 'ls -la | grep ".py" | wc -l'],
    capture_output=True,
    text=True
)

Security warning — never use shell=True with user input:

# DANGEROUS — command injection vulnerability
user_input = "file.txt; rm -rf /"
result = subprocess.run(f"cat {user_input}", shell=True)  # Executes rm -rf /

# SAFE — pass user input as list argument (no shell injection possible)
result = subprocess.run(['cat', user_input], capture_output=True, text=True)

Warning: Only use shell=True when the command string is fully under your control and doesn’t include any user-supplied input.

When you actually need shell features, use pipes explicitly:

import subprocess

# Instead of: "ps aux | grep python"
ps = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE)
grep = subprocess.Popen(
    ['grep', 'python'],
    stdin=ps.stdout,
    stdout=subprocess.PIPE,
    text=True
)
ps.stdout.close()  # Allow ps to receive SIGPIPE if grep exits early
output, _ = grep.communicate()
print(output)

Fix 4: Set Timeouts to Prevent Hanging

Long-running commands can hang indefinitely without a timeout:

import subprocess

# WRONG — hangs if command never finishes
result = subprocess.run(['ping', '-c', '1000', 'example.com'])

# CORRECT — raises TimeoutExpired after 10 seconds
try:
    result = subprocess.run(
        ['ping', '-c', '1000', 'example.com'],
        capture_output=True,
        text=True,
        timeout=10
    )
except subprocess.TimeoutExpired as e:
    print(f"Command timed out after {e.timeout}s")
    print(f"Partial stdout: {e.stdout}")

Timeouts with Popen for streaming output:

import subprocess
import threading

def run_with_timeout(cmd, timeout):
    proc = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True
    )

    try:
        stdout, stderr = proc.communicate(timeout=timeout)
        return proc.returncode, stdout, stderr
    except subprocess.TimeoutExpired:
        proc.kill()
        stdout, stderr = proc.communicate()
        raise subprocess.TimeoutExpired(cmd, timeout, output=stdout, stderr=stderr)

returncode, out, err = run_with_timeout(['long_command'], timeout=30)

Fix 5: Stream Output in Real Time with Popen

subprocess.run() buffers all output until the command finishes. Use Popen when you need to see output as it arrives:

import subprocess

# subprocess.run() — waits for everything, then returns
# Good for: short commands where you need the full output at once

# Popen — gives you a process handle for streaming
# Good for: long commands, progress updates, interactive processes

def run_streaming(cmd):
    with subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1  # Line-buffered
    ) as proc:
        for line in proc.stdout:
            print(line, end='')  # Print each line as it arrives
        proc.wait()
        return proc.returncode

# Usage — see build output in real time
returncode = run_streaming(['make', 'build'])

Async streaming with asyncio:

import asyncio

async def run_async(cmd):
    proc = await asyncio.create_subprocess_exec(
        *cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE
    )

    async def read_stream(stream, callback):
        while True:
            line = await stream.readline()
            if not line:
                break
            callback(line.decode())

    await asyncio.gather(
        read_stream(proc.stdout, lambda l: print('OUT:', l, end='')),
        read_stream(proc.stderr, lambda l: print('ERR:', l, end='')),
    )

    await proc.wait()
    return proc.returncode

# Run async subprocess
asyncio.run(run_async(['npm', 'run', 'build']))

Fix 6: Common Subprocess Patterns

Run a command and get output as a list of lines:

def get_lines(cmd):
    result = subprocess.run(cmd, capture_output=True, text=True, check=True)
    return result.stdout.strip().splitlines()

files = get_lines(['git', 'diff', '--name-only'])
print(files)  # ['src/main.py', 'tests/test_main.py']

Check if a command exists:

import shutil

def command_exists(cmd):
    return shutil.which(cmd) is not None

if command_exists('ffmpeg'):
    subprocess.run(['ffmpeg', '-version'])
else:
    print("ffmpeg not installed")

Run a command with input (stdin):

# Pass input to the process's stdin
result = subprocess.run(
    ['python', '-c', 'import sys; data = sys.stdin.read(); print(data.upper())'],
    input='hello world\n',
    capture_output=True,
    text=True
)
print(result.stdout)  # HELLO WORLD

Retry logic for flaky commands:

import time

def run_with_retry(cmd, retries=3, delay=2):
    for attempt in range(retries):
        result = subprocess.run(cmd, capture_output=True, text=True)
        if result.returncode == 0:
            return result
        if attempt < retries - 1:
            print(f"Attempt {attempt + 1} failed, retrying in {delay}s...")
            time.sleep(delay)
    raise subprocess.CalledProcessError(result.returncode, cmd, result.stdout, result.stderr)

Version History: How the subprocess API Evolved

Most of the confusion around subprocess comes from version drift between snippets. Knowing what landed when saves hours of debugging.

Python 3.5 (Sept 2015) — subprocess.run() and CompletedProcess. Before this, you stitched together Popen, communicate(), and manual returncode checks. subprocess.run() collapses all of that into one call that returns a CompletedProcess with .returncode, .stdout, .stderr, and .args. PEP 503 also raised the minimum supported behaviour: passing check=True raises CalledProcessError on non-zero exit, and timeout= raises TimeoutExpired. If you are still using subprocess.call() you are leaving on the table all of the structured error reporting that came in 3.5.

Python 3.6 — os.PathLike support and encoding= consistency. You can pass pathlib.Path objects directly as the command or cwd=. The encoding= argument on Popen was made to behave the same way text=True does in 3.7, so encoding='utf-8' is a forward-compatible way to get text-mode output on 3.6.

Python 3.7 (June 2018) — capture_output=True and text= alias. This is the version most production code targets. capture_output=True is a shortcut for stdout=PIPE, stderr=PIPE. text=True was added as a clearer alias for universal_newlines=True. Both keywords still exist — if you maintain a library that needs 3.5/3.6 compatibility, prefer universal_newlines=True; otherwise use text=True.

Python 3.8 — pipesize= on Popen. You can now hint at the OS pipe buffer size when creating a Popen. This matters when a child process writes a lot of data faster than the parent reads it: the default 64KB pipe on Linux can fill up and deadlock. Bumping pipesize=1024*1024 (1MB) avoids the deadlock without switching to threads or asyncio.

Python 3.9 — signal_handler improvements. Cleaner behaviour when the parent receives SIGINT while waiting on a subprocess; KeyboardInterrupt now propagates more reliably during communicate().

Python 3.11 (Oct 2022) — process_cpu_count() and hardened Windows behaviour. os.process_cpu_count() reports CPUs available to the process (respecting cgroup limits), which is what you actually want when sizing subprocess pools inside containers. Windows shell quoting was tightened to mitigate command injection through shell=True.

Python 3.12 — _use_posix_spawn faster path. On Linux and macOS, simple subprocess.run() calls without preexec_fn now use posix_spawn() instead of fork()+exec(). This is invisible to your code but cuts startup overhead by roughly 2-5x for short-lived commands.

Python 3.13 (Oct 2024) — subprocess removes LookupError regressions. Minor bug fixes around how text-mode streams handle replacement on undecodable bytes — errors='replace' is now respected on all platforms uniformly.

Practical migration rules. If a tutorial uses subprocess.check_output(...), mentally rewrite it as subprocess.run(..., capture_output=True, text=True, check=True).stdout. If it uses subprocess.call(...), rewrite as subprocess.run(...).returncode. If it constructs a Popen just to call .communicate() once, switch to subprocess.run() unless you genuinely need streaming or stdin interaction.

Still Not Working?

Output buffering causes empty or delayed output — some programs buffer their output when not connected to a terminal. If streaming output appears empty or delayed, force line buffering with PYTHONUNBUFFERED=1 (for Python subprocesses) or use the program’s own unbuffered flag (e.g., python -u). For C programs, stdbuf -oL command forces line buffering.

Unicode errors in output — if the command outputs non-UTF-8 characters, text=True raises a UnicodeDecodeError. Use encoding='latin-1' or errors='replace' to handle it:

result = subprocess.run(
    ['some_command'],
    capture_output=True,
    encoding='utf-8',
    errors='replace'  # Replace undecodable bytes with ?
)

FileNotFoundError on Windows for commands that work in CMD — on Windows, some commands like dir, copy, and del are shell builtins and require shell=True. Also, executable extensions (.exe, .bat, .cmd) may need to be included explicitly:

# Windows shell builtin
result = subprocess.run('dir', shell=True, capture_output=True, text=True)

# Explicit .exe extension
result = subprocess.run(['python.exe', 'script.py'], capture_output=True, text=True)

Process inherits open file descriptors — by default, child processes inherit the parent’s open file descriptors, which can cause issues in long-running servers. Pass close_fds=True (the default on Unix since Python 3.2) or use pass_fds=() to control which FDs are inherited.

Locale is C inside containers and cron, breaking text=True — without LANG or LC_ALL set, Python falls back to ASCII for stream decoding on some platforms. A command that emits UTF-8 then crashes with UnicodeDecodeError even though it works on your laptop. Set env={'LANG': 'C.UTF-8', **os.environ} or pass encoding='utf-8', errors='replace' explicitly. This is the single most common cause of “works on my machine but fails in Docker.”

Permission denied on a file that is clearly executable — Linux requires both the file and every parent directory to be readable+executable for the calling user. If you scripted a copy that lost the executable bit (scp, unzip, S3 downloads), os.chmod(path, 0o755) before invoking. On Windows, the issue is usually the file extension: subprocess.run(['myscript']) fails when myscript.bat exists — pass the full filename or set shell=True.

Subprocess output works in shell but is empty when redirected — many CLI tools (git, docker, kubectl, npm) detect whether stdout is a TTY and switch to a machine-friendly format when it isn’t. If you want the colored, paged output you see in your terminal, set env={'FORCE_COLOR': '1', 'TERM': 'xterm-256color', **os.environ} or use the tool’s own --color=always flag. If you want stable, scriptable output, this auto-detection is actually doing you a favor — parse the redirected form.

For related Python issues, see Fix: Python asyncio Not Running, Fix: Python Logging Not Working, Fix: Python multiprocessing Not Working, and Fix: Python virtualenv Wrong Python.