Python Subprocess Tutorial [Beginner To Advanced]

The Python subprocess module is a powerful tool that allows developers to interact with external processes and execute system commands from within their Python programs.

It provides a way to create, manage, and communicate with subprocesses, making it an essential part of Python’s standard library for system administration, automation, and various other tasks.

Mastering Python Subprocess: A Comprehensive Tutorial

This comprehensive tutorial will walk you through the fundamentals of using the subprocess module in Python, starting with the basics and gradually diving into more advanced topics.

Whether you need to run simple shell commands, manage complex pipelines of processes, or interact with long-running external programs, this tutorial will equip you with the knowledge and skills to do so effectively.

Throughout this tutorial, you will learn how to:

  1. Run simple commands and capture their output.
  2. Handle input and output streams for subprocesses.
  3. Manage errors and exceptions when working with external processes.
  4. Redirect input and output to and from files.
  5. Set environment variables for subprocesses.
  6. Handle timeouts and asynchronous subprocess execution.
  7. Create pipelines of processes for data processing.
  8. Send signals to running processes for control and termination.
  9. Deal with platform-specific considerations.

By the end of this tutorial, you’ll have a solid understanding of how to harness the full potential of the subprocess module in Python, enabling you to automate tasks, integrate with external tools and services, and perform system-level operations with confidence.

Let’s dive into the world of Python subprocesses and unlock a new realm of possibilities for your Python programming journey.

Here’s a tutorial on how to use the subprocess module effectively:

1. Import the subprocess module

First, import the subprocess module:

import subprocess
Code language: Python (python)

Learn More: What Is Subprocess In Python? 

2. Running a Simple Command

You can run a simple command by using the function. For example, to run the ls command to list files in a directory:

result =["ls", "-l"])
Code language: Python (python)

This will execute the ls -l command and store the result in the result variable. You can access the return code, standard output, and standard error like this:

print("Return Code:", result.returncode)
print("Standard Output:", result.stdout)
print("Standard Error:", result.stderr)
Code language: Python (python)

Learn More: Learn in Python [Step-by-Step Examples]

3. Handling Input and Output

You can also pass input to the command and capture its output. For example, let’s run a simple Python script that reads input from standard input and prints it to standard output:

input_data = "Hello, subprocess!"
result =["python", "-c", "print(input())"], input=input_data.encode(), text=True, capture_output=True)

print("Return Code:", result.returncode)
print("Standard Output:", result.stdout)
print("Standard Error:", result.stderr)
Code language: Python (python)

In this example, we use the input parameter to provide input data as bytes, the text parameter to specify that the data is in text format, and the capture_output parameter to capture the standard output.

Learn More:

4. Running Shell Commands

You can also run shell commands by setting the shell parameter to True. However, be cautious about shell injection vulnerabilities if you’re including user-provided input in the command.

command = "echo Hello, subprocess!"
result =, shell=True, text=True, capture_output=True)

print("Return Code:", result.returncode)
print("Standard Output:", result.stdout)
print("Standard Error:", result.stderr)
Code language: Python (python)

Learn More: Python Execute Shell Command And Get Output

5. Handling Errors

To check if the command executed successfully, you can examine the return code. Typically, a return code of 0 indicates success, while non-zero values indicate errors.

if result.returncode == 0:
    print("Command executed successfully")
    print("Command failed with return code:", result.returncode)
Code language: Python (python)

6. Advanced Features

The subprocess module offers many more features for more complex scenarios, such as redirecting input/output, setting environment variables, and handling timeouts.

6.1. Redirecting Input/Output/Error

You can control where the standard input, standard output, and standard error of a subprocess are directed. Use the stdin, stdout, and stderr parameters when creating a subprocess.

import subprocess

with open("output.txt", "w") as outfile:["echo", "Hello, subprocess!"], stdout=outfile)

with open("input.txt", "r") as infile:["cat"], stdin=infile)
Code language: Python (python)

In this example, we redirect the standard output of the echo command to a file called “output.txt” and provide input to the cat command from “input.txt”.

Learn More: Python Subprocess’ Stdin [Full Guide With Examples]

6.2. Setting Environment Variables

You can set environment variables for the subprocess using the env parameter. This is useful when you need to customize the environment in which the subprocess runs.

import subprocess

custom_env = {"CUSTOM_VARIABLE": "custom_value"}
result =["python", "-c", "import os; print(os.getenv('CUSTOM_VARIABLE'))"], env=custom_env, text=True, capture_output=True)
Code language: Python (python)

Here, we set the CUSTOM_VARIABLE environment variable for the subprocess and use it within a Python script.

6.3. Handling Timeouts

You can specify a timeout for the subprocess using the timeout parameter. If the subprocess doesn’t complete within the specified time, a TimeoutExpired exception is raised.

import subprocess

    result =["sleep", "5"], timeout=3, text=True, capture_output=True)
except subprocess.TimeoutExpired:
    print("Command timed out")
Code language: Python (python)

In this example, we attempt to run the sleep command for 5 seconds but set a timeout of 3 seconds. The code will catch the TimeoutExpired exception when the command takes too long to complete.

6.4. Piping Between Processes

You can create pipelines of processes by connecting them using pipes. For example, you can pipe the output of one command to another using the stdout of one subprocess as the stdin of another.

import subprocess

p1 = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", ".txt"], stdin=p1.stdout, stdout=subprocess.PIPE, text=True)

output = p2.communicate()[0]
Code language: Python (python)

In this example, we list files in a directory with ls -l and then pipe the output to grep to filter for .txt files.

Learn More: Python Subprocess Pipe With Example

6.5. Handling Exceptions

Make sure to handle exceptions that might occur during subprocess execution. Common exceptions include subprocess.CalledProcessError, FileNotFoundError, and PermissionError. Robust error handling ensures that your program doesn’t crash unexpectedly.

import subprocess

    result =["non_existent_command"], check=True, text=True, capture_output=True)
except subprocess.CalledProcessError as e:
    print("Command failed with error:", e)
except FileNotFoundError as e:
    print("Command not found:", e)
except PermissionError as e:
    print("Permission denied:", e)Code language: Python (python)

These are some of the advanced features of the subprocess module. Depending on your specific use case, you may need to use different combinations of these features to achieve your desired functionality when interacting with external processes in Python.

6.6. Running Background Processes

If you want to run a subprocess in the background and continue with your program without waiting for it to finish, you can use the subprocess.Popen class. This class allows you to start a process and continue with other tasks.

import subprocess

# Start a background process
background_process = subprocess.Popen(["python", ""])

# Continue with other tasks

# Optionally, you can wait for the background process to finish later if needed
Code language: Python (python)

Learn More: Python Subprocess Run In Background [With Example]

6.7. Communicating with Long-Running Processes

For long-running processes where you need to interact with the subprocess during its execution, you can use subprocess.Popen in combination with communicate() to send input and receive output interactively.

import subprocess

process = subprocess.Popen(["python", "-u", ""], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

# Send input to the subprocess

# Read output from the subprocess
output = process.stdout.readline()

# Optionally, wait for the subprocess to finish
Code language: Python (python)

Learn More: How to use the subprocess Popen.communicate() method

6.8. Cross-Platform Considerations

Keep in mind that certain commands and behaviors may vary across different operating systems (Windows, Linux, macOS). Be aware of these platform-specific differences when using subprocess. You can use the platform module to detect the current operating system and adjust your code accordingly.

import platform

if platform.system() == "Windows":
    # Windows-specific code
elif platform.system() == "Linux":
    # Linux-specific code
elif platform.system() == "Darwin":
    # macOS-specific code
    # Code for other operating systems
Code language: Python (python)

6.9. Asynchronous Subprocesses

You can use Python’s asyncio library to run subprocesses asynchronously. This is particularly useful when you want to run multiple subprocesses concurrently and efficiently handle their outputs.

import asyncio
import subprocess

async def run_command(command):
    process = await asyncio.create_subprocess_shell(command, stdout=asyncio.subprocess.PIPE, text=True)
    output, _ = await process.communicate()
    return output

async def main():
    command1 = "ls -l"
    command2 = "echo 'Hello, subprocess!'"

    result1 = await run_command(command1)
    result2 = await run_command(command2)


if __name__ == "__main__":
Code language: Python (python)

This example uses asyncio to run two subprocesses concurrently and asynchronously.

Learn More: Python Asyncio Subprocess

6.10. Interacting with Streams

When dealing with processes that produce continuous streams of data, such as real-time logs or live data feeds, you can use the stdout and stderr streams to read data as it’s produced, rather than waiting for the process to complete.

import subprocess

process = subprocess.Popen(["tail", "-f", "my_log_file.log"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

while True:
    line = process.stdout.readline()
    if not line:
    print("stdout:", line.strip())

    error_line = process.stderr.readline()
    if not error_line:
    print("stderr:", error_line.strip())

Code language: Python (python)

This code continuously reads from both stdout and stderr streams of a subprocess.

Learn More: Subprocess Python Stdout [In-Depth Tutorial]

6.11. Handling Signals

You can send signals to running processes using the send_signal() method of a Popen object. For example, to send the SIGTERM signal to gracefully terminate a process:

import subprocess
import signal
import time

process = subprocess.Popen(["python", ""])

# Allow some time for the process to run

# Send a SIGTERM signal to gracefully terminate the process

# Wait for the process to complete
Code language: Python (python)

This allows you to control and manage the lifecycle of subprocesses more precisely.

6.12. Creating Custom Shells

You can create custom shells to execute multiple commands within the same shell environment. This is useful for complex sequences of commands.

import subprocess

commands = [
    "echo 'Step 1'",
    "echo 'Step 2'",
    "echo 'Step 3'",

shell_command = " && ".join(commands)

process = subprocess.Popen(shell_command, shell=True, text=True, stdout=subprocess.PIPE)

for line in process.stdout:

Code language: Python (python)

In this example, the && operator is used to run multiple commands within the same shell session.

Learn More: Python Execute Shell Command And Get Output [In-Depth Guide]

These advanced topics should give you more control and flexibility when working with the subprocess module in Python, especially in complex scenarios or when dealing with more challenging subprocess interactions.

The official Python documentation for subprocess is a valuable resource for diving deeper into these topics:

Read More;

  • Aniket Singh

    Aniket Singh holds a B.Tech in Computer Science & Engineering from Oriental University. He is a skilled programmer with a strong coding background, having hands-on experience in developing advanced projects, particularly in Python and the Django framework. Aniket has worked on various real-world industry projects and has a solid command of Python, Django, REST API, PostgreSQL, as well as proficiency in C and C++. He is eager to collaborate with experienced professionals to further enhance his skills.

Leave a Comment