subprocess stdin.write() Deadlock Without communicate() (tested)

Writing to proc.stdin and later reading proc.stdout looks like a reasonable way to talk to a subprocess by hand instead of using communicate(). We measured what happens once the data is large enough: the program hangs forever, with no exception, no timeout, and no indication anything is wrong.

1. Writing enough data directly to stdin hangs indefinitely

import subprocess

proc = subprocess.Popen(["cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
big_data = b"x" * (1024 * 1024)  # 1MB

proc.stdin.write(big_data)   # never returns
proc.stdin.close()
print("done")

Measured: run under a shell-level timeout 10 as a safety net, this script never reaches print("done") and gets killed by the outer timeout (exit code 124) — proc.stdin.write() itself never returns. With a smaller amount of data (we measured 128KB working fine), the exact same code completes instantly, which is what makes this bug so easy to ship: it only appears once input crosses a size threshold that depends on the OS pipe buffer and how the child process buffers its own output.

2. Why: the child's stdout pipe fills up because nobody is reading it

cat reads from stdin and immediately tries to write the same bytes to stdout. Since this script never reads proc.stdout, that pipe fills to its OS buffer limit, and cat's own write to stdout blocks. A blocked cat stops reading stdin entirely — which means proc.stdin.write() on the Python side eventually blocks too, once the stdin pipe's buffer also fills. Both processes end up waiting on each other with no way out: a classic pipe deadlock, not a slow operation that will eventually finish.

3. communicate() avoids it because it reads and writes concurrently

proc = subprocess.Popen(["cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = proc.communicate(input=big_data)
print(len(out))
1048576

Measured: the identical 1MB payload completes in under 3 milliseconds through communicate(). Per the subprocess docs, communicate() explicitly exists to solve this: it uses separate threads internally to write stdin and read stdout/stderr at the same time, so neither pipe can ever fill up while the other side is blocked waiting.

4. Fix: never write to stdin and read stdout/stderr by hand on the same process

Use communicate(input=...) whenever a subprocess needs both stdin data and its stdout/stderr captured — it is the built-in, deadlock-safe equivalent of manual proc.stdin.write() + proc.stdout.read(). If you genuinely need to stream data in both directions over time (not a single write-then-read), use select/selectors or a background thread per pipe so reads and writes can never block each other. The docs' own warning covers exactly this: "this will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data."

Frequently asked questions

Why does writing to subprocess stdin sometimes hang forever?

We measured it happening when the data is large enough that the child process's own stdout pipe fills up (because nothing is reading it concurrently), causing the child to block on its own output write and stop reading further input — deadlocking both sides.

Does this deadlock happen with small amounts of data?

Not in our test — we measured 128KB completing instantly with the same code, while 1MB hung indefinitely. The threshold depends on OS pipe buffer size and the child's own output buffering, which is exactly why this bug often passes small-scale testing and then hangs in production.

How do I avoid subprocess stdin/stdout deadlocks in Python?

We confirmed that subprocess.Popen.communicate(input=...) handles the identical case instantly by reading and writing to the child's pipes concurrently on separate threads, which is exactly what the Python docs recommend instead of manual proc.stdin.write()/proc.stdout.read() calls.

Does Python raise an exception or timeout when this deadlock happens?

No — we measured proc.stdin.write() simply never returning, with no exception and no built-in timeout. We only recovered control by wrapping the whole script in a shell-level timeout command as an external safety net.

Is this deadlock specific to using the cat command?

No — cat just makes the mechanism easy to demonstrate. Any child process that both reads from stdin and writes a large amount to stdout (compilers, image/video tools, data pipelines) can trigger the same deadlock if you write to its stdin and read its stdout separately instead of using communicate().

Written with AI assistance. Measured on 2026-10-03 with Python 3.13.5's built-in subprocess module on Linux: launched Popen(['cat'], stdin=PIPE, stdout=PIPE), wrote 1MB directly to proc.stdin without reading proc.stdout concurrently, bounded the whole script with the shell's own `timeout 10` so a true hang couldn't actually freeze anything, then repeated with communicate(). PlainHeaders is our extension; Python's subprocess module is built in.