gzip/bz2/lzma Crash on a Non-Blocking None (Tested)

A non-blocking raw stream signals "no data ready yet" by returning None from readinto() -- distinct from the empty read that means real end-of-stream. Plain io.BufferedReader.read() handles this correctly, returning None itself so a caller can retry. We measured that Python's three built-in compression file wrappers do not: gzip.GzipFile, bz2.BZ2File and lzma.LZMAFile each crash outright on that same transient None, even when the underlying stream has real, complete, valid compressed data waiting on the very next read call. None of them retry; each fails with a different, misleading exception.

1. The source: real compressed data, one call away, behind a single transient None

import io

def make_flaky_raw(payload):
    class FlakyRaw(io.RawIOBase):
        def __init__(self):
            self.calls = 0
            self.sent = False
        def readable(self):
            return True
        def readinto(self, b):
            self.calls += 1
            if self.calls == 1:
                return None            # transient: no data ready yet
            if not self.sent:
                self.sent = True
                n = len(payload)
                b[:n] = payload
                return n               # the real data, one call later
            return 0                   # real EOF after that
    return FlakyRaw()

The payload is genuinely valid, complete compressed data for each format -- nothing is actually truncated or corrupt. The only flaw is a single None on the very first read call.

2. gzip.GzipFile: crashes with an opaque TypeError

import gzip

compressed = gzip.compress(b"hello world, this is real data")
raw = make_flaky_raw(compressed)
gz = gzip.GzipFile(fileobj=io.BufferedReader(raw))
try:
    print(gz.read())
except Exception as e:
    print("CRASHED:", type(e).__name__, e)

CRASHED: TypeError can't concat NoneType to bytes

Measured: the crash happens deep inside gzip.py's internal header reader, which concatenates self._buffer[read:] + self.file.read(...) without checking for None. The traceback points entirely at standard library internals, with no hint that the real cause was a transient, retryable signal from the caller's own stream.

3. bz2.BZ2File and lzma.LZMAFile: crash with a misleading EOFError instead

import bz2, lzma

for mod_name, compress, cls in [("bz2", bz2.compress, bz2.BZ2File), ("lzma", lzma.compress, lzma.LZMAFile)]:
    compressed = compress(b"hello world, this is real data")
    raw = make_flaky_raw(compressed)
    try:
        f = cls(io.BufferedReader(raw))
        print(mod_name, "result:", f.read())
    except Exception as e:
        print(mod_name, "CRASHED:", type(e).__name__, e)

bz2 CRASHED: EOFError Compressed file ended before the end-of-stream marker was reached
lzma CRASHED: EOFError Compressed file ended before the end-of-stream marker was reached

Measured: both treat the single None as if the stream had truly ended, raising EOFError claiming the file was truncated -- actively misleading, since the file is not truncated at all and the real data is sitting one read call away. A developer debugging this would reasonably suspect a corrupt file or a broken producer, not a transient non-blocking hiccup.

4. Contrast: plain BufferedReader.read() on the identical raw stream

raw2 = make_flaky_raw(b"irrelevant for this check")
br = io.BufferedReader(raw2)
print("plain read():", repr(br.read(5)))

plain read(): None

Measured: at the raw io layer, the exact same signal is preserved correctly as None, letting a caller retry. All three compression wrappers sit directly on top of this correctly-behaving layer, yet none of them propagate or handle the None themselves -- each lets it corrupt a different piece of internal bookkeeping instead.

5. The fix: don't hand a None-capable stream straight to these wrappers

None of gzip.GzipFile, bz2.BZ2File or lzma.LZMAFile expose a way to retry past a transient None -- the only robust fix is to read the compressed bytes yourself with a loop that retries on None, buffer them into a plain io.BytesIO, and only then hand the complete, in-memory buffer to the compression wrapper:

def drain_retrying_on_none(raw_stream, chunk_size=65536):
    buf = io.BytesIO()
    br = io.BufferedReader(raw_stream)
    while True:
        chunk = br.read(chunk_size)
        if chunk is None:
            continue              # retry: not ready yet, not EOF
        if not chunk:
            break                  # real EOF
        buf.write(chunk)
    buf.seek(0)
    return buf

raw3 = make_flaky_raw(gzip.compress(b"hello world, this is real data"))
safe_buf = drain_retrying_on_none(raw3)
gz2 = gzip.GzipFile(fileobj=safe_buf)
print("safe result:", gz2.read())

safe result: b'hello world, this is real data'

Measured: once the None/EOF distinction is handled before the data ever reaches the compression wrapper, decompression succeeds correctly.

Written with AI assistance. Measured on 2026-10-09 with Python 3.13.5's built-in gzip.GzipFile, bz2.BZ2File and lzma.LZMAFile, each wrapping an io.BufferedReader over a custom non-blocking RawIOBase whose readinto() returns None once (no data ready yet) before returning real, valid compressed data. PlainHeaders is our extension; Python's gzip, bz2 and lzma modules are built in.