zlib decompressobj max_length Silent Data Loss (tested)

decompressobj().decompress(data, max_length) caps how much decompressed output you get back per call — and quietly leaves compressed bytes behind in unconsumed_tail for you to resubmit. We measured what happens when that resubmission step is skipped: no error, just most of your data missing.

1. max_length leaves bytes behind in unconsumed_tail

import zlib

data = b"hello world" * 1000
c = zlib.compress(data)

d = zlib.decompressobj()
out1 = d.decompress(c, 100)
len(out1)
100
len(d.unconsumed_tail)
27

Measured: capping output at 100 bytes with max_length doesn't mean 100 bytes of the compressed input were consumed — the decompressor stops as soon as it has enough output, and whatever compressed bytes it already read but didn't need yet sit in unconsumed_tail, waiting to be passed into the next call.

2. Skip unconsumed_tail and the rest of your data silently vanishes

out2 = d.decompress(b"")
len(out2)
170
out3 = d.flush()
len(out3)
0

len(out1) + len(out2) + len(out3)
270
len(data)
11000

Measured: calling decompress(b"") (or feeding it fresh data) instead of d.unconsumed_tail gets you only 270 bytes of a stream that should produce 11000 — the decompressor just stops silently once it runs out of what little it was actually given, with no exception and no warning that most of the data was never processed. flush() doesn't recover it either, because from the decompressor's point of view, the stream legitimately ended early; it has no way to know you were holding back the real continuation.

3. Feeding unconsumed_tail back gets everything

d2 = zlib.decompressobj()
chunks = []
out = d2.decompress(c, 100)
chunks.append(out)
while d2.unconsumed_tail:
    out = d2.decompress(d2.unconsumed_tail, 100)
    chunks.append(out)
chunks.append(d2.flush())
sum(len(x) for x in chunks)
11000

Measured: looping on d2.unconsumed_tail — feeding each call's leftover compressed bytes into the next call — reproduces the full 11000 bytes exactly. The loop condition itself is the fix: keep calling decompress(unconsumed_tail, max_length) until unconsumed_tail is empty, not until you've called it once.

4. What to actually do

If you pass max_length to decompressobj().decompress(), you must check unconsumed_tail after every call and feed it back into the next decompress() call until it's empty — it is not optional bookkeeping, it's the only thing carrying forward the compressed bytes you haven't processed yet. If you don't actually need to bound memory per call, skip max_length entirely and just call decompress(data) with no limit, which consumes everything in one call and avoids this failure mode altogether.

Frequently asked questions

Does zlib's decompressobj.decompress(data, max_length) raise an error if you ignore unconsumed_tail?

No — we measured it silently returning far less data than the original (270 bytes instead of 11000) with no exception and no warning when unconsumed_tail wasn't fed back in.

What is unconsumed_tail actually for?

We measured it holding compressed input bytes that decompressobj already read but didn't need to produce the current max_length-capped output — it's the only place those bytes exist; they're not re-derivable afterward.

Does flush() recover data lost by skipping unconsumed_tail?

No — we measured flush() returning 0 extra bytes in that case, because from the decompressor's perspective the stream legitimately ended; it has no way to know more compressed data was being withheld.

How do I correctly decompress in bounded chunks with max_length?

We'd loop: call decompress(data, max_length), then keep calling decompress(unconsumed_tail, max_length) — feeding each call's leftover back in — until unconsumed_tail is empty, then call flush() once at the end.

Is this unconsumed_tail behavior documented?

Yes — the Python zlib docs describe max_length as bounding output size and unconsumed_tail as the data not yet consumed after such a call. We confirmed skipping it causes exactly the silent truncation described here in Python 3.13.5.

Written with AI assistance. Measured on 2026-10-03 with Python 3.13.5's built-in zlib module: compressed 11000 bytes, decompressed it in chunks using decompressobj().decompress(data, max_length), then fed the next decompress() call empty bytes instead of the object's own unconsumed_tail, and measured the total output size against the original. PlainHeaders is our extension; Python's zlib module is built in.