base64.encode()/decode() Silently Return Empty Output on None (tested)

A non-blocking raw stream signals "no data ready yet" by returning None from readinto() -- distinct from the empty read that means real end-of-stream. Both base64.decode() and base64.encode() loop with a truthy while check (while line := input.readline(): and while s := input.read(MAXBINSIZE):), and None is just as falsy as b'' to that check -- so a single transient None stops either loop immediately. Even though io.BufferedReader.read() correctly returns None (unlike readline(), which collapses it to b''), the common while chunk := f.read(n): idiom used by encode() mishandles the correctly-returned None just as badly -- both functions write zero bytes with no exception, even with a complete, valid payload one read call away.

1. decode(): readline()'s None-to-b'' collapse silently empties the output

import io, base64

class FlakyRaw(io.RawIOBase):
    def __init__(self, payload):
        self.calls = 0
        self.sent = False
        self.payload = payload
    def readable(self):
        return True
    def readinto(self, b):
        self.calls += 1
        if self.calls == 1:
            return None            # transient: no data ready yet
        if not self.sent:
            self.sent = True
            n = len(self.payload)
            b[:n] = self.payload
            return n               # the real, complete data
        return 0                   # real EOF after that

payload = base64.b64encode(b"hello world, this is a real payload") + b"\n"
raw = FlakyRaw(payload)
br = io.BufferedReader(raw)
out = io.BytesIO()
base64.decode(br, out)
print("decoded:", repr(out.getvalue()))
print("raw calls made:", raw.calls)

decoded: b''
raw calls made: 1

Measured: no exception anywhere. decode()'s while line := input.readline(): loop made one call, got back b'' (readline()'s collapse of the transient None), and silently concluded the file was empty.

2. encode(): even read()'s correct None return breaks the same truthy-while idiom

payload2 = b"hello world, this is a real payload for encoding"
raw2 = FlakyRaw(payload2)
br2 = io.BufferedReader(raw2)
out2 = io.BytesIO()
base64.encode(br2, out2)
print("encoded:", repr(out2.getvalue()))
print("raw calls made:", raw2.calls)

encoded: b''
raw calls made: 1

Measured: encode() uses while s := input.read(MAXBINSIZE):, and read() correctly returns None for the transient signal here -- but None is exactly as falsy as b'' to a while check. The one documented, correct way io has to signal "not ready yet" is silently swallowed by the single most common idiom for draining a file-like object in Python.

3. The fix: drain into a real buffer first, retrying on None explicitly

def drain_retrying_on_none(raw_stream, chunk_size=65536):
    out = io.BytesIO()
    br = io.BufferedReader(raw_stream)
    while True:
        chunk = br.read(chunk_size)
        if chunk is None:
            continue              # retry: not ready yet, not EOF
        if not chunk:
            break                  # real EOF
        out.write(chunk)
    out.seek(0)
    return out

raw3 = FlakyRaw(payload)
safe_buf = drain_retrying_on_none(raw3)
out3 = io.BytesIO()
base64.decode(safe_buf, out3)
print("decoded:", repr(out3.getvalue()))

raw4 = FlakyRaw(payload2)
safe_buf2 = drain_retrying_on_none(raw4)
out4 = io.BytesIO()
base64.encode(safe_buf2, out4)
print("encoded:", repr(out4.getvalue()))

decoded: b'hello world, this is a real payload'
encoded: b'aGVsbG8gd29ybGQsIHRoaXMgaXMgYSByZWFsIHBheWxvYWQgZm9yIGVuY29kaW5n\n'

Measured: resolving the None/EOF distinction explicitly before the bytes reach either function produces correct output for both.

Frequently asked questions

Is this documented behavior?

The base64 module's documentation for encode() and decode() says they operate on binary files; it says nothing about non-blocking streams or a None read result. We measured that a None-returning source produces silently empty results on both, with no error of any kind.

Doesn't io.BufferedReader.read() correctly return None, unlike readline()?

Yes -- we measured that elsewhere separately. The surprise here is that encode() uses the standard while chunk := f.read(n): idiom, and that idiom treats None exactly like b'' because both are falsy. Having read() return the 'correct' signal doesn't help if the calling code's loop condition can't tell None apart from real EOF.

So is there any safe way to loop on read() results?

Yes -- check explicitly: while True: chunk = f.read(n); if chunk is None: continue (retry); if not chunk: break (real EOF); process(chunk). The walrus-while shorthand is concise but silently conflates the two cases.

Is this related to the other None/EOF findings?

Yes -- decode()'s failure is the same io.BufferedReader.readline() collapse already confirmed in http.client, smtplib, and poplib. encode()'s failure is new: it's the first case in the family where read()'s correct None return still causes a silent failure, because of how the calling code's loop is written -- a reminder that the None convention only helps if every caller checks for it explicitly.

Written with AI assistance. Measured on 2026-10-09 with Python 3.13.5's built-in base64.encode() and base64.decode(), reading genuinely valid, complete payloads through an io.BufferedReader over a custom non-blocking RawIOBase whose readinto() returns None once before returning the real bytes. PlainHeaders is our extension; Python's base64 and io modules are built in.