codecs.StreamReader Crashes Concatenating None to Its Byte Buffer (tested)

A non-blocking raw stream signals "no data ready yet" by returning None from readinto() -- distinct from the empty read that means real end-of-stream. We measured that codecs.open() and codecs.getreader() -- the mechanism used for encodings that io.TextIOWrapper's fast path doesn't cover -- read via a pure-Python StreamReader.read() implementation that calls newdata = self.stream.read() and then immediately does data = self.bytebuffer + newdata with no check in between. When the underlying read returns a transient None, that line crashes with TypeError: can't concat NoneType to bytes -- an error that references neither codecs, encodings, nor files, on a payload that is valid and complete one read call away.

1. A real, valid text payload behind one transient None

import codecs, io

class FlakyRaw(io.RawIOBase):
    def __init__(self, payload):
        self.calls = 0
        self.sent = False
        self.payload = payload
    def readable(self):
        return True
    def readinto(self, b):
        self.calls += 1
        if self.calls == 1:
            return None            # transient: no data ready yet
        if not self.sent:
            self.sent = True
            n = len(self.payload)
            b[:n] = self.payload
            return n               # the real, complete text
        return 0                   # real EOF after that

raw = FlakyRaw(b"hello world\nline2\n")
br = io.BufferedReader(raw)
reader = codecs.getreader('utf-8')(br)
try:
    line = reader.readline()
    print("LINE:", repr(line))
except Exception as e:
    print("CRASHED:", type(e).__name__, "--", e)
print("raw calls made:", raw.calls)
CRASHED: TypeError -- can't concat NoneType to bytes
raw calls made: 1

Measured: the text is valid and complete, one read call away, and only one raw read happened before the crash. codecs.StreamReader.read() calls self.stream.read() and immediately concatenates the result onto its internal bytebuffer with self.bytebuffer + newdata -- no None check anywhere in that path, so the crash comes straight out of a bytes.__add__ call with no reference to codecs, encodings, or the file at all.

2. The fix: drain into a real buffer first, retrying on None

def read_retrying_on_none(buffered_reader, chunk_size=65536):
    out = io.BytesIO()
    while True:
        chunk = buffered_reader.read(chunk_size)
        if chunk is None:
            continue   # retry: not ready yet, not real EOF
        if not chunk:
            break      # real EOF
        out.write(chunk)
    out.seek(0)
    return out

raw2 = FlakyRaw(b"hello world\nline2\n")
br2 = io.BufferedReader(raw2)
safe_buf = read_retrying_on_none(br2)
reader2 = codecs.getreader('utf-8')(safe_buf)
print("line 1:", repr(reader2.readline()))
print("line 2:", repr(reader2.readline()))
line 1: 'hello world\n'
line 2: 'line2\n'

Measured: resolving the None/EOF distinction before the bytes reach the StreamReader decodes correctly. If you see TypeError: can't concat NoneType to bytes coming from code that uses codecs.open() or codecs.getreader(), the real cause is almost certainly a non-blocking stream behind the file object, not a bug in your encoding choice.

Frequently asked questions

Is this documented behavior?

codecs's documentation describes StreamReader as wrapping a file-like object that supports read(); it says nothing about None or non-blocking streams. We measured that a None-returning read() reaches a raw bytes concatenation with no type check, crashing before any decoding logic runs.

Why doesn't io.TextIOWrapper have this exact crash?

We measured that TextIOWrapper (the C-accelerated default for text mode) has its own separate buffering implementation that collapses a transient None from readline() into '' instead of crashing -- the already-confirmed silent-EOF flavor. codecs.StreamReader is a different, pure-Python implementation used for encodings TextIOWrapper's fast path doesn't natively handle, and it has no equivalent None-to-empty collapse at all -- it just crashes on the raw concatenation instead.

Does this affect codecs.open() the same way?

Yes -- we measured that codecs.open(filename, encoding=...) constructs exactly this StreamReader internally around the file it opens. Any non-blocking file-like object passed through that path hits the identical TypeError on a transient None.

Is this related to the other None/EOF findings?

Yes -- same family already confirmed in json.load(), quopri.decode(), shutil.copyfileobj, and others: a correctly-returned non-blocking None gets mishandled one layer above io itself. codecs.StreamReader is the second case in the family (after quopri.decode()'s binascii path) where the crash comes from a raw, module-unaware operation -- here a bytes concatenation -- rather than from any explicit type check.

Written with AI assistance. Measured on 2026-10-09 with Python 3.13.5's built-in codecs.getreader('utf-8') (the same StreamReader class codecs.open() uses internally), reading a genuinely valid, complete text payload through an io.BufferedReader over a custom non-blocking RawIOBase whose readinto() returns None once before returning the real bytes. PlainHeaders is our extension; Python's codecs and io modules are built in.