http.client.HTTPResponse Mistakes None for RemoteDisconnected (tested)

A non-blocking raw stream signals "no data ready yet" by returning None from readinto() -- distinct from the empty read that means real end-of-stream. We measured that http.client.HTTPResponse.begin() reads its status line with self.fp.readline(), and io.BufferedReader.readline() itself collapses that transient None into b'' -- indistinguishable from a socket that closed before sending anything. begin() sees the empty line and raises http.client.RemoteDisconnected: Remote end closed connection without response, even though a complete, valid 200 OK response is sitting one read call away and the connection was never actually closed.

1. A real, complete response behind one transient None

import io, http.client

class FlakyRaw(io.RawIOBase):
    def __init__(self, payload):
        self.calls = 0
        self.sent = False
        self.payload = payload
    def readable(self):
        return True
    def readinto(self, b):
        self.calls += 1
        if self.calls == 1:
            return None            # transient: no data ready yet
        if not self.sent:
            self.sent = True
            n = len(self.payload)
            b[:n] = self.payload
            return n               # the real, complete response
        return 0                   # real EOF after that

class FakeSocket:
    def __init__(self, payload):
        self.raw = FlakyRaw(payload)
        self.br = io.BufferedReader(self.raw)
    def makefile(self, mode, *a, **kw):
        return self.br

payload = (b"HTTP/1.1 200 OK\r\n"
           b"Content-Type: text/plain\r\n"
           b"Content-Length: 5\r\n"
           b"\r\n"
           b"hello")

sock = FakeSocket(payload)
resp = http.client.HTTPResponse(sock)
try:
    resp.begin()
    print("status:", resp.status)
except Exception as e:
    print("CRASHED:", type(e).__name__, e)
print("raw calls made:", sock.raw.calls)
CRASHED: RemoteDisconnected Remote end closed connection without response
raw calls made: 1

Measured: the response is valid and complete, one read call away. HTTPResponse.begin() made exactly one read call, got back an empty status line, and concluded the server closed the connection -- a conclusion that looks like a real network failure but is actually just an unhandled non-blocking retry signal.

2. The root cause: BufferedReader.readline() collapses None into b'', unlike read()

raw1 = FlakyRaw(b"HTTP/1.1 200 OK\r\n")
br1 = io.BufferedReader(raw1)
print("readline():", repr(br1.readline()))

raw2 = FlakyRaw(b"HTTP/1.1 200 OK\r\n")
br2 = io.BufferedReader(raw2)
print("read(5):", repr(br2.read(5)))
readline(): b''
read(5): None

Measured: the same class handles the identical transient None two different ways depending on which method is called. read() propagates it correctly so a caller can retry; readline() -- what http.client actually uses to read the status line -- silently turns it into b'', the same value a genuinely closed connection would produce.

3. The fix: drain into a real buffer first, retrying on None

def drain_retrying_on_none(raw_stream, chunk_size=65536):
    out = io.BytesIO()
    br = io.BufferedReader(raw_stream)
    while True:
        chunk = br.read(chunk_size)
        if chunk is None:
            continue              # retry: not ready yet, not EOF
        if not chunk:
            break                  # real EOF
        out.write(chunk)
    out.seek(0)
    return out

class FakeSocket2:
    def __init__(self, fileobj):
        self.fileobj = fileobj
    def makefile(self, mode, *a, **kw):
        return self.fileobj

raw3 = FlakyRaw(payload)
safe_buf = drain_retrying_on_none(raw3)
sock2 = FakeSocket2(safe_buf)
resp2 = http.client.HTTPResponse(sock2)
resp2.begin()
print("status:", resp2.status)
print("body:", resp2.read())
status: 200
body: b'hello'

Measured: resolving the None/EOF distinction before the bytes reach HTTPResponse.begin() parses the status line and body correctly, with no spurious disconnection.

Frequently asked questions

Is this documented behavior?

The http.client documentation for HTTPResponse does not mention non-blocking sockets or a None read result; RemoteDisconnected is documented as meaning the server closed the connection before sending a response. We measured that a None-returning source triggers the identical exception with no actual disconnection involved.

Does this affect urllib.request.urlopen(), not just raw http.client?

Yes -- we measured it directly: urlopen() builds its request on top of the same http.client.HTTPConnection and HTTPResponse, and a non-blocking socket wrapped incorrectly at that layer surfaces the identical RemoteDisconnected exception through urlopen() too, with the same misleading message. We did not test requests or urllib3 directly, but both are also built on http.client's HTTPResponse for the underlying socket read, so the same failure is plausible through them as well (possibly surfaced as a subclass like requests' ConnectionError).

Why does this look like a real network problem?

Because RemoteDisconnected is also the genuine exception for an actually-closed connection, there is no way to distinguish the two causes from the exception alone. If you see this on a socket you know is still open (e.g. a non-blocking socket you're polling), suspect a transient None read instead of a real disconnection.

Is this related to the other None/EOF findings?

Yes -- same family already confirmed in shutil.copyfileobj, gzip/bz2/lzma, tarfile/zipfile, TextIOWrapper.readline(), ElementTree.parse(), tomllib.load(), email.message_from_binary_file(), and plistlib.load(): a correctly-returned non-blocking None gets mishandled one layer above io itself. This is the first case in the family traced all the way down to io.BufferedReader.readline() itself doing the collapsing, rather than a higher-level module's own loop.

Written with AI assistance. Measured on 2026-10-09 with Python 3.13.5's built-in http.client.HTTPResponse, reading a genuinely complete, valid HTTP response through a fake socket whose makefile() returns an io.BufferedReader over a custom non-blocking RawIOBase whose readinto() returns None once before returning the real response bytes. PlainHeaders is our extension; Python's http.client and io modules are built in.