http.client resp.read() on a Transient None: 3 Different Failures (tested)

A non-blocking raw stream signals "no data ready yet" by returning None -- distinct from the empty read that means real end-of-stream. We fed http.client.HTTPResponse.read() (no argument -- the call behind most urlopen(url).read() code) one such None before a valid body. With Content-Length it crashed with TypeError: object of type 'NoneType' has no len(); with chunked encoding it raised IncompleteRead(0 bytes read); with no length it returned None and closed the connection. Three framings, three unrelated-looking errors, one cause.

1. One transient None, three framings, three different results

import http.client, io

class FlakyRaw(io.RawIOBase):
    """Sends headers cleanly, then one transient None, then the body."""
    def __init__(self, header_bytes, body_bytes):
        self.header_bytes, self.body_bytes, self.stage = header_bytes, body_bytes, 0
    def readable(self):
        return True
    def readinto(self, b):
        if self.stage == 0:
            n = len(self.header_bytes)
            b[:n] = self.header_bytes
            self.stage = 1
            return n
        if self.stage == 1:
            self.stage = 2
            return None            # transient: no data ready yet
        if self.stage == 2:
            n = len(self.body_bytes)
            b[:n] = self.body_bytes
            self.stage = 3
            return n               # the real, complete body
        return 0                   # real EOF after that

class FakeSocket:
    def __init__(self, raw):
        self.raw = raw
    def makefile(self, mode, *a, **kw):
        return io.BufferedReader(self.raw)
    def close(self):
        pass
    def sendall(self, data):
        pass

body = b"hello body"
cases = [
    ("Content-Length", b"HTTP/1.1 200 OK\r\nContent-Length: 10\r\n\r\n", body),
    ("chunked", b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n", b"a\r\nhello body\r\n0\r\n\r\n"),
    ("no length", b"HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", body),
]
for framing, headers, wire_body in cases:
    conn = http.client.HTTPConnection("example.com")
    conn.sock = FakeSocket(FlakyRaw(headers, wire_body))
    conn.request("GET", "/")
    resp = conn.getresponse()
    try:
        data = resp.read()
        print(f"{framing:15} returned {data!r}, connection closed: {resp.fp is None}")
    except Exception as e:
        print(f"{framing:15} {type(e).__name__}: {e}")

Content-Length  TypeError: object of type 'NoneType' has no len()
chunked         IncompleteRead: IncompleteRead(0 bytes read)
no length       returned None, connection closed: True

Measured: every body is valid and complete, one read away. The same resp.read() call crashed with a generic TypeError, raised IncompleteRead as if the server had cut the body short, or returned None -- which read() is documented never to return -- and closed the connection. Which one you get depends only on how the server framed the body.

2. Why each framing fails differently

Only the chunked case produces an http.client exception, and it blames the server ("incomplete read"). The Content-Length case points at a len() call inside the standard library. The no-length case raises nothing, so the failure surfaces later, wherever your code first treats the None as bytes. In all three the connection is unusable afterwards, so the body cannot be re-read.

3. The fix: stop the None below http.client

Append to the script above:

class RetryOnNone(io.RawIOBase):
    """Wraps a non-blocking raw stream; turns None into 'wait and retry'."""
    def __init__(self, raw):
        self.raw = raw
    def readable(self):
        return True
    def readinto(self, b):
        while True:
            n = self.raw.readinto(b)
            if n is not None:
                return n
            # real code: wait for readability here (e.g. selectors) instead of spinning

class FixedSocket(FakeSocket):
    def makefile(self, mode, *a, **kw):
        return io.BufferedReader(RetryOnNone(self.raw))

for framing, headers, wire_body in cases:
    conn = http.client.HTTPConnection("example.com")
    conn.sock = FixedSocket(FlakyRaw(headers, wire_body))
    conn.request("GET", "/")
    print(f"{framing:15}", conn.getresponse().read())

Content-Length  b'hello body'
chunked         b'hello body'
no length       b'hello body'

Measured: same flaky streams, and all three framings return the full body. The retry has to live in the raw stream that makefile() hands to http.client; once a None reaches HTTPResponse the connection is already lost. Simpler: if you control the socket, keep it blocking (or use settimeout()), which never returns None.

Related: HTTPResponse.read(amt) closes the connection on None (the sized-read case) and xmlrpc.client ServerProxy raising ExpatError on a valid reply.

Written with AI assistance. Measured on 2026-10-09 with Python 3.13.5's built-in http.client.HTTPResponse.read() (no argument), reading a complete, valid body framed three ways -- Content-Length, Transfer-Encoding: chunked, and read-until-close -- through a custom non-blocking socket whose underlying stream returns None once before the body. PlainHeaders is our extension; Python's http.client and io modules are built in.