io.TextIOWrapper.detach() Leaves the Raw Stream Past Your Last Read (tested)

io.TextIOWrapper reads ahead from its underlying binary buffer in chunks for efficiency -- it doesn't pull exactly as many bytes as you ask for in characters. After detach() separates the raw buffer back out, that raw stream's position reflects how far the wrapper's internal read-ahead actually got, not how many characters you've consumed through the text interface. Code that reads a text header with the wrapper and then detaches to continue reading the rest of the stream in binary mode -- a common pattern for mixed text/binary protocols -- can silently skip or lose data, because the raw stream is already positioned well past where the text reading appeared to stop.

1. Reading 5 characters consumes the whole underlying buffer

import io

data = b"hello world, this is more than five bytes of data here"
buf = io.BytesIO(data)
text = io.TextIOWrapper(buf, encoding="utf-8")

chunk = text.read(5)
print("text.read(5):", repr(chunk))
print("buf.tell():", buf.tell(), "of", len(data))

text.read(5): 'hello'
buf.tell(): 54 of 54

Measured: text.read(5) returns exactly the 5 characters asked for, but the underlying BytesIO's position is already at byte 54 -- the very end of the 54-byte buffer. TextIOWrapper read the entire buffer into its own internal cache in one shot and served the first 5 characters out of that cache; buf's position reflects the read-ahead, not the 5 characters actually returned.

2. detach() hands back the raw stream at that same, already-advanced position

raw = text.detach()
print("raw is buf:", raw is buf)
print("raw.tell():", raw.tell())
print("raw.read():", raw.read())

raw is buf: True
raw.tell(): 54
raw.read(): b''

Measured: detach() returns the exact same BytesIO object, still sitting at position 54. Calling .read() on it afterward returns nothing at all (b'') -- not "the bytes after the 5 characters we already consumed," which is what code written to switch from text to binary mid-stream is usually assuming.

3. The read-ahead amount depends on buffer/chunk size, not a fixed rule

data2 = b"x" * 20000  # bigger than TextIOWrapper's default read-ahead chunk
buf2 = io.BytesIO(data2)
text2 = io.TextIOWrapper(buf2, encoding="utf-8")
text2.read(1)
print("buf2.tell() after reading 1 char from a 20000-byte buffer:", buf2.tell())

buf2.tell() after reading 1 char from a 20000-byte buffer: 8192

Measured: with a larger source, the raw position after a single-character text read lands at 8192 (the default read-ahead chunk size), not at the end of the buffer and not at byte 1 -- confirming the raw position after detach() is governed by internal buffering chunk size, not by anything about how much text you've actually read. There is no general formula to recover "my real text position" in raw-stream terms from the outside.

4. The fix: don't mix text reads with detach-and-continue-in-binary

buf3 = io.BytesIO(b"header line here
binary body bytes")
header = buf3.readline()  # read just the header in binary first
text3 = io.TextIOWrapper(io.BytesIO(header), encoding="utf-8")
print("header as text:", repr(text3.read()))
print("remaining binary body:", buf3.read())

header as text: 'header line here
'
remaining binary body: b'binary body bytes'

Measured: reading the exact bytes you need in binary mode first (e.g. with readline() on the raw stream, which does know its own precise byte position), then decoding only that slice into a separate, throwaway TextIOWrapper, avoids the problem entirely -- buf3's position stays accurate because it was never wrapped in a TextIOWrapper that could read ahead on it, so the rest of the binary body comes through intact.

Frequently asked questions

Why is the raw stream positioned past my last text read after detach()?

We measured that TextIOWrapper reads ahead from the underlying binary buffer in large chunks (its internal read-ahead size, by default up to 8192 bytes or the whole source if smaller) regardless of how many characters you actually request. detach() returns that same raw stream, still sitting wherever the read-ahead left it, not at a position corresponding to the characters you've consumed.

Can I compute the correct raw byte offset after a text read to fix this?

Not reliably from the public API -- we measured the raw position depending on TextIOWrapper's internal chunk size relative to the data size, which isn't something the text interface exposes or guarantees to stay consistent across reads.

How do I safely switch from text to binary reading mid-stream?

We measured that reading the exact bytes you need directly from the raw/binary stream first (e.g. via readline() on a buffered binary reader, which tracks its own byte position precisely) and only decoding that slice into text afterward avoids the problem -- never let TextIOWrapper read ahead on a stream you plan to detach and continue reading in binary.

Does this affect every TextIOWrapper, or only ones wrapping BytesIO?

The read-ahead behavior is part of TextIOWrapper itself, not specific to BytesIO -- we used BytesIO because its tell() makes the raw position directly observable, but the same read-ahead applies to any binary stream (files, sockets, pipes) wrapped by TextIOWrapper.

Written with AI assistance. Measured on 2026-10-09 with Python 3.13.5's built-in io.TextIOWrapper: wrapped a 54-byte io.BytesIO, read only the first 5 characters through the text wrapper, then called detach() and checked the underlying BytesIO's position directly with tell(). PlainHeaders is our extension; Python's io module is built in.