Python UnicodeDecodeError (tested)

UnicodeDecodeError means Python tried to interpret a sequence of bytes as text using a specific encoding (usually UTF-8, Python's default), and those bytes aren't valid under that encoding. The byte data itself is fine — it's just encoded differently than the code assumed. We wrote real byte files on disk and reproduced four distinct causes.

Getting invalid start byte with 0xff, 0x80 or 0x8b? See UTF-16, binary and cp1252 files: invalid start byte.

1. Four reproduced causes

Each message names the specific codec that failed, the exact byte value, its position in the data, and a short reason (invalid continuation byte, unexpected end of data, invalid start byte) — which, read together, usually tells you which real cause below you're hitting.

2. Cause 1: the file isn't UTF-8 at all

By far the most common real cause: a file was actually saved in Latin-1 (cp1252/ISO-8859-1, common on older Windows tools and some databases) or another encoding, but the code assumes UTF-8, Python's default for open():

>>> open("file.txt", encoding="utf-8").read()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte

Fix: open with the correct encoding if you know it (encoding="latin-1" or encoding="cp1252"), or detect it with a library like charset-normalizer if the source is unpredictable. "invalid continuation byte" specifically means a UTF-8 multi-byte sequence started correctly but the next byte didn't match what UTF-8 requires — a strong signal the file is single-byte-encoded, not UTF-8.

3. Cause 2: truncated or corrupted data

"unexpected end of data" means a multi-byte UTF-8 character was cut off before it finished — the file ends (or a network read stopped) mid-character:

>>> open("truncated.txt", encoding="utf-8").read()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 3: unexpected end of data

This happens with partial file downloads, a network stream read in fixed-size chunks that happens to split a multi-byte character across two chunks, or a file genuinely truncated by a crashed write. Fix: for streaming reads, decode incrementally with codecs.getincrementaldecoder("utf-8")() instead of decoding each chunk independently, so a split character is buffered until the next chunk completes it.

4. Cause 3: using the wrong codec on purpose (or by default)

ascii only accepts byte values 0-127. Anything outside that — including any accented letter, emoji, or non-Latin script — fails:

>>> b'Caf\xe9'.decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 3: ordinal not in range(128)

This can show up unexpectedly in older code, or in environments where Python 2-era str.decode() calls or a locale-based default encoding silently picked ascii. Fix: decode with utf-8 explicitly rather than relying on an implicit default, especially when deploying to a different machine with a different system locale.

5. Cause 4: the data is a different Unicode encoding entirely (UTF-16, UTF-32)

"invalid start byte", especially right at position 0, often means the file is a different multi-byte Unicode encoding, not a single-byte one:

>>> open("utf16file.txt", encoding="utf-8").read()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

0xff 0xfe (or 0xfe 0xff) at the very start of a file is a byte-order mark (BOM) that signals UTF-16. Fix: try encoding="utf-16" (Python auto-detects byte order from the BOM), or check the first 2-4 bytes of the file yourself before choosing a codec.

6. Fix checklist

• Know your data's actual encoding before choosing one in code — don't assume UTF-8 for files from Windows tools, legacy databases, or unknown external sources.
• Use errors="replace" or errors="ignore" in open()/.decode() only when losing or mangling the bad bytes is genuinely acceptable — it silently discards information otherwise.
• For a quick, honest fallback that doesn't lose data: errors="surrogateescape" round-trips unknown bytes losslessly if you'll re-encode with the same setting later.
• For streamed/chunked reads, use an incremental decoder instead of decoding each chunk on its own.
• Check the first few bytes for a BOM (\xff\xfe, \xfe\xff, \xef\xbb\xbf for UTF-8-with-BOM) before assuming a plain encoding.

This is purely about byte-level text decoding, unrelated to ValueError's type-conversion failures like int("abc") — though both report "this specific value isn't valid here" rather than a wrong Python type.

Written with AI assistance. Measured on 2026-10-03 with Python 3.13.5 on Linux, reading real byte files written under /tmp. Every error below is copy-pasted output, not paraphrased. PlainHeaders is our extension; Python is free.