UnicodeDecodeError: invalid start byte (tested fix)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte means the file is not UTF-8 text. The byte value tells you what it is instead: 0xff/0xfe is UTF-16, 0x80-0x9f is usually Windows-1252, and 0x8b or 0x89 means a binary file. We reproduced each case and tested the fix.

1. Byte 0xff in position 0: a UTF-16 file

# Excel/Windows "Unicode text" export: UTF-16 with a BOM (bytes ff fe)
with open("export.txt", "w", encoding="utf-16") as f:
    f.write("name\tcity\nZoë\tKöln\n")
print(open("export.txt", "rb").read()[:8])
try:
    open("export.txt").read()          # default encoding is UTF-8
except UnicodeDecodeError as e:
    print(f"{type(e).__name__}: {e}")
print(repr(open("export.txt", encoding="utf-16").read()))

Output:

b'\xff\xfen\x00a\x00m\x00'
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
'name\tcity\nZoë\tKöln\n'

0xff 0xfe at the very start is the UTF-16 little-endian byte order mark. Excel's "Unicode Text" export and many Windows tools write this, and every second byte is 0x00. No UTF-8 sequence can start with 0xff, so decoding fails at position 0. Open it with encoding="utf-16", which reads the BOM and picks the byte order. (0xfe in position 0 is the same thing, big-endian.)

2. 0x8b, 0x89, 0xa2...: a binary file opened as text

# Binary files opened in text mode: gzip, PNG, xlsx (a zip)
import gzip, zipfile
gzip.open("data.csv.gz", "wt").write("a,b\n1,2\n")
with zipfile.ZipFile("book.xlsx", "w") as z: z.writestr("x", "y")
open("logo.png", "wb").write(b"\x89PNG\r\n\x1a\n" + bytes(8))
for name in ["data.csv.gz", "book.xlsx", "logo.png"]:
    try:
        open(name).read()
    except UnicodeDecodeError as e:
        print(f"{name:12} {open(name,'rb').read(4)!r:22} {e}")
print(gzip.open("data.csv.gz", "rt").read())

Output:

data.csv.gz  b'\x1f\x8b\x08\x08'    'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
book.xlsx    b'PK\x03\x04'          'utf-8' codec can't decode byte 0xa2 in position 10: invalid start byte
logo.png     b'\x89PNG'             'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
a,b
1,2

A gzip file starts with 1f 8b, so it fails at position 1; a PNG starts with 0x89. An .xlsx file is a zip archive, so it fails at whatever compressed byte comes first (position 10 here; it varies by file). These are not text files with the wrong encoding, so no encoding= value fixes them. Open them with the right reader: gzip.open(path, "rt"), zipfile or pandas.read_excel, or open(path, "rb") for raw bytes.

3. Byte 0x80 (or 0x92, 0x96...): a Windows-1252 file

# Windows-1252 text: the euro sign is the single byte 0x80
open("prices.csv", "w", encoding="cp1252").write("item,price\nTea,€3\n")
try:
    open("prices.csv", encoding="utf-8").read()
except UnicodeDecodeError as e:
    print(e)
print(repr(open("prices.csv", encoding="cp1252").read()))

Output:

'utf-8' codec can't decode byte 0x80 in position 15: invalid start byte
'item,price\nTea,€3\n'

In Windows-1252 the euro sign, curly quotes and dashes are single bytes from 0x80 to 0x9f. UTF-8 only uses those bytes inside multi-byte characters, never at the start, so they produce "invalid start byte". Use encoding="cp1252". (Accented letters like 0xe9 give "invalid continuation byte" instead; see UnicodeDecodeError: all causes.)

4. Check the first bytes instead of guessing

# Sniff the first bytes before guessing an encoding
import codecs
def sniff(path):
    head = open(path, "rb").read(4)
    if head.startswith(codecs.BOM_UTF8): return "utf-8-sig"
    if head.startswith((codecs.BOM_UTF16_LE, codecs.BOM_UTF16_BE)): return "utf-16"
    if head.startswith(b"\x1f\x8b"): return "gzip"
    if head.startswith(b"PK\x03\x04"): return "zip (xlsx/docx)"
    if head.startswith(b"\x89PNG"): return "png"
    return "try utf-8, then cp1252"
for p in ["export.txt", "data.csv.gz", "book.xlsx", "logo.png"]:
    print(f"{p:12} -> {sniff(p)}")

Output:

export.txt   -> utf-16
data.csv.gz  -> gzip
book.xlsx    -> zip (xlsx/docx)
logo.png     -> png

Reading the first four bytes in binary mode tells you which case you have. pandas.read_csv takes the same encoding= argument as open(), so once you know the encoding, pass it there too.

5. Why errors="replace" or latin-1 is not a fix

# errors="replace" hides the problem: UTF-16 read as Latin-1 "works"
text = open("export.txt", encoding="latin-1").read()
print(repr(text[:12]))
text = open("export.txt", errors="replace").read()
print(repr(text[:12]))

Output:

'ÿþn\x00a\x00m\x00e\x00\t\x00'
'��n\x00a\x00m\x00e\x00\t\x00'

Both stop the exception, but the UTF-16 text comes out as mojibake with \x00 between letters. latin-1 never raises because every byte maps to some character, which is why it silently gives the wrong text.

Related: UnicodeDecodeError (4 causes) · subprocess text=True UnicodeDecodeError.

Written with AI assistance. Measured on 2026-10-23 with Python 3.13.5 on Debian: a UTF-16 file with a BOM, gzip/xlsx/PNG files opened as text, a Windows-1252 file with a euro sign, a first-bytes sniffer, and errors="replace". Every block was run verbatim and its output pasted unedited. PlainHeaders is our extension; Python is free.