pandas Error tokenizing data: Expected N fields (tested)

ParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 4 means one row has more separators than the header. error_bad_lines=False no longer exists; use on_bad_lines="skip", or better, fix the separator or quoting. Tested on pandas 2.3.3 and 3.1.0.

1. The error, and why error_bad_lines no longer works

import io
import pandas as pd

data = "a,b,c\n1,2,3\n4,5,6,7\n8,9,10\n"   # line 3 has 4 fields
try:
    pd.read_csv(io.StringIO(data))
except pd.errors.ParserError as e:
    print(type(e).__name__ + ":", e)

try:
    pd.read_csv(io.StringIO(data), error_bad_lines=False)
except TypeError as e:
    print("TypeError:", e)

df = pd.read_csv(io.StringIO(data), on_bad_lines="skip")
print(df.values.tolist())

Output (identical on pandas 2.3.3 and 3.1.0):

ParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 4

TypeError: read_csv() got an unexpected keyword argument 'error_bad_lines'. Did you mean 'on_bad_lines'?
[[1, 2, 3], [8, 9, 10]]

"Expected 3 fields in line 3, saw 4" counts lines from 1 including the header, so line 3 is the second data row. Old answers say error_bad_lines=False; that argument was removed in pandas 2.0 and now raises TypeError. The replacement is on_bad_lines="skip" (drop the row) or "warn" (drop it and print which line).

2. Keep or log bad rows instead of dropping them

import io
import pandas as pd

data = "a,b,c\n1,2,3\n4,5,6,7\n"
bad = []
def keep(fields):            # called with the split fields of each bad line
    bad.append(fields)
    return fields[:3]        # return a fixed row, or None to drop it

df = pd.read_csv(io.StringIO(data), on_bad_lines=keep, engine="python")
print(df.values.tolist())
print("bad:", bad)

Output (identical on pandas 2.3.3 and 3.1.0):

[[1, 2, 3], [4, 5, 6]]
bad: [['4', '5', '6', '7']]

on_bad_lines also accepts a function that receives the split fields and returns a repaired list, or None to drop the row. It needs engine="python" (or "pyarrow"); with the default C engine pandas raises ValueError: on_bad_line can only be a callable function if engine='python' or 'pyarrow' (we checked on both versions).

3. Fix the cause: separator, quoting, or header

import io
import pandas as pd

# 1) separator is ; but read with the default ,
semi = "name;note\nann;hi, there\nbob;ok\n"
print(pd.read_csv(io.StringIO(semi), sep=";").values.tolist())

# 2) a comma inside an unquoted field: on the FIRST data row there is no error,
#    pandas silently turns the extra leading field into the index
raw = "name,city\nann,Paris, France\n"
print(pd.read_csv(io.StringIO(raw)))
print(pd.read_csv(io.StringIO('name,city\nann,"Paris, France"\n')).values.tolist())

# 3) the header is shorter than later rows: give the column names yourself
short = "a,b\n1,2\n3,4,5\n"
print(pd.read_csv(io.StringIO(short), names=["a", "b", "c"], skiprows=1).values.tolist())

Output (identical on pandas 2.3.3 and 3.1.0):

[['ann', 'hi, there'], ['bob', 'ok']]
      name     city
ann  Paris   France
[['ann', 'Paris, France']]
[[1.0, 2.0, nan], [3.0, 4.0, 5.0]]

Skipping rows hides data loss, so check the cause first. European exports often use ;: pass sep=";". A comma inside a value needs quotes in the file. Note the second case: when every data row has the extra field there is no error at all; pandas makes the first field the index and shifts the rest, and on_bad_lines="skip" does not help (with two such rows it returned [['Paris', ' France'], ['Rome', ' Italy']]). If the header is simply shorter than the data, pass names= with skiprows=1.

Related: pandas KeyError but the column exists · csv: new-line character seen in unquoted field · csv: line contains NUL.

Written with AI assistance. Measured on 2026-10-23 in the python:3-alpine Docker image (Python 3.14) with pandas 2.3.3 and pandas 3.1.0. Each block was run verbatim on both versions; the outputs were identical and are pasted unedited. PlainHeaders is our extension; Python and pandas are free.