pandas PerformanceWarning: DataFrame is highly fragmented (tested)

PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times appears when you add columns one by one in a loop. We reproduced it (120 inserts, 21 warnings) and tested which fixes remove it, on pandas 2.3.3 and 3.1.0.

1. Reproducing the warning

import warnings
import pandas as pd
warnings.simplefilter("always")

df = pd.DataFrame({"base": range(3)})
with warnings.catch_warnings(record=True) as w:
    for i in range(120):
        df[f"c{i}"] = df["base"] * i          # one new column per iteration
print(len(w), "warnings;", w[0].category.__name__ if w else None)
print(str(w[0].message)[:120] if w else "")
print(df.shape)

Output (identical on pandas 2.3.3 and 3.1.0):

21 warnings; PerformanceWarning
DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor perform
(3, 121)

Each df["new"] = ... (or df.insert) stores the new column as a separate internal block. Once a DataFrame has more than about 100 of them pandas emits PerformanceWarning on every further insert: here 120 inserts gave 21 warnings. The data is correct; the warning is about speed, because later operations have to combine all those blocks.

2. Fix: add the columns in one step

import warnings
import pandas as pd
warnings.simplefilter("always")

df = pd.DataFrame({"base": range(3)})
with warnings.catch_warnings(record=True) as w:
    new = {f"c{i}": df["base"] * i for i in range(120)}   # build all columns first
    df = pd.concat([df, pd.DataFrame(new)], axis=1)       # add them in one step
print(len(w), "warnings", df.shape)

df2 = pd.DataFrame({"base": range(3)})
with warnings.catch_warnings(record=True) as w:
    for i in range(120):
        df2[f"c{i}"] = df2["base"] * i
    df2 = df2.copy()                                      # de-fragment afterwards
print(len(w) > 0, df2.shape)

Output (identical on pandas 2.3.3 and 3.1.0):

0 warnings (3, 121)
True (3, 121)

Build the new columns in a dict (or a list of Series), then add them with a single pd.concat([df, pd.DataFrame(new)], axis=1): no warning at all on either version. If the loop is hard to restructure, call df = df.copy() after it to consolidate the blocks; the warnings during the loop still fire (True above), but later operations are fast again. Hiding the warning with warnings.filterwarnings leaves the slowdown in place.

Related: FutureWarning: concat with empty or all-NA entries · SettingWithCopyWarning · groupby FutureWarning.

Written with AI assistance. Measured on 2026-10-23 in the python:3-alpine Docker image (Python 3.14) with pandas 2.3.3 and pandas 3.1.0. Each block was run verbatim on both versions with warnings.simplefilter("always"); the outputs were identical and are pasted unedited. PlainHeaders is our extension; Python and pandas are free.