pandas groupby FutureWarning: apply on grouping columns, observed=False (tested)

pandas 2 warns DataFrameGroupBy.apply operated on the grouping columns and The default of observed=False is deprecated. We ran the same code on pandas 3.1.0: both warnings are gone and the results are different. Here are the tested fixes that give the same output on both versions.

1. Two groupby FutureWarnings, and what pandas 3 does instead

import warnings
import pandas as pd
warnings.simplefilter("always")

df = pd.DataFrame({"g": ["a", "a", "b"], "v": [1, 2, 3]})
r = df.groupby("g").apply(lambda x: x["v"].sum() + len(x.columns))
print(r.tolist())

c = df.assign(g=df["g"].astype(pd.CategoricalDtype(["a", "b", "z"])))
print(c.groupby("g")["v"].sum().to_dict())

Output with pandas 2.3.3:

/w/b1.py:6: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  r = df.groupby("g").apply(lambda x: x["v"].sum() + len(x.columns))
/w/b1.py:10: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  print(c.groupby("g")["v"].sum().to_dict())
[5, 5]
{'a': 3, 'b': 3, 'z': 0}

Output with pandas 3.1.0:

[4, 4]
{'a': 3, 'b': 3}

Neither warning is cosmetic: on pandas 3.1.0 the same code returns different results without any warning. apply no longer passes the grouping column g to your function (so len(x.columns) drops from 2 to 1 and the sums change from 5 to 4), and grouping by a categorical column no longer lists empty categories ('z': 0 is gone).

2. Fix: say which behaviour you want

import warnings
import pandas as pd
warnings.simplefilter("always")

df = pd.DataFrame({"g": ["a", "a", "b"], "v": [1, 2, 3]})
r = df.groupby("g")[["v"]].apply(lambda x: x["v"].sum() + len(x.columns))   # select columns
print(r.tolist())
r = df.groupby("g").apply(lambda x: x["v"].sum() + len(x.columns), include_groups=False)
print(r.tolist())

c = df.assign(g=df["g"].astype(pd.CategoricalDtype(["a", "b", "z"])))
print(c.groupby("g", observed=True)["v"].sum().to_dict())    # only groups that occur
print(c.groupby("g", observed=False)["v"].sum().to_dict())   # keep empty categories

Output (identical on pandas 2.3.3 and 3.1.0, no warnings):

[4, 4]
[4, 4]
{'a': 3, 'b': 3}
{'a': 3, 'b': 3, 'z': 0}

For "DataFrameGroupBy.apply operated on the grouping columns": select the columns you need after groupby (df.groupby("g")[["v"]].apply(...)) or pass include_groups=False. If your function really needs the group key, read it from x.name inside the function. Do not pass include_groups=True to keep the old behaviour: on pandas 3.1.0 that raises ValueError: include_groups=True is no longer allowed. (we checked).

For "The default of observed=False is deprecated": pass observed=True to get only the groups that occur (the pandas 3 default), or observed=False if a report needs a row for every category, including empty ones. The warning only appears when grouping by a category column.

Related: all pandas 3 breaking changes · FutureWarning: downcasting on fillna · FutureWarning: concat with empty or all-NA entries · SettingWithCopyWarning.

Written with AI assistance. Measured on 2026-10-23 in the python:3-alpine Docker image (Python 3.14) with pandas 2.3.3 and pandas 3.1.0. Each block was run verbatim on both versions with warnings.simplefilter("always") and the output pasted unedited. PlainHeaders is our extension; Python and pandas are free.