MemoryError: Unable to allocate X GiB for an array (NumPy / pandas, tested)

MemoryError: Unable to allocate 298. GiB for an array with shape (200000, 200000) and data type float64 is NumPy refusing an allocation it computed up front. If the size is absurd, the shape is wrong, often from a merge on duplicate keys. Otherwise shrink dtypes or read in chunks. Measured on pandas 2.3.3 and 3.1.0.

1. What the number means

import numpy as np
try:
    a = np.zeros((200_000, 200_000))
except MemoryError as e:
    print(type(e).__name__, "is a MemoryError:", isinstance(e, MemoryError))
    print(e)
print(200_000 * 200_000 * 8 / 2**30, "GiB = rows * cols * 8 bytes (float64)")

Output (NumPy 2.5.4; identical with pandas 2.3.3 and 3.1.0 installed):

MemoryError is a MemoryError: True
Unable to allocate 298. GiB for an array with shape (200000, 200000) and data type float64
298.0232238769531 GiB = rows * cols * 8 bytes (float64)

Uncaught, the last traceback line reads numpy._core._exceptions._ArrayMemoryError: Unable to allocate 298. GiB .... It is a subclass of MemoryError, so except MemoryError catches it. The size is just rows × columns × bytes per item, computed before any memory is used. When the number is far bigger than your data could possibly be, the shape is wrong, not your RAM: look at the shape in the message and find what made it that big.

2. The usual hidden cause: a merge on duplicate keys

import pandas as pd
orders = pd.DataFrame({"cust": [1] * 2000, "order": range(2000)})
visits = pd.DataFrame({"cust": [1] * 3000, "page": range(3000)})
# rows the merge will produce, computed before merging
n = (orders["cust"].value_counts() * visits["cust"].value_counts()).sum()
print("merge would create", n, "rows")
print(len(orders.merge(visits, on="cust")))
try:
    orders.merge(visits, on="cust", validate="many_to_one")
except pd.errors.MergeError as e:
    print("MergeError:", e)

Output with pandas 2.3.3:

merge would create 6000000 rows
6000000
MergeError: Merge keys are not unique in right dataset; not a many-to-one merge

Output with pandas 3.1.0:

merge would create 6000000 rows
6000000
MergeError: Merge keys are not unique in right dataset; not a many-to-one merge

Duplicates in right:
  cust
    1
    1
    1
    1
    1 ...

2,000 rows merged with 3,000 rows became 6,000,000, because every duplicate key pairs with every other. On real data that turns into "Unable to allocate" inside the merge or in the next step. Multiply the per-key counts before merging to see the row count, and pass validate="one_to_one" or "many_to_one" so pandas raises MergeError instead of quietly building a huge frame. pandas 3.1.0 also prints the duplicate keys. The same kind of blow-up can come from pivot_table, get_dummies or crosstab on a column with millions of unique values.

3. Shrink the dtypes

import numpy as np, pandas as pd
n = 1_000_000
df = pd.DataFrame({
    "city": np.random.default_rng(0).choice(["Paris", "Lagos", "Lima", "Oslo"], n),
    "qty": np.arange(n) % 100,
    "price": np.arange(n) * 0.5,
})
def mib(d): return round(d.memory_usage(deep=True).sum() / 2**20, 1)
print(df.dtypes.to_dict(), mib(df), "MiB")
df["city"] = df["city"].astype("category")
df["qty"] = pd.to_numeric(df["qty"], downcast="integer")
df["price"] = df["price"].astype("float32")
print(df.dtypes.to_dict(), mib(df), "MiB")

Output with pandas 2.3.3:

{'city': dtype('O'), 'qty': dtype('int64'), 'price': dtype('float64')} 66.3 MiB
{'city': CategoricalDtype(categories=['Lagos', 'Lima', 'Oslo', 'Paris'], ordered=False, categories_dtype=object), 'qty': dtype('int8'), 'price': dtype('float32')} 5.7 MiB

Output with pandas 3.1.0:

{'city': <StringDtype(storage='python', na_value=nan)>, 'qty': dtype('int64'), 'price': dtype('float64')} 66.3 MiB
{'city': CategoricalDtype(categories=['Lagos', 'Lima', 'Oslo', 'Paris'], ordered=False, categories_dtype=str), 'qty': dtype('int8'), 'price': dtype('float32')} 5.7 MiB

66.3 MiB fell to 5.7 MiB (about 12×) on both versions. Repeated strings become category, small integers become int8 (downcast="integer" picks the smallest type that fits), and float32 is enough when 7 significant digits are. pandas 3 stores text as str by default instead of object, but without pyarrow installed it used the same memory here.

4. Read less from read_csv, or read it in chunks

import numpy as np, pandas as pd
n = 1_000_000
pd.DataFrame({
    "city": np.random.default_rng(0).choice(["Paris", "Lagos", "Lima", "Oslo"], n),
    "qty": np.arange(n) % 100,
    "note": ["x" * 40] * n,
}).to_csv("big.csv", index=False)
# load only the columns you need, with small dtypes
df = pd.read_csv("big.csv", usecols=["city", "qty"], dtype={"city": "category", "qty": "int8"})
print(round(df.memory_usage(deep=True).sum() / 2**20, 1), "MiB")
# or never hold the whole file: aggregate chunk by chunk
total = None
for chunk in pd.read_csv("big.csv", usecols=["city", "qty"], chunksize=200_000):
    part = chunk.groupby("city")["qty"].sum()
    total = part if total is None else total.add(part, fill_value=0)
print(total.astype(int).to_dict())

Output, identical on pandas 2.3.3 and 3.1.0:

1.9 MiB
{'Lagos': 12366120, 'Lima': 12373822, 'Oslo': 12408502, 'Paris': 12351556}

usecols skips columns you do not need (the 40-character note column here) and dtype parses straight into small types, so the full-size frame never exists. If even that is too big, chunksize returns an iterator of DataFrames: aggregate each chunk and combine the results, as above. For a plain-Python file version of this problem, see MemoryError when reading a large file.

Related: merge KeyError / not matching · pivot: Index contains duplicate entries · DataFrame is highly fragmented.

Written with AI assistance. Measured on 2026-10-23 in the python:3-alpine Docker image (Python 3.14.7, NumPy 2.5.4) with pandas 2.3.3 and pandas 3.1.0, 1 GB container memory limit. Outputs pasted unedited. PlainTables is our extension; Python, NumPy and pandas are free.