Python MemoryError reading a large file (tested)

MemoryError when reading a file means you loaded more of it at once than the process can hold. We capped a Python 3.13.5 process at 200 MB, read a 300 MB file five ways, and measured how much memory each really uses.

1. MemoryError has no message

for expr in ["bytearray(10**13)", "[0] * 10**13", "'x' * 10**13"]:
    try:
        eval(expr)
    except MemoryError as e:
        print(f"{expr:20} -> MemoryError({e!r}) args={e.args}")

Output:

bytearray(10**13)    -> MemoryError(MemoryError()) args=()
[0] * 10**13         -> MemoryError(MemoryError()) args=()
'x' * 10**13         -> MemoryError(MemoryError()) args=()

Measured on Python 3.13.5: a single allocation that can never fit raises MemoryError straight away, with empty args, so the traceback ends in a bare MemoryError. The line it points to is the allocation that failed, which is not always the thing using most of the memory.

2. Reading a file bigger than your memory

import resource
# Two 300 MB files: one with normal lines, one with no newline at all
with open("lines.txt", "w") as f:
    for _ in range(300):
        f.write(("x" * 99 + "\n") * 10_000)
with open("oneline.txt", "w") as f:
    for _ in range(300):
        f.write("x" * 1_000_000)

# Cap this process at 200 MB of address space
cap = 200 * 1024 * 1024
resource.setrlimit(resource.RLIMIT_AS, (cap, cap))

def attempt(name, fn):
    try:
        print(f"{name:32} -> {fn()}")
    except MemoryError as e:
        print(f"{name:32} -> MemoryError({e!r})")

attempt("lines.txt: f.read()", lambda: len(open("lines.txt").read()))
attempt("lines.txt: f.readlines()", lambda: len(open("lines.txt").readlines()))
attempt("lines.txt: for line in f", lambda: sum(1 for _ in open("lines.txt")))
attempt("oneline.txt: for line in f", lambda: sum(1 for _ in open("oneline.txt")))
def chunks(path, size=1 << 20):
    with open(path) as f:
        while chunk := f.read(size):
            yield chunk

attempt("oneline.txt: 1 MB chunks", lambda: sum(1 for _ in chunks("oneline.txt")))

Output:

lines.txt: f.read()              -> MemoryError(MemoryError())
lines.txt: f.readlines()         -> MemoryError(MemoryError())
lines.txt: for line in f         -> 3000000
oneline.txt: for line in f       -> MemoryError(MemoryError())
oneline.txt: 1 MB chunks         -> 287

Measured: with the process capped at 200 MB, read() and readlines() on a 300 MB file raised MemoryError, while for line in f counted all 3,000,000 lines. But line iteration only helps if the file has line breaks: on a 300 MB file with no newline, for line in f is one 300 MB line and fails the same way. Reading fixed-size chunks works for any file.

3. How much memory each approach really uses

import subprocess, sys
# Peak memory (max RSS) of each approach, each in a fresh process, no cap
snippets = {
    "f.read()":       "len(open('lines.txt').read())",
    "f.readlines()":  "len(open('lines.txt').readlines())",
    "for line in f":  "sum(1 for _ in open('lines.txt'))",
}
for name, code in snippets.items():
    prog = code + "\nimport resource; print(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024)"
    mb = subprocess.run([sys.executable, "-c", prog], capture_output=True, text=True).stdout.strip()
    print(f"{name:15} peak {mb} MB for a 300 MB file")

Output:

f.read()        peak 580 MB for a 300 MB file
f.readlines()   peak 446 MB for a 300 MB file
for line in f   peak 11 MB for a 300 MB file

Measured: read() peaked at almost twice the file size and readlines() at about 1.5x; iterating line by line stayed at 11 MB. Python's text decoding and per-line string objects add overhead on top of the raw bytes, so "the file fits in RAM" is not enough. Process the file as a stream, and write results out as you go instead of collecting them in a list.

On Linux without a memory cap, a process that runs out of memory is often killed by the kernel instead of getting a MemoryError; see our out of memory killed guide.

Written with AI assistance. Measured on 2026-10-09 with Python 3.13.5 on Linux: 10**13-sized allocations, two 300 MB files read five ways under a 200 MB RLIMIT_AS cap, and peak RSS of read(), readlines() and line iteration. Every block was run verbatim and its output pasted from that run. PlainHeaders is our extension; everything else here is built-in Python.