Out of Memory: Killed Process (tested)

Out of memory: Killed process is a line the Linux kernel writes when it kills a process to free RAM. Seeing it for mysqld, a Python script, Java or any other process usually means something hit a memory limit, not that your code crashed on its own. We built a small allocator, ran it inside a 50 MB memory cgroup until it got killed, and captured exactly what that looks like — then ran the same script with a generous limit to prove the limit, not a real system-wide shortage, caused the kill.

1. What "Out of memory: Killed process" actually means

This message comes from the Linux kernel's OOM (out-of-memory) killer. When available memory (RAM plus swap, within whatever boundary applies) runs out, the kernel does not let the system grind to a halt. Instead it picks one process, logs a line like the one below, and sends it SIGKILL. The process gets no chance to clean up; it just disappears.

The exact wording readers see for mysqld, mariadb, python3 or java is a per-process line the kernel prints when it fires. We did not have permission to read the live kernel ring buffer on this box (dmesg returned Operation not permitted for an unprivileged user — normal on a locked-down machine), so this exact format is per the kernel source / docs, not captured by us today:

Out of memory: Killed process 12345 (mysqld) total-vm:2145332kB, anon-rss:1823456kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:4568kB oom_score_adj:0

total-vm is the process's total virtual address space; anon-rss is the anonymous (heap/stack, non-file-backed) resident memory it actually held in RAM — that second number is the one that matters for "how much RAM did it really use."

2. We reproduced a real OOM kill, safely, with a memory cgroup

Letting a process actually exhaust a shared machine's RAM is destructive to everything else running on it, so we did not do that. Instead we used the same mechanism Docker's --memory and Kubernetes pod memory limits use under the hood: a Linux memory cgroup with a hard cap, created with systemd-run --user --scope. This confines the kill to one disposable process tree; nothing else on the box was touched.

Our script, hog.py, just grows a list of 5 MB bytearray chunks in a loop and prints progress:

chunks = []
total_mb = 0
while True:
    chunks.append(bytearray(5 * 1024 * 1024))
    total_mb += 5
    print(f"allocated {total_mb} MB", flush=True)

We ran it capped at 50 MB, with swap disabled for that cgroup so the kill happens fast and cleanly instead of being delayed by swapping:

$ systemd-run --user --scope -p MemoryMax=50M -p MemorySwapMax=0 -- python3 hog.py
Running as unit: run-p1953945-i6146989.scope
allocated 5 MB
allocated 10 MB
allocated 15 MB
allocated 20 MB
allocated 25 MB
allocated 30 MB
allocated 35 MB
allocated 40 MB
allocated 45 MB
Killed
$ echo $?
137

Exit code 137 is the standard signature of this kind of kill: 128 + 9 (SIGKILL). It stopped at 45–50 MB, right at the 50 MB cap we set — the Python interpreter itself, plus the already-allocated list, used up the rest of the headroom.

3. The systemd journal confirms it was the OOM killer, not a crash

Because this is a user-scoped unit, we checked only our own user journal (journalctl --user), not the system-wide log. It recorded the kill plainly:

$ journalctl --user -xe -n 20 --no-pager
systemd[965]: run-p1953945-i6146989.scope: A process of this unit has been killed by the OOM killer.
systemd[965]: run-p1953945-i6146989.scope: Failed with result 'oom-kill'.

That result 'oom-kill' field is the reliable way to tell "my process was OOM-killed" apart from a normal crash or an uncaught exception — check it before assuming a bug in your own code.

4. Proof it's the limit, not a real system-wide shortage

To show the 50 MB cap — not a genuine lack of RAM on the box — caused the kill, we ran the identical script again with a 2 GB cap instead, cut off after 5 seconds by our own test harness:

$ timeout 5 systemd-run --user --scope -p MemoryMax=2G -p MemorySwapMax=0 -- python3 hog.py
allocated 5 MB
...
allocated 440 MB
allocated 445 MB
$ echo $?
124

Exit 124 is timeout ending our own test, not an OOM kill. The same code sailed past 445 MB — nearly 9x the point where it died under the 50 MB cap — with no sign of memory pressure. The box has 15 GB of RAM; at no point was the machine itself short of memory. The only thing that changed between the two runs was the cgroup's MemoryMax, which is exactly how Docker's --memory=50m or a Kubernetes resources.limits.memory: 50Mi produce the same kind of kill in containers: a per-container (per-cgroup) ceiling, enforced the same way we enforced ours.

5. A different mechanism: RLIMIT_AS / `ulimit -v` gives a clean exception instead

Readers also hit memory errors from a process's own soft limit (ulimit -v, RLIMIT_AS), and it's easy to confuse this with an OOM kill — the symptoms look similar but the mechanism and the error you get back are different. We ran the same allocator under a 50,000 KB virtual-memory limit instead of a cgroup:

$ bash -c 'ulimit -v 51200; python3 hog.py'
allocated 20 MB
allocated 25 MB
allocated 30 MB
allocated 35 MB
MemoryError at ~35 MB

No SIGKILL, no exit 137, no journal entry — Python's own allocator asked the kernel for more address space, got refused, and raised a catchable MemoryError inside the process. The process exits cleanly (or keeps running if you catch the exception). That is a world away from the kernel killer: with ulimit -v your own process enforces its own ceiling and can react; with a cgroup OOM kill, something outside the process (the kernel) ends it with no warning and no chance to catch anything.

Mechanism	Trigger	What the process sees	Typical exit
Cgroup OOM kill (Docker `--memory`, k8s limits, systemd `MemoryMax`)	cgroup's memory accounting hits its cap	Nothing — killed instantly by `SIGKILL`	137 (measured)
System-wide kernel OOM killer	whole machine's RAM+swap exhausted	Nothing — killed instantly by `SIGKILL`, kernel logs total-vm/anon-rss	137 (per kernel docs, not reproduced here — would require exhausting the whole box)
`ulimit -v` / RLIMIT_AS	process's own virtual memory soft limit	Catchable allocation failure (`MemoryError` in Python, `malloc` returning NULL in C)	Whatever your code does with the exception (measured: clean exit)

6. Container vs whole-machine OOM, and how the kernel picks a victim

Everything in sections 2–4 is the same mechanism Docker and Kubernetes use: a memory cgroup with a hard limit, scoped to one container or pod. That is overwhelmingly the kind of "out of memory killed" report people hit in practice (mysqld in a small container, a Java service with a tight pod limit, a Python worker in a 512 MB dyno) because container memory limits are deliberately set tight to pack more workloads onto shared hardware.

A genuine whole-machine OOM — where the kernel kills something because the entire box, not a container, is out of RAM and swap — is rarer in day-to-day operation, because most servers either have enough headroom or are themselves inside a container/VM limit that gets hit first. Per the kernel docs (not something we could safely reproduce on a shared box), when this does happen the kernel's OOM killer scores every process using (per the Linux docs) its oom_score, which factors in memory usage and is adjusted by oom_score_adj (range -1000 to 1000, readable/writable at /proc//oom_score_adj). A process at -1000 is never killed by the OOM killer; one at 1000 is killed first. This is how systems protect critical daemons (set a very negative oom_score_adj) while letting large, replaceable batch jobs or caches be sacrificed first (a less negative or positive value). Databases are frequently the victim in low-memory systems simply because they tend to hold the most anon-rss of anything running.

7. What to do about it

Check which mechanism you hit first. Exit 137 plus a cgroup/container log (docker inspect shows OOMKilled: true; kubectl describe pod shows OOMKilled; journalctl --user or journalctl -k shows "killed by the OOM killer") means a memory limit, not a code bug. A Python MemoryError / Java OutOfMemoryError exception you can catch means your own process's allocator gave up, which is a different fix (handle it, or raise your own limit).
Raise or right-size the container/cgroup limit (Docker --memory, k8s resources.limits.memory) once you know the real working-set size, rather than guessing.
Find the real peak usage before changing limits: for a cgroup, systemd-run and systemctl --user status report a "memory peak"; for containers, docker stats.
For databases specifically (mysqld, mariadb, postgres), tune the engine's own memory settings (e.g. MySQL's innodb_buffer_pool_size) to fit comfortably inside the container limit — the database doesn't know about the cgroup ceiling and will keep allocating until it hits it.
For a genuinely memory-tight host, set oom_score_adj deliberately on processes you never want killed, and add swap or more RAM rather than relying on the OOM killer's default choice.

Frequently asked questions

What does exit code 137 mean?

137 = 128 + 9, where 9 is SIGKILL. We measured this exact exit code when our test process was killed by a 50 MB memory cgroup limit. It shows up for both container/cgroup OOM kills and a genuine system-wide kernel OOM kill — check your container runtime's logs or journalctl to tell them apart.

Is 'Killed' with no error message always an OOM kill?

In our test, yes — the process printed nothing itself, just stopped, and the shell reported 'Killed'. The confirmation came from journalctl ('A process of this unit has been killed by the OOM killer'), Docker's OOMKilled flag, or dmesg — not from the process's own output, since SIGKILL gives it no chance to log anything.

Why did mysqld / mariadb get killed instead of another process?

Per the kernel docs, the OOM killer scores processes mainly by how much resident memory (anon-rss) they hold, adjusted by each process's oom_score_adj. Databases typically hold the largest working set on a box, so they're statistically likely victims when memory is genuinely tight — independent of whether they're actually the cause.

Is a Python MemoryError the same as an OOM kill?

No, and we measured both to show the difference. A cgroup OOM kill ends the process instantly via SIGKILL (exit 137, nothing catchable). A MemoryError happens when a process hits its own RLIMIT_AS virtual-memory limit (ulimit -v) or genuinely fails an allocation — it's a normal Python exception your code can catch and handle.

How do I stop my container from being OOM-killed?

First confirm it's actually hitting a limit (docker inspect shows OOMKilled: true; kubectl describe pod shows OOMKilled). Then either raise the container's memory limit to match real peak usage, or reduce the app's own memory footprint (for databases, lower buffer/cache settings so the engine stays under the ceiling itself instead of letting the kernel enforce it with SIGKILL).

Written with AI assistance. Measured on 2026-10-02 with Python 3.13, systemd 257 (systemd-run --user --scope), on Debian 13 / Linux 6.12. We triggered a real cgroup OOM kill with MemoryMax, read systemd's own journal record of it, and contrasted it with Python's RLIMIT_AS MemoryError and a generous-limit control run. We could not read dmesg (Operation not permitted, unprivileged user) so the classic kernel log line with total-vm/anon-rss is quoted from the kernel docs, not captured live. PlainHeaders is our extension; curl, Python and systemd are free.