Diagnosing Linux Memory Exhaustion and OOM Kills

Hello. In the previous lesson, you diagnosed CPU saturation by separating a clue such as high load from stronger evidence such as run-queue pressure, CPU utilization, and process attribution. Memory troubleshooting needs the same discipline: low “free” memory or a large process alone does not prove memory exhaustion.

In this lesson, you will learn to distinguish normal Linux caching from genuine memory pressure, recognize active swapping, identify the process killed by the kernel, and communicate a bounded conclusion: whether this was a host-wide OOM event, a service-level memory limit, or merely a warning sign requiring further investigation.

---

Memory accounting: what is actually scarce?

Linux deliberately uses otherwise idle RAM for useful work, especially the page cache: recently read file data retained in memory so it can be reused without another disk read. Therefore, a host with very little MemFree is often healthy.

The first question is not, “How much memory is free?” It is:

How much memory can the kernel make available for a new workload without substantial reclaiming or swapping?

On modern Linux systems, the best quick answer is MemAvailable, also shown in the available column of free.

free -h

A representative output might look like this:

               total        used        free      shared  buff/cache   available
Mem:            16Gi        12Gi       420Mi       310Mi       3.6Gi        2.9Gi
Swap:           2.0Gi       120Mi       1.9Gi

Do not conclude that this host has only 420 MiB left. The key figure is 2.9 GiB available: Linux estimates it can reclaim enough cache and reclaimable kernel memory to start new applications without immediately swapping.

A useful working model separates four categories:

Category	Meaning	Diagnostic importance
Anonymous memory	Process-private memory such as heaps and stacks	Often where application leaks and oversized runtimes appear
File-backed memory / page cache	Executables, libraries, and cached file data	Usually reclaimable; high values are not automatically bad
Kernel memory	Slab caches, page tables, kernel stacks, networking structures	Can become important when process RSS does not explain pressure
Swap	Memory pages moved to disk	Used swap is historical state; active swap I/O is the concern

The Linux kernel also distinguishes an allocation promise from memory physically in use:

• Virtual memory is address space a process may use.
• Resident Set Size (RSS) is the portion currently resident in RAM.
• Committed memory is memory the kernel has promised to make available under its overcommit policy.

That distinction prevents a common misdiagnosis: a process with 20 GiB of virtual memory has not necessarily consumed 20 GiB of RAM.

The image demonstrates why VSZ, total-vm, and Committed_AS need context. Allocation may reserve or commit address space before an application actually accesses the corresponding pages. A process that progressively touches pages will increase its RSS and create real pressure.

A compact interpretation rule is:

• Low MemFree, high MemAvailable: normally healthy cache use.
• Falling MemAvailable over time: rising pressure; investigate the trend.
• Low MemAvailable plus sustained swap activity or memory stalls: active memory contention.
• Low MemAvailable followed by kernel OOM logs: confirmed allocation failure, not merely a monitoring threshold breach.

---

Detect active memory pressure rather than a harmless snapshot

Begin a live investigation by preserving basic context:

date
hostname
free -h
grep -E 'MemAvailable|SwapTotal|SwapFree|Slab|SReclaimable|SUnreclaim|Committed_AS|CommitLimit' /proc/meminfo

free -h gives a readable snapshot. /proc/meminfo provides detail when you need to determine whether pressure comes from processes, cache, or kernel allocations.

Next, sample the host over time:

vmstat 1

As in the CPU lesson, ignore the first vmstat row because it is an average since boot. For memory diagnosis, focus on:

• swpd: total swap currently used. This is a snapshot, not a rate.
• si: swap pages read back into RAM per second.
• so: pages written from RAM to swap per second.
• free, buff, and cache: supporting context only.
• b and wa: blocked tasks and I/O wait; swapping can turn memory pressure into an I/O and latency problem.

The distinction between used swap and active swapping is crucial. Consider these two cases:

Case A
swpd: 1024M
si: 0
so: 0
MemAvailable: 5G

This host has swapped pages at some earlier time, but there is no evidence of current paging pressure.

Case B
swpd: increasing
si: 18000
so: 24000
MemAvailable: near zero
wa: elevated

This is active paging. The host repeatedly evicts pages to disk and later reads them back because processes need them again. Requests may slow dramatically before an OOM kill occurs.

On hosts with Linux Pressure Stall Information enabled, inspect:

cat /proc/pressure/memory

Typical output includes some and possibly full pressure measurements:

some avg10=12.40 avg60=6.18 avg300=2.11 total=...
full avg10=4.60 avg60=1.72 avg300=0.51 total=...

• some means at least one task was stalled because it could not obtain memory promptly.
• full means all non-idle tasks were stalled at the same time.

Elevated memory PSI that overlaps with latency or timeout reports is valuable evidence. It is stronger than merely observing a low free-memory number because it measures the operational consequence: work waiting on memory reclaim.

Find the consumers, but interpret process memory carefully

To identify large resident processes:

ps -eo pid,ppid,user,rss,vsz,stat,comm --sort=-rss | head -n 20

Here, rss and vsz are usually reported in KiB. A large RSS is a useful lead; a large VSZ alone is not.

For a live view, use:

top

In top, sort by memory usage with Shift + M. Inspect the RES or resident-memory column, alongside the command, PID, state, and %MEM. For a suspected process, record a short trend rather than trusting one point in time:

watch -n 5 'ps -p <PID> -o pid,ppid,rss,vsz,etime,cmd'

A steadily rising RSS across many samples supports a leak hypothesis. A stable high RSS during a traffic peak may instead be expected workload growth, excessive concurrency, or an intentionally large cache.

Do not sum RSS values across every process and expect an exact host total. Shared libraries and shared memory can be counted in multiple processes’ RSS. Use the sorted list to identify suspects, then correlate with application behavior, deployment changes, request volume, and the timing of the incident.

Also check whether kernel memory explains the missing capacity:

grep -E 'Slab|SReclaimable|SUnreclaim|KernelStack|PageTables' /proc/meminfo

Slab is kernel cache memory. SReclaimable may be freed under pressure; SUnreclaim cannot readily be reclaimed. If process RSS is modest but SUnreclaim, page tables, or kernel stacks are unusually large and growing, the investigation must extend beyond application processes.

---

OOM kills: prove them from kernel evidence

An Out Of Memory (OOM) kill is the kernel’s last-resort action when it cannot satisfy a memory allocation. The kernel selects a victim process to terminate with SIGKILL to restore memory and preserve the rest of the system.

A service that disappears with a SIGKILL status is suspicious, but not conclusive. An operator, an orchestrator, or a timeout supervisor can also send SIGKILL. The decisive evidence is in the kernel log.

sudo journalctl -k -b --since "2 hours ago" \
  | grep -Ei 'out of memory|oom-kill|killed process'

For a currently running host where the ring buffer still contains the event:

sudo dmesg -T | grep -Ei 'out of memory|oom-kill|killed process'

Then inspect the application service over the same incident window:

sudo systemctl status <service-name>
sudo journalctl -u <service-name> --since "2 hours ago"

A container platform may report exit code 137, conventionally meaning termination by signal 9 (SIGKILL). Treat it as a trigger to examine kernel and platform logs, not as proof by itself.

A simplified OOM record can look like this:

oom-kill: constraint=CONSTRAINT_NONE, task=java, pid=1842
Out of memory: Killed process 1842 (java)
total-vm:12582912kB, anon-rss:7350000kB, file-rss:18000kB,
shmem-rss:0kB, pgtables:16000kB, oom_score_adj:0

Interpret it carefully:

Field	Meaning	What it can tell you
constraint	Scope of the allocation failure	CONSTRAINT_NONE commonly indicates a host-wide event; a memory-cgroup constraint indicates a limit local to a workload
Killed process	The kernel’s selected victim	Identifies what died, not necessarily what originally caused pressure
total-vm	Virtual address space	Not equivalent to RAM used
anon-rss	Resident private anonymous memory	Often the strongest clue to a process heap or private allocation growth
file-rss	Resident file-backed memory	Libraries and mapped files; may be reclaimable
oom_score_adj	Explicit bias in victim selection	A protected or deprioritized process can alter which victim is chosen

The victim is not automatically the root cause. For example, a batch job may exhaust the node while the kernel kills an API process with a more favorable victim score. Conversely, a large Java process killed during an OOM event may genuinely have a poorly sized heap or a leak. You need trend data and workload context to decide.

Do not respond by broadly setting critical services to oom_score_adj=-1000. That can make a service effectively unkillable; if it continues consuming memory, the host may become unusable or force the kernel to kill less appropriate processes. Victim selection is damage control, not capacity planning.

Host OOM versus service or container OOM

Not every OOM event means the entire host ran out of usable memory.

A service can hit its cgroup memory limit while the node still has plenty of MemAvailable. In kernel logs, this often appears as a memory-cgroup constraint rather than a global constraint. On a cgroup v2 system managed by systemd, you can inspect a service’s configured group and event counters:

cg=$(systemctl show -p ControlGroup --value <service-name>)
cat "/sys/fs/cgroup${cg}/memory.events"
systemctl show <service-name> -p MemoryCurrent -p MemoryMax

An incrementing oom_kill counter in that service’s memory.events file supports a local cgroup OOM diagnosis. This distinction determines the remedy:

• Global host OOM: investigate node capacity, all major consumers, swap/paging behavior, and workload placement.
• Service or container OOM: investigate that workload’s memory limit, current usage, request concurrency, and runtime configuration.

---

A defensible incident workflow

Use this sequence when a service has disappeared, latency is rising, or an alert reports low available memory.

1. Set the hypothesis and time window.
   “I will determine whether this is normal cache use, active memory pressure, a confirmed OOM kill, or a workload-specific memory-limit failure.”

2. Capture current host evidence.
   Run free -h, record key /proc/meminfo fields, and sample vmstat 1. If present, inspect /proc/pressure/memory.

3. Separate dormant swap from thrashing.
   Used swap with si=0 and so=0 is not an emergency. Sustained swap-in/out activity, falling MemAvailable, elevated memory PSI, and I/O wait support a pressure diagnosis.

4. Attribute resident memory.
   Sort processes by RSS, not VSZ. Trend a suspected process. Check whether kernel slab or page-table memory accounts for a material portion of RAM.

5. Confirm or rule out OOM.
   Search journalctl -k and dmesg for oom-kill, Out of memory, and Killed process. Correlate the timestamp with service restart logs and monitoring data.

6. Determine the OOM scope.
   Read the kernel log’s constraint details. Check cgroup counters and configured memory limits when the affected workload is containerized or systemd-managed.

7. State the conclusion with an appropriate confidence level.
   Distinguish the observed event, its likely immediate cause, and the root-cause hypothesis still needing validation.

For example:

“The API service was terminated by a kernel OOM kill at 14:07; the kernel log identifies PID 1842 and shows approximately 7 GiB of anonymous resident memory. During the preceding ten minutes, MemAvailable fell near zero, vmstat showed sustained swap-in and swap-out, and memory PSI rose, so this was active host-level memory pressure rather than normal page-cache use. The API process is a primary suspect, but it is the victim rather than proven root cause. I would compare its RSS trend with traffic and the deployment at 13:50, then verify its runtime heap limit and request-concurrency settings.”

---

Key takeaways

• Low MemFree is normal on Linux. Use MemAvailable to estimate practical headroom.
• Used swap is not necessarily an incident. Persistent vmstat si and so activity indicates active paging and likely performance impact.
• RSS is the primary process-level starting point; VSZ and total-vm do not equal physical RAM consumption.
• Kernel logs prove OOM kills. Search for oom-kill, Out of memory, and Killed process, then correlate them with service logs and metrics.
• The killed process is not automatically the root cause. Establish scope, trends, memory limits, and changes around the incident.
• Separate host-wide OOM from cgroup-limited OOM. The remediation may be node capacity in one case and a specific workload’s limit or behavior in the other.

Next, you will move from resource exhaustion to process-level diagnosis: tracing a failing process through its file descriptors, system calls, and service logs.