Domain 5Containers & RuntimesCORE

(Genova) Troubleshooting Cgroup v2 Memory Limits and OOM Killer Isolation in Linux

A deep-dive troubleshooting guide on diagnosing why a cgroup v2 memory cap failed to prevent system-wide OOM kills, comparing memory.high vs. memory.max, and isolating noisy-neighbor workloads.

Domain: Domain 5: Containers & Runtimes•Updated: 2026-11-12
#Linux#cgroups#cgroup-v2#systemd#OOM-Killer#Memory-Management#Troubleshooting#SRE

1. Introduction

Resource contention—often referred to as the “noisy neighbor” problem—is one of the most common causes of service outages on shared Linux hosts. When an asynchronous, memory-intensive batch job runs alongside a latency-sensitive web API on the same machine, an unbounded memory spike in the batch process can exhaust physical RAM and trigger the Linux kernel’s Out-Of-Memory (OOM) killer. Without strict resource boundaries, the kernel may terminate the critical API instead of the runaway batch worker.

In modern Linux environments, Control Groups (cgroups v2) provide the kernel mechanism for partitioning and limiting hardware resources such as memory, CPU, and I/O across groups of processes. However, misconfiguring cgroup v2 control files—or confusing soft throttling thresholds with hard limits—can silently undermine workload isolation.

In this article, we walk through a real-world troubleshooting exercise based on the SadServers “Genova” scenario. You will learn how to:

  • Systematically triage a cgroup v2 configuration failure using read-only diagnostics before modifying system state.
  • Distinguish between cgroup v1 legacy artifacts, cgroup v2 soft limits (memory.high), and cgroup v2 hard limits (memory.max).
  • Trace how systemd units interact with custom cgroup management scripts during boot and service execution.
  • Apply and verify a targeted, persistent fix that contains OOM events within a specific cgroup hierarchy.

2. Problem Description and Environment

Scenario Overview

A small Linux virtual machine hosts two primary workloads:

  1. sad-api: A lightweight Python health endpoint listening locally on 127.0.0.1:9090.
  2. sad-batch: A nightly ETL-style worker script that allocates large amounts of RAM.

Following a recent deployment, starting sad-batch caused memory consumption to spike rapidly, resulting in sad-api being terminated by the kernel OOM killer. An on-call engineer stopped and disabled sad-batch prior to handing off the host and left context in /home/admin/incident-notes.txt.

Expected vs. Actual Behaviour

  • Expected Behaviour: sad-batch executes inside a dedicated cgroup v2 hierarchy (/sys/fs/cgroup/sad-batch) capped at a hard limit of 128 MiB (134217728 bytes). If sad-batch attempts to allocate more than 128 MiB, the kernel OOM killer terminates only processes within the sad-batch cgroup, leaving sad-api healthy and responsive on 127.0.0.1:9090.
  • Actual Behaviour: The 128 MiB cap never takes effect. /sys/fs/cgroup/sad-batch/memory.max remains set to max (unlimited), allowing sad-batch to consume host RAM until the global OOM killer terminates sad-api.

Environment Details & Constraints

Based on the diagnostic evidence collected during the session:

  • Hostname: i-0285bb94fbc615e56
  • User Context: admin with passwordless sudo privileges.
  • Init & Service Manager: systemd
  • Cgroup Hierarchy: Unified cgroup v2 mounted at /sys/fs/cgroup/.
  • OS & Kernel Version: Specific distribution name and kernel release were not queried during the session (uname -a and /etc/os-release were not executed), though the unified cgroup v2 mount confirms a modern Linux kernel supporting cgroups v2.
  • Operational Constraints:
  • Do not modify sad-api; it must remain active and return SadServers - API OK from [http://127.0.0.1:9090/](http://127.0.0.1:9090/).
  • /sys/fs/cgroup/sad-batch/memory.max must be configured to 134217728 before restarting sad-batch.
  • Do not modify the grading script /home/admin/agent/check.sh.

3. Initial Investigation and Hypotheses

When approaching an unfamiliar system after an incident, resisting the urge to immediately edit files is critical. Changing configuration before understanding how services, scripts, and kernel interfaces interact often introduces secondary failures or masks the true root cause.

What Was Initially Known vs. Unknown

  • Known: sad-api was killed by the OOM killer when sad-batch ran; a legacy cgroup v2 launcher under /opt/sad/ is supposed to enforce a 128 MiB limit; sad-batch is currently stopped and disabled.
  • Unknown: Whether /sys/fs/cgroup/sad-batch exists at boot, what value memory.max currently holds, how systemd orchestrates the cgroup setup relative to sad-batch, and why the launcher’s limit fails to apply.

Forming Initial Hypotheses

Before inspecting the scripts, four plausible hypotheses could explain why a cgroup memory cap fails to apply:

Hypothesis Supporting Evidence Evidence Against Investigation Step
1. Soft limit (memory.high) used instead of hard limit (memory.max) Problem states a 128 MiB limit is configured in /opt/sad/ but /sys/fs/cgroup/sad-batch/memory.max is not 134217728. None yet; requires inspecting the setup script. Inspect /opt/sad/setup-cgroup.sh and /sys/fs/cgroup/sad-batch/memory.max.
2. Cgroup v1 syntax used on a cgroup v2 host Problem description explicitly calls /opt/sad/ a “legacy” cgroup launcher. Problem mentions /sys/fs/cgroup/sad-batch/memory.max exists as a path structure. Read incident-notes.txt and /opt/sad/setup-cgroup.sh to check target paths and filenames.
3. Boot setup service failed or was not triggered If the setup script never ran, memory.max would remain at its default (max). None yet; service status has not been checked. Run systemctl status on the sad-* units.
4. Worker process never joins the target cgroup Even if memory.max were set, a process running outside /sys/fs/cgroup/sad-batch would bypass the limit. Problem states memory.max itself does not show 134217728, pointing first to cgroup setup. Inspect /opt/sad/run-batch.sh to verify cgroup.procs assignment.

4. Troubleshooting Process

Step 1: Reviewing Incident Notes and Launcher Directory

Objective

Gather operational context left by the on-call engineer in /home/admin/incident-notes.txt and list the files inside /opt/sad/ to map out the custom launcher architecture.

Command

cat incident-notes.txt
ls -la /opt/sad/

Command Explanation

  • cat incident-notes.txt: Reads and prints the contents of incident-notes.txt from the current working directory (/home/admin) to standard output.
  • ls -la /opt/sad/: Lists all directory entries inside /opt/sad/.
  • -l: Uses the long listing format, displaying file permissions, link count, owner, group, size in bytes, and modification timestamp.
  • -a: Includes hidden files (entries beginning with .).

Output and Analysis

Incident: 2026-06-10 — nightly batch run took down the API

After the batch job was redeployed, sad-batch started allocating hundreds of
megabytes during its run. sad-api (port 9090) was killed by the OOM killer and
stayed down until someone restarted it manually.

This host does NOT use systemd resource limits for the batch job. Memory control
is handled by a legacy cgroup v2 launcher under /opt/sad/ (see setup-cgroup.sh
and run-batch.sh). On-call disabled sad-batch before handing the VM to you.
...
Ops ticket excerpt:
  "We set a 128M cap in setup-cgroup.sh — why is memory.max still max?"

Legacy wiki snippet (2023, cgroup v1 — do not trust blindly):
  echo 134217728 > /sys/fs/cgroup/memory/sad-batch/memory.limit_in_bytes
total 24
drwxr-xr-x 2 root root 4096 Jun 16 18:54 .
drwxr-xr-x 4 root root 4096 Jun 16 18:54 ..
-rwxr-xr-x 1 root root  194 Jun 16 18:54 run-batch.sh
-rwxr-xr-x 1 root root  447 Jun 16 18:54 sad-api.sh
-rwxr-xr-x 1 root root  219 Jun 16 18:54 sad-batch.sh
-rwxr-xr-x 1 root root  200 Jun 16 18:54 setup-cgroup.sh

The incident notes provide three critical clues:

  1. Systemd native memory directives (like MemoryMax=) are intentionally not used for sad-batch; memory enforcement is delegated to /opt/sad/setup-cgroup.sh and /opt/sad/run-batch.sh.
  2. The ops ticket explicitly confirms that memory.max is still reading max despite a 128M cap being configured in setup-cgroup.sh.
  3. The legacy wiki snippet (memory.limit_in_bytes) is a classic cgroup v1 red herring, warning us not to confuse cgroup v1 filenames with cgroup v2.

The ls -la output shows four root-owned executable (-rwxr-xr-x) shell scripts in /opt/sad/.

Decision

Check the current status of the systemd services (sad-api, sad-cgroup-setup, and sad-batch) and verify that sad-api is currently serving traffic before inspecting the scripts.


Step 2: Auditing Systemd Unit States and Baseline API Health

Objective

Verify the runtime state of all three related systemd services and confirm that sad-api is healthy on 127.0.0.1:9090.

Command

systemctl status sad-api sad-cgroup-setup sad-batch --no-pager
curl -s http://127.0.0.1:9090

Command Explanation

  • systemctl status <units> --no-pager: Queries systemd for the loaded state, active state, main PID, memory footprint, and recent journal entries for multiple units simultaneously.

  • --no-pager: Prints output directly to stdout rather than piping through less, preventing truncation or interactive blocking.

  • curl -s [http://127.0.0.1:9090](http://127.0.0.1:9090): Performs an HTTP GET request against the local API endpoint.

  • -s (--silent): Suppresses progress meters and error output so only the response body is displayed.

Output and Analysis

● sad-api.service - SadServers API health endpoint
     Loaded: loaded (/etc/systemd/system/sad-api.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-11-11 14:16:14 UTC; 2min 8s ago
   Main PID: 762 (python3)
     Memory: 8.6M (max: 48M, available: 39.3M, peak: 8.9M)
     CGroup: /system.slice/sad-api.service
             └─762 python3 /opt/sad/sad-api.sh

● sad-cgroup-setup.service - Create sad-batch cgroup at boot
     Loaded: loaded (/etc/systemd/system/sad-cgroup-setup.service; enabled; preset: enabled)
     Active: active (exited) since Wed 2026-11-11 14:15:58 UTC; 2min 24s ago
   Main PID: 265 (code=exited, status=0/SUCCESS)

○ sad-batch.service - SadServers batch ETL worker
     Loaded: loaded (/etc/systemd/system/sad-batch.service; disabled; preset: enabled)
     Active: inactive (dead)
SadServers - API OK
  • sad-api.service is active (running) under /system.slice/sad-api.service (with its own systemd memory cap of 48M) and responds with SadServers - API OK.
  • sad-cgroup-setup.service ran at boot and exited cleanly (status=0/SUCCESS), ruling out Hypothesis 3 (boot setup script crash).
  • sad-batch.service is disabled and inactive (dead), confirming it is safe to inspect the cgroup filesystem without active memory pressure.

Decision

Inspect the live cgroup control files under /sys/fs/cgroup/sad-batch/ and read the contents of /opt/sad/setup-cgroup.sh, run-batch.sh, and sad-batch.sh.


Step 3: Inspecting the Live Cgroup v2 Control File

Objective

Confirm the directory structure under /sys/fs/cgroup/sad-batch/ and check the current value of memory.max.

Command

cat /sys/fs/cgroup/sad-batch/memory.max

Command Explanation

  • cat /sys/fs/cgroup/sad-batch/memory.max: Reads the kernel virtual file representing the hard memory limit for the sad-batch cgroup. In Linux, /sys (sysfs / cgroup2) exposes kernel data structures as plain-text files.

Output and Analysis

max

Tab-completion on /sys/fs/cgroup/sad-batch/memory. also confirmed the presence of standard cgroup v2 memory controller files (memory.current, memory.events, memory.high, memory.low, memory.max, memory.min, memory.oom.group, memory.peak, memory.pressure, memory.stat, etc.).

The directory /sys/fs/cgroup/sad-batch exists (created by sad-cgroup-setup.service), confirming the host is running cgroup v2, but memory.max is indeed max (unlimited).

Decision

Read the shell scripts in /opt/sad/ to see what setup-cgroup.sh writes and how run-batch.sh launches the worker.


Step 4: Examining the Legacy Launcher and Worker Scripts

Objective

Inspect the exact implementation of the cgroup setup script, the process launcher, and the batch workload.

Command

cat /opt/sad/setup-cgroup.sh
cat /opt/sad/run-batch.sh
cat /opt/sad/sad-batch.sh

Command Explanation

  • cat /opt/sad/<script>: Prints the source code of each script to examine how cgroup directories, limits, process assignments, and memory allocations are handled.

Output and Analysis

  1. /opt/sad/setup-cgroup.sh:
#!/usr/bin/env bash
# Ensure the sad-batch cgroup exists with the intended memory budget.
set -euo pipefail

CGROUP=/sys/fs/cgroup/sad-batch

mkdir -p "$CGROUP"
echo 134217728 > "$CGROUP/memory.high"
  1. /opt/sad/run-batch.sh:
#!/usr/bin/env bash
# Launch the batch worker inside the sad-batch cgroup.
set -euo pipefail

/opt/sad/setup-cgroup.sh
echo $$ > /sys/fs/cgroup/sad-batch/cgroup.procs
exec /opt/sad/sad-batch.sh
  1. /opt/sad/sad-batch.sh:
#!/usr/bin/env bash
# Memory-heavy batch worker; must stay inside its cgroup budget.
exec python3 - <<'PY'
import time

chunks = []
for _ in range(256):
    chunks.append(bytearray(1024 * 1024))
    time.sleep(0.05)
PY

This reveals the exact defect:

  • In setup-cgroup.sh, 134217728 (128 MiB) is written to $CGROUP/memory.high rather than $CGROUP/memory.max. This confirms Hypothesis 1.
  • run-batch.sh properly ensures the cgroup exists, writes its own PID ($$) into /sys/fs/cgroup/sad-batch/cgroup.procs, and uses exec to replace the shell with sad-batch.sh (which in turn uses exec python3), ensuring the Python worker inherits membership in /sys/fs/cgroup/sad-batch. This rules out Hypothesis 4.
  • sad-batch.sh allocates 256 chunks of 1 MiB (1024 * 1024 bytes) each—a total of 256 MiB—over ~12.8 seconds.

Decision

Before modifying setup-cgroup.sh, inspect the systemd unit files and the validation script (/home/admin/agent/check.sh) to ensure there are no hidden dependencies, override drop-ins, or side effects.


Step 5: Auditing Systemd Unit Definitions and Validation Criteria

Objective

Verify the ordering and dependency relationship between sad-cgroup-setup.service and sad-batch.service, and check the exact pass criteria in check.sh.

Command

cat /home/admin/agent/check.sh
cat /etc/systemd/system/sad-cgroup-setup.service
cat /etc/systemd/system/sad-batch.service

Command Explanation

  • cat /home/admin/agent/check.sh: Inspects the automated evaluation script (marked read-only/do not modify).
  • cat /etc/systemd/system/sad-*.service: Reads the unit definitions to check Type=, ExecStart=, Requires=, Before=, After=, and Restart= directives.

Output and Analysis

  1. /home/admin/agent/check.sh:
#!/usr/bin/env bash
# DO NOT MODIFY THIS FILE ("Check My Solution" will fail)

fail() {
  echo -n "NO"
  exit 0
}

systemctl is-active --quiet sad-api.service || fail

if ! curl -s -m 2 http://127.0.0.1:9090/ | grep -q 'SadServers - API OK'; then
  fail
fi

if [ ! -f /sys/fs/cgroup/sad-batch/memory.max ]; then
  fail
fi

max=$(tr -d '[:space:]' < /sys/fs/cgroup/sad-batch/memory.max 2>/dev/null)
if [ -z "$max" ] || [ "$max" = "max" ]; then
  fail
fi

if [ "$max" != "134217728" ]; then
  fail
fi

echo -n "OK"
  1. /etc/systemd/system/sad-cgroup-setup.service:
[Unit]
Description=Create sad-batch cgroup at boot
Before=sad-batch.service
DefaultDependencies=no

[Service]
Type=oneshot
ExecStart=/opt/sad/setup-cgroup.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
  1. /etc/systemd/system/sad-batch.service:
[Unit]
Description=SadServers batch ETL worker
After=network.target sad-cgroup-setup.service
Requires=sad-cgroup-setup.service

[Service]
Type=simple
ExecStart=/opt/sad/run-batch.sh
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

The systemd wiring is sound: sad-cgroup-setup.service uses Type=oneshot with RemainAfterExit=yes and runs Before=sad-batch.service. Meanwhile, sad-batch.service enforces Requires=sad-cgroup-setup.service and After=sad-cgroup-setup.service, and /opt/sad/run-batch.sh also invokes /opt/sad/setup-cgroup.sh idempotently on startup.

Decision

Proceed to remediation: update /opt/sad/setup-cgroup.sh to write 134217728 to $CGROUP/memory.max and restart sad-cgroup-setup.service.


5. Root Cause Analysis

Confirmed Root Cause

The root cause of the outage was a misconfiguration in /opt/sad/setup-cgroup.sh, which wrote the 128 MiB memory budget (134217728 bytes) to the cgroup v2 memory.high control file instead of memory.max.

Underlying Kernel Mechanism

In the Linux cgroup v2 memory controller, memory.high and memory.max serve fundamentally different purposes:

  1. memory.high (Throttling / Best-Effort Reclamation Threshold): When a cgroup’s memory footprint exceeds memory.high, the kernel places its processes under aggressive direct memory reclaim and throttles their execution speed so background reclaimers (kswapd) can free page cache. However, memory.high never invokes the OOM killer. If processes allocate anonymous memory (like Python bytearray objects) that cannot be reclaimed or swapped out, the kernel permits the cgroup to keep growing beyond memory.high.
  2. memory.max (Hard Allocation Ceiling): memory.max defines the absolute maximum memory a cgroup can consume. When usage reaches memory.max and kernel page reclamation fails to free sufficient pages, the kernel immediately triggers a cgroup-scoped OOM event, killing a process inside that specific cgroup rather than allowing it to consume host memory.

Sequence of Events Leading to Failure

flowchart TD
    A[Boot: sad-cgroup-setup.service runs] --> B["setup-cgroup.sh sets memory.high = 134217728<br/>(memory.max remains 'max')"]
    B --> C[sad-batch.service starts run-batch.sh]
    C --> D[Worker PID moves into /sys/fs/cgroup/sad-batch]
    D --> E[sad-batch.sh allocates 256 MiB anonymous RAM]
    E --> F{Usage > memory.high<br/>128 MiB?}
    F -- Yes --> G[Kernel throttles process but allows allocation to continue]
    G --> H[Host physical RAM exhausted]
    H --> I[Global Kernel OOM Killer invoked]
    I --> J[sad-api.service killed; API goes down]

Contributing Factors

  • Anonymous Memory Allocation: sad-batch.sh allocates bytearray buffers in heap memory rather than file-backed page cache, meaning the kernel cannot simply drop clean pages under memory.high pressure.
  • Unprotected Co-location on a Small VM: Because the host has limited RAM, allowing a single batch worker to grow to 256 MiB starved the rest of the OS, making sad-api a prime candidate for the global OOM killer.
  • Documentation Drift: The presence of a 2023 cgroup v1 wiki snippet (memory.limit_in_bytes) alongside cgroup v2 scripts increased the likelihood of confusion around cgroup control file semantics.

6. Solution and Remediation

Changes Applied

  1. Modified /opt/sad/setup-cgroup.sh: Replaced memory.high with memory.max using sed in-place editing.
  2. Restarted sad-cgroup-setup.service: Re-ran the oneshot systemd unit so the updated script immediately wrote 134217728 to /sys/fs/cgroup/sad-batch/memory.max.

Commands Executed

sudo sed -i 's/memory.high/memory.max/' /opt/sad/setup-cgroup.sh
sudo systemctl restart sad-cgroup-setup.service

Before and After Configuration (/opt/sad/setup-cgroup.sh)

Before:

#!/usr/bin/env bash
# Ensure the sad-batch cgroup exists with the intended memory budget.
set -euo pipefail

CGROUP=/sys/fs/cgroup/sad-batch

mkdir -p "$CGROUP"
echo 134217728 > "$CGROUP/memory.high"

After:

#!/usr/bin/env bash
# Ensure the sad-batch cgroup exists with the intended memory budget.
set -euo pipefail

CGROUP=/sys/fs/cgroup/sad-batch

mkdir -p "$CGROUP"
echo 134217728 > "$CGROUP/memory.max"

Rationale, Side Effects, and Rollback

  • Why This Solution Was Selected: It is the smallest, cleanest, and most reversible change that directly addresses the root cause while respecting the host’s architectural constraint (using /opt/sad/setup-cgroup.sh rather than systemd unit directives) and leaving sad-api untouched.
  • Operational Side Effects: With memory.max set to 128 MiB, sad-batch.sh (which attempts to allocate 256 MiB) will be killed with SIGKILL (exit code 137) by the cgroup OOM killer once it reaches ~128 MiB. Because sad-batch.service is configured with Restart=on-failure and RestartSec=5, systemd will restart it every 5 seconds while the unit is active—each time safely contained inside the 128 MiB cgroup boundary without threatening sad-api.
  • Rollback Procedure: Reverting the change requires running sudo sed -i 's/memory.max/memory.high/' /opt/sad/setup-cgroup.sh, writing max back to /sys/fs/cgroup/sad-batch/memory.max, and restarting sad-cgroup-setup.service.

7. Verification and Validation

Confirming that sed and systemctl restart returned exit code 0 is not enough; we must verify both the kernel control file state and actual runtime isolation when the memory-heavy batch job executes.

Step 1: Verify Cgroup Control File Value

cat /sys/fs/cgroup/sad-batch/memory.max

Output:

134217728

This confirms the kernel cgroup v2 memory controller registered the exact 128 MiB (128 * 1024 * 1024) hard cap.

Step 2: Start sad-batch and Test sad-api Survival

sudo systemctl start sad-batch.service
curl -s http://127.0.0.1:9090/

Output:

SadServers - API OK

Starting sad-batch.service launches the 256 MiB allocation loop inside /sys/fs/cgroup/sad-batch. Unlike before, sad-api remains uninterrupted and responds immediately on port 9090.

Step 3: Run the Automated Validation Suite

agent/check.sh

Output:

OK

Persistence Check

Because sad-cgroup-setup.service is enabled (WantedBy=multi-user.target) and runs ExecStart=/opt/sad/setup-cgroup.sh with Before=sad-batch.service, the 134217728 byte limit on /sys/fs/cgroup/sad-batch/memory.max will persist across service restarts and full VM reboots.

(Note: In a production environment, an SRE would also inspect journalctl -u sad-batch.service and /sys/fs/cgroup/sad-batch/memory.events after ~10 seconds to verify the oom_kill counter incremented cleanly while host memory remained stable.)


8. Alternative Troubleshooting Approaches

The steps above reflect the exact investigation performed during the session. Below are alternative diagnostic and remediation approaches (not executed in the original session) that an experienced sysadmin or SRE could use in similar scenarios.

1. Kernel OOM Log Forensics (dmesg / journalctl)

  • Underlying Idea: Whenever the Linux OOM killer terminates a process, the kernel logs detailed diagnostics—including whether it was a global OOM or a cgroup-scoped OOM (Memory cgroup out of memory), the victim PID, oom_score, and RSS memory usage.
  • Commands:
sudo dmesg -T | grep -i -E "oom|killed process"
sudo journalctl -k -g "Out of memory" --no-pager
  • When to Use: Essential during post-incident forensics when you need proof of which process triggered the OOM event and which cgroup hit its limit.

2. Real-Time Cgroup Telemetry (memory.current, memory.events, systemd-cgtop)

  • Underlying Idea: Cgroup v2 exposes real-time memory consumption and OOM event counters directly in /sys/fs/cgroup/<name>/.
  • Commands:
watch -n 1 'cat /sys/fs/cgroup/sad-batch/memory.current /sys/fs/cgroup/sad-batch/memory.events'
systemd-cgtop -m
  • When to Use: Useful during live load testing to watch memory.current climb toward memory.max and verify that high, max, oom, and oom_kill counters in memory.events behave as expected.

3. Verifying Process Cgroup Membership via /proc

  • Underlying Idea: Even when memory.max is configured properly, a service can escape limits if its launcher forks without migrating the PID into cgroup.procs. Checking /proc/<PID>/cgroup proves which cgroup a running process actually belongs to.
  • Commands:
pgrep -f sad-batch.sh
cat /proc/<PID>/cgroup
  • When to Use: Whenever a process appears to exceed its cgroup limit despite memory.max being set correctly.

4. Native Systemd Resource Directives (Alternative Remediation)

  • Underlying Idea: Instead of maintaining custom bash scripts (setup-cgroup.sh and run-batch.sh) to manually mkdir directories and echo PIDs into cgroup.procs, systemd can manage cgroup v2 limits natively.
  • Example Configuration (/etc/systemd/system/sad-batch.service):
[Service]
MemoryMax=128M

Or dynamically via CLI:

sudo systemctl set-property sad-batch.service MemoryMax=128M

Comparison: Custom Cgroup Scripts vs. Native Systemd Resource Control

Criterion Custom Cgroup v2 Script (/opt/sad/) Native Systemd (MemoryMax=128M)
Complexity Higher; requires separate setup/wrapper scripts and manual cgroup.procs migration. Low; declarative single-line directive inside the .service unit.
Reliability Prone to race conditions if unit ordering (Before=/After=) is misconfigured. High; systemd configures the cgroup before execve() of the service binary.
Maintainability Non-standard; requires custom documentation so operators know not to look in systemd. Standard across modern Linux distributions; visible directly in systemctl status.
Suitability for Exercise Required here because incident-notes.txt and check.sh explicitly test /sys/fs/cgroup/sad-batch/memory.max. Best practice for greenfield production services managed by systemd.

9. Linux Command Deep Dive

Command: systemctl

Purpose

systemctl is the primary control utility for systemd, used to inspect, start, stop, enable, disable, and configure system services, targets, slices, and resource properties.

General Syntax

systemctl [OPTIONS...] COMMAND [UNIT...]

Options and Arguments

Command / Option Purpose
systemctl status <unit> Show runtime state, main PID, cgroup path, memory usage, and recent logs.
--no-pager Print full output directly to stdout without invoking less.
systemctl restart <unit> Stop and immediately start a unit (re-runs ExecStart= for oneshot units).
systemctl is-active --quiet <unit> Return exit code 0 if the unit is active, non-zero otherwise (ideal for health-check scripts).
systemctl show <unit> -p MemoryMax Print the resolved value of a specific unit property.
systemctl daemon-reload Reload unit files from disk after editing /etc/systemd/system/*.service.

Practical Examples

# List all failed systemd units on the host
systemctl --failed

# Check resolved memory limits on a systemd-managed service
systemctl show sad-api.service -p MemoryCurrent -p MemoryMax

# View dependency tree for a service
systemctl list-dependencies sad-batch.service

Common Mistakes and Limitations

  • Editing a .service file on disk without running sudo systemctl daemon-reload causes systemd to continue using the cached unit definition in memory (note: editing a bash script called by ExecStart=, like /opt/sad/setup-cgroup.sh, does not require daemon-reload, only restarting the unit).
  • Confusing enabled (configured to start at boot) with active (currently running in memory).

Command: sed

Purpose

sed (Stream Editor) parses and transforms text streams or files line by line using regular expressions. It is widely used in sysadmin automation for non-interactive configuration updates.

General Syntax

sed [OPTIONS...] 'SCRIPT' [FILE...]

Options and Arguments

Option / Flag Purpose
-i Edit files in-place instead of printing modified text to stdout.
-i.bak Edit in-place while saving a backup copy of the original file with a .bak suffix.
-E Enable Extended Regular Expressions (ERE), avoiding the need to backslash +, ?, `
-n Suppress automatic printing of pattern space (often paired with p to print matching lines).
's/old/new/g' Substitute old with new; the g flag replaces all occurrences on a line rather than just the first.

Practical Examples

# Replace text in-place while creating a timestamped backup
sudo sed -i.bak 's/memory.high/memory.max/' /opt/sad/setup-cgroup.sh

# Print lines 1 through 15 of a configuration file
sed -n '1,15p' /etc/systemd/system/sad-batch.service

# Strip comments and blank lines when reading a config file
sed -E '/^\s*#|^\s*$/d' /etc/ssh/sshd_config

Common Mistakes and Limitations

  • GNU vs. BSD sed: On Linux (GNU sed), -i takes an optional suffix directly attached (-i.bak). On macOS/BSD, -i requires an explicit argument (e.g., sed -i '' 's/a/b/' file).
  • Running sed -i without testing the regex first (without -i) can corrupt configuration files if the pattern matches unintended lines.

Command: curl

Purpose

curl transfers data to or from a server over HTTP, HTTPS, and other network protocols. It is the standard CLI tool for probing local and remote API health endpoints.

General Syntax

curl [OPTIONS...] <URL>

Options and Arguments

Option Purpose
-s (--silent) Silent mode; hides the progress bar and error messages.
-S (--show-error) When paired with -s (-sS), hides progress bars but still prints errors if the request fails.
-m <seconds> Maximum time allowed for the entire operation (prevents scripts from hanging).
-I (--head) Fetch HTTP response headers only (HEAD request).
-i (--include) Include HTTP response headers alongside the response body.
-f (--fail) Return exit code 22 on HTTP errors (4xx/5xx) instead of silently outputting the error HTML/JSON.

Practical Examples

# Health check returning only the HTTP status code
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9090/

# Probe an endpoint with a strict 2-second timeout and fail on HTTP >= 400
curl -fsSm 2 http://127.0.0.1:9090/

Common Mistakes and Limitations

  • By default, curl exits with code 0 even if the server returns HTTP 500 Internal Server Error or 404 Not Found. Always pass -f (--fail) or inspect %{http_code} when using curl in health checks.

10. Underlying Linux Concepts and Architecture

1. Control Groups: cgroup v1 vs. cgroup v2

Control Groups are a Linux kernel feature that organizes processes hierarchically and distributes system resources in a controlled manner:

  • cgroup v1 (Legacy): Mounted separate directory trees per resource controller (e.g., /sys/fs/cgroup/memory/, /sys/fs/cgroup/cpu/). A process could belong to completely different hierarchies for memory and CPU, making coordinated resource management (such as buffered disk I/O accounting) difficult. Memory limits were set via memory.limit_in_bytes.
  • cgroup v2 (Unified Hierarchy): Introduced a single, unified tree mounted at /sys/fs/cgroup/. Every cgroup directory exposes control files for all enabled controllers in one place. Memory limits use memory.max and memory.high.

2. Cgroup v2 Memory Controller Hierarchy

Cgroup v2 provides four primary memory thresholds, ordered from strongest protection to hard ceiling:

  1. memory.min (Hard Memory Protection): Guarantees a minimum amount of memory that the kernel will never reclaim under normal pressure.
  2. memory.low (Best-Effort Memory Protection): Soft guarantee; the kernel avoids reclaiming memory below this threshold unless non-protected cgroups have no reclaimable pages left.
  3. memory.high (Throttle Threshold): Soft ceiling; exceeding this triggers aggressive direct page reclaim and CPU throttling of allocating threads, without invoking the OOM killer.
  4. memory.max (Hard Ceiling): Absolute limit; exceeding this when pages cannot be reclaimed triggers the cgroup OOM killer.

3. Process Migration via cgroup.procs and exec

In /opt/sad/run-batch.sh, two shell mechanisms work together to ensure the Python process is constrained:

echo $$ > /sys/fs/cgroup/sad-batch/cgroup.procs
exec /opt/sad/sad-batch.sh
  • Writing a PID to cgroup.procs moves that process into the target cgroup. Any child processes subsequently created via fork() automatically inherit their parent’s cgroup membership.
  • Using exec replaces the current bash shell process image in-place without changing its PID ($$), ensuring sad-batch.sh (and subsequently python3) runs directly inside /sys/fs/cgroup/sad-batch with zero extra wrapper processes left behind.

4. Global OOM Killer vs. Cgroup-Scoped OOM Killer

  • Global OOM Killer: Triggered when the entire host runs out of physical RAM (and swap). The kernel evaluates every process on the machine using a heuristic (oom_score, roughly proportional to percentage of total RAM used, adjusted by /proc/<PID>/oom_score_adj) and kills the highest-scoring victim. On a small VM, a critical API process can easily be chosen.
  • Cgroup-Scoped OOM Killer: Triggered when a cgroup hits memory.max. The kernel restricts victim selection exclusively to processes within that cgroup (or kills all processes in the cgroup atomically if memory.oom.group is set to 1), completely insulating the rest of the system.

5. Systemd Dependency & Ordering Semantics

Notice how sad-batch.service specifies both Requires= and After=:

  • Requires=sad-cgroup-setup.service: Declares a hard activation dependency—if sad-cgroup-setup.service fails, sad-batch.service will not start. However, Requires= alone does not dictate order; systemd starts required units in parallel by default.
  • After=sad-cgroup-setup.service: Enforces sequential ordering, guaranteeing sad-cgroup-setup.service finishes (Type=oneshot completes ExecStart=) before sad-batch.service launches.

11. Real-World Sysadmin and SRE Applications

Detection and Observability in Production

In production environments, OOM kills should never be discovered only when a user reports that an API is down. SRE teams monitor cgroup telemetry continuously using tools like Prometheus (node_exporter and cAdvisor):

  • memory.events Counters: Monitor high, max, oom, and oom_kill inside /sys/fs/cgroup/<slice>/<service>/memory.events. A rising high or max count provides early warning of memory pressure before oom_kill events occur.
  • Pressure Stall Information (PSI): Modern Linux kernels expose /sys/fs/cgroup/<name>/memory.pressure (and /proc/pressure/memory), measuring the exact percentage of wall-clock time tasks spend stalled waiting for memory allocation or reclaim (some and full metrics).

Alerting and SLO Protection

  • Protect Tier-0 APIs: Customer-facing APIs directly impact availability and latency Service Level Objectives (SLOs). Co-locating batch ETL jobs on the same host without strict MemoryMax= boundaries violates fault isolation.
  • Actionable Alerts: Alert on increase(container_oom_events_total[5m]) > 0 (or systemd unit OOMKills property changes) routed to the batch workload’s owning team, rather than paging the API on-call engineer for a batch job’s memory regression.

Preventive Engineering & Toil Reduction

  • Eliminate Custom Cgroup Scripts: Maintaining bespoke bash scripts under /opt/sad/ to manipulate /sys/fs/cgroup/ introduces operational drift and hidden dependencies. Standardizing on systemd unit properties (MemoryMax=, MemoryHigh=, OOMPolicy=kill) managed via Infrastructure as Code (Ansible, Terraform, or Puppet) makes resource limits auditable and uniform.
  • Pre-Production Memory Profiling: Include memory regression checks in CI/CD pipelines for batch jobs so a code change that doubles heap consumption (e.g., loading an entire dataset into memory instead of streaming chunks) is caught before reaching production.

12. Further Learning and Experiments

Hands-on Experiments

You can perform the following safe experiments on a non-production Linux VM to observe cgroup v2 memory controls firsthand. (Warning: Do not run memory exhaustion tests on production hosts.)

Experiment 1: Comparing MemoryHigh Throttling vs. MemoryMax OOM Kill

  1. Objective: Observe how systemd-run creates transient cgroup v2 scopes and see the difference between MemoryHigh and MemoryMax.
  2. Commands:
# Test MemoryHigh (process slows down under reclaim pressure, may complete if host has RAM)
systemd-run --user --scope -p MemoryHigh=32M python3 -c 'a = bytearray(64*1024*1024); print("Survived MemoryHigh")'

# Test MemoryMax (kernel immediately kills the process with SIGKILL / exit code 137)
systemd-run --user --scope -p MemoryMax=32M python3 -c 'a = bytearray(64*1024*1024); print("Survived MemoryMax")'
echo $?
  1. What You Will Learn: The first command prints Survived MemoryHigh (assuming free host RAM exists), whereas the second command is terminated by the cgroup OOM killer and prints exit status 137 (128 + 9 for SIGKILL).

Experiment 2: Inspecting oom_score and oom_score_adj

  1. Objective: Understand how the kernel ranks processes when selecting an OOM victim.
  2. Commands:
# View the OOM score of your current shell and systemd-journald
cat /proc/$$/oom_score
cat /proc/$$/oom_score_adj
  1. What You Will Learn: Scores range from 0 to 1000 (plus oom_score_adj from -1000 to +1000). Critical system daemons often set negative oom_score_adj values (-250 or -1000) to protect themselves from global OOM events.

Topics to Explore Next

  • Linux Pressure Stall Information (PSI): Measuring CPU, memory, and I/O contention via /proc/pressure/*.
  • Systemd Slice Hierarchy: Organizing workloads into system.slice, user.slice, and custom .slice units (e.g., batch.slice vs. api.slice) to share aggregate resource budgets.
  • Container Runtime Cgroup Drivers: How Docker, containerd, and Kubernetes map pod requests.memory and limits.memory directly onto cgroup v2 memory.low/memory.min and memory.max.

Authoritative Documentation and References

  • man 7 cgroups: Comprehensive overview of Linux cgroup v1 and v2 architecture.
  • man 5 systemd.resource-control: Official reference for MemoryMax=, MemoryHigh=, MemoryLow=, and OOMPolicy= in systemd units.
  • man 5 proc: Detailed documentation of /proc/<PID>/cgroup, /proc/<PID>/oom_score, and /proc/pressure/memory.
  • Linux Kernel Documentation: Documentation/admin-guide/cgroup-v2.rst in the official Linux kernel source tree.

13. Key Takeaways

  • Soft vs. Hard Limits in Cgroup v2: memory.high only throttles and reclaims memory; memory.max is required to enforce a hard ceiling and isolate OOM kills to a specific cgroup.
  • Don’t Confuse Cgroup v1 and v2: Cgroup v1 used per-controller paths and memory.limit_in_bytes; cgroup v2 uses a unified hierarchy at /sys/fs/cgroup/ with memory.max.
  • Verify Both State and Behaviour: Checking that memory.max contains 134217728 confirms configuration state, while running sad-batch and verifying sad-api remains responsive confirms runtime fault isolation.
  • Understand Requires= vs. After= in Systemd: Requires= enforces dependency activation, while After= enforces sequential execution order—both are needed when a oneshot setup unit must finish before a worker unit starts.
  • Prefer Declarative Resource Controls: Whenever architectural constraints allow, use native systemd directives (MemoryMax=) instead of custom bash scripts manipulating /sys/fs/cgroup/.

14. Quick Troubleshooting Reference / Runbook

Use this checklist when investigating suspected cgroup memory limit failures or unexpected OOM kills on Linux:

  1. Confirm Cgroup Version Mounted on Host
stat -fc %T /sys/fs/cgroup/
# Output "cgroup2fs" = cgroup v2 unified hierarchy; "tmpfs" = cgroup v1
  1. Check Kernel Logs for Recent OOM Events
sudo dmesg -T | grep -i -E "oom|killed process"
  1. Inspect Service Status and Systemd Cgroup Path
systemctl status <service-name> --no-pager
  1. Verify Process Membership in Target Cgroup
cat /proc/<PID>/cgroup
  1. Inspect Live Cgroup v2 Memory Limits and Usage
cat /sys/fs/cgroup/<cgroup-path>/memory.max
cat /sys/fs/cgroup/<cgroup-path>/memory.high
cat /sys/fs/cgroup/<cgroup-path>/memory.current
cat /sys/fs/cgroup/<cgroup-path>/memory.events
  1. Check Systemd Unit Resource Properties (If Managed via Systemd)
systemctl show <service-name> -p MemoryMax -p MemoryHigh -p OOMPolicy
  1. Apply Hard Limit Fix & Reload/Restart Unit
  • For custom cgroup v2 scripts: write byte limit to memory.max and restart the setup unit.
  • For native systemd units: set MemoryMax=<size> in the [Service] block, run sudo systemctl daemon-reload, and restart the service.
  1. Validate Isolation Under Load
  • Trigger the memory-intensive workload, verify memory.events (oom_kill) isolates the failure to the target cgroup, and confirm critical co-located services remain healthy.