1. Introduction
Resource contention—often referred to as the “noisy neighbor” problem—is one of the most common causes of service outages on shared Linux hosts. When an asynchronous, memory-intensive batch job runs alongside a latency-sensitive web API on the same machine, an unbounded memory spike in the batch process can exhaust physical RAM and trigger the Linux kernel’s Out-Of-Memory (OOM) killer. Without strict resource boundaries, the kernel may terminate the critical API instead of the runaway batch worker.
In modern Linux environments, Control Groups (cgroups v2) provide the kernel mechanism for partitioning and limiting hardware resources such as memory, CPU, and I/O across groups of processes. However, misconfiguring cgroup v2 control files—or confusing soft throttling thresholds with hard limits—can silently undermine workload isolation.
In this article, we walk through a real-world troubleshooting exercise based on the SadServers “Genova” scenario. You will learn how to:
- Systematically triage a cgroup v2 configuration failure using read-only diagnostics before modifying system state.
- Distinguish between cgroup v1 legacy artifacts, cgroup v2 soft limits (
memory.high), and cgroup v2 hard limits (memory.max). - Trace how
systemdunits interact with custom cgroup management scripts during boot and service execution. - Apply and verify a targeted, persistent fix that contains OOM events within a specific cgroup hierarchy.
2. Problem Description and Environment
Scenario Overview
A small Linux virtual machine hosts two primary workloads:
sad-api: A lightweight Python health endpoint listening locally on127.0.0.1:9090.sad-batch: A nightly ETL-style worker script that allocates large amounts of RAM.
Following a recent deployment, starting sad-batch caused memory consumption to spike rapidly, resulting in sad-api being terminated by the kernel OOM killer. An on-call engineer stopped and disabled sad-batch prior to handing off the host and left context in /home/admin/incident-notes.txt.
Expected vs. Actual Behaviour
- Expected Behaviour:
sad-batchexecutes inside a dedicated cgroup v2 hierarchy (/sys/fs/cgroup/sad-batch) capped at a hard limit of 128 MiB (134217728bytes). Ifsad-batchattempts to allocate more than 128 MiB, the kernel OOM killer terminates only processes within thesad-batchcgroup, leavingsad-apihealthy and responsive on127.0.0.1:9090. - Actual Behaviour: The 128 MiB cap never takes effect.
/sys/fs/cgroup/sad-batch/memory.maxremains set tomax(unlimited), allowingsad-batchto consume host RAM until the global OOM killer terminatessad-api.
Environment Details & Constraints
Based on the diagnostic evidence collected during the session:
- Hostname:
i-0285bb94fbc615e56 - User Context:
adminwith passwordlesssudoprivileges. - Init & Service Manager:
systemd - Cgroup Hierarchy: Unified cgroup v2 mounted at
/sys/fs/cgroup/. - OS & Kernel Version: Specific distribution name and kernel release were not queried during the session (
uname -aand/etc/os-releasewere not executed), though the unified cgroup v2 mount confirms a modern Linux kernel supporting cgroups v2. - Operational Constraints:
- Do not modify
sad-api; it must remain active and returnSadServers - API OKfrom[http://127.0.0.1:9090/](http://127.0.0.1:9090/). /sys/fs/cgroup/sad-batch/memory.maxmust be configured to134217728before restartingsad-batch.- Do not modify the grading script
/home/admin/agent/check.sh.
3. Initial Investigation and Hypotheses
When approaching an unfamiliar system after an incident, resisting the urge to immediately edit files is critical. Changing configuration before understanding how services, scripts, and kernel interfaces interact often introduces secondary failures or masks the true root cause.
What Was Initially Known vs. Unknown
- Known:
sad-apiwas killed by the OOM killer whensad-batchran; a legacy cgroup v2 launcher under/opt/sad/is supposed to enforce a 128 MiB limit;sad-batchis currently stopped and disabled. - Unknown: Whether
/sys/fs/cgroup/sad-batchexists at boot, what valuememory.maxcurrently holds, howsystemdorchestrates the cgroup setup relative tosad-batch, and why the launcher’s limit fails to apply.
Forming Initial Hypotheses
Before inspecting the scripts, four plausible hypotheses could explain why a cgroup memory cap fails to apply:
| Hypothesis | Supporting Evidence | Evidence Against | Investigation Step |
|---|---|---|---|
1. Soft limit (memory.high) used instead of hard limit (memory.max) |
Problem states a 128 MiB limit is configured in /opt/sad/ but /sys/fs/cgroup/sad-batch/memory.max is not 134217728. |
None yet; requires inspecting the setup script. | Inspect /opt/sad/setup-cgroup.sh and /sys/fs/cgroup/sad-batch/memory.max. |
| 2. Cgroup v1 syntax used on a cgroup v2 host | Problem description explicitly calls /opt/sad/ a “legacy” cgroup launcher. |
Problem mentions /sys/fs/cgroup/sad-batch/memory.max exists as a path structure. |
Read incident-notes.txt and /opt/sad/setup-cgroup.sh to check target paths and filenames. |
| 3. Boot setup service failed or was not triggered | If the setup script never ran, memory.max would remain at its default (max). |
None yet; service status has not been checked. | Run systemctl status on the sad-* units. |
| 4. Worker process never joins the target cgroup | Even if memory.max were set, a process running outside /sys/fs/cgroup/sad-batch would bypass the limit. |
Problem states memory.max itself does not show 134217728, pointing first to cgroup setup. |
Inspect /opt/sad/run-batch.sh to verify cgroup.procs assignment. |
4. Troubleshooting Process
Step 1: Reviewing Incident Notes and Launcher Directory
Objective
Gather operational context left by the on-call engineer in /home/admin/incident-notes.txt and list the files inside /opt/sad/ to map out the custom launcher architecture.
Command
cat incident-notes.txt
ls -la /opt/sad/
Command Explanation
cat incident-notes.txt: Reads and prints the contents ofincident-notes.txtfrom the current working directory (/home/admin) to standard output.ls -la /opt/sad/: Lists all directory entries inside/opt/sad/.-l: Uses the long listing format, displaying file permissions, link count, owner, group, size in bytes, and modification timestamp.-a: Includes hidden files (entries beginning with.).
Output and Analysis
Incident: 2026-06-10 — nightly batch run took down the API
After the batch job was redeployed, sad-batch started allocating hundreds of
megabytes during its run. sad-api (port 9090) was killed by the OOM killer and
stayed down until someone restarted it manually.
This host does NOT use systemd resource limits for the batch job. Memory control
is handled by a legacy cgroup v2 launcher under /opt/sad/ (see setup-cgroup.sh
and run-batch.sh). On-call disabled sad-batch before handing the VM to you.
...
Ops ticket excerpt:
"We set a 128M cap in setup-cgroup.sh — why is memory.max still max?"
Legacy wiki snippet (2023, cgroup v1 — do not trust blindly):
echo 134217728 > /sys/fs/cgroup/memory/sad-batch/memory.limit_in_bytes
total 24
drwxr-xr-x 2 root root 4096 Jun 16 18:54 .
drwxr-xr-x 4 root root 4096 Jun 16 18:54 ..
-rwxr-xr-x 1 root root 194 Jun 16 18:54 run-batch.sh
-rwxr-xr-x 1 root root 447 Jun 16 18:54 sad-api.sh
-rwxr-xr-x 1 root root 219 Jun 16 18:54 sad-batch.sh
-rwxr-xr-x 1 root root 200 Jun 16 18:54 setup-cgroup.sh
The incident notes provide three critical clues:
- Systemd native memory directives (like
MemoryMax=) are intentionally not used forsad-batch; memory enforcement is delegated to/opt/sad/setup-cgroup.shand/opt/sad/run-batch.sh. - The ops ticket explicitly confirms that
memory.maxis still readingmaxdespite a 128M cap being configured insetup-cgroup.sh. - The legacy wiki snippet (
memory.limit_in_bytes) is a classic cgroup v1 red herring, warning us not to confuse cgroup v1 filenames with cgroup v2.
The ls -la output shows four root-owned executable (-rwxr-xr-x) shell scripts in /opt/sad/.
Decision
Check the current status of the systemd services (sad-api, sad-cgroup-setup, and sad-batch) and verify that sad-api is currently serving traffic before inspecting the scripts.
Step 2: Auditing Systemd Unit States and Baseline API Health
Objective
Verify the runtime state of all three related systemd services and confirm that sad-api is healthy on 127.0.0.1:9090.
Command
systemctl status sad-api sad-cgroup-setup sad-batch --no-pager
curl -s http://127.0.0.1:9090
Command Explanation
-
systemctl status <units> --no-pager: Queriessystemdfor the loaded state, active state, main PID, memory footprint, and recent journal entries for multiple units simultaneously. -
--no-pager: Prints output directly to stdout rather than piping throughless, preventing truncation or interactive blocking. -
curl -s [http://127.0.0.1:9090](http://127.0.0.1:9090): Performs an HTTP GET request against the local API endpoint. -
-s(--silent): Suppresses progress meters and error output so only the response body is displayed.
Output and Analysis
● sad-api.service - SadServers API health endpoint
Loaded: loaded (/etc/systemd/system/sad-api.service; enabled; preset: enabled)
Active: active (running) since Wed 2026-11-11 14:16:14 UTC; 2min 8s ago
Main PID: 762 (python3)
Memory: 8.6M (max: 48M, available: 39.3M, peak: 8.9M)
CGroup: /system.slice/sad-api.service
└─762 python3 /opt/sad/sad-api.sh
● sad-cgroup-setup.service - Create sad-batch cgroup at boot
Loaded: loaded (/etc/systemd/system/sad-cgroup-setup.service; enabled; preset: enabled)
Active: active (exited) since Wed 2026-11-11 14:15:58 UTC; 2min 24s ago
Main PID: 265 (code=exited, status=0/SUCCESS)
○ sad-batch.service - SadServers batch ETL worker
Loaded: loaded (/etc/systemd/system/sad-batch.service; disabled; preset: enabled)
Active: inactive (dead)
SadServers - API OK
sad-api.serviceisactive (running)under/system.slice/sad-api.service(with its own systemd memory cap of 48M) and responds withSadServers - API OK.sad-cgroup-setup.serviceran at boot and exited cleanly (status=0/SUCCESS), ruling out Hypothesis 3 (boot setup script crash).sad-batch.serviceisdisabledandinactive (dead), confirming it is safe to inspect the cgroup filesystem without active memory pressure.
Decision
Inspect the live cgroup control files under /sys/fs/cgroup/sad-batch/ and read the contents of /opt/sad/setup-cgroup.sh, run-batch.sh, and sad-batch.sh.
Step 3: Inspecting the Live Cgroup v2 Control File
Objective
Confirm the directory structure under /sys/fs/cgroup/sad-batch/ and check the current value of memory.max.
Command
cat /sys/fs/cgroup/sad-batch/memory.max
Command Explanation
cat /sys/fs/cgroup/sad-batch/memory.max: Reads the kernel virtual file representing the hard memory limit for thesad-batchcgroup. In Linux,/sys(sysfs/cgroup2) exposes kernel data structures as plain-text files.
Output and Analysis
max
Tab-completion on /sys/fs/cgroup/sad-batch/memory. also confirmed the presence of standard cgroup v2 memory controller files (memory.current, memory.events, memory.high, memory.low, memory.max, memory.min, memory.oom.group, memory.peak, memory.pressure, memory.stat, etc.).
The directory /sys/fs/cgroup/sad-batch exists (created by sad-cgroup-setup.service), confirming the host is running cgroup v2, but memory.max is indeed max (unlimited).
Decision
Read the shell scripts in /opt/sad/ to see what setup-cgroup.sh writes and how run-batch.sh launches the worker.
Step 4: Examining the Legacy Launcher and Worker Scripts
Objective
Inspect the exact implementation of the cgroup setup script, the process launcher, and the batch workload.
Command
cat /opt/sad/setup-cgroup.sh
cat /opt/sad/run-batch.sh
cat /opt/sad/sad-batch.sh
Command Explanation
cat /opt/sad/<script>: Prints the source code of each script to examine how cgroup directories, limits, process assignments, and memory allocations are handled.
Output and Analysis
/opt/sad/setup-cgroup.sh:
#!/usr/bin/env bash
# Ensure the sad-batch cgroup exists with the intended memory budget.
set -euo pipefail
CGROUP=/sys/fs/cgroup/sad-batch
mkdir -p "$CGROUP"
echo 134217728 > "$CGROUP/memory.high"
/opt/sad/run-batch.sh:
#!/usr/bin/env bash
# Launch the batch worker inside the sad-batch cgroup.
set -euo pipefail
/opt/sad/setup-cgroup.sh
echo $$ > /sys/fs/cgroup/sad-batch/cgroup.procs
exec /opt/sad/sad-batch.sh
/opt/sad/sad-batch.sh:
#!/usr/bin/env bash
# Memory-heavy batch worker; must stay inside its cgroup budget.
exec python3 - <<'PY'
import time
chunks = []
for _ in range(256):
chunks.append(bytearray(1024 * 1024))
time.sleep(0.05)
PY
This reveals the exact defect:
- In
setup-cgroup.sh,134217728(128 MiB) is written to$CGROUP/memory.highrather than$CGROUP/memory.max. This confirms Hypothesis 1. run-batch.shproperly ensures the cgroup exists, writes its own PID ($$) into/sys/fs/cgroup/sad-batch/cgroup.procs, and usesexecto replace the shell withsad-batch.sh(which in turn usesexec python3), ensuring the Python worker inherits membership in/sys/fs/cgroup/sad-batch. This rules out Hypothesis 4.sad-batch.shallocates 256 chunks of 1 MiB (1024 * 1024bytes) each—a total of 256 MiB—over ~12.8 seconds.
Decision
Before modifying setup-cgroup.sh, inspect the systemd unit files and the validation script (/home/admin/agent/check.sh) to ensure there are no hidden dependencies, override drop-ins, or side effects.
Step 5: Auditing Systemd Unit Definitions and Validation Criteria
Objective
Verify the ordering and dependency relationship between sad-cgroup-setup.service and sad-batch.service, and check the exact pass criteria in check.sh.
Command
cat /home/admin/agent/check.sh
cat /etc/systemd/system/sad-cgroup-setup.service
cat /etc/systemd/system/sad-batch.service
Command Explanation
cat /home/admin/agent/check.sh: Inspects the automated evaluation script (marked read-only/do not modify).cat /etc/systemd/system/sad-*.service: Reads the unit definitions to checkType=,ExecStart=,Requires=,Before=,After=, andRestart=directives.
Output and Analysis
/home/admin/agent/check.sh:
#!/usr/bin/env bash
# DO NOT MODIFY THIS FILE ("Check My Solution" will fail)
fail() {
echo -n "NO"
exit 0
}
systemctl is-active --quiet sad-api.service || fail
if ! curl -s -m 2 http://127.0.0.1:9090/ | grep -q 'SadServers - API OK'; then
fail
fi
if [ ! -f /sys/fs/cgroup/sad-batch/memory.max ]; then
fail
fi
max=$(tr -d '[:space:]' < /sys/fs/cgroup/sad-batch/memory.max 2>/dev/null)
if [ -z "$max" ] || [ "$max" = "max" ]; then
fail
fi
if [ "$max" != "134217728" ]; then
fail
fi
echo -n "OK"
/etc/systemd/system/sad-cgroup-setup.service:
[Unit]
Description=Create sad-batch cgroup at boot
Before=sad-batch.service
DefaultDependencies=no
[Service]
Type=oneshot
ExecStart=/opt/sad/setup-cgroup.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
/etc/systemd/system/sad-batch.service:
[Unit]
Description=SadServers batch ETL worker
After=network.target sad-cgroup-setup.service
Requires=sad-cgroup-setup.service
[Service]
Type=simple
ExecStart=/opt/sad/run-batch.sh
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
The systemd wiring is sound: sad-cgroup-setup.service uses Type=oneshot with RemainAfterExit=yes and runs Before=sad-batch.service. Meanwhile, sad-batch.service enforces Requires=sad-cgroup-setup.service and After=sad-cgroup-setup.service, and /opt/sad/run-batch.sh also invokes /opt/sad/setup-cgroup.sh idempotently on startup.
Decision
Proceed to remediation: update /opt/sad/setup-cgroup.sh to write 134217728 to $CGROUP/memory.max and restart sad-cgroup-setup.service.
5. Root Cause Analysis
Confirmed Root Cause
The root cause of the outage was a misconfiguration in /opt/sad/setup-cgroup.sh, which wrote the 128 MiB memory budget (134217728 bytes) to the cgroup v2 memory.high control file instead of memory.max.
Underlying Kernel Mechanism
In the Linux cgroup v2 memory controller, memory.high and memory.max serve fundamentally different purposes:
memory.high(Throttling / Best-Effort Reclamation Threshold): When a cgroup’s memory footprint exceedsmemory.high, the kernel places its processes under aggressive direct memory reclaim and throttles their execution speed so background reclaimers (kswapd) can free page cache. However,memory.highnever invokes the OOM killer. If processes allocate anonymous memory (like Pythonbytearrayobjects) that cannot be reclaimed or swapped out, the kernel permits the cgroup to keep growing beyondmemory.high.memory.max(Hard Allocation Ceiling):memory.maxdefines the absolute maximum memory a cgroup can consume. When usage reachesmemory.maxand kernel page reclamation fails to free sufficient pages, the kernel immediately triggers a cgroup-scoped OOM event, killing a process inside that specific cgroup rather than allowing it to consume host memory.
Sequence of Events Leading to Failure
flowchart TD
A[Boot: sad-cgroup-setup.service runs] --> B["setup-cgroup.sh sets memory.high = 134217728<br/>(memory.max remains 'max')"]
B --> C[sad-batch.service starts run-batch.sh]
C --> D[Worker PID moves into /sys/fs/cgroup/sad-batch]
D --> E[sad-batch.sh allocates 256 MiB anonymous RAM]
E --> F{Usage > memory.high<br/>128 MiB?}
F -- Yes --> G[Kernel throttles process but allows allocation to continue]
G --> H[Host physical RAM exhausted]
H --> I[Global Kernel OOM Killer invoked]
I --> J[sad-api.service killed; API goes down]
Contributing Factors
- Anonymous Memory Allocation:
sad-batch.shallocatesbytearraybuffers in heap memory rather than file-backed page cache, meaning the kernel cannot simply drop clean pages undermemory.highpressure. - Unprotected Co-location on a Small VM: Because the host has limited RAM, allowing a single batch worker to grow to 256 MiB starved the rest of the OS, making
sad-apia prime candidate for the global OOM killer. - Documentation Drift: The presence of a 2023 cgroup v1 wiki snippet (
memory.limit_in_bytes) alongside cgroup v2 scripts increased the likelihood of confusion around cgroup control file semantics.
6. Solution and Remediation
Changes Applied
- Modified
/opt/sad/setup-cgroup.sh: Replacedmemory.highwithmemory.maxusingsedin-place editing. - Restarted
sad-cgroup-setup.service: Re-ran theoneshotsystemd unit so the updated script immediately wrote134217728to/sys/fs/cgroup/sad-batch/memory.max.
Commands Executed
sudo sed -i 's/memory.high/memory.max/' /opt/sad/setup-cgroup.sh
sudo systemctl restart sad-cgroup-setup.service
Before and After Configuration (/opt/sad/setup-cgroup.sh)
Before:
#!/usr/bin/env bash
# Ensure the sad-batch cgroup exists with the intended memory budget.
set -euo pipefail
CGROUP=/sys/fs/cgroup/sad-batch
mkdir -p "$CGROUP"
echo 134217728 > "$CGROUP/memory.high"
After:
#!/usr/bin/env bash
# Ensure the sad-batch cgroup exists with the intended memory budget.
set -euo pipefail
CGROUP=/sys/fs/cgroup/sad-batch
mkdir -p "$CGROUP"
echo 134217728 > "$CGROUP/memory.max"
Rationale, Side Effects, and Rollback
- Why This Solution Was Selected: It is the smallest, cleanest, and most reversible change that directly addresses the root cause while respecting the host’s architectural constraint (using
/opt/sad/setup-cgroup.shrather than systemd unit directives) and leavingsad-apiuntouched. - Operational Side Effects: With
memory.maxset to 128 MiB,sad-batch.sh(which attempts to allocate 256 MiB) will be killed withSIGKILL(exit code137) by the cgroup OOM killer once it reaches ~128 MiB. Becausesad-batch.serviceis configured withRestart=on-failureandRestartSec=5,systemdwill restart it every 5 seconds while the unit is active—each time safely contained inside the 128 MiB cgroup boundary without threateningsad-api. - Rollback Procedure: Reverting the change requires running
sudo sed -i 's/memory.max/memory.high/' /opt/sad/setup-cgroup.sh, writingmaxback to/sys/fs/cgroup/sad-batch/memory.max, and restartingsad-cgroup-setup.service.
7. Verification and Validation
Confirming that sed and systemctl restart returned exit code 0 is not enough; we must verify both the kernel control file state and actual runtime isolation when the memory-heavy batch job executes.
Step 1: Verify Cgroup Control File Value
cat /sys/fs/cgroup/sad-batch/memory.max
Output:
134217728
This confirms the kernel cgroup v2 memory controller registered the exact 128 MiB (128 * 1024 * 1024) hard cap.
Step 2: Start sad-batch and Test sad-api Survival
sudo systemctl start sad-batch.service
curl -s http://127.0.0.1:9090/
Output:
SadServers - API OK
Starting sad-batch.service launches the 256 MiB allocation loop inside /sys/fs/cgroup/sad-batch. Unlike before, sad-api remains uninterrupted and responds immediately on port 9090.
Step 3: Run the Automated Validation Suite
agent/check.sh
Output:
OK
Persistence Check
Because sad-cgroup-setup.service is enabled (WantedBy=multi-user.target) and runs ExecStart=/opt/sad/setup-cgroup.sh with Before=sad-batch.service, the 134217728 byte limit on /sys/fs/cgroup/sad-batch/memory.max will persist across service restarts and full VM reboots.
(Note: In a production environment, an SRE would also inspect journalctl -u sad-batch.service and /sys/fs/cgroup/sad-batch/memory.events after ~10 seconds to verify the oom_kill counter incremented cleanly while host memory remained stable.)
8. Alternative Troubleshooting Approaches
The steps above reflect the exact investigation performed during the session. Below are alternative diagnostic and remediation approaches (not executed in the original session) that an experienced sysadmin or SRE could use in similar scenarios.
1. Kernel OOM Log Forensics (dmesg / journalctl)
- Underlying Idea: Whenever the Linux OOM killer terminates a process, the kernel logs detailed diagnostics—including whether it was a global OOM or a cgroup-scoped OOM (
Memory cgroup out of memory), the victim PID,oom_score, and RSS memory usage. - Commands:
sudo dmesg -T | grep -i -E "oom|killed process"
sudo journalctl -k -g "Out of memory" --no-pager
- When to Use: Essential during post-incident forensics when you need proof of which process triggered the OOM event and which cgroup hit its limit.
2. Real-Time Cgroup Telemetry (memory.current, memory.events, systemd-cgtop)
- Underlying Idea: Cgroup v2 exposes real-time memory consumption and OOM event counters directly in
/sys/fs/cgroup/<name>/. - Commands:
watch -n 1 'cat /sys/fs/cgroup/sad-batch/memory.current /sys/fs/cgroup/sad-batch/memory.events'
systemd-cgtop -m
- When to Use: Useful during live load testing to watch
memory.currentclimb towardmemory.maxand verify thathigh,max,oom, andoom_killcounters inmemory.eventsbehave as expected.
3. Verifying Process Cgroup Membership via /proc
- Underlying Idea: Even when
memory.maxis configured properly, a service can escape limits if its launcher forks without migrating the PID intocgroup.procs. Checking/proc/<PID>/cgroupproves which cgroup a running process actually belongs to. - Commands:
pgrep -f sad-batch.sh
cat /proc/<PID>/cgroup
- When to Use: Whenever a process appears to exceed its cgroup limit despite
memory.maxbeing set correctly.
4. Native Systemd Resource Directives (Alternative Remediation)
- Underlying Idea: Instead of maintaining custom bash scripts (
setup-cgroup.shandrun-batch.sh) to manuallymkdirdirectories and echo PIDs intocgroup.procs,systemdcan manage cgroup v2 limits natively. - Example Configuration (
/etc/systemd/system/sad-batch.service):
[Service]
MemoryMax=128M
Or dynamically via CLI:
sudo systemctl set-property sad-batch.service MemoryMax=128M
Comparison: Custom Cgroup Scripts vs. Native Systemd Resource Control
| Criterion | Custom Cgroup v2 Script (/opt/sad/) |
Native Systemd (MemoryMax=128M) |
|---|---|---|
| Complexity | Higher; requires separate setup/wrapper scripts and manual cgroup.procs migration. |
Low; declarative single-line directive inside the .service unit. |
| Reliability | Prone to race conditions if unit ordering (Before=/After=) is misconfigured. |
High; systemd configures the cgroup before execve() of the service binary. |
| Maintainability | Non-standard; requires custom documentation so operators know not to look in systemd. |
Standard across modern Linux distributions; visible directly in systemctl status. |
| Suitability for Exercise | Required here because incident-notes.txt and check.sh explicitly test /sys/fs/cgroup/sad-batch/memory.max. |
Best practice for greenfield production services managed by systemd. |
9. Linux Command Deep Dive
Command: systemctl
Purpose
systemctl is the primary control utility for systemd, used to inspect, start, stop, enable, disable, and configure system services, targets, slices, and resource properties.
General Syntax
systemctl [OPTIONS...] COMMAND [UNIT...]
Options and Arguments
| Command / Option | Purpose |
|---|---|
systemctl status <unit> |
Show runtime state, main PID, cgroup path, memory usage, and recent logs. |
--no-pager |
Print full output directly to stdout without invoking less. |
systemctl restart <unit> |
Stop and immediately start a unit (re-runs ExecStart= for oneshot units). |
systemctl is-active --quiet <unit> |
Return exit code 0 if the unit is active, non-zero otherwise (ideal for health-check scripts). |
systemctl show <unit> -p MemoryMax |
Print the resolved value of a specific unit property. |
systemctl daemon-reload |
Reload unit files from disk after editing /etc/systemd/system/*.service. |
Practical Examples
# List all failed systemd units on the host
systemctl --failed
# Check resolved memory limits on a systemd-managed service
systemctl show sad-api.service -p MemoryCurrent -p MemoryMax
# View dependency tree for a service
systemctl list-dependencies sad-batch.service
Common Mistakes and Limitations
- Editing a
.servicefile on disk without runningsudo systemctl daemon-reloadcausessystemdto continue using the cached unit definition in memory (note: editing a bash script called byExecStart=, like/opt/sad/setup-cgroup.sh, does not requiredaemon-reload, only restarting the unit). - Confusing
enabled(configured to start at boot) withactive(currently running in memory).
Command: sed
Purpose
sed (Stream Editor) parses and transforms text streams or files line by line using regular expressions. It is widely used in sysadmin automation for non-interactive configuration updates.
General Syntax
sed [OPTIONS...] 'SCRIPT' [FILE...]
Options and Arguments
| Option / Flag | Purpose |
|---|---|
-i |
Edit files in-place instead of printing modified text to stdout. |
-i.bak |
Edit in-place while saving a backup copy of the original file with a .bak suffix. |
-E |
Enable Extended Regular Expressions (ERE), avoiding the need to backslash +, ?, ` |
-n |
Suppress automatic printing of pattern space (often paired with p to print matching lines). |
's/old/new/g' |
Substitute old with new; the g flag replaces all occurrences on a line rather than just the first. |
Practical Examples
# Replace text in-place while creating a timestamped backup
sudo sed -i.bak 's/memory.high/memory.max/' /opt/sad/setup-cgroup.sh
# Print lines 1 through 15 of a configuration file
sed -n '1,15p' /etc/systemd/system/sad-batch.service
# Strip comments and blank lines when reading a config file
sed -E '/^\s*#|^\s*$/d' /etc/ssh/sshd_config
Common Mistakes and Limitations
- GNU vs. BSD
sed: On Linux (GNUsed),-itakes an optional suffix directly attached (-i.bak). On macOS/BSD,-irequires an explicit argument (e.g.,sed -i '' 's/a/b/' file). - Running
sed -iwithout testing the regex first (without-i) can corrupt configuration files if the pattern matches unintended lines.
Command: curl
Purpose
curl transfers data to or from a server over HTTP, HTTPS, and other network protocols. It is the standard CLI tool for probing local and remote API health endpoints.
General Syntax
curl [OPTIONS...] <URL>
Options and Arguments
| Option | Purpose |
|---|---|
-s (--silent) |
Silent mode; hides the progress bar and error messages. |
-S (--show-error) |
When paired with -s (-sS), hides progress bars but still prints errors if the request fails. |
-m <seconds> |
Maximum time allowed for the entire operation (prevents scripts from hanging). |
-I (--head) |
Fetch HTTP response headers only (HEAD request). |
-i (--include) |
Include HTTP response headers alongside the response body. |
-f (--fail) |
Return exit code 22 on HTTP errors (4xx/5xx) instead of silently outputting the error HTML/JSON. |
Practical Examples
# Health check returning only the HTTP status code
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9090/
# Probe an endpoint with a strict 2-second timeout and fail on HTTP >= 400
curl -fsSm 2 http://127.0.0.1:9090/
Common Mistakes and Limitations
- By default,
curlexits with code0even if the server returnsHTTP 500 Internal Server Erroror404 Not Found. Always pass-f(--fail) or inspect%{http_code}when usingcurlin health checks.
10. Underlying Linux Concepts and Architecture
1. Control Groups: cgroup v1 vs. cgroup v2
Control Groups are a Linux kernel feature that organizes processes hierarchically and distributes system resources in a controlled manner:
- cgroup v1 (Legacy): Mounted separate directory trees per resource controller (e.g.,
/sys/fs/cgroup/memory/,/sys/fs/cgroup/cpu/). A process could belong to completely different hierarchies for memory and CPU, making coordinated resource management (such as buffered disk I/O accounting) difficult. Memory limits were set viamemory.limit_in_bytes. - cgroup v2 (Unified Hierarchy): Introduced a single, unified tree mounted at
/sys/fs/cgroup/. Every cgroup directory exposes control files for all enabled controllers in one place. Memory limits usememory.maxandmemory.high.
2. Cgroup v2 Memory Controller Hierarchy
Cgroup v2 provides four primary memory thresholds, ordered from strongest protection to hard ceiling:
memory.min(Hard Memory Protection): Guarantees a minimum amount of memory that the kernel will never reclaim under normal pressure.memory.low(Best-Effort Memory Protection): Soft guarantee; the kernel avoids reclaiming memory below this threshold unless non-protected cgroups have no reclaimable pages left.memory.high(Throttle Threshold): Soft ceiling; exceeding this triggers aggressive direct page reclaim and CPU throttling of allocating threads, without invoking the OOM killer.memory.max(Hard Ceiling): Absolute limit; exceeding this when pages cannot be reclaimed triggers the cgroup OOM killer.
3. Process Migration via cgroup.procs and exec
In /opt/sad/run-batch.sh, two shell mechanisms work together to ensure the Python process is constrained:
echo $$ > /sys/fs/cgroup/sad-batch/cgroup.procs
exec /opt/sad/sad-batch.sh
- Writing a PID to
cgroup.procsmoves that process into the target cgroup. Any child processes subsequently created viafork()automatically inherit their parent’s cgroup membership. - Using
execreplaces the current bash shell process image in-place without changing its PID ($$), ensuringsad-batch.sh(and subsequentlypython3) runs directly inside/sys/fs/cgroup/sad-batchwith zero extra wrapper processes left behind.
4. Global OOM Killer vs. Cgroup-Scoped OOM Killer
- Global OOM Killer: Triggered when the entire host runs out of physical RAM (and swap). The kernel evaluates every process on the machine using a heuristic (
oom_score, roughly proportional to percentage of total RAM used, adjusted by/proc/<PID>/oom_score_adj) and kills the highest-scoring victim. On a small VM, a critical API process can easily be chosen. - Cgroup-Scoped OOM Killer: Triggered when a cgroup hits
memory.max. The kernel restricts victim selection exclusively to processes within that cgroup (or kills all processes in the cgroup atomically ifmemory.oom.groupis set to1), completely insulating the rest of the system.
5. Systemd Dependency & Ordering Semantics
Notice how sad-batch.service specifies both Requires= and After=:
Requires=sad-cgroup-setup.service: Declares a hard activation dependency—ifsad-cgroup-setup.servicefails,sad-batch.servicewill not start. However,Requires=alone does not dictate order;systemdstarts required units in parallel by default.After=sad-cgroup-setup.service: Enforces sequential ordering, guaranteeingsad-cgroup-setup.servicefinishes (Type=oneshotcompletesExecStart=) beforesad-batch.servicelaunches.
11. Real-World Sysadmin and SRE Applications
Detection and Observability in Production
In production environments, OOM kills should never be discovered only when a user reports that an API is down. SRE teams monitor cgroup telemetry continuously using tools like Prometheus (node_exporter and cAdvisor):
memory.eventsCounters: Monitorhigh,max,oom, andoom_killinside/sys/fs/cgroup/<slice>/<service>/memory.events. A risinghighormaxcount provides early warning of memory pressure beforeoom_killevents occur.- Pressure Stall Information (PSI): Modern Linux kernels expose
/sys/fs/cgroup/<name>/memory.pressure(and/proc/pressure/memory), measuring the exact percentage of wall-clock time tasks spend stalled waiting for memory allocation or reclaim (someandfullmetrics).
Alerting and SLO Protection
- Protect Tier-0 APIs: Customer-facing APIs directly impact availability and latency Service Level Objectives (SLOs). Co-locating batch ETL jobs on the same host without strict
MemoryMax=boundaries violates fault isolation. - Actionable Alerts: Alert on
increase(container_oom_events_total[5m]) > 0(or systemd unitOOMKillsproperty changes) routed to the batch workload’s owning team, rather than paging the API on-call engineer for a batch job’s memory regression.
Preventive Engineering & Toil Reduction
- Eliminate Custom Cgroup Scripts: Maintaining bespoke bash scripts under
/opt/sad/to manipulate/sys/fs/cgroup/introduces operational drift and hidden dependencies. Standardizing onsystemdunit properties (MemoryMax=,MemoryHigh=,OOMPolicy=kill) managed via Infrastructure as Code (Ansible, Terraform, or Puppet) makes resource limits auditable and uniform. - Pre-Production Memory Profiling: Include memory regression checks in CI/CD pipelines for batch jobs so a code change that doubles heap consumption (e.g., loading an entire dataset into memory instead of streaming chunks) is caught before reaching production.
12. Further Learning and Experiments
Hands-on Experiments
You can perform the following safe experiments on a non-production Linux VM to observe cgroup v2 memory controls firsthand. (Warning: Do not run memory exhaustion tests on production hosts.)
Experiment 1: Comparing MemoryHigh Throttling vs. MemoryMax OOM Kill
- Objective: Observe how
systemd-runcreates transient cgroup v2 scopes and see the difference betweenMemoryHighandMemoryMax. - Commands:
# Test MemoryHigh (process slows down under reclaim pressure, may complete if host has RAM)
systemd-run --user --scope -p MemoryHigh=32M python3 -c 'a = bytearray(64*1024*1024); print("Survived MemoryHigh")'
# Test MemoryMax (kernel immediately kills the process with SIGKILL / exit code 137)
systemd-run --user --scope -p MemoryMax=32M python3 -c 'a = bytearray(64*1024*1024); print("Survived MemoryMax")'
echo $?
- What You Will Learn: The first command prints
Survived MemoryHigh(assuming free host RAM exists), whereas the second command is terminated by the cgroup OOM killer and prints exit status137(128 + 9forSIGKILL).
Experiment 2: Inspecting oom_score and oom_score_adj
- Objective: Understand how the kernel ranks processes when selecting an OOM victim.
- Commands:
# View the OOM score of your current shell and systemd-journald
cat /proc/$$/oom_score
cat /proc/$$/oom_score_adj
- What You Will Learn: Scores range from
0to1000(plusoom_score_adjfrom-1000to+1000). Critical system daemons often set negativeoom_score_adjvalues (-250or-1000) to protect themselves from global OOM events.
Topics to Explore Next
- Linux Pressure Stall Information (PSI): Measuring CPU, memory, and I/O contention via
/proc/pressure/*. - Systemd Slice Hierarchy: Organizing workloads into
system.slice,user.slice, and custom.sliceunits (e.g.,batch.slicevs.api.slice) to share aggregate resource budgets. - Container Runtime Cgroup Drivers: How Docker, containerd, and Kubernetes map pod
requests.memoryandlimits.memorydirectly onto cgroup v2memory.low/memory.minandmemory.max.
Authoritative Documentation and References
man 7 cgroups: Comprehensive overview of Linux cgroup v1 and v2 architecture.man 5 systemd.resource-control: Official reference forMemoryMax=,MemoryHigh=,MemoryLow=, andOOMPolicy=in systemd units.man 5 proc: Detailed documentation of/proc/<PID>/cgroup,/proc/<PID>/oom_score, and/proc/pressure/memory.- Linux Kernel Documentation:
Documentation/admin-guide/cgroup-v2.rstin the official Linux kernel source tree.
13. Key Takeaways
- Soft vs. Hard Limits in Cgroup v2:
memory.highonly throttles and reclaims memory;memory.maxis required to enforce a hard ceiling and isolate OOM kills to a specific cgroup. - Don’t Confuse Cgroup v1 and v2: Cgroup v1 used per-controller paths and
memory.limit_in_bytes; cgroup v2 uses a unified hierarchy at/sys/fs/cgroup/withmemory.max. - Verify Both State and Behaviour: Checking that
memory.maxcontains134217728confirms configuration state, while runningsad-batchand verifyingsad-apiremains responsive confirms runtime fault isolation. - Understand
Requires=vs.After=in Systemd:Requires=enforces dependency activation, whileAfter=enforces sequential execution order—both are needed when aoneshotsetup unit must finish before a worker unit starts. - Prefer Declarative Resource Controls: Whenever architectural constraints allow, use native
systemddirectives (MemoryMax=) instead of custom bash scripts manipulating/sys/fs/cgroup/.
14. Quick Troubleshooting Reference / Runbook
Use this checklist when investigating suspected cgroup memory limit failures or unexpected OOM kills on Linux:
- Confirm Cgroup Version Mounted on Host
stat -fc %T /sys/fs/cgroup/
# Output "cgroup2fs" = cgroup v2 unified hierarchy; "tmpfs" = cgroup v1
- Check Kernel Logs for Recent OOM Events
sudo dmesg -T | grep -i -E "oom|killed process"
- Inspect Service Status and Systemd Cgroup Path
systemctl status <service-name> --no-pager
- Verify Process Membership in Target Cgroup
cat /proc/<PID>/cgroup
- Inspect Live Cgroup v2 Memory Limits and Usage
cat /sys/fs/cgroup/<cgroup-path>/memory.max
cat /sys/fs/cgroup/<cgroup-path>/memory.high
cat /sys/fs/cgroup/<cgroup-path>/memory.current
cat /sys/fs/cgroup/<cgroup-path>/memory.events
- Check Systemd Unit Resource Properties (If Managed via Systemd)
systemctl show <service-name> -p MemoryMax -p MemoryHigh -p OOMPolicy
- Apply Hard Limit Fix & Reload/Restart Unit
- For custom cgroup v2 scripts: write byte limit to
memory.maxand restart the setup unit. - For native systemd units: set
MemoryMax=<size>in the[Service]block, runsudo systemctl daemon-reload, and restart the service.
- Validate Isolation Under Load
- Trigger the memory-intensive workload, verify
memory.events(oom_kill) isolates the failure to the target cgroup, and confirm critical co-located services remain healthy.