1. Introduction
In modern cloud infrastructure and Site Reliability Engineering (SRE), metric integrity and process table health are non-negotiable. System dashboards rely on continuous, accurate telemetry from host operating systems to trigger automated scaling, alert on anomalies, and maintain visibility into application health. When published metrics diverge from true kernel metrics, or when processes fail to clean up after themselves, system stability and observability degrade rapidly.
This technical article explores a real-world Linux troubleshooting scenario based on the “Philadelphia” host scenario. We investigate two co-occurring system failures:
- A metrics exporter regression: An internal metrics daemon reports hardcoded, stale host load and uptime numbers to a status board while interactive monitoring tools (
htop,btop) show live, changing metrics. - A daemon process leakage: A persistent background lab service creates child processes that terminate and remain stuck as zombie (defunct) processes indefinitely.
Through this post-mortem and technical reference guide, you will learn how the Linux virtual filesystem (/proc) provides live kernel state, how POSIX process creation and termination function at the system call level, how to diagnose process leaks using standard Linux diagnostic tools, and how to resolve both issues permanently.
2. Problem Description and Environment
Scenario Background
On the host Philadelphia, an internal exporter script /opt/sad/sysinfo-reporter.sh runs periodically via a systemd background service (sad-sysinfo.service) to publish metrics to /var/lib/sad/sysinfo.txt. A second service, sad-zombie-lab.service, executes a binary (/opt/sad/zombie-lab) compiled from /opt/sad/zombie.c.
Expected vs. Actual Behaviour
| Component | Expected Behaviour | Actual Behaviour |
|---|---|---|
| Metrics Exporter | /var/lib/sad/sysinfo.txt publishes live 1-minute load average and uptime matching /proc/loadavg and /proc/uptime. |
Published values remain permanently stuck at load_1m=0.00 and uptime_seconds=86400 despite fluctuating host activity. |
| Lab Daemon | sad-zombie-lab.service remains active without accumulating defunct child processes in the kernel process table. |
A child process spawned by /opt/sad/zombie-lab exits immediately and remains in state Z (<defunct>) indefinitely. |
Environment Details
- Host Name:
i-01f4e41baaa55a740(Philadelphia) - Operating System: Linux (Debian/Ubuntu-based distribution)
- Installed Utilities:
procps(ps,uptime),htop,btop,strace,gcc,gdb,systemd - Target Files & Paths:
/var/lib/sad/sysinfo.txt(Published output metric file)/opt/sad/sysinfo-reporter.sh(Metric generator script)/opt/sad/zombie.c&/opt/sad/zombie-lab(Source code and binary for daemon)/home/admin/agent/check.sh(Automated verification script)
3. Initial Investigation and Hypotheses
To approach this problem systematically, we must avoid changing system states until we establish clear evidence. We begin by formulating testable hypotheses for both observed symptoms.
Hypothesis Matrix
| Symptom | Potential Root Cause | Supporting Evidence | Contradicting Evidence | Diagnostic Step |
|---|---|---|---|---|
| Stale Dashboard Load & Uptime | Exporter script is reading from a static file path rather than the kernel /proc filesystem. |
htop shows changing CPU/load, but output text file never changes. |
Service running status shows active execution. | Inspect script source /opt/sad/sysinfo-reporter.sh with cat or strace. |
| Stale Dashboard Load & Uptime | System timer or loop service stopped updating the file. | File timestamp might be old. | Service status for sad-sysinfo is active and running a loop. |
Check systemctl status sad-sysinfo and output file timestamp. |
| Zombie Child Process | Parent process forks a child, but fails to call wait() / waitpid() after child exit. |
systemctl status shows Tasks: 2 under single main PID. |
Child process memory is released. | Inspect C source code /opt/sad/zombie.c. |
| Zombie Child Process | Parent process terminated unexpectedly, leaving an orphan. | systemctl status shows main parent PID is still running. |
Process is orphan (reparented to PID 1). | Run ps -ef or pgrep -P <PID> to verify parent-child relationship. |
4. Troubleshooting Process
Step 1: Reading Incident Notes and Verification Logic
Objective
Understand the reported problem, identify key paths, and examine the rules used by the automated verification harness to ensure complete compliance.
Commands
cat /home/admin/incident-notes.txt
cat /home/admin/agent/check.sh
Command Explanation
cat: Concatenates and prints file contents to standard output. Used here to read project documentation and verification shell scripts.
Output and Analysis
admin@i-01f4e41baaa55a740:~$ cat incident-notes.txt
Incident: 2026-07-01 — dashboard metrics disagree with htop
The Philadelphia host exports load and uptime to /var/lib/sad/sysinfo.txt for a
small status board. After a "performance" refactor, the board shows load stuck
at 0.00 and an uptime from last week, while htop and btop on the box look normal.
A separate lab service (sad-zombie-lab) was left running from a fork()/exit()
demo. htop shows a defunct child under /opt/sad/zombie-lab. That is not normal
for production — the parent should reap exited children.
...
admin@i-01f4e41baaa55a740:~$ cat agent/check.sh
#!/usr/bin/env bash
...
if ! head -n 1 /var/lib/sad/sysinfo.txt | grep -q 'SadServers - Philadelphia OK'; then
fail
fi
reported_load=$(grep '^load_1m=' /var/lib/sad/sysinfo.txt 2>/dev/null | cut -d= -f2 | tr -d '[:space:]')
actual_load=$(awk '{print $1}' /proc/loadavg 2>/dev/null | tr -d '[:space:]')
...
if ! awk -v r="$reported_load" -v a="$actual_load" 'BEGIN { exit !(r + 0 <= a + 0.20 && r + 0 >= a - 0.20) }'; then
fail
fi
reported_uptime=$(grep '^uptime_seconds=' /var/lib/sad/sysinfo.txt 2>/dev/null | cut -d= -f2 | tr -d '[:space:]')
actual_uptime=$(awk '{print int($1)}' /proc/uptime 2>/dev/null)
...
uptime_diff=$((actual_uptime - reported_uptime))
if [ "$uptime_diff" -gt 120 ]; then
fail
fi
systemctl is-active --quiet sad-zombie-lab.service || fail
parent=$(pgrep -f '^/opt/sad/zombie-lab$' 2>/dev/null | head -n 1)
...
while read -r cpid; do
[ -z "$cpid" ] && continue
child_stat=$(ps -o stat= -p "$cpid" 2>/dev/null | tr -d '[:space:]')
case "$child_stat" in
Z*) fail ;;
esac
done < <(pgrep -P "$parent" 2>/dev/null || true)
echo -n "OK"
Analysis
incident-notes.txtexplicitly mentions a past “performance refactor” that affected metrics generation, pointing to/opt/sad/sysinfo-reporter.sh.check.shenforces the following tolerances:
- First line of
/var/lib/sad/sysinfo.txtmust matchSadServers - Philadelphia OK. load_1min/var/lib/sad/sysinfo.txtmust match field 1 of/proc/loadavgwithin $\pm 0.20$.uptime_secondsin/var/lib/sad/sysinfo.txtmust match/proc/uptimewithin 120 seconds.sad-zombie-lab.servicemust be active, and no child processes under/opt/sad/zombie-labcan have process stateZ*(Zombie).
Decision
Proceed to inspect current published metric values and check the systemd service statuses.
Step 2: Inspecting Current Exporter Output and Service Statuses
Objective
Confirm the baseline failure state in /var/lib/sad/sysinfo.txt and verify service runtime parameters.
Commands
head -n 15 /var/lib/sad/sysinfo.txt
systemctl status sad-zombie-lab --no-pager
systemctl status sad-sysinfo --no-pager
Command Explanation
head -n 15: Displays the first 15 lines of a target file.systemctl status <unit>: Displays the operational status, main PID, control group process tree, and recent logs for a systemd unit.--no-pager: Forces output directly to stdout without piping throughlessor a terminal pager.
Output and Analysis
admin@i-01f4e41baaa55a740:~$ head -n 15 /var/lib/sad/sysinfo.txt
SadServers - Philadelphia OK
load_1m=0.00
load_5m=0.00
load_15m=0.00
uptime_seconds=86400
admin@i-01f4e41baaa55a740:~$ systemctl status sad-zombie-lab --no-pager
● sad-zombie-lab.service - SadServers Philadelphia zombie process lab
Loaded: loaded (/etc/systemd/system/sad-zombie-lab.service; enabled; preset: enabled)
Active: active (running) since Sun 2026-11-08 15:35:47 UTC; 1min 53s ago
Main PID: 781 (zombie-lab)
Tasks: 2 (limit: 501)
Memory: 208K (peak: 1.8M)
CPU: 13ms
CGroup: /system.slice/sad-zombie-lab.service
└─781 /opt/sad/zombie-lab
admin@i-01f4e41baaa55a740:~$ systemctl status sad-sysinfo --no-pager
● sad-sysinfo.service - SadServers Philadelphia sysinfo exporter
Loaded: loaded (/etc/systemd/system/sad-sysinfo.service; enabled; preset: enabled)
Active: active (running) since Sun 2026-11-08 15:35:47 UTC; 2min 46s ago
Main PID: 779 (bash)
Tasks: 2 (limit: 501)
Memory: 1016K (peak: 1.9M)
CPU: 190ms
CGroup: /system.slice/sad-sysinfo.service
├─ 779 bash /opt/sad/sysinfo-loop.sh
└─1089 sleep 10
Analysis
/var/lib/sad/sysinfo.txtdisplays static metrics (load_1m=0.00,uptime_seconds=86400, which is exactly $24 \times 3600$ seconds).sad-zombie-lab.servicereportsTasks: 2attached to PID 781, despite showing only one executable in the CGroup tree, indicating an untracked defunct child task in the process list.sad-sysinfo.serviceis actively executing/opt/sad/sysinfo-loop.sh.
Decision
Inspect /opt/sad/sysinfo-reporter.sh and /opt/sad/zombie.c to identify code-level errors.
Step 3: Source Code Inspection (sysinfo-reporter.sh and zombie.c)
Objective
Analyze the scripts and source code generating metrics and spawning child processes.
Commands
cat /opt/sad/sysinfo-reporter.sh
cat /opt/sad/zombie.c
Output and Analysis
admin@i-01f4e41baaa55a740:~$ cat /opt/sad/sysinfo-reporter.sh
#!/usr/bin/env bash
# Publish host load/uptime for the internal dashboard.
set -euo pipefail
OUT=/var/lib/sad/sysinfo.txt
mkdir -p /var/lib/sad
# "Fast path" — reads cached proc snapshots (refactor regression)
UPTIME=$(awk '{print $1}' /var/lib/sad/proc/uptime)
LOAD1=$(awk '{print $1}' /var/lib/sad/proc/loadavg)
LOAD5=$(awk '{print $2}' /var/lib/sad/proc/loadavg)
LOAD15=$(awk '{print $3}' /var/lib/sad/proc/loadavg)
{
echo 'SadServers - Philadelphia OK'
printf 'load_1m=%s\n' "$LOAD1"
printf 'load_5m=%s\n' "$LOAD5"
printf 'load_15m=%s\n' "$LOAD15"
printf 'uptime_seconds=%s\n' "${UPTIME%.*}"
} > "$OUT"
admin@i-01f4e41baaa55a740:~$ cat /opt/sad/zombie.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(void) {
int pid = fork();
if (pid == 0) {
exit(0);
}
while (1) {
sleep(60);
}
return 0;
}
Analysis
Issue 1: Path Regression in sysinfo-reporter.sh
The script reads metrics from /var/lib/sad/proc/uptime and /var/lib/sad/proc/loadavg. These are standard, static regular files created on disk during the “performance refactor”. Because they are static disk files rather than virtual procfs streams (/proc/uptime and /proc/loadavg), their values never update.
Issue 2: Unreaped Child Process in zombie.c
In zombie.c, fork() creates a child process:
- For the child (
pid == 0), it callsexit(0)immediately. - For the parent (
pid > 0), it skipsif (pid == 0)and enters an infinite loopwhile (1) { sleep(60); }without invokingwait()orwaitpid(). - The Linux kernel frees the child’s execution context (memory, file descriptors), but preserves its process control block (PCB) and PID in the kernel process table until the parent reads its exit status code via
wait(). Because the parent never callswait(), the child remains a zombie (TASK_DEAD/Zstate) forever.
5. Root Cause Analysis
Metric Exporter Root Cause
The metric exporter failed due to a path regression introduced during a software refactor. Regular files placed at /var/lib/sad/proc/ were being queried instead of kernel interfaces residing at /proc/.
In Linux, reading /proc/loadavg triggers a kernel function that queries live CPU runqueue statistics across scheduler cores. Reading /var/lib/sad/proc/loadavg simply reads static text bytes from disk blocks, returning fixed values.
Zombie Process Root Cause
The zombie child process in sad-zombie-lab stems from a violation of POSIX process management rules.
Parent Process (PID 781) Child Process (PID 782)
| |
+----- fork() --------------------->|
| | (Executes exit(0))
| v
| State: TASK_DEAD
| PCB retained in kernel task table
| Process status: Z (Zombie)
|
while(1) { sleep(60); }
(No wait() or SIGCHLD handler)
|
v
Child remains Z (Zombie) permanently!
- Child Exit: A child process invokes
exit(). The kernel transitions its state toTASK_DEAD(Z) and releases heap, stack, and file descriptors. - Status Retention: The kernel keeps the process entry (PID, exit code, CPU usage stats) in memory so the parent process can query how the child exited.
- Missing Reaping: The parent process loops indefinitely without calling
wait(),waitpid(), or registering a handler forSIGCHLD. - Leak: The zombie process accumulates in the kernel task list, consuming a task entry slot until the parent process terminates or is killed.
6. Solution and Remediation
Approach Applied
1. Fix the Metrics Exporter Script
Use sed to update /opt/sad/sysinfo-reporter.sh so awk queries /proc/ instead of /var/lib/sad/proc/.
sudo sed -i 's|/var/lib/sad/proc/|/proc/|g' /opt/sad/sysinfo-reporter.sh
Script Differences (/opt/sad/sysinfo-reporter.sh)
--- /opt/sad/sysinfo-reporter.sh.orig
+++ /opt/sad/sysinfo-reporter.sh
@@ -6,8 +6,8 @@
OUT=/var/lib/sad/sysinfo.txt
mkdir -p /var/lib/sad
-# "Fast path" — reads cached proc snapshots (refactor regression)
-UPTIME=$(awk '{print $1}' /var/lib/sad/proc/uptime)
-LOAD1=$(awk '{print $1}' /var/lib/sad/proc/loadavg)
-LOAD5=$(awk '{print $2}' /var/lib/sad/proc/loadavg)
-LOAD15=$(awk '{print $3}' /var/lib/sad/proc/loadavg)
+# Live kernel procfs interface
+UPTIME=$(awk '{print $1}' /proc/uptime)
+LOAD1=$(awk '{print $1}' /proc/loadavg)
+LOAD5=$(awk '{print $2}' /proc/loadavg)
+LOAD15=$(awk '{print $3}' /proc/loadavg)
Restart the exporter service to populate /var/lib/sad/sysinfo.txt with live metrics:
sudo systemctl restart sad-sysinfo
2. Fix the Zombie Process C Source & Recompile
Modify /opt/sad/zombie.c to include <sys/wait.h> and invoke wait(NULL); immediately after the fork() branch in the parent context.
sudo bash -c 'cat << "EOF" > /opt/sad/zombie.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
int main(void) {
int pid = fork();
if (pid == 0) {
exit(0);
}
wait(NULL);
while (1) {
sleep(60);
}
return 0;
}
EOF'
C Source Code Differences (/opt/sad/zombie.c)
--- /opt/sad/zombie.c.orig
+++ /opt/sad/zombie.c
@@ -1,6 +1,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
+#include <sys/wait.h>
int main(void) {
int pid = fork();
@@ -9,6 +10,8 @@
exit(0);
}
+ wait(NULL);
+
while (1) {
sleep(60);
}
Recompile the binary and restart the service:
sudo gcc -Wall -o /opt/sad/zombie-lab /opt/sad/zombie.c
sudo systemctl restart sad-zombie-lab
7. Verification and Validation
Step 1: Run System Verification Script
Execute the scenario check harness /home/admin/agent/check.sh:
admin@i-01f4e41baaa55a740:~$ /home/admin/agent/check.sh
OK
Step 2: Validate Metric Alignment
Compare the generated /var/lib/sad/sysinfo.txt file against live /proc/loadavg and /proc/uptime:
cat /var/lib/sad/sysinfo.txt
cat /proc/loadavg
cat /proc/uptime
Output
SadServers - Philadelphia OK
load_1m=0.01
load_5m=0.03
load_15m=0.01
uptime_seconds=345
The published values now update continuously and accurately reflect live kernel telemetry.
Step 3: Inspect Service Task List
Verify that sad-zombie-lab.service reports Tasks: 1 and has no defunct child processes:
systemctl status sad-zombie-lab --no-pager
pgrep -P $(pgrep -f '^/opt/sad/zombie-lab$')
Output
● sad-zombie-lab.service - SadServers Philadelphia zombie process lab
Loaded: loaded (/etc/systemd/system/sad-zombie-lab.service; enabled; preset: enabled)
Active: active (running) since Sun 2026-11-08 15:40:12 UTC; 45s ago
Main PID: 1240 (zombie-lab)
Tasks: 1 (limit: 501)
...
Tasks: 1 confirms that the child process was successfully reaped by wait(), leaving zero zombies in the process control block table.
8. Alternative Troubleshooting Approaches
Approach A: Tracing I/O and System Calls with strace
Instead of manually reading shell scripts, strace allows you to observe system calls made by daemons in real time.
sudo strace -e trace=openat,read /opt/sad/sysinfo-reporter.sh
- What it reveals: Shows exact file paths passed to
openat(). If the output showsopenat(AT_FDCWD, "/var/lib/sad/proc/loadavg", ...)instead of/proc/loadavg, the path regression is immediately obvious without reading shell code. - Advantages: Language-agnostic; works on compiled binaries where source code is unavailable.
- Limitations: High performance overhead; produces verbose output on busy processes.
Approach B: Asynchronous Zombie Reaping via Signal Handling (SIGCHLD)
In C programming, calling a blocking wait(NULL); prevents the parent from continuing until the child exits. For non-blocking asynchronous servers, alternative strategies exist:
Option B1: Explicit Signal Handling
Catch SIGCHLD and reap asynchronously using waitpid():
#include <signal.h>
#include <sys/wait.h>
void handle_sigchld(int sig) {
while (waitpid(-1, NULL, WNOHANG) > 0);
}
int main(void) {
signal(SIGCHLD, handle_sigchld);
if (fork() == 0) { exit(0); }
while (1) { sleep(60); }
}
Option B2: Ignore SIGCHLD (SIG_IGN)
Instruct the kernel to automatically reap exited children without requiring a wait() call:
#include <signal.h>
int main(void) {
signal(SIGCHLD, SIG_IGN);
if (fork() == 0) { exit(0); }
while (1) { sleep(60); }
}
- When to use:
SIG_IGNis ideal for long-running daemons that fork worker processes but do not care about their exit codes.
Comparison of Diagnostic Approaches
| Tool / Interface | Purpose | Primary Use Case | Output Complexity | Overhead |
|---|---|---|---|---|
ps aux / ps -ef |
Process table snapshot | Identify defunct processes (Z state flag) |
Low | Very Low |
pgrep -P <PID> |
Parent-child mapping | Identify process hierarchy and child PIDs | Low | Very Low |
/proc/<PID>/status |
Direct kernel process state | Inspect state (State: Z (zombie)), memory, and parent PID |
Medium | Minimal |
strace -f -e trace=process |
System call tracing | Follow fork(), clone(), exit(), and waitpid() calls across threads |
High | High |
systemctl status |
Unit health & CGroup overview | Check Tasks: count for unexpected process accumulation |
Low | Very Low |
9. Linux Command Deep Dive
1. sed (Stream Editor)
- Purpose: Performs text parsing and transformations on file streams or input files.
General Syntax
sed [options] 'script' [input-file]
Common Options & Arguments
| Option / Syntax | Description |
|---|---|
-i[SUFFIX], --in-place |
Edit files in place instead of printing output to stdout. |
| `s | regexp |
-n, --quiet |
Suppress automatic printing of pattern space. |
-e script |
Execute multiple script commands in sequence. |
Useful Examples
# Replace /var/log with /tmp/log in a config file
sed -i 's|/var/log|/tmp/log|g' /etc/app.conf
# Print lines 10 through 20 of a log file
sed -n '10,20p' /var/log/syslog
2. gcc (GNU Compiler Collection)
- Purpose: Compiles C, C++, and assembly source code into executable binary files.
General Syntax
gcc [options] [file...]
Common Options & Arguments
| Option / Syntax | Description |
|---|---|
-Wall |
Enables all common compiler warnings (e.g., unused variables, missing header includes). |
-Wextra |
Enables extra diagnostic warnings for strict code quality. |
-o <filename> |
Write output binary to <filename> instead of a.out. |
-g |
Generates debugging symbol information for gdb. |
-O2 |
Enables standard performance optimizations. |
Useful Examples
# Compile C file with strict warnings and output binary name
gcc -Wall -Wextra -o mydaemon daemon.c
# Compile with debugging symbols for gdb inspection
gcc -g -o mydaemon daemon.c
3. pgrep (Process Grep)
- Purpose: Searches currently running processes and lists PIDs matching specified criteria.
General Syntax
pgrep [options] pattern
Common Options & Arguments
| Option / Syntax | Description |
|---|---|
-P <PPID> |
Match only processes whose Parent Process ID is <PPID>. |
-f |
Match pattern against full command line instead of executable name. |
-u <user> |
Match only processes owned by specified username or UID. |
-l |
List process name alongside PID. |
-a |
List full command line alongside PID. |
Useful Examples
# Find all child PIDs belonging to parent PID 1240
pgrep -P 1240
# Find PID matching full binary path
pgrep -f '^/opt/sad/zombie-lab$'
4. ps (Process Status)
- Purpose: Provides a snapshot of currently running processes.
General Syntax
ps [options]
Options Reference
| Syntax Format | Flag | Description |
|---|---|---|
| UNIX style | ps -ef |
Standard full-format listing (UID, PID, PPID, C, STIME, TTY, TIME, CMD). |
| BSD style | ps aux |
Display all processes including user, CPU/MEM %, process state (STAT). |
| Custom Format | ps -o stat= -p <PID> |
Print specific output fields (e.g., stat) without printing the header line. |
| Sorting | ps aux --sort=-%cpu |
Sort processes by descending CPU usage. |
| Sorting | ps aux --sort=-%mem |
Sort processes by descending Memory usage. |
10. Underlying Linux Concepts and Architecture
1. Procfs (/proc Virtual Filesystem)
The /proc directory is not a real disk filesystem; it is a pseudo-filesystem (procfs) mounted dynamically by the Linux kernel.
+-------------------------------------------------------------+
| User Space |
| cat /proc/loadavg awk '{print $1}' /proc/uptime |
+-------------------------------------------------------------+
| Read Syscall
v
+-------------------------------------------------------------+
| Linux Kernel |
| procfs VFS Handler -> Query Scheduler & Timer Subsystems |
+-------------------------------------------------------------+
| Live Metrics Stream
v
Dynamic CPU Load & Uptime
- Dynamic Data Generation: When a process calls
read()on a file in/proc, the kernel’s virtual filesystem (VFS) intercepts the request and runs internal kernel code to generate text output on the fly. - Disk vs. Procfs: Real files stored on EXT4 or XFS filesystems read fixed blocks from persistent storage blocks. Procfs queries live RAM kernel data structures (such as
struct task_structand scheduler queue metrics).
2. POSIX Process Lifecycle & Zombie Reaping
Every process in Linux is assigned a Process Control Block (task_struct).
+-------------+ fork() +-------------+ exit() +-------------+
| Created | -----------------> | Running | ----------------> | TASK_DEAD |
| (Parent) | | (Child) | | (Zombie) |
+-------------+ +-------------+ +-------------+
|
| wait() called
v
+-------------+
| Removed |
| from Table |
+-------------+
fork(): Copies the parent process to create a child process with a unique PID.exit(): The child process terminates. Memory allocations, file descriptor tables, and thread structures are freed. The status is updated toTASK_DEAD(Z).- The Zombie State: The kernel maintains the PID entry in the task list so the parent process can collect its exit status via
wait()orwaitpid(). - Reaping: When the parent calls
wait(), the kernel reads the exit code and frees the remaining process control block entry. - Orphan Adoption: If a parent process terminates before reaping its children,
systemd(PID 1) or an init sub-reaper adopts the child and automatically callswait()when the child exits.
11. Real-World Sysadmin and SRE Applications
1. Metric Integrity Monitoring & Alerts
- Stale Metric Detection: In production, monitoring agents (e.g., Prometheus Node Exporter, Datadog) should export a counter metric like
node_exporter_scrape_time_seconds. Configure alerting rules to trigger when a host’s load average remains bit-for-bit identical across multiple scrape intervals, indicating a stuck collector daemon or snapshot path regression.
2. Process Leak Prevention in Production
- Systemd Task Limits (
TasksMax): SetTasksMax=in service unit files to cap the number of tasks (threads + processes) a unit can create. This prevents zombie process leaks from exhausting kernel process IDs (/proc/sys/kernel/pid_max).
[Service]
TasksMax=64
- Container Init Process (
tini): In Docker or Kubernetes containers, running application processes as PID 1 often causes zombie build-up because standard application processes do not include init-style process reaping logic. Using lightweight init systems liketiniordocker run --initensures orphaned zombie processes are automatically reaped.
12. Further Learning and Experiments
Hands-on Lab Experiments
Experiment 1: Inspecting Procfs Filesystem Behavior
Execute the following to observe how procfs generates live dynamic output:
# Continuously monitor kernel procfs virtual file vs text file
watch -n 1 "cat /proc/loadavg"
- Observation: Notice how
/proc/loadavgdynamically changes without requiring manual updates or disk writes.
Experiment 2: Reproducing Zombie Accumulation
- Create a test script named
test_zombie.c:
#include <unistd.h>
#include <stdlib.h>
int main() {
if (fork() == 0) exit(0);
sleep(30);
return 0;
}
- Compile and run in background:
gcc -o test_zombie test_zombie.c
./test_zombie &
- Check the process table:
ps aux | grep -E 'Z|defunct'
- Learning Objective: Observe how the child process transitions to
Zstate (<defunct>) during the parent’s 30-second sleep window, and automatically disappears once the parent exits and gets reaped by PID 1.
Documentation References
man 2 wait/man 2 waitpid: System calls for waiting for process state changes.man 5 proc: Manual page for the/procvirtual filesystem interface.man 2 fork: Process creation system call mechanics.
13. Key Takeaways
- Procfs is Virtual: Linux metrics in
/procare dynamic kernel streams. Never configure exporters to read static snapshot files on disk (/var/lib/...) unless explicitly implementing historical caching. - Always Reap Child Processes: Any C/Linux application invoking
fork()must handle child termination viawait(),waitpid(), or by explicitly settingSIGCHLDtoSIG_IGN. - Validate Task Counts: Use
systemctl statusto inspectTasks:metrics or checkps auxfor process stateZto quickly identify zombie leaks. - Use Standard System Calls: When troubleshooting path issues, tools like
straceprovide transparent visibility intoopenat()file descriptor calls.
14. Quick Troubleshooting Reference / Runbook
Metric & Process Failure Runbook
|
+-------------------------------+-------------------------------+
| |
v v
[ Issue 1: Metrics Exporter Stale ] [ Issue 2: Zombie Process Accumulation ]
| |
1. Compare output file vs kernel procfs: 1. Check for process state Z:
diff <(head -n5 /var/lib/sad/sysinfo.txt) ps -eo pid,ppid,stat,cmd | grep ' Z'
<(cat /proc/loadavg)
2. Identify parent PID:
2. Trace file read operations: pgrep -P <PARENT_PID>
strace -e openat /path/to/exporter.sh
3. Check systemd unit task count:
3. Update script path to /proc/<metric> systemctl status <unit-name>
4. Restart metric service: 4. Update code: Add wait() or SIGCHLD handler
systemctl restart <exporter-service>
5. Recompile & restart unit:
gcc -Wall -o <binary> <source.c>
systemctl restart <unit-name>