Domain 10Linux & SREINTERMEDIATE

(Philadelphia) Debugging Metrics Exporter Regressions and POSIX Zombie Process Leakage

A deep-dive technical post-mortem and guide on fixing a metrics exporter reading from static disk snapshots instead of kernel procfs, and resolving a zombie process leak by implementing POSIX process reaping with wait().

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-08
#linux#systemd#procfs#zombie-process#c-programming#troubleshooting#sadservers

1. Introduction

In modern cloud infrastructure and Site Reliability Engineering (SRE), metric integrity and process table health are non-negotiable. System dashboards rely on continuous, accurate telemetry from host operating systems to trigger automated scaling, alert on anomalies, and maintain visibility into application health. When published metrics diverge from true kernel metrics, or when processes fail to clean up after themselves, system stability and observability degrade rapidly.

This technical article explores a real-world Linux troubleshooting scenario based on the “Philadelphia” host scenario. We investigate two co-occurring system failures:

  1. A metrics exporter regression: An internal metrics daemon reports hardcoded, stale host load and uptime numbers to a status board while interactive monitoring tools (htop, btop) show live, changing metrics.
  2. A daemon process leakage: A persistent background lab service creates child processes that terminate and remain stuck as zombie (defunct) processes indefinitely.

Through this post-mortem and technical reference guide, you will learn how the Linux virtual filesystem (/proc) provides live kernel state, how POSIX process creation and termination function at the system call level, how to diagnose process leaks using standard Linux diagnostic tools, and how to resolve both issues permanently.


2. Problem Description and Environment

Scenario Background

On the host Philadelphia, an internal exporter script /opt/sad/sysinfo-reporter.sh runs periodically via a systemd background service (sad-sysinfo.service) to publish metrics to /var/lib/sad/sysinfo.txt. A second service, sad-zombie-lab.service, executes a binary (/opt/sad/zombie-lab) compiled from /opt/sad/zombie.c.

Expected vs. Actual Behaviour

Component Expected Behaviour Actual Behaviour
Metrics Exporter /var/lib/sad/sysinfo.txt publishes live 1-minute load average and uptime matching /proc/loadavg and /proc/uptime. Published values remain permanently stuck at load_1m=0.00 and uptime_seconds=86400 despite fluctuating host activity.
Lab Daemon sad-zombie-lab.service remains active without accumulating defunct child processes in the kernel process table. A child process spawned by /opt/sad/zombie-lab exits immediately and remains in state Z (<defunct>) indefinitely.

Environment Details

  • Host Name: i-01f4e41baaa55a740 (Philadelphia)
  • Operating System: Linux (Debian/Ubuntu-based distribution)
  • Installed Utilities: procps (ps, uptime), htop, btop, strace, gcc, gdb, systemd
  • Target Files & Paths:
  • /var/lib/sad/sysinfo.txt (Published output metric file)
  • /opt/sad/sysinfo-reporter.sh (Metric generator script)
  • /opt/sad/zombie.c & /opt/sad/zombie-lab (Source code and binary for daemon)
  • /home/admin/agent/check.sh (Automated verification script)

3. Initial Investigation and Hypotheses

To approach this problem systematically, we must avoid changing system states until we establish clear evidence. We begin by formulating testable hypotheses for both observed symptoms.

Hypothesis Matrix

Symptom Potential Root Cause Supporting Evidence Contradicting Evidence Diagnostic Step
Stale Dashboard Load & Uptime Exporter script is reading from a static file path rather than the kernel /proc filesystem. htop shows changing CPU/load, but output text file never changes. Service running status shows active execution. Inspect script source /opt/sad/sysinfo-reporter.sh with cat or strace.
Stale Dashboard Load & Uptime System timer or loop service stopped updating the file. File timestamp might be old. Service status for sad-sysinfo is active and running a loop. Check systemctl status sad-sysinfo and output file timestamp.
Zombie Child Process Parent process forks a child, but fails to call wait() / waitpid() after child exit. systemctl status shows Tasks: 2 under single main PID. Child process memory is released. Inspect C source code /opt/sad/zombie.c.
Zombie Child Process Parent process terminated unexpectedly, leaving an orphan. systemctl status shows main parent PID is still running. Process is orphan (reparented to PID 1). Run ps -ef or pgrep -P <PID> to verify parent-child relationship.

4. Troubleshooting Process

Step 1: Reading Incident Notes and Verification Logic

Objective

Understand the reported problem, identify key paths, and examine the rules used by the automated verification harness to ensure complete compliance.

Commands

cat /home/admin/incident-notes.txt
cat /home/admin/agent/check.sh

Command Explanation

  • cat: Concatenates and prints file contents to standard output. Used here to read project documentation and verification shell scripts.

Output and Analysis

admin@i-01f4e41baaa55a740:~$ cat incident-notes.txt 
Incident: 2026-07-01 — dashboard metrics disagree with htop

The Philadelphia host exports load and uptime to /var/lib/sad/sysinfo.txt for a
small status board. After a "performance" refactor, the board shows load stuck
at 0.00 and an uptime from last week, while htop and btop on the box look normal.

A separate lab service (sad-zombie-lab) was left running from a fork()/exit()
demo. htop shows a defunct child under /opt/sad/zombie-lab. That is not normal
for production — the parent should reap exited children.
...
admin@i-01f4e41baaa55a740:~$ cat agent/check.sh 
#!/usr/bin/env bash
...
if ! head -n 1 /var/lib/sad/sysinfo.txt | grep -q 'SadServers - Philadelphia OK'; then
  fail
fi

reported_load=$(grep '^load_1m=' /var/lib/sad/sysinfo.txt 2>/dev/null | cut -d= -f2 | tr -d '[:space:]')
actual_load=$(awk '{print $1}' /proc/loadavg 2>/dev/null | tr -d '[:space:]')
...
if ! awk -v r="$reported_load" -v a="$actual_load" 'BEGIN { exit !(r + 0 <= a + 0.20 && r + 0 >= a - 0.20) }'; then
  fail
fi

reported_uptime=$(grep '^uptime_seconds=' /var/lib/sad/sysinfo.txt 2>/dev/null | cut -d= -f2 | tr -d '[:space:]')
actual_uptime=$(awk '{print int($1)}' /proc/uptime 2>/dev/null)
...
uptime_diff=$((actual_uptime - reported_uptime))
if [ "$uptime_diff" -gt 120 ]; then
  fail
fi

systemctl is-active --quiet sad-zombie-lab.service || fail

parent=$(pgrep -f '^/opt/sad/zombie-lab$' 2>/dev/null | head -n 1)
...
while read -r cpid; do
  [ -z "$cpid" ] && continue
  child_stat=$(ps -o stat= -p "$cpid" 2>/dev/null | tr -d '[:space:]')
  case "$child_stat" in
    Z*) fail ;;
  esac
done < <(pgrep -P "$parent" 2>/dev/null || true)

echo -n "OK"

Analysis

  1. incident-notes.txt explicitly mentions a past “performance refactor” that affected metrics generation, pointing to /opt/sad/sysinfo-reporter.sh.
  2. check.sh enforces the following tolerances:
  • First line of /var/lib/sad/sysinfo.txt must match SadServers - Philadelphia OK.
  • load_1m in /var/lib/sad/sysinfo.txt must match field 1 of /proc/loadavg within $\pm 0.20$.
  • uptime_seconds in /var/lib/sad/sysinfo.txt must match /proc/uptime within 120 seconds.
  • sad-zombie-lab.service must be active, and no child processes under /opt/sad/zombie-lab can have process state Z* (Zombie).

Decision

Proceed to inspect current published metric values and check the systemd service statuses.


Step 2: Inspecting Current Exporter Output and Service Statuses

Objective

Confirm the baseline failure state in /var/lib/sad/sysinfo.txt and verify service runtime parameters.

Commands

head -n 15 /var/lib/sad/sysinfo.txt
systemctl status sad-zombie-lab --no-pager
systemctl status sad-sysinfo --no-pager

Command Explanation

  • head -n 15: Displays the first 15 lines of a target file.
  • systemctl status <unit>: Displays the operational status, main PID, control group process tree, and recent logs for a systemd unit.
  • --no-pager: Forces output directly to stdout without piping through less or a terminal pager.

Output and Analysis

admin@i-01f4e41baaa55a740:~$ head -n 15 /var/lib/sad/sysinfo.txt 
SadServers - Philadelphia OK
load_1m=0.00
load_5m=0.00
load_15m=0.00
uptime_seconds=86400
admin@i-01f4e41baaa55a740:~$ systemctl status sad-zombie-lab --no-pager
● sad-zombie-lab.service - SadServers Philadelphia zombie process lab
     Loaded: loaded (/etc/systemd/system/sad-zombie-lab.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-11-08 15:35:47 UTC; 1min 53s ago
   Main PID: 781 (zombie-lab)
      Tasks: 2 (limit: 501)
     Memory: 208K (peak: 1.8M)
        CPU: 13ms
     CGroup: /system.slice/sad-zombie-lab.service
             └─781 /opt/sad/zombie-lab
admin@i-01f4e41baaa55a740:~$ systemctl status sad-sysinfo --no-pager
● sad-sysinfo.service - SadServers Philadelphia sysinfo exporter
     Loaded: loaded (/etc/systemd/system/sad-sysinfo.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-11-08 15:35:47 UTC; 2min 46s ago
   Main PID: 779 (bash)
      Tasks: 2 (limit: 501)
     Memory: 1016K (peak: 1.9M)
        CPU: 190ms
     CGroup: /system.slice/sad-sysinfo.service
             ├─ 779 bash /opt/sad/sysinfo-loop.sh
             └─1089 sleep 10

Analysis

  • /var/lib/sad/sysinfo.txt displays static metrics (load_1m=0.00, uptime_seconds=86400, which is exactly $24 \times 3600$ seconds).
  • sad-zombie-lab.service reports Tasks: 2 attached to PID 781, despite showing only one executable in the CGroup tree, indicating an untracked defunct child task in the process list.
  • sad-sysinfo.service is actively executing /opt/sad/sysinfo-loop.sh.

Decision

Inspect /opt/sad/sysinfo-reporter.sh and /opt/sad/zombie.c to identify code-level errors.


Step 3: Source Code Inspection (sysinfo-reporter.sh and zombie.c)

Objective

Analyze the scripts and source code generating metrics and spawning child processes.

Commands

cat /opt/sad/sysinfo-reporter.sh
cat /opt/sad/zombie.c

Output and Analysis

admin@i-01f4e41baaa55a740:~$ cat /opt/sad/sysinfo-reporter.sh
#!/usr/bin/env bash
# Publish host load/uptime for the internal dashboard.
set -euo pipefail

OUT=/var/lib/sad/sysinfo.txt
mkdir -p /var/lib/sad

# "Fast path" — reads cached proc snapshots (refactor regression)
UPTIME=$(awk '{print $1}' /var/lib/sad/proc/uptime)
LOAD1=$(awk '{print $1}' /var/lib/sad/proc/loadavg)
LOAD5=$(awk '{print $2}' /var/lib/sad/proc/loadavg)
LOAD15=$(awk '{print $3}' /var/lib/sad/proc/loadavg)

{
  echo 'SadServers - Philadelphia OK'
  printf 'load_1m=%s\n' "$LOAD1"
  printf 'load_5m=%s\n' "$LOAD5"
  printf 'load_15m=%s\n' "$LOAD15"
  printf 'uptime_seconds=%s\n' "${UPTIME%.*}"
} > "$OUT"
admin@i-01f4e41baaa55a740:~$ cat /opt/sad/zombie.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(void) {
  int pid = fork();

  if (pid == 0) {
    exit(0);
  }

  while (1) {
    sleep(60);
  }

  return 0;
}

Analysis

Issue 1: Path Regression in sysinfo-reporter.sh

The script reads metrics from /var/lib/sad/proc/uptime and /var/lib/sad/proc/loadavg. These are standard, static regular files created on disk during the “performance refactor”. Because they are static disk files rather than virtual procfs streams (/proc/uptime and /proc/loadavg), their values never update.

Issue 2: Unreaped Child Process in zombie.c

In zombie.c, fork() creates a child process:

  • For the child (pid == 0), it calls exit(0) immediately.
  • For the parent (pid > 0), it skips if (pid == 0) and enters an infinite loop while (1) { sleep(60); } without invoking wait() or waitpid().
  • The Linux kernel frees the child’s execution context (memory, file descriptors), but preserves its process control block (PCB) and PID in the kernel process table until the parent reads its exit status code via wait(). Because the parent never calls wait(), the child remains a zombie (TASK_DEAD / Z state) forever.

5. Root Cause Analysis

Metric Exporter Root Cause

The metric exporter failed due to a path regression introduced during a software refactor. Regular files placed at /var/lib/sad/proc/ were being queried instead of kernel interfaces residing at /proc/.

In Linux, reading /proc/loadavg triggers a kernel function that queries live CPU runqueue statistics across scheduler cores. Reading /var/lib/sad/proc/loadavg simply reads static text bytes from disk blocks, returning fixed values.

Zombie Process Root Cause

The zombie child process in sad-zombie-lab stems from a violation of POSIX process management rules.

  Parent Process (PID 781)            Child Process (PID 782)
       |                                   |
       +----- fork() --------------------->|
       |                                   | (Executes exit(0))
       |                                   v
       |                             State: TASK_DEAD
       |                             PCB retained in kernel task table
       |                             Process status: Z (Zombie)
       |
  while(1) { sleep(60); }
  (No wait() or SIGCHLD handler)
       |
       v
  Child remains Z (Zombie) permanently!
  1. Child Exit: A child process invokes exit(). The kernel transitions its state to TASK_DEAD (Z) and releases heap, stack, and file descriptors.
  2. Status Retention: The kernel keeps the process entry (PID, exit code, CPU usage stats) in memory so the parent process can query how the child exited.
  3. Missing Reaping: The parent process loops indefinitely without calling wait(), waitpid(), or registering a handler for SIGCHLD.
  4. Leak: The zombie process accumulates in the kernel task list, consuming a task entry slot until the parent process terminates or is killed.

6. Solution and Remediation

Approach Applied

1. Fix the Metrics Exporter Script

Use sed to update /opt/sad/sysinfo-reporter.sh so awk queries /proc/ instead of /var/lib/sad/proc/.

sudo sed -i 's|/var/lib/sad/proc/|/proc/|g' /opt/sad/sysinfo-reporter.sh
Script Differences (/opt/sad/sysinfo-reporter.sh)
--- /opt/sad/sysinfo-reporter.sh.orig
+++ /opt/sad/sysinfo-reporter.sh
@@ -6,8 +6,8 @@
 OUT=/var/lib/sad/sysinfo.txt
 mkdir -p /var/lib/sad
 
-# "Fast path" — reads cached proc snapshots (refactor regression)
-UPTIME=$(awk '{print $1}' /var/lib/sad/proc/uptime)
-LOAD1=$(awk '{print $1}' /var/lib/sad/proc/loadavg)
-LOAD5=$(awk '{print $2}' /var/lib/sad/proc/loadavg)
-LOAD15=$(awk '{print $3}' /var/lib/sad/proc/loadavg)
+# Live kernel procfs interface
+UPTIME=$(awk '{print $1}' /proc/uptime)
+LOAD1=$(awk '{print $1}' /proc/loadavg)
+LOAD5=$(awk '{print $2}' /proc/loadavg)
+LOAD15=$(awk '{print $3}' /proc/loadavg)

Restart the exporter service to populate /var/lib/sad/sysinfo.txt with live metrics:

sudo systemctl restart sad-sysinfo

2. Fix the Zombie Process C Source & Recompile

Modify /opt/sad/zombie.c to include <sys/wait.h> and invoke wait(NULL); immediately after the fork() branch in the parent context.

sudo bash -c 'cat << "EOF" > /opt/sad/zombie.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>

int main(void) {
    int pid = fork();

    if (pid == 0) {
        exit(0);
    }

    wait(NULL);

    while (1) {
        sleep(60);
    }

    return 0;
}
EOF'
C Source Code Differences (/opt/sad/zombie.c)
--- /opt/sad/zombie.c.orig
+++ /opt/sad/zombie.c
@@ -1,6 +1,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <unistd.h>
+#include <sys/wait.h>
 
 int main(void) {
   int pid = fork();
@@ -9,6 +10,8 @@
     exit(0);
   }
 
+  wait(NULL);
+
   while (1) {
     sleep(60);
   }

Recompile the binary and restart the service:

sudo gcc -Wall -o /opt/sad/zombie-lab /opt/sad/zombie.c
sudo systemctl restart sad-zombie-lab

7. Verification and Validation

Step 1: Run System Verification Script

Execute the scenario check harness /home/admin/agent/check.sh:

admin@i-01f4e41baaa55a740:~$ /home/admin/agent/check.sh
OK

Step 2: Validate Metric Alignment

Compare the generated /var/lib/sad/sysinfo.txt file against live /proc/loadavg and /proc/uptime:

cat /var/lib/sad/sysinfo.txt
cat /proc/loadavg
cat /proc/uptime

Output

SadServers - Philadelphia OK
load_1m=0.01
load_5m=0.03
load_15m=0.01
uptime_seconds=345

The published values now update continuously and accurately reflect live kernel telemetry.

Step 3: Inspect Service Task List

Verify that sad-zombie-lab.service reports Tasks: 1 and has no defunct child processes:

systemctl status sad-zombie-lab --no-pager
pgrep -P $(pgrep -f '^/opt/sad/zombie-lab$')

Output

● sad-zombie-lab.service - SadServers Philadelphia zombie process lab
     Loaded: loaded (/etc/systemd/system/sad-zombie-lab.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-11-08 15:40:12 UTC; 45s ago
   Main PID: 1240 (zombie-lab)
      Tasks: 1 (limit: 501)
...

Tasks: 1 confirms that the child process was successfully reaped by wait(), leaving zero zombies in the process control block table.


8. Alternative Troubleshooting Approaches

Approach A: Tracing I/O and System Calls with strace

Instead of manually reading shell scripts, strace allows you to observe system calls made by daemons in real time.

sudo strace -e trace=openat,read /opt/sad/sysinfo-reporter.sh
  • What it reveals: Shows exact file paths passed to openat(). If the output shows openat(AT_FDCWD, "/var/lib/sad/proc/loadavg", ...) instead of /proc/loadavg, the path regression is immediately obvious without reading shell code.
  • Advantages: Language-agnostic; works on compiled binaries where source code is unavailable.
  • Limitations: High performance overhead; produces verbose output on busy processes.

Approach B: Asynchronous Zombie Reaping via Signal Handling (SIGCHLD)

In C programming, calling a blocking wait(NULL); prevents the parent from continuing until the child exits. For non-blocking asynchronous servers, alternative strategies exist:

Option B1: Explicit Signal Handling

Catch SIGCHLD and reap asynchronously using waitpid():

#include <signal.h>
#include <sys/wait.h>

void handle_sigchld(int sig) {
    while (waitpid(-1, NULL, WNOHANG) > 0);
}

int main(void) {
    signal(SIGCHLD, handle_sigchld);
    if (fork() == 0) { exit(0); }
    while (1) { sleep(60); }
}

Option B2: Ignore SIGCHLD (SIG_IGN)

Instruct the kernel to automatically reap exited children without requiring a wait() call:

#include <signal.h>

int main(void) {
    signal(SIGCHLD, SIG_IGN);
    if (fork() == 0) { exit(0); }
    while (1) { sleep(60); }
}
  • When to use: SIG_IGN is ideal for long-running daemons that fork worker processes but do not care about their exit codes.

Comparison of Diagnostic Approaches

Tool / Interface Purpose Primary Use Case Output Complexity Overhead
ps aux / ps -ef Process table snapshot Identify defunct processes (Z state flag) Low Very Low
pgrep -P <PID> Parent-child mapping Identify process hierarchy and child PIDs Low Very Low
/proc/<PID>/status Direct kernel process state Inspect state (State: Z (zombie)), memory, and parent PID Medium Minimal
strace -f -e trace=process System call tracing Follow fork(), clone(), exit(), and waitpid() calls across threads High High
systemctl status Unit health & CGroup overview Check Tasks: count for unexpected process accumulation Low Very Low

9. Linux Command Deep Dive

1. sed (Stream Editor)

  • Purpose: Performs text parsing and transformations on file streams or input files.

General Syntax

sed [options] 'script' [input-file]

Common Options & Arguments

Option / Syntax Description
-i[SUFFIX], --in-place Edit files in place instead of printing output to stdout.
`s regexp
-n, --quiet Suppress automatic printing of pattern space.
-e script Execute multiple script commands in sequence.

Useful Examples

# Replace /var/log with /tmp/log in a config file
sed -i 's|/var/log|/tmp/log|g' /etc/app.conf

# Print lines 10 through 20 of a log file
sed -n '10,20p' /var/log/syslog

2. gcc (GNU Compiler Collection)

  • Purpose: Compiles C, C++, and assembly source code into executable binary files.

General Syntax

gcc [options] [file...]

Common Options & Arguments

Option / Syntax Description
-Wall Enables all common compiler warnings (e.g., unused variables, missing header includes).
-Wextra Enables extra diagnostic warnings for strict code quality.
-o <filename> Write output binary to <filename> instead of a.out.
-g Generates debugging symbol information for gdb.
-O2 Enables standard performance optimizations.

Useful Examples

# Compile C file with strict warnings and output binary name
gcc -Wall -Wextra -o mydaemon daemon.c

# Compile with debugging symbols for gdb inspection
gcc -g -o mydaemon daemon.c

3. pgrep (Process Grep)

  • Purpose: Searches currently running processes and lists PIDs matching specified criteria.

General Syntax

pgrep [options] pattern

Common Options & Arguments

Option / Syntax Description
-P <PPID> Match only processes whose Parent Process ID is <PPID>.
-f Match pattern against full command line instead of executable name.
-u <user> Match only processes owned by specified username or UID.
-l List process name alongside PID.
-a List full command line alongside PID.

Useful Examples

# Find all child PIDs belonging to parent PID 1240
pgrep -P 1240

# Find PID matching full binary path
pgrep -f '^/opt/sad/zombie-lab$'

4. ps (Process Status)

  • Purpose: Provides a snapshot of currently running processes.

General Syntax

ps [options]

Options Reference

Syntax Format Flag Description
UNIX style ps -ef Standard full-format listing (UID, PID, PPID, C, STIME, TTY, TIME, CMD).
BSD style ps aux Display all processes including user, CPU/MEM %, process state (STAT).
Custom Format ps -o stat= -p <PID> Print specific output fields (e.g., stat) without printing the header line.
Sorting ps aux --sort=-%cpu Sort processes by descending CPU usage.
Sorting ps aux --sort=-%mem Sort processes by descending Memory usage.

10. Underlying Linux Concepts and Architecture

1. Procfs (/proc Virtual Filesystem)

The /proc directory is not a real disk filesystem; it is a pseudo-filesystem (procfs) mounted dynamically by the Linux kernel.

+-------------------------------------------------------------+
|                        User Space                           |
|  cat /proc/loadavg        awk '{print $1}' /proc/uptime     |
+-------------------------------------------------------------+
                              | Read Syscall
                              v
+-------------------------------------------------------------+
|                       Linux Kernel                          |
|  procfs VFS Handler -> Query Scheduler & Timer Subsystems   |
+-------------------------------------------------------------+
                              | Live Metrics Stream
                              v
                 Dynamic CPU Load & Uptime
  • Dynamic Data Generation: When a process calls read() on a file in /proc, the kernel’s virtual filesystem (VFS) intercepts the request and runs internal kernel code to generate text output on the fly.
  • Disk vs. Procfs: Real files stored on EXT4 or XFS filesystems read fixed blocks from persistent storage blocks. Procfs queries live RAM kernel data structures (such as struct task_struct and scheduler queue metrics).

2. POSIX Process Lifecycle & Zombie Reaping

Every process in Linux is assigned a Process Control Block (task_struct).

 +-------------+       fork()       +-------------+       exit()      +-------------+
 |  Created    | -----------------> |   Running   | ----------------> | TASK_DEAD   |
 |  (Parent)   |                    |   (Child)   |                   |  (Zombie)   |
 +-------------+                    +-------------+                   +-------------+
                                                                             |
                                                                             | wait() called
                                                                             v
                                                                      +-------------+
                                                                      |  Removed    |
                                                                      | from Table  |
                                                                      +-------------+
  1. fork(): Copies the parent process to create a child process with a unique PID.
  2. exit(): The child process terminates. Memory allocations, file descriptor tables, and thread structures are freed. The status is updated to TASK_DEAD (Z).
  3. The Zombie State: The kernel maintains the PID entry in the task list so the parent process can collect its exit status via wait() or waitpid().
  4. Reaping: When the parent calls wait(), the kernel reads the exit code and frees the remaining process control block entry.
  5. Orphan Adoption: If a parent process terminates before reaping its children, systemd (PID 1) or an init sub-reaper adopts the child and automatically calls wait() when the child exits.

11. Real-World Sysadmin and SRE Applications

1. Metric Integrity Monitoring & Alerts

  • Stale Metric Detection: In production, monitoring agents (e.g., Prometheus Node Exporter, Datadog) should export a counter metric like node_exporter_scrape_time_seconds. Configure alerting rules to trigger when a host’s load average remains bit-for-bit identical across multiple scrape intervals, indicating a stuck collector daemon or snapshot path regression.

2. Process Leak Prevention in Production

  • Systemd Task Limits (TasksMax): Set TasksMax= in service unit files to cap the number of tasks (threads + processes) a unit can create. This prevents zombie process leaks from exhausting kernel process IDs (/proc/sys/kernel/pid_max).
[Service]
TasksMax=64
  • Container Init Process (tini): In Docker or Kubernetes containers, running application processes as PID 1 often causes zombie build-up because standard application processes do not include init-style process reaping logic. Using lightweight init systems like tini or docker run --init ensures orphaned zombie processes are automatically reaped.

12. Further Learning and Experiments

Hands-on Lab Experiments

Experiment 1: Inspecting Procfs Filesystem Behavior

Execute the following to observe how procfs generates live dynamic output:

# Continuously monitor kernel procfs virtual file vs text file
watch -n 1 "cat /proc/loadavg"
  • Observation: Notice how /proc/loadavg dynamically changes without requiring manual updates or disk writes.

Experiment 2: Reproducing Zombie Accumulation

  1. Create a test script named test_zombie.c:
#include <unistd.h>
#include <stdlib.h>
int main() {
    if (fork() == 0) exit(0);
    sleep(30);
    return 0;
}
  1. Compile and run in background:
gcc -o test_zombie test_zombie.c
./test_zombie &
  1. Check the process table:
ps aux | grep -E 'Z|defunct'
  • Learning Objective: Observe how the child process transitions to Z state (<defunct>) during the parent’s 30-second sleep window, and automatically disappears once the parent exits and gets reaped by PID 1.

Documentation References

  • man 2 wait / man 2 waitpid: System calls for waiting for process state changes.
  • man 5 proc: Manual page for the /proc virtual filesystem interface.
  • man 2 fork: Process creation system call mechanics.

13. Key Takeaways

  1. Procfs is Virtual: Linux metrics in /proc are dynamic kernel streams. Never configure exporters to read static snapshot files on disk (/var/lib/...) unless explicitly implementing historical caching.
  2. Always Reap Child Processes: Any C/Linux application invoking fork() must handle child termination via wait(), waitpid(), or by explicitly setting SIGCHLD to SIG_IGN.
  3. Validate Task Counts: Use systemctl status to inspect Tasks: metrics or check ps aux for process state Z to quickly identify zombie leaks.
  4. Use Standard System Calls: When troubleshooting path issues, tools like strace provide transparent visibility into openat() file descriptor calls.

14. Quick Troubleshooting Reference / Runbook

                         Metric & Process Failure Runbook
                                         |
         +-------------------------------+-------------------------------+
         |                                                               |
         v                                                               v
[ Issue 1: Metrics Exporter Stale ]                             [ Issue 2: Zombie Process Accumulation ]
         |                                                               |
1. Compare output file vs kernel procfs:                        1. Check for process state Z:
   diff <(head -n5 /var/lib/sad/sysinfo.txt)                       ps -eo pid,ppid,stat,cmd | grep ' Z'
   <(cat /proc/loadavg)                                          
                                                                2. Identify parent PID:
2. Trace file read operations:                                     pgrep -P <PARENT_PID>
   strace -e openat /path/to/exporter.sh                         
                                                                3. Check systemd unit task count:
3. Update script path to /proc/<metric>                            systemctl status <unit-name>
                                                                 
4. Restart metric service:                                      4. Update code: Add wait() or SIGCHLD handler
   systemctl restart <exporter-service>                          
                                                                5. Recompile & restart unit:
                                                                   gcc -Wall -o <binary> <source.c>
                                                                   systemctl restart <unit-name>