Domain 10Linux & SREINTRO

Demystifying Linux Signals: A Sysadmin's Guide to SIGTERM, SIGKILL, and Process Management

A practical guide to understanding Linux signals, analyzing process termination issues, and knowing when to use SIGTERM, SIGKILL, and other IPC signals in production.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-02
#linux#signals#process-management#troubleshooting#systemd#sysadmin

1. Overview

In Linux system administration, managing running processes is a foundational task. Whether an application is unresponsive, consuming excess CPU, or hanging during a service stop operation, knowing how to communicate with processes via signals is essential.

This technical guide explores the mechanisms of Linux software signals—specifically focusing on standard signals like SIGTERM (Signal 15), SIGKILL (Signal 9), SIGHUP (Signal 1), and SIGINT (Signal 2). Through a practical troubleshooting scenario involving a hanging worker daemon, we break down how signals work, how the kernel delivers them, why processes sometimes ignore shutdown requests, and how to safely inspect and terminate stuck jobs without causing data corruption or resource leaks.



2. Problem Description

During a routine deployment update on an Ubuntu web server, a custom queue worker process (worker_service) refused to stop cleanly.

  • Expected Behavior: Issuing a service stop command (systemctl stop worker_service) or standard process termination (kill <pid>) sends a SIGTERM signal. The process should catch the signal, finish its active database transaction, release file locks, and exit cleanly with code 0.
  • Actual Behavior: The systemctl stop command hung for 90 seconds before hitting systemd’s default TimeoutStopSec limit. The service was forcefully killed by systemd, leaving database connections open and orphan temporary files in /tmp/worker_locks/.
  • Constraints: The system was in production; blindly issuing standard SIGKILL commands without identifying why SIGTERM failed risked corrupting downstream task state.

Information Gap Note: In this hands-on exercise, full kernel trace dumps (/proc/sys/kernel/sysrq) were not available. Analysis was conducted using standard userspace inspection utilities (ps, strace, lsof, and /proc).


3. Initial Investigation and Troubleshooting Strategy

When a process fails to respond to standard termination signals, system administrators should investigate in a structured sequence:

  1. Verify Process State: Is the process in a Runnable (R), Sleeping (S), or Uninterruptible Sleep (D) state? Processes in state D are waiting on disk or network I/O and cannot process any signals—including SIGKILL—until the I/O operation returns or times out.
  2. Inspect Signal Handling Masks: Examine /proc/<PID>/status to determine which signals the application has explicitly registered handlers for (SigCgt), which are blocked (SigBlk), and which are pending (SigPnd).
  3. Trace System Calls: Attach a system call tracer (strace) to check if the process is stuck in an infinite retry loop, a deadlocked mutex, or waiting on an unblocked socket read.
  4. Determine the Scope: Check whether child processes or thread pools are holding parent locks open.

4. Step-by-Step Troubleshooting

Step 1: Locating the Target Process and Checking State

Objective: Identify the Process ID (PID), parent process ID (PPID), current process state, and execution flags of the hung daemon.

ps aux | grep [w]orker_service

Command Breakdown

  • ps aux: Displays all processes (a = all users, u = user-oriented format, x = include processes without a controlling TTY).
  • grep [w]orker_service: Uses character class matching to prevent grep itself from appearing in the output list.

Observed Result

appuser   14285  0.1  0.4  152400 34200 ?        Ss   10:15   0:01 /usr/bin/python3 /opt/services/worker_service.py --config /etc/worker.conf

Interpretation

The process is running under PID 14285 in state Ss (S = Interruptible Sleep, s = Session leader). Because it is in interruptible sleep, the process is capable of receiving and processing signals from the kernel. It is not stuck in an Uninterruptible I/O state (D).

Key Takeaway

If a process is in state S, it is actively choosing not to terminate upon receiving SIGTERM because its application code has either caught the signal and gotten stuck in a handler, or is explicitly ignoring it.


Step 2: Attempting Standard Graceful Termination (SIGTERM)

Objective: Test if sending an explicit SIGTERM signal via kill triggers any response or error logging in standard streams.

kill -15 14285

Command Breakdown

  • kill: Built-in/standalone utility to send a POSIX signal to a process.
  • -15: Explicitly specifies signal 15 (SIGTERM). If omitted, 15 is the default value for kill.
  • 14285: Target PID.

Observed Result

No terminal output returned (standard exit code 0). Re-checking ps aux | grep 14285 showed PID 14285 was still running without state changes.

Interpretation

Sending SIGTERM did not terminate PID 14285. The application registered a custom signal handler for SIGTERM that caught the signal, but failed to complete its cleanup subroutine.


Step 3: Inspecting Process Signal Masks in /proc

Objective: Read the process’s internal bitmasks to confirm caught, blocked, and ignored signals.

cat /proc/14285/status | grep -i sig

Command Breakdown

  • /proc/14285/status: Human-readable summary of process execution state and signal masks maintained by the Linux kernel task structure.
  • grep -i sig: Filters signal fields (SigPnd, SigBlk, SigIgn, SigCgt).

Observed Result

SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000004
SigCgt: 0000000000004006

Interpretation

The values are represented in hexadecimal bitmasks:

  • SigIgn: 0000000000000004 $\rightarrow$ Bit 3 set = Signal 3 (SIGQUIT) is set to be Ignored.
  • SigCgt: 0000000000004006 $\rightarrow$ Bits set correspond to Signals 2 (SIGINT), 3 (SIGQUIT), 14 (SIGALRM), and 15 (SIGTERM). Custom handlers are actively attached (Caught).

Step 4: Tracing Execution Behavior with strace

Objective: Attach strace to PID 14285 to monitor active system calls and see where the execution thread is blocked.

sudo strace -p 14285

Command Breakdown

  • sudo: Required to trace processes running under other execution contexts or users.
  • strace: Diagnostics tool that intercepts and logs system calls made by a process.
  • -p 14285: Attaches to running PID 14285.

Observed Result

strace: Process 14285 attached
futex(0x55b4129a8340, FUTEX_WAIT_PRIVATE, 0, NULL ...

Interpretation

The process is stuck in a futex() (fast userspace mutex) system call waiting indefinitely on a lock that was never released by a worker thread. When SIGTERM was delivered, the custom python signal_handler attempted to acquire this locked resource before exiting, creating a deadlock.


5. Root Cause Analysis

The root cause of the hung service was a software deadlock within the application’s signal handling implementation:

  1. Custom Signal Handler Registration: The Python script explicitly registered a handler for SIGTERM (signal.signal(signal.SIGTERM, cleanup_handler)).
  2. Deadlock on Cleanup: Inside cleanup_handler(), the code attempted to acquire a global thread lock (db_lock.acquire()).
  3. Unreleased Lock: An unhandled internal exception in a secondary thread left db_lock locked.
  4. Indefinite Hang: When SIGTERM arrived, the main thread caught the signal, entered cleanup_handler(), blocked on db_lock.acquire(), and entered FUTEX_WAIT_PRIVATE.
  5. Inability to Exit via SIGTERM: Because the main thread was waiting on the deadlock inside the SIGTERM handler itself, subsequent SIGTERM signals merely re-queued or were ignored while waiting on the single-threaded execution lock.

6. Solution and Recovery

Since the application was deadlocked inside its own signal handler and could not process user-level requests, the non-catchable signal SIGKILL (Signal 9) had to be issued to allow kernel-level force termination.

Applied Solution

sudo kill -9 14285

Command Breakdown

  • kill -9: Sends signal SIGKILL. Unlike SIGTERM, SIGKILL cannot be caught, ignored, or blocked by userspace code. The kernel immediately destroys the task structure and frees allocated virtual memory pages, open file descriptors, and socket bindings.

7. Verification

After forcefully terminating the process, perform verification checks to ensure the system returns to a clean state:

# 1. Confirm process is no longer in process table
ps aux | grep [w]orker_service

# 2. Verify network ports and file locks were released by kernel
sudo lsof /tmp/worker_locks/

# 3. Clean up dangling state files manually if necessary
sudo rm -f /tmp/worker_locks/*.lock

# 4. Start service and check status via systemd
sudo systemctl start worker_service
sudo systemctl status worker_service

Verification Result

● worker_service.service - Queue Worker Service
     Loaded: loaded (/etc/systemd/system/worker_service.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-11-02 10:25:12 UTC; 15s ago
   Main PID: 14890 (python3)

8. Deeper Dive: Understanding the Linux Commands

kill

Sends specified signals to target processes or process groups.

General Syntax

kill [signal_or_option] <pid>

Flags Used in Exercise

  • -15 (-SIGTERM): Request graceful shutdown.
  • -9 (-SIGKILL): Immediate kernel-level force termination.

Additional Useful Flags

  • -l (List signals): Displays all available POSIX signals and their corresponding numbers.
kill -l
  • -s <SIGNAL_NAME>: Explicitly names the signal (e.g., kill -s SIGHUP 14285).
  • -1 or -- -PGID: Kills an entire Process Group ID rather than a single PID.
kill -9 -- -14285

ps

Displays snapshot information regarding active system processes.

General Syntax

ps [options]

Flags Used in Exercise

  • aux: BSD style syntax showing user, PID, CPU/MEM usage, process state, and full command-line parameters.

Additional Useful Flags

  • -ef (Standard UNIX format): Shows full process hierarchy with parent IDs (PPID).
ps -ef
  • -o <format> (Custom output formatting): Selects specific columns for script parsing.
ps -eo pid,ppid,stat,comm,args
  • --forest: Renders a visual tree showing parent-child process inheritance.
ps aux --forest

strace

Diagnostic utility for monitoring interactions between userspace applications and the Linux kernel.

General Syntax

strace [options] [-p PID | command]

Useful Troubleshooting Options

  • -c: Summarize time, calls, and errors for each system call rather than printing raw stream data.
sudo strace -c -p 14285
  • -e trace=<syscall>: Filters for specific system calls (e.g., openat, connect, write).
sudo strace -e trace=network,file -p 14285
  • -f: Follow child threads/processes created via fork() or clone().

9. Exploring Alternative Troubleshooting Methods

1. pkill and killall

Instead of finding the PID manually via ps and running kill, pattern-based tools can streamline signal delivery.

# Terminate matching process by name
pkill -15 -f worker_service.py

# Send signal to all instances
killall -SIGTERM worker_service
  • When to Use: Useful when dealing with multiple worker processes or auto-scaling thread pools.
  • Trade-off: Risk of accidentally matching unintended processes if the search string is too generic.

2. Inspecting Signals via /proc/[pid]/status Bitmasks

You can inspect the bitmask fields directly to decode signal configurations without third-party diagnostic tools:

Field Description
SigPnd Signals pending delivery to the specific thread.
ShdPnd Signals pending delivery to the process as a whole.
SigBlk Signals blocked by thread masks (sigprocmask).
SigIgn Signals explicitly set to SIG_IGN (Ignored).
SigCgt Signals caught by explicit handler functions (SIG_DFL overridden).

To convert a hex status mask like 0000000000004006 to signal names:

python3 -c '
mask = 0x0000000000004006
for i in range(1, 65):
    if (mask >> (i - 1)) & 1:
        print(f"Signal {i}")
'

3. Native systemctl kill Command

When managing services under systemd, systemd offers a native signal wrapper:

# Send SIGTERM to all processes in the service control group (cgroup)
sudo systemctl kill -s SIGTERM worker_service.service

# Force kill all processes in the cgroup
sudo systemctl kill -s SIGKILL worker_service.service
  • Advantage: Guarantees that spawned child processes and orphaned sub-threads inside the cgroup are killed together, preventing dangling processes.

10. Deeper Linux Concepts and Internals

The Linux Signal Architecture

Signals are standardized, asynchronous notifications delivered by the Linux kernel to a process to inform it of an event. When a signal is generated, the kernel updates the process descriptor (struct task_struct) in kernel space.

+--------------------------------------------------------------------+
|                             KERNEL                                 |
|  +---------------------+        +-------------------------------+  |
|  | Signal Generation   | -----> | Update task_struct Bitmask    |  |
|  | (Hardware/Software) |        | (SigPnd / SigCgt)             |  |
|  +---------------------+        +-------------------------------+  |
+--------------------------------------------------------------------+
                                   |
                                   v (On return to userspace)
+--------------------------------------------------------------------+
|                           USERSPACE                                |
|  +--------------------------------------------------------------+  |
|  | Default Action / Custom Handler Execution / Force Exit       |  |
|  +--------------------------------------------------------------+  |
+--------------------------------------------------------------------+
  1. Signal Generation: Triggered by hardware exceptions (e.g., SIGSEGV on bad memory access), software events (e.g., SIGPIPE writing to broken pipe), or user interaction (kill()).
  2. Signal Pending: The signal rests in the target’s pending queue until the process is scheduled to run.
  3. Signal Delivery: When the CPU transitions the process from kernel mode back to user mode, the kernel checks the pending mask against blocked masks and executes one of four standard actions:
  • Terminate: Process terminates immediately (e.g., SIGTERM, SIGKILL).
  • Core Dump: Process terminates and writes a memory image to disk (e.g., SIGQUIT, SIGSEGV).
  • Ignore: Signal is discarded (e.g., SIGCHLD).
  • Stop/Continue: Process execution is paused or resumed (e.g., SIGSTOP, SIGCONT).

Key POSIX Signals Reference

Signal Code Standard Action Can be Caught? Can be Blocked? Typical Use Case
SIGHUP 1 Terminate Yes Yes Terminal disconnect / Reload configuration files
SIGINT 2 Terminate Yes Yes Terminal interrupt (Ctrl+C)
SIGQUIT 3 Core Dump Yes Yes Terminal quit request (Ctrl+\)
SIGKILL 9 Terminate NO NO Force uncatchable immediate process destruction
SIGUSR1 10 Terminate Yes Yes Application-defined custom action
SIGUSR2 12 Terminate Yes Yes Application-defined custom action
SIGTERM 15 Terminate Yes Yes Standard request for graceful application exit
SIGSTOP 19 Stop NO NO Pause process execution (Job control Ctrl+Z)
SIGCONT 18 Continue Yes Yes Resume execution of stopped process

11. Common Mistakes and Troubleshooting Pitfalls

1. Reaching for SIGKILL (kill -9) Immediately

  • Pitfall: SIGKILL bypasses application cleanup routines entirely.
  • Impact: Temporary files remain, database connections linger, locks stay held, and shared memory segments (shm) remain allocated.
  • Prevention: Always attempt SIGTERM (15) or SIGINT (2) first. Reserve SIGKILL for cases where the process is demonstrably non-responsive.

2. Confusing Uninterruptible Sleep (D state) with Deadlocks

  • Pitfall: Issuing kill -9 against a process in state D and expecting it to disappear immediately.
  • Impact: The process remains in the process table until the underlying hardware/NFS I/O request finishes or fails.
  • Prevention: Use ps aux or top to check the process state column (STAT). If state D is observed, investigate the storage subsystem, mount points, or kernel dmesg logs.

3. Orphaned Child Processes

  • Pitfall: Killing a parent process with SIGKILL while its children are running.
  • Impact: Child processes become orphans and are re-parented to init (PID 1) or a systemd user manager, continuing to consume system resources.
  • Prevention: Use cgroup-aware signal tools (systemctl kill) or terminate process groups (kill -- -PGID).

12. Quick Reference

Command Quick-Sheet

Command Purpose Useful Flags / Arguments When to Use
kill Send signal to PID -15 (SIGTERM), -9 (SIGKILL), -1 (SIGHUP) Targeted process termination
pkill Send signal by pattern -f (Full command line matching), -u (User) Terminating processes by name
ps Inspect processes aux, -ef, -eo pid,stat,comm Checking process states & PIDs
systemctl kill Kill service cgroup -s SIGTERM, -s SIGKILL Service daemon management
strace Trace system calls -p <PID>, -f (Follow threads), -c (Summary) Diagnosing hung or deadlocked processes
lsof List open files -p <PID>, +D <dir> Checking held lockfiles or open sockets

Key Lessons to Remember

  1. SIGTERM is a request; SIGKILL is an instruction. SIGTERM asks the application nicely; SIGKILL tells the kernel to destroy it.
  2. Processes in state D cannot be killed by any signal until they exit kernel I/O wait.
  3. Always prefer cgroup-based signal delivery (systemctl kill) for managed services to avoid leaving orphan processes.

13. Further Learning and Practice

To deepen your understanding of Linux signals and process handling, try the following practical exercises in a test virtual machine:

  1. Simulate Configuration Reloads:
  • Install nginx or apache2.
  • Send SIGHUP (sudo kill -1 <PID>) and observe /var/log/nginx/error.log to watch worker processes re-read configuration without dropping connections.
  1. Signal Trapping in Bash:
  • Write a shell script that traps SIGTERM (trap 'echo Caught SIGTERM; exit 0' SIGTERM).
  • Run the script in the background, send signals using kill, and observe how custom handlers intercept execution.
  1. Investigating Process State Limits:
  • Mount an intentionally unresponsive NFS share or simulate high I/O latency to observe processes entering state D.