1. Overview
In Linux system administration, managing running processes is a foundational task. Whether an application is unresponsive, consuming excess CPU, or hanging during a service stop operation, knowing how to communicate with processes via signals is essential.
This technical guide explores the mechanisms of Linux software signals—specifically focusing on standard signals like SIGTERM (Signal 15), SIGKILL (Signal 9), SIGHUP (Signal 1), and SIGINT (Signal 2). Through a practical troubleshooting scenario involving a hanging worker daemon, we break down how signals work, how the kernel delivers them, why processes sometimes ignore shutdown requests, and how to safely inspect and terminate stuck jobs without causing data corruption or resource leaks.
2. Problem Description
During a routine deployment update on an Ubuntu web server, a custom queue worker process (worker_service) refused to stop cleanly.
- Expected Behavior: Issuing a service stop command (
systemctl stop worker_service) or standard process termination (kill <pid>) sends aSIGTERMsignal. The process should catch the signal, finish its active database transaction, release file locks, and exit cleanly with code0. - Actual Behavior: The
systemctl stopcommand hung for 90 seconds before hitting systemd’s defaultTimeoutStopSeclimit. The service was forcefully killed by systemd, leaving database connections open and orphan temporary files in/tmp/worker_locks/. - Constraints: The system was in production; blindly issuing standard
SIGKILLcommands without identifying whySIGTERMfailed risked corrupting downstream task state.
Information Gap Note: In this hands-on exercise, full kernel trace dumps (
/proc/sys/kernel/sysrq) were not available. Analysis was conducted using standard userspace inspection utilities (ps,strace,lsof, and/proc).
3. Initial Investigation and Troubleshooting Strategy
When a process fails to respond to standard termination signals, system administrators should investigate in a structured sequence:
- Verify Process State: Is the process in a Runnable (
R), Sleeping (S), or Uninterruptible Sleep (D) state? Processes in stateDare waiting on disk or network I/O and cannot process any signals—includingSIGKILL—until the I/O operation returns or times out. - Inspect Signal Handling Masks: Examine
/proc/<PID>/statusto determine which signals the application has explicitly registered handlers for (SigCgt), which are blocked (SigBlk), and which are pending (SigPnd). - Trace System Calls: Attach a system call tracer (
strace) to check if the process is stuck in an infinite retry loop, a deadlocked mutex, or waiting on an unblocked socket read. - Determine the Scope: Check whether child processes or thread pools are holding parent locks open.
4. Step-by-Step Troubleshooting
Step 1: Locating the Target Process and Checking State
Objective: Identify the Process ID (PID), parent process ID (PPID), current process state, and execution flags of the hung daemon.
ps aux | grep [w]orker_service
Command Breakdown
ps aux: Displays all processes (a= all users,u= user-oriented format,x= include processes without a controlling TTY).grep [w]orker_service: Uses character class matching to preventgrepitself from appearing in the output list.
Observed Result
appuser 14285 0.1 0.4 152400 34200 ? Ss 10:15 0:01 /usr/bin/python3 /opt/services/worker_service.py --config /etc/worker.conf
Interpretation
The process is running under PID 14285 in state Ss (S = Interruptible Sleep, s = Session leader). Because it is in interruptible sleep, the process is capable of receiving and processing signals from the kernel. It is not stuck in an Uninterruptible I/O state (D).
Key Takeaway
If a process is in state S, it is actively choosing not to terminate upon receiving SIGTERM because its application code has either caught the signal and gotten stuck in a handler, or is explicitly ignoring it.
Step 2: Attempting Standard Graceful Termination (SIGTERM)
Objective: Test if sending an explicit SIGTERM signal via kill triggers any response or error logging in standard streams.
kill -15 14285
Command Breakdown
kill: Built-in/standalone utility to send a POSIX signal to a process.-15: Explicitly specifies signal 15 (SIGTERM). If omitted,15is the default value forkill.14285: Target PID.
Observed Result
No terminal output returned (standard exit code 0). Re-checking ps aux | grep 14285 showed PID 14285 was still running without state changes.
Interpretation
Sending SIGTERM did not terminate PID 14285. The application registered a custom signal handler for SIGTERM that caught the signal, but failed to complete its cleanup subroutine.
Step 3: Inspecting Process Signal Masks in /proc
Objective: Read the process’s internal bitmasks to confirm caught, blocked, and ignored signals.
cat /proc/14285/status | grep -i sig
Command Breakdown
/proc/14285/status: Human-readable summary of process execution state and signal masks maintained by the Linux kernel task structure.grep -i sig: Filters signal fields (SigPnd,SigBlk,SigIgn,SigCgt).
Observed Result
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000004
SigCgt: 0000000000004006
Interpretation
The values are represented in hexadecimal bitmasks:
SigIgn: 0000000000000004$\rightarrow$ Bit 3 set = Signal 3 (SIGQUIT) is set to be Ignored.SigCgt: 0000000000004006$\rightarrow$ Bits set correspond to Signals 2 (SIGINT), 3 (SIGQUIT), 14 (SIGALRM), and 15 (SIGTERM). Custom handlers are actively attached (Caught).
Step 4: Tracing Execution Behavior with strace
Objective: Attach strace to PID 14285 to monitor active system calls and see where the execution thread is blocked.
sudo strace -p 14285
Command Breakdown
sudo: Required to trace processes running under other execution contexts or users.strace: Diagnostics tool that intercepts and logs system calls made by a process.-p 14285: Attaches to running PID14285.
Observed Result
strace: Process 14285 attached
futex(0x55b4129a8340, FUTEX_WAIT_PRIVATE, 0, NULL ...
Interpretation
The process is stuck in a futex() (fast userspace mutex) system call waiting indefinitely on a lock that was never released by a worker thread. When SIGTERM was delivered, the custom python signal_handler attempted to acquire this locked resource before exiting, creating a deadlock.
5. Root Cause Analysis
The root cause of the hung service was a software deadlock within the application’s signal handling implementation:
- Custom Signal Handler Registration: The Python script explicitly registered a handler for
SIGTERM(signal.signal(signal.SIGTERM, cleanup_handler)). - Deadlock on Cleanup: Inside
cleanup_handler(), the code attempted to acquire a global thread lock (db_lock.acquire()). - Unreleased Lock: An unhandled internal exception in a secondary thread left
db_locklocked. - Indefinite Hang: When
SIGTERMarrived, the main thread caught the signal, enteredcleanup_handler(), blocked ondb_lock.acquire(), and enteredFUTEX_WAIT_PRIVATE. - Inability to Exit via SIGTERM: Because the main thread was waiting on the deadlock inside the
SIGTERMhandler itself, subsequentSIGTERMsignals merely re-queued or were ignored while waiting on the single-threaded execution lock.
6. Solution and Recovery
Since the application was deadlocked inside its own signal handler and could not process user-level requests, the non-catchable signal SIGKILL (Signal 9) had to be issued to allow kernel-level force termination.
Applied Solution
sudo kill -9 14285
Command Breakdown
kill -9: Sends signalSIGKILL. UnlikeSIGTERM,SIGKILLcannot be caught, ignored, or blocked by userspace code. The kernel immediately destroys the task structure and frees allocated virtual memory pages, open file descriptors, and socket bindings.
7. Verification
After forcefully terminating the process, perform verification checks to ensure the system returns to a clean state:
# 1. Confirm process is no longer in process table
ps aux | grep [w]orker_service
# 2. Verify network ports and file locks were released by kernel
sudo lsof /tmp/worker_locks/
# 3. Clean up dangling state files manually if necessary
sudo rm -f /tmp/worker_locks/*.lock
# 4. Start service and check status via systemd
sudo systemctl start worker_service
sudo systemctl status worker_service
Verification Result
● worker_service.service - Queue Worker Service
Loaded: loaded (/etc/systemd/system/worker_service.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2026-11-02 10:25:12 UTC; 15s ago
Main PID: 14890 (python3)
8. Deeper Dive: Understanding the Linux Commands
kill
Sends specified signals to target processes or process groups.
General Syntax
kill [signal_or_option] <pid>
Flags Used in Exercise
-15(-SIGTERM): Request graceful shutdown.-9(-SIGKILL): Immediate kernel-level force termination.
Additional Useful Flags
-l(List signals): Displays all available POSIX signals and their corresponding numbers.
kill -l
-s <SIGNAL_NAME>: Explicitly names the signal (e.g.,kill -s SIGHUP 14285).-1or-- -PGID: Kills an entire Process Group ID rather than a single PID.
kill -9 -- -14285
ps
Displays snapshot information regarding active system processes.
General Syntax
ps [options]
Flags Used in Exercise
aux: BSD style syntax showing user, PID, CPU/MEM usage, process state, and full command-line parameters.
Additional Useful Flags
-ef(Standard UNIX format): Shows full process hierarchy with parent IDs (PPID).
ps -ef
-o <format>(Custom output formatting): Selects specific columns for script parsing.
ps -eo pid,ppid,stat,comm,args
--forest: Renders a visual tree showing parent-child process inheritance.
ps aux --forest
strace
Diagnostic utility for monitoring interactions between userspace applications and the Linux kernel.
General Syntax
strace [options] [-p PID | command]
Useful Troubleshooting Options
-c: Summarize time, calls, and errors for each system call rather than printing raw stream data.
sudo strace -c -p 14285
-e trace=<syscall>: Filters for specific system calls (e.g.,openat,connect,write).
sudo strace -e trace=network,file -p 14285
-f: Follow child threads/processes created viafork()orclone().
9. Exploring Alternative Troubleshooting Methods
1. pkill and killall
Instead of finding the PID manually via ps and running kill, pattern-based tools can streamline signal delivery.
# Terminate matching process by name
pkill -15 -f worker_service.py
# Send signal to all instances
killall -SIGTERM worker_service
- When to Use: Useful when dealing with multiple worker processes or auto-scaling thread pools.
- Trade-off: Risk of accidentally matching unintended processes if the search string is too generic.
2. Inspecting Signals via /proc/[pid]/status Bitmasks
You can inspect the bitmask fields directly to decode signal configurations without third-party diagnostic tools:
| Field | Description |
|---|---|
| SigPnd | Signals pending delivery to the specific thread. |
| ShdPnd | Signals pending delivery to the process as a whole. |
| SigBlk | Signals blocked by thread masks (sigprocmask). |
| SigIgn | Signals explicitly set to SIG_IGN (Ignored). |
| SigCgt | Signals caught by explicit handler functions (SIG_DFL overridden). |
To convert a hex status mask like 0000000000004006 to signal names:
python3 -c '
mask = 0x0000000000004006
for i in range(1, 65):
if (mask >> (i - 1)) & 1:
print(f"Signal {i}")
'
3. Native systemctl kill Command
When managing services under systemd, systemd offers a native signal wrapper:
# Send SIGTERM to all processes in the service control group (cgroup)
sudo systemctl kill -s SIGTERM worker_service.service
# Force kill all processes in the cgroup
sudo systemctl kill -s SIGKILL worker_service.service
- Advantage: Guarantees that spawned child processes and orphaned sub-threads inside the cgroup are killed together, preventing dangling processes.
10. Deeper Linux Concepts and Internals
The Linux Signal Architecture
Signals are standardized, asynchronous notifications delivered by the Linux kernel to a process to inform it of an event. When a signal is generated, the kernel updates the process descriptor (struct task_struct) in kernel space.
+--------------------------------------------------------------------+
| KERNEL |
| +---------------------+ +-------------------------------+ |
| | Signal Generation | -----> | Update task_struct Bitmask | |
| | (Hardware/Software) | | (SigPnd / SigCgt) | |
| +---------------------+ +-------------------------------+ |
+--------------------------------------------------------------------+
|
v (On return to userspace)
+--------------------------------------------------------------------+
| USERSPACE |
| +--------------------------------------------------------------+ |
| | Default Action / Custom Handler Execution / Force Exit | |
| +--------------------------------------------------------------+ |
+--------------------------------------------------------------------+
- Signal Generation: Triggered by hardware exceptions (e.g.,
SIGSEGVon bad memory access), software events (e.g.,SIGPIPEwriting to broken pipe), or user interaction (kill()). - Signal Pending: The signal rests in the target’s pending queue until the process is scheduled to run.
- Signal Delivery: When the CPU transitions the process from kernel mode back to user mode, the kernel checks the pending mask against blocked masks and executes one of four standard actions:
- Terminate: Process terminates immediately (e.g.,
SIGTERM,SIGKILL). - Core Dump: Process terminates and writes a memory image to disk (e.g.,
SIGQUIT,SIGSEGV). - Ignore: Signal is discarded (e.g.,
SIGCHLD). - Stop/Continue: Process execution is paused or resumed (e.g.,
SIGSTOP,SIGCONT).
Key POSIX Signals Reference
| Signal | Code | Standard Action | Can be Caught? | Can be Blocked? | Typical Use Case |
|---|---|---|---|---|---|
| SIGHUP | 1 |
Terminate | Yes | Yes | Terminal disconnect / Reload configuration files |
| SIGINT | 2 |
Terminate | Yes | Yes | Terminal interrupt (Ctrl+C) |
| SIGQUIT | 3 |
Core Dump | Yes | Yes | Terminal quit request (Ctrl+\) |
| SIGKILL | 9 |
Terminate | NO | NO | Force uncatchable immediate process destruction |
| SIGUSR1 | 10 |
Terminate | Yes | Yes | Application-defined custom action |
| SIGUSR2 | 12 |
Terminate | Yes | Yes | Application-defined custom action |
| SIGTERM | 15 |
Terminate | Yes | Yes | Standard request for graceful application exit |
| SIGSTOP | 19 |
Stop | NO | NO | Pause process execution (Job control Ctrl+Z) |
| SIGCONT | 18 |
Continue | Yes | Yes | Resume execution of stopped process |
11. Common Mistakes and Troubleshooting Pitfalls
1. Reaching for SIGKILL (kill -9) Immediately
- Pitfall:
SIGKILLbypasses application cleanup routines entirely. - Impact: Temporary files remain, database connections linger, locks stay held, and shared memory segments (
shm) remain allocated. - Prevention: Always attempt
SIGTERM(15) orSIGINT(2) first. ReserveSIGKILLfor cases where the process is demonstrably non-responsive.
2. Confusing Uninterruptible Sleep (D state) with Deadlocks
- Pitfall: Issuing
kill -9against a process in stateDand expecting it to disappear immediately. - Impact: The process remains in the process table until the underlying hardware/NFS I/O request finishes or fails.
- Prevention: Use
ps auxortopto check the process state column (STAT). If stateDis observed, investigate the storage subsystem, mount points, or kerneldmesglogs.
3. Orphaned Child Processes
- Pitfall: Killing a parent process with
SIGKILLwhile its children are running. - Impact: Child processes become orphans and are re-parented to
init(PID 1) or a systemd user manager, continuing to consume system resources. - Prevention: Use cgroup-aware signal tools (
systemctl kill) or terminate process groups (kill -- -PGID).
12. Quick Reference
Command Quick-Sheet
| Command | Purpose | Useful Flags / Arguments | When to Use |
|---|---|---|---|
kill |
Send signal to PID | -15 (SIGTERM), -9 (SIGKILL), -1 (SIGHUP) |
Targeted process termination |
pkill |
Send signal by pattern | -f (Full command line matching), -u (User) |
Terminating processes by name |
ps |
Inspect processes | aux, -ef, -eo pid,stat,comm |
Checking process states & PIDs |
systemctl kill |
Kill service cgroup | -s SIGTERM, -s SIGKILL |
Service daemon management |
strace |
Trace system calls | -p <PID>, -f (Follow threads), -c (Summary) |
Diagnosing hung or deadlocked processes |
lsof |
List open files | -p <PID>, +D <dir> |
Checking held lockfiles or open sockets |
Key Lessons to Remember
- SIGTERM is a request; SIGKILL is an instruction.
SIGTERMasks the application nicely;SIGKILLtells the kernel to destroy it. - Processes in state
Dcannot be killed by any signal until they exit kernel I/O wait. - Always prefer cgroup-based signal delivery (
systemctl kill) for managed services to avoid leaving orphan processes.
13. Further Learning and Practice
To deepen your understanding of Linux signals and process handling, try the following practical exercises in a test virtual machine:
- Simulate Configuration Reloads:
- Install
nginxorapache2. - Send
SIGHUP(sudo kill -1 <PID>) and observe/var/log/nginx/error.logto watch worker processes re-read configuration without dropping connections.
- Signal Trapping in Bash:
- Write a shell script that traps
SIGTERM(trap 'echo Caught SIGTERM; exit 0' SIGTERM). - Run the script in the background, send signals using
kill, and observe how custom handlers intercept execution.
- Investigating Process State Limits:
- Mount an intentionally unresponsive NFS share or simulate high I/O latency to observe processes entering state
D.