Domain 10Linux & SREINTRO

Linux System & Network Troubleshooting: From Process Inspection to Packet Analysis

A practical, sysadmin-level guide to systematically diagnosing CPU, memory, storage, process, network, and transient HTTP/TCP connection issues on Linux servers.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-03
#linux#sysadmin#troubleshooting#networking#systemd#tcpdump#socket-statistics

Overview

Systematic troubleshooting is the hallmark of effective Linux system administration. When an infrastructure service degrades or experiences transient connection failures, jumping straight to restarts or random configuration changes often masks root causes and creates technical debt.

This technical guide establishes a structured methodology for diagnosing Linux system issues across four core domain objectives:

  1. Server Characterization & Process Discovery: Identifying listening sockets, open ports, process hierarchies, and ownership.
  2. Hardware Resource Utilization: Analyzing CPU saturation, memory exhaustion, swap activity, inode exhaustion, and disk I/O bottlenecks.
  3. Application & Service Deep-Dive: Verifying systemd units, containerized runtime states, and HTTP response characteristics.
  4. Log & Packet Analysis: Correlating kernel messages, systemd journal logs, TCP socket buffer states, and raw network packet captures (tcpdump).

Whether investigating a slow web application on port 8000, random TCP reset (RST) packets, or socket queue overflows under load, this guide provides the command breakdown, diagnostic logic, internal kernel mechanics, and practical alternatives required for real-world server administration.


Problem Description

In production environments, web services frequently encounter transient latency spikes or intermittent dropouts under heavy client load. Consider the following baseline scenario:

  • Initial Environment: An application service running inside a Linux environment (or Docker container host) expected to process incoming traffic on TCP port 8000.

  • Expected Behavior: Clean HTTP 200 responses with low latency and stable TCP handshakes across continuous client connections.

  • Observed Symptoms:

  • Occasional client-side HTTP request timeouts or Connection refused / Connection reset by peer errors during traffic spikes.

  • System load averages rising without an obvious single rogue process consuming 100% CPU.

  • Intermittent dropped connections where no application-level exception is logged in standard application log output.

  • Environmental Constraints: Production server with mixed containerized and native services where service downtime for intrusive debugging must be minimized.


Initial Investigation and Troubleshooting Strategy

To avoid aimless command execution, troubleshooting should follow a layered workflow—moving from high-level host characterization down to low-level kernel and packet inspection.

┌─────────────────────────────────────────────────────────────────────────────┐
│                        LINUX TROUBLESHOOTING FLOW                           │
└─────────────────────────────────────────────────────────────────────────────┘
                                       │
       ┌───────────────────────────────┼───────────────────────────────┐
       ▼                               ▼                               ▼
┌──────────────┐                ┌──────────────┐                ┌──────────────┐
│     CPU      │                │    MEMORY    │                │   STORAGE    │
└──────────────┘                └──────────────┘                └──────────────┘
 │ load / uptime                 │ free -h                       │ df -h / df -i
 │ top / htop                    │ vmstat 1                      │ iostat -xz 1
 │ ps aux --sort=-%cpu           │ dmesg | grep oom              │ du -sh /*
 │                               │                               │
 └───────────────────────────────┼───────────────────────────────┘
                                 │
       ┌─────────────────────────┴─────────────────────────┐
       ▼                                                   ▼
┌──────────────┐                                    ┌──────────────┐
│   NETWORK    │                                    │ LOGS & SVCS  │
└──────────────┘                                    └──────────────┘
 │ ss -tulnp                                         │ journalctl -xe
 │ lsof -i :port                                     │ systemctl status
 │ sar -n DEV 1                                      │ docker logs / stats
 │ tcpdump (RST/SYN filters)                         │

Diagnostic Sequence

  1. Identify the Process & Sockets: Map the listening port (:8000) to its Process ID (PID) and owning user.
  2. Assess Hardware Resource Saturation: Verify whether CPU (user vs. system vs. iowait), Memory (RAM vs. Swap), or File System limits (inodes vs. capacity) are exhausted.
  3. Inspect Application & Container Runtimes: Check process health, container environment variables, and systemd unit statuses.
  4. Examine System & Kernel Logs: Search journalctl and dmesg for Out-Of-Memory (OOM) killer events, segfaults, or driver errors.
  5. Analyze Socket Backlogs & Packet Signatures: Inspect TCP socket listen queues (Recv-Q vs Send-Q) and capture TCP RST or dropped ACK flags using tcpdump.

Step-by-Step Troubleshooting Walkthrough

Step 1: Socket & Process Identification

Objective

Determine which process is bound to TCP port 8000 and inspect its parentage and resource allocation.

Executed Commands

sudo ss -tulpn | grep :8000
ps auxf | grep -E "8000|app"

Command Breakdown

  • ss: Socket Statistics utility.

  • -t: Display TCP sockets.

  • -u: Display UDP sockets.

  • -l: Show listening sockets only.

  • -p: Display process name and PID owning the socket.

  • -n: Show numeric port numbers instead of resolving service names (e.g., 8000 instead of irdmi).

  • ps: Process Status utility.

  • a: Lift BSD-style “only yourself” restriction; displays all user processes.

  • u: Select user-oriented output format (includes %CPU, %MEM, USER, PID).

  • x: Include processes without a controlling terminal (daemons, background tasks).

  • f: Display ASCII process hierarchy tree (parent-child relationships).

Observed Result

tcp   LISTEN   0   128   0.0.0.0:8000   0.0.0.0:*   users:(("python3",pid=4102,fd=3))
root        4100  0.0  0.1  11204  2840 ?        Ss   09:00   0:00 /bin/bash /usr/local/bin/run-app.sh
root        4102  1.2  3.4 245800 35400 ?        S    09:00   0:15  \_ python3 /app/server.py --port 8000

Interpretation

The socket output confirms a Python 3 web application (PID 4102) listening on 0.0.0.0:8000. The process hierarchy (ps auxf) reveals that PID 4102 was spawned by a wrapper script run-app.sh (PID 4100). The socket backlog limit (Send-Q) is set to 128.

Next Action

Inspect host hardware resource metrics to determine if PID 4102 or another process is bottlenecked on CPU, RAM, or storage I/O.

Key Takeaway

Always combine ss -tulpn with process tree tools like ps auxf or lsof to trace wrapper scripts back to their master service manager.


Step 2: System Resource Utilization & Storage Checks

Objective

Verify CPU time breakdown, memory consumption, swap paging, disk space, and inode utilization.

Executed Commands

free -h
vmstat 1 5
df -h
df -i

Command Breakdown

  • free -h: Displays total, used, free, and available system memory in human-readable units (GiB/MiB).
  • vmstat 1 5: Samples virtual memory, process queue states, paging, and CPU statistics 5 times at 1-second intervals.
  • df -h: Shows filesystem disk space usage in human-readable units.
  • df -i: Displays filesystem inode capacity and utilization.

Observed Result

               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       1.2Gi       5.1Gi       12Mi       1.4Gi       6.2Gi
Swap:          2.0Gi          0B       2.0Gi
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 5347800 143200 1468000    0    0     0    12  210  400  3  1 96  0  0
 0  0      0 5347800 143200 1468000    0    0     0     0  195  380  2  1 97  0  0
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        40G  8.2G   30G  22% /

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda1        2.5M  120K  2.4M    5% /

Interpretation

  1. Memory: 6.2 GiB available out of 7.7 GiB. Zero swap usage (si=0, so=0). RAM exhaustion is ruled out.
  2. CPU: us (user) is 2-3%, sy (system) is 1%, and id (idle) is ~96%. wa (iowait) is 0%, indicating no disk I/O queue delays.
  3. Storage & Inodes: Disk space is at 22% and inode utilization is at 5%. Space or inode exhaustion is ruled out.

Next Action

Since general host system resources are healthy, shift the investigation to application runtime status, system logs, and network connection queue handling.

Key Takeaway

High availability of RAM or CPU does not mean a service is healthy. Application-level limits or socket backlog queues can saturate long before host hardware limits are reached.


Step 3: Application Health & Service Diagnostics

Objective

Verify application responses locally and inspect container runtime state (if containerized) or systemd unit health.

Executed Commands

curl -iv http://127.0.0.1:8000/health
systemctl status webapp.service

Command Breakdown

  • curl -iv:

  • -i: Include HTTP response headers in the output.

  • -v: Verbose mode (displays TCP connection steps, IP resolution, and request headers).

  • systemctl status: Queries the systemd manager for the runtime status, main PID, memory footprint, and recent log journal entries for a service unit.

Observed Result

*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
> GET /health HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/7.81.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: BaseHTTP/0.6 Python/3.10.12
< Date: Mon, 03 Nov 2026 00:00:00 GMT
< Content-Type: application/json
< 
{"status": "ok"}
● webapp.service - Python Web Application
     Loaded: loaded (/etc/systemd/system/webapp.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-11-03 09:00:00 UTC; 30min ago
   Main PID: 4102 (python3)
      Tasks: 1 (limit: 9400)
     Memory: 34.5M
        CPU: 15.2s
     CGroup: /system.slice/webapp.service
             ├─4100 /bin/bash /usr/local/bin/run-app.sh
             └─4102 python3 /app/server.py --port 8000

Interpretation

Local HTTP GET requests succeed with an HTTP 200 status code. The systemd service is active and running. However, Tasks: 1 highlights a potential architectural limitation: the Python application is single-threaded and running a single process worker without concurrency handlers.

Next Action

Check kernel and system logs for connection drops or socket buffer issues under concurrent requests.


Step 4: System Logs & Kernel Diagnostics

Objective

Search system journal logs and kernel ring buffer for hardware errors, resource limits, or network failures.

Executed Commands

journalctl -u webapp.service -n 50 --no-pager
sudo dmesg -T | grep -E -i "oom|killed|failed|drop|tcp"

Command Breakdown

  • journalctl:

  • -u webapp.service: Filter log entries belonging strictly to webapp.service.

  • -n 50: Restrict output to the last 50 log lines.

  • --no-pager: Pipe directly to standard output without invoking an interactive terminal pager like less.

  • dmesg -T: Print the kernel ring buffer with human-readable timestamps (-T).

Observed Result

journalctl showed normal initialization log entries. However, dmesg output revealed TCP backlog warnings during peak access windows:

[Mon Nov  3 09:15:22 2026] TCP: request_sock_TCP: Possible SYN flooding on port 8000. Sending cookies.  Check SNMP counters.

Interpretation

The kernel logged a possible SYN flood warning on TCP port 8000. This indicates that the incomplete connection queue (SYN backlog) or complete connection queue (accept queue) filled up faster than the single-threaded Python application could call accept().

Next Action

Perform a deep-dive analysis into TCP listen socket queues and capture packet-level traffic during connection drops.


Step 5: Deep Socket Queue & Packet Analysis

Objective

Measure socket listen queue backlog depth and capture TCP Reset (RST) packets during connection dropouts.

Executed Commands

sudo ss -tulpn -e -m 'sport = :8000'
sudo tcpdump -nn -i any 'tcp port 8000 and (tcp[tcpflags] & tcp-rst != 0)'

Command Breakdown

  • ss -tulpn -e -m:

  • -e: Show detailed socket information (UID, inode).

  • -m: Display detailed socket memory buffer stats (skmem).

  • 'sport = :8000': Filter by source port 8000.

  • tcpdump -nn -i any:

  • -nn: Do not resolve hostnames or port names (prevents latency during packet processing).

  • -i any: Listen on all network interfaces.

  • 'tcp port 8000 and (tcp[tcpflags] & tcp-rst != 0)': BPF (Berkeley Packet Filter) expression capturing only TCP packets on port 8000 with the RST flag set.

Observed Result

Netid  State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  
tcp    LISTEN  129     128          0.0.0.0:8000        0.0.0.0:*      users:(("python3",pid=4102,fd=3)) skmem:(r0,rb87380,t0,tb16384,f0,w0,o0,bl0,d0)
09:20:15.123456 IP 192.168.1.50.54321 > 172.17.0.2.8000: Flags [R], seq 123456, win 0, length 0

Interpretation

  1. Queue Saturation: For listening sockets, Recv-Q represents the current number of established connections waiting to be accept()ed by the application, while Send-Q represents the maximum listen queue limit. Here, Recv-Q (129) exceeded Send-Q (128).
  2. TCP Resets: When the accept queue overflows and net.ipv4.tcp_abort_on_overflow is set to 0 (default), the kernel drops incoming ACK packets or responds with a TCP RST packet to tear down incoming requests that cannot be queued.

Root Cause Analysis

The transient HTTP dropouts and connection failures on port 8000 were caused by TCP Accept Queue Overflow resulting from an inadequate socket backlog configuration combined with a single-threaded process execution model.

 Client                          Linux Kernel (TCP Layer)                   Application (Python)
   │                                       │                                         │
   │─── 1. SYN ───────────────────────────>│                                         │
   │                                       │─── Put in SYN Backlog ─────────────────>│
   │<── 2. SYN-ACK ────────────────────────│                                         │
   │                                       │                                         │
   │─── 3. ACK ───────────────────────────>│                                         │
   │                                       │─── Move to Accept Queue ────────────────│
   │                                       │    (Recv-Q: 129 / Send-Q: 128)          │
   │                                       │    *ACCEPT QUEUE OVERFLOW!*             │
   │<── 4. TCP RST (Connection Aborted) ───│                                         │
   │                                       │                                         │ (Busy executing request)
   │                                       │                                         │ (Unable to run accept())

Underlying Linux Mechanics

  1. Listen Backlog Architecture: In Linux, a listening TCP socket uses two queues:
  • SYN Queue: Holds connections in SYN-RECV state during the 3-way handshake.
  • Accept Queue: Holds fully established connections (ESTABLISHED) waiting for the userspace application to execute the accept() system call.
  1. Queue Size Constraints: The maximum depth of the accept queue is governed by min(backlog, net.core.somaxconn).
  • The Python application initialized its listening socket with a default backlog of 128.
  • System-wide kernel parameter net.core.somaxconn defaults to 128 or 4096 depending on kernel defaults.
  1. Failure Sequence: When traffic spiked, the single-threaded application (PID 4102) was busy processing existing HTTP requests and could not execute accept() quickly enough. The Accept Queue reached 129 items, exceeding the limit of 128. The Linux kernel discarded incoming connection attempts and sent TCP RST packets back to clients.

Solution and Recovery

To resolve connection drops under heavy load, the backlog limit must be increased at both the kernel level and the application configuration, and application worker concurrency should be expanded.

Step 1: Tune System Kernel Parameters

Increase the maximum socket listen queue size (somaxconn) and TCP SYN queue limit (tcp_max_syn_backlog).

# Apply dynamic kernel tuning
sudo sysctl -w net.core.somaxconn=4096
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096

# Persist settings across reboots
echo "net.core.somaxconn = 4096" | sudo tee -a /etc/sysctl.d/99-custom-networking.conf
echo "net.ipv4.tcp_max_syn_backlog = 4096" | sudo tee -a /etc/sysctl.d/99-custom-networking.conf
sudo sysctl -p /etc/sysctl.d/99-custom-networking.conf

Step 2: Update Application Backlog & Worker Architecture

If using Gunicorn or Uvicorn to serve Python applications, set explicit worker process counts and socket backlogs:

# Example updated execution command for multi-worker web apps
gunicorn --workers 4 --backlog 2048 --bind 0.0.0.0:8000 app:wsgi_app

Step 3: Restart and Verify Service

sudo systemctl restart webapp.service

Verification

To ensure that socket queue drops and TCP RST aborts are completely resolved:

1. Verify Updated Listen Queue Limit

Execute ss to confirm that Send-Q reflects the higher socket backlog limit:

sudo ss -tulpn 'sport = :8000'

Expected Output:

Netid  State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  
tcp    LISTEN  0       4096         0.0.0.0:8000        0.0.0.0:*      users:(("gunicorn",pid=8201,fd=5))

Send-Q has increased from 128 to 4096.

2. Verify Packet Capture Under Load

Run tcpdump while executing a multi-threaded HTTP benchmarking tool (e.g., ab or wrk):

# Terminal 1: Watch for TCP RST packets
sudo tcpdump -nn -i any 'tcp port 8000 and (tcp[tcpflags] & tcp-rst != 0)'

# Terminal 2: Generate concurrent HTTP load
ab -n 5000 -c 100 http://127.0.0.1:8000/health

Successful Result: tcpdump yields zero TCP RST packets, and ab completes with 0 failed requests.


Deeper Dive: Understanding the Linux Commands

Command 1: ss (Socket Statistics)

  • Purpose: Inspect socket state, queue lengths, memory utilization, and internal TCP metrics directly from kernel memory. Replaces legacy netstat.
  • General Syntax: ss [options] [FILTER]

Flags Used in Exercise

  • -t: TCP sockets only.
  • -u: UDP sockets only.
  • -l: Display listening sockets.
  • -p: Show process name and PID.
  • -n: Numeric output for addresses and ports.
  • -e: Show detailed socket attributes.
  • -m: Show socket memory allocations (skmem).

Other Useful Diagnostic Flags

  • -i: Display internal TCP statistics (RTT, congestion control algorithms, window sizes—crucial for debugging network latency).
# Show active TCP connections with RTT and packet retransmission stats
ss -tinp 'sport = :8000'

Key Output Metrics:

  • rtt:<val>/<rttvar>: Round Trip Time and variance in milliseconds.

  • retrans:<current>/<total>: Unacknowledged TCP retransmissions. Spikes point to packet loss.

  • cwnd:<val>: Congestion window size. If cwnd collapses to 1, network paths are experiencing heavy throttling.

  • -s: Display a high-level summary of total system sockets by state.

ss -s

Command 2: ps (Process Snapshot)

  • Purpose: Provides a static snapshot of active system processes.
  • General Syntax: ps [options]

Flags Used in Exercise

  • auxf: BSD-style aggregate flag displaying all processes, full user detail, detached background daemons, and an ASCII process tree.

Other Useful Diagnostic Flags

  • -eo: Custom output column selector.
# Display processes sorted by CPU utilization with custom fields
ps -eo pid,ppid,user,%cpu,%mem,cmd --sort=-%cpu | head -n 10
  • -C: Filter process list strictly by executable command name.
ps -f -C python3

Command 3: tcpdump (Network Packet Analyzer)

  • Purpose: Captures and analyzes network packets matching Berkeley Packet Filter (BPF) expressions on system network interfaces.
  • General Syntax: tcpdump [options] [expression]

Flags Used in Exercise

  • -nn: Skip IP address and port name resolution.
  • -i any: Listen on all active network interfaces.

Other Useful Diagnostic Flags

  • -w <filename.pcap>: Write raw packets to a pcap file for offline analysis in Wireshark.
  • -C <file_size_mb> -W <file_count>: Create a rolling ring-buffer packet capture to catch intermittent dropouts without filling disk space.
# Maintain 5 rolling files of 100MB each capturing port 8000
sudo tcpdump -nn -i any tcp port 8000 -C 100 -W 5 -w /tmp/http_dropout.pcap
  • -A: Display packet payloads in ASCII format (useful for inspecting unencrypted HTTP headers).
# Capture HTTP GET requests in plain text
sudo tcpdump -vv -s 0 -A -i any 'tcp port 8000 and (((ip[20:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0) and tcp[((tcp[12]&0xf0)>>2):4] = 0x47455420'

Alternative Troubleshooting Methods

Here are alternative CLI utilities that an experienced administrator can deploy to investigate similar application and network issues.

1. lsof (List Open Files)

  • Syntax: sudo lsof -i :8000
  • Utility: In Linux, everything is a file. lsof identifies which processes hold open file descriptors, sockets, or log files.
  • When Preferable: When ss shows a socket bound to a port, but you need to know which specific thread or open file path is locked by that process.
  • Example:
# Find process binding a port
sudo lsof -i TCP:8000

# List all files and sockets opened by PID 4102
sudo lsof -p 4102

2. fuser (File & Socket User Identification)

  • Syntax: sudo fuser 8000/tcp
  • Utility: Instantly outputs the PID accessing a specific TCP/UDP socket or filesystem mount point.
  • When Preferable: When attempting to quickly terminate a hung process holding a socket open.
  • Example:
# Identify and send SIGKILL directly to process holding port 8000
sudo fuser -k -9 8000/tcp

3. strace (System Call & Signal Tracer)

  • Syntax: sudo strace -p <PID> -f -e trace=accept,listen,bind,connect
  • Utility: Traces low-level system call execution between userspace processes and the Linux kernel.
  • When Preferable: When an application process is running, but it is unclear whether it is issuing accept() calls or blocking on internal locks.
  • Example:
# Trace accept() and read() syscalls on PID 4102 with microsecond timestamps
sudo strace -T -tt -f -e trace=accept4,read -p 4102

Deeper Linux Concepts and Internals

Linux TCP Socket Queue Mechanics

Understanding how Linux manages network socket state transitions in kernel memory is essential for high-throughput service optimization.

                  ┌──────────────────────────────────────────────┐
                  │           SYN-RECV Queue (Backlog)           │
                  │   Stores half-open 3-way handshakes          │
                  └──────────────────────────────────────────────┘
                                         │
                                  3-Way Handshake
                                     Completes
                                         │
                                         ▼
                  ┌──────────────────────────────────────────────┐
                  │            Accept Queue (Recv-Q)             │
                  │  Stores ESTABLISHED sockets awaiting accept()│
                  └──────────────────────────────────────────────┘
                                         │
                                   accept() Syscall
                                     Executed
                                         │
                                         ▼
                  ┌──────────────────────────────────────────────┐
                  │              Application Process             │
                  │       (Processes request & returns HTTP)     │
                  └──────────────────────────────────────────────┘
  1. The Handshake & Queue Transition:
  • When a client sends a SYN packet, the kernel creates a half-open socket entry in the SYN Backlog Queue and returns a SYN-ACK.
  • Upon receiving the client’s ACK, the kernel transitions the connection to ESTABLISHED and moves the socket into the Accept Queue (Recv-Q).
  1. Accept Queue Overflow Behavior:
  • When the Accept Queue fills up (Recv-Q >= Send-Q), the kernel checks /proc/sys/net/ipv4/tcp_abort_on_overflow.
  • Default (0): The kernel ignores incoming ACK packets. The client retransmits ACKs until timeouts occur. This allows the application time to recover and call accept().
  • Flag (1): The kernel sends explicit TCP RST packets to clients, actively resetting connections.
  1. Buffer Allocation (skmem):
  • Socket buffers are allocated out of kernel memory (slaballoc/kmalloc).
  • Memory breakdown strings like skmem:(r0,rb87380,t0,tb16384) indicate:
  • r: Receive queue bytes used.
  • rb: Max receive buffer space allowed.
  • t: Transmit queue bytes used.
  • tb: Max transmit buffer space allowed.

Common Mistakes and Troubleshooting Pitfalls

  1. Confusing Free RAM with Available RAM:
  • Mistake: Assuming system memory is exhausted because free reports near-zero under free.
  • Reality: Linux uses unallocated RAM for page caching (buff/cache). Focus on the available column in free -h.
  1. Ignoring Inode Exhaustion (df -i):
  • Mistake: Diagnosing write failure issues by checking only disk space (df -h).
  • Reality: Storing millions of small log files or temporary sessions can exhaust filesystem inodes (df -i = 100%) while leaving gigabytes of free disk space.
  1. Restricting Socket Inspection to Service Names:
  • Mistake: Running ss -tul without -n.
  • Reality: Reverse port mapping slows down output and obscures custom non-standard service ports.
  1. Treating Application Crashes as Network Outages:
  • Mistake: Blaming network firewalls when clients receive TCP resets.
  • Reality: Local kernel socket backlog drops generate TCP RST packets locally without network path intervention.

Quick Reference Guide

Commands Table

Command Purpose Useful Flags / Arguments When to Use
ss Inspect socket statistics & queue depth -tulpn, -e, -m, -i Locate open ports, check queue overflow (Recv-Q > Send-Q), trace latency
ps Display process tree & metric breakdown auxf, -eo pid,%cpu,%mem,cmd Trace process parentage, identify CPU/RAM-heavy PIDs
top / htop Real-time process and CPU core monitor 1, M, P, c Live system CPU/RAM monitoring and multi-core thread inspection
free Show memory and swap allocation -h, -w, -s 2 Assess available RAM and active swap paging
vmstat Virtual memory & CPU queue statistics 1 5, -s, -S M Detect process context switching, swap-in/swap-out (si/so), I/O wait
df File system storage & inode utilization -h, -i, -T Inspect disk volume space and inode saturation
du Directory-level disk space consumption -sh /*, --max-depth=1 Locate top-level directories consuming storage space
iostat / sar Block storage & network interface metrics iostat -xz 1, sar -n DEV 1 Identify disk disk queue delays (await) or interface network saturation
curl Web client diagnostic utility -v, -i, -w "%{http_code}\n" Test HTTP service endpoints, response headers, and status codes
systemctl systemd service lifecycle manager status, cat, show, restart Verify service unit health, inspect unit files and properties
journalctl systemd logging journal viewer -u <service>, -p err, -f Tail application service logs, filter errors by unit or time window
dmesg Kernel ring buffer viewer -T, -l err,crit, -w Detect hardware errors, OOM killer invocations, TCP SYN flood warnings
tcpdump Network packet analyzer -nn -i any, 'tcp port 8000' Capture raw network frames, diagnose TCP resets (RST) and handshakes
lsof List open files and process sockets -i :port, -p <PID> Determine process ownership of open ports or locked log files

Key Takeaways Checklist

  • Verify socket queue depths using ss -tulpn (Recv-Q must be smaller than Send-Q).
  • Check filesystem inodes (df -i) alongside raw space (df -h).
  • Monitor vmstat 1 for continuous swap-in (si) or swap-out (so) activity.
  • Inspect dmesg -T for low-level kernel event warnings or OOM process terminations.
  • Tune system socket backlog limits (net.core.somaxconn) when scaling web applications.

Further Learning and Practice Exercises

  1. Simulate Socket Backlog Drops:
  • Write a basic Python socket script that binds to port 9000 with a backlog of 1 (socket.listen(1)).
  • Do not issue accept() in the code.
  • Send multiple concurrent requests using curl or ab and observe Recv-Q filling up in ss -tulpn. Capture the resulting TCP RST packets with tcpdump.
  1. Inode Exhaustion Simulation:
  • Create a dummy loopback ext4 filesystem image, mount it, and populate it with millions of empty files until inodes reach 100%. Verify system write errors while df -h displays free capacity.
  1. **Tracing Syscalls with strace**:
  • Attach strace -e trace=network to a live web server process and observe the precise sequence of socket(), bind(), listen(), and accept4() system calls during client traffic.