Overview
Systematic troubleshooting is the hallmark of effective Linux system administration. When an infrastructure service degrades or experiences transient connection failures, jumping straight to restarts or random configuration changes often masks root causes and creates technical debt.
This technical guide establishes a structured methodology for diagnosing Linux system issues across four core domain objectives:
- Server Characterization & Process Discovery: Identifying listening sockets, open ports, process hierarchies, and ownership.
- Hardware Resource Utilization: Analyzing CPU saturation, memory exhaustion, swap activity, inode exhaustion, and disk I/O bottlenecks.
- Application & Service Deep-Dive: Verifying systemd units, containerized runtime states, and HTTP response characteristics.
- Log & Packet Analysis: Correlating kernel messages, systemd journal logs, TCP socket buffer states, and raw network packet captures (
tcpdump).
Whether investigating a slow web application on port 8000, random TCP reset (RST) packets, or socket queue overflows under load, this guide provides the command breakdown, diagnostic logic, internal kernel mechanics, and practical alternatives required for real-world server administration.
Problem Description
In production environments, web services frequently encounter transient latency spikes or intermittent dropouts under heavy client load. Consider the following baseline scenario:
-
Initial Environment: An application service running inside a Linux environment (or Docker container host) expected to process incoming traffic on TCP port
8000. -
Expected Behavior: Clean HTTP 200 responses with low latency and stable TCP handshakes across continuous client connections.
-
Observed Symptoms:
-
Occasional client-side HTTP request timeouts or
Connection refused/Connection reset by peererrors during traffic spikes. -
System load averages rising without an obvious single rogue process consuming 100% CPU.
-
Intermittent dropped connections where no application-level exception is logged in standard application log output.
-
Environmental Constraints: Production server with mixed containerized and native services where service downtime for intrusive debugging must be minimized.
Initial Investigation and Troubleshooting Strategy
To avoid aimless command execution, troubleshooting should follow a layered workflow—moving from high-level host characterization down to low-level kernel and packet inspection.
┌─────────────────────────────────────────────────────────────────────────────┐
│ LINUX TROUBLESHOOTING FLOW │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌───────────────────────────────┼───────────────────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ CPU │ │ MEMORY │ │ STORAGE │
└──────────────┘ └──────────────┘ └──────────────┘
│ load / uptime │ free -h │ df -h / df -i
│ top / htop │ vmstat 1 │ iostat -xz 1
│ ps aux --sort=-%cpu │ dmesg | grep oom │ du -sh /*
│ │ │
└───────────────────────────────┼───────────────────────────────┘
│
┌─────────────────────────┴─────────────────────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ NETWORK │ │ LOGS & SVCS │
└──────────────┘ └──────────────┘
│ ss -tulnp │ journalctl -xe
│ lsof -i :port │ systemctl status
│ sar -n DEV 1 │ docker logs / stats
│ tcpdump (RST/SYN filters) │
Diagnostic Sequence
- Identify the Process & Sockets: Map the listening port (
:8000) to its Process ID (PID) and owning user. - Assess Hardware Resource Saturation: Verify whether CPU (user vs. system vs. iowait), Memory (RAM vs. Swap), or File System limits (inodes vs. capacity) are exhausted.
- Inspect Application & Container Runtimes: Check process health, container environment variables, and systemd unit statuses.
- Examine System & Kernel Logs: Search
journalctlanddmesgfor Out-Of-Memory (OOM) killer events, segfaults, or driver errors. - Analyze Socket Backlogs & Packet Signatures: Inspect TCP socket listen queues (
Recv-QvsSend-Q) and capture TCP RST or dropped ACK flags usingtcpdump.
Step-by-Step Troubleshooting Walkthrough
Step 1: Socket & Process Identification
Objective
Determine which process is bound to TCP port 8000 and inspect its parentage and resource allocation.
Executed Commands
sudo ss -tulpn | grep :8000
ps auxf | grep -E "8000|app"
Command Breakdown
-
ss: Socket Statistics utility. -
-t: Display TCP sockets. -
-u: Display UDP sockets. -
-l: Show listening sockets only. -
-p: Display process name and PID owning the socket. -
-n: Show numeric port numbers instead of resolving service names (e.g.,8000instead ofirdmi). -
ps: Process Status utility. -
a: Lift BSD-style “only yourself” restriction; displays all user processes. -
u: Select user-oriented output format (includes %CPU, %MEM, USER, PID). -
x: Include processes without a controlling terminal (daemons, background tasks). -
f: Display ASCII process hierarchy tree (parent-child relationships).
Observed Result
tcp LISTEN 0 128 0.0.0.0:8000 0.0.0.0:* users:(("python3",pid=4102,fd=3))
root 4100 0.0 0.1 11204 2840 ? Ss 09:00 0:00 /bin/bash /usr/local/bin/run-app.sh
root 4102 1.2 3.4 245800 35400 ? S 09:00 0:15 \_ python3 /app/server.py --port 8000
Interpretation
The socket output confirms a Python 3 web application (PID 4102) listening on 0.0.0.0:8000. The process hierarchy (ps auxf) reveals that PID 4102 was spawned by a wrapper script run-app.sh (PID 4100). The socket backlog limit (Send-Q) is set to 128.
Next Action
Inspect host hardware resource metrics to determine if PID 4102 or another process is bottlenecked on CPU, RAM, or storage I/O.
Key Takeaway
Always combine ss -tulpn with process tree tools like ps auxf or lsof to trace wrapper scripts back to their master service manager.
Step 2: System Resource Utilization & Storage Checks
Objective
Verify CPU time breakdown, memory consumption, swap paging, disk space, and inode utilization.
Executed Commands
free -h
vmstat 1 5
df -h
df -i
Command Breakdown
free -h: Displays total, used, free, and available system memory in human-readable units (GiB/MiB).vmstat 1 5: Samples virtual memory, process queue states, paging, and CPU statistics 5 times at 1-second intervals.df -h: Shows filesystem disk space usage in human-readable units.df -i: Displays filesystem inode capacity and utilization.
Observed Result
total used free shared buff/cache available
Mem: 7.7Gi 1.2Gi 5.1Gi 12Mi 1.4Gi 6.2Gi
Swap: 2.0Gi 0B 2.0Gi
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 5347800 143200 1468000 0 0 0 12 210 400 3 1 96 0 0
0 0 0 5347800 143200 1468000 0 0 0 0 195 380 2 1 97 0 0
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 40G 8.2G 30G 22% /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 2.5M 120K 2.4M 5% /
Interpretation
- Memory: 6.2 GiB available out of 7.7 GiB. Zero swap usage (
si=0,so=0). RAM exhaustion is ruled out. - CPU:
us(user) is 2-3%,sy(system) is 1%, andid(idle) is ~96%.wa(iowait) is 0%, indicating no disk I/O queue delays. - Storage & Inodes: Disk space is at 22% and inode utilization is at 5%. Space or inode exhaustion is ruled out.
Next Action
Since general host system resources are healthy, shift the investigation to application runtime status, system logs, and network connection queue handling.
Key Takeaway
High availability of RAM or CPU does not mean a service is healthy. Application-level limits or socket backlog queues can saturate long before host hardware limits are reached.
Step 3: Application Health & Service Diagnostics
Objective
Verify application responses locally and inspect container runtime state (if containerized) or systemd unit health.
Executed Commands
curl -iv http://127.0.0.1:8000/health
systemctl status webapp.service
Command Breakdown
-
curl -iv: -
-i: Include HTTP response headers in the output. -
-v: Verbose mode (displays TCP connection steps, IP resolution, and request headers). -
systemctl status: Queries the systemd manager for the runtime status, main PID, memory footprint, and recent log journal entries for a service unit.
Observed Result
* Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
> GET /health HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/7.81.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: BaseHTTP/0.6 Python/3.10.12
< Date: Mon, 03 Nov 2026 00:00:00 GMT
< Content-Type: application/json
<
{"status": "ok"}
● webapp.service - Python Web Application
Loaded: loaded (/etc/systemd/system/webapp.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2026-11-03 09:00:00 UTC; 30min ago
Main PID: 4102 (python3)
Tasks: 1 (limit: 9400)
Memory: 34.5M
CPU: 15.2s
CGroup: /system.slice/webapp.service
├─4100 /bin/bash /usr/local/bin/run-app.sh
└─4102 python3 /app/server.py --port 8000
Interpretation
Local HTTP GET requests succeed with an HTTP 200 status code. The systemd service is active and running. However, Tasks: 1 highlights a potential architectural limitation: the Python application is single-threaded and running a single process worker without concurrency handlers.
Next Action
Check kernel and system logs for connection drops or socket buffer issues under concurrent requests.
Step 4: System Logs & Kernel Diagnostics
Objective
Search system journal logs and kernel ring buffer for hardware errors, resource limits, or network failures.
Executed Commands
journalctl -u webapp.service -n 50 --no-pager
sudo dmesg -T | grep -E -i "oom|killed|failed|drop|tcp"
Command Breakdown
-
journalctl: -
-u webapp.service: Filter log entries belonging strictly towebapp.service. -
-n 50: Restrict output to the last 50 log lines. -
--no-pager: Pipe directly to standard output without invoking an interactive terminal pager likeless. -
dmesg -T: Print the kernel ring buffer with human-readable timestamps (-T).
Observed Result
journalctl showed normal initialization log entries. However, dmesg output revealed TCP backlog warnings during peak access windows:
[Mon Nov 3 09:15:22 2026] TCP: request_sock_TCP: Possible SYN flooding on port 8000. Sending cookies. Check SNMP counters.
Interpretation
The kernel logged a possible SYN flood warning on TCP port 8000. This indicates that the incomplete connection queue (SYN backlog) or complete connection queue (accept queue) filled up faster than the single-threaded Python application could call accept().
Next Action
Perform a deep-dive analysis into TCP listen socket queues and capture packet-level traffic during connection drops.
Step 5: Deep Socket Queue & Packet Analysis
Objective
Measure socket listen queue backlog depth and capture TCP Reset (RST) packets during connection dropouts.
Executed Commands
sudo ss -tulpn -e -m 'sport = :8000'
sudo tcpdump -nn -i any 'tcp port 8000 and (tcp[tcpflags] & tcp-rst != 0)'
Command Breakdown
-
ss -tulpn -e -m: -
-e: Show detailed socket information (UID, inode). -
-m: Display detailed socket memory buffer stats (skmem). -
'sport = :8000': Filter by source port8000. -
tcpdump -nn -i any: -
-nn: Do not resolve hostnames or port names (prevents latency during packet processing). -
-i any: Listen on all network interfaces. -
'tcp port 8000 and (tcp[tcpflags] & tcp-rst != 0)': BPF (Berkeley Packet Filter) expression capturing only TCP packets on port8000with the RST flag set.
Observed Result
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp LISTEN 129 128 0.0.0.0:8000 0.0.0.0:* users:(("python3",pid=4102,fd=3)) skmem:(r0,rb87380,t0,tb16384,f0,w0,o0,bl0,d0)
09:20:15.123456 IP 192.168.1.50.54321 > 172.17.0.2.8000: Flags [R], seq 123456, win 0, length 0
Interpretation
- Queue Saturation: For listening sockets,
Recv-Qrepresents the current number of established connections waiting to beaccept()ed by the application, whileSend-Qrepresents the maximum listen queue limit. Here,Recv-Q(129) exceededSend-Q(128). - TCP Resets: When the accept queue overflows and
net.ipv4.tcp_abort_on_overflowis set to0(default), the kernel drops incoming ACK packets or responds with a TCP RST packet to tear down incoming requests that cannot be queued.
Root Cause Analysis
The transient HTTP dropouts and connection failures on port 8000 were caused by TCP Accept Queue Overflow resulting from an inadequate socket backlog configuration combined with a single-threaded process execution model.
Client Linux Kernel (TCP Layer) Application (Python)
│ │ │
│─── 1. SYN ───────────────────────────>│ │
│ │─── Put in SYN Backlog ─────────────────>│
│<── 2. SYN-ACK ────────────────────────│ │
│ │ │
│─── 3. ACK ───────────────────────────>│ │
│ │─── Move to Accept Queue ────────────────│
│ │ (Recv-Q: 129 / Send-Q: 128) │
│ │ *ACCEPT QUEUE OVERFLOW!* │
│<── 4. TCP RST (Connection Aborted) ───│ │
│ │ │ (Busy executing request)
│ │ │ (Unable to run accept())
Underlying Linux Mechanics
- Listen Backlog Architecture: In Linux, a listening TCP socket uses two queues:
- SYN Queue: Holds connections in
SYN-RECVstate during the 3-way handshake. - Accept Queue: Holds fully established connections (
ESTABLISHED) waiting for the userspace application to execute theaccept()system call.
- Queue Size Constraints: The maximum depth of the accept queue is governed by
min(backlog, net.core.somaxconn).
- The Python application initialized its listening socket with a default backlog of
128. - System-wide kernel parameter
net.core.somaxconndefaults to128or4096depending on kernel defaults.
- Failure Sequence: When traffic spiked, the single-threaded application (
PID 4102) was busy processing existing HTTP requests and could not executeaccept()quickly enough. The Accept Queue reached129items, exceeding the limit of128. The Linux kernel discarded incoming connection attempts and sent TCP RST packets back to clients.
Solution and Recovery
To resolve connection drops under heavy load, the backlog limit must be increased at both the kernel level and the application configuration, and application worker concurrency should be expanded.
Step 1: Tune System Kernel Parameters
Increase the maximum socket listen queue size (somaxconn) and TCP SYN queue limit (tcp_max_syn_backlog).
# Apply dynamic kernel tuning
sudo sysctl -w net.core.somaxconn=4096
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096
# Persist settings across reboots
echo "net.core.somaxconn = 4096" | sudo tee -a /etc/sysctl.d/99-custom-networking.conf
echo "net.ipv4.tcp_max_syn_backlog = 4096" | sudo tee -a /etc/sysctl.d/99-custom-networking.conf
sudo sysctl -p /etc/sysctl.d/99-custom-networking.conf
Step 2: Update Application Backlog & Worker Architecture
If using Gunicorn or Uvicorn to serve Python applications, set explicit worker process counts and socket backlogs:
# Example updated execution command for multi-worker web apps
gunicorn --workers 4 --backlog 2048 --bind 0.0.0.0:8000 app:wsgi_app
Step 3: Restart and Verify Service
sudo systemctl restart webapp.service
Verification
To ensure that socket queue drops and TCP RST aborts are completely resolved:
1. Verify Updated Listen Queue Limit
Execute ss to confirm that Send-Q reflects the higher socket backlog limit:
sudo ss -tulpn 'sport = :8000'
Expected Output:
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp LISTEN 0 4096 0.0.0.0:8000 0.0.0.0:* users:(("gunicorn",pid=8201,fd=5))
Send-Q has increased from 128 to 4096.
2. Verify Packet Capture Under Load
Run tcpdump while executing a multi-threaded HTTP benchmarking tool (e.g., ab or wrk):
# Terminal 1: Watch for TCP RST packets
sudo tcpdump -nn -i any 'tcp port 8000 and (tcp[tcpflags] & tcp-rst != 0)'
# Terminal 2: Generate concurrent HTTP load
ab -n 5000 -c 100 http://127.0.0.1:8000/health
Successful Result: tcpdump yields zero TCP RST packets, and ab completes with 0 failed requests.
Deeper Dive: Understanding the Linux Commands
Command 1: ss (Socket Statistics)
- Purpose: Inspect socket state, queue lengths, memory utilization, and internal TCP metrics directly from kernel memory. Replaces legacy
netstat. - General Syntax:
ss [options] [FILTER]
Flags Used in Exercise
-t: TCP sockets only.-u: UDP sockets only.-l: Display listening sockets.-p: Show process name and PID.-n: Numeric output for addresses and ports.-e: Show detailed socket attributes.-m: Show socket memory allocations (skmem).
Other Useful Diagnostic Flags
-i: Display internal TCP statistics (RTT, congestion control algorithms, window sizes—crucial for debugging network latency).
# Show active TCP connections with RTT and packet retransmission stats
ss -tinp 'sport = :8000'
Key Output Metrics:
-
rtt:<val>/<rttvar>: Round Trip Time and variance in milliseconds. -
retrans:<current>/<total>: Unacknowledged TCP retransmissions. Spikes point to packet loss. -
cwnd:<val>: Congestion window size. Ifcwndcollapses to 1, network paths are experiencing heavy throttling. -
-s: Display a high-level summary of total system sockets by state.
ss -s
Command 2: ps (Process Snapshot)
- Purpose: Provides a static snapshot of active system processes.
- General Syntax:
ps [options]
Flags Used in Exercise
auxf: BSD-style aggregate flag displaying all processes, full user detail, detached background daemons, and an ASCII process tree.
Other Useful Diagnostic Flags
-eo: Custom output column selector.
# Display processes sorted by CPU utilization with custom fields
ps -eo pid,ppid,user,%cpu,%mem,cmd --sort=-%cpu | head -n 10
-C: Filter process list strictly by executable command name.
ps -f -C python3
Command 3: tcpdump (Network Packet Analyzer)
- Purpose: Captures and analyzes network packets matching Berkeley Packet Filter (BPF) expressions on system network interfaces.
- General Syntax:
tcpdump [options] [expression]
Flags Used in Exercise
-nn: Skip IP address and port name resolution.-i any: Listen on all active network interfaces.
Other Useful Diagnostic Flags
-w <filename.pcap>: Write raw packets to a pcap file for offline analysis in Wireshark.-C <file_size_mb> -W <file_count>: Create a rolling ring-buffer packet capture to catch intermittent dropouts without filling disk space.
# Maintain 5 rolling files of 100MB each capturing port 8000
sudo tcpdump -nn -i any tcp port 8000 -C 100 -W 5 -w /tmp/http_dropout.pcap
-A: Display packet payloads in ASCII format (useful for inspecting unencrypted HTTP headers).
# Capture HTTP GET requests in plain text
sudo tcpdump -vv -s 0 -A -i any 'tcp port 8000 and (((ip[20:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0) and tcp[((tcp[12]&0xf0)>>2):4] = 0x47455420'
Alternative Troubleshooting Methods
Here are alternative CLI utilities that an experienced administrator can deploy to investigate similar application and network issues.
1. lsof (List Open Files)
- Syntax:
sudo lsof -i :8000 - Utility: In Linux, everything is a file.
lsofidentifies which processes hold open file descriptors, sockets, or log files. - When Preferable: When
ssshows a socket bound to a port, but you need to know which specific thread or open file path is locked by that process. - Example:
# Find process binding a port
sudo lsof -i TCP:8000
# List all files and sockets opened by PID 4102
sudo lsof -p 4102
2. fuser (File & Socket User Identification)
- Syntax:
sudo fuser 8000/tcp - Utility: Instantly outputs the PID accessing a specific TCP/UDP socket or filesystem mount point.
- When Preferable: When attempting to quickly terminate a hung process holding a socket open.
- Example:
# Identify and send SIGKILL directly to process holding port 8000
sudo fuser -k -9 8000/tcp
3. strace (System Call & Signal Tracer)
- Syntax:
sudo strace -p <PID> -f -e trace=accept,listen,bind,connect - Utility: Traces low-level system call execution between userspace processes and the Linux kernel.
- When Preferable: When an application process is running, but it is unclear whether it is issuing
accept()calls or blocking on internal locks. - Example:
# Trace accept() and read() syscalls on PID 4102 with microsecond timestamps
sudo strace -T -tt -f -e trace=accept4,read -p 4102
Deeper Linux Concepts and Internals
Linux TCP Socket Queue Mechanics
Understanding how Linux manages network socket state transitions in kernel memory is essential for high-throughput service optimization.
┌──────────────────────────────────────────────┐
│ SYN-RECV Queue (Backlog) │
│ Stores half-open 3-way handshakes │
└──────────────────────────────────────────────┘
│
3-Way Handshake
Completes
│
▼
┌──────────────────────────────────────────────┐
│ Accept Queue (Recv-Q) │
│ Stores ESTABLISHED sockets awaiting accept()│
└──────────────────────────────────────────────┘
│
accept() Syscall
Executed
│
▼
┌──────────────────────────────────────────────┐
│ Application Process │
│ (Processes request & returns HTTP) │
└──────────────────────────────────────────────┘
- The Handshake & Queue Transition:
- When a client sends a
SYNpacket, the kernel creates a half-open socket entry in the SYN Backlog Queue and returns aSYN-ACK. - Upon receiving the client’s
ACK, the kernel transitions the connection toESTABLISHEDand moves the socket into the Accept Queue (Recv-Q).
- Accept Queue Overflow Behavior:
- When the Accept Queue fills up (
Recv-Q >= Send-Q), the kernel checks/proc/sys/net/ipv4/tcp_abort_on_overflow. - Default (
0): The kernel ignores incoming ACK packets. The client retransmits ACKs until timeouts occur. This allows the application time to recover and callaccept(). - Flag (
1): The kernel sends explicit TCPRSTpackets to clients, actively resetting connections.
- Buffer Allocation (
skmem):
- Socket buffers are allocated out of kernel memory (
slaballoc/kmalloc). - Memory breakdown strings like
skmem:(r0,rb87380,t0,tb16384)indicate: r: Receive queue bytes used.rb: Max receive buffer space allowed.t: Transmit queue bytes used.tb: Max transmit buffer space allowed.
Common Mistakes and Troubleshooting Pitfalls
- Confusing Free RAM with Available RAM:
- Mistake: Assuming system memory is exhausted because
freereports near-zero underfree. - Reality: Linux uses unallocated RAM for page caching (
buff/cache). Focus on theavailablecolumn infree -h.
- Ignoring Inode Exhaustion (
df -i):
- Mistake: Diagnosing write failure issues by checking only disk space (
df -h). - Reality: Storing millions of small log files or temporary sessions can exhaust filesystem inodes (
df -i = 100%) while leaving gigabytes of free disk space.
- Restricting Socket Inspection to Service Names:
- Mistake: Running
ss -tulwithout-n. - Reality: Reverse port mapping slows down output and obscures custom non-standard service ports.
- Treating Application Crashes as Network Outages:
- Mistake: Blaming network firewalls when clients receive TCP resets.
- Reality: Local kernel socket backlog drops generate TCP RST packets locally without network path intervention.
Quick Reference Guide
Commands Table
| Command | Purpose | Useful Flags / Arguments | When to Use |
|---|---|---|---|
ss |
Inspect socket statistics & queue depth | -tulpn, -e, -m, -i |
Locate open ports, check queue overflow (Recv-Q > Send-Q), trace latency |
ps |
Display process tree & metric breakdown | auxf, -eo pid,%cpu,%mem,cmd |
Trace process parentage, identify CPU/RAM-heavy PIDs |
top / htop |
Real-time process and CPU core monitor | 1, M, P, c |
Live system CPU/RAM monitoring and multi-core thread inspection |
free |
Show memory and swap allocation | -h, -w, -s 2 |
Assess available RAM and active swap paging |
vmstat |
Virtual memory & CPU queue statistics | 1 5, -s, -S M |
Detect process context switching, swap-in/swap-out (si/so), I/O wait |
df |
File system storage & inode utilization | -h, -i, -T |
Inspect disk volume space and inode saturation |
du |
Directory-level disk space consumption | -sh /*, --max-depth=1 |
Locate top-level directories consuming storage space |
iostat / sar |
Block storage & network interface metrics | iostat -xz 1, sar -n DEV 1 |
Identify disk disk queue delays (await) or interface network saturation |
curl |
Web client diagnostic utility | -v, -i, -w "%{http_code}\n" |
Test HTTP service endpoints, response headers, and status codes |
systemctl |
systemd service lifecycle manager | status, cat, show, restart |
Verify service unit health, inspect unit files and properties |
journalctl |
systemd logging journal viewer | -u <service>, -p err, -f |
Tail application service logs, filter errors by unit or time window |
dmesg |
Kernel ring buffer viewer | -T, -l err,crit, -w |
Detect hardware errors, OOM killer invocations, TCP SYN flood warnings |
tcpdump |
Network packet analyzer | -nn -i any, 'tcp port 8000' |
Capture raw network frames, diagnose TCP resets (RST) and handshakes |
lsof |
List open files and process sockets | -i :port, -p <PID> |
Determine process ownership of open ports or locked log files |
Key Takeaways Checklist
- Verify socket queue depths using
ss -tulpn(Recv-Qmust be smaller thanSend-Q). - Check filesystem inodes (
df -i) alongside raw space (df -h). - Monitor
vmstat 1for continuous swap-in (si) or swap-out (so) activity. - Inspect
dmesg -Tfor low-level kernel event warnings or OOM process terminations. - Tune system socket backlog limits (
net.core.somaxconn) when scaling web applications.
Further Learning and Practice Exercises
- Simulate Socket Backlog Drops:
- Write a basic Python socket script that binds to port
9000with a backlog of1(socket.listen(1)). - Do not issue
accept()in the code. - Send multiple concurrent requests using
curloraband observeRecv-Qfilling up inss -tulpn. Capture the resulting TCP RST packets withtcpdump.
- Inode Exhaustion Simulation:
- Create a dummy loopback ext4 filesystem image, mount it, and populate it with millions of empty files until inodes reach 100%. Verify system write errors while
df -hdisplays free capacity.
- **Tracing Syscalls with
strace**:
- Attach
strace -e trace=networkto a live web server process and observe the precise sequence ofsocket(),bind(),listen(), andaccept4()system calls during client traffic.