Domain 10Linux & SREINTERMEDIATE

(Tokyo) Troubleshooting Quiet Drops & Permission Denials: Resolving the SadServers 'Tokyo' Web Server Challenge

A comprehensive deep dive into diagnosing silent network timeouts caused by Netfilter/iptables rules and secondary HTTP 403 access issues stemming from file permission constraints on Linux web servers.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-08
#linux#apache#iptables#networking#security#troubleshooting#sadservers#sre

1. Introduction

In modern Site Reliability Engineering (SRE) and Linux system administration, web server failures rarely stem from a single broken configuration file. Production incidents often present overlapping failure modes across multiple layers of the Linux operating system—ranging from kernel network packet filtering at Layer 3/4 to file permission constraints at Layer 7.

This article documents the complete investigation and resolution of the “Tokyo” scenario from SadServers. The challenge describes a web server serving static content where local HTTP requests fail silently without returning data.

By following this guide, you will learn:

  • How to systematically diagnose socket connection hangs versus connection refusals.
  • The internal workings of Linux Netfilter and iptables packet filtering.
  • How Apache’s privilege-separation model interacts with POSIX file permissions.
  • Advanced diagnostic tools including ss, iptables, stat, lsof, and tcpdump.
  • Practical SRE techniques to detect, remediate, and prevent similar issues in production.

2. Problem Description and Environment

Scenario Specifications

  • Scenario Name: Tokyo
  • Difficulty: Medium
  • Type: Fix
  • Tags: apache, realistic-interviews
  • Target Output: Executing curl 127.0.0.1:80 must return hello sadserver.

Problem Statement

A web server on the system is configured to serve the file /var/www/html/index.html containing the string "hello sadserver". However, when checking the server locally using an HTTP client (curl 127.0.0.1:80), nothing is returned, and the command hangs indefinitely.

Expected Output:
hello sadserver

Actual Behavior:
$ curl http://127.0.0.1:80
^C (Hangs indefinitely until interrupted)

System Environment

  • Operating System: Debian/Ubuntu-based Linux environment.
  • Web Server: Apache HTTP Server (apache2).
  • Document Root: /var/www/html/index.html.
  • Privileges: Root (sudo) access available.

3. Initial Investigation and Hypotheses

When troubleshooting local network service failures, it is vital to distinguish between four distinct failure modes before modifying system state:

                          Client Connection Attempt (curl)
                                         │
                         ┌───────────────┴───────────────┐
                         ▼                               ▼
                 Socket Level                     Application Level
          ┌──────────────┴──────────────┐                 │
          ▼                             ▼                 ▼
   Connection Refused           Connection Hang     HTTP 403 / 500
(No process listening or     (Packets dropped by   (Service answers,
  TCP RST returned)           firewall/Netfilter)  file access denied)
  1. Connection Refused (ECONNREFUSED): The client receives an explicit TCP RST packet. This indicates that no socket is listening on that IP/port combination or a firewall actively resets the attempt.
  2. Connection Hang / Timeout (ETIMEDOUT): The client sends a TCP SYN packet but receives no response whatsoever. The packet is being silently dropped.
  3. Empty Response (0 Bytes): The TCP connection succeeds, but the remote process immediately closes the stream without writing data.
  4. HTTP Status Code Error (e.g., 403 Forbidden): The TCP connection and HTTP request succeed, but the web server returns an application-level error payload.

Hypotheses Matrix

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
H1: Apache is stopped or not running curl returns no content. curl hung instead of returning “Connection refused” immediately. Inspect listening ports with ss -tulpn.
H2: Netfilter / iptables dropping traffic curl hangs indefinitely. None at initial observation. Check firewall rules with iptables -L -n -v.
H3: Apache listening on wrong interface Sockets might bind to external IP or IPv6 only. Local bind configuration needs verification. Inspect socket bindings with ss.
H4: File permissions unreadable /var/www/html/index.html created by root. Does not explain connection timeout at TCP level. Inspect file mode with stat.

4. Troubleshooting Process

Step 1: Initial Client Diagnostic

Objective

Verify the client-side behavior when attempting an HTTP request to 127.0.0.1:80.

Command

curl http://127.0.0.1:80

Command Explanation

  • curl: Transfer data to or from a server using supported protocols (HTTP, HTTPS, FTP, etc.).
  • [http://127.0.0.1:80](http://127.0.0.1:80): Explicitly requests an HTTP connection over IPv4 loopback on port 80.

Output & Analysis

admin@i-0fc157fbad9745c3e:~$ curl http://127.0.0.1:80
^C

The output showed no initial text and hung indefinitely until manually interrupted with Control-C (^C). This confirmed a Connection Timeout symptom, indicating that TCP packets were being silently dropped before reaching the web server process, or the kernel stack was withholding responses.

Decision

Proceed to audit active listening sockets on the host to verify whether a process was bound to port 80.


Step 2: Audit Listening Sockets

Objective

Determine if any process is actively listening on TCP port 80.

Command

ss -tulpn | grep :80

Command Explanation

  • ss: Socket statistics utility used to dump socket information.
  • -t: Display TCP sockets.
  • -u: Display UDP sockets.
  • -l: Display listening sockets only.
  • -p: Show process using socket.
  • -n: Do not resolve service names (show numeric port 80 instead of http).
  • | grep :80: Filter output for lines matching port 80.

Output & Analysis

admin@i-0fc157fbad9745c3e:~$ ss -tulpn | grep :80
tcp   LISTEN 0      4096                  *:8080             *:*    users:(("gotty",pid=765,fd=6))   
tcp   LISTEN 0      511                   *:80               *:*                                     
  • Line 2 shows that a TCP socket is in state LISTEN on *:80 (all IPv4 interfaces, backlog queue 511).
  • Because the command was run without sudo privileges initially, the process name and PID column were blank for port 80.

Decision

A process is indeed listening on port 80. Because curl hangs despite an active listener, the issue is located in the network stack filter (firewall) sitting in front of the application socket.


Step 3: Inspect Filesystem Metadata & Permissions

Objective

Verify the target index file’s existence, size, and permission bits to ensure no secondary application issues exist.

Command

stat /var/www/html/index.html

Command Explanation

  • stat: Displays detailed file or filesystem status, including size, inode, octal permissions, UID, GID, and timestamps.

Output & Analysis

admin@i-0fc157fbad9745c3e:~$ stat /var/www/html/index.html 
  File: /var/www/html/index.html
  Size: 15              Blocks: 8          IO Block: 4096   regular file
Device: 259,1   Inode: 273659      Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-09-04 15:07:21.283280849 +0000
Modify: 2026-09-04 15:07:21.283281326 +0000
Change: 2026-09-04 15:07:21.279372349 +0000
 Birth: 2026-09-04 15:07:20.219361455 +0000
  • Size: 15 bytes (matches "hello sadserver" plus newline).
  • Access Mode: 0600 (-rw-------).
  • Ownership: root (UID 0), root (GID 0).

Critical Finding

File permissions are set to octal 0600, granting read/write access only to root. Apache HTTP Server runs its request worker threads under an unprivileged user (typically www-data). Once network access to port 80 is restored, Apache will receive EACCES (Permission Denied) when reading this file, returning an HTTP 403 Forbidden error.

Decision

Note this permission constraint as a secondary blocker. Inspect firewall rules next to resolve the primary network hang.


Step 4: Audit Kernel Firewall Rules (iptables)

Objective

Inspect the IPv4 packet filtering rules in the kernel to locate any drop or reject rules targeting port 80.

Command

sudo iptables -L -n -v

Command Explanation

  • sudo: Execute command with superuser privileges (required to read kernel firewall structures).
  • iptables: Administration tool for IPv4 packet filtering and NAT.
  • -L: List all rules in all chains.
  • -n: Numeric output (suppresses reverse DNS lookups and port name translations).
  • -v: Verbose output (shows packet/byte counters and interface details).

Output & Analysis

admin@i-0fc157fbad9745c3e:~$ sudo iptables -L -n -v
Chain INPUT (policy ACCEPT 457 packets, 36188 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   13   780 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 365 packets, 386K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Critical Finding

Line 1 of the INPUT chain contains an explicit rule:

  • Target: DROP
  • Protocol: tcp
  • Destination Port: 80 (tcp dpt:80)
  • Counters: 13 packets, 780 bytes dropped.

This rule instructs the Linux kernel Netfilter framework to silently discard all incoming TCP traffic addressed to port 80 on any interface. This directly explains why curl hung indefinitely.


Step 5: Privilege-Escalated Socket Audit

Objective

Re-run ss with superuser privileges to confirm the process holding port 80 open.

Command

sudo ss -tulpn | grep :80

Output & Analysis

tcp   LISTEN 0      511                   *:80               *:*    users:(("apache2",pid=860,fd=4),("apache2",pid=858,fd=4),("apache2",pid=856,fd=4))

sudo privilege reveals that apache2 (PIDs 856, 858, 860) owns file descriptor 4 listening on port 80.


Step 6: Identify Rule Line Numbers for Deletion

Objective

Retrieve the exact rule line number in the INPUT chain to perform a clean deletion.

Command

sudo iptables -L INPUT -n --line-numbers

Output & Analysis

admin@i-0fc157fbad9745c3e:~$ sudo iptables -L INPUT -n --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80

The DROP rule targeting port 80 is index 1 in the INPUT chain.


5. Root Cause Analysis

The failure in this scenario was caused by two cascading issues spanning distinct layers of the operating system stack:

                            Client: curl http://127.0.0.1:80
                                          │
                                          ▼
 [Layer 4: Netfilter] ──► Chain INPUT Rule 1: DROP tcp dpt:80 ──► [Packet Discarded] (Symptom 1: Timeout)
                                          │ (If rule removed)
                                          ▼
 [Layer 7: Apache Server] ──► Worker Process (User: www-data)
                                          │
                                          ▼
 [Layer 1: Filesystem] ──► /var/www/html/index.html (Mode: 0600 root:root) ──► [EACCES Denied] (Symptom 2: HTTP 403)

1. Primary Root Cause: Packet Filtering (Network Layer)

  • Mechanism: The kernel firewall contained an explicit iptables rule (-A INPUT -p tcp --dport 80 -j DROP).
  • Effect: During the TCP 3-way handshake, the client sends a SYN packet. When Netfilter evaluates the INPUT chain, it hits the DROP target. The kernel discards the packet without sending a SYN-ACK or RST back to the client. The client TCP stack retransmits SYN packets until the client application times out.

2. Secondary Root Cause: File Access Control (Application Layer)

  • Mechanism: The file /var/www/html/index.html was provisioned with permissions 0600 (-rw-------) owned by root:root.
  • Effect: Apache runs its master process as root (to bind to privileged port 80), but spawns worker child processes running under the unprivileged user account www-data. When www-data attempts to service an HTTP GET request for index.html, POSIX Discretionary Access Control (DAC) evaluates the “Others” permission bits (---). Access is denied (EACCES), resulting in an HTTP 403 Forbidden error.

6. Solution and Remediation

Remediation Actions

Action 1: Delete the Blocking iptables Rule

Remove rule 1 from the INPUT chain to allow TCP traffic to port 80.

sudo iptables -D INPUT 1
  • Command Breakdown:
  • iptables: Netfilter administration tool.
  • -D INPUT 1: Deletes (-D) rule position 1 from the INPUT chain.

Action 2: Update Index File Permissions

Grant global read permissions to /var/www/html/index.html so the www-data user can read it.

sudo chmod 644 /var/www/html/index.html
  • Command Breakdown:
  • chmod: Change file access permissions.
  • 644: Octal mode setting Owner read/write (6 = rw-), Group read (4 = r--), Others read (4 = r--).

Alternative Remediation Methods

Alternative A: Delete iptables Rule by Specification

Instead of referencing line numbers (which can shift if rules are added/deleted concurrently), delete by exact matching parameters:

sudo iptables -D INPUT -p tcp --dport 80 -j DROP

Alternative B: Change File Ownership to www-data

Keep permissions locked down (0600 or 0640) but assign ownership directly to Apache’s execution group:

sudo chown www-data:www-data /var/www/html/index.html
sudo chmod 640 /var/www/html/index.html

7. Verification and Validation

Execution

Run curl against the local endpoint to confirm successful content delivery.

curl http://127.0.0.1:80

Output

admin@i-0fc157fbad9745c3e:~$ curl http://127.0.0.1:80
hello sadserveradmin@i-0fc157fbad9745c3e:~$ 
  • The output returns immediately with hello sadserver.
  • No trailing newline was present in the target file, appending the prompt directly after the string, confirming raw exact content delivery.

Post-Fix System State Validation

To ensure no lingering firewall rules or access anomalies remain:

  1. Verify Firewall Policy:
sudo iptables -L INPUT -n -v

Expected Result: INPUT chain contains no active DROP rules for destination port 80. 2. Verify File Security Settings:

stat -c "%a %U:%G" /var/www/html/index.html

Expected Result: 644 root:root.


8. Alternative Troubleshooting Approaches

In complex environment deployments, command availability or permissions may prevent direct use of iptables -L. The following alternative diagnostic techniques provide equivalent visibility:

                            Alternative Diagnostic Tools
                                         │
        ┌───────────────────┬────────────┴───────┬───────────────────┐
        ▼                   ▼                    ▼                   ▼
   Packet Capture     System Call Trace   Process File Audit   Modern Firewall
   (tcpdump / tshark)     (strace)             (lsof)             (nftables)

1. Low-Level Packet Capture (tcpdump)

If curl hangs, capture live network interface traffic to observe the TCP handshake:

sudo tcpdump -i lo port 80 -n -vv
  • Interpretation:
  • If you see Flags [S] (SYN) sent by 127.0.0.1 but no returning Flags [S.] (SYN-ACK) or Flags [R] (RST), packets are being dropped by Netfilter before reaching the application socket layer.

2. System Call Tracing (strace)

Trace Apache worker processes to diagnose file access failures in real time:

sudo strace -f -p $(pgrep -d',' apache2) -e trace=openat,access,read
  • Interpretation:
  • Look for system calls returning -1 EACCES (Permission denied) when accessing /var/www/html/index.html.

3. Open Files and Sockets Inspection (lsof)

Verify which process holds open file descriptors for port 80 and the index file:

sudo lsof -i :80
sudo lsof /var/www/html/index.html

4. Modern Netfilter Inspection (nftables)

On modern Linux distributions (Debian 10+, Ubuntu 20.04+, RHEL 8+), iptables often operates as a compatibility layer over nftables:

sudo nft list ruleset

Comparison Matrix of Diagnostic Tools

Tool Subsystem Evaluated Primary Use Case Output Complexity
ss Linux Socket Layer Checking listening ports & process owners. Low
**iptables / nft** Kernel Netfilter Inspecting firewall drop/accept rules. Medium
**stat / ls** VFS / File System Verifying ownership & permission bits. Low
tcpdump Network Interface / PCAP Analyzing packet flags & dropped handshakes. High
strace Kernel System Calls Debugging application file access errors. High

9. Linux Command Deep Dive

1. curl

  • Purpose: Command-line tool for transferring data with URLs using various network protocols.
  • Syntax: curl [options] <URL>

Frequently Used Options

Option Long Option Description
-v --verbose Verbose mode; prints request/response headers and connection steps.
-I --head Fetch HTTP response headers only.
-i --include Include HTTP response headers in the output.
-m <secs> --max-time Set maximum time allowed for the transfer before timing out.
-s --silent Silent mode; suppresses progress meters and error messages.
`-X <GET POST>` --request

Examples

# Debug HTTP connection details and headers
curl -v http://127.0.0.1:80

# Fail fast if connection hangs longer than 3 seconds
curl -m 3 http://127.0.0.1:80

2. ss (Socket Statistics)

  • Purpose: Modern utility to inspect network sockets; supersedes legacy netstat.
  • Syntax: ss [options] [FILTER]

Options Reference

Option Description
-t Show TCP sockets.
-u Show UDP sockets.
-l Display listening sockets only.
-p Show process using socket (requires sudo for all processes).
-n Show numerical address and port numbers.
-a Show both listening and non-listening (established) sockets.
-s Display summary statistics of active sockets.

Practical Examples

# Find all listening TCP/UDP ports with PIDs
sudo ss -tulpn

# Show established TCP connections to port 80
ss -t state established '( dport = :80 or sport = :80 )'

3. iptables

  • Purpose: IPv4 packet filter administration utility for Linux Netfilter.
  • Syntax: iptables [-t table] -ACTION chain [rule-specification]

Chain & Table Overview

  • Tables: filter (default), nat, mangle, raw, security.
  • Chains: INPUT, OUTPUT, FORWARD, PREROUTING, POSTROUTING.

Options Reference

Option Description
-L List rules in selected chain or all chains.
-A Append one or more rules to the end of the selected chain.
-I Insert one or more rules into the specified chain at rule position.
-D Delete one or more rules from specified chain.
-F Flush (delete all rules) in selected chain or all chains.
-n Output IP addresses and port numbers in numerical format.
-v Verbose output showing packet and byte counters.
--line-numbers Print line numbers when listing rules.

Examples

# Append a rule allowing incoming HTTP traffic on port 80
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# Delete rule position 1 in INPUT chain
sudo iptables -D INPUT 1

4. stat

  • Purpose: Displays detailed file or filesystem status.
  • Syntax: stat [options] <filename>

Useful Format Options

  • stat -c "%a": Display access rights in octal (e.g., 644).
  • stat -c "%U:%G": Display owner user name and group name.

10. Underlying Linux Concepts and Architecture

1. Linux Netfilter Architecture & Packet Traversal

Netfilter is a framework inside the Linux kernel that provides packet filtering, network address translation (NAT), and packet mangling.

                   ┌───────────────────────┐
                   │  Incoming IP Packet   │
                   └───────────┬───────────┘
                               │
                               ▼
                   ┌───────────────────────┐
                   │  PREROUTING (mangle)  │
                   └───────────┬───────────┘
                               │
                      Routing Decision
                     /                \
      (Local Destination)         (Forward Packet)
           /                                \
          ▼                                  ▼
┌──────────────────┐               ┌──────────────────┐
│  INPUT Chain     │               │  FORWARD Chain   │
│  (iptables rules)│               │  (iptables rules)│
└─────────┬────────┘               └─────────┬────────┘
          │                                  │
          ▼                                  │
┌──────────────────┐                         │
│  Local Socket    │                         │
│ (e.g., Apache)   │                         │
└──────────────────┘                         │
                                             │
                                             ▼
                                   ┌──────────────────┐
                                   │ POSTROUTING Chain│
                                   └──────────────────┘

When an incoming packet hits the network card interface:

  1. The kernel makes a Routing Decision. If the destination IP matches a local host interface (127.0.0.1), the packet moves to the INPUT chain.
  2. Rules in the INPUT chain are evaluated sequentially from top to bottom.
  3. If a packet matches a rule with target DROP, the kernel frees the packet memory buffer immediately.

2. TCP Handshake Dynamics: DROP vs. REJECT

Understanding kernel response mechanisms is essential for rapid SRE troubleshooting:

  • Target DROP: The kernel discards the packet. No response is sent back. The client application waits for a TCP timeout, causing command hangs (curl ... ^C).
  • Target REJECT: The kernel discards the packet but actively returns a TCP RST (or ICMP Destination Unreachable) packet to the client. The client immediately fails with Connection refused.

3. Apache Process Model & Privilege Separation

Apache uses a multi-process architecture to enhance security:

[System Startup / Root Privilege]
           │
           ▼
 Master Process (PID 856, User: root)  ──► Binds to Port 80 (Privileged <1024)
           │
           ├───────────────┬───────────────┐
           ▼               ▼               ▼
    Worker Child 1   Worker Child 2   Worker Child 3
   (PID 858, www-data) (PID 860, www-data) (...)
           │
           ▼
 Tries to read /var/www/html/index.html (Evaluates POSIX permissions)
  1. Master Process: Starts as root to bind to restricted low-number ports (ports < 1024 require CAP_NET_BIND_SERVICE or root).
  2. Worker Processes: Drop privileges to an unprivileged account defined in apache2.conf (User www-data, Group www-data).
  3. Permission Resolution: When servicing HTTP requests, worker processes read static files using the permissions granted to www-data, not root.

11. Real-World Sysadmin and SRE Applications

1. Monitoring & Detection Strategy

In production environments, silent firewall drops and file permission issues should be caught automatically by multi-layered monitoring:

  • Blackbox Probes (Synthetics): Configure Prometheus blackbox_exporter or Datadog Synthetic Monitoring to probe HTTP endpoints (http://localhost:80/) every 15 seconds. Alert if latency exceeds 2 seconds or status code is non-200.
  • TCP Connection State Metrics: Monitor node_netstat_Tcp_RetransSegs metrics. A sharp spike in retransmissions indicates dropped packets across network paths.

2. Infrastructure as Code (IaC) & Firewall Persistence

Manual changes made via iptables -D are ephemeral and cleared upon reboot. In production, firewall configurations must be persisted:

Debian / Ubuntu Persistence

sudo apt-get install iptables-persistent
sudo netfilter-persistent save

Ansible Declarative Enforcement

Ensure firewall rules and permissions are maintained declaratively using configuration management:

- name: Ensure port 80 is open in iptables
  ansible.builtin.iptables:
    chain: INPUT
    protocol: tcp
    destination_port: "80"
    jump: ACCEPT
    state: present

- name: Ensure correct web document permissions
  ansible.builtin.file:
    path: /var/www/html/index.html
    owner: root
    group: root
    mode: '0644'

3. Incident Runbook Design

When responding to a web server timeout incident:

  1. Verify Layer 3/4: Can the client reach the host port? (nc -zvw3 127.0.0.1 80 or curl -m 5 ...).
  2. Verify Process State: Is the daemon running and bound to socket? (ss -tulpn).
  3. Verify Firewall Layer: Are packet drops occurring? (iptables -L -n -v | grep DROP).
  4. Verify Application Log: Are access denials occurring? (tail -n 50 /var/log/apache2/error.log).

12. Further Learning and Experiments

To deepen your understanding of Linux networking and process security, execute these safe experiments in a test virtual machine or container.

Warning: Do not run these commands in a live production environment!

Experiment 1: Simulating DROP vs. REJECT Behavior

  1. Apply a DROP Rule:
sudo iptables -A INPUT -p tcp --dport 8080 -j DROP
  1. Test connection and measure execution time:
time curl -m 3 http://127.0.0.1:8080

Observation: Command waits for the full 3-second timeout. 3. Replace with a REJECT Rule:

sudo iptables -D INPUT -p tcp --dport 8080 -j DROP
sudo iptables -A INPUT -p tcp --dport 8080 -j REJECT
  1. Test connection again:
time curl http://127.0.0.1:8080

Observation: Command fails instantly with curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused. 5. Clean up:

sudo iptables -D INPUT -p tcp --dport 8080 -j REJECT

Experiment 2: Observing Permission Denied in Web Server Logs

  1. Set index.html permissions to restrictive octal 0600:
sudo chmod 600 /var/www/html/index.html
  1. Request content with curl:
curl -i http://127.0.0.1:80

Observation: HTTP/1.1 status returned is 403 Forbidden. 3. Inspect Apache error logs:

sudo tail -n 5 /var/log/apache2/error.log

Observation: Log records: (13)Permission denied: [client 127.0.0.1:XXXXX] AH00132: file permissions deny server access: /var/www/html/index.html. 4. Restore permissions:

sudo chmod 644 /var/www/html/index.html

13. Key Takeaways

  • Distinguish Connection Failure Symptoms: Immediately check whether a request times out (packet drop), resets (connection refused), or throws an application error (HTTP status code).
  • Network vs. Application Ownership: Low-numbered ports require root or specialized capabilities (CAP_NET_BIND_SERVICE) to bind, but worker processes process content under unprivileged user contexts (www-data).
  • Verbose Flags are Mandatory: Always run diagnostic utilities with verbose and numerical flags (curl -v, iptables -L -n -v, ss -tulpn) to prevent misleading host resolution delays and unhide operational data.
  • Persist Runtime Changes: Runtime modifications to iptables or file permission flags take effect immediately in kernel space, but will revert upon system reboot unless written to persistent configuration files.

14. Quick Troubleshooting Reference / Runbook

                         Quick Web Server Diagnostic Checklist
                                           │
  1. Test HTTP Client Connection ──────────┼──► curl -v -m 5 http://<IP>:<PORT>
                                           │
  2. Audit Listening Sockets & PIDs ───────┼──► sudo ss -tulpn | grep :<PORT>
                                           │
  3. Inspect Firewall Drops ───────────────┼──► sudo iptables -L INPUT -n -v --line-numbers
                                           │
  4. Inspect Document Metadata ────────────┼──► stat /path/to/file
                                           │
  5. Check Web Server Logs ────────────────┼──► sudo tail -f /var/log/apache2/error.log

Emergency Command Summary

# Delete iptables DROP rule by line number
sudo iptables -D INPUT <LINE_NUMBER>

# Fix standard web document permissions
sudo chmod 644 /var/www/html/index.html
sudo chown www-data:www-data /var/www/html/index.html

# Verify socket and process ownership
sudo ss -tulpn | grep :80