1. Overview
In modern multi-tier web architectures, a single incoming HTTP request often traverses multiple security and application boundaries—from netfilter kernel firewall rules to edge web servers, microservices, and backend databases. When an endpoint fails to respond, pinpointing the exact layer of failure requires a systematic, bottom-up troubleshooting methodology.
This article explores a hands-on Linux sysadmin troubleshooting exercise (“Nerdearla Buenos Aires: Troubleshoot ‘A’ does not connect with ‘B’”). The scenario involves a local HTTP service hosted behind a Caddy web server that fails to respond when queried locally via curl [http://127.0.0.1](http://127.0.0.1).
Key Learning Objectives
Through this post-mortem and educational breakdown, you will learn:
- How to systematically isolate failures across the OSI stack—distinguishing packet drops (firewall Layer 3/4) from socket connection refusals and upstream backend application failures (Layer 7).
- How to inspect network sockets and listening daemons using
ssandsystemctl. - How to identify and modify packet-filtering rules in
iptableswithout disrupting legitimate host traffic. - How reverse proxy architectures (such as Caddy) propagate upstream connection failures back to the client.
- How to inspect and reconfigure PostgreSQL database socket bindings (
postgresql.conf). - Advanced CLI flags, diagnostic tools (
lsof,nftables,tcpdump), and command alternatives to broaden your sysadmin toolkit.
2. Problem Description
Initial Scenario & Environment
The host ip-10-1-13-5 was configured to run a Caddy web server intended to serve requests on HTTP port 80. The web server proxies requests to a local Python backend application listening on port 5050, which in turn connects to a database via an unmodifiable script located at /home/admin/db_connector.py.
Expected vs. Actual Behavior
-
Expected: Executing
curl [http://127.0.0.1](http://127.0.0.1)should successfully route through Caddy to the backend service and output a target verification URL. -
Actual: Running
curl [http://127.0.0.1](http://127.0.0.1)hung indefinitely without returning any response or HTTP headers.
Constraints & Requirements
-
Immutable Backend File: The file
/home/admin/db_connector.pymust not be modified under any circumstances. -
Loopback IP Binding Requirement: The web server must accept connections specifically on IP address
127.0.0.1(IPv4 loopback), not exclusively onlocalhostor IPv6[::1]. -
Success Criteria: The command
curl [http://127.0.0.1](http://127.0.0.1)exits cleanly with status0and outputs a valid URL.
3. Initial Investigation and Troubleshooting Strategy
When diagnosing network service outages on Linux, jumping directly to editing configuration files often leads to incorrect fixes or secondary issues. A structured, evidence-based investigation sequence minimizes downtime and isolates the root cause quickly:
+-------------------------------------------------------------------+
| Step 1: Process & Socket Layer |
| Is the web server process alive and listening on TCP port 80? |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| Step 2: Packet Filtering & Network Firewall Layer |
| Are packets reaching port 80, or are iptables rules dropping them?|
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| Step 3: Web Server & Reverse Proxy Configuration |
| Is Caddy correctly bound to 127.0.0.1 and proxying to port 5050? |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| Step 4: Upstream Backend & Database Dependencies |
| Is the Python backend active, and is PostgreSQL listening on |
| the exact TCP port requested by the application? |
+-------------------------------------------------------------------+
General Sysadmin Diagnostic Rule: Hangs vs. Refusals
- Connection Hangs (Timeout): Typically indicates a Layer 3/4 packet drop. The TCP SYN packet is sent by the client (
curl), but no SYN-ACK or RST packet is returned because a firewall (iptables/nftables) silently drops (DROP) the packet. - Connection Refused (
ECONNREFUSED): Indicates that the network path and firewall are clear, and the target IP is reachable, but no process is listening on the requested destination TCP port (the kernel returns a TCPRSTpacket). - HTTP 50x Errors or Script Exceptions: Indicates that the network path and edge web server are functional, but the upstream backend application failed to process the request.
4. Step-by-Step Troubleshooting
Step 1: Inspect Active Network Sockets and Service Status
Objective
Verify whether the Caddy web server is active and confirm which local IP addresses and TCP ports it is listening on.
Commands
sudo ss -tulpn
sudo systemctl status caddy
Command Breakdown
-
ss: Displays socket statistics directly from kernel space. -
-t: Restricts output to TCP sockets. -
-u: Restricts output to UDP sockets. -
-l: Displays listening sockets only. -
-p: Displays the process ID (PID) and process name owning the socket. -
-n: Prints numeric port numbers instead of resolving service names (prevents slow DNS lookups). -
systemctl status caddy: Queriessystemdto check the execution state, PID, uptime, and recent journal log output for thecaddyservice unit.
Observed Result
admin@ip-10-1-13-5:~$ sudo ss -tulpn
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=655,fd=5))
udp UNCONN 0 0 0.0.0.0:68 0.0.0.0:* users:(("dhclient",pid=415,fd=9))
tcp LISTEN 0 1 0.0.0.0:5050 0.0.0.0:* users:(("python3",pid=619,fd=3))
tcp LISTEN 0 4096 127.0.0.1:2019 0.0.0.0:* users:(("caddy",pid=614,fd=3))
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=642,fd=3))
tcp LISTEN 0 4096 *:6767 *:* users:(("sadagent",pid=613,fd=7))
tcp LISTEN 0 4096 *:8080 *:* users:(("gotty",pid=612,fd=6))
tcp LISTEN 0 4096 *:80 *:* users:(("caddy",pid=614,fd=7))
tcp LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=642,fd=4))
admin@ip-10-1-13-5:~$ sudo systemctl status caddy
● caddy.service - Caddy
Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2026-11-01 14:26:15 UTC; 1min 10s ago
Main PID: 614 (caddy)
Interpretation
- Caddy is active (PID 614) and bound to
*:80(all IPv4 interfaces on port 80) as well as internal management port127.0.0.1:2019. - A Python 3 application (PID 619) is listening on
0.0.0.0:5050. - Since Caddy is listening on port 80, a local connection to
[http://127.0.0.1](http://127.0.0.1)should complete a TCP handshake. The fact thatcurl [http://127.0.0.1](http://127.0.0.1)hangs (^C) strongly suggests traffic is being intercepted before reaching Caddy.
Next Action
Inspect the Caddy configuration (Caddyfile) to verify site block definitions, then test for packet filtering rules.
Key Takeaway
A process listing on *:80 in ss confirms kernel socket binding, but does not guarantee packet delivery if a firewall rule drops incoming SYN packets beforehand.
Step 2: Inspect Caddy Configuration and Initial Binding Attempt
Objective
Examine /etc/caddy/Caddyfile to verify site addresses and reverse proxy parameters.
Commands
cat /etc/caddy/Caddyfile
sudo vim /etc/caddy/Caddyfile
sudo systemctl reload caddy
curl http://127.0.0.1
Command Breakdown
-
cat /etc/caddy/Caddyfile: Displays Caddy’s configuration file. -
sudo systemctl reload caddy: Sends a reload signal (SIGHUP/ administrative API call) to Caddy to apply configuration updates gracefully without dropping established connections.
Observed Result
# Initial /etc/caddy/Caddyfile content
:80 {
reverse_proxy localhost:5050
}
Updating /etc/caddy/Caddyfile to explicitly bind to 127.0.0.1:80:
127.0.0.1:80 {
reverse_proxy localhost:5050
}
Reloading Caddy and re-testing:
admin@ip-10-1-13-5:~$ sudo systemctl reload caddy
admin@ip-10-1-13-5:~$ curl http://127.0.0.1
^C
Interpretation
Even with Caddy explicitly configured for 127.0.0.1:80, curl continues to hang. This confirms that Caddy configuration syntax is not the root cause of the initial connection timeout. The packet drop occurs at the netfilter/firewall layer before Caddy processes the TCP packet.
Next Action
Check host firewall rules using iptables.
Key Takeaway
Do not spend excessive time tweaking application configuration files when connection behavior exhibits classic packet drop characteristics (indefinite hanging).
Step 3: Inspect Firewall Rules and Remove Port 80 Block
Objective
Inspect active IPv4 netfilter packet filtering rules in the INPUT chain and eliminate any rules dropping port 80 traffic.
Commands
sudo iptables -L INPUT -v -n --line-numbers
sudo iptables -D INPUT 1
Command Breakdown
-
iptables: Administration tool for IPv4 packet filtering and NAT. -
-L INPUT: Lists all rules in theINPUTchain (packets destined for local host sockets). -
-v: Verbose output; shows packet and byte counters, interface bindings (in,out), and rule options. -
-n: Numeric output; displays IP addresses and port numbers numerically. -
--line-numbers: Displays line numbers at the beginning of each rule entry, required for targeted deletion. -
sudo iptables -D INPUT 1: Deletes (-D) rule number1from theINPUTchain.
Observed Result
admin@ip-10-1-13-5:~$ sudo iptables -L INPUT -v -n --line-numbers
Chain INPUT (policy ACCEPT 994 packets, 63922 bytes)
num pkts bytes target prot opt in out source destination
1 15 900 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
admin@ip-10-1-13-5:~$ sudo iptables -D INPUT 1
Re-running curl [http://127.0.0.1](http://127.0.0.1):
admin@ip-10-1-13-5:~$ curl http://127.0.0.1
An error occurred: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5433?
Interpretation
- Firewall Unblocked: Removing
iptablesrule 1 immediately resolved the connection timeout! Packets now reach Caddy on port 80. - New Upstream Symptom: The output changed from hanging to returning an explicit database connection error message:
could not connect to server: Connection refused on port 5433. - Flow Breakdown: Caddy received the HTTP request on port 80, proxied it over
localhost:5050to the Python backend, and the Python backend attempted to connect to a PostgreSQL database on TCP port5433. PostgreSQL refused the connection.
Next Action
Investigate PostgreSQL socket bindings to determine why port 5433 is refusing connections.
Key Takeaway
Resolving one troubleshooting layer (firewall) often reveals downstream application errors (database connection refusal). Always re-evaluate symptoms after resolving each layer.
Step 4: Investigate Backend Database Connection and Update PostgreSQL Port
Objective
Verify PostgreSQL socket status, locate postgresql.conf, and reconfigure PostgreSQL to listen on port 5433 to satisfy backend application requirements.
Commands
sudo ss -tulpn | grep postgres
sudo find /etc/postgresql/ -name postgresql.conf
sudo sed -i "s/^port = 5432/port = 5433/" $(sudo find /etc/postgresql/ -name postgresql.conf)
sudo systemctl restart postgresql
Command Breakdown
-
sudo ss -tulpn | grep postgres: Checks if PostgreSQL is running and which port it is bound to. -
sudo find /etc/postgresql/ -name postgresql.conf: Searches/etc/postgresql/recursively for the main cluster configuration file. -
sed -i "s/^port = 5432/port = 5433/" ...: -
sed: Stream editor for filtering and transforming text. -
-i: Modifies the target file in-place. -
"s/^port = 5432/port = 5433/": Substitutes (s) any line starting withport = 5432withport = 5433. -
sudo systemctl restart postgresql: Restarts the PostgreSQL database service to re-bind network sockets to port 5433.
Observed Result
admin@ip-10-1-13-5:~$ sudo ss -tulpn | grep postgres
# (No output returned - PostgreSQL was not currently listening on TCP sockets)
admin@ip-10-1-13-5:~$ sudo find /etc/postgresql/ -name postgresql.conf
/etc/postgresql/13/main/postgresql.conf
admin@ip-10-1-13-5:~$ cat /etc/postgresql/13/main/postgresql.conf | grep "^port"
port = 5432
admin@ip-10-1-13-5:~$ sudo sed -i "s/^port = 5432/port = 5433/" /etc/postgresql/13/main/postgresql.conf
admin@ip-10-1-13-5:~$ sudo systemctl restart postgresql
Interpretation
- The backend script
/home/admin/db_connector.pywas hardcoded to connect to PostgreSQL on port 5433. Per scenario rules,db_connector.pycould not be modified. - PostgreSQL default installation listens on port
5432. - Updating
postgresql.conftoport = 5433and restarting PostgreSQL aligns the database listening port with the backend application’s requirement.
Next Action
Perform final end-to-end verification.
Key Takeaway
When application source code is immutable due to compliance or operational constraints, service infrastructure parameters (such as database configuration ports) must be adjusted to match application expectations.
Step 5: End-to-End Verification
Objective
Confirm that curl [http://127.0.0.1](http://127.0.0.1) successfully returns the expected scenario URL and validate completion using the challenge check script.
Commands
curl http://127.0.0.1
/home/admin/agent/check.sh
Observed Result
admin@ip-10-1-13-5:~$ curl http://127.0.0.1
https://sadservers.com/nerdearla2024
admin@ip-10-1-13-5:~$ /home/admin/agent/check.sh
OK
Interpretation
The HTTP request successfully passed through iptables to Caddy on port 80, proxied to the Python backend on port 5050, connected to PostgreSQL on port 5433, and returned the target URL [https://sadservers.com/nerdearla2024](https://sadservers.com/nerdearla2024).
5. Root Cause Analysis
The overall service failure was caused by two distinct issues across different layers of the infrastructure stack:
CLIENT (curl http://127.0.0.1)
|
| 1. Layer 4 Firewall Block (iptables DROP tcp dpt:80)
v [FIX: sudo iptables -D INPUT 1]
CADDY WEB SERVER (127.0.0.1:80)
|
| 2. Reverse Proxy Pass (localhost:5050)
v
PYTHON BACKEND APP (0.0.0.0:5050)
|
| 3. DB Connection Attempt (127.0.0.1:5433 - Connection Refused)
v [FIX: port = 5433 in postgresql.conf & systemctl restart postgresql]
POSTGRESQL DATABASE (Port 5433)
1. Primary Ingress Block: Netfilter Rule Drop
An explicit iptables rule (DROP tcp dpt:80) was present in the INPUT chain. In Linux, netfilter evaluates INPUT chain rules sequentially for incoming packets destined for local sockets. Because the DROP rule matched TCP destination port 80, the kernel silently discarded incoming SYN packets from curl. No SYN-ACK or RST was generated, causing curl to hang while awaiting a TCP response.
2. Secondary Backend Failure: Database Port Mismatch
Once the firewall rule was removed, Caddy successfully received client requests and proxied them to the Python application on port 5050. However, the Python application depends on /home/admin/db_connector.py, which initiates a TCP socket connection to PostgreSQL on port 5433. Because PostgreSQL was configured with port = 5432 in /etc/postgresql/13/main/postgresql.conf, the kernel issued an ECONNREFUSED (TCP RST) error back to the Python process, resulting in an upstream HTTP 500 error.
6. Solution and Recovery
Summary of Applied Fixes
-
Firewall Rule Removal: Deleted the blocking rule from netfilter:
sudo iptables -D INPUT 1 -
Caddy Site Binding Configuration: Ensured
/etc/caddy/Caddyfileproperly defines the loopback HTTP listener:127.0.0.1:80 { reverse_proxy localhost:5050 }Reloaded Caddy:
sudo systemctl reload caddy -
PostgreSQL Listening Port Reconfiguration: Updated PostgreSQL port configuration from
5432to5433:sudo sed -i "s/^port = 5432/port = 5433/" /etc/postgresql/13/main/postgresql.conf sudo systemctl restart postgresql
7. Verification
To ensure long-term stability and confirm the fix across system reboots, execute the following verification steps:
# 1. Verify firewall rules contain no port 80 drop entries
sudo iptables -L INPUT -v -n | grep dpt:80
# 2. Confirm PostgreSQL is actively listening on TCP port 5433
sudo ss -tulpn | grep 5433
# 3. Test end-to-end HTTP response on loopback IP
curl -sS http://127.0.0.1
# 4. Execute automated scenario check script
/home/admin/agent/check.sh
Expected Verification Output
iptablesoutput: Returns empty (no drop rules for port 80).ssoutput: Displayspostgresbound to127.0.0.1:5433.curloutput: Displays[https://sadservers.com/nerdearla2024](https://sadservers.com/nerdearla2024).check.shoutput: DisplaysOK.
8. Deeper Dive: Understanding the Linux Commands
Command: ss (Socket Statistics)
Purpose
ss is a modern utility used to dump socket statistics. It retrieves information directly from kernel netlink sockets, making it significantly faster than legacy tools like netstat on systems with thousands of active connections.
General Syntax
ss [options] [FILTER]
Flags Used in Exercise
-
-t: Displays TCP sockets. -
-u: Displays UDP sockets. -
-l: Displays listening sockets. -
-p: Displays process names and PIDs. -
-n: Displays numeric addresses and ports.
Other Useful Flags for Sysadmins
-
-s(Summary Statistics): Prints a high-level summary of socket allocations by protocol family. -
Syntax:
ss -s -
When useful: Essential during high-load scenarios to check for socket exhaustion without rendering millions of individual connection lines.
-
-i(Internal TCP Information): Displays detailed TCP metrics (rtt, send/recv window size, retransmissions, congestion windowcwnd). -
Syntax:
ss -ti 'sport = :80' -
When useful: Debugging packet loss, high latency, or TCP performance bottlenecks.
-
state <STATE-NAME>(Filter by TCP State): Filters sockets by TCP state (established,syn-sent,time-wait,close-wait). -
Syntax:
ss -tn state time-wait -
When useful: Identifying connection leaks or ephemeral port exhaustion.
Command: iptables
Purpose
iptables is the user-space administration utility used to configure IPv4 packet filtering rules in the Linux kernel netfilter framework.
General Syntax
iptables [-t table] -ACTION chain [rule-specification]
Flags Used in Exercise
-
-L: Lists rules in the selected chain. -
-v: Enables verbose output (packet/byte counters). -
-n: Employs numeric output for IPs and ports. -
--line-numbers: Prints line numbers alongside rules. -
-D: Deletes a rule by chain and line number.
Other Useful Flags for Sysadmins
-
-S(Print Rules in Save Format): Outputs all active rules formatted exactly asiptables-savesyntax. -
Syntax:
sudo iptables -S INPUT -
When useful: Quick copy-pasting of rules or creating backup shell scripts.
-
-C(Check Existing Rule): Tests whether a specific rule exists in a chain without modifying rules. -
Syntax:
sudo iptables -C INPUT -p tcp --dport 80 -j DROP -
When useful: Automation scripts verifying firewall state before applying changes.
-
-F(Flush Chain): Deletes all rules in the specified chain (or all chains if none specified). -
Syntax:
sudo iptables -F INPUT -
Caution: Flushing rules when default policy is set to
DROPcan immediately lock out SSH sessions!
Command: sed (Stream Editor)
Purpose
sed performs non-interactive text parsing and transformation on data streams or files using regular expressions.
General Syntax
sed [options] 'script' [input-file]
Flags Used in Exercise
-
-i: Edits files in-place. -
s/regexp/replacement/: Substitutes the first occurrence ofregexpwithreplacementon matching lines.
Other Useful Flags for Sysadmins
-
-i.bak(In-place with Backup): Creates a backup copy before modifying the target file. -
Example:
sudo sed -i.bak "s/^port = 5432/port = 5433/" /etc/postgresql/13/main/postgresql.conf -
When useful: Prevents accidental configuration destruction during sysadmin edits.
-
-nwith/p(Print Matching Lines Only): Suppresses default output printing and displays only regex matches (similar togrep). -
Example:
sed -n '/^port/p' /etc/postgresql/13/main/postgresql.conf
9. Explore Alternative Troubleshooting Methods
1. Alternative Socket & Process Diagnostics: lsof and fuser
Instead of using ss -tulpn, you can inspect file descriptors and port bindings using lsof or fuser:
# Find process bound to port 80 using lsof
sudo lsof -i :80
# Find process bound to port 80 using fuser
sudo fuser 80/tcp
- Comparison:
lsof(List Open Files) presents detailed file descriptor information, including working directory and open shared libraries.fuserquickly outputs PIDs tied to sockets or files.ssis faster for high socket counts, whilelsofprovides richer process context.
2. Alternative Firewall Management: nftables and iptables-save
Modern Linux distributions (Debian 11+, Ubuntu 20.04+, RHEL 8+) use nftables as the modern netfilter subsystem replacement for iptables.
# Inspect rules via nftables
sudo nft list ruleset
# Export iptables rules to stdout
sudo iptables-save
- Comparison:
iptables-saveprints all table chains (filter,nat,mangle) in a concise, readable format, making it easier to spot hidden drop rules across multiple chains than runningiptables -Lrepeatedly.
3. Alternative Database Connectivity Testing: pg_isready and psql
Rather than relying on curl to trigger database errors through the web backend, sysadmins can directly test PostgreSQL connection health:
# Check if PostgreSQL is accepting connections on port 5433
pg_isready -h 127.0.0.1 -p 5433
# Attempt CLI login over TCP socket
psql -h 127.0.0.1 -p 5433 -U postgres
- Comparison:
pg_isreadyreturns exit code0if PostgreSQL accepts connections,1if connections are refused, and2for syntax/network errors. It is ideal for automated health check scripts.
10. Deeper Linux Concepts and Internals
1. Packet Flow in Netfilter & iptables
When an IPv4 TCP packet enters the Linux network stack (even over the loopback interface lo), it passes through kernel netfilter hooks before reaching userspace sockets:
Incoming Packet (curl http://127.0.0.1)
|
v
[ PREROUTING Chain ]
|
v
[ Routing Decision ] ----------> (Forwarded to another host?)
|
v (Destined for local host)
[ INPUT Chain ] <--- (DROP rule matched here!)
|
(Passes if ACCEPT)
|
v
[ User Space Socket ] ---> (Caddy PID 614 listening on port 80)
If an INPUT chain rule matches with target DROP, netfilter frees the kernel socket buffer (sk_buff) immediately. No TCP control packets (RST or FIN) are sent to the originator, resulting in client connection timeouts.
2. Socket Binding Mechanics (0.0.0.0 vs 127.0.0.1)
0.0.0.0(INADDR_ANY): Instructs the kernel socket layer to accept incoming connection requests on all available IPv4 network interfaces (loopbacklo, local Etherneteth0, cloud virtual interfacesens5).127.0.0.1(INADDR_LOOPBACK): Restricts socket connection acceptance exclusively to traffic originating from the local loopback interface (lo). External network interfaces will reject attempts to reach this socket.[::](IPv6 Any): Listens across all IPv6 interfaces. Depending on kernel socket options (IPV6_V6ONLY), listening on[::]may also accept IPv4-mapped connections.
11. Common Mistakes and Troubleshooting Pitfalls
- Confusing TCP Timeouts with Connection Refusals:
- Mistake: Assuming application config errors when a connection hangs.
- Correction: If a connection hangs, check
iptables/nftablesfirewall rules first. If a connection returns “Connection Refused”, verify whether the process is running and bound to the expected port.
- Editing Configuration Files Without Verifying Service Reloads:
- Mistake: Updating
postgresql.conforCaddyfilewithout reloading or restarting the system service. - Correction: Always run
systemctl reload <service>orsystemctl restart <service>after editing/etc/configuration files.
- Mistake: Updating
- Flushing
iptablesRules Over Remote SSH Sessions:- Mistake: Running
iptables -Fon a remote server where default chain policies are set toDROP. - Correction: Always check default policy (
iptables -L | grep policy) before flushing rules to avoid instant SSH lockout.
- Mistake: Running
- Modifying Restricted Challenge Files:
- Mistake: Editing
/home/admin/db_connector.pyto change port5433back to5432. - Correction: Always read operational constraints carefully. Adjust underlying infrastructure services to match application constraints.
- Mistake: Editing
12. Quick Reference
Command Summary Sheet
| Command | Purpose | Useful Flags / Arguments | When to Use |
|---|---|---|---|
ss |
Inspect active socket state & ports | -tulpn, -s, -ti |
Verify service port bindings & network traffic. |
iptables |
Manage IPv4 packet filtering rules | -L INPUT -v -n --line-numbers, -D |
Inspect or delete firewall rules blocking traffic. |
systemctl |
Manage systemd services | status, reload, restart |
Reload configs or check daemon runtime state. |
sed |
Edit text configuration files | -i "s/old/new/" |
Automated in-place config file updating. |
find |
Locate system configuration files | -name "postgresql.conf" |
Search standard /etc/ directories for config paths. |
pg_isready |
Test PostgreSQL service readiness | -h <host> -p <port> |
Verify database socket listening status non-interactively. |
Key Lessons to Remember
- Bottom-Up Diagnostics: Check physical/virtual link $\rightarrow$ firewall rules $\rightarrow$ web server socket $\rightarrow$ reverse proxy $\rightarrow$ backend service $\rightarrow$ database.
- Hangs = Firewall Drops; Refusals = Closed Ports: Connection timeouts indicate dropped packets;
Connection Refusedindicates reaching an unassigned port. - Verify Changes Incrementally: Test connectivity (
curl) after fixing each layer to catch downstream issues early.
13. Further Learning and Practice
To deepen your understanding of multi-tier network troubleshooting, practice the following hands-on scenarios in a test VM or lab environment:
- Firewall Drop vs Reject Simulation:
- Practice creating both
DROPandREJECTrules iniptables:sudo iptables -A INPUT -p tcp --dport 8080 -j DROP sudo iptables -A INPUT -p tcp --dport 8081 -j REJECT - Compare
curloutput and latency between port 8080 (timeout) and port 8081 (instant refusal).
- Practice creating both
- Caddy Reverse Proxy Latency & Error Logging:
- Configure Caddy to proxy traffic to an invalid backend port and observe access/error logs using
journalctl -u caddy -f.
- Configure Caddy to proxy traffic to an invalid backend port and observe access/error logs using
- PostgreSQL Socket and Network Hardening:
- Practice updating
listen_addressesandportsettings inpostgresql.confalongsidepg_hba.confaccess rules. Validate connectivity usingpg_isreadyandpsql.
- Practice updating