Troubleshooting a Broken Golang Web Application: A Deep Dive into TLS/PKI, POSIX DAC, and AppArmor MAC
1. Overview
In Linux system administration, service outages are rarely caused by a single isolated bug. Production incidents frequently stem from overlapping failures across multiple subsystem layers—ranging from process management and Public Key Infrastructure (PKI) to file permission semantics and kernel-level Security Modules (LSM).
This technical post-mortem documents a comprehensive hands-on troubleshooting exercise involving a Golang web application serving static content over HTTPS on port 7000.
System & Service Context
-
Affected Service:
webapp.service(a compiled Golang HTTP web server managed bysystemd). -
Network Endpoint:
https://webapp:7000/users.html(listening on TCP port 7000). -
Primary Symptoms: Initial total failure to establish a TCP connection (
Connection refused), followed by persistent HTTP403 Forbiddenerrors even after standard POSIX file permissions were corrected. -
Core Subsystems Involved:
systemdprocess supervisor, OpenSSL / TLS trust chain, POSIX Discretionary Access Control (DAC), and AppArmor Mandatory Access Control (MAC).
Key Learning Objectives
By analyzing this exercise, system administrators will understand:
-
How to diagnose systemd service startup retry loops using
journalctl. -
How file permission restrictions on private keys block TLS socket initialization.
-
How to establish system-wide trust for private Certificate Authorities (CAs) and configure loopback hostname resolution.
-
The critical distinction between file read (
r) permissions and directory traversal (x) permissions in POSIX file systems. -
How Linux Security Modules (LSM) like AppArmor enforce Mandatory Access Control (MAC) independently of standard POSIX permissions.
2. Problem Description
A developer team (Mary and John) attempted to implement security measures on a Golang web application. Following their changes, the application broke completely and stopped responding to incoming client requests.
Expected Behavior
Clients must be able to securely connect to the application over HTTPS using standard command-line tools without bypassing security checks (e.g., executing curl https://webapp:7000/users.html without the -k or --insecure flags) and successfully retrieve static HTML content.
Actual Symptoms & Escalation Sequence
-
Phase 1 (Connection Refused): Executing
curl https://webapp:7000/users.htmlresulted incurl: (7) Failed to connect to webapp port 7000: Connection refused. The web server process was completely down and unable to bind to port 7000. -
Phase 2 (TLS & Hostname Failures): The application process continuously crashed or looped due to unreadable certificate files. Furthermore,
curlcould not establish a trusted TLS connection because the private Root CA was missing from the operating system’s trust store, and the hostnamewebappdid not resolve to a valid loopback IP address. -
Phase 3 (Persistent HTTP 403 Forbidden): After starting the web server and fixing TLS trust, requesting
https://webapp:7000/users.htmlreturned an HTTP403 Forbiddenresponse. Standard file permission modifications (chownandchmod) failed to grant access.
Environmental Constraints
-
Root (
sudo) privileges were available on the target host. -
Strict security validation required full certificate chain verification (no disabling of TLS checks allowed).
3. Initial Investigation and Troubleshooting Strategy
A systematic sysadmin approaches complex outages layer-by-layer, moving from the network edge down to the kernel:
[ Network / Transport Layer ] ────► Is port 7000 open & listening? (ss, lsof)
│
▼
[ Process / Service Layer ] ────► Is the daemon running or crashing? (systemctl, journalctl)
│
▼
[ TLS / Certificate Layer ] ────► Are certificates readable & trusted? (openssl, update-ca-certificates)
│
▼
[ POSIX Permission (DAC) ] ────► Does the service account have directory & file access? (ls, chmod, chown)
│
▼
[ Mandatory Access (MAC) ] ────► Is an LSM profile blocking access? (AppArmor / SELinux)
Diagnostic Plan
-
Inspect Process State: Check if the web service process is running and inspect its service manager logs (
journalctl) to identify why port 7000 is closed. -
Audit File Ownership & Permissions (DAC): Inspect the public keys, private keys, and static asset directories to ensure the application user has appropriate access rights.
-
Validate TLS Certificate Chain & Hostname: Verify certificate Subject Alternative Names (SAN) using
openssl s_clientand update the local CA trust store and/etc/hosts. -
Audit Mandatory Access Control (MAC): If standard POSIX permissions are correct but file access is still denied, inspect active Security Module profiles (AppArmor) in
/etc/apparmor.d/.
4. Step-by-Step Troubleshooting
Step 1: Diagnosing Service Startup Failure & Fixing PKI Key Permissions
Objective
Determine why curl https://webapp:7000/users.html fails with Connection refused and bring the web service online.
Command
sudo journalctl -u webapp -ef
Command Breakdown
-
journalctl: Queries thesystemd-journaldlogging service. -
-u webapp: Filters logs specifically for thewebapp.serviceunit. -
-e: Jumps immediately to the end of the log buffer to show the most recent entries. -
-f: Follows log output in real-time.
Observed Result
Apr 10 01:15:58 ip-10-1-13-82 webapp[3423]: 2024/04/10 01:15:58 open /home/webapp/pki/server.crt: permission denied
Apr 10 01:15:58 ip-10-1-13-82 webapp[3423]: 2024/04/10 01:15:58 open /home/webapp/pki/server.pem: permission denied
Apr 10 01:15:58 ip-10-1-13-82 webapp[3423]: 2024/04/10 01:15:58 can not access certificate/key file. sleeping for 10s and will retry
Interpretation
The process webapp runs under the service user webapp (systemctl show -p User webapp returned User=webapp). However, the TLS certificate (server.crt) and private key (server.pem) located in /home/webapp/pki/ were unreadable by the webapp process due to restricted permissions. The application entered an infinite 10-second retry loop, leaving TCP port 7000 unhandled.
Corrective Action & Commands
Assign ownership of the PKI directory to webapp:webapp and restrict the private key file permissions to 0600 (rw-------):
sudo chown -vR webapp.webapp /home/webapp/pki
sudo chmod -v 0600 /home/webapp/pki/server.*
sudo systemctl restart webapp
Inspecting the logs confirmed the daemon successfully initialized:
Nov 01 15:01:40 ip-10-1-13-82 webapp[2532]: 2026/11/01 15:01:40 Starting webserver, listening on port 7000
Key Takeaway
A daemon failing to read its TLS private key will fail to initialize its HTTPS network listener, presenting as a transport-layer connection refusal rather than an application-layer error.
Step 2: Installing the Private Certificate Authority (CA) into System Trust Store
Objective
Enable client tools (curl) to validate the server’s TLS certificate chain without using the insecure -k flag.
Command
sudo cp -v /home/webapp/pki/CA.crt /usr/local/share/ca-certificates/CA.crt
sudo chmod 0644 /usr/local/share/ca-certificates/CA.crt
sudo update-ca-certificates
Command Breakdown
cp -v: Copies the private Root CA certificate to Debian/Ubuntu’s designated custom CA directory (/usr/local/share/ca-certificates/).chmod 0644: Ensures the custom certificate is world-readable (rw-r--r--).update-ca-certificates: Rebuilds the system-wide bundled certificate store (/etc/ssl/certs/ca-certificates.crt) and updates symbolic links.
Observed Result
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Interpretation
The system trust store now accepts certificates issued by the custom CA.crt. Any local HTTP client relying on OpenSSL or System CA bundles will trust the web application’s identity.
Key Takeaway
Adding custom CA certificates to system anchor paths allows native verification across CLI tools and applications without disabling security flags.
Step 3: SAN Verification and Local Hostname Resolution
Objective
Ensure the hostname requested by curl (webapp) matches the Subject Alternative Name (SAN) configured in the server’s TLS certificate.
Command
echo QUIT | openssl s_client -connect localhost:7000 -showcerts 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName
Command Breakdown
echo QUIT: Sends aQUITsignal to terminate the interactiveopenssl s_clientsession immediately after handshake completion.openssl s_client -connect localhost:7000: Connects to port 7000 and dumps the raw TLS certificate chain.openssl x509 -noout -subject -ext subjectAltName: Parses the certificate data, printing the Subject field and Subject Alternative Name extensions without outputting the encoded certificate block.
Observed Result
subject=CN = webapp
X509v3 Subject Alternative Name: critical
DNS:webapp
Interpretation
The certificate requires connections to address the host specifically as webapp (due to DNS:webapp). Connecting to localhost or 127.0.0.1 will result in a TLS domain mismatch error.
Corrective Action & Commands
Map the domain name webapp to a loopback address in /etc/hosts:
sudo vim /etc/hosts
Add the line:
127.0.10.1 webapp
Key Takeaway
Modern TLS implementations enforce strict SAN matching. Local testing requires /etc/hosts mapping when internal DNS is absent.
Step 4: Correcting POSIX File Permissions (DAC)
Objective
Resolve the HTTP 403 Forbidden error resulting from file permission and ownership mismatches on static assets.
Command
sudo chown -Rv webapp.webapp /home/webapp/static-files
sudo chmod -Rv 0640 /home/webapp/static-files/*
sudo chmod 0755 /home/webapp
sudo chmod 0755 /home/webapp/static-files
Command Breakdown
chown -Rv webapp.webapp: Recursively changes directory and file ownership to userwebappand groupwebapp.chmod -Rv 0640: Sets file permissions torw-r-----(Owner read/write, Group read, Others no access).chmod 0755: Sets directory permissions torwxr-xr-x. Crucially, the execute (x) bit on directories permits directory traversal.
Observed Result
Testing with curl still returned an HTTP 403 Forbidden error:
curl https://webapp:7000/users.html
Forbidden
Interpretation
Even though all POSIX Discretionary Access Control (DAC) requirements were fully met (webapp owned the files, files were readable, and parent directories had +x traversal bits set), access was still denied. This strongly indicated that a Mandatory Access Control (MAC) layer was intercepting system calls.
Key Takeaway
If a file request is rejected despite correct POSIX permissions (0755 directories and 0644/0640 files), an active Linux Security Module (AppArmor or SELinux) is enforcing stricter kernel-level restrictions.
Step 5: Modifying and Reloading the AppArmor MAC Security Profile
Objective
Update the kernel’s AppArmor confinement policy for the webapp binary to grant read access to static web assets.
Command
Inspect the profile file located at /etc/apparmor.d/usr.local.bin.webapp and add the required path rules:
sudo nano /etc/apparmor.d/usr.local.bin.webapp
Add the following rules within the profile definition block:
/home/webapp/static-files/ r,
/home/webapp/static-files/* r,
Reload the modified profile into the kernel:
sudo apparmor_parser -r /etc/apparmor.d/usr.local.bin.webapp
Command Breakdown
apparmor_parser: The utility responsible for compiling and loading AppArmor security profiles into the Linux kernel.-r: Replaces/reloads an existing active profile in kernel space.
Observed Result
curl https://webapp:7000/users.html
<html>
<head> </head>
<body>
<p>From Users Page</p>
</body>
</html>
Interpretation
The HTTP 403 Forbidden error was successfully resolved. AppArmor previously intercepted the open() system call made by /usr/local/bin/webapp when trying to access /home/webapp/static-files/users.html because the path was absent from its access control list. Updating and reloading the profile instructed the kernel to allow the operation.
Key Takeaway
AppArmor rules override traditional POSIX root and user permissions. Any new file path accessed by a confined binary must be explicitly declared in its profile.
5. Root Cause Analysis
The failure of the Golang web application was caused by four distinct, cascading technical issues across different system layers:
┌────────────────────────────────────────────────────────────────────────┐
│ SYSTEM CALL ATTEMPT │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 1. Process Layer (Daemon Retry Loop) │
│ Cause: Restricted permissions (0700/0600 root) on /home/webapp/pki/ │
│ Effect: Daemon couldn't read server.pem -> TCP 7000 Connection Refused │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 2. Transport/TLS Layer (Trust & Resolution Failure) │
│ Cause: CA.crt unimported; missing /etc/hosts mapping for "webapp" │
│ Effect: curl rejected self-signed chain & failed SAN domain match │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 3. POSIX DAC Layer (Directory & File Permissions) │
│ Cause: Owner admin:admin & missing directory search/execute (+x) │
│ Effect: Process blocked at filesystem permission level │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 4. Mandatory Access Control Layer (AppArmor Policy) │
│ Cause: Profile /etc/apparmor.d/usr.local.bin.webapp lacked asset path│
│ Effect: Kernel LSM intercepted open() system call -> HTTP 403 │
└────────────────────────────────────────────────────────────────────────┘
Layer-by-Layer Root Cause Breakdown
- Daemon Startup Failure: Systemd started
webapp.service, but the binary returnedpermission deniedwhile trying to open/home/webapp/pki/server.crtandserver.pem. Because the process userwebappcould not open the private key, it slept continuously, preventing the network stack from opening a listening socket on TCP port 7000. - TLS Trust Infrastructure: The application used a custom private Root CA. Because this CA was not installed in
/usr/local/share/ca-certificates/, standard OpenSSL verification failed. Additionally, the TLS certificate contained aSubjectAlternativeNamerestricting access towebapp. Without a/etc/hostsmapping (127.0.10.1 webapp), name resolution failed to validate the TLS handshake. - Discretionary Access Control (DAC): The static asset directory
/home/webapp/static-fileswas originally owned byadmin:adminwith0750permissions. Thewebappsystem user lacked read access tousers.htmland traversal privileges (+x) on the folder path. - Mandatory Access Control (MAC - AppArmor): Linux Security Modules (LSM) act as an absolute authority after POSIX permissions pass. The binary
/usr/local/bin/webappwas confined by AppArmor profile/etc/apparmor.d/usr.local.bin.webapp. Because/home/webapp/static-files/*was not explicitly listed with read (r) permissions, the kernel denied system call execution, returning403 Forbiddenregardless of file ownership.
6. Solution and Recovery
Comprehensive Remediation Script
To restore the service safely following minimal privilege principles, execute the following commands as root:
#!/usr/bin/env bash
set -euo pipefail
# 1. Configure Hostname Resolution
grep -q "webapp" /etc/hosts || echo '127.0.10.1 webapp' >> /etc/hosts
# 2. Fix PKI Permissions (Minimum Read Privilege)
chown -R webapp:webapp /home/webapp/pki
chmod 0700 /home/webapp/pki
chmod 0600 /home/webapp/pki/server.*
# 3. Install Private CA into Operating System Trust Store
cp /home/webapp/pki/CA.crt /usr/local/share/ca-certificates/CA.crt
chmod 0644 /usr/local/share/ca-certificates/CA.crt
update-ca-certificates
# 4. Fix POSIX Permissions for Static Content
chown -R webapp:webapp /home/webapp/static-files
chmod 0755 /home/webapp
chmod 0755 /home/webapp/static-files
chmod 0640 /home/webapp/static-files/*
# 5. Update AppArmor MAC Profile
cat << 'EOF' > /etc/apparmor.d/usr.local.bin.webapp
#include <tunables/global>
/usr/local/bin/webapp {
#include <abstractions/base>
#include <abstractions/nameservice>
/home/webapp/pki/server.crt r,
/home/webapp/pki/server.pem r,
/home/webapp/static-files/ r,
/home/webapp/static-files/* r,
}
EOF
# 6. Reload AppArmor Profile and Restart Service
apparmor_parser -r /etc/apparmor.d/usr.local.bin.webapp
systemctl restart webapp
7. Verification
Verify complete end-to-end functionality using the following tests:
1. Verify Service & Socket Listener
sudo systemctl status webapp
sudo ss -tulpn | grep 7000
Expected Result: webapp.service is active (running), and ss shows webapp listening on *:7000 or 127.0.0.1:7000.
2. Verify AppArmor Profile Enforcement
sudo aa-status | grep usr.local.bin.webapp
Expected Result: Profile is listed under enforce mode.
3. Verification of Secure Endpoint Content Delivery
curl https://webapp:7000/users.html
Expected Output:
<html>
<head> </head>
<body>
<p>From Users Page</p>
</body>
</html>
Exit Status: 0 (indicating successful TLS validation, name resolution, and HTTP 200 payload delivery).
8. Deeper Dive: Understanding the Linux Commands
journalctl
- Purpose: Queries and displays system logs generated by
systemd-journald. - General Syntax:
journalctl [options] [matches] - Options Used:
-u webapp(unit filter),-e(jump to end),-f(follow live output). - Other Useful Flags:
-p err..emerg: Filters messages by severity priority (err,crit,alert,emerg).--since "1 hour ago": Filters logs within a relative time range.-k: Shows kernel logs only (useful for detecting AppArmor/SELinux denials).
update-ca-certificates
- Purpose: Updates the
/etc/ssl/certsdirectory and/etc/ssl/certs/ca-certificates.crtcomposite file to include custom SSL/TLS certificates. - General Syntax:
update-ca-certificates [options] - Flags Worth Knowing:
-f/--fresh: Wipes existing symlinks and rebuilds the trust store completely from scratch.
openssl s_client
- Purpose: A generic SSL/TLS client tool for establishing raw encrypted connections to remote endpoints to diagnose handshakes, protocols, and certificate chains.
- General Syntax:
openssl s_client -connect host:port [options] - Flags Worth Knowing:
-tls1_3: Forces TLS 1.3 protocol handshake.-CAfile /path/to/ca.crt: Specifies a custom CA file directly during handshake testing.-servername hostname: Sends Server Name Indication (SNI) extension during TLS client hello.
apparmor_parser
- Purpose: Loads, reloads, or unloads AppArmor MAC policy profiles into the Linux Kernel.
- General Syntax:
apparmor_parser [options] <profile_file> - Options Used:
-r(replace/reload existing profile). - Flags Worth Knowing:
-C: Compiles profile into binary format without loading into kernel.-R: Removes/unloads profile from kernel space.-q: Quiet mode; suppresses warnings.
9. Explore Alternative Troubleshooting Methods
| Diagnostic Need | Alternative Utility | Practical Example Command | Scenario Advantage |
|---|---|---|---|
| Detect LSM Denials | dmesg / auditd |
sudo dmesg | grep -i apparmor |
Directly displays kernel-level access denial events (audit: type=1400 DENIED) when HTTP logs are silent. |
| Inspect System Calls | strace |
sudo strace -p <PID> -e trace=openat,access |
Tracks raw kernel system calls of a running binary to see exact paths failing with EACCES. |
| Audit Socket Owners | lsof / fuser |
sudo lsof -i :7000 |
Instantly reveals which binary and user ID own an open network port. |
| AppArmor Status | aa-status |
sudo aa-status |
Summarizes all active, enforced, and complaining AppArmor profiles across running processes. |
10. Deeper Linux Concepts and Internals
Discretionary Access Control (DAC) vs. Mandatory Access Control (MAC)
System Access Request (e.g., open file)
│
▼
┌──────────────────────────────────┐
│ Layer 1: POSIX DAC Evaluation │ ──► Checks UID/GID against rwx permissions bits
└──────────────────────────────────┘
│ Passes
▼
┌──────────────────────────────────┐
│ Layer 2: Kernel LSM (MAC) │ ──► Checks Process Path/Label against AppArmor/SELinux Policy
└──────────────────────────────────┘
│ Passes
▼
Access Granted
- POSIX DAC (Discretionary Access Control): Traditional Linux permission model. Access is determined by ownership (
user:group) and permission flags (rwx). DAC is discretionary because resource owners can modify permissions at will. - Linux Security Modules (LSM / MAC): Mandatory Access Control operates at the Linux kernel level via LSM hooks. AppArmor enforces path-based rules, while SELinux enforces label-based rules. MAC overrides DAC—even if
rootor a file owner attempts access, the kernel denies the system call if the active MAC profile omits the operation.
POSIX Directory Traversal (+x) Semantics
In POSIX file systems, permissions on directories differ fundamentally from regular files:
- Directory Read (
r): Allows listing file names within the directory (e.g.,ls). - Directory Execute (
x): Acts as a search/traversal bit. Allows a process to pass through the directory to access nested subdirectories or files. - Rule: A process cannot read
/home/webapp/static-files/users.htmlunless every parent directory (/,/home,/home/webapp,/home/webapp/static-files) grants+xto that user or process group.
11. Common Mistakes and Troubleshooting Pitfalls
- Over-Granting File Permissions (
chmod 777): Applying777to fix permission errors is a severe security anti-pattern. It makes private TLS keys world-readable, compromising host security. Always use minimum required permissions (0600for keys,0640for static files,0755for directories). - Confusing File Read (
r) with Directory Traversal (x): Giving0644permissions to a file fails if any parent directory lacks the+xexecute bit. - Bypassing Security Flags (
curl -k): Using--insecurehides certificate trust and domain name errors during testing, leaving system problems undetected until production deployment. - Ignoring LSM Security Modules: System administrators often spend hours debugging file permissions without checking
dmesgoraa-statusto see if AppArmor or SELinux is blocking system calls. - Shell Redirection Privileges with
sudo: Executingsudo echo '127.0.10.1 webapp' >> /etc/hostsfails withPermission deniedbecause shell redirection (>>) runs under the unprivileged user’s shell context. Useecho '127.0.10.1 webapp' | sudo tee -a /etc/hostsinstead.
12. Quick Reference
Command Quick Reference Table
| Command | Purpose | Key Flags / Arguments | Practical Use Case |
|---|---|---|---|
journalctl |
Query system logs | -u <unit> -ef |
Debug service crashes and startup loops. |
update-ca-certificates |
Rebuild system CA trust store | -f |
Install private enterprise Root CAs. |
openssl s_client |
Debug TLS handshakes | -connect host:port |
Verify remote SSL certificates and SAN domain names. |
chmod |
Modify POSIX permissions | 0600 (key), 0755 (dir) |
Enforce principle of least privilege. |
chown |
Modify file ownership | -R user:group |
Align resource ownership with service accounts. |
apparmor_parser |
Reload AppArmor profiles | -r <profile> |
Apply updated kernel MAC rules without rebooting. |
dmesg |
Print kernel ring buffer | -T | grep -i apparmor |
Audit LSM kernel security denial events. |
13. Further Learning and Practice
To reinforce the techniques covered in this post-mortem, try these hands-on exercises on a local test system:
- AppArmor Profile Creation: Create a minimal C or Go HTTP web server, generate an AppArmor profile using
aa-autodep, and restrict file system access. Intentionally attempt to access unlisted paths and monitor denial messages indmesg. - PKI & CA Trust Setup: Set up a local private CA using
openssl, issue server certificates with customSubjectAlternativeNamefields, and configure client validation without usingcurl -k. - Least Privilege Auditing: Write a script to audit permissions across web root directories, verifying that no private key files possess world-readable flags or unnecessary execute bits.