Domain 9Security & PKIINTRO

(Buenos Aires) Troubleshooting a Broken Golang Web Application: A Deep Dive into TLS/PKI, POSIX DAC, and AppArmor MAC

A comprehensive post-mortem and technical guide on resolving a multi-layered Linux web application failure involving TLS certificate permissions, private CA trust, loopback DNS resolution, POSIX permissions, and AppArmor security profiles.

Domain: Domain 9: Security & PKI•Updated: 2026-11-02
#linux#troubleshooting#apparmor#systemd#tls-pki#posix-permissions#golang#networking

Troubleshooting a Broken Golang Web Application: A Deep Dive into TLS/PKI, POSIX DAC, and AppArmor MAC

1. Overview

In Linux system administration, service outages are rarely caused by a single isolated bug. Production incidents frequently stem from overlapping failures across multiple subsystem layers—ranging from process management and Public Key Infrastructure (PKI) to file permission semantics and kernel-level Security Modules (LSM).

This technical post-mortem documents a comprehensive hands-on troubleshooting exercise involving a Golang web application serving static content over HTTPS on port 7000.

System & Service Context

  • Affected Service: webapp.service (a compiled Golang HTTP web server managed by systemd).

  • Network Endpoint: https://webapp:7000/users.html (listening on TCP port 7000).

  • Primary Symptoms: Initial total failure to establish a TCP connection (Connection refused), followed by persistent HTTP 403 Forbidden errors even after standard POSIX file permissions were corrected.

  • Core Subsystems Involved: systemd process supervisor, OpenSSL / TLS trust chain, POSIX Discretionary Access Control (DAC), and AppArmor Mandatory Access Control (MAC).

Key Learning Objectives

By analyzing this exercise, system administrators will understand:

  1. How to diagnose systemd service startup retry loops using journalctl.

  2. How file permission restrictions on private keys block TLS socket initialization.

  3. How to establish system-wide trust for private Certificate Authorities (CAs) and configure loopback hostname resolution.

  4. The critical distinction between file read (r) permissions and directory traversal (x) permissions in POSIX file systems.

  5. How Linux Security Modules (LSM) like AppArmor enforce Mandatory Access Control (MAC) independently of standard POSIX permissions.


2. Problem Description

A developer team (Mary and John) attempted to implement security measures on a Golang web application. Following their changes, the application broke completely and stopped responding to incoming client requests.

Expected Behavior

Clients must be able to securely connect to the application over HTTPS using standard command-line tools without bypassing security checks (e.g., executing curl https://webapp:7000/users.html without the -k or --insecure flags) and successfully retrieve static HTML content.

Actual Symptoms & Escalation Sequence

  1. Phase 1 (Connection Refused): Executing curl https://webapp:7000/users.html resulted in curl: (7) Failed to connect to webapp port 7000: Connection refused. The web server process was completely down and unable to bind to port 7000.

  2. Phase 2 (TLS & Hostname Failures): The application process continuously crashed or looped due to unreadable certificate files. Furthermore, curl could not establish a trusted TLS connection because the private Root CA was missing from the operating system’s trust store, and the hostname webapp did not resolve to a valid loopback IP address.

  3. Phase 3 (Persistent HTTP 403 Forbidden): After starting the web server and fixing TLS trust, requesting https://webapp:7000/users.html returned an HTTP 403 Forbidden response. Standard file permission modifications (chown and chmod) failed to grant access.

Environmental Constraints

  • Root (sudo) privileges were available on the target host.

  • Strict security validation required full certificate chain verification (no disabling of TLS checks allowed).


3. Initial Investigation and Troubleshooting Strategy

A systematic sysadmin approaches complex outages layer-by-layer, moving from the network edge down to the kernel:

  [ Network / Transport Layer ] ────► Is port 7000 open & listening? (ss, lsof)
               │
               ▼
  [ Process / Service Layer ]   ────► Is the daemon running or crashing? (systemctl, journalctl)
               │
               ▼
  [ TLS / Certificate Layer ]   ────► Are certificates readable & trusted? (openssl, update-ca-certificates)
               │
               ▼
  [ POSIX Permission (DAC) ]    ────► Does the service account have directory & file access? (ls, chmod, chown)
               │
               ▼
  [ Mandatory Access (MAC) ]    ────► Is an LSM profile blocking access? (AppArmor / SELinux)

Diagnostic Plan

  1. Inspect Process State: Check if the web service process is running and inspect its service manager logs (journalctl) to identify why port 7000 is closed.

  2. Audit File Ownership & Permissions (DAC): Inspect the public keys, private keys, and static asset directories to ensure the application user has appropriate access rights.

  3. Validate TLS Certificate Chain & Hostname: Verify certificate Subject Alternative Names (SAN) using openssl s_client and update the local CA trust store and /etc/hosts.

  4. Audit Mandatory Access Control (MAC): If standard POSIX permissions are correct but file access is still denied, inspect active Security Module profiles (AppArmor) in /etc/apparmor.d/.


4. Step-by-Step Troubleshooting

Step 1: Diagnosing Service Startup Failure & Fixing PKI Key Permissions

Objective

Determine why curl https://webapp:7000/users.html fails with Connection refused and bring the web service online.

Command

sudo journalctl -u webapp -ef

Command Breakdown

  • journalctl: Queries the systemd-journald logging service.

  • -u webapp: Filters logs specifically for the webapp.service unit.

  • -e: Jumps immediately to the end of the log buffer to show the most recent entries.

  • -f: Follows log output in real-time.

Observed Result

Apr 10 01:15:58 ip-10-1-13-82 webapp[3423]: 2024/04/10 01:15:58 open /home/webapp/pki/server.crt: permission denied
Apr 10 01:15:58 ip-10-1-13-82 webapp[3423]: 2024/04/10 01:15:58 open /home/webapp/pki/server.pem: permission denied
Apr 10 01:15:58 ip-10-1-13-82 webapp[3423]: 2024/04/10 01:15:58 can not access certificate/key file. sleeping for 10s and will retry

Interpretation

The process webapp runs under the service user webapp (systemctl show -p User webapp returned User=webapp). However, the TLS certificate (server.crt) and private key (server.pem) located in /home/webapp/pki/ were unreadable by the webapp process due to restricted permissions. The application entered an infinite 10-second retry loop, leaving TCP port 7000 unhandled.

Corrective Action & Commands

Assign ownership of the PKI directory to webapp:webapp and restrict the private key file permissions to 0600 (rw-------):

sudo chown -vR webapp.webapp /home/webapp/pki
sudo chmod -v 0600 /home/webapp/pki/server.*
sudo systemctl restart webapp

Inspecting the logs confirmed the daemon successfully initialized:

Nov 01 15:01:40 ip-10-1-13-82 webapp[2532]: 2026/11/01 15:01:40 Starting webserver, listening on port 7000

Key Takeaway

A daemon failing to read its TLS private key will fail to initialize its HTTPS network listener, presenting as a transport-layer connection refusal rather than an application-layer error.


Step 2: Installing the Private Certificate Authority (CA) into System Trust Store

Objective

Enable client tools (curl) to validate the server’s TLS certificate chain without using the insecure -k flag.

Command

sudo cp -v /home/webapp/pki/CA.crt /usr/local/share/ca-certificates/CA.crt
sudo chmod 0644 /usr/local/share/ca-certificates/CA.crt
sudo update-ca-certificates

Command Breakdown

  • cp -v: Copies the private Root CA certificate to Debian/Ubuntu’s designated custom CA directory (/usr/local/share/ca-certificates/).
  • chmod 0644: Ensures the custom certificate is world-readable (rw-r--r--).
  • update-ca-certificates: Rebuilds the system-wide bundled certificate store (/etc/ssl/certs/ca-certificates.crt) and updates symbolic links.

Observed Result

Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

Interpretation

The system trust store now accepts certificates issued by the custom CA.crt. Any local HTTP client relying on OpenSSL or System CA bundles will trust the web application’s identity.

Key Takeaway

Adding custom CA certificates to system anchor paths allows native verification across CLI tools and applications without disabling security flags.


Step 3: SAN Verification and Local Hostname Resolution

Objective

Ensure the hostname requested by curl (webapp) matches the Subject Alternative Name (SAN) configured in the server’s TLS certificate.

Command

echo QUIT | openssl s_client -connect localhost:7000 -showcerts 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName

Command Breakdown

  • echo QUIT: Sends a QUIT signal to terminate the interactive openssl s_client session immediately after handshake completion.
  • openssl s_client -connect localhost:7000: Connects to port 7000 and dumps the raw TLS certificate chain.
  • openssl x509 -noout -subject -ext subjectAltName: Parses the certificate data, printing the Subject field and Subject Alternative Name extensions without outputting the encoded certificate block.

Observed Result

subject=CN = webapp
X509v3 Subject Alternative Name: critical
DNS:webapp

Interpretation

The certificate requires connections to address the host specifically as webapp (due to DNS:webapp). Connecting to localhost or 127.0.0.1 will result in a TLS domain mismatch error.

Corrective Action & Commands

Map the domain name webapp to a loopback address in /etc/hosts:

sudo vim /etc/hosts

Add the line:

127.0.10.1 webapp

Key Takeaway

Modern TLS implementations enforce strict SAN matching. Local testing requires /etc/hosts mapping when internal DNS is absent.


Step 4: Correcting POSIX File Permissions (DAC)

Objective

Resolve the HTTP 403 Forbidden error resulting from file permission and ownership mismatches on static assets.

Command

sudo chown -Rv webapp.webapp /home/webapp/static-files
sudo chmod -Rv 0640 /home/webapp/static-files/*
sudo chmod 0755 /home/webapp
sudo chmod 0755 /home/webapp/static-files

Command Breakdown

  • chown -Rv webapp.webapp: Recursively changes directory and file ownership to user webapp and group webapp.
  • chmod -Rv 0640: Sets file permissions to rw-r----- (Owner read/write, Group read, Others no access).
  • chmod 0755: Sets directory permissions to rwxr-xr-x. Crucially, the execute (x) bit on directories permits directory traversal.

Observed Result

Testing with curl still returned an HTTP 403 Forbidden error:

curl https://webapp:7000/users.html
Forbidden

Interpretation

Even though all POSIX Discretionary Access Control (DAC) requirements were fully met (webapp owned the files, files were readable, and parent directories had +x traversal bits set), access was still denied. This strongly indicated that a Mandatory Access Control (MAC) layer was intercepting system calls.

Key Takeaway

If a file request is rejected despite correct POSIX permissions (0755 directories and 0644/0640 files), an active Linux Security Module (AppArmor or SELinux) is enforcing stricter kernel-level restrictions.


Step 5: Modifying and Reloading the AppArmor MAC Security Profile

Objective

Update the kernel’s AppArmor confinement policy for the webapp binary to grant read access to static web assets.

Command

Inspect the profile file located at /etc/apparmor.d/usr.local.bin.webapp and add the required path rules:

sudo nano /etc/apparmor.d/usr.local.bin.webapp

Add the following rules within the profile definition block:

  /home/webapp/static-files/ r,
  /home/webapp/static-files/* r,

Reload the modified profile into the kernel:

sudo apparmor_parser -r /etc/apparmor.d/usr.local.bin.webapp

Command Breakdown

  • apparmor_parser: The utility responsible for compiling and loading AppArmor security profiles into the Linux kernel.
  • -r: Replaces/reloads an existing active profile in kernel space.

Observed Result

curl https://webapp:7000/users.html
<html>
  <head> </head>
  <body>
    <p>From Users Page</p>
  </body>
</html>

Interpretation

The HTTP 403 Forbidden error was successfully resolved. AppArmor previously intercepted the open() system call made by /usr/local/bin/webapp when trying to access /home/webapp/static-files/users.html because the path was absent from its access control list. Updating and reloading the profile instructed the kernel to allow the operation.

Key Takeaway

AppArmor rules override traditional POSIX root and user permissions. Any new file path accessed by a confined binary must be explicitly declared in its profile.


5. Root Cause Analysis

The failure of the Golang web application was caused by four distinct, cascading technical issues across different system layers:


┌────────────────────────────────────────────────────────────────────────┐
│                           SYSTEM CALL ATTEMPT                          │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 1. Process Layer (Daemon Retry Loop)                                   │
│    Cause: Restricted permissions (0700/0600 root) on /home/webapp/pki/  │
│    Effect: Daemon couldn't read server.pem -> TCP 7000 Connection Refused │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 2. Transport/TLS Layer (Trust & Resolution Failure)                    │
│    Cause: CA.crt unimported; missing /etc/hosts mapping for "webapp"   │
│    Effect: curl rejected self-signed chain & failed SAN domain match   │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 3. POSIX DAC Layer (Directory & File Permissions)                      │
│    Cause: Owner admin:admin & missing directory search/execute (+x)     │
│    Effect: Process blocked at filesystem permission level               │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 4. Mandatory Access Control Layer (AppArmor Policy)                    │
│    Cause: Profile /etc/apparmor.d/usr.local.bin.webapp lacked asset path│
│    Effect: Kernel LSM intercepted open() system call -> HTTP 403        │
└────────────────────────────────────────────────────────────────────────┘

Layer-by-Layer Root Cause Breakdown

  1. Daemon Startup Failure: Systemd started webapp.service, but the binary returned permission denied while trying to open /home/webapp/pki/server.crt and server.pem. Because the process user webapp could not open the private key, it slept continuously, preventing the network stack from opening a listening socket on TCP port 7000.
  2. TLS Trust Infrastructure: The application used a custom private Root CA. Because this CA was not installed in /usr/local/share/ca-certificates/, standard OpenSSL verification failed. Additionally, the TLS certificate contained a SubjectAlternativeName restricting access to webapp. Without a /etc/hosts mapping (127.0.10.1 webapp), name resolution failed to validate the TLS handshake.
  3. Discretionary Access Control (DAC): The static asset directory /home/webapp/static-files was originally owned by admin:admin with 0750 permissions. The webapp system user lacked read access to users.html and traversal privileges (+x) on the folder path.
  4. Mandatory Access Control (MAC - AppArmor): Linux Security Modules (LSM) act as an absolute authority after POSIX permissions pass. The binary /usr/local/bin/webapp was confined by AppArmor profile /etc/apparmor.d/usr.local.bin.webapp. Because /home/webapp/static-files/* was not explicitly listed with read (r) permissions, the kernel denied system call execution, returning 403 Forbidden regardless of file ownership.

6. Solution and Recovery

Comprehensive Remediation Script

To restore the service safely following minimal privilege principles, execute the following commands as root:

#!/usr/bin/env bash
set -euo pipefail

# 1. Configure Hostname Resolution
grep -q "webapp" /etc/hosts || echo '127.0.10.1 webapp' >> /etc/hosts

# 2. Fix PKI Permissions (Minimum Read Privilege)
chown -R webapp:webapp /home/webapp/pki
chmod 0700 /home/webapp/pki
chmod 0600 /home/webapp/pki/server.*

# 3. Install Private CA into Operating System Trust Store
cp /home/webapp/pki/CA.crt /usr/local/share/ca-certificates/CA.crt
chmod 0644 /usr/local/share/ca-certificates/CA.crt
update-ca-certificates

# 4. Fix POSIX Permissions for Static Content
chown -R webapp:webapp /home/webapp/static-files
chmod 0755 /home/webapp
chmod 0755 /home/webapp/static-files
chmod 0640 /home/webapp/static-files/*

# 5. Update AppArmor MAC Profile
cat << 'EOF' > /etc/apparmor.d/usr.local.bin.webapp
#include <tunables/global>

/usr/local/bin/webapp {
  #include <abstractions/base>
  #include <abstractions/nameservice>

  /home/webapp/pki/server.crt r,
  /home/webapp/pki/server.pem r,
  /home/webapp/static-files/ r,
  /home/webapp/static-files/* r,
}
EOF

# 6. Reload AppArmor Profile and Restart Service
apparmor_parser -r /etc/apparmor.d/usr.local.bin.webapp
systemctl restart webapp

7. Verification

Verify complete end-to-end functionality using the following tests:

1. Verify Service & Socket Listener

sudo systemctl status webapp
sudo ss -tulpn | grep 7000

Expected Result: webapp.service is active (running), and ss shows webapp listening on *:7000 or 127.0.0.1:7000.

2. Verify AppArmor Profile Enforcement

sudo aa-status | grep usr.local.bin.webapp

Expected Result: Profile is listed under enforce mode.

3. Verification of Secure Endpoint Content Delivery

curl https://webapp:7000/users.html

Expected Output:

<html>
  <head> </head>
  <body>
    <p>From Users Page</p>
  </body>
</html>

Exit Status: 0 (indicating successful TLS validation, name resolution, and HTTP 200 payload delivery).


8. Deeper Dive: Understanding the Linux Commands

journalctl

  • Purpose: Queries and displays system logs generated by systemd-journald.
  • General Syntax: journalctl [options] [matches]
  • Options Used: -u webapp (unit filter), -e (jump to end), -f (follow live output).
  • Other Useful Flags:
    • -p err..emerg: Filters messages by severity priority (err, crit, alert, emerg).
    • --since "1 hour ago": Filters logs within a relative time range.
    • -k: Shows kernel logs only (useful for detecting AppArmor/SELinux denials).

update-ca-certificates

  • Purpose: Updates the /etc/ssl/certs directory and /etc/ssl/certs/ca-certificates.crt composite file to include custom SSL/TLS certificates.
  • General Syntax: update-ca-certificates [options]
  • Flags Worth Knowing:
    • -f / --fresh: Wipes existing symlinks and rebuilds the trust store completely from scratch.

openssl s_client

  • Purpose: A generic SSL/TLS client tool for establishing raw encrypted connections to remote endpoints to diagnose handshakes, protocols, and certificate chains.
  • General Syntax: openssl s_client -connect host:port [options]
  • Flags Worth Knowing:
    • -tls1_3: Forces TLS 1.3 protocol handshake.
    • -CAfile /path/to/ca.crt: Specifies a custom CA file directly during handshake testing.
    • -servername hostname: Sends Server Name Indication (SNI) extension during TLS client hello.

apparmor_parser

  • Purpose: Loads, reloads, or unloads AppArmor MAC policy profiles into the Linux Kernel.
  • General Syntax: apparmor_parser [options] <profile_file>
  • Options Used: -r (replace/reload existing profile).
  • Flags Worth Knowing:
    • -C: Compiles profile into binary format without loading into kernel.
    • -R: Removes/unloads profile from kernel space.
    • -q: Quiet mode; suppresses warnings.

9. Explore Alternative Troubleshooting Methods

Diagnostic Need Alternative Utility Practical Example Command Scenario Advantage
Detect LSM Denials dmesg / auditd sudo dmesg | grep -i apparmor Directly displays kernel-level access denial events (audit: type=1400 DENIED) when HTTP logs are silent.
Inspect System Calls strace sudo strace -p <PID> -e trace=openat,access Tracks raw kernel system calls of a running binary to see exact paths failing with EACCES.
Audit Socket Owners lsof / fuser sudo lsof -i :7000 Instantly reveals which binary and user ID own an open network port.
AppArmor Status aa-status sudo aa-status Summarizes all active, enforced, and complaining AppArmor profiles across running processes.

10. Deeper Linux Concepts and Internals

Discretionary Access Control (DAC) vs. Mandatory Access Control (MAC)


System Access Request (e.g., open file)
│
▼
┌──────────────────────────────────┐
│  Layer 1: POSIX DAC Evaluation   │ ──► Checks UID/GID against rwx permissions bits
└──────────────────────────────────┘
│ Passes
▼
┌──────────────────────────────────┐
│   Layer 2: Kernel LSM (MAC)      │ ──► Checks Process Path/Label against AppArmor/SELinux Policy
└──────────────────────────────────┘
│ Passes
▼
Access Granted
  1. POSIX DAC (Discretionary Access Control): Traditional Linux permission model. Access is determined by ownership (user:group) and permission flags (rwx). DAC is discretionary because resource owners can modify permissions at will.
  2. Linux Security Modules (LSM / MAC): Mandatory Access Control operates at the Linux kernel level via LSM hooks. AppArmor enforces path-based rules, while SELinux enforces label-based rules. MAC overrides DAC—even if root or a file owner attempts access, the kernel denies the system call if the active MAC profile omits the operation.

POSIX Directory Traversal (+x) Semantics

In POSIX file systems, permissions on directories differ fundamentally from regular files:

  • Directory Read (r): Allows listing file names within the directory (e.g., ls).
  • Directory Execute (x): Acts as a search/traversal bit. Allows a process to pass through the directory to access nested subdirectories or files.
  • Rule: A process cannot read /home/webapp/static-files/users.html unless every parent directory (/, /home, /home/webapp, /home/webapp/static-files) grants +x to that user or process group.

11. Common Mistakes and Troubleshooting Pitfalls

  1. Over-Granting File Permissions (chmod 777): Applying 777 to fix permission errors is a severe security anti-pattern. It makes private TLS keys world-readable, compromising host security. Always use minimum required permissions (0600 for keys, 0640 for static files, 0755 for directories).
  2. Confusing File Read (r) with Directory Traversal (x): Giving 0644 permissions to a file fails if any parent directory lacks the +x execute bit.
  3. Bypassing Security Flags (curl -k): Using --insecure hides certificate trust and domain name errors during testing, leaving system problems undetected until production deployment.
  4. Ignoring LSM Security Modules: System administrators often spend hours debugging file permissions without checking dmesg or aa-status to see if AppArmor or SELinux is blocking system calls.
  5. Shell Redirection Privileges with sudo: Executing sudo echo '127.0.10.1 webapp' >> /etc/hosts fails with Permission denied because shell redirection (>>) runs under the unprivileged user’s shell context. Use echo '127.0.10.1 webapp' | sudo tee -a /etc/hosts instead.

12. Quick Reference

Command Quick Reference Table

Command Purpose Key Flags / Arguments Practical Use Case
journalctl Query system logs -u <unit> -ef Debug service crashes and startup loops.
update-ca-certificates Rebuild system CA trust store -f Install private enterprise Root CAs.
openssl s_client Debug TLS handshakes -connect host:port Verify remote SSL certificates and SAN domain names.
chmod Modify POSIX permissions 0600 (key), 0755 (dir) Enforce principle of least privilege.
chown Modify file ownership -R user:group Align resource ownership with service accounts.
apparmor_parser Reload AppArmor profiles -r <profile> Apply updated kernel MAC rules without rebooting.
dmesg Print kernel ring buffer -T | grep -i apparmor Audit LSM kernel security denial events.

13. Further Learning and Practice

To reinforce the techniques covered in this post-mortem, try these hands-on exercises on a local test system:

  1. AppArmor Profile Creation: Create a minimal C or Go HTTP web server, generate an AppArmor profile using aa-autodep, and restrict file system access. Intentionally attempt to access unlisted paths and monitor denial messages in dmesg.
  2. PKI & CA Trust Setup: Set up a local private CA using openssl, issue server certificates with custom SubjectAlternativeName fields, and configure client validation without using curl -k.
  3. Least Privilege Auditing: Write a script to audit permissions across web root directories, verifying that no private key files possess world-readable flags or unnecessary execute bits.