1. Introduction
Public Key Infrastructure (PKI) and Transport Layer Security (TLS) outages are among the most common—and most disruptive—production incidents in Linux environments. When a TLS-enabled web server fails to start or rejects client handshakes, every downstream service and user attempting to connect over HTTPS experiences an immediate availability failure.
In this troubleshooting guide, we investigate a realistic TLS outage from the SadServers scenario “Salzburg”: The TLS key that still matches. The Salzburg Festival box-office web application should be served over HTTPS on port 443 by Nginx, but the service is down, port 443 has no listener, and the existing X.509 certificate has expired. Complicating recovery, an upstream edge Access Control List (ACL) pins the server’s existing public key, prohibiting administrators from simply generating a fresh key pair, while a chaotic key-archive cleanup scattered 29 unsorted files into a dump directory.
This incident involves several core Linux subsystems and tools:
- Systemd service management (
nginx.servicelifecycle andExecStartPrevalidation gates) - Nginx configuration and privilege separation (master vs. worker process key loading)
- OpenSSL cryptographic primitives (asymmetric key derivation, X.509v3 certificate inspection, and Subject Alternative Name extensions)
- Linux filesystem permissions and ownership (protecting
0600private key material)
By working through this article, you will learn how to separate symptoms from root causes during TLS failures, mathematically match an Elliptic Curve (ECDSA) private key to a public key without trusting misleading filenames, renew a self-signed certificate while preserving its Subject and SANs, and apply production SRE practices to prevent certificate and key-rotation outages.
2. Problem Description and Environment
Original Challenge Summary
- Scenario:
"Salzburg": The TLS key that still matches(Medium) - Target Endpoint:
[https://www.festival.salzburg.local/](https://www.festival.salzburg.local/)(local host, TCP port443) - Required Deliverables:
- Locate the private key inside
/opt/festival/keys/that corresponds to the public key at/home/admin/festival.pub. - Place a copy of the matching private key at
/home/admin/festival.key(for on-call handover and automated test validation) and restore it for Nginx. - Issue a new, non-expired X.509 certificate for the original festival hostname using that exact same key pair (do not generate a new key pair).
- Restore HTTPS service on port
443socurl -k [https://127.0.0.1/](https://127.0.0.1/)returns the box-office HTML page.
Expected vs. Actual Behaviour
| Component | Expected Behaviour | Actual Behaviour |
|---|---|---|
| Nginx Service | nginx.service is active (running) and listening on 0.0.0.0:443 |
nginx.service is in failed (Result: exit-code) state; ExecStartPre (nginx -t) exits with status 1 |
| Network Socket | Kernel TCP socket bound to port 443 |
`ss -tulpn |
| Key Material | Matching private key present in /etc/nginx/ssl/ and /home/admin/festival.key |
Only festival.crt remains in /etc/nginx/ssl/; 29 unsorted files exist in /opt/festival/keys/ |
| X.509 Certificate | Valid, non-expired certificate matching www.festival.salzburg.local and festival.pub |
Existing /etc/nginx/ssl/festival.crt expired on Jun 1 00:00:00 2024 GMT |
Environment Details
- Operating System: Debian 13 (Kernel version was not captured during the session)
- Host Identity & Privileges:
admin@i-05870ea37ff767d56with fullsudo(root) privileges - Web Server / Reverse Proxy: Nginx (
/usr/sbin/nginx, configuration rooted at/etc/nginx/nginx.conf) - Operational Constraints:
- Public Key Pinning: An upstream edge ACL pins
/home/admin/festival.pub. Generating a new key pair would break upstream trust even if Nginx started cleanly. - Archive Preservation: Operator notes in
/opt/festival/keys/NOTES.txtexplicitly instruct engineers not to delete files from/opt/festival/keys/.
3. Initial Investigation and Hypotheses
Separating Symptoms from Root Causes
When an incident description mentions both an expired TLS certificate and a web server that refuses to start, a common mistake is assuming the expired certificate caused the daemon crash.
In practice, Nginx does not refuse to start when an X.509 certificate is expired. OpenSSL loads expired certificates into Nginx’s SSL context without error during startup; the failure occurs later during the TLS handshake when clients reject the notAfter timestamp. The fact that nginx.service failed during ExecStartPre indicated a structural or filesystem problem—such as a missing private key file, a certificate/key mismatch, a file permission error, or a configuration syntax mistake.
What Was Known vs. Unknown
-
Initially Known:
-
Port
443was closed (Connection refused). -
/home/admin/HANDOVER.txtstated that a key-archive cleanup moved files into/opt/festival/keys/and that the matching private key was supposed to remain in/etc/nginx/ssl/. -
/home/admin/festival.pubwas intact on disk. -
Initially Unknown:
-
Why
nginx -tfailed (the exact error line was truncated insystemctl status). -
The exact filesystem path Nginx expected for its private key.
-
The cryptographic algorithm and format of
/home/admin/festival.pub(RSA vs. ECDSA vs. Ed25519). -
Whether
/etc/nginx/ssl/festival.crtwas self-signed or issued by a local Certificate Authority (CA) present on the machine, and what Subject / Subject Alternative Names (SANs) it contained.
Initial Hypotheses
| Hypothesis | Supporting Evidence | Evidence Against | Diagnostic Step |
|---|---|---|---|
**H1: Missing private key file in /etc/nginx/ssl/** |
HANDOVER.txt noted that a cleanup dumped keys into /opt/festival/keys/; ls /etc/nginx/ssl/ showed only festival.crt. |
None yet, though permissions or syntax errors could also contribute. | Run sudo nginx -t to view the untruncated OpenSSL/Nginx error message. |
| H2: Expired certificate caused Nginx startup failure | Problem description and HANDOVER.txt highlighted that the certificate expired overnight. |
Nginx normally loads expired certificates at startup and leaves expiry validation to TLS clients. | Inspect festival.crt dates with openssl x509 and verify exact failure in nginx -t. |
| H3: Internal or local CA signed the original certificate | Enterprise staging environments (www.festival.salzburg.local) often use an internal intermediate CA (a file named ca-intermediate.key existed in /opt/festival/keys/). |
Could also be a standalone self-signed certificate. | Compare -subject and -issuer fields on /etc/nginx/ssl/festival.crt. |
Gathering read-only evidence before copying files or generating certificates ensures we do not overwrite critical metadata (such as original Distinguished Name fields or SAN extensions) or choose the wrong signing model.
4. Troubleshooting Process
Step 1: Initial Triage, Handover Review, and Service State
Objective
Confirm the operational requirements in HANDOVER.txt, inventory /etc/nginx/ssl/ and /opt/festival/keys/, verify whether anything is listening on port 443, and check the nginx systemd unit state.
Commands
cat HANDOVER.txt
ls /etc/nginx/ssl/
ls /opt/festival/keys/
curl https://127.0.0.1:443
ss -tulpn | grep :443
systemctl status nginx --no-pager
Command Explanation
cat HANDOVER.txtreads the operator handover note directly tostdout.ls /etc/nginx/ssl/andls /opt/festival/keys/inspect the live TLS directory and the archive dump directory without modifying timestamps or contents.curl [https://127.0.0.1:443](https://127.0.0.1:443)tests whether a TLS listener responds on the loopback interface.ss -tulpn | grep :443queries kernel netlink socket diagnostics for any TCP (-t) or UDP (-u) listening (-l) sockets on port443with numeric ports (-n) and process info (-p).systemctl status nginx --no-pagerchecks the unit state, recent journal entries, and exit status ofnginx.servicewithout invoking a pager (less).
Output and Analysis
admin@i-05870ea37ff767d56:~$ cat HANDOVER.txt
Salzburg Festival box office — staging handover
===============================================
Site: https://www.festival.salzburg.local/ (this host, :443)
The certificate expired overnight. Do not mint a new key pair: the
edge ACL already pins the current public key. Renew the cert with
the same private key.
...
admin@i-05870ea37ff767d56:~$ ls /etc/nginx/ssl/
festival.crt
admin@i-05870ea37ff767d56:~$ ls /opt/festival/keys/
2022-03-18-www.pem id_rsa scratch-17.key
NOTES.txt id_rsa.old server.key
backup-15.key incident-4812 staging.csr
boxoffice-2023.pem letsencrypt-backup.key staging.key
ca-intermediate.key nginx.key.bak ticket-1088.key
deploy_ed25519 old-prod.pem tmp-gen.key
ec-p256-current.pem old-selfsigned.crt vpn-client.pem
festival.key pkcs8-exported.pem wildcard.festival.salzburg.local.key
github-deploy privkey.pem www.festival.salzburg.local.key
id_ed25519 rotate-20241103.pem
admin@i-05870ea37ff767d56:~$ systemctl status nginx --no-pager
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Thu 2026-11-12 14:45:32 UTC; 2min 34s ago
Process: 763 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
...
Nov 12 14:45:32 i-05870ea37ff767d56 nginx[763]: 2026/11/12 14:45:32 [emerg] 763#763: cannot load…file)
Key observations:
/etc/nginx/ssl/contains onlyfestival.crt; the private key is absent./opt/festival/keys/contains 29 files, including obvious decoy names such asfestival.key,www.festival.salzburg.local.key,server.key,privkey.pem, andec-p256-current.pem.nginx.servicefailed insideExecStartPre=/usr/sbin/nginx -t ..., preventingExecStartfrom ever launching the daemon or binding:443.systemctl statusellipsized (…) the critical[emerg]log message at the terminal width boundary.
Decision
Run sudo nginx -t directly to view the full untruncated error message, read /opt/festival/keys/NOTES.txt, inspect /home/admin/festival.pub, and extract metadata from /etc/nginx/ssl/festival.crt.
Step 2: Inspecting the Nginx Error, Notes, Public Key, and Existing Certificate
Objective
Determine the exact file path Nginx is trying to load, check for operational notes in the key dump directory, identify the public key’s algorithm and encoding, and inspect the existing certificate’s Subject, Issuer, validity dates, and Subject Alternative Names (SANs).
Commands
sudo nginx -t
cat /opt/festival/keys/NOTES.txt
cat festival.pub
openssl x509 -in /etc/nginx/ssl/festival.crt -noout -subject -issuer -dates -ext subjectAltName
Command Explanation
sudo nginx -t: Runs Nginx’s configuration test asrootso permission errors on0600files do not mask configuration or missing-file errors, printing full error messages tostderr.openssl x509: Parses X.509 certificates.-in /etc/nginx/ssl/festival.crt: Specifies the input PEM certificate.-noout: Suppresses printing the raw Base64-encoded certificate block.-subjectand-issuer: Print the certificate owner and the signing authority Distinguished Names (DNs).-dates: PrintsnotBeforeandnotAftervalidity timestamps.-ext subjectAltName: Prints the X.509v3 Subject Alternative Name extension.
Output and Analysis
admin@i-05870ea37ff767d56:~$ sudo nginx -t
2026/11/12 14:49:09 [emerg] 982#982: cannot load certificate key "/etc/nginx/ssl/festival.key": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/nginx/ssl/festival.key, r) error:10000080:BIO routines::no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed
admin@i-05870ea37ff767d56:~$ cat /opt/festival/keys/NOTES.txt
Festival private-key dump, 2022-2025.
Filenames are ticket ids and dates, not hostnames. Do not delete.
The live key was supposed to stay under /etc/nginx/ssl/.
admin@i-05870ea37ff767d56:~$ cat festival.pub
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEh+NTqS4SNs2dy0FpGp3iC9tQUv44
hlmYBc7bxbXjq11tzz78JJ+9ztjvsbZ7UwqMYVvaRIcjOodK6sSx/nOJxA==
-----END PUBLIC KEY-----
admin@i-05870ea37ff767d56:~$ openssl x509 -in /etc/nginx/ssl/festival.crt -noout -subject -issuer -dates -ext subjectAltName
subject=CN=www.festival.salzburg.local, O=Salzburg Festival, OU=Box Office, L=Salzburg, ST=Salzburg, C=AT
issuer=CN=www.festival.salzburg.local, O=Salzburg Festival, OU=Box Office, L=Salzburg, ST=Salzburg, C=AT
notBefore=Jan 1 00:00:00 2024 GMT
notAfter=Jun 1 00:00:00 2024 GMT
X509v3 Subject Alternative Name:
DNS:www.festival.salzburg.local, DNS:localhost, IP Address:127.0.0.1
Critical conclusions from this output:
- Confirmed Nginx Path (Hypothesis H1 confirmed): Nginx explicitly expects
/etc/nginx/ssl/festival.keyand failed becausefopen("/etc/nginx/ssl/festival.key", "r")returnedNo such file or directory(ENOENT). - Confirmed Decoy Filenames:
NOTES.txtconfirms filenames in/opt/festival/keys/reflect ticket IDs and dates rather than hostnames, and warns not to delete anything. - Key Algorithm Identified (
ECDSA P-256):festival.pubuses X.509SubjectPublicKeyInfo(SPKI) PEM format (-----BEGIN PUBLIC KEY-----). Its ASN.1 header (MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...) encodes an Elliptic Curve P-256 (prime256v1) public key—meaning legacy RSA modulus comparisons (openssl rsa -modulus) would fail. - Self-Signed Certificate with SANs (Hypothesis H3 ruled out): Because
subjectandissuerare identical (CN=www.festival.salzburg.local, O=Salzburg Festival, OU=Box Office, L=Salzburg, ST=Salzburg, C=AT), the certificate is self-signed—we do not needca-intermediate.key. It expired onJun 1 2024and must retain all three SANs (DNS:www.festival.salzburg.local,DNS:localhost,IP Address:127.0.0.1).
Decision
Iterate through all files in /opt/festival/keys/, derive the public key from each candidate private key in SPKI format, and perform a byte-for-byte comparison against /home/admin/festival.pub.
Step 3: Mathematically Matching the Private Key
Objective
Identify which of the 29 files in /opt/festival/keys/ mathematically pairs with /home/admin/festival.pub without modifying any files on disk.
Command
for f in /opt/festival/keys/*; do
if diff -q <(openssl pkey -in "$f" -pubout -passin pass: 2>/dev/null) /home/admin/festival.pub >/dev/null 2>&1; then
echo "MATCH: $f"
fi
done
(Note: During interactive entry, two initial one-liner attempts hit Bash syntax errors due to a space between < and ( in < (openssl ...) and missing semicolons before fi and done. Properly formatted process substitution <(...) resolved the syntax issue.)
Command Explanation
-
for f in /opt/festival/keys/*; do ... done: Loops safely over every entry in/opt/festival/keys/using double-quoted"$f"to prevent word-splitting issues. -
openssl pkey -in "$f" -pubout -passin pass:: -
pkeyis OpenSSL’s algorithm-agnostic asymmetric key utility (supporting RSA, ECDSA, Ed25519, and DSA across PKCS#1, SEC1, and PKCS#8 PEM formats). -
-puboutcomputes and outputs the corresponding public key in standardized SPKI PEM format (-----BEGIN PUBLIC KEY-----), matchingfestival.pub. -
-passin pass:supplies an empty passphrase non-interactively so any passphrase-protected key fails immediately instead of hanging the loop on a password prompt. -
2>/dev/nulldiscards OpenSSL parse errors when the loop encounters non-PEM files (NOTES.txt,.crt,.csr, or OpenSSH-format keys). -
diff -q <(...) /home/admin/festival.pub: Compares the derived public key stream via Bash process substitution against/home/admin/festival.pub, returning exit code0if and only if the outputs are identical.
Output and Analysis
MATCH: /opt/festival/keys/scratch-17.key
Out of 29 files in /opt/festival/keys/, **only /opt/festival/keys/scratch-17.key** derived the exact public key stored in /home/admin/festival.pub. All other .key and .pem files—including festival.key and www.festival.salzburg.local.key—were stale or unrelated keys.
Decision
Back up the expired certificate, deploy /opt/festival/keys/scratch-17.key to /home/admin/festival.key and /etc/nginx/ssl/festival.key with strict 0600 permissions, generate a new self-signed certificate using that existing private key, and start Nginx.
5. Root Cause Analysis
Primary Root Cause
The immediate cause of the service outage was the **removal of /etc/nginx/ssl/festival.key** during an archival cleanup script run that relocated the active ECDSA P-256 private key to /opt/festival/keys/scratch-17.key.
When systemd attempted to start nginx.service, it executed the pre-start validation hook defined in /usr/lib/systemd/system/nginx.service:
ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on;
During configuration parsing, Nginx invoked OpenSSL’s BIO_new_file() wrapper, which called the C standard library fopen("/etc/nginx/ssl/festival.key", "r") -> kernel openat() syscall. The Linux Virtual File System (VFS) returned -ENOENT (No such file or directory), causing nginx -t to exit with status 1 (FAILURE). Because ExecStartPre failed, systemd aborted unit startup before ExecStart could spawn the master process or bind a socket to TCP port 443.
Secondary Failure Condition
Even after restoring the missing private key, the certificate at /etc/nginx/ssl/festival.crt had passed its notAfter expiration timestamp (Jun 1 00:00:00 2024 GMT). Because an upstream edge ACL pinned the public key in /home/admin/festival.pub, regenerating a fresh key pair would have caused upstream TLS validation failures. Renewing the self-signed certificate with the recovered private key and original Subject/SANs was required to restore full end-to-end validity.
Failure Chain
Key-archive cleanup moves active key -> /opt/festival/keys/scratch-17.key
│
▼
/etc/nginx/ssl/festival.key no longer exists (ENOENT)
│
▼
systemctl start nginx -> runs ExecStartPre (/usr/sbin/nginx -t)
│
▼
OpenSSL BIO_new_file() fails with "No such file or directory" (exit code 1)
│
▼
systemd marks nginx.service as failed; TCP :443 remains unbound
6. Solution and Remediation
Safety, Backup, and Rollback Strategy
Before modifying any files under /etc/nginx/ssl/:
- Preservation: We copied
/opt/festival/keys/scratch-17.keyrather than moving it, honoringNOTES.txt’s “Do not delete” instruction. - Backup: We created a timestamp-preserving backup (
/etc/nginx/ssl/festival.crt.expired.bak) before overwriting/etc/nginx/ssl/festival.crt. - Rollback Plan: If the newly issued certificate caused issues, restoring the previous state required only moving
festival.crt.expired.bakback tofestival.crtand removing the copied key.
Commands Executed
# 1. Back up the expired certificate before overwriting
sudo cp -p /etc/nginx/ssl/festival.crt /etc/nginx/ssl/festival.crt.expired.bak
# 2. Place the matching private key at /home/admin/festival.key with restricted permissions
cp /opt/festival/keys/scratch-17.key /home/admin/festival.key
chmod 600 /home/admin/festival.key
# 3. Install the matching private key for Nginx owned by root:root with mode 0600
sudo install -o root -g root -m 600 /opt/festival/keys/scratch-17.key /etc/nginx/ssl/festival.key
# 4. Issue a new self-signed X.509v3 certificate using the recovered key, original Subject, and original SANs
sudo openssl req -new -x509 \
-key /etc/nginx/ssl/festival.key \
-out /etc/nginx/ssl/festival.crt \
-days 365 \
-subj "/C=AT/ST=Salzburg/L=Salzburg/O=Salzburg Festival/OU=Box Office/CN=www.festival.salzburg.local" \
-addext "subjectAltName = DNS:www.festival.salzburg.local, DNS:localhost, IP:127.0.0.1"
# 5. Set standard world-readable permissions on the public certificate
sudo chmod 644 /etc/nginx/ssl/festival.crt
# 6. Validate Nginx configuration and start the service
sudo nginx -t
sudo systemctl start nginx
Why This Solution Works
install -o root -g root -m 600: Atomically copies the private key to/etc/nginx/ssl/festival.keywhile ensuring onlyrootcan read or write it, preventing unprivileged local users from compromising the private key.openssl req -new -x509 -key ...(without-newkey): Passing-keyreuses the existing ECDSA P-256 private key rather than minting a new key pair, preserving the edge ACL public-key pin.-subjand-addext: Reconstructs the exact Distinguished Name and all three Subject Alternative Names (DNS:www.festival.salzburg.local,DNS:localhost,IP:127.0.0.1) required by modern TLS hostname verification.- No Configuration File Edits Needed: Because
/etc/nginx/nginx.conf(and its included site files) already pointed to/etc/nginx/ssl/festival.crtand/etc/nginx/ssl/festival.key, restoring both files in place satisfied Nginx’s existing directives.
7. Verification and Validation
Executed Verification Steps
- Configuration Syntax and File Readability Check:
sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
- Systemd Service and Process Health Check:
systemctl status nginx --no-pager
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
Active: active (running) since Thu 2026-11-12 14:59:18 UTC; 19s ago
Process: 1216 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 1218 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 1219 (nginx)
Tasks: 3 (limit: 500)
Memory: 2.9M (peak: 4M)
CGroup: /system.slice/nginx.service
├─1219 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
├─1220 "nginx: worker process"
└─1221 "nginx: worker process"
Loaded: ... enabledconfirmsnginx.serviceis configured to start automatically across system reboots.Main PID: 1219with two worker processes confirms the daemon initialized cleanly.
- HTTPS Endpoint Response Check:
curl -k https://127.0.0.1/
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Salzburg Festival Box Office</title>
</head>
<body>
<h1>Salzburg Festival Box Office</h1>
<p>Tickets and season passes.</p>
</body>
</html>
(Note: In the live session, curl was invoked as curl -k https:/127.0.0.1/ with a single slash, which curl’s URL parser normalizes to [https://127.0.0.1/](https://127.0.0.1/).)
Command Success vs. End-to-End TLS Health
Confirming that openssl req exited 0 and nginx is active (running) proves the daemon started, but in production environments you should also verify the live certificate served over TCP (to confirm workers did not retain an older cached certificate in memory):
# Suggested post-change live socket inspection (not executed during session)
openssl s_client -connect 127.0.0.1:443 -servername www.festival.salzburg.local </dev/null 2>/dev/null \
| openssl x509 -noout -subject -dates -ext subjectAltName
8. Alternative Troubleshooting Approaches
Experienced sysadmins and SREs have several alternative ways to investigate and solve key-matching and Nginx TLS failures depending on scale and environment constraints.
| Approach | Core Mechanism | Best Used When | Limitations |
|---|---|---|---|
SPKI Byte Comparison (openssl pkey -pubout + diff) (Used) |
Derives SPKI PEM public key from each candidate and diffs against .pub |
Local directories with tens or hundreds of mixed-algorithm keys | Spawns one openssl subprocess per file |
SHA-256 Digest Comparison (sha256sum) |
Hashes derived SPKI public keys and compares 64-char hex fingerprints | Auditing thousands of keys across remote hosts or matching HPKP/SPKI pins | Requires normalizing line endings/headers consistently |
**Two-Step CSR + openssl x509 -req** |
Generates a .csr first, then signs it using -signkey and an -extfile |
Older systems (OpenSSL < 1.1.1) lacking -addext, or when handing CSRs to a CA |
Leaves intermediate .csr and .ext files to clean up |
Full Nginx Config Dump (nginx -T) |
Dumps merged configuration across all include files |
Complex Nginx setups with dozens of vhosts where ssl_certificate paths are unknown |
Prints entire config to stdout; requires grep filtering |
Alternative 1: SHA-256 Fingerprint Matching (Proposed Alternative)
Instead of comparing multi-line PEM blocks with diff, you can compute the SHA-256 digest of /home/admin/festival.pub and compare single-line hashes across candidate keys:
# Proposed alternative (not executed during session)
TARGET_HASH=$(sha256sum /home/admin/festival.pub | awk '{print $1}')
for f in /opt/festival/keys/*; do
KEY_HASH=$(openssl pkey -in "$f" -pubout -passin pass: 2>/dev/null | sha256sum | awk '{print $1}')
if [ "$KEY_HASH" = "$TARGET_HASH" ]; then
echo "MATCH: $f"
fi
done
- Advantages: Produces compact, loggable fingerprints that can be compared across air-gapped systems without transferring public key files.
- When to use: Large-scale key inventory audits or automated configuration management verification scripts.
Alternative 2: Two-Step CSR and openssl x509 -req Signing (Proposed Alternative)
On older distributions (such as CentOS/RHEL 7 running OpenSSL 1.0.2) where openssl req -addext is unavailable, you generate a Certificate Signing Request (.csr) and sign it using an extension file:
# Proposed alternative (not executed during session)
sudo openssl req -new -key /etc/nginx/ssl/festival.key -out /tmp/festival.csr \
-subj "/C=AT/ST=Salzburg/L=Salzburg/O=Salzburg Festival/OU=Box Office/CN=www.festival.salzburg.local"
printf "subjectAltName=DNS:www.festival.salzburg.local,DNS:localhost,IP:127.0.0.1\n" > /tmp/san.ext
sudo openssl x509 -req -in /tmp/festival.csr \
-signkey /etc/nginx/ssl/festival.key \
-out /etc/nginx/ssl/festival.crt \
-days 365 -extfile /tmp/san.ext
rm -f /tmp/festival.csr /tmp/san.ext
- Advantages: Works on legacy OpenSSL versions and mirrors the exact workflow used when submitting a CSR to an internal or external CA.
- Limitations: Requires managing temporary
.csrand.extfiles.
Alternative 3: Auditing All Nginx TLS Directives via nginx -T (Proposed Alternative)
When troubleshooting multi-tenant Nginx instances with complex sites-enabled/ includes, sudo nginx -T dumps the entire effective configuration:
# Proposed alternative (not executed during session)
sudo nginx -T 2>/dev/null | grep -E "server_name|ssl_certificate"
This reveals every virtual host’s ssl_certificate and ssl_certificate_key path in one pass.
9. Linux Command Deep Dive
Command: openssl pkey
Purpose
openssl pkey is OpenSSL’s unified asymmetric key management utility. It parses, validates, converts, and extracts public keys from private keys across all modern algorithms (RSA, ECDSA, Ed25519, Ed448, DSA) and PEM container formats (PKCS#1, SEC1, PKCS#8).
General Syntax
openssl pkey [OPTIONS]
Important Options
| Option | Purpose |
|---|---|
-in <file> |
Reads the input private or public key file. |
-pubout |
Extracts and outputs the corresponding public key in SPKI PEM format (-----BEGIN PUBLIC KEY-----). |
-passin pass:<pwd> |
Supplies the decryption passphrase non-interactively (an empty pass: prevents hanging on encrypted keys in loops). |
-text -noout |
Displays human-readable key parameters (algorithm, curve name such as prime256v1, or modulus bit size) without printing the Base64 block. |
-check |
Verifies the mathematical consistency of a private key (e.g., confirms an EC private scalar falls within the valid curve order). |
Practical Examples
# Check key algorithm and curve/bit-length of a private key
openssl pkey -in /etc/nginx/ssl/festival.key -text -noout | head -n 3
# Verify that a certificate and private key belong to the same key pair
diff -u <(openssl x509 -in festival.crt -pubkey -noout) <(openssl pkey -in festival.key -pubout)
Common Mistakes and Limitations
- Using
openssl rsa -moduluson ECDSA keys: Legacy guides frequently compareopenssl rsa -noout -moduluswithopenssl x509 -noout -modulus. That technique only works for RSA keys and fails on Elliptic Curve (EC) orEd25519keys. Always preferopenssl pkey -puboutandopenssl x509 -pubkeyfor algorithm-agnostic verification. - OpenSSH Key Format:
openssl pkeycannot parse OpenSSH-proprietary private keys (-----BEGIN OPENSSH PRIVATE KEY-----), such asid_ed25519. For OpenSSH keys, usessh-keygen -y -f <keyfile>.
Command: openssl x509
Purpose
Inspects, validates, converts, and signs X.509 digital certificates.
General Syntax
openssl x509 [OPTIONS]
Important Options
| Option | Purpose |
|---|---|
-in <file> |
Reads the input certificate file. |
-noout |
Suppresses outputting the encoded Base64 PEM certificate block. |
-subject / -issuer |
Displays the Subject DN and Issuer DN (if identical, the cert is self-signed). |
-dates |
Prints notBefore and notAfter validity timestamps. |
-ext subjectAltName |
Extracts the X.509v3 Subject Alternative Name extension. |
-pubkey |
Extracts the embedded public key in SPKI PEM format. |
-checkend <seconds> |
Exits 0 if the certificate remains valid for at least <seconds>, or 1 if it expires sooner. |
Practical Examples
# Check whether a certificate expires within the next 30 days (2,592,000 seconds)
openssl x509 -in /etc/nginx/ssl/festival.crt -noout -checkend 2592000 || echo "Expiring within 30 days!"
# Inspect a certificate chain inside a bundle file
openssl crl2pkcs7 -nocrl -certfile fullchain.pem | openssl pkcs7 -print_certs -noout
Command: openssl req
Purpose
Creates PKCS#10 Certificate Signing Requests (CSRs) or, when combined with -x509, directly generates self-signed X.509 certificates.
General Syntax
openssl req [OPTIONS]
Important Options
| Option | Purpose |
|---|---|
-new |
Generates a new certificate or CSR structure. |
-x509 |
Outputs a self-signed X.509 certificate instead of a CSR. |
-key <file> |
Uses an existing private key rather than generating a new key pair. |
-newkey <alg:param> |
Generates a brand-new key pair alongside the request (e.g., ec:<(openssl ecparam -name prime256v1) or rsa:2048). |
-days <N> |
Sets the validity period in days when -x509 is specified. |
-subj "<DN>" |
Sets the Subject Distinguished Name non-interactively (/C=.../ST=.../O=.../CN=...). |
-addext "<ext>" |
Adds X.509v3 extensions (such as subjectAltName) directly via CLI flags (OpenSSL 1.1.1+). |
-noenc (-nodes) |
Skips passphrase encryption when generating a new private key so daemons can start unattended. |
Common Mistakes and Limitations
- Confusing
-keywith-newkey: Using-newkeywhen public-key pinning is active creates a brand-new key pair and breaks upstream ACLs. - SAN IP Syntax: Inside
-addext, IP addresses must be prefixed withIP:(e.g.,IP:127.0.0.1), whereas OpenSSL’s human-readable output displaysIP Address:127.0.0.1.
Command: install
Purpose
Part of GNU coreutils; copies files to a destination while atomically setting ownership (chown) and file mode permissions (chmod) in a single command.
General Syntax
install [OPTIONS] SOURCE DEST
Important Options
| Option | Purpose |
|---|---|
-o <user> |
Sets the destination file owner (requires root). |
-g <group> |
Sets the destination file group. |
-m <mode> |
Sets octal permissions (e.g., 600 for private keys, 755 for binaries). |
-d <dir> |
Creates all leading directory components with the specified owner, group, and mode. |
-b |
Creates a backup of the destination file if it already exists. |
Common Mistakes and Limitations
- Default
0755Mode: Unlikecp(which respects the user’sumask),installdefaults to0755(world-readable and executable) if-mis omitted. Always pass-m 600when deploying private keys or credentials.
Command: nginx
Purpose
Controls the Nginx master process and validates configuration syntax and referenced assets.
Important Options
| Option | Purpose |
|---|---|
-t |
Tests configuration syntax and verifies that referenced files (certs, keys, logs) exist and are readable. |
-T |
Tests configuration and dumps the full merged configuration across all include files to stdout. |
-s reload |
Sends SIGHUP to the running master process to gracefully reload configuration and TLS certificates with zero dropped connections. |
-V |
Displays Nginx version, linked OpenSSL version, and compile-time paths (--conf-path, --prefix). |
10. Underlying Linux Concepts and Architecture
1. Nginx Privilege Separation and Private Key Permissions
A frequent point of confusion for newer administrators is how Nginx worker processes (which typically run as the unprivileged www-data user on Debian) can use TLS private keys owned by root:root with restrictive 0600 (-rw-------) permissions.
The mechanism relies on Unix process inheritance across fork():
systemdstarts the Nginx master process (PID 1219in our session) asroot(UID 0).- Because
UID 0bypasses discretionary read permission checks (and holdsCAP_NET_BIND_SERVICE), the master process binds privileged ports (< 1024, including TCP443) and reads/etc/nginx/ssl/festival.keyinto its user-space memory to initialize OpenSSL’sSSL_CTXstructures. - The master process then calls
fork()to spawn unprivileged worker processes (PIDs 1220and1221), which drop privileges viasetuid()/setgid()towww-data. - Because
fork()duplicates the parent’s virtual memory space and open file descriptors, workers inherit the initializedSSL_CTXobjects and listening sockets in memory without ever needing filesystem read access to/etc/nginx/ssl/festival.key. Even if an attacker compromises a worker process via an HTTP vulnerability, they cannot read the private key file directly from/etc/nginx/ssl/.
2. Systemd ExecStartPre Validation Gates
In Debian’s /usr/lib/systemd/system/nginx.service, the unit defines:
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
Systemd executes ExecStartPre commands sequentially before ExecStart. If any ExecStartPre command returns a non-zero exit code, systemd immediately transitions the unit into the failed state without running ExecStart. Similarly, ExecReload runs nginx -t before sending SIGHUP—protecting a running production server from terminating if a broken TLS configuration is deployed.
3. SubjectPublicKeyInfo (SPKI) and Elliptic Curve Key Derivation
In Elliptic Curve Cryptography (ECC) over the NIST P-256 curve (prime256v1), a private key is a 256-bit random scalar integer $d$, and the corresponding public key is a point $Q = d \cdot G$ on the elliptic curve (where $G$ is the curve’s standardized generator point).
When OpenSSL writes a public key in -----BEGIN PUBLIC KEY----- format, it wraps the algorithm OID (1.2.840.10045.2.1 for ecPublicKey and 1.2.840.10045.3.1.7 for prime256v1) together with the uncompressed point $Q$ into an ASN.1 SubjectPublicKeyInfo (SPKI) structure. Because $Q$ is deterministically derived from $d$, openssl pkey -pubout produces the exact same Base64 SPKI output regardless of whether the private key file is stored in legacy SEC1 (BEGIN EC PRIVATE KEY) or PKCS#8 (BEGIN PRIVATE KEY) format.
4. Subject Alternative Names (SANs) vs. Common Name (CN)
Historically, TLS clients checked the CN= field in a certificate’s Subject DN against the requested hostname. Under RFC 6125 and modern TLS stacks (OpenSSL 3.x, curl, Go crypto/tls, and all modern browsers), CN matching is deprecated and ignored when validating hostnames. Clients strictly require a matching entry in the X.509v3 subjectAltName extension (DNS:www.festival.salzburg.local, DNS:localhost, or IP:127.0.0.1). Omitting -addext "subjectAltName = ..." when renewing a certificate causes strict TLS clients to fail hostname verification even when the certificate is unexpired and trusted.
11. Real-World Sysadmin and SRE Applications
1. Detection, Monitoring, and Alerting
Certificate expiration is a 100% deterministic failure mode—every certificate carries its exact failure timestamp in notAfter. In production environments:
- Synthetic Blackbox Probing: Deploy Prometheus
blackbox_exporterto probe HTTPS endpoints externally over TCP443and exportprobe_ssl_earliest_cert_expiry. Alerting on live network probes catches cases where a renewed certificate was written to disk, butnginx -s reloadwas never executed. - Alert Thresholds: Configure a warning alert when
probe_ssl_earliest_cert_expiry - time() < 21 * 86400(21 days remaining) and a critical page at< 7 * 86400(7 days remaining). - Config Drift & Daemon Health: Monitor
systemd_unit_state{name="nginx.service",state="active"}and alert ifnginx -texits non-zero during automated configuration runs.
2. Safe Key-Rotation and Cleanup Hygiene
The root cause of this outage—an ad-hoc cleanup script moving active key material into an unsorted archive directory—is a classic operational hazard:
- Never Move Active Secrets Blindly: Archival scripts should inspect active service configurations (
nginx -T) or track symlink targets (such as Let’s Encrypt’s/etc/letsencrypt/live/<domain>/privkey.pem->../../archive/...pattern) before relocating files. - Immutable Secrets Management: Store TLS keys and certificates in a dedicated secrets manager (HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets) deployed via Infrastructure as Code (Ansible/Terraform) rather than manually copying files between
/optand/etc.
3. Documenting Public Key Pinning in Runbooks
When upstream load balancers, mobile apps, or edge ACLs pin an SPKI fingerprint:
- Record the pinning constraint prominently in the service’s runbook and in comments inside
/etc/nginx/configuration files. - Configure ACME clients (such as Certbot) with
--reuse-key(reuse_key = Truein/etc/letsencrypt/renewal/<domain>.conf) so automated renewals preserve the existing private key across rotations.
12. Further Learning and Experiments
Hands-on Experiments
Perform these safe experiments in a non-production test directory (such as /tmp/tls-lab) to reinforce your understanding of OpenSSL and Nginx TLS mechanics:
Experiment 1: Test Strict Hostname Verification Without curl -k
- Objective: Verify that your renewed certificate passes full trust-chain and SAN hostname verification without disabling TLS checks (
-k). - Command:
curl --cacert /etc/nginx/ssl/festival.crt \
--resolve www.festival.salzburg.local:443:127.0.0.1 \
https://www.festival.salzburg.local/
- What to Observe:
curlmapswww.festival.salzburg.local:443to127.0.0.1without editing/etc/hosts, trusts the self-signed certificate via--cacert, validatesDNS:www.festival.salzburg.localagainst the SAN extension, and returns the HTML page with exit code0.
Experiment 2: Observe What Happens When SANs Are Omitted
- Objective: See firsthand why modern TLS clients reject certificates that rely solely on
CN=without asubjectAltNameextension. - Commands:
mkdir -p /tmp/tls-lab && cd /tmp/tls-lab
openssl ecparam -name prime256v1 -genkey -noout -out test.key
openssl req -new -x509 -key test.key -out no-san.crt -days 1 -subj "/CN=www.festival.salzburg.local"
openssl x509 -in no-san.crt -noout -subject -ext subjectAltName
rm -rf /tmp/tls-lab
- What to Observe:
openssl x509printsNo extensions in certificateforsubjectAltName. Modern TLS clients like Gonet/httpand moderncurlbuilds rejectno-san.crtwith a hostname mismatch error despiteCN=www.festival.salzburg.local.
Experiment 3: Compare PKCS#8 and SEC1 Private Key Formats
- Objective: Understand how different PEM private key headers map to the same public key.
- Commands:
mkdir -p /tmp/tls-lab && cd /tmp/tls-lab
openssl ecparam -name prime256v1 -genkey -noout -out sec1.key
openssl pkcs8 -topk8 -nocrypt -in sec1.key -out pkcs8.key
head -n 1 sec1.key pkcs8.key
diff -s <(openssl pkey -in sec1.key -pubout) <(openssl pkey -in pkcs8.key -pubout)
rm -rf /tmp/tls-lab
- What to Observe:
sec1.keybegins with-----BEGIN EC PRIVATE KEY-----whilepkcs8.keybegins with-----BEGIN PRIVATE KEY-----, yetopenssl pkey -puboutoutputs identical SPKI public keys for both.
Topics to Explore Next
- Zero-Downtime Certificate Reloads (
nginx -s reload): How Nginx spawns new worker processes with updatedSSL_CTXstructures while older workers finish serving in-flight HTTPS connections. - ACME Protocol & Certbot
--reuse-key: Automating certificate issuance and renewal while preserving pinned public keys. - OCSP Stapling and TLS 1.3 Cipher Suites: Optimizing TLS handshake latency and hardening Nginx SSL configurations (
ssl_protocols TLSv1.2 TLSv1.3).
Authoritative Documentation and References
man openssl-pkey,man openssl-x509,man openssl-req, andman x509v3_config(OpenSSL manual pages)man nginxand the official Nginx HTTP SSL Module documentation (ngx_http_ssl_module)man systemd.service(behavior ofExecStartPre,ExecStart, andExecReload)- RFC 6125: Representation and Verification of Domain-Based Application Service Identity within Internet Public Key Infrastructure Using X.509 (PKIX) Certificates in the Context of Transport Layer Security (TLS)
13. Key Takeaways
- Separate Startup Failures from Handshake Failures: An expired certificate alone does not stop Nginx from starting; a missing or unreadable
ssl_certificate_keyfile causesnginx -t(ExecStartPre) to abort immediately. - Never Trust Key Filenames: Always verify key pairs mathematically (
openssl pkey -in <key> -puboutvs.openssl x509 -in <cert> -pubkey -nooutor a.pubfile). - Use Algorithm-Agnostic OpenSSL Commands: Prefer
openssl pkeyoveropenssl rsaso your commands work seamlessly on modern ECDSA (prime256v1) and Ed25519 keys. - Preserve Public Keys When Pinning Is Active: Pass
-key <existing_key>(never-newkey) toopenssl req -new -x509when upstream ACLs or clients pin the server’s public key. - Always Include Subject Alternative Names (SANs): Replicate all
DNS:andIP:SAN entries using-addext "subjectAltName = ..."when renewing certificates. - Lock Down Private Key Permissions: Deploy private keys with
sudo install -o root -g root -m 600so only therootmaster process can read them before dropping worker privileges.
14. Quick Troubleshooting Reference / Runbook
Use this checklist when diagnosing Nginx TLS startup or certificate issues in Linux environments:
- Check port listener and service state:
ss -tulpn | grep :443
systemctl status nginx --no-pager
- Run full Nginx configuration validation as
root:
sudo nginx -t
- Locate all configured certificate and key paths across included configs:
sudo nginx -T 2>/dev/null | grep -E "server_name|ssl_certificate"
- Inspect existing certificate Subject, Issuer, expiration dates, and SANs:
openssl x509 -in <cert_path> -noout -subject -issuer -dates -ext subjectAltName
- Verify that the certificate, private key, and any
.pubfile share the same public key:
diff -u <(openssl x509 -in <cert_path> -pubkey -noout) <(openssl pkey -in <key_path> -pubout)
- Search a directory of candidate keys for a match against a known
.pubfile:
for f in <key_dir>/*; do
diff -q <(openssl pkey -in "$f" -pubout -passin pass: 2>/dev/null) <pub_path> >/dev/null 2>&1 && echo "MATCH: $f"
done
- Install recovered private key securely and renew a self-signed certificate using the same key:
sudo cp -p <cert_path> <cert_path>.bak
sudo install -o root -g root -m 600 <matched_key> <key_path>
sudo openssl req -new -x509 -key <key_path> -out <cert_path> -days 365 \
-subj "<original_subject>" -addext "subjectAltName = <original_sans>"
sudo chmod 644 <cert_path>
- Validate configuration, start/reload Nginx, and verify the live TLS handshake:
sudo nginx -t && sudo systemctl restart nginx
openssl s_client -connect 127.0.0.1:443 -servername <hostname> </dev/null 2>/dev/null \
| openssl x509 -noout -subject -dates -ext subjectAltName