1. Overview
Transport Layer Security (TLS) and its deprecated predecessor, Secure Sockets Layer (SSL), form the cryptographic backbone of secure communication across modern computer networks. In modern Linux system administration and Site Reliability Engineering (SRE), TLS is universally employed to secure web applications (HTTPS), API endpoints, internal service-to-service communication (mTLS), database connections, and mail transport.
What Problem SSL/TLS Solves
Unencrypted network communication passes data over the wire in plain text, exposing systems to packet sniffing, man-in-the-middle (MitM) inspection, and request tampering. TLS addresses three fundamental security requirements:
- Authentication: Proves the identity of the remote server (and optionally the client) using digital certificates signed by trusted Certificate Authorities (CAs).
- Confidentiality: Encrypts application payloads using high-speed symmetric ciphers so intermediate network nodes cannot read the data.
- Integrity: Appends cryptographic Message Authentication Codes (MAC) or uses Authenticated Encryption with Associated Data (AEAD) to detect payload alteration or replay attacks in transit.
Everyday Practical Example
When an HTTP client (such as curl or a web browser) sends an unencrypted request over TCP port 80, any router on the transit path can record sensitive session tokens or inject malicious payloads. By wrapping the connection in TLS on port 443, the client and server complete an initial cryptographic handshake to verify server identity and establish temporary session keys before exchanging application data.
Unencrypted (HTTP):
[ Client ] --- ( Plaintext HTTP GET /login ) ---> [ Router / Sniffer ] ---> [ Web Server ]
Encrypted (HTTPS over TLS):
[ Client ] <=== Cryptographic Handshake & Key Agreement ===> [ Web Server ]
[ Client ] === ( Encrypted Application Payload ) =========> [ Web Server ]
Advantages and Operational Limitations
| Aspect | Benefits & Strengths | Operational Considerations & Limitations |
|---|---|---|
| Security | Prevents eavesdropping, tampering, and session hijacking across untrusted networks. | Does not protect application-layer vulnerabilities (e.g., SQL injection, XSS) or server compromise. |
| Trust Model | Leverages a hierarchical PKI trust store pre-installed across OS and browser distributions. | Requires continuous tracking of certificate validity dates and automated renewal pipelines to avoid service outages. |
| Performance | Modern TLS 1.3 completes handshakes in a single round trip (1-RTT) with negligible CPU overhead via hardware acceleration (AES-NI). | Initial connection setup introduces network round trips; legacy cipher configurations can cause handshake delays or failures. |
2. Architecture and Mental Model
To effectively diagnose TLS failures, Linux engineers must understand the hybrid cryptographic architecture and the precise sequence of events during a connection handshake.
Hybrid Cryptography: Asymmetric vs Symmetric
TLS combines asymmetric (public key) and symmetric (secret key) cryptography to balance security and performance:
- Asymmetric Cryptography (Public Key PKI): Each server holds a public-private key pair. The private key (
privkey.pem) is kept strictly secret on the server. The public key is embedded in an X.509 certificate (cert.pem) signed by a CA. Data signed with a private key can be verified using the public key, proving identity during authentication. Because asymmetric operations (RSA/ECDSA) are computationally expensive, they are restricted to authentication and initial session key negotiation. - Symmetric Cryptography (Secret Key Encryption): Once identity is established, the client and server derive identical ephemeral session keys. Subsequent application traffic (e.g., HTTP) is encrypted using fast symmetric ciphers such as AES-GCM or ChaCha20-Poly1305.
+-----------------------------------------------------------------------------------+
| TLS SESSION |
| |
| +---------------------------------------------------------------------------+ |
| | 1. Asymmetric Phase (Handshake) | |
| | - Authenticate Server Identity via Certificate Chain & CA | |
| | - Negotiate Ephemeral Symmetric Keys via ECDHE | |
| +---------------------------------------------------------------------------+ |
| | |
| v |
| +---------------------------------------------------------------------------+ |
| | 2. Symmetric Phase (Application Data Stream) | |
| | - High-Speed Bulk Payload Encryption using AES-GCM or ChaCha20 | |
| +---------------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------------+
The TLS Handshake Lifecycle
When an HTTPS connection is initiated (e.g., curl -v [https://example.com](https://example.com)), the client and server perform the following handshake sequence:
sequenceDiagram
autonumber
participant Client as Client (Browser / curl)
participant Server as Web Server / Reverse Proxy
participant CA as CA Trust Store
Client->>Server: ClientHello (TLS versions, Cipher Suites, SNI: example.com)
Server->>Client: ServerHello (Selected TLS version & Cipher Suite)
Server->>Client: Certificate (Leaf Cert + Intermediate CA Chain)
Note over Client,CA: Client verifies cert signature, expiry date,<br/>hostname match (SAN), and root chain
Server->>Client: ServerKeyExchange (ECDHE Public Parameters & Signature)
Client->>Server: ClientKeyExchange (Client ECDHE Public Parameters)
Note over Client,Server: Both sides derive shared Symmetric Session Key
Client->>Server: Finished (Encrypted verification check)
Server->>Client: Finished (Encrypted verification check)
Note over Client,Server: Secure Symmetric Channel Established (HTTP Traffic Flows)
- ClientHello: The client sends its maximum supported TLS version, supported cipher suites, compression methods, and the Server Name Indication (SNI) extension containing the targeted domain name.
- ServerHello: The server selects the highest mutually supported TLS version and preferred cipher suite.
- Certificate Presentation: The server presents its X.509 certificate and intermediate CA chain (
fullchain.pem). - Certificate Verification: The client checks that the certificate is signed by a trusted root CA in its system trust store, is within its validity window (
notBefore/notAfter), matching the requested domain in the Subject Alternative Name (SAN) extension, and is not revoked. - Key Exchange (ECDHE): Using Elliptic Curve Diffie-Hellman Ephemeral (ECDHE) exchange, both parties exchange key parameters to independently generate identical symmetric session keys.
- Finished: Both endpoints send encrypted validation messages to verify that the key exchange succeeded without interception. Encrypted application payloads begin flowing over the established channel.
Server Name Indication (SNI) and Virtual Hosting
In HTTP/1.1, multiple domains are hosted on a single IP address using the Host: request header. However, during a TLS handshake, the TCP connection and TLS session are established before any HTTP request header is sent.
Server Name Indication (SNI) solves this bootstrap problem by requiring the client to send the desired hostname inside the initial ClientHello packet. This enables reverse proxies (Nginx, HAProxy, AWS ALB) to select and present the correct TLS certificate before negotiating the session.
ClientHello (SNI: app1.example.com) ----> [ Reverse Proxy / ALB ]
|
v Selects Certificate:
app1.example.com.crt
3. Core Concepts and Terminology
X.509 Certificate Structure
An X.509 certificate is a digital document formatted in ASN.1 and encoded in PEM (Base64) or DER (Binary). It contains standard fields required for validation:
- Subject / Common Name (CN): The legacy field identifying the primary hostname (e.g.,
example.com). Modern browsers ignore the CN field and strictly mandate the Subject Alternative Name (SAN) extension. - Subject Alternative Names (SAN): An extension listing all valid hostnames, wildcards (
*.example.com), or subdomains covered by the certificate. - Issuer: The Certificate Authority that digitally signed the certificate (e.g.,
Let's Encrypt Authority X3,DigiCert). - Validity Window: The
notBeforeandnotAfterUTC timestamps defining the operational lifecycle. - Public Key: The public key material (RSA or ECDSA) and key parameters.
- Certificate Chain: Intermediate CA certificates that bridge trust between the server’s leaf certificate and a trusted Root CA pre-installed in the operating system’s trust store.
RSA vs. ECDSA Algorithms
- RSA (Rivest-Shamir-Adleman): The legacy public-key standard based on prime factorization. Commonly uses key sizes of 2048-bit or 4096-bit.
- ECDSA (Elliptic Curve Digital Signature Algorithm): Modern asymmetric cryptography based on elliptic curve mathematics (e.g., Curve25519 or P-256). Provides equivalent security to RSA at much smaller key sizes (256-bit ECDSA ~ 3072-bit RSA), leading to faster handshakes and lower CPU/memory consumption.
Automated Certificate Management Environment (ACME)
ACME is an IETF-standardized protocol (RFC 8555) used by free CAs such as Let’s Encrypt to automate certificate issuance, domain verification, and renewal. Verification is completed via challenges:
- HTTP-01 Challenge: The ACME client (e.g., Certbot) places a token at
http://<domain>/.well-known/acme-challenge/<TOKEN>. Let’s Encrypt validates the token over port 80. - DNS-01 Challenge: The ACME client creates a DNS
TXTrecord at_acme-challenge.<domain>containing a cryptographic digest. This is required for issuing wildcard certificates (*.example.com) and securing servers inaccessible from the public internet.
Certificate Transparency (CT) Logs and crt.sh
Certificate Transparency is an open framework of publicly auditable, append-only logs. Public CAs must log every issued certificate to CT logs before handing it to the requester. SREs use CT search engines like crt.sh to discover unauthorized issuances, inspect historical certs, and audit subdomains.
OCSP and OCSP Stapling
The Online Certificate Status Protocol (OCSP) allows clients to check if a certificate has been revoked by the CA prior to expiration.
- Standard OCSP: The client contacts the CA’s OCSP responder directly during every TLS connection, causing privacy leaks and handshake delays.
- OCSP Stapling: The web server periodically fetches a signed, time-stamped OCSP response from the CA and “staples” it to the initial TLS handshake, eliminating extra client-side DNS and HTTP lookups.
Glossary of Common File Extensions
.pem(Privacy Enhanced Mail): Base64 encoded text file enclosed between-----BEGIN ...-----and-----END ...-----headers. Used for keys, certs, and chains..crt/.cer: Public certificate files (usually PEM format)..key: Private key files (PEM format)..csr: Certificate Signing Request submitted to a CA for signing.
4. Installation and Version Management
Installing OpenSSL and Certbot
Debian / Ubuntu Systems
# Update package repositories
sudo apt update
# Install OpenSSL diagnostic utilities and Certbot with Nginx/Apache plugins
sudo apt install -y openssl certbot python3-certbot-nginx python3-certbot-apache curl jq
RHEL / Rocky Linux / AlmaLinux Systems
# Enable EPEL repository (required for Certbot on RHEL family)
sudo dnf install -y epel-release
# Install OpenSSL and Certbot plugins
sudo dnf install -y openssl certbot python3-certbot-nginx python3-certbot-apache curl jq
Kubernetes Environments
Deploy cert-manager via Helm to handle containerized ingress certificate lifecycle management:
# Add Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io --force-update
helm repo update
# Install cert-manager Custom Resource Definitions (CRDs) and chart
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true
Identifying Installed Tool Versions
# Inspect OpenSSL binary version and build flags
openssl version -a
# Inspect Certbot CLI version
certbot --version
# Verify Kubernetes cert-manager deployment version
kubectl get pods -n cert-manager
5. Configuration Files and Directory Structure
Central Directory and File Reference
| File / Directory Path | Purpose | Important Details & Security Requirements |
|---|---|---|
/etc/letsencrypt/live/<domain>/ |
Active symlinks pointing to current certificate assets. | Contains fullchain.pem, privkey.pem, cert.pem, and chain.pem. |
/etc/letsencrypt/archive/<domain>/ |
Versioned storage of all historical keys and certs. | Symlinks in /live/ point to numbered files in this directory. |
/etc/letsencrypt/renewal/<domain>.conf |
Certbot renewal parameters and configuration settings. | Stores ACME account details, webroot paths, and renewal hooks. |
/etc/ssl/certs/ |
Default CA trust store directory on Debian/Ubuntu systems. | Contains bundled root CA certificates used by system utilities. |
/etc/pki/tls/certs/ |
Default CA trust store directory on RHEL/CentOS systems. | Managed by update-ca-trust package utilities. |
fullchain.pem |
Server certificate concatenated with intermediate CA certificates. | Must be served to clients by web servers to prevent trust errors. |
privkey.pem / .key |
Server private key file. | Must be restricted to permissions 0600 or 0640 owned by root:root or web service user. |
Web Server Configuration Syntax
Nginx TLS Configuration Example (/etc/nginx/conf.d/example.conf)
server {
listen 80;
listen [::]:80;
server_name example.com www.example.com;
# Redirect all HTTP traffic to HTTPS
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name example.com www.example.com;
# Certificate & Private Key Paths
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; #
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; #
# Protocol & Cipher Hardening
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
# OCSP Stapling Configuration
ssl_stapling on; #
ssl_stapling_verify on; #
ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem; #
# HTTP Strict Transport Security (HSTS)
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
location / {
root /var/www/html;
index index.html;
}
}
HAProxy TLS Configuration Example (/etc/haproxy/haproxy.cfg)
global
# Define default SSL settings
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
frontend https_in
# HAProxy requires cert and private key combined in one .pem file
bind :443 ssl crt /etc/haproxy/certs/example.com.pem alpn h2,http/1.1
mode http
default_backend web_servers
6. Service Management and Common Operations
Checking Service Timers and Processes
Certbot automatically manages renewal tasks via systemd timers or cron jobs.
# Check status of Certbot systemd renewal timer
systemctl status certbot.timer
# List all active systemd timers to verify next renewal execution
systemctl list-timers | grep certbot #
# Inspect running web server processes bound to TLS ports
ss -tulpn | grep -E ':80|:443'
Validating Configuration Changes Before Reload
Modifying TLS certificates or configuration directives can break web services if syntax is invalid or key moduli mismatch. Always validate configurations before reloading daemons.
# Validate Nginx configuration syntax
sudo nginx -t
# Validate HAProxy configuration syntax
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
# Reload web services without dropping active client connections
sudo systemctl reload nginx
sudo systemctl reload haproxy
7. Common Administrative Tasks and Workflows
Workflow 1: Generate Self-Signed Certificate for Local Development
Use OpenSSL to generate a 2048-bit RSA private key and self-signed certificate valid for 365 days without passphrase prompting (-nodes):
openssl req -x509 -newkey rsa:2048 -nodes \
-keyout /etc/ssl/private/dev-selfsigned.key \
-out /etc/ssl/certs/dev-selfsigned.crt \
-days 365 \
-subj "/CN=localhost/O=Development" #
Workflow 2: Generate Private Key and Certificate Signing Request (CSR)
When requesting a certificate from a commercial CA (e.g., DigiCert, Sectigo), generate a private key and CSR file locally:
# 1. Generate RSA 2048-bit key and CSR
openssl req -new -newkey rsa:2048 -nodes \
-keyout example.com.key \
-out example.com.csr \
-subj "/C=US/ST=California/L=SanFrancisco/O=MyCompany/CN=example.com" #
# 2. Inspect CSR contents to verify subject and extensions
openssl req -in example.com.csr -text -noout #
Workflow 3: Obtain Let’s Encrypt Certificate via Certbot (HTTP-01)
Obtain a certificate using the Nginx plugin, which automatically alters config files and requests the certificate:
# Interactive issuance using Nginx web server plugin
sudo certbot certonly --nginx -d example.com -d www.example.com #
# Non-interactive webroot issuance (useful for custom web server setups)
sudo certbot certonly --webroot -w /var/www/html -d example.com -d www.example.com #
# Test automatic renewal workflow via dry-run simulation
sudo certbot renew --dry-run #
Workflow 4: Configure cert-manager ClusterIssuer in Kubernetes
Deploy an ACME ClusterIssuer in Kubernetes to manage cluster-wide TLS certificates:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@example.com
privateKeySecretRef:
name: letsencrypt-prod-account-key
solvers:
- http01:
ingress:
class: nginx
Apply an Ingress manifest annotated for automated provisioning:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
namespace: default
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod" #
spec:
ingressClassName: nginx
tls:
- hosts:
- app.example.com
secretName: app-example-tls #
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app-service
port:
number: 80
Workflow 5: Validate Private Key and Certificate Modulus Match
If a server fails to start after certificate replacement, verify that the public modulus of the private key and certificate are identical:
cert_md5=$(openssl x509 -noout -modulus -in /etc/letsencrypt/live/example.com/fullchain.pem | openssl md5) #
key_md5=$(openssl rsa -noout -modulus -in /etc/letsencrypt/live/example.com/privkey.pem | openssl md5) #
if [ "$cert_md5" = "$key_md5" ]; then
echo "MATCH: Private key corresponds to certificate." #
else
echo "MISMATCH: Private key does NOT belong to this certificate!" #
fi
8. CLI Command Reference and Cheatsheet
Remote TLS Testing with openssl s_client
| Command | Purpose | Important Flags | Example | Explanation |
|---|---|---|---|---|
openssl s_client |
Initiates TLS handshake to remote host. | -connect: host:port |
-servername: SNI hostname | openssl s_client -connect example.com:443 -servername example.com | Executes full TLS handshake using SNI to inspect returned certificates. |
| openssl s_client | Displays complete certificate chain. | -showcerts: print raw PEM certificates | openssl s_client -connect example.com:443 -showcerts | Prints leaf and intermediate certificates presented by the server. |
| openssl s_client | Tests specific TLS protocol version. | -tls1_2, -tls1_3: force protocol version | openssl s_client -connect example.com:443 -tls1_2 | Validates whether server supports legacy or required protocol versions. |
| openssl s_client | Verifies OCSP Stapling. | -status: request OCSP status in handshake | openssl s_client -status -connect example.com:443 -servername example.com | Checks if server staples valid OCSP revocation response. |
Inspecting Local Certificate Files with openssl x509
| Command | Purpose | Important Flags | Example | Explanation |
|---|---|---|---|---|
openssl x509 |
Prints certificate details. | -text, -noout: suppress encoded dump |
openssl x509 -in cert.pem -text -noout |
Displays human-readable subject, issuer, SANs, and key parameters. |
openssl x509 |
Checks certificate expiry dates. | -dates: display notBefore and notAfter |
openssl x509 -in cert.pem -noout -dates |
Outputs exact validity timestamps to diagnose expired certificates. |
openssl x509 |
Lists Subject Alternative Names. | -ext subjectAltName: inspect SAN extension |
openssl x509 -in cert.pem -noout -ext subjectAltName |
Displays all hostnames covered by the certificate. |
openssl verify |
Validates cert against CA chain. | -CAfile: path to CA bundle |
openssl verify -CAfile chain.pem cert.pem |
Verifies leaf certificate trust chain against intermediate/root CAs. |
Certbot Management Commands
| Command | Purpose | Important Flags | Example | Explanation |
|---|---|---|---|---|
certbot certonly |
Obtains certificate without editing config. | --nginx, --webroot, --standalone |
certbot certonly --webroot -w /var/www/html -d example.com |
Solves ACME challenge and saves certificates to /etc/letsencrypt/live/. |
certbot renew |
Renews certificates near expiration. | --dry-run: simulate renewal |
certbot renew --dry-run |
Tests ACME challenge workflows and hooks without requesting real certs. |
certbot certificates |
Lists all managed certificates. | None | certbot certificates |
Prints domain names, paths, expiry dates, and validation status. |
Kubernetes cert-manager Diagnostic Commands
| Command | Purpose | Example | Explanation |
|---|---|---|---|
kubectl get certificate |
Summarize certificate status in namespace. | kubectl get certificate -A |
Displays ready status, secret names, and renewal windows. |
kubectl describe certificate |
Inspect certificate issuance events. | kubectl describe certificate my-tls -n prod |
Lists detailed events, challenge references, and failure error messages. |
kubectl describe challenge |
Diagnose ACME challenge failures. | kubectl describe challenge -n prod |
Displays status of HTTP-01 or DNS-01 verification attempts. |
9. Logs, Monitoring and Diagnostics
Log File Locations and Systemd Integration
- Certbot Execution Logs: Logged to
/var/log/letsencrypt/letsencrypt.log. - Systemd Timer Logs: Inspected using
journalctl -u certbot.service. - Nginx TLS Handshake Errors: Logged to
/var/log/nginx/error.log. - Kubernetes cert-manager Controller Logs: Inspected via
kubectl logs -n cert-manager deploy/cert-manager.
Key Diagnostic Commands for Troubleshooting
1. Checking Expiry Date of a Remote Endpoint
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
| openssl x509 -noout -dates #
2. Querying Public Certificate Transparency (CT) Logs via Shell
# Query crt.sh API for all issued subdomains matching a target domain
curl -s "https://crt.sh/?q=%25.example.com&output=json" | jq -r '.[].name_value' | sort -u #
3. Simulating HTTP-01 Challenge Endpoint Reachability
# Verify port 80 path is accessible locally for ACME verification
curl -I http://example.com/.well-known/acme-challenge/test-token
10. Troubleshooting Guide and Common Failure Scenarios
flowchart TD
A[TLS Connection Failure or Warning] --> B{What is the symptom?}
B -->|Browser: Expired Cert| C[Check Expiry Dates: openssl s_client / x509]
B -->|Browser: ERR_CERT_COMMON_NAME_INVALID| D[Check SAN Extension: openssl x509 -ext subjectAltName]
B -->|Browser: Trust Unknown CA| E[Check Cert Chain: openssl s_client -showcerts]
B -->|Handshake Failure / Reset| F[Check Protocol & Cipher Compatibility / Port 443]
C --> C1[Renew via Certbot / cert-manager]
D --> D1[Re-issue Cert with missing -d domain or check SNI config]
E --> E1[Serve fullchain.pem instead of cert.pem]
F --> F1[Adjust ssl_protocols & ciphers in web server config]
Scenario 1: Certificate Expired (ERR_CERT_DATE_INVALID)
Symptoms
Browsers display a prominent security warning stating NET::ERR_CERT_DATE_INVALID or “Your connection is not private”. Automated HTTP clients return TLS verification errors (SSL certificate problem: certificate has expired).
Possible Causes
- Certbot systemd renewal timer failed or was disabled.
- The web server was not reloaded after Certbot renewed certificate files on disk.
- ACME HTTP-01 challenge failed due to network firewalls blocking port 80.
Investigation and Diagnosis
- Inspect remote endpoint validity timestamps:
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -dates #
- Inspect local certificate files managed by Certbot:
sudo certbot certificates #
- Check Certbot renewal timer status:
systemctl status certbot.timer #
journalctl -u certbot.service --since "7 days ago"
Resolution
Force immediate manual renewal and reload the web server:
sudo certbot renew --force-renewal #
sudo systemctl reload nginx #
Verification
Re-run the remote checking command to verify that notAfter reflects a future date:
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -dates #
Prevention
Ensure deploy hooks are configured in /etc/letsencrypt/renewal/example.conf to automatically reload web servers upon successful renewal: renew_hook = systemctl reload nginx.
Scenario 2: Hostname Mismatch (ERR_CERT_COMMON_NAME_INVALID)
Symptoms
Browsers refuse connection with ERR_CERT_COMMON_NAME_INVALID. curl commands return curl: (60) SSL: certificate subject name does not match target host name.
Possible Causes
- The domain requested by the client is missing from the certificate’s SAN list.
- The certificate was issued only for
example.com, but the user accessed[www.example.com](https://www.example.com).
Investigation and Diagnosis
Inspect the Subject Alternative Name (SAN) extension presented by the server:
openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
| openssl x509 -noout -ext subjectAltName #
If output displays DNS:example.com but omits DNS:[www.example.com](https://www.example.com), the SAN list is incomplete.
Resolution
Re-issue the certificate with all required hostnames explicitly included using Certbot:
sudo certbot certonly --nginx -d example.com -d www.example.com --allow-subset-of-names #
sudo systemctl reload nginx
Verification
Query SAN extension again to confirm all domain variants are present:
openssl s_client -connect www.example.com:443 -servername www.example.com 2>/dev/null \
| openssl x509 -noout -ext subjectAltName #
Scenario 3: SNI vs. Bare IP Access Confusion
Symptoms
Connecting to a server via domain name succeeds (curl [https://example.com](https://example.com)), but accessing the server directly via IP address (curl [https://203.0.113.10](https://203.0.113.10)) fails with hostname mismatch warnings or presents a default catch-all certificate.
Possible Causes
When an HTTP client connects to a raw IP address, no host domain name is sent in the SNI extension during the TLS handshake. The web server cannot match an SNI header and falls back to serving its default virtual host certificate.
curl https://203.0.113.10/ --> Sends NO SNI --> Server serves default (wrong) cert
curl https://example.com/ --> Sends SNI: example.com --> Server serves correct cert
Investigation and Diagnosis
- Test connection via raw IP without SNI:
curl -vk https://203.0.113.10/ #
- Test connection to IP while explicitly forcing the SNI hostname header:
openssl s_client -connect 203.0.113.10:443 -servername example.com #
If the second command succeeds, the issue is purely caused by missing SNI during direct IP access.
Resolution
- Update health checks, monitoring tools, and internal scripts to use domain names rather than bare IP addresses.
- If testing against a specific IP before DNS propagation, use
curl’s--resolveflag to pass SNI correctly:
curl --resolve example.com:443:203.0.113.10 https://example.com/ #
Scenario 4: Wrong Certificate Served in Multi-Vhost or Load Balancer Environment
Symptoms
The correct certificate exists on disk or in AWS ACM, but external clients receive a certificate belonging to a completely different domain hosted on the same infrastructure.
Possible Causes
- Nginx: Multiple
serverblocks share port 443, but a catch-all block is set asdefault_server, or SNI matching fails due to missingserver_namedirectives. - AWS ALB: The listener certificate list does not include the target domain, causing the ALB to fall back to the default listener certificate.
Investigation and Diagnosis
- Compare certificates returned via public DNS vs. direct connection to load balancer backend IPs:
# Query via DNS
openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
| openssl x509 -noout -subject -issuer #
# Query direct load balancer IP with SNI
openssl s_client -connect 203.0.113.10:443 -servername example.com 2>/dev/null \
| openssl x509 -noout -subject -issuer #
- Audit active Nginx server configuration blocks:
nginx -T | grep -E 'server_name|ssl_certificate|listen' #
Resolution
- Nginx Fix: Ensure the
server_namedirective precisely matches the SNI domain, and verify that default catch-all blocks do not intercept port 443 traffic unexpectedly. - AWS ALB Fix: Add the domain certificate to the ALB listener’s SNI Certificate List via AWS Console or CLI:
aws elbv2 add-listener-certificates \
--listener-arn <LISTENER_ARN> \
--certificates CertificateArn=<ACM_CERT_ARN>
Scenario 5: Incomplete Certificate Chain (Missing Intermediates)
Symptoms
Modern desktop web browsers open the website without errors (because they cache intermediate CAs), but mobile browsers, CLI tools (curl), or automated API clients fail with SEC_ERROR_UNKNOWN_ISSUER or unable to get local issuer certificate.
Possible Causes
The web server is configured to serve only the leaf server certificate (cert.pem) instead of the combined server and intermediate CA bundle (fullchain.pem).
Investigation and Diagnosis
Execute openssl s_client with -showcerts to count the certificates presented during the handshake:
openssl s_client -connect example.com:443 -servername example.com -showcerts #
If the output displays only one BEGIN CERTIFICATE block, the intermediate CA chain is missing. A complete chain contains two or more certificate blocks.
Resolution
Update web server configuration files to point to the full certificate chain:
- Nginx: Change
ssl_certificate /etc/letsencrypt/live/[example.com/cert.pem](https://example.com/cert.pem);tofullchain.pem. - Apache: Ensure
SSLCertificateChainFileorSSLCertificateFilepoints tofullchain.pem.
Reload the service:
sudo systemctl reload nginx #
Scenario 6: Private Key Does Not Match Certificate
Symptoms
Web server daemons fail to start or reload, logging errors such as SSL_CTX_use_PrivateKey_file failed: error:0B080074:x509 certificate routines:key values mismatch.
Possible Causes
During certificate renewal or manual editing, the certificate file was updated but the private key file on disk belongs to a different keypair.
Investigation and Diagnosis
Compare the MD5 cryptographic hashes of the public moduli derived from both files:
cert_hash=$(openssl x509 -noout -modulus -in /etc/letsencrypt/live/example.com/fullchain.pem | openssl md5) #
key_hash=$(openssl rsa -noout -modulus -in /etc/letsencrypt/live/example.com/privkey.pem | openssl md5) #
echo "Cert Modulus MD5: $cert_hash" #
echo "Key Modulus MD5: $key_hash" #
If the hash strings do not match identically, the files are mismatched.
Resolution
Restore the matching private key from backup or re-issue the certificate keypair using Certbot:
sudo certbot certonly --force-renewal --nginx -d example.com #
Scenario 7: Protocol or Cipher Mismatch (Legacy Clients)
Symptoms
Modern desktop web browsers connect successfully, but older embedded clients, legacy Java applications, or partner API webhooks fail during TLS negotiation with handshake_failure or connection reset by peer.
Possible Causes
The web server was hardened to require TLS 1.3 or strict cipher suites that legacy client libraries do not support.
Investigation and Diagnosis
Test handshakes explicitly enforcing older protocol versions:
# Test TLS 1.2 handshake
openssl s_client -connect example.com:443 -servername example.com -tls1_2 #
# Test TLS 1.1 handshake (legacy)
openssl s_client -connect example.com:443 -servername example.com -tls1_1 #
Inspect verbose handshake messaging using -state and -msg flags:
openssl s_client -connect example.com:443 -servername example.com -tls1_2 -state -msg #
Resolution
If legacy client support is mandatory, update web server settings to enable TLS 1.2 alongside TLS 1.3 and include compatible ciphers:
ssl_protocols TLSv1.2 TLSv1.3; #
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384; #
Scenario 8: Broken or Missing OCSP Stapling
Symptoms
TLS connections experience noticeable delays (1-3 seconds) during initial handshake, or strict security scanners flag missing OCSP responses.
Possible Causes
- The web server cannot reach the CA’s external OCSP responder over outbound port 443/80.
- Missing intermediate chain configuration required for verifying OCSP stapled responses.
Investigation and Diagnosis
Query OCSP status explicitly using openssl s_client:
openssl s_client -status -connect example.com:443 -servername example.com #
Inspect the output for the OCSP response block:
OCSP Response Data:
OCSP Response Status: successful (0x0) #
If the response reads OCSP response: no response sent, stapling is not functioning.
Resolution
Configure Nginx with ssl_trusted_certificate pointing explicitly to the CA intermediate chain:
ssl_stapling on; #
ssl_stapling_verify on; #
ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem; #
Ensure egress firewall rules permit outbound traffic to CA OCSP endpoints.
Scenario 9: Certbot Renewal Failure (ACME Challenge Fails)
Symptoms
Certbot renewal commands exit with errors: Certbot failed to authenticate some domains... Unauthorized / Connection refused.
Possible Causes
- Inbound HTTP traffic on port 80 is blocked by network firewalls or security groups.
- Nginx configuration blocks requests to
.well-known/acme-challenge/.
Investigation and Diagnosis
- Run Certbot renewal in dry-run mode:
sudo certbot renew --dry-run #
- Inspect log files for detailed ACME API error traces:
less /var/log/letsencrypt/letsencrypt.log #
Resolution
Ensure port 80 is open to external networks and explicitly allow ACME challenge locations in web server configs:
location /.well-known/acme-challenge/ {
root /var/www/html;
}
Scenario 10: Kubernetes cert-manager Certificate Stuck in Pending
Symptoms
Kubernetes ingress resources fail to present valid TLS certificates, and kubectl get certificate shows READY: False.
Possible Causes
- Misconfigured
ClusterIssuercredentials or ACME email. - Ingress path rules blocking HTTP-01 challenge pods created by
cert-manager.
Investigation and Diagnosis
Inspect the status and event logs across cert-manager resources:
kubectl describe certificate app-tls-cert -n prod #
kubectl describe certificaterequest -n prod #
kubectl describe challenge -n prod #
Resolution
If challenges fail due to ingress path mismatches, switch to DNS-01 challenge solvers using cloud DNS provider credentials (e.g., AWS Route53, Cloudflare).
11. Security and Best Practices
1. Protocol Restrictions
- Disable Outdated Protocols: Disable SSLv2, SSLv3, TLS 1.0, and TLS 1.1 completely. Allow only TLS 1.2 and TLS 1.3.
2. Strict Key Protection and Permissions
- Private key files (
.key/privkey.pem) must be restricted:
chmod 600 /etc/letsencrypt/live/example.com/privkey.pem #
chown root:root /etc/letsencrypt/live/example.com/privkey.pem
3. Enforce HTTP Strict Transport Security (HSTS)
HSTS prevents SSL stripping attacks by forcing browsers to interact with the site strictly over HTTPS:
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
4. Perfect Forward Secrecy (PFS)
Use ephemeral Diffie-Hellman key exchange ciphers (ECDHE). PFS ensures that even if a server’s private key is compromised in the future, past recorded encrypted session traffic cannot be decrypted.
12. Performance and Resource Optimisation
1. ECDSA vs. RSA Key Performance
Prefer ECDSA (P-256) keys over standard RSA 2048/4096-bit keys. ECDSA key generation and signature operations require significantly fewer CPU cycles, accelerating handshakes and increasing server throughput.
# Generate an ECDSA P-256 private key and CSR
openssl req -new -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
-nodes -keyout ecdsa.key -out ecdsa.csr -subj "/CN=example.com"
2. TLS Session Resumption
Reduce CPU load and network round trips by enabling session caching and session tickets:
# Nginx Session Caching Settings
ssl_session_cache shared:SSL:10m; # ~40,000 sessions
ssl_session_timeout 1d;
ssl_session_tickets off; # Disable tickets if key rotation isn't managed
13. Backup, Recovery and Maintenance
Essential Directories for Backup
To restore TLS operations in a disaster recovery scenario, back up the following directories:
/etc/letsencrypt/(Contains active certs, private keys, archive files, and renewal configs).- Custom web server certificates (
/etc/ssl/private/,/etc/haproxy/certs/).
# Create an encrypted backup archive of Let's Encrypt directory
sudo tar -czvf /safe/location/letsencrypt_backup_$(date +%F).tar.gz /etc/letsencrypt/
Key and Certificate Restoration
# Extract backup archive
sudo tar -xzvf letsencrypt_backup_2026-11-03.tar.gz -C /
# Verify ownership and file permissions
sudo chmod -R 755 /etc/letsencrypt/live/
sudo chmod -R 600 /etc/letsencrypt/live/*/privkey.pem #
# Reload web server
sudo systemctl reload nginx #
14. Automation and Integration
Certbot Deploy Hooks for Web Server Auto-Reload
Ensure web services reload automatically whenever Certbot renews a certificate by placing executable scripts in /etc/letsencrypt/renewal-hooks/deploy/:
#!/bin/bash
# File: /etc/letsencrypt/renewal-hooks/deploy/01-reload-nginx.sh
systemctl reload nginx #
Make the hook executable:
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/01-reload-nginx.sh
15. Practical Labs and Exercises
Lab 1: Certificate Modulus Matching Exercise
Objective: Practice verifying private key and certificate pairings.
- Generate a self-signed certificate keypair:
openssl req -x509 -newkey rsa:2048 -nodes -keyout test.key -out test.crt -days 30 -subj "/CN=test.local" #
- Generate a second keypair:
openssl req -x509 -newkey rsa:2048 -nodes -keyout wrong.key -out wrong.crt -days 30 -subj "/CN=test.local" #
- Extract and compare public moduli MD5 hashes using bash shell commands:
[ "$(openssl x509 -noout -modulus -in test.crt | openssl md5)" = "$(openssl rsa -noout -modulus -in test.key | openssl md5)" ] && echo "Match!" || echo "Mismatch!" #
Lab 2: Troubleshooting an Incomplete Certificate Chain
Objective: Diagnose and fix a missing intermediate certificate using openssl s_client.
- Configure Nginx to serve
cert.peminstead offullchain.pem. - Execute remote inspection:
openssl s_client -connect localhost:443 -showcerts #
- Observe that only one certificate block is returned.
- Update configuration to serve
fullchain.pemand reload Nginx. - Re-run command to confirm multiple certificate blocks are presented.
16. Quick Reference and Cheat Sheet
Important Configuration Directories
| Path | Purpose |
|---|---|
/etc/letsencrypt/live/<domain>/ |
Active symlinks for Let’s Encrypt certificates. |
/etc/letsencrypt/renewal/ |
Certbot renewal configuration files. |
/etc/ssl/certs/ |
System trust store on Debian/Ubuntu. |
Important Log Locations
| Service / Tool | Log Path |
|---|---|
| Certbot ACME Engine | /var/log/letsencrypt/letsencrypt.log |
| Nginx Web Server | /var/log/nginx/error.log |
| Kubernetes cert-manager | kubectl logs -n cert-manager deploy/cert-manager |
Essential OpenSSL Commands
| Task | Command |
|---|---|
| Test Remote Handshake & SNI | openssl s_client -connect example.com:443 -servername example.com |
| Show Remote Cert Chain | openssl s_client -connect example.com:443 -showcerts |
| Check Cert Expiry Dates | openssl x509 -in cert.pem -noout -dates |
| Inspect SAN Extension | openssl x509 -in cert.pem -noout -ext subjectAltName |
| Compare Modulus Hash | `openssl x509 -noout -modulus -in cert.pem |
Troubleshooting Quick Reference
| Symptom | Initial Diagnostic Command | Primary Cause |
|---|---|---|
ERR_CERT_DATE_INVALID |
`openssl s_client -connect host:443 | openssl x509 -noout -dates` |
ERR_CERT_COMMON_NAME_INVALID |
openssl x509 -in cert.pem -noout -ext subjectAltName |
Target domain missing from SAN list. |
| Unknown CA Warning | openssl s_client -connect host:443 -showcerts |
Server serving cert.pem instead of fullchain.pem. |
| Key Mismatch Error | Compare MD5 moduli of .crt and .key |
Private key does not correspond to cert. |
Important Ports and Protocols
| Port | Protocol | Usage |
|---|---|---|
| 80 | TCP | HTTP / Let’s Encrypt HTTP-01 Challenge |
| 443 | TCP | HTTPS / TLS Encrypted Application Data |
| 53 | UDP/TCP | DNS / Let’s Encrypt DNS-01 Challenge |
17. Further Reading and References
- Let’s Encrypt Documentation - How It Works: letsencrypt.org/how-it-works
- Certbot Official Documentation: eff-certbot.readthedocs.io
- cert-manager Kubernetes Documentation: cert-manager.io/docs
- OpenSSL s_client Manual Page: man7.org - openssl s_client
- crt.sh Certificate Transparency Search: crt.sh