1. Overview
Caddy is a modern, open-source web server and reverse proxy written in Go. It is designed from the ground up around memory safety, modularity, and automated infrastructure operations. While traditional servers like NGINX or Apache HTTP Server require external tools (such as Certbot or acme.sh) and explicit cron jobs to provision and renew TLS certificates, Caddy integrates public key infrastructure (PKI) management directly into its core engine.
+-----------------------------------------------------------------------+
| CADDY SERVER |
| |
| +-------------------+ +--------------------+ +----------------+ |
| | HTTP/HTTPS Core | | Automated ACME Engine | | Static Server | |
| | (TLS 1.3 / QUIC) | | (Let's Encrypt/ZeroSSL)| | & Reverse Proxy| |
| +---------+---------+ +---------+----------+ +-------+--------+ |
+------------|-----------------------|----------------------|-----------+
| | |
v v v
Port 80/443 Traffic Automated TLS Certs Upstream Applications
Core Responsibilities in Production
- HTTP/HTTPS Termination: Accepts incoming HTTP/1.1, HTTP/2, and HTTP/3 (QUIC) client connections, terminates TLS encryption, and handles cipher negotiation.
- Reverse Proxying: Forwards client traffic to backend application servers (written in Go, Python, Node.js, Rust, Java, or PHP-FPM) with support for load balancing and active/passive health checks.
- Static Asset Serving: Delivers static files from local disk with high performance using modern compression algorithms like Zstandard and Gzip.
- Automatic Certificate Management: Communicates automatically with ACME certificate authorities (such as Let’s Encrypt or ZeroSSL) to request, obtain, and renew TLS certificates without service interruption.
Problems Solved by Caddy
- Certbot Complexity & Expiry Outages: Traditional setups depend on shell scripts, certbot renewals, and reload hooks. If a renewal hook fails, certificates silently expire. Caddy manages the complete certificate lifecycle in memory and on disk automatically.
- Verbose Configuration Boilerplate: Complex NGINX or Apache configurations often run into hundreds of lines for basic SSL settings, redirect rules, and security headers. Caddy accomplishes the same output with concise site blocks.
- Configuration Reload Disruptions: Caddy utilizes atomic, zero-downtime configuration reloads. When configuration changes occur, active TCP sockets are not abruptly dropped.
Feature Comparison
| Feature | Caddy | NGINX | Apache HTTP Server |
|---|---|---|---|
| Language & Safety | Go (Memory safe) | C (Manual memory management) | C (Manual memory management) |
| Automatic TLS (ACME) | Built-in by default | Requires Certbot / external scripts | Requires Certbot / external scripts |
| HTTP/3 (QUIC) Support | Native out-of-the-box | Requires specific compile flags | Requires external modules |
| Default Config Format | Caddyfile or JSON | NGINX DSL | Directive-based (httpd.conf) |
| Dynamic API Control | REST API on Port 2019 | NGINX Plus (Paid) or NJS | Limited / mod_status |
| Reload Mechanism | In-memory atomic swap | Worker binary signal | Process grace restart |
Production Use Case Example
Consider a backend Node.js microservice listening locally on 127.0.0.1:8080. To expose this application to the public internet securely at api.example.com, Caddy requires only two lines of configuration:
api.example.com {
reverse_proxy 127.0.0.1:8080
}
When Caddy starts with this block, it binds to ports 80 and 443, detects that api.example.com is a public domain, contacts Let’s Encrypt to complete ACME challenges, provisions a valid TLS certificate, sets up HTTPS redirection on port 80, and begins proxying traffic to port 8080.
2. Architecture and Mental Model
To operate Caddy reliably in production, SysAdmins and SREs must understand how Caddy processes requests internally and how it manages configuration state.
graph TD
Client[Client Browser / API Client] -->|TCP Connect :80 / :443| Socket[Socket Accept Listener]
Socket -->|Port 80 HTTP| Redirect[HTTP to HTTPS Redirect]
Socket -->|Port 443 HTTPS| TLS[TLS Handshake & SNI Check]
TLS -->|Obtain / Validate Cert| ACME[ACME Engine / Cert Cache]
TLS --> Engine[Caddy HTTP Core Engine]
Engine --> Matcher{Site Block & Path Matcher}
Matcher -->|/api/*| Proxy[reverse_proxy Handler]
Matcher -->|/static/*| Static[file_server Handler]
Matcher -->|Redirect Rule| Redir[redir Handler]
Proxy -->|TCP Connection| Upstream[App Server 127.0.0.1:8000]
Static -->|Disk Read| Filesystem[/var/www/html]
Upstream --> Logger[Structured JSON Logger]
Filesystem --> Logger
Redir --> Logger
Logger --> Client
1. Modular Core Architecture
At its core, Caddy is a light runtime module loader. The Caddy core engine manages the server lifecycle and dynamic JSON configurations. Every piece of functional code—including the HTTP web server, the TLS manager, the reverse proxy, and custom plugins—is an independent module loaded into the core engine.
- Host Modules: Parent modules that load guest modules (e.g., the
httpapp module). - Guest Modules: Specialized sub-components pluggable into host modules (e.g., specific HTTP matchers or handlers).
2. Request Handling Lifecycle
When a client connects to Caddy, the request undergoes six explicit stages:
- TCP Accept: Caddy’s network listeners accept incoming TCP connections on configured network interfaces (typically
:80and:443). - TLS Handshake & SNI Selection: For HTTPS requests, Caddy inspects the Server Name Indication (SNI) header in the ClientHello packet. It selects the corresponding TLS certificate from its internal memory cache or triggers On-Demand TLS if configured.
- Site Block Matching: Caddy matches the requested host name and port against defined site blocks.
- Directive Chain Execution: Directives within the matched site block are executed sequentially or in accordance with Caddy’s default directive sorting order (e.g.,
root->rewrite->route->handle->reverse_proxy->file_server). - Upstream / Disk Execution: The terminal handler executes the primary action: proxying the request over a TCP socket to an upstream backend or reading static files from the file system.
- Structured Response Logging: Caddy emits high-performance, structured JSON access logs capturing latency, status codes, bytes sent, and upstream response times.
3. Dynamic Configuration Architecture (Caddyfile vs. JSON)
Caddy’s native internal configuration is represented strictly as a JSON document. The human-friendly Caddyfile is an ergonomic abstraction.
When Caddy starts or reloads a Caddyfile, an internal Adapter parses the text, translates it into the corresponding JSON configuration structure, and sends it to the administrative API.
+------------------+ caddy adapt +--------------------+
| /etc/caddy/ | ------------------> | Native Caddy JSON |
| Caddyfile | | Configuration |
+------------------+ +---------+----------+
|
v POST /load
+--------------------+
| Admin REST API |
| (Port 2019) |
+--------------------+
Because Caddy manages state via an internal dynamic engine, configurations can be updated in real time through HTTP API calls or via file reloads.
3. Core Concepts and Terminology
Understanding Caddy requires mastering several key architectural terms and concepts:
Directives
A directive is a single keyword block in a Caddyfile that configures a specific behavior of the server.
reverse_proxy: Forwards requests to backend servers.file_server: Serves static files from the file system.root: Sets the base document directory for file operations.redir: Issues HTTP redirect responses (e.g., 301/302) to clients.encode: Enables response compression algorithms such as Gzip or Zstandard.
Matchers
A matcher acts as a filter to determine whether a directive should be applied to a specific incoming request. Matchers can filter by URI path, HTTP method, request headers, client IP addresses, or environment flags.
- Standard Path Matcher:
reverse_proxy /api/* 127.0.0.1:8000 - Named Matcher:
@post_requests {
method POST
path /submit
}
handle @post_requests {
reverse_proxy 127.0.0.1:9000
}
Handlers & Blocks (handle vs. route)
handleBlock: Defines mutually exclusive routing paths. Only the first matchinghandleblock executes.routeBlock: Forces directives inside it to execute strictly in the literal order they are written, overriding Caddy’s default directive sorting algorithm.
# Example using handle for mutually exclusive logic
example.com {
handle /api/* {
reverse_proxy 127.0.0.1:8000
}
handle {
root * /var/www/html
file_server
}
}
Directive Sorting Precedence
Caddy automatically sorts directives within a site block into a deterministic execution pipeline. For instance, rewrite rules always execute before reverse_proxy, and root executes before file_server. If custom execution ordering is required, place directives within a route { ... } block.
Glossary of Essential Terms
| Term | Definition |
|---|---|
| ACME | Automatic Certificate Management Environment protocol used to issue TLS certificates from Let’s Encrypt or ZeroSSL. |
| Adapter | An internal module that converts non-JSON configuration formats (like Caddyfile) into Caddy native JSON. |
| Admin API | The default REST endpoint listening on localhost:2019 used for runtime configuration management. |
| SNI | Server Name Indication, an extension to the TLS protocol allowing clients to specify the domain name during connection setup. |
| Upstream | A backend application server destination to which Caddy proxies requests. |
4. Installation and Version Management
Prerequisites
- A supported Linux distribution (Debian, Ubuntu, RHEL, CentOS, Rocky Linux, Arch Linux).
- Network firewall rules permitting inbound traffic on TCP ports
80and443. sudoorrootadministrative privileges on the target host.
Installation Method 1: Official Package Repository (Recommended for Production)
On Debian / Ubuntu:
# Install required dependencies
sudo apt update && sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
# Import official Caddy GPG signing key
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
# Add Caddy APT repository source list
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
# Update package index and install Caddy
sudo apt update
sudo apt install caddy
On RHEL / Fedora / Rocky Linux:
# Install DNF COPR plugin and enable Caddy repository
sudo dnf install -y 'dnf-command(copr)'
sudo dnf copr enable @caddy/caddy
sudo dnf install -y caddy
Installation Method 2: Custom Binary Build with xcaddy
When custom third-party modules (such as DNS provider plugins for DNS-01 ACME challenges) are required, compile Caddy using the official build tool xcaddy.
# Install xcaddy binary
go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
# Build Caddy with the Cloudflare DNS module integrated
xcaddy build --with github.com/caddy-dns/cloudflare
# Move binary to system path
sudo mv caddy /usr/local/bin/caddy
sudo chown root:root /usr/local/bin/caddy
Version Verification & Post-Install File Inspection
Verify that Caddy is installed and check the running binary version:
caddy version
Inspect default installed system paths:
- Binary Location:
/usr/bin/caddy(or/usr/local/bin/caddy) - Default Configuration:
/etc/caddy/Caddyfile - Systemd Service Unit:
/lib/systemd/system/caddy.service - Data Storage Directory:
/var/lib/caddy
5. Configuration Files and Directory Structure
Directory Layout & Permissions Reference
| File / Directory | Recommended Owner:Group | Default Mode | Purpose & Details |
|---|---|---|---|
/etc/caddy/Caddyfile |
root:root |
0644 |
Primary Caddyfile configuration file. |
/etc/caddy/Caddyfile.d/ |
root:root |
0755 |
Directory for modular site config snippets imported via import. |
/var/lib/caddy/ |
caddy:caddy |
0700 |
State directory holding certificates, ACME keys, and active runtime data. |
/var/lib/caddy/.local/share/caddy/certificates/ |
caddy:caddy |
0700 |
ACME TLS certificates storage path. |
/var/log/caddy/ |
caddy:caddy |
0755 |
Recommended directory for access and error log files. |
/run/caddy/ |
caddy:caddy |
0755 |
Process PID socket storage directory. |
Main Configuration Structure (/etc/caddy/Caddyfile)
A Caddyfile consists of three primary structural areas: Global Options, Snippets, and Site Blocks.
# 1. GLOBAL OPTIONS BLOCK (Applies to the entire process)
{
admin localhost:2019
email admin@example.com
log {
output file /var/log/caddy/system.log
level INFO
}
}
# 2. REUSABLE SNIPPET (Importable elsewhere)
(security_headers) {
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains"
X-Content-Type-Options "nosniff"
X-Frame-Options "DENY"
}
}
# 3. SITE BLOCKS
app.example.com {
import security_headers
encode zstd gzip
reverse_proxy 127.0.0.1:8000 {
header_up Host {upstream_hostport}
header_up X-Real-IP {remote_host}
}
}
Configuration Validation and Formatting
Always format and validate the Caddyfile prior to issuing service reload commands to avoid service startup failures:
# Format Caddyfile to standardize indentation and formatting
caddy fmt --overwrite /etc/caddy/Caddyfile
# Validate syntax without applying changes
caddy validate --config /etc/caddy/Caddyfile
6. Service Management and Common Operations
Caddy is managed as a background service via systemd on standard Linux systems.
+-----------------------------------------------------------------------+
| SYSTEMD ENGINE |
| |
| systemctl reload caddy ----> Sends SIGUSR1 to Caddy |
| (Zero Connection Loss) In-Memory Configuration Swap |
| |
| systemctl restart caddy ----> Terminates Master Process |
| (Drops Socket Connections) Re-binds Network Ports |
+-----------------------------------------------------------------------+
Core Operations
# Check service status and health state
systemctl status caddy
# Start the Caddy background daemon
systemctl start caddy
# Stop the Caddy service
systemctl stop caddy
# Enable automatic startup at system boot
systemctl enable caddy
# Graceful reload configuration without dropping connections
systemctl reload caddy
Graceful Reload vs. Process Restart
- Reload (
systemctl reload caddy/caddy reload): Performs an in-memory configuration swap. The existing process validates the new configuration JSON. If valid, new worker context initialized, active TCP socket handlers seamlessly hand off connections, and the old context closes. Zero downtime occurs. - Restart (
systemctl restart caddy): Unconditionally terminates the running Caddy process, tears down listeners on ports 80/443, and starts a fresh process. Active client requests are dropped immediately.
Process & Listener Inspection
# Confirm Caddy processes are active
ps aux | grep caddy
# Inspect network sockets held by Caddy
ss -tlnp | grep caddy
7. Common Administrative Tasks and Workflows
Workflow 1: Reverse Proxying a Microservice with Automatic HTTPS
Objective: Securely proxy api.example.com to an internal service on 127.0.0.1:3000.
- Add site block to
/etc/caddy/Caddyfile:
api.example.com {
reverse_proxy 127.0.0.1:3000
}
- Validate syntax:
caddy validate --config /etc/caddy/Caddyfile
- Apply changes gracefully:
systemctl reload caddy
- Verify HTTPS response and certificate headers:
curl -vI https://api.example.com
Workflow 2: Serving a Static Website with Compression & Path Routing
Objective: Serve static HTML/JS from /var/www/my-site, proxy /api/* requests to port 8000, and enable Zstandard compression.
my-site.example.com {
root * /var/www/my-site
encode zstd gzip
# Route backend requests
handle /api/* {
reverse_proxy 127.0.0.1:8000
}
# Fallback to static files
handle {
try_files {path} {path}/ /index.html
file_server
}
}
Workflow 3: Configuring Upstream Load Balancing & Active Health Checks
Objective: Distribute incoming requests across three backend application instances with automatic failover.
app.example.com {
reverse_proxy 10.0.0.10:8080 10.0.0.11:8080 10.0.0.12:8080 {
lb_policy round_robin
health_uri /health
health_interval 10s
health_timeout 2s
fail_duration 30s
}
}
Workflow 4: Internal Certificate Management with DNS-01 ACME Challenge
Objective: Obtain trusted TLS certificates for internal infrastructure servers (internal.corp.local) that are not publicly reachable from the internet on port 80/443.
internal.corp.local {
tls {
dns cloudflare {$CLOUDFLARE_API_TOKEN}
}
reverse_proxy 127.0.0.1:9000
}
8. CLI Command Reference and Cheatsheet
1. Version & Environment Inspection
| Command | Purpose | Important Flags / Arguments | Practical Example | Explanation |
|---|---|---|---|---|
caddy version |
Displays binary version and module details | --build-info |
caddy version |
Shows the exact commit hash, Go compiler version, and embedded modules. |
caddy list-modules |
Lists installed Caddy modules | --standard |
caddy list-modules |
Displays available standard and third-party plugins in the running binary. |
2. Service Management Commands
| Command | Purpose | Important Flags / Arguments | Practical Example | Explanation |
|---|---|---|---|---|
caddy run |
Executes Caddy in foreground (blocking) | --config <file>, --adapter |
caddy run --config /etc/caddy/Caddyfile |
Primary command for foreground execution and live debugging. |
caddy start |
Launches Caddy as a background daemon | --config <file> |
caddy start --config /etc/caddy/Caddyfile |
Starts process detached in background (used when systemd is not present). |
caddy stop |
Stops running background Caddy daemon | --address |
caddy stop |
Sends shutdown signal via Admin API to stop background server process. |
caddy reload |
Gracefully reloads active configuration | --config <file> |
caddy reload --config /etc/caddy/Caddyfile |
Performs zero-downtime dynamic configuration swap. |
WARNING: Avoid running
caddy runwhile the systemdcaddy.serviceis actively running. Binding conflicts on ports 80/443 will occur. Stop systemd first viasystemctl stop caddy.
3. Configuration & Validation Commands
| Command | Purpose | Important Flags / Arguments | Practical Example | Explanation |
|---|---|---|---|---|
caddy validate |
Tests configuration file syntax | --config <file> |
caddy validate --config /etc/caddy/Caddyfile |
Validates syntax without applying config to production engine. |
caddy fmt |
Formats Caddyfile according to standard layout | --overwrite |
caddy fmt --overwrite /etc/caddy/Caddyfile |
Standardizes spacing, indentation, and structure. |
caddy adapt |
Converts Caddyfile into native JSON structure | --config <file>, --pretty |
caddy adapt --config /etc/caddy/Caddyfile --pretty |
Vital diagnostic command to view parsed internal routing logic. |
9. Logs, Monitoring and Diagnostics
Structured JSON Logging Setup
Caddy outputs raw logs in structured JSON format. Configure dedicated site access logs within the site block:
example.com {
log {
output file /var/log/caddy/example_access.log {
roll_size 10MiB
roll_keep 10
roll_keep_for 720h
}
format json
}
reverse_proxy 127.0.0.1:8080
}
Real-Time Journald Inspection
# Follow Caddy service logs live
journalctl -u caddy -f
# Inspect the last 100 log lines without pagination
journalctl -u caddy -n 100 --no-pager
# Filter logs for specific errors
journalctl -u caddy -g "error" --since "1 hour ago"
Monitoring & Prometheus Integration
Enable Caddy’s internal Prometheus metrics endpoint in the global options block:
{
metrics
}
Scrape metrics via curl or point Prometheus to port 2019:
curl http://localhost:2019/metrics
Key Prometheus Metrics to Monitor:
caddy_http_requests_total: Total counter of HTTP requests broken down by handler and status code.caddy_http_request_duration_seconds: Histogram of request latencies.caddy_http_responses_total: Output responses split by status class (2xx, 4xx, 5xx).
10. Troubleshooting Guide and Common Failure Scenarios
Scenario 1: Caddy Service Fails to Start or Reload
Symptoms
Executing systemctl reload caddy or systemctl start caddy returns a failure state. Service status shows failed (Result: exit-code).
Possible Causes
- Syntax error or typo in
/etc/caddy/Caddyfile(e.g., missing brace, invalid directive). - Port binding conflict: Port 80 or 443 is already claimed by NGINX, Apache, or another process.
- Incorrect directory permissions on Caddy log paths or config files.
Investigation & Diagnosis
- Validate Caddyfile syntax:
caddy validate --config /etc/caddy/Caddyfile
- Check network port bindings:
ss -tlnp | grep -E ':80|:443'
- Inspect detailed systemd logs:
journalctl -u caddy -n 50 --no-pager
Resolution
- If syntax error: Correct the broken directive or missing brace identified by
caddy validate. - If port conflict: Stop conflicting web server services:
sudo systemctl stop nginx
sudo systemctl disable nginx
- Re-run validation and start service:
caddy validate --config /etc/caddy/Caddyfile
systemctl start caddy
Verification
systemctl status caddy
Scenario 2: Split-Brain State (API vs. Caddyfile Reload Overwrites)
Symptoms
Runtime API changes issued via curl localhost:2019/load suddenly vanish after a system restart or server reboot.
+-------------------------------------------------------------------------+
| SPLIT-BRAIN CONFIG TRAP |
| |
| 1. Admin updates server via REST API --> Active Runtime Modified |
| (curl -X POST localhost:2019/load) |
| |
| 2. SysAdmin runs systemctl reload caddy --> Reads /etc/caddy/Caddyfile|
| (Disk file silently overwrites runtime state!) |
+-------------------------------------------------------------------------+
Possible Causes
Mixing configuration control methods. Caddy allows active runtime modifications via its REST API, but running systemctl reload caddy forces Caddy to re-parse /etc/caddy/Caddyfile from disk, wiping out in-memory API modifications.
Investigation & Diagnosis
- Query active running JSON configuration:
curl http://localhost:2019/config/
- Compare running configuration output against disk Caddyfile:
caddy adapt --config /etc/caddy/Caddyfile
Resolution
Establish a single source of truth per operational environment.
- Option A (File-Based): Disallow runtime API updates. Make all changes directly in
/etc/caddy/Caddyfileand executesystemctl reload caddy. - Option B (API-Based): Export running JSON state back to disk before reloading:
curl http://localhost:2019/config/ > /etc/caddy/saved_config.json
Scenario 3: ACME Certificate Provisioning Failure
Symptoms
Clients receive browser security warnings (ERR_CERT_AUTHORITY_INVALID) or connections hang during TLS handshake. Journal logs report ACME challenge failed.
Possible Causes
- External DNS A/AAAA records do not point to this server’s public IP address.
- Network firewall blocks inbound TCP port 80 (required for ACME HTTP-01 challenge validation).
- Server system clock is drifted or out of sync.
- Let’s Encrypt API rate limits exceeded due to repeated failed retry loops.
Investigation & Diagnosis
- Verify public DNS resolution:
dig +short example.com
- Test port 80 accessibility from an external system:
curl -I http://example.com/.well-known/acme-challenge/test
- Check system time status:
timedatectl
- Search journal logs for ACME protocol errors:
journalctl -u caddy -g "acme" --no-pager
Resolution
- Correct DNS records at domain registrar.
- Open firewall ports on host and cloud provider security groups:
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
- If domain is internal, switch to DNS-01 challenge mode using
tls { dns ... }.
Scenario 4: HTTP 502 Bad Gateway Error on Proxy Request
Symptoms
Client browsers receive 502 Bad Gateway when trying to access proxied application endpoints.
Client ------> Caddy (Proxy) --X (TCP Refused)--> Upstream App (127.0.0.1:8000)
Possible Causes
- Backend application service is stopped or crashed.
- Backend application is listening on a different host interface or port than configured in Caddyfile (e.g., listening on
127.0.0.1while Caddy targets10.0.0.5). - Host firewall (iptables/ufw) is blocking internal connection between Caddy and upstream.
Investigation & Diagnosis
- Test upstream app directly from Caddy host using curl:
curl -v http://127.0.0.1:8000/
- Inspect listening ports on local machine:
ss -tlnp | grep 8000
- Check Caddy logs for upstream failure specifics:
journalctl -u caddy -g "dial tcp" --no-pager
Resolution
- Start or restart backend application service.
- Update Caddyfile target address to match upstream listener interface:
reverse_proxy 127.0.0.1:8000
- Reload Caddy:
systemctl reload caddy
Scenario 5: HTTP 404 Errors on Valid Routing Paths
Symptoms
Requests to expected paths return 404 Not Found.
Possible Causes
handle_pathdirective was used instead ofhandle, inadvertently stripping the URI prefix before forwarding to backend.- Document
rootdirectory path is wrong or missing files. - Incorrect file system permissions prevent
caddyuser from reading static files.
Investigation & Diagnosis
- Parse actual generated URI handling via
caddy adapt:
caddy adapt --config /etc/caddy/Caddyfile --pretty
- Verify directory existence and file readability:
ls -la /var/www/my-site
Resolution
- If using
handle_path /api/*, note that/api/usersis forwarded to upstream as/users. Switch tohandle /api/*if upstream expects the/apiprefix. - Fix static directory ownership:
sudo chown -R caddy:caddy /var/www/my-site
sudo chmod -R 755 /var/www/my-site
Scenario 6: Infinite HTTPS Redirect Loops
Symptoms
Browser fails to load site and displays TOO_MANY_REDIRECTS.
Possible Causes
Caddy is behind an edge CDN/Load Balancer (e.g., Cloudflare Flexible SSL) terminating HTTPS and connecting to Caddy over HTTP on port 80. Caddy attempts to issue an automatic HTTP-to-HTTPS redirect back to the client, creating a continuous loop.
Investigation & Diagnosis
Trace redirect hops using curl:
curl -vIL http://example.com
Resolution
Explicitly set site block address to http:// or adjust incoming request handling:
http://example.com {
reverse_proxy 127.0.0.1:8000
}
Or configure CDN SSL mode to Full (Strict) so CDN connects to Caddy over HTTPS (port 443).
Scenario 7: Data Directory Permission Denied
Symptoms
Caddy log displays write permission failures to storage path /var/lib/caddy during certificate renewal.
Possible Causes
Directory permissions modified manually or owned by root instead of unprivileged caddy service user.
Investigation & Diagnosis
ls -la /var/lib/caddy
Resolution
Restore correct recursive ownership to caddy user and group:
sudo chown -R caddy:caddy /var/lib/caddy
sudo chmod 700 /var/lib/caddy
11. Security and Best Practices
1. Privilege Minimization & Service Hardening
- Never run Caddy directly as the
rootadministrative superuser in production. - Systemd service should employ Linux capabilities (
CAP_NET_BIND_SERVICE) to allow binding low-numbered ports (80/443) as unprivileged usercaddy.
2. Mandatory TLS Hardening
Restrict acceptable TLS protocols to standard modern specs within global options:
{
tls {
protocols tls1.2 tls1.3
}
}
3. HTTP Basic Authentication (basic_auth)
Protect staging or internal administrative endpoints with bcrypt-hashed credentials:
Generate hash using Caddy CLI:
caddy hash-password --plaintext "MySecurePassword123"
Apply in Caddyfile:
staging.example.com {
basic_auth {
admin $2a$14$Z8...hashed_password_string...
}
reverse_proxy 127.0.0.1:8080
}
4. Injecting Standard Production Security Headers
(security_headers) {
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
X-Content-Type-Options "nosniff"
X-Frame-Options "SAMEORIGIN"
Referrer-Policy "strict-origin-when-cross-origin"
}
}
example.com {
import security_headers
reverse_proxy 127.0.0.1:8080
}
12. Performance and Resource Optimisation
1. Upstream Keep-Alive Connection Pooling
Maintain persistent backend HTTP connections in reverse_proxy to eliminate TCP connection setup overhead for high-traffic microservices:
example.com {
reverse_proxy 127.0.0.1:8080 {
transport http {
keepalive 30s
keepalive_idle_conns 100
}
}
}
2. High-Performance Static Compression
Zstandard (zstd) delivers superior compression ratios and speed compared to traditional Gzip:
example.com {
encode zstd gzip
file_server
}
3. System Limits Hardening (ulimit & systemd)
Ensure systemd unit /lib/systemd/system/caddy.service allows high open file descriptor counts for concurrent TCP sockets:
[Service]
LimitNOFILE=65536
LimitNPROC=51200
Apply limit changes:
sudo systemctl daemon-reload
sudo systemctl restart caddy
13. Backup, Recovery and Maintenance
Essential Files for Backup
- Configuration Files:
/etc/caddy/Caddyfileand custom included modules. - ACME Data & Certificates:
/var/lib/caddy/(Backing up this directory prevents rate limit hits with ACME providers upon re-deployment).
Automated Backup Script Example
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="/var/backups/caddy"
DATE=$(date +%Y%m%m_%H%M%S)
TARGET_ARCHIVE="${BACKUP_DIR}/caddy_backup_${DATE}.tar.gz"
mkdir -p "${BACKUP_DIR}"
# Archive configuration and ACME state
tar -czf "${TARGET_ARCHIVE}" \
/etc/caddy/Caddyfile \
/var/lib/caddy
echo "Backup created successfully at ${TARGET_ARCHIVE}"
Upgrade & Maintenance Procedure
# 1. Backup existing configuration and certificates
sudo tar -czf /tmp/caddy_pre_upgrade.tar.gz /etc/caddy /var/lib/caddy
# 2. Update repository package
sudo apt update && sudo apt install --only-upgrade caddy
# 3. Validate syntax post-upgrade
caddy validate --config /etc/caddy/Caddyfile
# 4. Graceful reload
systemctl reload caddy
14. Automation and Integration
Docker Compose Production Setup
Deploying Caddy alongside application containers using persistent volume mounts:
version: '3.8'
services:
caddy:
image: caddy:2-alpine
restart: unless-stopped
ports:
- "80:80"
- "443:443"
- "443:443/udp" # HTTP/3 (QUIC) support
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config
volumes:
caddy_data:
caddy_config:
15. Practical Labs and Exercises
Exercise 1 (Beginner): Setup Static Site with Dynamic Compression
- Objective: Serve HTML content out of
/var/www/lab1on domainlab1.localwith Gzip enabled. - Setup Instructions:
- Create folder
/var/www/lab1and place anindex.htmlfile inside. - Create site block in
/etc/caddy/Caddyfile:
http://lab1.local {
root * /var/www/lab1
encode gzip
file_server
}
- Validate and reload Caddy.
- Validation: Execute
curl -H "Accept-Encoding: gzip" -I [http://lab1.local](http://lab1.local)and confirmContent-Encoding: gzipheader exists.
Exercise 2 (Intermediate): High-Availability Failover Proxy
- Objective: Configure a load-balanced proxy pointing to primary application
127.0.0.1:8001and backup127.0.0.1:8002. - Setup Instructions:
- Start two simple HTTP python servers:
python3 -m http.server 8001 &
python3 -m http.server 8002 &
- Write proxy site block with health checking:
http://lab2.local {
reverse_proxy 127.0.0.1:8001 127.0.0.1:8002 {
lb_policy first
health_uri /
health_interval 5s
}
}
- Validation: Stop process on 8001 (
kill %1) and observe Caddy routing requests to 8002 without client failure.
Exercise 3 (Troubleshooting Lab): Recovering from Port Conflict Crash
- Objective: Diagnose and fix a failed Caddy startup caused by an unexpected process holding port 80.
- Lab Setup:
- Run
nc -l -p 80 &to occupy port 80. - Attempt to restart Caddy:
systemctl restart caddy.
- Troubleshooting Steps:
- Execute
systemctl status caddyto observe failure state. - Run
ss -tlnp | grep ':80'to identify conflicting process PID. - Kill conflicting process and restart Caddy:
sudo kill -9 <PID>
sudo systemctl start caddy
- Confirm operational status.
16. Quick Reference and Cheat Sheet
Important Configuration Files & Directories
| Path | Purpose |
|---|---|
/etc/caddy/Caddyfile |
Primary configuration file. |
/var/lib/caddy |
Data directory for certificates and ACME state. |
/var/log/caddy |
Standard log directory. |
Essential Operational Commands
| Administrative Goal | Command |
|---|---|
| Service Status | systemctl status caddy |
| Validate Config | caddy validate --config /etc/caddy/Caddyfile |
| Format Caddyfile | caddy fmt --overwrite /etc/caddy/Caddyfile |
| Graceful Reload | systemctl reload caddy OR caddy reload |
| Debug Foreground Run | caddy run --config /etc/caddy/Caddyfile |
| Convert to JSON | caddy adapt --config /etc/caddy/Caddyfile |
Network Ports & Protocols
| Port | Protocol | Usage |
|---|---|---|
80 |
TCP | Plain HTTP traffic and ACME HTTP-01 challenge validation. |
443 |
TCP / UDP | Encrypted HTTPS traffic (TCP for HTTP/1.1 & HTTP/2; UDP for HTTP/3 QUIC). |
2019 |
TCP | Local Administrative REST API endpoint (bound to localhost). |
17. Further Reading and References
- Official Caddy Documentation: https://caddyserver.com/docs
- Caddyfile Tutorial & Syntax: https://caddyserver.com/docs/caddyfile
- Caddy reverse_proxy Directive Reference: https://caddyserver.com/docs/caddyfile/directives/reverse_proxy
- Caddy Automatic HTTPS Architecture: https://caddyserver.com/docs/automatic-https