Domain 7Web & Application LayerINTERMEDIATE

Mastering Caddy: Architecture, Configuration, Administration, and Production Troubleshooting

A comprehensive guide to the Caddy web server covering modular architecture, ACME automatic HTTPS, Caddyfile directives, JSON API state management, service administration, CLI commands, and production SRE troubleshooting workflows.

Domain: Domain 7: Web & Application Layer•Updated: 2026-11-03
#linux#sysadmin#caddy#reverse-proxy#tls

1. Overview

Caddy is a modern, open-source web server and reverse proxy written in Go. It is designed from the ground up around memory safety, modularity, and automated infrastructure operations. While traditional servers like NGINX or Apache HTTP Server require external tools (such as Certbot or acme.sh) and explicit cron jobs to provision and renew TLS certificates, Caddy integrates public key infrastructure (PKI) management directly into its core engine.

+-----------------------------------------------------------------------+
|                             CADDY SERVER                              |
|                                                                       |
|  +-------------------+   +--------------------+   +----------------+  |
|  | HTTP/HTTPS Core   |   | Automated ACME Engine |   | Static Server  |  |
|  | (TLS 1.3 / QUIC)  |   | (Let's Encrypt/ZeroSSL)|  | & Reverse Proxy|  |
|  +---------+---------+   +---------+----------+   +-------+--------+  |
+------------|-----------------------|----------------------|-----------+
             |                       |                      |
             v                       v                      v
    Port 80/443 Traffic     Automated TLS Certs     Upstream Applications

Core Responsibilities in Production

  • HTTP/HTTPS Termination: Accepts incoming HTTP/1.1, HTTP/2, and HTTP/3 (QUIC) client connections, terminates TLS encryption, and handles cipher negotiation.
  • Reverse Proxying: Forwards client traffic to backend application servers (written in Go, Python, Node.js, Rust, Java, or PHP-FPM) with support for load balancing and active/passive health checks.
  • Static Asset Serving: Delivers static files from local disk with high performance using modern compression algorithms like Zstandard and Gzip.
  • Automatic Certificate Management: Communicates automatically with ACME certificate authorities (such as Let’s Encrypt or ZeroSSL) to request, obtain, and renew TLS certificates without service interruption.

Problems Solved by Caddy

  1. Certbot Complexity & Expiry Outages: Traditional setups depend on shell scripts, certbot renewals, and reload hooks. If a renewal hook fails, certificates silently expire. Caddy manages the complete certificate lifecycle in memory and on disk automatically.
  2. Verbose Configuration Boilerplate: Complex NGINX or Apache configurations often run into hundreds of lines for basic SSL settings, redirect rules, and security headers. Caddy accomplishes the same output with concise site blocks.
  3. Configuration Reload Disruptions: Caddy utilizes atomic, zero-downtime configuration reloads. When configuration changes occur, active TCP sockets are not abruptly dropped.

Feature Comparison

Feature Caddy NGINX Apache HTTP Server
Language & Safety Go (Memory safe) C (Manual memory management) C (Manual memory management)
Automatic TLS (ACME) Built-in by default Requires Certbot / external scripts Requires Certbot / external scripts
HTTP/3 (QUIC) Support Native out-of-the-box Requires specific compile flags Requires external modules
Default Config Format Caddyfile or JSON NGINX DSL Directive-based (httpd.conf)
Dynamic API Control REST API on Port 2019 NGINX Plus (Paid) or NJS Limited / mod_status
Reload Mechanism In-memory atomic swap Worker binary signal Process grace restart

Production Use Case Example

Consider a backend Node.js microservice listening locally on 127.0.0.1:8080. To expose this application to the public internet securely at api.example.com, Caddy requires only two lines of configuration:

api.example.com {
    reverse_proxy 127.0.0.1:8080
}

When Caddy starts with this block, it binds to ports 80 and 443, detects that api.example.com is a public domain, contacts Let’s Encrypt to complete ACME challenges, provisions a valid TLS certificate, sets up HTTPS redirection on port 80, and begins proxying traffic to port 8080.


2. Architecture and Mental Model

To operate Caddy reliably in production, SysAdmins and SREs must understand how Caddy processes requests internally and how it manages configuration state.

graph TD
    Client[Client Browser / API Client] -->|TCP Connect :80 / :443| Socket[Socket Accept Listener]
    Socket -->|Port 80 HTTP| Redirect[HTTP to HTTPS Redirect]
    Socket -->|Port 443 HTTPS| TLS[TLS Handshake & SNI Check]
    TLS -->|Obtain / Validate Cert| ACME[ACME Engine / Cert Cache]
    TLS --> Engine[Caddy HTTP Core Engine]
    Engine --> Matcher{Site Block & Path Matcher}
    Matcher -->|/api/*| Proxy[reverse_proxy Handler]
    Matcher -->|/static/*| Static[file_server Handler]
    Matcher -->|Redirect Rule| Redir[redir Handler]
    Proxy -->|TCP Connection| Upstream[App Server 127.0.0.1:8000]
    Static -->|Disk Read| Filesystem[/var/www/html]
    Upstream --> Logger[Structured JSON Logger]
    Filesystem --> Logger
    Redir --> Logger
    Logger --> Client

1. Modular Core Architecture

At its core, Caddy is a light runtime module loader. The Caddy core engine manages the server lifecycle and dynamic JSON configurations. Every piece of functional code—including the HTTP web server, the TLS manager, the reverse proxy, and custom plugins—is an independent module loaded into the core engine.

  • Host Modules: Parent modules that load guest modules (e.g., the http app module).
  • Guest Modules: Specialized sub-components pluggable into host modules (e.g., specific HTTP matchers or handlers).

2. Request Handling Lifecycle

When a client connects to Caddy, the request undergoes six explicit stages:

  1. TCP Accept: Caddy’s network listeners accept incoming TCP connections on configured network interfaces (typically :80 and :443).
  2. TLS Handshake & SNI Selection: For HTTPS requests, Caddy inspects the Server Name Indication (SNI) header in the ClientHello packet. It selects the corresponding TLS certificate from its internal memory cache or triggers On-Demand TLS if configured.
  3. Site Block Matching: Caddy matches the requested host name and port against defined site blocks.
  4. Directive Chain Execution: Directives within the matched site block are executed sequentially or in accordance with Caddy’s default directive sorting order (e.g., root -> rewrite -> route -> handle -> reverse_proxy -> file_server).
  5. Upstream / Disk Execution: The terminal handler executes the primary action: proxying the request over a TCP socket to an upstream backend or reading static files from the file system.
  6. Structured Response Logging: Caddy emits high-performance, structured JSON access logs capturing latency, status codes, bytes sent, and upstream response times.

3. Dynamic Configuration Architecture (Caddyfile vs. JSON)

Caddy’s native internal configuration is represented strictly as a JSON document. The human-friendly Caddyfile is an ergonomic abstraction.

When Caddy starts or reloads a Caddyfile, an internal Adapter parses the text, translates it into the corresponding JSON configuration structure, and sends it to the administrative API.

+------------------+      caddy adapt      +--------------------+
|  /etc/caddy/     |  ------------------>  | Native Caddy JSON  |
|    Caddyfile     |                       | Configuration      |
+------------------+                       +---------+----------+
                                                     |
                                                     v  POST /load
                                           +--------------------+
                                           | Admin REST API     |
                                           | (Port 2019)        |
                                           +--------------------+

Because Caddy manages state via an internal dynamic engine, configurations can be updated in real time through HTTP API calls or via file reloads.


3. Core Concepts and Terminology

Understanding Caddy requires mastering several key architectural terms and concepts:

Directives

A directive is a single keyword block in a Caddyfile that configures a specific behavior of the server.

  • reverse_proxy: Forwards requests to backend servers.
  • file_server: Serves static files from the file system.
  • root: Sets the base document directory for file operations.
  • redir: Issues HTTP redirect responses (e.g., 301/302) to clients.
  • encode: Enables response compression algorithms such as Gzip or Zstandard.

Matchers

A matcher acts as a filter to determine whether a directive should be applied to a specific incoming request. Matchers can filter by URI path, HTTP method, request headers, client IP addresses, or environment flags.

  • Standard Path Matcher: reverse_proxy /api/* 127.0.0.1:8000
  • Named Matcher:
@post_requests {
    method POST
    path /submit
}
handle @post_requests {
    reverse_proxy 127.0.0.1:9000
}

Handlers & Blocks (handle vs. route)

  • handle Block: Defines mutually exclusive routing paths. Only the first matching handle block executes.
  • route Block: Forces directives inside it to execute strictly in the literal order they are written, overriding Caddy’s default directive sorting algorithm.
# Example using handle for mutually exclusive logic
example.com {
    handle /api/* {
        reverse_proxy 127.0.0.1:8000
    }
    handle {
        root * /var/www/html
        file_server
    }
}

Directive Sorting Precedence

Caddy automatically sorts directives within a site block into a deterministic execution pipeline. For instance, rewrite rules always execute before reverse_proxy, and root executes before file_server. If custom execution ordering is required, place directives within a route { ... } block.

Glossary of Essential Terms

Term Definition
ACME Automatic Certificate Management Environment protocol used to issue TLS certificates from Let’s Encrypt or ZeroSSL.
Adapter An internal module that converts non-JSON configuration formats (like Caddyfile) into Caddy native JSON.
Admin API The default REST endpoint listening on localhost:2019 used for runtime configuration management.
SNI Server Name Indication, an extension to the TLS protocol allowing clients to specify the domain name during connection setup.
Upstream A backend application server destination to which Caddy proxies requests.

4. Installation and Version Management

Prerequisites

  • A supported Linux distribution (Debian, Ubuntu, RHEL, CentOS, Rocky Linux, Arch Linux).
  • Network firewall rules permitting inbound traffic on TCP ports 80 and 443.
  • sudo or root administrative privileges on the target host.

On Debian / Ubuntu:

# Install required dependencies
sudo apt update && sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl

# Import official Caddy GPG signing key
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg

# Add Caddy APT repository source list
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list

# Update package index and install Caddy
sudo apt update
sudo apt install caddy

On RHEL / Fedora / Rocky Linux:

# Install DNF COPR plugin and enable Caddy repository
sudo dnf install -y 'dnf-command(copr)'
sudo dnf copr enable @caddy/caddy
sudo dnf install -y caddy

Installation Method 2: Custom Binary Build with xcaddy

When custom third-party modules (such as DNS provider plugins for DNS-01 ACME challenges) are required, compile Caddy using the official build tool xcaddy.

# Install xcaddy binary
go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest

# Build Caddy with the Cloudflare DNS module integrated
xcaddy build --with github.com/caddy-dns/cloudflare

# Move binary to system path
sudo mv caddy /usr/local/bin/caddy
sudo chown root:root /usr/local/bin/caddy

Version Verification & Post-Install File Inspection

Verify that Caddy is installed and check the running binary version:

caddy version

Inspect default installed system paths:

  • Binary Location: /usr/bin/caddy (or /usr/local/bin/caddy)
  • Default Configuration: /etc/caddy/Caddyfile
  • Systemd Service Unit: /lib/systemd/system/caddy.service
  • Data Storage Directory: /var/lib/caddy

5. Configuration Files and Directory Structure

Directory Layout & Permissions Reference

File / Directory Recommended Owner:Group Default Mode Purpose & Details
/etc/caddy/Caddyfile root:root 0644 Primary Caddyfile configuration file.
/etc/caddy/Caddyfile.d/ root:root 0755 Directory for modular site config snippets imported via import.
/var/lib/caddy/ caddy:caddy 0700 State directory holding certificates, ACME keys, and active runtime data.
/var/lib/caddy/.local/share/caddy/certificates/ caddy:caddy 0700 ACME TLS certificates storage path.
/var/log/caddy/ caddy:caddy 0755 Recommended directory for access and error log files.
/run/caddy/ caddy:caddy 0755 Process PID socket storage directory.

Main Configuration Structure (/etc/caddy/Caddyfile)

A Caddyfile consists of three primary structural areas: Global Options, Snippets, and Site Blocks.

# 1. GLOBAL OPTIONS BLOCK (Applies to the entire process)
{
    admin localhost:2019
    email admin@example.com
    log {
        output file /var/log/caddy/system.log
        level INFO
    }
}

# 2. REUSABLE SNIPPET (Importable elsewhere)
(security_headers) {
    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
    }
}

# 3. SITE BLOCKS
app.example.com {
    import security_headers
    encode zstd gzip
    
    reverse_proxy 127.0.0.1:8000 {
        header_up Host {upstream_hostport}
        header_up X-Real-IP {remote_host}
    }
}

Configuration Validation and Formatting

Always format and validate the Caddyfile prior to issuing service reload commands to avoid service startup failures:

# Format Caddyfile to standardize indentation and formatting
caddy fmt --overwrite /etc/caddy/Caddyfile

# Validate syntax without applying changes
caddy validate --config /etc/caddy/Caddyfile

6. Service Management and Common Operations

Caddy is managed as a background service via systemd on standard Linux systems.

+-----------------------------------------------------------------------+
|                           SYSTEMD ENGINE                              |
|                                                                       |
|   systemctl reload caddy        ---->   Sends SIGUSR1 to Caddy        |
|   (Zero Connection Loss)               In-Memory Configuration Swap   |
|                                                                       |
|   systemctl restart caddy       ---->   Terminates Master Process     |
|   (Drops Socket Connections)           Re-binds Network Ports         |
+-----------------------------------------------------------------------+

Core Operations

# Check service status and health state
systemctl status caddy

# Start the Caddy background daemon
systemctl start caddy

# Stop the Caddy service
systemctl stop caddy

# Enable automatic startup at system boot
systemctl enable caddy

# Graceful reload configuration without dropping connections
systemctl reload caddy

Graceful Reload vs. Process Restart

  • Reload (systemctl reload caddy / caddy reload): Performs an in-memory configuration swap. The existing process validates the new configuration JSON. If valid, new worker context initialized, active TCP socket handlers seamlessly hand off connections, and the old context closes. Zero downtime occurs.
  • Restart (systemctl restart caddy): Unconditionally terminates the running Caddy process, tears down listeners on ports 80/443, and starts a fresh process. Active client requests are dropped immediately.

Process & Listener Inspection

# Confirm Caddy processes are active
ps aux | grep caddy

# Inspect network sockets held by Caddy
ss -tlnp | grep caddy

7. Common Administrative Tasks and Workflows

Workflow 1: Reverse Proxying a Microservice with Automatic HTTPS

Objective: Securely proxy api.example.com to an internal service on 127.0.0.1:3000.

  1. Add site block to /etc/caddy/Caddyfile:
api.example.com {
    reverse_proxy 127.0.0.1:3000
}
  1. Validate syntax:
caddy validate --config /etc/caddy/Caddyfile
  1. Apply changes gracefully:
systemctl reload caddy
  1. Verify HTTPS response and certificate headers:
curl -vI https://api.example.com

Workflow 2: Serving a Static Website with Compression & Path Routing

Objective: Serve static HTML/JS from /var/www/my-site, proxy /api/* requests to port 8000, and enable Zstandard compression.

my-site.example.com {
    root * /var/www/my-site
    encode zstd gzip

    # Route backend requests
    handle /api/* {
        reverse_proxy 127.0.0.1:8000
    }

    # Fallback to static files
    handle {
        try_files {path} {path}/ /index.html
        file_server
    }
}

Workflow 3: Configuring Upstream Load Balancing & Active Health Checks

Objective: Distribute incoming requests across three backend application instances with automatic failover.

app.example.com {
    reverse_proxy 10.0.0.10:8080 10.0.0.11:8080 10.0.0.12:8080 {
        lb_policy round_robin
        health_uri /health
        health_interval 10s
        health_timeout 2s
        fail_duration 30s
    }
}

Workflow 4: Internal Certificate Management with DNS-01 ACME Challenge

Objective: Obtain trusted TLS certificates for internal infrastructure servers (internal.corp.local) that are not publicly reachable from the internet on port 80/443.

internal.corp.local {
    tls {
        dns cloudflare {$CLOUDFLARE_API_TOKEN}
    }
    reverse_proxy 127.0.0.1:9000
}

8. CLI Command Reference and Cheatsheet

1. Version & Environment Inspection

Command Purpose Important Flags / Arguments Practical Example Explanation
caddy version Displays binary version and module details --build-info caddy version Shows the exact commit hash, Go compiler version, and embedded modules.
caddy list-modules Lists installed Caddy modules --standard caddy list-modules Displays available standard and third-party plugins in the running binary.

2. Service Management Commands

Command Purpose Important Flags / Arguments Practical Example Explanation
caddy run Executes Caddy in foreground (blocking) --config <file>, --adapter caddy run --config /etc/caddy/Caddyfile Primary command for foreground execution and live debugging.
caddy start Launches Caddy as a background daemon --config <file> caddy start --config /etc/caddy/Caddyfile Starts process detached in background (used when systemd is not present).
caddy stop Stops running background Caddy daemon --address caddy stop Sends shutdown signal via Admin API to stop background server process.
caddy reload Gracefully reloads active configuration --config <file> caddy reload --config /etc/caddy/Caddyfile Performs zero-downtime dynamic configuration swap.

WARNING: Avoid running caddy run while the systemd caddy.service is actively running. Binding conflicts on ports 80/443 will occur. Stop systemd first via systemctl stop caddy.

3. Configuration & Validation Commands

Command Purpose Important Flags / Arguments Practical Example Explanation
caddy validate Tests configuration file syntax --config <file> caddy validate --config /etc/caddy/Caddyfile Validates syntax without applying config to production engine.
caddy fmt Formats Caddyfile according to standard layout --overwrite caddy fmt --overwrite /etc/caddy/Caddyfile Standardizes spacing, indentation, and structure.
caddy adapt Converts Caddyfile into native JSON structure --config <file>, --pretty caddy adapt --config /etc/caddy/Caddyfile --pretty Vital diagnostic command to view parsed internal routing logic.

9. Logs, Monitoring and Diagnostics

Structured JSON Logging Setup

Caddy outputs raw logs in structured JSON format. Configure dedicated site access logs within the site block:

example.com {
    log {
        output file /var/log/caddy/example_access.log {
            roll_size 10MiB
            roll_keep 10
            roll_keep_for 720h
        }
        format json
    }
    reverse_proxy 127.0.0.1:8080
}

Real-Time Journald Inspection

# Follow Caddy service logs live
journalctl -u caddy -f

# Inspect the last 100 log lines without pagination
journalctl -u caddy -n 100 --no-pager

# Filter logs for specific errors
journalctl -u caddy -g "error" --since "1 hour ago"

Monitoring & Prometheus Integration

Enable Caddy’s internal Prometheus metrics endpoint in the global options block:

{
    metrics
}

Scrape metrics via curl or point Prometheus to port 2019:

curl http://localhost:2019/metrics

Key Prometheus Metrics to Monitor:

  • caddy_http_requests_total: Total counter of HTTP requests broken down by handler and status code.
  • caddy_http_request_duration_seconds: Histogram of request latencies.
  • caddy_http_responses_total: Output responses split by status class (2xx, 4xx, 5xx).

10. Troubleshooting Guide and Common Failure Scenarios

Scenario 1: Caddy Service Fails to Start or Reload

Symptoms

Executing systemctl reload caddy or systemctl start caddy returns a failure state. Service status shows failed (Result: exit-code).

Possible Causes

  1. Syntax error or typo in /etc/caddy/Caddyfile (e.g., missing brace, invalid directive).
  2. Port binding conflict: Port 80 or 443 is already claimed by NGINX, Apache, or another process.
  3. Incorrect directory permissions on Caddy log paths or config files.

Investigation & Diagnosis

  1. Validate Caddyfile syntax:
caddy validate --config /etc/caddy/Caddyfile
  1. Check network port bindings:
ss -tlnp | grep -E ':80|:443'
  1. Inspect detailed systemd logs:
journalctl -u caddy -n 50 --no-pager

Resolution

  • If syntax error: Correct the broken directive or missing brace identified by caddy validate.
  • If port conflict: Stop conflicting web server services:
sudo systemctl stop nginx
sudo systemctl disable nginx
  • Re-run validation and start service:
caddy validate --config /etc/caddy/Caddyfile
systemctl start caddy

Verification

systemctl status caddy

Scenario 2: Split-Brain State (API vs. Caddyfile Reload Overwrites)

Symptoms

Runtime API changes issued via curl localhost:2019/load suddenly vanish after a system restart or server reboot.

+-------------------------------------------------------------------------+
|                         SPLIT-BRAIN CONFIG TRAP                         |
|                                                                         |
|  1. Admin updates server via REST API    -->  Active Runtime Modified   |
|     (curl -X POST localhost:2019/load)                                  |
|                                                                         |
|  2. SysAdmin runs systemctl reload caddy -->  Reads /etc/caddy/Caddyfile|
|     (Disk file silently overwrites runtime state!)                      |
+-------------------------------------------------------------------------+

Possible Causes

Mixing configuration control methods. Caddy allows active runtime modifications via its REST API, but running systemctl reload caddy forces Caddy to re-parse /etc/caddy/Caddyfile from disk, wiping out in-memory API modifications.

Investigation & Diagnosis

  1. Query active running JSON configuration:
curl http://localhost:2019/config/
  1. Compare running configuration output against disk Caddyfile:
caddy adapt --config /etc/caddy/Caddyfile

Resolution

Establish a single source of truth per operational environment.

  • Option A (File-Based): Disallow runtime API updates. Make all changes directly in /etc/caddy/Caddyfile and execute systemctl reload caddy.
  • Option B (API-Based): Export running JSON state back to disk before reloading:
curl http://localhost:2019/config/ > /etc/caddy/saved_config.json

Scenario 3: ACME Certificate Provisioning Failure

Symptoms

Clients receive browser security warnings (ERR_CERT_AUTHORITY_INVALID) or connections hang during TLS handshake. Journal logs report ACME challenge failed.

Possible Causes

  1. External DNS A/AAAA records do not point to this server’s public IP address.
  2. Network firewall blocks inbound TCP port 80 (required for ACME HTTP-01 challenge validation).
  3. Server system clock is drifted or out of sync.
  4. Let’s Encrypt API rate limits exceeded due to repeated failed retry loops.

Investigation & Diagnosis

  1. Verify public DNS resolution:
dig +short example.com
  1. Test port 80 accessibility from an external system:
curl -I http://example.com/.well-known/acme-challenge/test
  1. Check system time status:
timedatectl
  1. Search journal logs for ACME protocol errors:
journalctl -u caddy -g "acme" --no-pager

Resolution

  • Correct DNS records at domain registrar.
  • Open firewall ports on host and cloud provider security groups:
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
  • If domain is internal, switch to DNS-01 challenge mode using tls { dns ... }.

Scenario 4: HTTP 502 Bad Gateway Error on Proxy Request

Symptoms

Client browsers receive 502 Bad Gateway when trying to access proxied application endpoints.

Client  ------>  Caddy (Proxy)  --X (TCP Refused)-->  Upstream App (127.0.0.1:8000)

Possible Causes

  1. Backend application service is stopped or crashed.
  2. Backend application is listening on a different host interface or port than configured in Caddyfile (e.g., listening on 127.0.0.1 while Caddy targets 10.0.0.5).
  3. Host firewall (iptables/ufw) is blocking internal connection between Caddy and upstream.

Investigation & Diagnosis

  1. Test upstream app directly from Caddy host using curl:
curl -v http://127.0.0.1:8000/
  1. Inspect listening ports on local machine:
ss -tlnp | grep 8000
  1. Check Caddy logs for upstream failure specifics:
journalctl -u caddy -g "dial tcp" --no-pager

Resolution

  • Start or restart backend application service.
  • Update Caddyfile target address to match upstream listener interface:
reverse_proxy 127.0.0.1:8000
  • Reload Caddy:
systemctl reload caddy

Scenario 5: HTTP 404 Errors on Valid Routing Paths

Symptoms

Requests to expected paths return 404 Not Found.

Possible Causes

  1. handle_path directive was used instead of handle, inadvertently stripping the URI prefix before forwarding to backend.
  2. Document root directory path is wrong or missing files.
  3. Incorrect file system permissions prevent caddy user from reading static files.

Investigation & Diagnosis

  1. Parse actual generated URI handling via caddy adapt:
caddy adapt --config /etc/caddy/Caddyfile --pretty
  1. Verify directory existence and file readability:
ls -la /var/www/my-site

Resolution

  • If using handle_path /api/*, note that /api/users is forwarded to upstream as /users. Switch to handle /api/* if upstream expects the /api prefix.
  • Fix static directory ownership:
sudo chown -R caddy:caddy /var/www/my-site
sudo chmod -R 755 /var/www/my-site

Scenario 6: Infinite HTTPS Redirect Loops

Symptoms

Browser fails to load site and displays TOO_MANY_REDIRECTS.

Possible Causes

Caddy is behind an edge CDN/Load Balancer (e.g., Cloudflare Flexible SSL) terminating HTTPS and connecting to Caddy over HTTP on port 80. Caddy attempts to issue an automatic HTTP-to-HTTPS redirect back to the client, creating a continuous loop.

Investigation & Diagnosis

Trace redirect hops using curl:

curl -vIL http://example.com

Resolution

Explicitly set site block address to http:// or adjust incoming request handling:

http://example.com {
    reverse_proxy 127.0.0.1:8000
}

Or configure CDN SSL mode to Full (Strict) so CDN connects to Caddy over HTTPS (port 443).


Scenario 7: Data Directory Permission Denied

Symptoms

Caddy log displays write permission failures to storage path /var/lib/caddy during certificate renewal.

Possible Causes

Directory permissions modified manually or owned by root instead of unprivileged caddy service user.

Investigation & Diagnosis

ls -la /var/lib/caddy

Resolution

Restore correct recursive ownership to caddy user and group:

sudo chown -R caddy:caddy /var/lib/caddy
sudo chmod 700 /var/lib/caddy

11. Security and Best Practices

1. Privilege Minimization & Service Hardening

  • Never run Caddy directly as the root administrative superuser in production.
  • Systemd service should employ Linux capabilities (CAP_NET_BIND_SERVICE) to allow binding low-numbered ports (80/443) as unprivileged user caddy.

2. Mandatory TLS Hardening

Restrict acceptable TLS protocols to standard modern specs within global options:

{
    tls {
        protocols tls1.2 tls1.3
    }
}

3. HTTP Basic Authentication (basic_auth)

Protect staging or internal administrative endpoints with bcrypt-hashed credentials:

Generate hash using Caddy CLI:

caddy hash-password --plaintext "MySecurePassword123"

Apply in Caddyfile:

staging.example.com {
    basic_auth {
        admin $2a$14$Z8...hashed_password_string...
    }
    reverse_proxy 127.0.0.1:8080
}

4. Injecting Standard Production Security Headers

(security_headers) {
    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "SAMEORIGIN"
        Referrer-Policy "strict-origin-when-cross-origin"
    }
}

example.com {
    import security_headers
    reverse_proxy 127.0.0.1:8080
}

12. Performance and Resource Optimisation

1. Upstream Keep-Alive Connection Pooling

Maintain persistent backend HTTP connections in reverse_proxy to eliminate TCP connection setup overhead for high-traffic microservices:

example.com {
    reverse_proxy 127.0.0.1:8080 {
        transport http {
            keepalive 30s
            keepalive_idle_conns 100
        }
    }
}

2. High-Performance Static Compression

Zstandard (zstd) delivers superior compression ratios and speed compared to traditional Gzip:

example.com {
    encode zstd gzip
    file_server
}

3. System Limits Hardening (ulimit & systemd)

Ensure systemd unit /lib/systemd/system/caddy.service allows high open file descriptor counts for concurrent TCP sockets:

[Service]
LimitNOFILE=65536
LimitNPROC=51200

Apply limit changes:

sudo systemctl daemon-reload
sudo systemctl restart caddy

13. Backup, Recovery and Maintenance

Essential Files for Backup

  1. Configuration Files: /etc/caddy/Caddyfile and custom included modules.
  2. ACME Data & Certificates: /var/lib/caddy/ (Backing up this directory prevents rate limit hits with ACME providers upon re-deployment).

Automated Backup Script Example

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR="/var/backups/caddy"
DATE=$(date +%Y%m%m_%H%M%S)
TARGET_ARCHIVE="${BACKUP_DIR}/caddy_backup_${DATE}.tar.gz"

mkdir -p "${BACKUP_DIR}"

# Archive configuration and ACME state
tar -czf "${TARGET_ARCHIVE}" \
    /etc/caddy/Caddyfile \
    /var/lib/caddy

echo "Backup created successfully at ${TARGET_ARCHIVE}"

Upgrade & Maintenance Procedure

# 1. Backup existing configuration and certificates
sudo tar -czf /tmp/caddy_pre_upgrade.tar.gz /etc/caddy /var/lib/caddy

# 2. Update repository package
sudo apt update && sudo apt install --only-upgrade caddy

# 3. Validate syntax post-upgrade
caddy validate --config /etc/caddy/Caddyfile

# 4. Graceful reload
systemctl reload caddy

14. Automation and Integration

Docker Compose Production Setup

Deploying Caddy alongside application containers using persistent volume mounts:

version: '3.8'

services:
  caddy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp" # HTTP/3 (QUIC) support
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config

volumes:
  caddy_data:
  caddy_config:

15. Practical Labs and Exercises

Exercise 1 (Beginner): Setup Static Site with Dynamic Compression

  • Objective: Serve HTML content out of /var/www/lab1 on domain lab1.local with Gzip enabled.
  • Setup Instructions:
  1. Create folder /var/www/lab1 and place an index.html file inside.
  2. Create site block in /etc/caddy/Caddyfile:
http://lab1.local {
    root * /var/www/lab1
    encode gzip
    file_server
}
  1. Validate and reload Caddy.
  • Validation: Execute curl -H "Accept-Encoding: gzip" -I [http://lab1.local](http://lab1.local) and confirm Content-Encoding: gzip header exists.

Exercise 2 (Intermediate): High-Availability Failover Proxy

  • Objective: Configure a load-balanced proxy pointing to primary application 127.0.0.1:8001 and backup 127.0.0.1:8002.
  • Setup Instructions:
  1. Start two simple HTTP python servers:
python3 -m http.server 8001 &
python3 -m http.server 8002 &
  1. Write proxy site block with health checking:
http://lab2.local {
    reverse_proxy 127.0.0.1:8001 127.0.0.1:8002 {
        lb_policy first
        health_uri /
        health_interval 5s
    }
}
  • Validation: Stop process on 8001 (kill %1) and observe Caddy routing requests to 8002 without client failure.

Exercise 3 (Troubleshooting Lab): Recovering from Port Conflict Crash

  • Objective: Diagnose and fix a failed Caddy startup caused by an unexpected process holding port 80.
  • Lab Setup:
  1. Run nc -l -p 80 & to occupy port 80.
  2. Attempt to restart Caddy: systemctl restart caddy.
  • Troubleshooting Steps:
  1. Execute systemctl status caddy to observe failure state.
  2. Run ss -tlnp | grep ':80' to identify conflicting process PID.
  3. Kill conflicting process and restart Caddy:
sudo kill -9 <PID>
sudo systemctl start caddy
  1. Confirm operational status.

16. Quick Reference and Cheat Sheet

Important Configuration Files & Directories

Path Purpose
/etc/caddy/Caddyfile Primary configuration file.
/var/lib/caddy Data directory for certificates and ACME state.
/var/log/caddy Standard log directory.

Essential Operational Commands

Administrative Goal Command
Service Status systemctl status caddy
Validate Config caddy validate --config /etc/caddy/Caddyfile
Format Caddyfile caddy fmt --overwrite /etc/caddy/Caddyfile
Graceful Reload systemctl reload caddy OR caddy reload
Debug Foreground Run caddy run --config /etc/caddy/Caddyfile
Convert to JSON caddy adapt --config /etc/caddy/Caddyfile

Network Ports & Protocols

Port Protocol Usage
80 TCP Plain HTTP traffic and ACME HTTP-01 challenge validation.
443 TCP / UDP Encrypted HTTPS traffic (TCP for HTTP/1.1 & HTTP/2; UDP for HTTP/3 QUIC).
2019 TCP Local Administrative REST API endpoint (bound to localhost).

17. Further Reading and References

  1. Official Caddy Documentation: https://caddyserver.com/docs
  2. Caddyfile Tutorial & Syntax: https://caddyserver.com/docs/caddyfile
  3. Caddy reverse_proxy Directive Reference: https://caddyserver.com/docs/caddyfile/directives/reverse_proxy
  4. Caddy Automatic HTTPS Architecture: https://caddyserver.com/docs/automatic-https