1. Overview
Traefik is a modern HTTP reverse proxy and load balancer engineered for cloud-native architectures and microservices environments. Positioned at the edge of an infrastructure stack, Traefik acts as the ingress controller or edge gateway, terminating incoming HTTP and HTTPS traffic and forwarding requests to downstream application backends, container networks, or Kubernetes pods.
Traditional reverse proxies like Nginx or HAProxy relied heavily on static, file-based configuration models where server blocks and backend IP addresses were explicitly declared and reloaded whenever infrastructure changes occurred. Traefik fundamentally differs by introducing dynamic configuration discovery. By continuously listening to infrastructure orchestration APIs—such as the Docker daemon, Kubernetes API server, or HashiCorp Consul—Traefik automatically detects when containers or pods are launched, updated, or stopped, reconfiguring its routing tables in real time without dropping active connections or requiring manual process reloads.
+---------------------------------------+
| Client Request |
+---------------------------------------+
|
v
+---------------------------------------+
| Traefik Reverse Proxy |
| (Edge Gateway / Port 80 & 443) |
+---------------------------------------+
/ | \
Dynamic API / | \ Dynamic API
Discovery v v v Discovery
+--------------------+ +--------------------+ +--------------------+
| Docker Container A | | Docker Container B | | Kubernetes Pod C |
| (api.example.com) | | (app.example.com) | | (web.example.com) |
+--------------------+ +--------------------+ +--------------------+
Core Problem Solved
In containerized, ephemerally scaled environments, instances frequently scale up, down, or migrate across host nodes, generating unpredictable IP addresses and internal ports. Manually maintaining static upstream server lists in legacy proxies under high-frequency deployments leads to configuration drift, deployment delays, and service outages during configuration reloads. Traefik solves this by shifting routing configuration directly onto the workloads using metadata (such as Docker container labels or Kubernetes Custom Resources), eliminating manual proxy reloads entirely.
Common Use Cases
- Edge Ingress Gateway for Container Clusters: Serving as the entry point for Docker Compose, Docker Swarm, and Kubernetes deployments.
- Automated TLS Termination: Automatically issuing, configuring, and renewing wildcard or single-domain SSL/TLS certificates using Let’s Encrypt ACME protocols without external scripts or cron jobs.
- Microservices Routing & Traffic Management: Routing requests based on HTTP headers, path prefixes, hostnames, or query parameters while applying request transforms, rate limiting, and authentication middleware.
- Multi-Tenant Edge Gateways: Isolating traffic across namespaces or environments via dynamic path and hostname rules.
Advantages and Limitations
| Metric / Aspect | Traefik | Traditional Nginx / HAProxy | Caddy |
|---|---|---|---|
| Primary Use Case | Cloud-native microservices, Docker, Kubernetes | Static infrastructure, high-throughput raw reverse proxying, legacy web servers | Developer-friendly static sites, automatic TLS, simple file configuration |
| Config Discovery | Fully dynamic via API integration (Docker, K8s, Consul) | Static configuration files requiring SIGHUP/reload | Dynamic JSON API or static Caddyfile |
| ACME Integration | Built-in native support for HTTP, TLS-ALPN, and DNS challenges | Requires external tools (Certbot, acme.sh) | Built-in native support |
| Performance Profile | High throughput (Go runtime), optimized for rapid routing updates | Ultra-high raw HTTP throughput & minimal memory footprint (C runtime) | Moderate to high throughput (Go runtime) |
| Configuration Model | Dual: Static startup file + Dynamic Provider metadata | Single or nested configuration files | Caddyfile or JSON REST API |
2. Architecture and Mental Model
To effectively operate and troubleshoot Traefik in production, SREs must understand its internal packet processing model and component architecture. Traefik disconnects network listener definitions from request handling rules and backend upstream destinations, organizing traffic handling into four distinct conceptual primitives: Entrypoints, Routers, Middlewares, and Services.
graph TD
Client[Client Request] -->|Port 80 / 443| EP[Entrypoint: web / websecure]
EP --> Router{Router Rule Match?}
Router -->|No Match| R404[Return HTTP 404 Not Found]
Router -->|Match: Host / Path| MW[Middleware Chain]
MW -->|Auth / Headers / Redirect / RateLimit| MW
MW --> Svc[Service / Load Balancer]
Svc -->|Round Robin / Weighted| Backend1[Backend: Container / Pod IP:8080]
Svc -->|Round Robin / Weighted| Backend2[Backend: Container / Pod IP:8080]
Main Components and Responsibilities
- Entrypoints (Network Listeners):
- Entrypoints define the physical network sockets (IP addresses and TCP/UDP ports) on which Traefik listens for incoming client connections. Common entrypoints include
web(listening on port 80) andwebsecure(listening on port 443). - Entrypoints act as the entry boundary and pass incoming raw traffic into Traefik’s routing core.
- Routers (Traffic Matching Engine):
- Routers evaluate incoming requests from entrypoints against user-defined matching rules (such as
Host(),PathPrefix(),Headers(), orQuery()). - When a rule matches, the router routes the request through associated Middlewares to a target Service. Routers also bind TLS configurations and ACME certificate resolvers to specific domains.
- Middlewares (Request/Response Pipeline):
- Middlewares modify requests before they reach the service backend, or adjust responses before returning them to the client.
- Executed sequentially in a defined chain, middlewares perform functions such as HTTP basic authentication, header injection, rate limiting, URL path stripping (
stripPrefix), and HTTPS redirection.
- Services (Load Balancers & Target Forwarders):
- Traefik Services configure how requests are load-balanced across real backend workloads.
- A service defines backend server addresses (IP and port pairs discovered dynamically from containers or pods), health checking criteria, sticky sessions, and load-balancing algorithms (such as weighted round-robin).
Data Flow and Request Lifecycle
- Connection Acceptance: A client establishes a TCP connection to an Entrypoint (e.g., port 443). If TLS is configured, Traefik completes the TLS handshake using the certificate managed by the active certificate resolver or dynamic configuration.
- Router Evaluation: Traefik examines HTTP request metadata (Host header, path, protocol scheme) and selects the matching Router with the highest priority rule. If no router matches, Traefik immediately returns an HTTP 404 error.
- Middleware Processing: The request passes through the Middleware chain assigned to the router sequentially. If a middleware rejects the request (e.g., failed authentication or rate limit reached), execution halts, and the middleware returns an appropriate error response (e.g., HTTP 401 or 429).
- Service Load Balancing: The passed request is dispatched to the router’s assigned Service. The service selects an active, healthy backend server IP and port from its dynamically updated endpoint pool.
- Backend Dispatch & Response: Traefik proxies the request over the internal network to the chosen container or pod IP, waits for the downstream response, applies any response-modifying middlewares, and streams the response back to the client.
3. Core Concepts and Terminology
1. Static Configuration vs. Dynamic Configuration
Traefik enforces a strict operational boundary between static and dynamic configuration models:
- Static Configuration: Defines startup-time system parameters, including network entrypoints, active infrastructure providers (e.g., Docker, Kubernetes, File), ACME certificate resolvers, logging formats, metrics exporters, and dashboard settings. Static configuration is read once during binary execution; any modifications require a full process restart.
- Dynamic Configuration: Defines runtime routing rules, routers, services, middlewares, and TLS certificates. Dynamic configuration is discovered live from external infrastructure providers without restarting Traefik or interrupting active client connections.
2. Configuration Providers
Providers are external infrastructure engines that supply dynamic routing metadata to Traefik:
- Docker Provider: Scans container labels attached to Docker containers or Swarm services to dynamically build routers and services.
- Kubernetes Providers: Watches Kubernetes API resources, supporting standard
Ingress, TraefikIngressRouteCRDs, and the KubernetesGateway API. - File Provider: Monitors a directory or explicit YAML/TOML file on local storage for dynamic routing rules, reloading configurations instantly upon file save operations.
3. Rule Matching Syntax
Routers use expressive matching expressions to filter traffic:
Host(api.example.com): Matches exact HTTP Host request headers.PathPrefix(/v1): Matches requests whose URL paths start with/v1.Headers(X-Custom-Header,value): Matches specific request headers.- Combining rules with logical operators:
Host(api.example.com) && PathPrefix(/v1).
4. ACME & Automatic Certificate Resolvers
Traefik contains built-in Automated Certificate Management Environment (ACME) clients for requesting, validating, and automatically renewing SSL/TLS certificates from Let’s Encrypt or ZeroSSL. Resolvers support HTTP-01, TLS-ALPN-01, and DNS-01 challenge types.
4. Installation and Version Management
Prerequisites and Dependencies
- Linux kernel 3.10+ (64-bit architecture:
amd64,arm64). - Root or
sudoprivileges for binding to privileged ports (:80,:443). - System container runtime (Docker v20.10+ or Kubernetes v1.22+) if using container providers.
Installation Methods
Method A: Direct Binary Download (Systemd Managed)
# 1. Download official binary release
curl -sL https://github.com/traefik/traefik/releases/download/v3.3.0/traefik_v3.3.0_linux_amd64.tar.gz -o traefik.tar.gz
# 2. Extract binary and install to system path
tar -xvf traefik.tar.gz
sudo mv traefik /usr/local/bin/
sudo chmod +x /usr/local/bin/traefik
# 3. Verify binary version
traefik version
Method B: Docker / Docker Compose Deployment
# docker-compose.yml
version: '3.8'
services:
traefik:
image: traefik:v3.3
container_name: traefik
restart: always
command:
- "--configFile=/etc/traefik/traefik.yml"
ports:
- "80:80"
- "443:443"
- "8080:8080" # Traefik Dashboard / API
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./traefik.yml:/etc/traefik/traefik.yml:ro
- ./dynamic:/etc/traefik/dynamic:ro
- ./acme.json:/letsencrypt/acme.json
networks:
- proxy-net
networks:
proxy-net:
external: true
Method C: Kubernetes Deployment via Helm
# Add Traefik Helm repository
helm repo add traefik https://traefik.github.io/charts
helm repo update
# Install Traefik into dedicated namespace
helm install traefik traefik/traefik \
--namespace traefik \
--create-namespace \
--set ports.web.redirectTo=websecure \
--set ingressClass.enabled=true
Installation Files and Directory Structure Standard
| Location | Purpose |
|---|---|
/usr/local/bin/traefik |
Main compiled executable binary. |
/etc/traefik/traefik.yml |
Static configuration file (YAML/TOML). |
/etc/traefik/dynamic/ |
Directory containing dynamic file configuration manifests. |
/var/log/traefik/ |
Directory storing access logs and process stdout/stderr output. |
/etc/traefik/acme.json |
Encrypted JSON storage holding ACME private keys and certificates. |
5. Configuration Files and Directory Structure
Traefik accepts static configuration in YAML, TOML, or CLI flag formats. YAML is the standard choice for modern DevOps deployments.
Comprehensive File Reference Table
| Path / File | Configuration Type | Purpose | Reload Required on Change? |
|---|---|---|---|
/etc/traefik/traefik.yml |
Static | Defines entrypoints, active providers, ACME resolvers, dashboard, and logs. | Yes (Requires binary restart) |
/etc/traefik/dynamic/*.yml |
Dynamic (File Provider) | Defines custom routers, middleware chains, static TLS certs, and manual services. | No (Hot-reloaded automatically) |
/etc/systemd/system/traefik.service |
System Unit | Systemd unit file controlling daemon lifecycle. | systemctl daemon-reload |
/letsencrypt/acme.json |
State Storage | Encrypted certificate store for ACME resolvers. | No (Managed by Traefik engine) |
Static Configuration Example (/etc/traefik/traefik.yml)
# Global configuration
global:
checkNewVersion: false
sendAnonymousUsage: false
# Operational Logging
log:
level: INFO # DEBUG, INFO, WARN, ERROR
format: json
# HTTP Access Logs
accessLog:
filePath: "/var/log/traefik/access.log"
format: json
bufferingSize: 100
# Entrypoints (Network Listeners)
entryPoints:
web:
address: ":80"
http:
redirections:
entryPoint:
to: websecure
scheme: https
permanent: true
websecure:
address: ":443"
http:
tls:
certResolver: letsencrypt
# Built-in Administrative API & Web Dashboard
api:
dashboard: true
insecure: false # Set false in production; bind securely via router
# Dynamic Providers Setup
providers:
docker:
endpoint: "unix:///var/run/docker.sock"
exposedByDefault: false # Requires traefik.enable=true on containers
watch: true
file:
directory: "/etc/traefik/dynamic"
watch: true
# Certificate Resolvers (ACME / Let's Encrypt)
certificatesResolvers:
letsencrypt:
acme:
email: ops@example.com
storage: /letsencrypt/acme.json
httpChallenge:
entryPoint: web
Dynamic File Configuration Example (/etc/traefik/dynamic/middleware_and_routes.yml)
http:
# Dynamic Routers
routers:
secure-dashboard:
entryPoints:
- websecure
rule: "Host(`traefik.example.com`)"
service: api@internal # Built-in internal service for API/Dashboard
middlewares:
- admin-auth
- security-headers
tls:
certResolver: letsencrypt
external-api:
entryPoints:
- websecure
rule: "Host(`api.example.com`) && PathPrefix(`/v1`)"
service: api-service
middlewares:
- api-ratelimit
- strip-api-prefix
tls:
certResolver: letsencrypt
# Dynamic Middlewares
middlewares:
admin-auth:
basicAuth:
users:
# Generated using htpasswd -nb admin securepassword
- "admin:$apr1$q86I8P45$0G5Yv1wXqYd12.Jk9/yvI."
strip-api-prefix:
stripPrefix:
prefixes:
- "/v1"
api-ratelimit:
rateLimit:
average: 100
burst: 50
security-headers:
headers:
sslRedirect: true
stsSeconds: 31536000
stsIncludeSubdomains: true
frameDeny: true
contentTypeNosniff: true
# Dynamic Services
services:
api-service:
loadBalancer:
servers:
- url: "http://10.0.1.50:8080"
- url: "http://10.0.1.51:8080"
healthCheck:
path: /healthz
interval: "10s"
timeout: "3s"
6. Service Management and Common Operations
Systemd Service Definition (/etc/systemd/system/traefik.service)
[Unit]
Description=Traefik Edge Reverse Proxy
Documentation=https://doc.traefik.io/traefik/
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
User=traefik
Group=traefik
ExecStart=/usr/local/bin/traefik --configFile=/etc/traefik/traefik.yml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
Essential System Operations Commands
- Start Service:
sudo systemctl start traefik
- Stop Service:
sudo systemctl stop traefik
- Enable on Boot:
sudo systemctl enable traefik
- Check Running Status:
sudo systemctl status traefik
- Validate Listening Ports:
sudo ss -tlnp | grep traefik
- Reload vs. Restart Distinction:
- Restart (
systemctl restart traefik): Destroys and recreates the Traefik process. Required when modifying static configurations (traefik.yml) such as adding new entrypoints, changing provider socket paths, or modifying ACME resolvers. - Dynamic Reload (Automatic): Traefik watches dynamic file directories, Docker sockets, and Kubernetes API events. Modifying dynamic files, labels, or CRDs causes Traefik to hot-reload routing tables in memory instantaneously without dropping active connections or restarting.
7. Common Administrative Tasks and Workflows
Workflow A: Exposing a New Microservice via Docker Labels
Objective
Deploy a backend web application using Docker Compose and safely expose it through Traefik with automatic TLS termination and rate limiting.
Prerequisites
- Traefik running on a shared Docker network named
proxy-net. - Static configuration has
certificatesResolvers.letsencryptconfigured.
Step-by-Step Procedure
- Create the
docker-compose.ymlfor the target application:
version: '3.8'
services:
web-app:
image: nginx:alpine
container_name: web-app
restart: always
networks:
- proxy-net
labels:
# Explicitly enable Traefik discovery for this container
- "traefik.enable=true"
# Define the Router matching rule
- "traefik.http.routers.webapp.rule=Host(`app.example.com`)"
# Assign entrypoint and TLS resolver
- "traefik.http.routers.webapp.entrypoints=websecure"
- "traefik.http.routers.webapp.tls.certresolver=letsencrypt"
# Define target container internal port
- "traefik.http.services.webapp.loadbalancer.server.port=80"
# Assign a dynamic rate-limiting middleware
- "traefik.http.routers.webapp.middlewares=app-ratelimit@docker"
- "traefik.http.middlewares.app-ratelimit.ratelimit.average=50"
networks:
proxy-net:
external: true
- Launch the application:
docker compose up -d
- Validation: Query the endpoint end-to-end to verify TLS issuance and HTTP routing:
curl -vI https://app.example.com
Workflow B: Exposing Services on Kubernetes via IngressRoute CRD
Objective
Route traffic to a Kubernetes deployment using Traefik’s native custom resource definition (IngressRoute).
Step-by-Step Procedure
- Verify Traefik CRDs are installed in the cluster:
kubectl get crd | grep traefik.io
- Create the
ingressroute.yamldefinition:
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: payment-service-route
namespace: production
spec:
entryPoints:
- websecure
routes:
- match: Host(`pay.example.com`) && PathPrefix(`/api`)
kind: Rule
services:
- name: payment-backend-svc
port: 8080
middlewares:
- name: strip-api-prefix
namespace: production
tls:
certResolver: letsencrypt
- Apply the resource:
kubectl apply -f ingressroute.yaml
- Validation: Check the operational status of the IngressRoute:
kubectl describe ingressroute payment-service-route -n production
8. CLI Command Reference and Cheatsheet
1. Process and Version Inspection
| Command | Purpose | Important Flags / Arguments | Example | Explanation |
|---|---|---|---|---|
traefik version |
Displays binary build version and execution architecture. | None | traefik version |
Outputs version, commit hash, and build date. |
traefik |
Launches the Traefik daemon process. | --configFile= |
traefik --configFile=/etc/traefik/traefik.yml |
Starts proxy with specified static config. |
ss / netstat |
Inspects active listening ports for network sockets. | -tlnp |
`ss -tlnp | grep traefik` |
2. Runtime API and Diagnostic Endpoints
Traefik exposes an administrative REST API (typically on port 8080) for real-time state inspection.
| Endpoint / Command | Purpose | Response Type | Example Usage |
|---|---|---|---|
/api/http/routers |
Lists all active HTTP routers and matching rules. | JSON | `curl -s http://localhost:8080/api/http/routers |
/api/http/services |
Lists discovered dynamic load balancers & backend IPs. | JSON | `curl -s http://localhost:8080/api/http/services |
/api/http/middlewares |
Inspects deployed request/response middlewares. | JSON | `curl -s http://localhost:8080/api/http/middlewares |
/api/rawdata |
Dumps complete combined dynamic configuration snapshot. | JSON | `curl -s http://localhost:8080/api/rawdata |
/ping |
Health-check endpoint for load balancers. | Plain Text | curl -i http://localhost:8080/ping |
3. Kubernetes Diagnostics Reference
| Command | Purpose | Key Flags | Example |
|---|---|---|---|
kubectl get ingressroute |
Lists all Traefik IngressRoute CRDs. | -A (All Namespaces) |
kubectl get ingressroute -A |
kubectl describe ingressroute |
Inspects route status, events, and syntax errors. | -n <namespace> |
kubectl describe ingressroute myroute -n production |
kubectl logs |
Fetches controller operational logs. | -l app.kubernetes.io/name=traefik |
kubectl logs -n traefik -l app.kubernetes.io/name=traefik --tail=100 |
kubectl get endpoints |
Confirms downstream pod IPs exist for target Service. | -n <namespace> |
kubectl get endpoints backend-svc -n production |
4. Edge Connectivity and Route Testing
# Test router host matching locally by overriding HTTP Host Header
curl -vI -H "Host: api.example.com" http://127.0.0.1/
# Trace full HTTPS TLS handshake and certificate chain end-to-end
curl -vIL https://api.example.com
# Verify public DNS resolution prior to requesting Let's Encrypt certificates
dig +short api.example.com
9. Logs, Monitoring and Diagnostics
Log File Locations and Formats
Traefik emits two distinct log streams:
- Traefik System Logs: Process output containing internal state updates, provider discovery events, ACME certificate logs, and error traces.
- Access Logs: Structured HTTP request records details including client IPs, requested paths, response status codes, execution latency, and assigned router names.
Access Log Example (JSON Format)
{
"ClientAddr": "192.168.1.100:54321",
"ClientHost": "192.168.1.100",
"RequestMethod": "GET",
"RequestPath": "/api/v1/users",
"OriginStatus": 200,
"OriginContentSize": 1042,
"RequestCount": 4210,
"RouterName": "api-service-router@docker",
"ServiceName": "api-service@docker",
"Duration": 12450122,
"StartLocal": "2026-11-03T10:15:30.123456Z"
}
Live Log Streaming and Filtering
- Follow Systemd Journal Logs:
journalctl -u traefik -f -o cat
- Follow Docker Container Logs:
docker logs traefik -f --tail 100
- Filter Access Logs for HTTP 5xx Server Errors:
tail -f /var/log/traefik/access.log | grep -E '"OriginStatus":(500|502|503|504)'
10. Troubleshooting Guide and Common Failure Scenarios
Scenario 1: HTTP 404 Not Found — Router Not Found
Symptoms
Clients receive an instant HTTP 404 Not Found response generated directly by Traefik when requesting a hosted application.
$ curl -i http://api.example.com/
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
Length: 19
404 page not found
Possible Causes
- Traefik has not loaded a router matching the incoming
Host()orPathPrefix()rules. - On Docker: The container lacks the required
traefik.enable=truelabel, or Traefik is set toexposedByDefault: false. - Entrypoint mismatch: The router is bound to entrypoint
websecure(:443), but the client sent a request to entrypointweb(:80) without HTTPS redirection. - On Kubernetes: Traefik is not configured to watch the namespace in which the
IngressRouteresides.
Investigation and Diagnosis
Step 1: Query Traefik API for Active Routers
State check: Determine if Traefik registered the router in its in-memory routing table.
curl -s http://localhost:8080/api/http/routers | jq '.[] | select(.name | contains("myapp"))'
- If output is empty: Traefik failed to discover the router from the provider. Move to Step 2.
- If router exists: Verify that the rule (
Host(...)) exactly matches the hostname requested by the client.
Step 2: Verify Docker Container Labels and Networks
Inspect container metadata directly from the Docker daemon:
docker inspect my-container-name --format '{{json .Config.Labels}}' | jq .
Verify:
"traefik.enable"is set to"true".- Traefik and the container share a common Docker network.
Step 3: Verify Kubernetes Namespace Scope
For Kubernetes deployments, confirm Traefik RBAC permissions and watched namespaces:
kubectl get ingressroute -n my-app-namespace
Resolution
- For Docker: Ensure correct label syntax and shared network attachment:
labels:
- "traefik.enable=true"
- "traefik.http.routers.myapp.rule=Host(`api.example.com`)"
- "traefik.docker.network=proxy-net"
- For Entrypoint Mismatches: Ensure the router specifies both entrypoints if dual-binding is intended:
traefik.http.routers.myapp.entrypoints=web,websecure
Verification
Execute a request explicitly enforcing the Host header:
curl -vI -H "Host: api.example.com" http://127.0.0.1/
Expected: HTTP/1.1 200 OK (or appropriate backend response code).
Scenario 2: HTTP 502 Bad Gateway / HTTP 503 Service Unavailable
Symptoms
Traefik successfully matches the request to a router, but returns HTTP 502 Bad Gateway or HTTP 503 Service Unavailable to the client.
$ curl -i https://app.example.com/
HTTP/2 502
content-length: 11
Bad Gateway
Possible Causes
- Port Mismatch: The service load balancer port configured in Traefik points to the host-mapped port rather than the container’s internal listening port.
- Unreachable Backend Network: Traefik cannot reach the backend IP address due to network segmentation or missing Docker network attachments.
- Failed Health Checks: Traefik’s health checker has marked all downstream backend instances as unhealthy.
- Backend Process Crashed: The container or application inside the pod crashed or is not accepting connections.
Investigation and Diagnosis
Step 1: Check Traefik Internal Logs
Search system logs for connection attempt errors:
docker logs traefik 2>&1 | grep -E '502|503|dial tcp|connection refused'
Look for: dial tcp 172.18.0.5:8080: connect: connection refused.
Step 2: Inspect Discovered Service Port
Query the Traefik API to see what IP and port Traefik is trying to contact:
curl -s http://localhost:8080/api/http/services | jq '.[].loadBalancer.servers'
Compare the resulting IP/port against the container’s actual internal exposed port (EXPOSE).
Step 3: Test Direct Network Connectivity
Execute a direct request from inside the Traefik container to the target backend IP:
docker exec -it traefik wget --spider -T 3 http://172.18.0.5:8080/healthz
Resolution
Explicitly declare the container’s internal listening port using the server port label:
# Correct: Target internal container port (e.g., 8080), NOT host port (e.g., 80)
- "traefik.http.services.myapp.loadbalancer.server.port=8080"
Verification
Re-query the dynamic service endpoint and confirm active communication:
curl -s http://localhost:8080/api/http/services | jq '.[].serverStatus'
Scenario 3: TLS / Let’s Encrypt Certificate Generation Failure
Symptoms
Traefik serves a self-signed fallback certificate (TRAEFIK DEFAULT CERTIFCATE) instead of a valid Let’s Encrypt TLS certificate, causing browser security warnings.
Possible Causes
- Public Reachability: Let’s Encrypt HTTP-01 challenge cannot reach Traefik on port 80 from the public internet.
- File Permissions: Storage file
/letsencrypt/acme.jsonhas overly permissive file permissions (must be strictly600). - DNS Misconfiguration: Public DNS record does not point to the external IP address of the Traefik host.
- Let’s Encrypt Rate Limits: Exceeded failed validation rate limits for the target domain.
Investigation and Diagnosis
Step 1: Check ACME Operational Logs
Filter Traefik logs specifically for ACME validation events:
docker logs traefik 2>&1 | grep -i "acme"
Common error logs:
ACME error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from...unable to get ACME account: permissions on /letsencrypt/acme.json are 0644 instead of 0600
Step 2: Verify acme.json Permissions
Inspect file permission mode on host storage:
ls -l /path/to/acme.json
Resolution
- Correct ownership and restrict file permissions on
acme.json:
sudo chmod 600 /path/to/acme.json
sudo chown 1000:1000 /path/to/acme.json
- Ensure Port 80 is open in firewalls (UFW/iptables) and forwarded to Traefik.
- Verify public DNS record resolution using
dig:
dig +short app.example.com
Verification
Inspect the SSL certificate presented by the live endpoint:
echo | openssl s_client -servername app.example.com -connect app.example.com:443 2>/dev/null | openssl x509 -noout -dates -issuer
Scenario 4: Dynamic HTTP Redirect Loops (Too Many Redirects)
Symptoms
Web browsers display ERR_TOO_MANY_REDIRECTS, and curl outputs an infinite sequence of HTTP 301/302 redirects.
Possible Causes
- Conflicting HTTPS Redirects: Traefik entrypoint forces HTTPS redirection, while an upstream application or secondary middleware also issues an HTTPS redirect.
- X-Forwarded-Proto Misinterpretation: Downstream backend receives plain HTTP requests from Traefik without recognizing
X-Forwarded-Proto: https, prompting the backend to repeatedly issue redirects.
Investigation and Diagnosis
Trace the HTTP redirect location chain using curl:
curl -vIL https://app.example.com
Observe whether the Location: header toggles repeatedly between http:// and https://.
Resolution
Ensure that the HTTPS redirection is managed strictly at the Traefik entrypoint level and disable duplicate HTTPS enforcement inside the application backend code. Configure backend applications to trust X-Forwarded-Proto headers supplied by Traefik.
11. Security and Best Practices
- Secure the Traefik API and Dashboard:
- Never set
api.insecure: truein production environments. - Expose the dashboard using a dedicated, secure dynamic router protected with
basicAuthorforwardAuthmiddleware and TLS.
- Restrict Docker Socket Access:
- Mounting
/var/run/docker.sockexposes the host to elevated security risks if Traefik is compromised. - Use a read-only socket mount (
/var/run/docker.sock:ro) or deploy a secure proxy sidecar likedocker-socket-proxyto restrict API endpoints to read-only container listings.
- Enforce Storage File Permissions:
- Enforce strict file mode
600on/letsencrypt/acme.jsonto safeguard private TLS keys.
- Hardening HTTP Security Headers: Apply dynamic security headers middleware across all public routers to enforce modern browser protections:
http:
middlewares:
secure-headers:
headers:
stsSeconds: 31536000
stsIncludeSubdomains: true
stsPreload: true
frameDeny: true
contentTypeNosniff: true
browserXssFilter: true
referrerPolicy: "strict-origin-when-cross-origin"
- Resource Rate Limiting:
Attach
rateLimitmiddlewares to public endpoints to defend backends against Denial-of-Service (DoS) attacks and brute-force attempts.
12. Performance and Resource Optimisation
- Tune Go Runtime Limits:
Set Linux open file descriptor limits (
ulimit -n 65536) inside systemd units or container environments to handle high numbers of concurrent TCP connections. - Access Log Buffering: In high-throughput environments, writing access logs synchronously to disk causes disk I/O bottlenecks. Enable log buffering in static configuration:
accessLog:
filePath: "/var/log/traefik/access.log"
format: json
bufferingSize: 500 # Buffer up to 500 log lines in memory before disk flush
- HTTP/2 and HTTP/3 Enablement: Traefik supports HTTP/2 by default over TLS. Enable experimental HTTP/3 (QUIC) support on entrypoints to minimize latency for mobile and high-latency clients.
- Keep-Alive Connection Pooling: Optimize upstream transport parameters in static config to reuse persistent TCP connections to backend workloads:
serversTransport:
maxIdleConnsPerHost: 200
forwardingTimeouts:
dialTimeout: "3s"
responseHeaderTimeout: "10s"
13. Backup, Recovery and Maintenance
Backup Procedures
- Static Configuration & Dynamic Files:
Back up static configuration files (
traefik.yml), dynamic file directories, and Docker Compose manifests using standard version control (Git). - ACME TLS Storage (
acme.json): Theacme.jsonfile contains private account keys and active TLS certificates. Back up this file to prevent hitting Let’s Encrypt rate limits during system recovery.
# Safely snapshot acme.json maintaining strict permissions
sudo tar -czvf traefik-acme-backup-$(date +%F).tar.gz /etc/traefik/acme.json
Upgrade and Patching Workflow
- Review release notes for breaking syntax changes between major versions.
- Pull the updated binary or container image (
traefik:v3.3). - Validate static configuration syntax offline:
traefik healthcheck --configFile=/etc/traefik/traefik.yml
- Perform zero-downtime rolling updates in Kubernetes or deploy updated system container images.
14. Automation and Integration
Prometheus Metrics Integration
Traefik natively exports detailed Prometheus metrics on an entrypoint or internal path.
Static Enablement (traefik.yml)
metrics:
prometheus:
entryPoint: metrics
addEntryPointsLabels: true
addRoutersLabels: true
addServicesLabels: true
entryPoints:
metrics:
address: ":9100"
Scraping with Prometheus (prometheus.yml)
scrape_configs:
- job_name: 'traefik'
static_configs:
- targets: ['traefik.internal.example.com:9100']
15. Practical Labs and Exercises
Exercise 1 (Beginner): Deploy Traefik and Secure a Local Web Application
Objective
Deploy Traefik using Docker Compose alongside a basic web container (nginxdemos/hello), expose it on hello.local, and inspect the router via the Traefik API.
Steps
- Create a workspace directory and define
docker-compose.yml:
version: '3.8'
services:
traefik:
image: traefik:v3.3
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
ports:
- "80:80"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
demo-app:
image: nginxdemos/hello
labels:
- "traefik.enable=true"
- "traefik.http.routers.demo.rule=Host(`hello.local`)"
- "traefik.http.routers.demo.entrypoints=web"
- "traefik.http.services.demo.loadbalancer.server.port=80"
- Launch stack:
docker compose up -d. - Test locally:
curl -H "Host: hello.local" http://localhost. - Query active routers via API:
curl -s http://localhost:8080/api/http/routers | jq.
Exercise 2 (Intermediate): Troubleshooting a Bad Gateway Outage
Objective
Investigate and resolve an intentional HTTP 502 Bad Gateway error in a broken microservices deployment.
Scenario Setup
Deploy a broken compose file where the service target port is intentionally misconfigured:
labels:
- "traefik.enable=true"
- "traefik.http.routers.broken.rule=Host(`broken.local`)"
- "traefik.http.services.broken.loadbalancer.server.port=9999" # Incorrect internal port
Student Task
- Execute
curl -i -H "Host: broken.local" http://localhostand observeHTTP 502 Bad Gateway. - Inspect Traefik container logs to identify the connection failure.
- Query
http://localhost:8080/api/http/servicesto observe target backend IP/port registration. - Fix the label to point to valid internal port
80and re-test untilHTTP 200 OKis achieved.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| Path / Location | Purpose |
|---|---|
/etc/traefik/traefik.yml |
Primary static configuration file. |
/etc/traefik/dynamic/ |
Directory for dynamic file routing rules. |
/letsencrypt/acme.json |
Encrypted Let’s Encrypt certificate store. |
Important Log Locations
| Log | Location / Command | Purpose |
|---|---|---|
| Traefik System Log | docker logs traefik or journalctl -u traefik |
Process state and error events. |
| HTTP Access Log | /var/log/traefik/access.log |
Structured incoming request records. |
Common CLI & API Commands
| Task | Command / Endpoint |
|---|---|
| Binary Version | traefik version |
| List HTTP Routers | `curl -s http://localhost:8080/api/http/routers |
| List Services | `curl -s http://localhost:8080/api/http/services |
| Dump Full Runtime Config | `curl -s http://localhost:8080/api/rawdata |
Troubleshooting Quick Reference
| Symptom | Initial Investigation Step | Primary Cause |
|---|---|---|
| HTTP 404 Not Found | `curl -s http://localhost:8080/api/http/routers | jq` |
| HTTP 502 Bad Gateway | `docker logs traefik | grep “dial tcp”` |
| TLS Default Cert Warning | Check permissions on acme.json & verify port 80 |
ACME challenge failed, port 80 blocked, or permissions not 600. |
| Redirect Loop | curl -vIL [https://app.example.com](https://app.example.com) |
Conflicting HTTPS redirects or unhandled X-Forwarded-Proto. |
Important Ports and Protocols
| Port | Protocol | Purpose |
|---|---|---|
80 |
HTTP | Ingress entrypoint web & ACME HTTP-01 challenges. |
443 |
HTTPS | Secure ingress entrypoint websecure. |
8080 |
HTTP | Administrative Web Dashboard and REST API. |
9100 |
HTTP | Dedicated Prometheus metrics scraping endpoint. |
17. Further Reading and References
- Official Traefik Documentation: doc.traefik.io/traefik
- Traefik Routing Overview: doc.traefik.io — routing
- Traefik Docker Provider Reference: doc.traefik.io — Docker
- Traefik Kubernetes IngressRoute CRD: doc.traefik.io — Kubernetes CRD