Domain 7Web & Application LayerINTERMEDIATE

Traefik Architecture, Configuration, and Troubleshooting Guide for SREs

A comprehensive operational handbook for Traefik: architecture, static vs. dynamic configuration, Docker and Kubernetes providers, ACME TLS automation, CLI diagnostics, deep troubleshooting, performance tuning, and security best practices.

Domain: Domain 7: Web & Application Layer•Updated: 2026-11-03
#linux#sysadmin#traefik#reverse-proxy#devops

1. Overview

Traefik is a modern HTTP reverse proxy and load balancer engineered for cloud-native architectures and microservices environments. Positioned at the edge of an infrastructure stack, Traefik acts as the ingress controller or edge gateway, terminating incoming HTTP and HTTPS traffic and forwarding requests to downstream application backends, container networks, or Kubernetes pods.

Traditional reverse proxies like Nginx or HAProxy relied heavily on static, file-based configuration models where server blocks and backend IP addresses were explicitly declared and reloaded whenever infrastructure changes occurred. Traefik fundamentally differs by introducing dynamic configuration discovery. By continuously listening to infrastructure orchestration APIs—such as the Docker daemon, Kubernetes API server, or HashiCorp Consul—Traefik automatically detects when containers or pods are launched, updated, or stopped, reconfiguring its routing tables in real time without dropping active connections or requiring manual process reloads.

                     +---------------------------------------+
                     |             Client Request            |
                     +---------------------------------------+
                                         |
                                         v
                     +---------------------------------------+
                     |        Traefik Reverse Proxy          |
                     |  (Edge Gateway / Port 80 & 443)       |
                     +---------------------------------------+
                        /                |                \
       Dynamic API     /                 |                 \   Dynamic API
       Discovery      v                  v                  v  Discovery
       +--------------------+  +--------------------+  +--------------------+
       | Docker Container A |  | Docker Container B |  | Kubernetes Pod C   |
       |  (api.example.com) |  | (app.example.com)  |  | (web.example.com)  |
       +--------------------+  +--------------------+  +--------------------+

Core Problem Solved

In containerized, ephemerally scaled environments, instances frequently scale up, down, or migrate across host nodes, generating unpredictable IP addresses and internal ports. Manually maintaining static upstream server lists in legacy proxies under high-frequency deployments leads to configuration drift, deployment delays, and service outages during configuration reloads. Traefik solves this by shifting routing configuration directly onto the workloads using metadata (such as Docker container labels or Kubernetes Custom Resources), eliminating manual proxy reloads entirely.

Common Use Cases

  • Edge Ingress Gateway for Container Clusters: Serving as the entry point for Docker Compose, Docker Swarm, and Kubernetes deployments.
  • Automated TLS Termination: Automatically issuing, configuring, and renewing wildcard or single-domain SSL/TLS certificates using Let’s Encrypt ACME protocols without external scripts or cron jobs.
  • Microservices Routing & Traffic Management: Routing requests based on HTTP headers, path prefixes, hostnames, or query parameters while applying request transforms, rate limiting, and authentication middleware.
  • Multi-Tenant Edge Gateways: Isolating traffic across namespaces or environments via dynamic path and hostname rules.

Advantages and Limitations

Metric / Aspect Traefik Traditional Nginx / HAProxy Caddy
Primary Use Case Cloud-native microservices, Docker, Kubernetes Static infrastructure, high-throughput raw reverse proxying, legacy web servers Developer-friendly static sites, automatic TLS, simple file configuration
Config Discovery Fully dynamic via API integration (Docker, K8s, Consul) Static configuration files requiring SIGHUP/reload Dynamic JSON API or static Caddyfile
ACME Integration Built-in native support for HTTP, TLS-ALPN, and DNS challenges Requires external tools (Certbot, acme.sh) Built-in native support
Performance Profile High throughput (Go runtime), optimized for rapid routing updates Ultra-high raw HTTP throughput & minimal memory footprint (C runtime) Moderate to high throughput (Go runtime)
Configuration Model Dual: Static startup file + Dynamic Provider metadata Single or nested configuration files Caddyfile or JSON REST API

2. Architecture and Mental Model

To effectively operate and troubleshoot Traefik in production, SREs must understand its internal packet processing model and component architecture. Traefik disconnects network listener definitions from request handling rules and backend upstream destinations, organizing traffic handling into four distinct conceptual primitives: Entrypoints, Routers, Middlewares, and Services.

graph TD
    Client[Client Request] -->|Port 80 / 443| EP[Entrypoint: web / websecure]
    EP --> Router{Router Rule Match?}
    Router -->|No Match| R404[Return HTTP 404 Not Found]
    Router -->|Match: Host / Path| MW[Middleware Chain]
    MW -->|Auth / Headers / Redirect / RateLimit| MW
    MW --> Svc[Service / Load Balancer]
    Svc -->|Round Robin / Weighted| Backend1[Backend: Container / Pod IP:8080]
    Svc -->|Round Robin / Weighted| Backend2[Backend: Container / Pod IP:8080]

Main Components and Responsibilities

  1. Entrypoints (Network Listeners):
  • Entrypoints define the physical network sockets (IP addresses and TCP/UDP ports) on which Traefik listens for incoming client connections. Common entrypoints include web (listening on port 80) and websecure (listening on port 443).
  • Entrypoints act as the entry boundary and pass incoming raw traffic into Traefik’s routing core.
  1. Routers (Traffic Matching Engine):
  • Routers evaluate incoming requests from entrypoints against user-defined matching rules (such as Host(), PathPrefix(), Headers(), or Query()).
  • When a rule matches, the router routes the request through associated Middlewares to a target Service. Routers also bind TLS configurations and ACME certificate resolvers to specific domains.
  1. Middlewares (Request/Response Pipeline):
  • Middlewares modify requests before they reach the service backend, or adjust responses before returning them to the client.
  • Executed sequentially in a defined chain, middlewares perform functions such as HTTP basic authentication, header injection, rate limiting, URL path stripping (stripPrefix), and HTTPS redirection.
  1. Services (Load Balancers & Target Forwarders):
  • Traefik Services configure how requests are load-balanced across real backend workloads.
  • A service defines backend server addresses (IP and port pairs discovered dynamically from containers or pods), health checking criteria, sticky sessions, and load-balancing algorithms (such as weighted round-robin).

Data Flow and Request Lifecycle

  1. Connection Acceptance: A client establishes a TCP connection to an Entrypoint (e.g., port 443). If TLS is configured, Traefik completes the TLS handshake using the certificate managed by the active certificate resolver or dynamic configuration.
  2. Router Evaluation: Traefik examines HTTP request metadata (Host header, path, protocol scheme) and selects the matching Router with the highest priority rule. If no router matches, Traefik immediately returns an HTTP 404 error.
  3. Middleware Processing: The request passes through the Middleware chain assigned to the router sequentially. If a middleware rejects the request (e.g., failed authentication or rate limit reached), execution halts, and the middleware returns an appropriate error response (e.g., HTTP 401 or 429).
  4. Service Load Balancing: The passed request is dispatched to the router’s assigned Service. The service selects an active, healthy backend server IP and port from its dynamically updated endpoint pool.
  5. Backend Dispatch & Response: Traefik proxies the request over the internal network to the chosen container or pod IP, waits for the downstream response, applies any response-modifying middlewares, and streams the response back to the client.

3. Core Concepts and Terminology

1. Static Configuration vs. Dynamic Configuration

Traefik enforces a strict operational boundary between static and dynamic configuration models:

  • Static Configuration: Defines startup-time system parameters, including network entrypoints, active infrastructure providers (e.g., Docker, Kubernetes, File), ACME certificate resolvers, logging formats, metrics exporters, and dashboard settings. Static configuration is read once during binary execution; any modifications require a full process restart.
  • Dynamic Configuration: Defines runtime routing rules, routers, services, middlewares, and TLS certificates. Dynamic configuration is discovered live from external infrastructure providers without restarting Traefik or interrupting active client connections.

2. Configuration Providers

Providers are external infrastructure engines that supply dynamic routing metadata to Traefik:

  • Docker Provider: Scans container labels attached to Docker containers or Swarm services to dynamically build routers and services.
  • Kubernetes Providers: Watches Kubernetes API resources, supporting standard Ingress, Traefik IngressRoute CRDs, and the Kubernetes Gateway API.
  • File Provider: Monitors a directory or explicit YAML/TOML file on local storage for dynamic routing rules, reloading configurations instantly upon file save operations.

3. Rule Matching Syntax

Routers use expressive matching expressions to filter traffic:

  • Host(api.example.com): Matches exact HTTP Host request headers.
  • PathPrefix(/v1): Matches requests whose URL paths start with /v1.
  • Headers(X-Custom-Header, value): Matches specific request headers.
  • Combining rules with logical operators: Host(api.example.com) && PathPrefix(/v1).

4. ACME & Automatic Certificate Resolvers

Traefik contains built-in Automated Certificate Management Environment (ACME) clients for requesting, validating, and automatically renewing SSL/TLS certificates from Let’s Encrypt or ZeroSSL. Resolvers support HTTP-01, TLS-ALPN-01, and DNS-01 challenge types.


4. Installation and Version Management

Prerequisites and Dependencies

  • Linux kernel 3.10+ (64-bit architecture: amd64, arm64).
  • Root or sudo privileges for binding to privileged ports (:80, :443).
  • System container runtime (Docker v20.10+ or Kubernetes v1.22+) if using container providers.

Installation Methods

Method A: Direct Binary Download (Systemd Managed)

# 1. Download official binary release
curl -sL https://github.com/traefik/traefik/releases/download/v3.3.0/traefik_v3.3.0_linux_amd64.tar.gz -o traefik.tar.gz

# 2. Extract binary and install to system path
tar -xvf traefik.tar.gz
sudo mv traefik /usr/local/bin/
sudo chmod +x /usr/local/bin/traefik

# 3. Verify binary version
traefik version

Method B: Docker / Docker Compose Deployment

# docker-compose.yml
version: '3.8'

services:
  traefik:
    image: traefik:v3.3
    container_name: traefik
    restart: always
    command:
      - "--configFile=/etc/traefik/traefik.yml"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080" # Traefik Dashboard / API
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.yml:/etc/traefik/traefik.yml:ro
      - ./dynamic:/etc/traefik/dynamic:ro
      - ./acme.json:/letsencrypt/acme.json
    networks:
      - proxy-net

networks:
  proxy-net:
    external: true

Method C: Kubernetes Deployment via Helm

# Add Traefik Helm repository
helm repo add traefik https://traefik.github.io/charts
helm repo update

# Install Traefik into dedicated namespace
helm install traefik traefik/traefik \
  --namespace traefik \
  --create-namespace \
  --set ports.web.redirectTo=websecure \
  --set ingressClass.enabled=true

Installation Files and Directory Structure Standard

Location Purpose
/usr/local/bin/traefik Main compiled executable binary.
/etc/traefik/traefik.yml Static configuration file (YAML/TOML).
/etc/traefik/dynamic/ Directory containing dynamic file configuration manifests.
/var/log/traefik/ Directory storing access logs and process stdout/stderr output.
/etc/traefik/acme.json Encrypted JSON storage holding ACME private keys and certificates.

5. Configuration Files and Directory Structure

Traefik accepts static configuration in YAML, TOML, or CLI flag formats. YAML is the standard choice for modern DevOps deployments.

Comprehensive File Reference Table

Path / File Configuration Type Purpose Reload Required on Change?
/etc/traefik/traefik.yml Static Defines entrypoints, active providers, ACME resolvers, dashboard, and logs. Yes (Requires binary restart)
/etc/traefik/dynamic/*.yml Dynamic (File Provider) Defines custom routers, middleware chains, static TLS certs, and manual services. No (Hot-reloaded automatically)
/etc/systemd/system/traefik.service System Unit Systemd unit file controlling daemon lifecycle. systemctl daemon-reload
/letsencrypt/acme.json State Storage Encrypted certificate store for ACME resolvers. No (Managed by Traefik engine)

Static Configuration Example (/etc/traefik/traefik.yml)

# Global configuration
global:
  checkNewVersion: false
  sendAnonymousUsage: false

# Operational Logging
log:
  level: INFO # DEBUG, INFO, WARN, ERROR
  format: json

# HTTP Access Logs
accessLog:
  filePath: "/var/log/traefik/access.log"
  format: json
  bufferingSize: 100

# Entrypoints (Network Listeners)
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: letsencrypt

# Built-in Administrative API & Web Dashboard
api:
  dashboard: true
  insecure: false # Set false in production; bind securely via router

# Dynamic Providers Setup
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false # Requires traefik.enable=true on containers
    watch: true
  file:
    directory: "/etc/traefik/dynamic"
    watch: true

# Certificate Resolvers (ACME / Let's Encrypt)
certificatesResolvers:
  letsencrypt:
    acme:
      email: ops@example.com
      storage: /letsencrypt/acme.json
      httpChallenge:
        entryPoint: web

Dynamic File Configuration Example (/etc/traefik/dynamic/middleware_and_routes.yml)

http:
  # Dynamic Routers
  routers:
    secure-dashboard:
      entryPoints:
        - websecure
      rule: "Host(`traefik.example.com`)"
      service: api@internal # Built-in internal service for API/Dashboard
      middlewares:
        - admin-auth
        - security-headers
      tls:
        certResolver: letsencrypt

    external-api:
      entryPoints:
        - websecure
      rule: "Host(`api.example.com`) && PathPrefix(`/v1`)"
      service: api-service
      middlewares:
        - api-ratelimit
        - strip-api-prefix
      tls:
        certResolver: letsencrypt

  # Dynamic Middlewares
  middlewares:
    admin-auth:
      basicAuth:
        users:
          # Generated using htpasswd -nb admin securepassword
          - "admin:$apr1$q86I8P45$0G5Yv1wXqYd12.Jk9/yvI."

    strip-api-prefix:
      stripPrefix:
        prefixes:
          - "/v1"

    api-ratelimit:
      rateLimit:
        average: 100
        burst: 50

    security-headers:
      headers:
        sslRedirect: true
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        frameDeny: true
        contentTypeNosniff: true

  # Dynamic Services
  services:
    api-service:
      loadBalancer:
        servers:
          - url: "http://10.0.1.50:8080"
          - url: "http://10.0.1.51:8080"
        healthCheck:
          path: /healthz
          interval: "10s"
          timeout: "3s"

6. Service Management and Common Operations

Systemd Service Definition (/etc/systemd/system/traefik.service)

[Unit]
Description=Traefik Edge Reverse Proxy
Documentation=https://doc.traefik.io/traefik/
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=traefik
Group=traefik
ExecStart=/usr/local/bin/traefik --configFile=/etc/traefik/traefik.yml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Essential System Operations Commands

  • Start Service:
sudo systemctl start traefik
  • Stop Service:
sudo systemctl stop traefik
  • Enable on Boot:
sudo systemctl enable traefik
  • Check Running Status:
sudo systemctl status traefik
  • Validate Listening Ports:
sudo ss -tlnp | grep traefik
  • Reload vs. Restart Distinction:
  • Restart (systemctl restart traefik): Destroys and recreates the Traefik process. Required when modifying static configurations (traefik.yml) such as adding new entrypoints, changing provider socket paths, or modifying ACME resolvers.
  • Dynamic Reload (Automatic): Traefik watches dynamic file directories, Docker sockets, and Kubernetes API events. Modifying dynamic files, labels, or CRDs causes Traefik to hot-reload routing tables in memory instantaneously without dropping active connections or restarting.

7. Common Administrative Tasks and Workflows

Workflow A: Exposing a New Microservice via Docker Labels

Objective

Deploy a backend web application using Docker Compose and safely expose it through Traefik with automatic TLS termination and rate limiting.

Prerequisites

  • Traefik running on a shared Docker network named proxy-net.
  • Static configuration has certificatesResolvers.letsencrypt configured.

Step-by-Step Procedure

  1. Create the docker-compose.yml for the target application:
version: '3.8'

services:
  web-app:
    image: nginx:alpine
    container_name: web-app
    restart: always
    networks:
      - proxy-net
    labels:
      # Explicitly enable Traefik discovery for this container
      - "traefik.enable=true"
      
      # Define the Router matching rule
      - "traefik.http.routers.webapp.rule=Host(`app.example.com`)"
      
      # Assign entrypoint and TLS resolver
      - "traefik.http.routers.webapp.entrypoints=websecure"
      - "traefik.http.routers.webapp.tls.certresolver=letsencrypt"
      
      # Define target container internal port
      - "traefik.http.services.webapp.loadbalancer.server.port=80"
      
      # Assign a dynamic rate-limiting middleware
      - "traefik.http.routers.webapp.middlewares=app-ratelimit@docker"
      - "traefik.http.middlewares.app-ratelimit.ratelimit.average=50"

networks:
  proxy-net:
    external: true
  1. Launch the application:
docker compose up -d
  1. Validation: Query the endpoint end-to-end to verify TLS issuance and HTTP routing:
curl -vI https://app.example.com

Workflow B: Exposing Services on Kubernetes via IngressRoute CRD

Objective

Route traffic to a Kubernetes deployment using Traefik’s native custom resource definition (IngressRoute).

Step-by-Step Procedure

  1. Verify Traefik CRDs are installed in the cluster:
kubectl get crd | grep traefik.io
  1. Create the ingressroute.yaml definition:
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: payment-service-route
  namespace: production
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`pay.example.com`) && PathPrefix(`/api`)
      kind: Rule
      services:
        - name: payment-backend-svc
          port: 8080
      middlewares:
        - name: strip-api-prefix
          namespace: production
  tls:
    certResolver: letsencrypt
  1. Apply the resource:
kubectl apply -f ingressroute.yaml
  1. Validation: Check the operational status of the IngressRoute:
kubectl describe ingressroute payment-service-route -n production

8. CLI Command Reference and Cheatsheet

1. Process and Version Inspection

Command Purpose Important Flags / Arguments Example Explanation
traefik version Displays binary build version and execution architecture. None traefik version Outputs version, commit hash, and build date.
traefik Launches the Traefik daemon process. --configFile= traefik --configFile=/etc/traefik/traefik.yml Starts proxy with specified static config.
ss / netstat Inspects active listening ports for network sockets. -tlnp `ss -tlnp grep traefik`

2. Runtime API and Diagnostic Endpoints

Traefik exposes an administrative REST API (typically on port 8080) for real-time state inspection.

Endpoint / Command Purpose Response Type Example Usage
/api/http/routers Lists all active HTTP routers and matching rules. JSON `curl -s http://localhost:8080/api/http/routers
/api/http/services Lists discovered dynamic load balancers & backend IPs. JSON `curl -s http://localhost:8080/api/http/services
/api/http/middlewares Inspects deployed request/response middlewares. JSON `curl -s http://localhost:8080/api/http/middlewares
/api/rawdata Dumps complete combined dynamic configuration snapshot. JSON `curl -s http://localhost:8080/api/rawdata
/ping Health-check endpoint for load balancers. Plain Text curl -i http://localhost:8080/ping

3. Kubernetes Diagnostics Reference

Command Purpose Key Flags Example
kubectl get ingressroute Lists all Traefik IngressRoute CRDs. -A (All Namespaces) kubectl get ingressroute -A
kubectl describe ingressroute Inspects route status, events, and syntax errors. -n <namespace> kubectl describe ingressroute myroute -n production
kubectl logs Fetches controller operational logs. -l app.kubernetes.io/name=traefik kubectl logs -n traefik -l app.kubernetes.io/name=traefik --tail=100
kubectl get endpoints Confirms downstream pod IPs exist for target Service. -n <namespace> kubectl get endpoints backend-svc -n production

4. Edge Connectivity and Route Testing

# Test router host matching locally by overriding HTTP Host Header
curl -vI -H "Host: api.example.com" http://127.0.0.1/

# Trace full HTTPS TLS handshake and certificate chain end-to-end
curl -vIL https://api.example.com

# Verify public DNS resolution prior to requesting Let's Encrypt certificates
dig +short api.example.com

9. Logs, Monitoring and Diagnostics

Log File Locations and Formats

Traefik emits two distinct log streams:

  1. Traefik System Logs: Process output containing internal state updates, provider discovery events, ACME certificate logs, and error traces.
  2. Access Logs: Structured HTTP request records details including client IPs, requested paths, response status codes, execution latency, and assigned router names.

Access Log Example (JSON Format)

{
  "ClientAddr": "192.168.1.100:54321",
  "ClientHost": "192.168.1.100",
  "RequestMethod": "GET",
  "RequestPath": "/api/v1/users",
  "OriginStatus": 200,
  "OriginContentSize": 1042,
  "RequestCount": 4210,
  "RouterName": "api-service-router@docker",
  "ServiceName": "api-service@docker",
  "Duration": 12450122,
  "StartLocal": "2026-11-03T10:15:30.123456Z"
}

Live Log Streaming and Filtering

  • Follow Systemd Journal Logs:
journalctl -u traefik -f -o cat
  • Follow Docker Container Logs:
docker logs traefik -f --tail 100
  • Filter Access Logs for HTTP 5xx Server Errors:
tail -f /var/log/traefik/access.log | grep -E '"OriginStatus":(500|502|503|504)'

10. Troubleshooting Guide and Common Failure Scenarios


Scenario 1: HTTP 404 Not Found — Router Not Found

Symptoms

Clients receive an instant HTTP 404 Not Found response generated directly by Traefik when requesting a hosted application.

$ curl -i http://api.example.com/
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
Length: 19

404 page not found

Possible Causes

  1. Traefik has not loaded a router matching the incoming Host() or PathPrefix() rules.
  2. On Docker: The container lacks the required traefik.enable=true label, or Traefik is set to exposedByDefault: false.
  3. Entrypoint mismatch: The router is bound to entrypoint websecure (:443), but the client sent a request to entrypoint web (:80) without HTTPS redirection.
  4. On Kubernetes: Traefik is not configured to watch the namespace in which the IngressRoute resides.

Investigation and Diagnosis

Step 1: Query Traefik API for Active Routers

State check: Determine if Traefik registered the router in its in-memory routing table.

curl -s http://localhost:8080/api/http/routers | jq '.[] | select(.name | contains("myapp"))'
  • If output is empty: Traefik failed to discover the router from the provider. Move to Step 2.
  • If router exists: Verify that the rule (Host(...)) exactly matches the hostname requested by the client.

Step 2: Verify Docker Container Labels and Networks

Inspect container metadata directly from the Docker daemon:

docker inspect my-container-name --format '{{json .Config.Labels}}' | jq .

Verify:

  1. "traefik.enable" is set to "true".
  2. Traefik and the container share a common Docker network.

Step 3: Verify Kubernetes Namespace Scope

For Kubernetes deployments, confirm Traefik RBAC permissions and watched namespaces:

kubectl get ingressroute -n my-app-namespace

Resolution

  • For Docker: Ensure correct label syntax and shared network attachment:
labels:
  - "traefik.enable=true"
  - "traefik.http.routers.myapp.rule=Host(`api.example.com`)"
  - "traefik.docker.network=proxy-net"
  • For Entrypoint Mismatches: Ensure the router specifies both entrypoints if dual-binding is intended:
traefik.http.routers.myapp.entrypoints=web,websecure

Verification

Execute a request explicitly enforcing the Host header:

curl -vI -H "Host: api.example.com" http://127.0.0.1/

Expected: HTTP/1.1 200 OK (or appropriate backend response code).


Scenario 2: HTTP 502 Bad Gateway / HTTP 503 Service Unavailable

Symptoms

Traefik successfully matches the request to a router, but returns HTTP 502 Bad Gateway or HTTP 503 Service Unavailable to the client.

$ curl -i https://app.example.com/
HTTP/2 502 
content-length: 11

Bad Gateway

Possible Causes

  1. Port Mismatch: The service load balancer port configured in Traefik points to the host-mapped port rather than the container’s internal listening port.
  2. Unreachable Backend Network: Traefik cannot reach the backend IP address due to network segmentation or missing Docker network attachments.
  3. Failed Health Checks: Traefik’s health checker has marked all downstream backend instances as unhealthy.
  4. Backend Process Crashed: The container or application inside the pod crashed or is not accepting connections.

Investigation and Diagnosis

Step 1: Check Traefik Internal Logs

Search system logs for connection attempt errors:

docker logs traefik 2>&1 | grep -E '502|503|dial tcp|connection refused'

Look for: dial tcp 172.18.0.5:8080: connect: connection refused.

Step 2: Inspect Discovered Service Port

Query the Traefik API to see what IP and port Traefik is trying to contact:

curl -s http://localhost:8080/api/http/services | jq '.[].loadBalancer.servers'

Compare the resulting IP/port against the container’s actual internal exposed port (EXPOSE).

Step 3: Test Direct Network Connectivity

Execute a direct request from inside the Traefik container to the target backend IP:

docker exec -it traefik wget --spider -T 3 http://172.18.0.5:8080/healthz

Resolution

Explicitly declare the container’s internal listening port using the server port label:

# Correct: Target internal container port (e.g., 8080), NOT host port (e.g., 80)
- "traefik.http.services.myapp.loadbalancer.server.port=8080"

Verification

Re-query the dynamic service endpoint and confirm active communication:

curl -s http://localhost:8080/api/http/services | jq '.[].serverStatus'

Scenario 3: TLS / Let’s Encrypt Certificate Generation Failure

Symptoms

Traefik serves a self-signed fallback certificate (TRAEFIK DEFAULT CERTIFCATE) instead of a valid Let’s Encrypt TLS certificate, causing browser security warnings.

Possible Causes

  1. Public Reachability: Let’s Encrypt HTTP-01 challenge cannot reach Traefik on port 80 from the public internet.
  2. File Permissions: Storage file /letsencrypt/acme.json has overly permissive file permissions (must be strictly 600).
  3. DNS Misconfiguration: Public DNS record does not point to the external IP address of the Traefik host.
  4. Let’s Encrypt Rate Limits: Exceeded failed validation rate limits for the target domain.

Investigation and Diagnosis

Step 1: Check ACME Operational Logs

Filter Traefik logs specifically for ACME validation events:

docker logs traefik 2>&1 | grep -i "acme"

Common error logs:

  • ACME error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from...
  • unable to get ACME account: permissions on /letsencrypt/acme.json are 0644 instead of 0600

Step 2: Verify acme.json Permissions

Inspect file permission mode on host storage:

ls -l /path/to/acme.json

Resolution

  1. Correct ownership and restrict file permissions on acme.json:
sudo chmod 600 /path/to/acme.json
sudo chown 1000:1000 /path/to/acme.json
  1. Ensure Port 80 is open in firewalls (UFW/iptables) and forwarded to Traefik.
  2. Verify public DNS record resolution using dig:
dig +short app.example.com

Verification

Inspect the SSL certificate presented by the live endpoint:

echo | openssl s_client -servername app.example.com -connect app.example.com:443 2>/dev/null | openssl x509 -noout -dates -issuer

Scenario 4: Dynamic HTTP Redirect Loops (Too Many Redirects)

Symptoms

Web browsers display ERR_TOO_MANY_REDIRECTS, and curl outputs an infinite sequence of HTTP 301/302 redirects.

Possible Causes

  1. Conflicting HTTPS Redirects: Traefik entrypoint forces HTTPS redirection, while an upstream application or secondary middleware also issues an HTTPS redirect.
  2. X-Forwarded-Proto Misinterpretation: Downstream backend receives plain HTTP requests from Traefik without recognizing X-Forwarded-Proto: https, prompting the backend to repeatedly issue redirects.

Investigation and Diagnosis

Trace the HTTP redirect location chain using curl:

curl -vIL https://app.example.com

Observe whether the Location: header toggles repeatedly between http:// and https://.

Resolution

Ensure that the HTTPS redirection is managed strictly at the Traefik entrypoint level and disable duplicate HTTPS enforcement inside the application backend code. Configure backend applications to trust X-Forwarded-Proto headers supplied by Traefik.


11. Security and Best Practices

  1. Secure the Traefik API and Dashboard:
  • Never set api.insecure: true in production environments.
  • Expose the dashboard using a dedicated, secure dynamic router protected with basicAuth or forwardAuth middleware and TLS.
  1. Restrict Docker Socket Access:
  • Mounting /var/run/docker.sock exposes the host to elevated security risks if Traefik is compromised.
  • Use a read-only socket mount (/var/run/docker.sock:ro) or deploy a secure proxy sidecar like docker-socket-proxy to restrict API endpoints to read-only container listings.
  1. Enforce Storage File Permissions:
  • Enforce strict file mode 600 on /letsencrypt/acme.json to safeguard private TLS keys.
  1. Hardening HTTP Security Headers: Apply dynamic security headers middleware across all public routers to enforce modern browser protections:
http:
  middlewares:
    secure-headers:
      headers:
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        stsPreload: true
        frameDeny: true
        contentTypeNosniff: true
        browserXssFilter: true
        referrerPolicy: "strict-origin-when-cross-origin"
  1. Resource Rate Limiting: Attach rateLimit middlewares to public endpoints to defend backends against Denial-of-Service (DoS) attacks and brute-force attempts.

12. Performance and Resource Optimisation

  1. Tune Go Runtime Limits: Set Linux open file descriptor limits (ulimit -n 65536) inside systemd units or container environments to handle high numbers of concurrent TCP connections.
  2. Access Log Buffering: In high-throughput environments, writing access logs synchronously to disk causes disk I/O bottlenecks. Enable log buffering in static configuration:
accessLog:
  filePath: "/var/log/traefik/access.log"
  format: json
  bufferingSize: 500 # Buffer up to 500 log lines in memory before disk flush
  1. HTTP/2 and HTTP/3 Enablement: Traefik supports HTTP/2 by default over TLS. Enable experimental HTTP/3 (QUIC) support on entrypoints to minimize latency for mobile and high-latency clients.
  2. Keep-Alive Connection Pooling: Optimize upstream transport parameters in static config to reuse persistent TCP connections to backend workloads:
serversTransport:
  maxIdleConnsPerHost: 200
  forwardingTimeouts:
    dialTimeout: "3s"
    responseHeaderTimeout: "10s"

13. Backup, Recovery and Maintenance

Backup Procedures

  1. Static Configuration & Dynamic Files: Back up static configuration files (traefik.yml), dynamic file directories, and Docker Compose manifests using standard version control (Git).
  2. ACME TLS Storage (acme.json): The acme.json file contains private account keys and active TLS certificates. Back up this file to prevent hitting Let’s Encrypt rate limits during system recovery.
# Safely snapshot acme.json maintaining strict permissions
sudo tar -czvf traefik-acme-backup-$(date +%F).tar.gz /etc/traefik/acme.json

Upgrade and Patching Workflow

  1. Review release notes for breaking syntax changes between major versions.
  2. Pull the updated binary or container image (traefik:v3.3).
  3. Validate static configuration syntax offline:
traefik healthcheck --configFile=/etc/traefik/traefik.yml
  1. Perform zero-downtime rolling updates in Kubernetes or deploy updated system container images.

14. Automation and Integration

Prometheus Metrics Integration

Traefik natively exports detailed Prometheus metrics on an entrypoint or internal path.

Static Enablement (traefik.yml)

metrics:
  prometheus:
    entryPoint: metrics
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true

entryPoints:
  metrics:
    address: ":9100"

Scraping with Prometheus (prometheus.yml)

scrape_configs:
  - job_name: 'traefik'
    static_configs:
      - targets: ['traefik.internal.example.com:9100']

15. Practical Labs and Exercises


Exercise 1 (Beginner): Deploy Traefik and Secure a Local Web Application

Objective

Deploy Traefik using Docker Compose alongside a basic web container (nginxdemos/hello), expose it on hello.local, and inspect the router via the Traefik API.

Steps

  1. Create a workspace directory and define docker-compose.yml:
version: '3.8'
services:
  traefik:
    image: traefik:v3.3
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  demo-app:
    image: nginxdemos/hello
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.demo.rule=Host(`hello.local`)"
      - "traefik.http.routers.demo.entrypoints=web"
      - "traefik.http.services.demo.loadbalancer.server.port=80"
  1. Launch stack: docker compose up -d.
  2. Test locally: curl -H "Host: hello.local" http://localhost.
  3. Query active routers via API: curl -s http://localhost:8080/api/http/routers | jq.

Exercise 2 (Intermediate): Troubleshooting a Bad Gateway Outage

Objective

Investigate and resolve an intentional HTTP 502 Bad Gateway error in a broken microservices deployment.

Scenario Setup

Deploy a broken compose file where the service target port is intentionally misconfigured:

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.broken.rule=Host(`broken.local`)"
  - "traefik.http.services.broken.loadbalancer.server.port=9999" # Incorrect internal port

Student Task

  1. Execute curl -i -H "Host: broken.local" http://localhost and observe HTTP 502 Bad Gateway.
  2. Inspect Traefik container logs to identify the connection failure.
  3. Query http://localhost:8080/api/http/services to observe target backend IP/port registration.
  4. Fix the label to point to valid internal port 80 and re-test until HTTP 200 OK is achieved.

16. Quick Reference and Cheat Sheet

Important Configuration Files

Path / Location Purpose
/etc/traefik/traefik.yml Primary static configuration file.
/etc/traefik/dynamic/ Directory for dynamic file routing rules.
/letsencrypt/acme.json Encrypted Let’s Encrypt certificate store.

Important Log Locations

Log Location / Command Purpose
Traefik System Log docker logs traefik or journalctl -u traefik Process state and error events.
HTTP Access Log /var/log/traefik/access.log Structured incoming request records.

Common CLI & API Commands

Task Command / Endpoint
Binary Version traefik version
List HTTP Routers `curl -s http://localhost:8080/api/http/routers
List Services `curl -s http://localhost:8080/api/http/services
Dump Full Runtime Config `curl -s http://localhost:8080/api/rawdata

Troubleshooting Quick Reference

Symptom Initial Investigation Step Primary Cause
HTTP 404 Not Found `curl -s http://localhost:8080/api/http/routers jq`
HTTP 502 Bad Gateway `docker logs traefik grep “dial tcp”`
TLS Default Cert Warning Check permissions on acme.json & verify port 80 ACME challenge failed, port 80 blocked, or permissions not 600.
Redirect Loop curl -vIL [https://app.example.com](https://app.example.com) Conflicting HTTPS redirects or unhandled X-Forwarded-Proto.

Important Ports and Protocols

Port Protocol Purpose
80 HTTP Ingress entrypoint web & ACME HTTP-01 challenges.
443 HTTPS Secure ingress entrypoint websecure.
8080 HTTP Administrative Web Dashboard and REST API.
9100 HTTP Dedicated Prometheus metrics scraping endpoint.

17. Further Reading and References

  1. Official Traefik Documentation: doc.traefik.io/traefik
  2. Traefik Routing Overview: doc.traefik.io — routing
  3. Traefik Docker Provider Reference: doc.traefik.io — Docker
  4. Traefik Kubernetes IngressRoute CRD: doc.traefik.io — Kubernetes CRD