Domain 7Web & Application LayerINTERMEDIATE

(San Juan) Troubleshooting Traefik Round-Robin Load Balancing in Docker Compose

A comprehensive technical deep dive into diagnosing and fixing intermittent 502 Bad Gateway errors and backend container network timeouts in a Traefik-managed Docker Compose setup.

Domain: Domain 7: Web & Application Layer•Updated: 2026-11-10
#Traefik#Docker#Docker-Compose#Networking#Load-Balancing#SRE

1. Introduction

In microservices architectures, reverse proxies and load balancers serve as the critical gateway for client traffic. Traefik has become one of the industry-standard edge routers for containerized workloads due to its native integration with container orchestrators and engine sockets, enabling automatic service discovery through metadata labels.

However, automatic service discovery relies strictly on accurate declarative configuration. When containers are misconfigured—such as exposing invalid internal service ports or being deployed onto isolated network bridges—the load balancer can exhibit non-deterministic behavior. Symptoms often manifest as intermittent 502 Bad Gateway errors, client request timeouts, or partial pool rotation.

This article walks through a step-by-step investigation of a broken Traefik load-balancing setup deployed via Docker Compose on Debian 13. You will learn how to systematically trace HTTP failures down through network namespaces, analyze reverse proxy logs, fix declarative Docker Compose labels, and apply Site Reliability Engineering (SRE) principles to prevent similar failure modes in production.


2. Problem Description and Environment

Scenario Context

The scenario (“San Juan”) features an edge load balancer managed by Traefik v3.6.1 sitting in front of four backend container replicas running traefik/whoami. The backends are intended to process HTTP traffic in a round-robin rotation when requests hit [http://app.sadserver](http://app.sadserver).

Environment Specifications

  • Operating System: Debian 13 (Trixie)
  • Privileges: Root / Sudo access available (admin user)
  • Container Engine: Docker Engine with Docker Compose
  • Edge Proxy: Traefik v3.6.1
  • Backend Image: traefik/whoami (a tiny HTTP server that returns container OS metadata and hostname)

Reported Symptoms vs. Expected Behavior

  • Expected Behavior: Executing curl -s app.sadserver | head -n1 should consistently return the host ID (container hostname) of one of the backend replicas, cycling evenly across all four backends on consecutive requests.
  • Actual Symptoms:
  1. Initial request execution against app.sadserver failed or returned blank.
  2. Repeated requests intermittently returned 502 Bad Gateway.
  3. Requests hung and timed out after several seconds.
  4. Only two of the four backend containers (app01 and app03) received traffic.

3. Initial Investigation and Hypotheses

Before modifying any configuration files or restarting services, a systematic SRE approach requires gathering baseline metrics and formulating verifiable hypotheses.

Initial Observations & Gaps

  1. Container State: Are all containers running, or are backend instances stuck in a CrashLoopBackOff?
  2. Proxy Routing: Is Traefik receiving requests on port 80?
  3. Backend Reachability: Why are only select containers answering requests?

Hypothesis Matrix

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
H1: Backend containers are crashing or offline curl fails or hangs intermittently. None yet. Inspect container state with docker ps -a.
H2: Port misconfiguration in Traefik labels 502 Bad Gateway error suggests proxy reached an IP but refused connection. Traffic partially works for app01 and app03. Check Traefik container logs (docker logs) and Compose service labels.
H3: Container network isolation Request timeouts indicate packets are dropped or unroutable. Traefik is actively listening on exposed ports 80/443. Inspect Docker bridge networks (docker network inspect).

4. Troubleshooting Process

Step 1: Evaluating Container Health and Base Connectivity

Objective Check the running status of all containers in the stack to verify if any containers are stopped or restarting continuously.

Command

curl -s app.sad.server | head -n1
docker ps -a

Command Explanation

  • curl -s app.sad.server: Performs a silent HTTP GET request.
  • docker ps -a: Lists all containers on the host engine. The -a (--all) flag is crucial because standard docker ps hides exited or dead containers.

Output and Analysis

admin@i-06b169eb07f44d33d:~$ curl -s app.sad.server | head -n1
admin@i-06b169eb07f44d33d:~$ docker ps -a
CONTAINER ID   IMAGE            COMMAND                  CREATED        STATUS              PORTS                                                                       NAMES
a2f3f16b0928   traefik:v3.6.1   "/entrypoint.sh --ap…"   8 months ago   Up About a minute   0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp   app-traefik-1
2990d7e439ce   traefik/whoami   "/whoami"                8 months ago   Up About a minute   80/tcp                                                                      app-app03-1
049eb38a4bf4   traefik/whoami   "/whoami"                8 months ago   Up About a minute   80/tcp                                                                      app-app01-1
5b3c8c3c22bd   traefik/whoami   "/whoami"                8 months ago   Up About a minute   80/tcp                                                                      app-app04-1
2fc39d0d7412   traefik/whoami   "/whoami"                8 months ago   Up About a minute   80/tcp                                                                      app-app02-1

Analysis:

  1. All five containers (app-traefik-1, app-app01-1, app-app02-1, app-app03-1, app-app04-1) report an Up status. This rules out Hypothesis H1 (crashing containers).
  2. The initial curl attempt against app.sad.server returned empty, indicating a typo in the domain (app.sad.server vs app.sadserver) or an unroutable attempt.

Step 2: Reproducing the Failure Pattern via Loop Probing

Objective Automate sequential HTTP requests to test Traefik’s Weighted Round Robin (WRR) load-balancing algorithm across the backend pool.

Command

for i in {1..6}; do curl -s --max-time 3 http://app.sadserver | head -n1; sleep 0.5; done

Command Explanation

  • for i in {1..6}; do ... done: Runs a sequential Bash loop 6 times.
  • curl -s --max-time 3: Connects to [http://app.sadserver](http://app.sadserver). --max-time 3 sets a hard limit of 3 seconds per connection attempt to catch hanging requests without blocking the shell indefinitely.
  • head -n1: Extracts the first line of the whoami output (which displays Hostname: <container_id>).

Output and Analysis

Bad GatewayHostname: 049eb38a4bf4
Hostname: 2990d7e439ce
Hostname: 049eb38a4bf4
Hostname: 2990d7e439ce

Analysis:

  • Request 1 failed immediately with Bad Gateway.
  • Requests 2 through 5 succeeded, but only rotated between two containers: 049eb38a4bf4 (app01) and 2990d7e439ce (app03).
  • Requests to app02 (2fc39d0d7412) and app04 (5b3c8c3c22bd) failed to render valid hostnames.

Step 3: Inspecting Traefik Log Diagnostics

Objective Inspect Traefik’s internal runtime logs to analyze backend registration and connection error details.

Command

docker logs app-traefik-1 --tail 30

Command Explanation

  • docker logs: Fetches stdout/stderr logs from a target container.
  • --tail 30: Limits output to the last 30 log lines for quick focused troubleshooting.

Output and Analysis

2026-11-10T15:18:34Z DBG ... > Creating server URL=http://172.19.0.3:81 entryPointName=web routerName=app@docker serverIndex=0 serviceName=app@docker
2026-11-10T15:18:34Z DBG ... > Creating server URL=http://172.19.0.5:80 entryPointName=web routerName=app@docker serverIndex=1 serviceName=app@docker
2026-11-10T15:18:34Z DBG ... > Creating server URL=http://172.19.0.4:80 entryPointName=web routerName=app@docker serverIndex=2 serviceName=app@docker
2026-11-10T15:18:34Z DBG ... > Creating server URL=http://172.18.0.2:80 entryPointName=web routerName=app@docker serverIndex=3 serviceName=app@docker
...
2026-11-10T15:19:31Z DBG ... > Service selected by WRR: http://172.19.0.3:81
2026-11-10T15:19:31Z DBG ... > 502 Bad Gateway error="dial tcp 172.19.0.3:81: connect: connection refused"
2026-11-10T15:19:32Z DBG ... > Service selected by WRR: http://172.18.0.2:80
2026-11-10T15:19:35Z DBG ... > 499 Client Closed Request error="context canceled"

Analysis:

  1. Finding A (app02): Traefik dynamically registered app02 at [http://172.19.0.3:81](http://172.19.0.3:81). When forwarding requests, it encountered connect: connection refused on port 81. This confirms Hypothesis H2 (Port Misconfiguration).
  2. Finding B (app04): Traefik dynamically registered app04 at [http://172.18.0.2:80](http://172.18.0.2:80). Notice that app04 resides on subnet 172.18.0.0/16, whereas Traefik and the other apps are on subnet 172.19.0.0/16. The request timed out (context canceled), causing HTTP 499/504 errors. This confirms Hypothesis H3 (Network Isolation).

Step 4: Inspecting Declarative Configuration (docker-compose.yml)

Objective Review the deployment specifications in docker-compose.yml to locate the source of the invalid labels and network definitions.

Command

cd ~/app && cat docker-compose.yml

Output and Analysis

services:
  traefik:
    image: traefik:v3.6.1
    restart: unless-stopped
    command:
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--log.level=DEBUG"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - web

  app01:
    image: traefik/whoami
    restart: unless-stopped
    labels:
      traefik.enable: "true"
      traefik.http.routers.app.rule: Host(`app.sadserver`)
      traefik.http.services.app.loadbalancer.server.port: "80"
    networks:
      - web

  app02:
    image: traefik/whoami
    restart: unless-stopped
    labels:
      traefik.enable: "true"
      traefik.http.routers.app.rule: Host(`app.sadserver`)
      traefik.http.services.app.loadbalancer.server.port: "81"
    networks:
      - web

  app03:
    image: traefik/whoami
    restart: unless-stopped
    labels:
      traefik.enable: "true"
      traefik.http.routers.app.rule: Host(`app.sadserver`)
      traefik.http.services.app.loadbalancer.server.port: "80"
    networks:
      - web

  app04:
    image: traefik/whoami
    restart: unless-stopped
    labels:
      traefik.enable: "true"
      traefik.http.routers.app.rule: Host(`app.sadserver`)
      traefik.http.services.app.loadbalancer.server.port: "80"
    networks:
      - internal

networks:
  web:
  internal:

Analysis:

  • app02 Defect: The label traefik.http.services.app.loadbalancer.server.port is explicitly set to "81". The whoami application natively listens on port 80. Traefik attempted to open a socket to port 81, receiving a TCP RST packet from app02.
  • app04 Defect: app04 is attached exclusively to network internal. traefik is attached exclusively to network web. Because Traefik shares no common network interface with app04, packet routing between Traefik and app04 fails silently at the bridge boundary.

5. Root Cause Analysis

The failure consists of two distinct root causes operating in tandem:

                  +-------------------------------------------------+
                  |              Client (curl request)              |
                  +------------------------+------------------------+
                                           |
                                           v
                  +-------------------------------------------------+
                  |           traefik (app-traefik-1)               |
                  |             Network: web (172.19.0.0/16)       |
                  +----+-------------------+-------------------+----+
                       |                   |                   |
        +--------------+                   |                   +---------------+
        | Port 80 (OK)                     | Port 81 (WRONG)                   | Isolated Net (WRONG)
        v                                  v                                   v
+---------------+                  +---------------+                   +---------------+
|     app01     |                  |     app02     |                   |     app04     |
| (172.19.0.4)  |                  | (172.19.0.3)  |                   | (172.18.0.2)  |
| Network: web  |                  | Network: web  |                   | Net: internal |
+---------------+                  +---------------+                   +---------------+
  Status: 200 OK                     TCP Connection                      Packet Timeout
                                    Refused (502)                      (Context Canceled)
  1. Port Misconfiguration (app02): Traefik dynamic provider reads Docker labels to construct destination target URLs. Setting server.port: 81 overrode Traefik’s auto-detection. Since whoami has no service bound to port 81 inside its network namespace, the kernel immediately returned a TCP RST flag. Traefik logged connection refused and bubbled up an HTTP 502 Bad Gateway error to the client.
  2. Bridge Network Isolation (app04): Docker builds distinct Linux bridge networks for isolated compose definitions. Traefik was placed on bridge web (172.19.0.0/16), while app04 was placed on bridge internal (172.18.0.0/16). Without a common network attachment or cross-bridge routing rule, IP packets sent from Traefik to 172.18.0.2 hit a dead end, timing out after 3 seconds (499 Client Closed Request / context canceled).

6. Solution and Remediation

Modifying Configuration

Edit ~/app/docker-compose.yml to correct the port label on app02 and change app04’s network attachment to web.

vim ~/app/docker-compose.yml

Declarative Configuration Snippet (Diff)

  app02:
    image: traefik/whoami
    restart: unless-stopped
    labels:
      traefik.enable: "true"
      traefik.http.routers.app.rule: Host(`app.sadserver`)
-     traefik.http.services.app.loadbalancer.server.port: "81"
+     traefik.http.services.app.loadbalancer.server.port: "80"
    networks:
      - web

  app04:
    image: traefik/whoami
    restart: unless-stopped
    labels:
      traefik.enable: "true"
      traefik.http.routers.app.rule: Host(`app.sadserver`)
      traefik.http.services.app.loadbalancer.server.port: "80"
    networks:
-     - internal
+     - web

Applying the Fix

Recreate affected containers using Docker Compose:

docker compose up -d

Output:

admin@i-06b169eb07f44d33d:~/app$ docker compose up -d
[+] Running 5/5
 ✔ Container app-app03-1    Running                                                0.0s 
 ✔ Container app-app01-1    Running                                                0.0s 
 ✔ Container app-app02-1    Started                                                1.0s 
 ✔ Container app-app04-1    Started                                                1.0s 
 ✔ Container app-traefik-1  Running                                                0.0s 

7. Verification and Validation

Step 1: Sequential Load-Balancer Probe

Verify that requests to [http://app.sadserver](http://app.sadserver) now rotate across all four distinct container hostnames without errors or timeouts.

for i in {1..8}; do curl -s --max-time 3 http://app.sadserver | head -n1; sleep 0.5; done

Output:

Hostname: f627cafcbe6c
Hostname: 049eb38a4bf4
Hostname: e90a18f1276d
Hostname: 2990d7e439ce
Hostname: e90a18f1276d
Hostname: 2990d7e439ce
Hostname: f627cafcbe6c
Hostname: 049eb38a4bf4

Analysis: All four backend container IDs (f627cafcbe6c, 049eb38a4bf4, e90a18f1276d, 2990d7e439ce) are actively responding in round-robin fashion. No 502 Bad Gateway or timing-out responses occurred.

Step 2: Automated Verification Script

Run the verification test script provided by the platform:

/home/admin/agent/check.sh

Output:

OK

8. Alternative Troubleshooting Approaches

While reviewing Compose files and tailing logs quickly resolved this scenario, SREs frequently encounter more complex network edge cases. Below are alternative diagnostic approaches:

Diagnostic Approach Commands Used Key Information Provided Ideal Usage Scenario
Docker Inspect Subnet Correlation docker inspect app-app04-1 Details container IP allocation, gateways, and attached network bridges. Debugging inter-container routing issues and missing interfaces.
In-Container Socket Probe docker exec -it app-traefik-1 nc -zv app02 80 Tests direct L4 TCP connectivity from inside the proxy container namespace. Validating firewall rules, port bindings, and DNS resolution within containers.
Traefik API / Dashboard Query curl http://localhost:8080/api/rawdata Exposes internal Traefik dynamic router tables, server weights, and status. Diagnosing missing backend registration or health-check drops.
Network Packet Capture tcpdump -i any host 172.19.0.3 Captures raw TCP handshakes (SYN, RST, ACK) across container virtual interfaces (veth). Tracing silent drops, MTU mismatches, or TCP reset causes at the kernel level.

9. Linux & Container Command Deep Dive

Command: docker ps

Purpose: Interrogates the local Docker daemon engine via UNIX socket (/var/run/docker.sock) to report container status, image metadata, and exposed network socket bindings.

Syntax & Essential Flags:

docker ps [OPTIONS]
Flag Description Troubleshooting Value
-a, --all Show all containers (default shows just running). Essential for spotting crashed containers (Exited (1)).
-f, --filter Filter output based on conditions (e.g., status=restarting). Isolates unstable containers in high-density hosts.
--format Pretty-print container outputs using Go templates. E.g., docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}".

Command: docker logs

Purpose: Streams stdout and stderr file logs generated by container PID 1 processes.

Syntax & Essential Flags:

docker logs [OPTIONS] CONTAINER
Flag Description Troubleshooting Value
--tail N Output only the last N lines. Filters out historical noise to highlight recent failures.
-f, --follow Stream live log output in real time. Real-time debugging while generating synthetic test traffic.
-t, --timestamps Append ISO-8601 timestamps to each log entry. Helps correlate proxy drops with external alerting events.
--since Show logs since timestamp or relative duration (e.g., 5m). Focuses on specific incident time windows.

Command: curl

Purpose: Command-line tool for transferring data with URLs using supported network protocols (HTTP, HTTPS, FTP, etc.).

Syntax & Essential Flags:

curl [OPTIONS] URL
Flag Description Troubleshooting Value
-s, --silent Suppress progress meters and non-error output. Clean parsing when piping output into scripts or head/grep.
-v, --verbose Print HTTP headers, TLS handshakes, and DNS resolution steps. Debugging HTTP response status codes (502, 504, 403).
--max-time N Set maximum time in seconds that the transfer is allowed to take. Prevents automation scripts from hanging on drop-tail timeouts.
-H "Host: ..." Send custom HTTP headers to test virtual host routing. Essential for testing Traefik host rule matching (Host(...)).

10. Underlying Linux Concepts and Architecture

1. Docker Networking & Linux Network Namespaces

Docker isolates containers using Linux Network Namespaces (netns). Each namespace maintains its own independent routing tables, iptables chains, network interfaces, and socket bindings.

When Docker Compose spins up a bridge network (e.g., web):

  1. Docker creates a virtual bridge interface on the host kernel (e.g., br-a1b2c3d4e5f6).
  2. A Virtual Ethernet Pair (veth) is allocated for each container. One end attaches to the host bridge; the other end enters the container’s network namespace as eth0.
  3. Containers assigned to different bridges reside on entirely separate IP subnets (e.g., 172.19.0.0/16 vs 172.18.0.0/16). By default, Docker does not enable cross-bridge IP forwarding without explicit network attachments.

2. Traefik Dynamic Provider Discovery

Unlike traditional static load balancers (such as Nginx or HAProxy), Traefik uses event-driven dynamic discovery:

  1. Traefik mounts /var/run/docker.sock to listen to the Docker daemon event stream.
  2. When containers start or update, Traefik inspects their labels (traefik.enable, traefik.http.routers...).
  3. If labels declare a custom server.port, Traefik bypasses the default exposed container port and routes traffic directly to the user-specified port. If that port is wrong, the kernel’s network stack returns an ECONNREFUSED error, resulting in a 502 Bad Gateway.

11. Real-World Sysadmin and SRE Applications

Production Health Check Configuration

To prevent Traefik from routing production traffic to failing or misconfigured backend pods, configure explicit Traefik active health checks in docker-compose.yml:

  app01:
    image: traefik/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`app.sadserver`)"
      - "traefik.http.services.app.loadbalancer.server.port=80"
      - "traefik.http.services.app.loadbalancer.healthcheck.path=/"
      - "traefik.http.services.app.loadbalancer.healthcheck.interval=5s"
      - "traefik.http.services.app.loadbalancer.healthcheck.timeout=2s"

If a backend fails its health check, Traefik automatically removes the instance from the Weighted Round-Robin rotation, preventing 502 Bad Gateway errors from reaching end users.

CI/CD Linting and Static Analysis

Prevent broken configurations from entering production by validating Docker Compose definitions prior to deployment:

# Validate syntax and network definitions
docker compose config --quiet

12. Further Learning and Experiments

Hands-on Experiments

Experiment 1: Inspect Container Network Namespaces

Gain deep visibility into container bridge attachments directly from the host system:

# List all active Docker bridges
ip link show type bridge

# Inspect IP allocations across all services in the stack
docker inspect --format='{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}} ({{.NetworkID}}){{end}}' $(docker ps -q)

Experiment 2: Simulate a Backend Partial Outage

Stop a single backend container (app02) and observe how Traefik handles backend drops when passive or active health checks are configured.

docker stop app-app02-1
curl -i -H "Host: app.sadserver" http://localhost

13. Key Takeaways

  1. Verify Exposed Container Ports: Never assume a container listens on non-standard ports unless explicitly declared in the container’s Dockerfile or entrypoint binary (whoami listens on 80).
  2. Match Network Namespaces: Edge proxies like Traefik must share at least one common Docker network bridge with every backend service they route traffic to.
  3. Use Logs to Isolate Errors: Traefik’s debug logs quickly distinguish between TCP level errors (connection refused -> wrong port) and IP/Layer 3 errors (context canceled -> unroutable network).
  4. Always Limit Probing Timeouts: Always use curl --max-time when probing unknown endpoints to prevent operational scripts from hanging indefinitely.

14. Quick Troubleshooting Reference / Runbook

                  +-----------------------------------+
                  |  Traefik Load Balancing Issue     |
                  +-----------------+-----------------+
                                    |
                                    v
                  +-----------------------------------+
                  | Run: docker ps -a                 |
                  | Are all containers Up?            |
                  +--------+-----------------+--------+
                           |                 |
                  No       |                 | Yes
        +------------------+                 +------------------+
        |                                                       |
        v                                                       v
+-----------------------+                       +-------------------------------+
| Check container logs: |                       | Probe proxy endpoints:        |
| docker logs <app>     |                       | curl -v http://<domain>       |
+-----------------------+                       +---------------+---------------+
                                                                |
                                             +------------------+------------------+
                                             |                                     |
                                             v                                     v
                                   HTTP 502 Bad Gateway                   HTTP 504 Timeout / Hang
                                             |                                     |
                                             v                                     v
                                 +-----------------------+             +-----------------------+
                                 | Check Traefik Logs:   |             | Inspect Networks:     |
                                 | Connection Refused?   |             | Traefik & App on same |
                                 | -> Fix port label in  |             | Docker bridge?        |
                                 |    docker-compose.yml |             | -> Fix networks list  |
                                 +-----------------------+             +-----------------------+

Emergency Command Cheat Sheet

# 1. Check all container statuses
docker ps -a

# 2. Tail Traefik debug logs
docker logs app-traefik-1 --tail 50 -f

# 3. Test endpoint loop with timeout
for i in {1..5}; do curl -s --max-time 2 -H "Host: app.sadserver" http://localhost; done

# 4. Re-apply docker compose changes gracefully
docker compose up -d --force-recreate