1. Introduction
In microservices architectures, reverse proxies and load balancers serve as the critical gateway for client traffic. Traefik has become one of the industry-standard edge routers for containerized workloads due to its native integration with container orchestrators and engine sockets, enabling automatic service discovery through metadata labels.
However, automatic service discovery relies strictly on accurate declarative configuration. When containers are misconfigured—such as exposing invalid internal service ports or being deployed onto isolated network bridges—the load balancer can exhibit non-deterministic behavior. Symptoms often manifest as intermittent 502 Bad Gateway errors, client request timeouts, or partial pool rotation.
This article walks through a step-by-step investigation of a broken Traefik load-balancing setup deployed via Docker Compose on Debian 13. You will learn how to systematically trace HTTP failures down through network namespaces, analyze reverse proxy logs, fix declarative Docker Compose labels, and apply Site Reliability Engineering (SRE) principles to prevent similar failure modes in production.
2. Problem Description and Environment
Scenario Context
The scenario (“San Juan”) features an edge load balancer managed by Traefik v3.6.1 sitting in front of four backend container replicas running traefik/whoami. The backends are intended to process HTTP traffic in a round-robin rotation when requests hit [http://app.sadserver](http://app.sadserver).
Environment Specifications
- Operating System: Debian 13 (Trixie)
- Privileges: Root / Sudo access available (
adminuser) - Container Engine: Docker Engine with Docker Compose
- Edge Proxy: Traefik v3.6.1
- Backend Image:
traefik/whoami(a tiny HTTP server that returns container OS metadata and hostname)
Reported Symptoms vs. Expected Behavior
- Expected Behavior: Executing
curl -s app.sadserver | head -n1should consistently return the host ID (container hostname) of one of the backend replicas, cycling evenly across all four backends on consecutive requests. - Actual Symptoms:
- Initial request execution against
app.sadserverfailed or returned blank. - Repeated requests intermittently returned
502 Bad Gateway. - Requests hung and timed out after several seconds.
- Only two of the four backend containers (
app01andapp03) received traffic.
3. Initial Investigation and Hypotheses
Before modifying any configuration files or restarting services, a systematic SRE approach requires gathering baseline metrics and formulating verifiable hypotheses.
Initial Observations & Gaps
- Container State: Are all containers running, or are backend instances stuck in a
CrashLoopBackOff? - Proxy Routing: Is Traefik receiving requests on port 80?
- Backend Reachability: Why are only select containers answering requests?
Hypothesis Matrix
| Hypothesis | Supporting Evidence | Evidence Against | Diagnostic Step |
|---|---|---|---|
| H1: Backend containers are crashing or offline | curl fails or hangs intermittently. |
None yet. | Inspect container state with docker ps -a. |
| H2: Port misconfiguration in Traefik labels | 502 Bad Gateway error suggests proxy reached an IP but refused connection. |
Traffic partially works for app01 and app03. |
Check Traefik container logs (docker logs) and Compose service labels. |
| H3: Container network isolation | Request timeouts indicate packets are dropped or unroutable. | Traefik is actively listening on exposed ports 80/443. | Inspect Docker bridge networks (docker network inspect). |
4. Troubleshooting Process
Step 1: Evaluating Container Health and Base Connectivity
Objective Check the running status of all containers in the stack to verify if any containers are stopped or restarting continuously.
Command
curl -s app.sad.server | head -n1
docker ps -a
Command Explanation
curl -s app.sad.server: Performs a silent HTTP GET request.docker ps -a: Lists all containers on the host engine. The-a(--all) flag is crucial because standarddocker pshides exited or dead containers.
Output and Analysis
admin@i-06b169eb07f44d33d:~$ curl -s app.sad.server | head -n1
admin@i-06b169eb07f44d33d:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a2f3f16b0928 traefik:v3.6.1 "/entrypoint.sh --ap…" 8 months ago Up About a minute 0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp app-traefik-1
2990d7e439ce traefik/whoami "/whoami" 8 months ago Up About a minute 80/tcp app-app03-1
049eb38a4bf4 traefik/whoami "/whoami" 8 months ago Up About a minute 80/tcp app-app01-1
5b3c8c3c22bd traefik/whoami "/whoami" 8 months ago Up About a minute 80/tcp app-app04-1
2fc39d0d7412 traefik/whoami "/whoami" 8 months ago Up About a minute 80/tcp app-app02-1
Analysis:
- All five containers (
app-traefik-1,app-app01-1,app-app02-1,app-app03-1,app-app04-1) report anUpstatus. This rules out Hypothesis H1 (crashing containers). - The initial
curlattempt againstapp.sad.serverreturned empty, indicating a typo in the domain (app.sad.servervsapp.sadserver) or an unroutable attempt.
Step 2: Reproducing the Failure Pattern via Loop Probing
Objective Automate sequential HTTP requests to test Traefik’s Weighted Round Robin (WRR) load-balancing algorithm across the backend pool.
Command
for i in {1..6}; do curl -s --max-time 3 http://app.sadserver | head -n1; sleep 0.5; done
Command Explanation
for i in {1..6}; do ... done: Runs a sequential Bash loop 6 times.curl -s --max-time 3: Connects to[http://app.sadserver](http://app.sadserver).--max-time 3sets a hard limit of 3 seconds per connection attempt to catch hanging requests without blocking the shell indefinitely.head -n1: Extracts the first line of thewhoamioutput (which displaysHostname: <container_id>).
Output and Analysis
Bad GatewayHostname: 049eb38a4bf4
Hostname: 2990d7e439ce
Hostname: 049eb38a4bf4
Hostname: 2990d7e439ce
Analysis:
- Request 1 failed immediately with
Bad Gateway. - Requests 2 through 5 succeeded, but only rotated between two containers:
049eb38a4bf4(app01) and2990d7e439ce(app03). - Requests to
app02(2fc39d0d7412) andapp04(5b3c8c3c22bd) failed to render valid hostnames.
Step 3: Inspecting Traefik Log Diagnostics
Objective Inspect Traefik’s internal runtime logs to analyze backend registration and connection error details.
Command
docker logs app-traefik-1 --tail 30
Command Explanation
docker logs: Fetches stdout/stderr logs from a target container.--tail 30: Limits output to the last 30 log lines for quick focused troubleshooting.
Output and Analysis
2026-11-10T15:18:34Z DBG ... > Creating server URL=http://172.19.0.3:81 entryPointName=web routerName=app@docker serverIndex=0 serviceName=app@docker
2026-11-10T15:18:34Z DBG ... > Creating server URL=http://172.19.0.5:80 entryPointName=web routerName=app@docker serverIndex=1 serviceName=app@docker
2026-11-10T15:18:34Z DBG ... > Creating server URL=http://172.19.0.4:80 entryPointName=web routerName=app@docker serverIndex=2 serviceName=app@docker
2026-11-10T15:18:34Z DBG ... > Creating server URL=http://172.18.0.2:80 entryPointName=web routerName=app@docker serverIndex=3 serviceName=app@docker
...
2026-11-10T15:19:31Z DBG ... > Service selected by WRR: http://172.19.0.3:81
2026-11-10T15:19:31Z DBG ... > 502 Bad Gateway error="dial tcp 172.19.0.3:81: connect: connection refused"
2026-11-10T15:19:32Z DBG ... > Service selected by WRR: http://172.18.0.2:80
2026-11-10T15:19:35Z DBG ... > 499 Client Closed Request error="context canceled"
Analysis:
- Finding A (
app02): Traefik dynamically registeredapp02at[http://172.19.0.3:81](http://172.19.0.3:81). When forwarding requests, it encounteredconnect: connection refusedon port 81. This confirms Hypothesis H2 (Port Misconfiguration). - Finding B (
app04): Traefik dynamically registeredapp04at[http://172.18.0.2:80](http://172.18.0.2:80). Notice thatapp04resides on subnet172.18.0.0/16, whereas Traefik and the other apps are on subnet172.19.0.0/16. The request timed out (context canceled), causing HTTP 499/504 errors. This confirms Hypothesis H3 (Network Isolation).
Step 4: Inspecting Declarative Configuration (docker-compose.yml)
Objective
Review the deployment specifications in docker-compose.yml to locate the source of the invalid labels and network definitions.
Command
cd ~/app && cat docker-compose.yml
Output and Analysis
services:
traefik:
image: traefik:v3.6.1
restart: unless-stopped
command:
- "--api.dashboard=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--log.level=DEBUG"
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- web
app01:
image: traefik/whoami
restart: unless-stopped
labels:
traefik.enable: "true"
traefik.http.routers.app.rule: Host(`app.sadserver`)
traefik.http.services.app.loadbalancer.server.port: "80"
networks:
- web
app02:
image: traefik/whoami
restart: unless-stopped
labels:
traefik.enable: "true"
traefik.http.routers.app.rule: Host(`app.sadserver`)
traefik.http.services.app.loadbalancer.server.port: "81"
networks:
- web
app03:
image: traefik/whoami
restart: unless-stopped
labels:
traefik.enable: "true"
traefik.http.routers.app.rule: Host(`app.sadserver`)
traefik.http.services.app.loadbalancer.server.port: "80"
networks:
- web
app04:
image: traefik/whoami
restart: unless-stopped
labels:
traefik.enable: "true"
traefik.http.routers.app.rule: Host(`app.sadserver`)
traefik.http.services.app.loadbalancer.server.port: "80"
networks:
- internal
networks:
web:
internal:
Analysis:
app02Defect: The labeltraefik.http.services.app.loadbalancer.server.portis explicitly set to"81". Thewhoamiapplication natively listens on port 80. Traefik attempted to open a socket to port 81, receiving a TCPRSTpacket fromapp02.app04Defect:app04is attached exclusively to networkinternal.traefikis attached exclusively to networkweb. Because Traefik shares no common network interface withapp04, packet routing between Traefik andapp04fails silently at the bridge boundary.
5. Root Cause Analysis
The failure consists of two distinct root causes operating in tandem:
+-------------------------------------------------+
| Client (curl request) |
+------------------------+------------------------+
|
v
+-------------------------------------------------+
| traefik (app-traefik-1) |
| Network: web (172.19.0.0/16) |
+----+-------------------+-------------------+----+
| | |
+--------------+ | +---------------+
| Port 80 (OK) | Port 81 (WRONG) | Isolated Net (WRONG)
v v v
+---------------+ +---------------+ +---------------+
| app01 | | app02 | | app04 |
| (172.19.0.4) | | (172.19.0.3) | | (172.18.0.2) |
| Network: web | | Network: web | | Net: internal |
+---------------+ +---------------+ +---------------+
Status: 200 OK TCP Connection Packet Timeout
Refused (502) (Context Canceled)
- Port Misconfiguration (
app02): Traefik dynamic provider reads Docker labels to construct destination target URLs. Settingserver.port: 81overrode Traefik’s auto-detection. Sincewhoamihas no service bound to port 81 inside its network namespace, the kernel immediately returned a TCPRSTflag. Traefik loggedconnection refusedand bubbled up an HTTP502 Bad Gatewayerror to the client. - Bridge Network Isolation (
app04): Docker builds distinct Linux bridge networks for isolated compose definitions. Traefik was placed on bridgeweb(172.19.0.0/16), whileapp04was placed on bridgeinternal(172.18.0.0/16). Without a common network attachment or cross-bridge routing rule, IP packets sent from Traefik to172.18.0.2hit a dead end, timing out after 3 seconds (499 Client Closed Request/context canceled).
6. Solution and Remediation
Modifying Configuration
Edit ~/app/docker-compose.yml to correct the port label on app02 and change app04’s network attachment to web.
vim ~/app/docker-compose.yml
Declarative Configuration Snippet (Diff)
app02:
image: traefik/whoami
restart: unless-stopped
labels:
traefik.enable: "true"
traefik.http.routers.app.rule: Host(`app.sadserver`)
- traefik.http.services.app.loadbalancer.server.port: "81"
+ traefik.http.services.app.loadbalancer.server.port: "80"
networks:
- web
app04:
image: traefik/whoami
restart: unless-stopped
labels:
traefik.enable: "true"
traefik.http.routers.app.rule: Host(`app.sadserver`)
traefik.http.services.app.loadbalancer.server.port: "80"
networks:
- - internal
+ - web
Applying the Fix
Recreate affected containers using Docker Compose:
docker compose up -d
Output:
admin@i-06b169eb07f44d33d:~/app$ docker compose up -d
[+] Running 5/5
✔ Container app-app03-1 Running 0.0s
✔ Container app-app01-1 Running 0.0s
✔ Container app-app02-1 Started 1.0s
✔ Container app-app04-1 Started 1.0s
✔ Container app-traefik-1 Running 0.0s
7. Verification and Validation
Step 1: Sequential Load-Balancer Probe
Verify that requests to [http://app.sadserver](http://app.sadserver) now rotate across all four distinct container hostnames without errors or timeouts.
for i in {1..8}; do curl -s --max-time 3 http://app.sadserver | head -n1; sleep 0.5; done
Output:
Hostname: f627cafcbe6c
Hostname: 049eb38a4bf4
Hostname: e90a18f1276d
Hostname: 2990d7e439ce
Hostname: e90a18f1276d
Hostname: 2990d7e439ce
Hostname: f627cafcbe6c
Hostname: 049eb38a4bf4
Analysis:
All four backend container IDs (f627cafcbe6c, 049eb38a4bf4, e90a18f1276d, 2990d7e439ce) are actively responding in round-robin fashion. No 502 Bad Gateway or timing-out responses occurred.
Step 2: Automated Verification Script
Run the verification test script provided by the platform:
/home/admin/agent/check.sh
Output:
OK
8. Alternative Troubleshooting Approaches
While reviewing Compose files and tailing logs quickly resolved this scenario, SREs frequently encounter more complex network edge cases. Below are alternative diagnostic approaches:
| Diagnostic Approach | Commands Used | Key Information Provided | Ideal Usage Scenario |
|---|---|---|---|
| Docker Inspect Subnet Correlation | docker inspect app-app04-1 |
Details container IP allocation, gateways, and attached network bridges. | Debugging inter-container routing issues and missing interfaces. |
| In-Container Socket Probe | docker exec -it app-traefik-1 nc -zv app02 80 |
Tests direct L4 TCP connectivity from inside the proxy container namespace. | Validating firewall rules, port bindings, and DNS resolution within containers. |
| Traefik API / Dashboard Query | curl http://localhost:8080/api/rawdata |
Exposes internal Traefik dynamic router tables, server weights, and status. | Diagnosing missing backend registration or health-check drops. |
| Network Packet Capture | tcpdump -i any host 172.19.0.3 |
Captures raw TCP handshakes (SYN, RST, ACK) across container virtual interfaces (veth). |
Tracing silent drops, MTU mismatches, or TCP reset causes at the kernel level. |
9. Linux & Container Command Deep Dive
Command: docker ps
Purpose: Interrogates the local Docker daemon engine via UNIX socket (/var/run/docker.sock) to report container status, image metadata, and exposed network socket bindings.
Syntax & Essential Flags:
docker ps [OPTIONS]
| Flag | Description | Troubleshooting Value |
|---|---|---|
-a, --all |
Show all containers (default shows just running). | Essential for spotting crashed containers (Exited (1)). |
-f, --filter |
Filter output based on conditions (e.g., status=restarting). |
Isolates unstable containers in high-density hosts. |
--format |
Pretty-print container outputs using Go templates. | E.g., docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}". |
Command: docker logs
Purpose: Streams stdout and stderr file logs generated by container PID 1 processes.
Syntax & Essential Flags:
docker logs [OPTIONS] CONTAINER
| Flag | Description | Troubleshooting Value |
|---|---|---|
--tail N |
Output only the last N lines. |
Filters out historical noise to highlight recent failures. |
-f, --follow |
Stream live log output in real time. | Real-time debugging while generating synthetic test traffic. |
-t, --timestamps |
Append ISO-8601 timestamps to each log entry. | Helps correlate proxy drops with external alerting events. |
--since |
Show logs since timestamp or relative duration (e.g., 5m). |
Focuses on specific incident time windows. |
Command: curl
Purpose: Command-line tool for transferring data with URLs using supported network protocols (HTTP, HTTPS, FTP, etc.).
Syntax & Essential Flags:
curl [OPTIONS] URL
| Flag | Description | Troubleshooting Value |
|---|---|---|
-s, --silent |
Suppress progress meters and non-error output. | Clean parsing when piping output into scripts or head/grep. |
-v, --verbose |
Print HTTP headers, TLS handshakes, and DNS resolution steps. | Debugging HTTP response status codes (502, 504, 403). |
--max-time N |
Set maximum time in seconds that the transfer is allowed to take. | Prevents automation scripts from hanging on drop-tail timeouts. |
-H "Host: ..." |
Send custom HTTP headers to test virtual host routing. | Essential for testing Traefik host rule matching (Host(...)). |
10. Underlying Linux Concepts and Architecture
1. Docker Networking & Linux Network Namespaces
Docker isolates containers using Linux Network Namespaces (netns). Each namespace maintains its own independent routing tables, iptables chains, network interfaces, and socket bindings.
When Docker Compose spins up a bridge network (e.g., web):
- Docker creates a virtual bridge interface on the host kernel (e.g.,
br-a1b2c3d4e5f6). - A Virtual Ethernet Pair (
veth) is allocated for each container. One end attaches to the host bridge; the other end enters the container’s network namespace aseth0. - Containers assigned to different bridges reside on entirely separate IP subnets (e.g.,
172.19.0.0/16vs172.18.0.0/16). By default, Docker does not enable cross-bridge IP forwarding without explicit network attachments.
2. Traefik Dynamic Provider Discovery
Unlike traditional static load balancers (such as Nginx or HAProxy), Traefik uses event-driven dynamic discovery:
- Traefik mounts
/var/run/docker.sockto listen to the Docker daemon event stream. - When containers start or update, Traefik inspects their labels (
traefik.enable,traefik.http.routers...). - If labels declare a custom
server.port, Traefik bypasses the default exposed container port and routes traffic directly to the user-specified port. If that port is wrong, the kernel’s network stack returns anECONNREFUSEDerror, resulting in a502 Bad Gateway.
11. Real-World Sysadmin and SRE Applications
Production Health Check Configuration
To prevent Traefik from routing production traffic to failing or misconfigured backend pods, configure explicit Traefik active health checks in docker-compose.yml:
app01:
image: traefik/whoami
labels:
- "traefik.enable=true"
- "traefik.http.routers.app.rule=Host(`app.sadserver`)"
- "traefik.http.services.app.loadbalancer.server.port=80"
- "traefik.http.services.app.loadbalancer.healthcheck.path=/"
- "traefik.http.services.app.loadbalancer.healthcheck.interval=5s"
- "traefik.http.services.app.loadbalancer.healthcheck.timeout=2s"
If a backend fails its health check, Traefik automatically removes the instance from the Weighted Round-Robin rotation, preventing 502 Bad Gateway errors from reaching end users.
CI/CD Linting and Static Analysis
Prevent broken configurations from entering production by validating Docker Compose definitions prior to deployment:
# Validate syntax and network definitions
docker compose config --quiet
12. Further Learning and Experiments
Hands-on Experiments
Experiment 1: Inspect Container Network Namespaces
Gain deep visibility into container bridge attachments directly from the host system:
# List all active Docker bridges
ip link show type bridge
# Inspect IP allocations across all services in the stack
docker inspect --format='{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}} ({{.NetworkID}}){{end}}' $(docker ps -q)
Experiment 2: Simulate a Backend Partial Outage
Stop a single backend container (app02) and observe how Traefik handles backend drops when passive or active health checks are configured.
docker stop app-app02-1
curl -i -H "Host: app.sadserver" http://localhost
13. Key Takeaways
- Verify Exposed Container Ports: Never assume a container listens on non-standard ports unless explicitly declared in the container’s
Dockerfileor entrypoint binary (whoamilistens on80). - Match Network Namespaces: Edge proxies like Traefik must share at least one common Docker network bridge with every backend service they route traffic to.
- Use Logs to Isolate Errors: Traefik’s debug logs quickly distinguish between TCP level errors (
connection refused-> wrong port) and IP/Layer 3 errors (context canceled-> unroutable network). - Always Limit Probing Timeouts: Always use
curl --max-timewhen probing unknown endpoints to prevent operational scripts from hanging indefinitely.
14. Quick Troubleshooting Reference / Runbook
+-----------------------------------+
| Traefik Load Balancing Issue |
+-----------------+-----------------+
|
v
+-----------------------------------+
| Run: docker ps -a |
| Are all containers Up? |
+--------+-----------------+--------+
| |
No | | Yes
+------------------+ +------------------+
| |
v v
+-----------------------+ +-------------------------------+
| Check container logs: | | Probe proxy endpoints: |
| docker logs <app> | | curl -v http://<domain> |
+-----------------------+ +---------------+---------------+
|
+------------------+------------------+
| |
v v
HTTP 502 Bad Gateway HTTP 504 Timeout / Hang
| |
v v
+-----------------------+ +-----------------------+
| Check Traefik Logs: | | Inspect Networks: |
| Connection Refused? | | Traefik & App on same |
| -> Fix port label in | | Docker bridge? |
| docker-compose.yml | | -> Fix networks list |
+-----------------------+ +-----------------------+
Emergency Command Cheat Sheet
# 1. Check all container statuses
docker ps -a
# 2. Tail Traefik debug logs
docker logs app-traefik-1 --tail 50 -f
# 3. Test endpoint loop with timeout
for i in {1..5}; do curl -s --max-time 2 -H "Host: app.sadserver" http://localhost; done
# 4. Re-apply docker compose changes gracefully
docker compose up -d --force-recreate