Domain 7Web & Application LayerINTERMEDIATE

HAProxy Production Administration: Architecture, Routing, Diagnostics, and Operational Handbook

A comprehensive operational handbook for HAProxy, covering L4/L7 load balancing architecture, configuration structures, ACL routing, runtime socket management, log analysis, and systematic troubleshooting workflows.

Domain: Domain 7: Web & Application Layer•Updated: 2026-11-03
#linux#sysadmin#haproxy#load-balancing#sre

1. Overview

HAProxy (High Availability Proxy) is an open-source, high-performance TCP and HTTP load balancer and reverse proxy. In production environments, HAProxy sits in front of application servers and distributes incoming network connections according to operator-defined rules. It handles advanced Layer 7 HTTP request routing (host, path, and header-based Access Control Lists), Layer 4 raw TCP connection passthrough, TLS termination, session persistence, and active health monitoring with automatic removal of unhealthy backend servers.

Unlike general-purpose web servers such as NGINX or Apache, HAProxy is purpose-built specifically for proxying, load balancing, and high-concurrency connection routing at scale.

                                 ┌─────────────────┐
                                 │  Backend App 1  │
                                 │  10.0.1.10:8080 │
                                 └─────────────────┘
                                        ▲
┌─────────┐      ┌──────────────┐       │
│ Client  │ ───► │   HAProxy    │ ──────┼─────────────────┐
└─────────┘      │ Load Balancer│       │                 │
                 └──────────────┘       ▼                 ▼
                                 ┌─────────────────┐ ┌─────────────────┐
                                 │  Backend App 2  │ │  Backend App 3  │
                                 │  10.0.1.11:8080 │ │  10.0.1.12:8080 │
                                 └─────────────────┘ └─────────────────┘

Problem Solved and Production Value

Modern distributed systems rely on HAProxy to achieve high availability, horizontal scalability, and fault tolerance:

  • High Availability: Prevents single-point-of-failure outages by actively checking application health and routing traffic away from failed nodes.
  • Horizontal Scaling: Allows operators to add or remove application nodes dynamically behind a single virtual IP or domain.
  • Traffic Control: Provides granular inspection of incoming requests to route traffic based on microservice routes, hostnames, client IP ranges, or headers.
  • TLS Offloading: Consolidates SSL/TLS management at the edge, freeing backend application servers from CPU-intensive cryptographic handshakes.

HAProxy vs. NGINX Comparison

While NGINX can perform reverse-proxying and basic load balancing, HAProxy is designed primarily for load balancing, deep protocol health checks, and maximum connection throughput. In enterprise architecture, HAProxy is often deployed at the outer network edge as the primary load balancer, forwarding traffic to NGINX instances or specialized application servers behind it.

Feature HAProxy NGINX
Primary Design Goal Specialized L4/L7 Load Balancer & Proxy Web Server, Content Host & Reverse Proxy
Static File Serving No (proxying only) Excellent (native file caching & static engine)
Health Checking Advanced active probes, custom payload evaluation, status expectation Basic active checks (mostly commercial NGINX Plus feature)
Runtime Administration Interactive UNIX socket (modify weights, drain servers live) Configuration reload required (or NGINX Plus API)
Connection Density Extremely high concurrency with minimal CPU/memory footprint High connection handling with static web serving features
Metrics & Diagnostics Detailed built-in stats interface & rich connection timing logs Standard access logging (advanced metrics via modules)

2. Architecture and Mental Model

HAProxy uses a single-threaded, event-driven, non-blocking architecture combined with an epoch-based priority scheduler. It uses low-level epoll (Linux) I/O multiplexing to manage thousands of concurrent connections over a minimal memory footprint.

Understanding HAProxy’s internal request processing lifecycle is critical for debugging routing failures and performance bottlenecks.

Request Lifecycle (Layer 7 HTTP Mode)

When an HTTP request traverses HAProxy, it passes through sequential processing stages:

  1. Bind: A frontend socket accepts incoming connections on one or more configured bind address:port definitions.
  2. Connection Acceptance & Mode Parsing: HAProxy accepts the socket. In HTTP mode, it parses request headers, URL paths, and cookies.
  3. ACL Evaluation: ACL rules evaluate request attributes (Host header, path prefix, IP address). use_backend conditions are evaluated sequentially until the first matching rule triggers.
  4. Backend Selection: The request is assigned to a target backend pool based on ACL matches or falls back to default_backend.
  5. Server Selection: The backend’s load-balancing algorithm (balance) selects an available server from the pool.
  6. Health Check Filter: HAProxy skips any servers currently flagged as DOWN by active health monitoring.
  7. Proxying & Streaming: HAProxy connects to the chosen server, applies header modifications (e.g., X-Forwarded-For), streams request/response payloads, and records timing metrics to system logs.

Request Lifecycle Flow

graph TD
    Client[Client Connection] --> Bind[1. Bind Socket Entry]
    Bind --> ModeCheck{Mode HTTP or TCP?}
    
    ModeCheck -- Mode TCP --> L4Select[L4 Pass-Through Engine]
    
    ModeCheck -- Mode HTTP --> L7Parse[2. Parse HTTP Headers & Path]
    L7Parse --> ACLEval[3. ACL Evaluation]
    ACLEval --> RuleMatch{use_backend Match?}
    RuleMatch -- Yes --> TargetBackend[4. Target Backend Pool]
    RuleMatch -- No --> DefaultBackend[4. default_backend Fallback]
    
    L4Select --> AlgSelect[5. Load-Balancing Algorithm]
    TargetBackend --> AlgSelect
    DefaultBackend --> AlgSelect
    
    AlgSelect --> HealthCheck{6. Server Health UP?}
    HealthCheck -- UP --> ProxyData[7. Proxy Data & Header Injection]
    HealthCheck -- DOWN --> NextServer[Skip DOWN Server & Pick Next]
    NextServer --> HealthCheck
    
    ProxyData --> BackendApp[Backend Application Server]
    BackendApp --> LogStats[Record Log Timings & Stats Counters]

3. Core Concepts and Terminology

Understanding these core concepts is essential for configuring and operating HAProxy effectively.

Frontend

The client-facing listener section in haproxy.cfg. Frontends define which network interfaces and ports to listen on (bind), define security constraints, process ACLs, and direct traffic to appropriate backends.

Backend

The server pool configuration section. Backends define the upstream application servers, the load-balancing algorithm, server weights, connection limits, and active health check rules.

Listen

A stanza combining both frontend and backend configurations into a single block. Listen blocks are commonly used for single-purpose setups, Layer 4 TCP proxying, or hosting the internal stats monitoring page.

Mode (mode http vs mode tcp)

  • mode http: Enables Layer 7 inspection. HAProxy parses HTTP requests, allowing header rewrites, path-based ACL routing, cookie-based sticky sessions, and detailed HTTP access logging.
  • mode tcp: Operates at Layer 4 passthrough. HAProxy proxies raw binary bytes between sockets without inspecting HTTP payloads. Necessary for non-HTTP protocols (databases, SSH) or end-to-end TLS passthrough.

Load Balancing Algorithms (balance)

  • roundrobin: Cycles sequentially through healthy servers. Adjusts for server weights dynamically. Ideal for short-lived, uniform HTTP requests.
  • leastconn: Directs traffic to the server with the lowest count of active connections. Preferred for long-lived sessions, WebSocket connections, or heavy database queries.
  • source: Hashes the client’s source IP address to pin that IP to a specific backend server. Useful for simple stateful session persistence.
  • uri: Hashes the URL path of the request to maximize cache hit ratios in upstream web proxy farms.

Health Checks

Active background probes configured with the check keyword. HAProxy sends periodic requests (TCP connections or HTTP probes) to each server. Servers failing a defined threshold of consecutive probes are marked DOWN and excluded from load balancing until they pass recovery checks.

Access Control Lists (ACLs)

Condition-testing mechanisms that evaluate request metadata—such as URL paths (path_beg), HTTP Host headers (hdr(host)), query parameters, or client source IPs (src). ACL outcomes trigger routing decisions (use_backend), header manipulations, or request denials.

Client IP Preservation (X-Forwarded-For / PROXY Protocol)

Because HAProxy terminates the client’s TCP socket, backend servers see HAProxy’s IP address as the source by default. Enabling option forwardfor appends the standard X-Forwarded-For HTTP header containing the original client IP. For TCP mode, the PROXY protocol (send-proxy) prepends connection metadata to the raw stream.


4. Installation and Version Management

Prerequisites

  • Operating System: Linux (Debian/Ubuntu, RHEL/Rocky Linux/CentOS)
  • Privileges: Root or sudo access
  • Dependencies: Systemd, OpenSSL (libssl), PCRE libraries

Package Installation

Debian / Ubuntu

sudo apt-get update
sudo apt-get install -y haproxy socat

RHEL / Rocky Linux / AlmaLinux

sudo dnf install -y haproxy socat

Identifying Installed Version and Compiled Capabilities

To inspect build features, SSL engine support, and compiled I/O engines, use the -vv flag:

haproxy -vv

Example Output Breakdown:

HAProxy version 2.8.3-1 2023/09/08 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc -O2
  OPTIONS = OPENSSL_WANT_REDUCE_MEMORY=YES USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Feature support : +EPOLL +SPLICE +OPENSSL +LUA +PCRE +THREAD
  • +OPENSSL: Confirms native SSL/TLS termination support.
  • +THREAD: Confirms multi-threading capabilities.
  • +EPOLL: Confirms high-performance Linux kernel I/O multiplexing.

System Directory Layout

  • Binary Path: /usr/sbin/haproxy
  • Main Configuration: /etc/haproxy/haproxy.cfg
  • Runtime Socket: /run/haproxy/admin.sock
  • Systemd Service: /lib/systemd/system/haproxy.service

5. Configuration Files and Directory Structure

The primary configuration file for HAProxy is /etc/haproxy/haproxy.cfg.

Configuration Directory Reference

File / Directory Purpose Important Details
/etc/haproxy/haproxy.cfg Primary configuration file. Houses global, defaults, frontend, backend, and listen stanzas.
/etc/haproxy/conf.d/ Supplementary configuration snippets. Used by modular deployment tools to split frontends/backends into separate files.
/run/haproxy/admin.sock Runtime API socket file. Enables non-disruptive query and state modifications via socat.
/etc/haproxy/certs/ TLS certificate store. Contains combined .pem files (SSL Certificate + Private Key + CA Chain).
/var/log/haproxy.log Dedicated log file (if configured in syslog). Receives structured system statistics and HTTP request access logs.

Section Hierarchy and Inheritances

  1. global: Configures process-wide security boundaries, system user/group drop privileges, logging endpoints, and socket definitions.
  2. defaults: Sets baseline configuration parameters inherited by all subsequent frontend, backend, and listen blocks. Explicit settings in specific blocks override defaults.
  3. frontend: Configures client-facing socket bindings (bind), ACL matching logic, and backend routing directives.
  4. backend: Defines target server IP arrays, load balancing rules, active check intervals, and server connection pools.
  5. listen: Combines frontend and backend logic into a consolidated configuration block.

Realistic Production Configuration Example

global
    log         /dev/log local0 info
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     10000
    user        haproxy
    group       haproxy
    daemon
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners

defaults
    log                     global
    mode                    http
    option                  httplog
    option                  dontlognull
    option                  forwardfor except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout connect         5s
    timeout client          50s
    timeout server          50s
    timeout tunnel          1h

frontend http_in
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/example.pem
    
    # ACL definitions
    acl is_api path_beg /api
    acl is_static path_end -i .jpg .png .css .js
    
    # Conditional backend routing
    use_backend api_servers if is_api
    use_backend static_servers if is_static
    default_backend app_servers

backend app_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server app1 10.0.1.10:8080 check maxconn 500
    server app2 10.0.1.11:8080 check maxconn 500

backend api_servers
    balance leastconn
    option httpchk GET /api/v1/health
    server api1 10.0.2.10:8080 check
    server api2 10.0.2.11:8080 check

backend static_servers
    balance roundrobin
    server static1 10.0.3.10:80 check

listen stats_page
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s

Configuration Syntax Validation

Always validate configuration syntax prior to initiating service reloads:

haproxy -c -f /etc/haproxy/haproxy.cfg

Expected Output: Configuration file is valid


6. Service Management and Common Operations

Systemd Service Control

# Check running status, process IDs, and active unit state
systemctl status haproxy

# Start HAProxy service
systemctl start haproxy

# Stop HAProxy service
systemctl stop haproxy

# Gracefully reload configuration (zero downtime)
systemctl reload haproxy

# Hard restart service (drops existing connections)
systemctl restart haproxy

Active Port and Process Inspection

# Inspect process PIDs and parent/child hierarchies
ps aux | grep haproxy

# Verify listening sockets and port bindings
ss -tlnp | grep haproxy

Reload vs. Restart Internal Mechanics

  • Graceful Reload (systemctl reload): The main control daemon parses and validates the disk configuration. If valid, it spawns new worker threads/processes while passing listening socket descriptors via the runtime socket (expose-fd listeners). New connections are immediately routed to the new workers. Old workers finish processing in-flight requests and exit naturally once connections drain.
  • Restart (systemctl restart): Immediately kills running worker processes, closes network sockets, and re-executes the service binary. In-flight TCP and HTTP sessions are abruptly terminated, resulting in client connection resets.

7. Common Administrative Tasks and Workflows

Workflow 1: Safely Modifying and Deploying Configurations

  1. Edit configuration file:
sudo nano /etc/haproxy/haproxy.cfg
  1. Validate syntax:
haproxy -c -f /etc/haproxy/haproxy.cfg
  1. Perform graceful reload:
sudo systemctl reload haproxy
  1. Confirm process status and system logs:
systemctl status haproxy
journalctl -u haproxy -n 20 --no-pager

Workflow 2: Zero-Downtime Server Maintenance (Drain Workflow)

Before taking a backend application server offline for maintenance, drain active sessions gracefully via the runtime admin socket:

  1. Connect to admin socket and set server state to DRAIN (disables new sessions while maintaining active connections):
echo "experimental-mode on; set server app_servers/app1 state drain" | sudo socat stdio /run/haproxy/admin.sock

Alternative legacy command:

echo "disable server app_servers/app1" | sudo socat stdio /run/haproxy/admin.sock
  1. Monitor stats UI or socket output until active connection count (scur) reaches 0.
  2. Perform server patching or application redeployment.
  3. Re-enable server traffic:
echo "enable server app_servers/app1" | sudo socat stdio /run/haproxy/admin.sock

Workflow 3: Updating SSL/TLS Certificates Live

HAProxy allows certificate updates without service restarts:

  1. Combine updated certificate, private key, and intermediate chain:
cat /etc/ssl/example.com.crt /etc/ssl/example.com.key /etc/ssl/chain.crt > /etc/haproxy/certs/example.pem
  1. Push new certificate payload directly via runtime socket:
echo -e "set ssl cert /etc/haproxy/certs/example.pem <<\n$(cat /etc/haproxy/certs/example.pem)\n" | sudo socat stdio /run/haproxy/admin.sock
echo "commit ssl cert /etc/haproxy/certs/example.pem" | sudo socat stdio /run/haproxy/admin.sock

8. CLI Command Reference and Cheatsheet

Core CLI Executable Commands

Command Purpose Key Flags / Arguments Example Usage Operational Context
haproxy Configuration syntax validation. -c (check syntax)


-f <file> (config path) | haproxy -c -f /etc/haproxy/haproxy.cfg | Run before any reload to prevent syntax errors from breaking the process. | | haproxy | Display version details and compile flags. | -v (short version)


-vv (verbose build details) | haproxy -vv | Verifies SSL/TLS, Lua, and epoll engine support. | | haproxy | Run in debug mode directly in foreground. | -d (debug mode)


-f <file> | haproxy -d -f /etc/haproxy/haproxy.cfg | Prints live connection state machine events directly to stdout. |

Runtime API Management Commands (via socat)

Interactive administration socket commands sent to /run/haproxy/admin.sock:

Command / Socket String Purpose Example Explanation / Impact
show info Displays runtime daemon statistics. `echo “show info” sudo socat stdio /run/haproxy/admin.sock`
show stat Exports tabular node and pool statistics. `echo “show stat” sudo socat stdio /run/haproxy/admin.sock`
show servers state <backend> Dumps detailed state for backend servers. `echo “show servers state app_servers” sudo socat stdio /run/haproxy/admin.sock`
disable server <backend>/<server> Places backend server into maintenance mode. `echo “disable server app_servers/app1” sudo socat stdio /run/haproxy/admin.sock`
enable server <backend>/<server> Restores maintenance server back to operational pool. `echo “enable server app_servers/app1” sudo socat stdio /run/haproxy/admin.sock`
set server <backend>/<server> weight <N> Dynamically alters load-balancing weight. `echo “set server app_servers/app1 weight 50” sudo socat stdio /run/haproxy/admin.sock`
show errors Displays detailed protocol error history. `echo “show errors” sudo socat stdio /run/haproxy/admin.sock`

9. Logs, Monitoring and Diagnostics

Syslog Integration

HAProxy logs directly to standard Unix syslog (/dev/log). Under Systemd-based operating systems, logs are consumed by journald.

# Follow HAProxy log entries in real-time
journalctl -u haproxy -f

# Filter log output by error severity level
journalctl -u haproxy -p err..emerg -n 50

Deciphering the Standard HTTP Access Log (httplog)

A standard option httplog log entry provides comprehensive connection metadata:

Oct 24 14:32:10 lb01 haproxy[2041]: 192.168.1.50:52134 [24/Oct/2026:14:32:10.112] http_in app_servers/app1 0/0/1/45/46 200 2412 - - -- 12/4/1/1/0 0/0 "GET /api/v1/users HTTP/1.1"

Field Decomposition Reference

192.168.1.50:52134 ── Client IP & Source Port
[24/Oct/2026:14:32:10.112] ── Log Timestamp
http_in ───────────── Frontend Name
app_servers/app1 ──── Backend Name / Server Choice
0/0/1/45/46 ───────── Session Timings (Tq/Tw/Tc/Tr/Tt)
200 ───────────────── HTTP Response Code
2412 ──────────────── Bytes Transferred to Client
-- ────────────────── Termination State Flags
12/4/1/1/0 ────────── Connection Counters (act/fe/be/srv/retries)
0/0 ───────────────── Queue Lengths (server/backend)
"GET /api/v1/users HTTP/1.1" ── HTTP Request Line

Timing Field Definitions (Tq/Tw/Tc/Tr/Tt)

  • Tq (Client Request Queue Time): Time spent waiting for the client to submit the complete HTTP request.
  • Tw (Queue Wait Time): Time spent waiting in the backend queue for a free server connection slot.
  • Tc (TCP Connect Time): Time required to establish the TCP connection to the backend server.
  • Tr (Response Header Time): Time taken by the backend server to return the first byte of HTTP headers after receiving the request.
  • Tt (Total Session Time): Total duration of the connection from receipt of first client byte to socket close.

Termination State Field Analysis

The two-character termination code indicates why a TCP or HTTP connection was closed:

Termination Code Meaning Primary Diagnostic Cause
-- Normal Termination. Request completed cleanly without errors.
cD Client Timeout During Data Transfer. Client connection dropped, network loss, or slow client stream.
sD Server Timeout During Data Transfer. Backend server crashed or exceeded timeout server limit during payload streaming.
SC Server Closed Connection. Backend server actively refused, reset (RST), or crashed during socket establishment.
CD Client Aborted Connection. User closed browser tab, or client curl command was interrupted before completion.

10. Troubleshooting Guide and Common Failure Scenarios

Problem: HAProxy Won’t Start or Reload

Symptoms

systemctl reload haproxy or systemctl start haproxy returns exit status failure. System logs report configuration parse errors or socket binding failures.

Possible Causes

  • Syntax error or unrecognized keyword in /etc/haproxy/haproxy.cfg.
  • Target network interface/port already occupied by another service (e.g., NGINX, Apache, or another HAProxy process).
  • SSL/TLS certificate path defined in configuration does not exist.
  • Duplicate backend or server identifiers.

Investigation and Diagnosis

  1. Run syntax verification explicitly to print offending line numbers:
haproxy -c -f /etc/haproxy/haproxy.cfg
  1. Inspect system journal entries for initialization exceptions:
journalctl -u haproxy -n 50 --no-pager
  1. Check port bindings to see if port 80/443 is already claimed by a rogue process:
ss -tlnp | grep -E ':80|:443'

Root Cause Identification

  • If syntax validation throws [ALERT] 296/141022 (2041) : parsing [/etc/haproxy/haproxy.cfg:42] : unknown keyword, line 42 contains a invalid keyword.
  • If logs report cannot bind socket [0.0.0.0:80], another service is listening on port 80.

Resolution

  • Correct syntax error on the identified file line.
  • Terminate conflicting service or rebind HAProxy to available ports:
systemctl stop nginx

Verification

haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl start haproxy
systemctl status haproxy

Prevention

Integrate haproxy -c checks into CI/CD deployment pipelines before pushing configurations to production nodes.


Problem: Reload Succeeded but Traffic Still Behaves Strangely

Symptoms

Running systemctl reload haproxy returns success, but updated ACL rules, timeouts, or backend modifications do not appear to apply.

Possible Causes

  • Long-lived connections delay old worker processes from exiting, leaving active traffic bound to old workers running the old configuration.
  • Editing a configuration file different from the one executed by the Systemd unit.
  • Runtime socket modifications (set server weight) were executed live but not saved to /etc/haproxy/haproxy.cfg on disk (a reload restores disk state).
  • Multiple HAProxy binary instances running concurrently on the host.

Investigation and Diagnosis

  1. Verify exact configuration file path targeted by systemd:
systemctl cat haproxy | grep ExecStart
  1. Inspect running processes to check for orphaned worker processes:
ps aux | grep haproxy
  1. Dump active server states via runtime socket:
echo "show info" | sudo socat stdio /run/haproxy/admin.sock

Root Cause Identification

  • If ps aux reveals worker processes running for hours alongside newly spawned PIDs, old workers are draining long-lived HTTP or WebSocket streams.
  • If systemctl cat haproxy shows /etc/haproxy/haproxy-custom.cfg, changes made to /etc/haproxy/haproxy.cfg are ignored.

Resolution

  • Ensure all socket-based server weight tweaks are committed to /etc/haproxy/haproxy.cfg on disk.
  • Terminate stuck legacy worker processes if drainage takes too long:
kill -9 <OLD_WORKER_PID>

Verification

Execute runtime socket show info query and verify process uptime and build dates match expectations.

Prevention

Set explicit hard limits on worker drainage times in global section: hard-stop-after 5m.


Problem: 503 Service Unavailable Returned to Clients

Symptoms

HTTP clients receive 503 Service Unavailable error pages from HAProxy.

HTTP/1.1 503 Service Unavailable
No server is available to handle this request.

Possible Causes

  • All backend servers in target pool are flagged DOWN by health checks.
  • ACL rule routes traffic to a backend name that contains a typo or lacks configured server IPs.
  • All servers in backend pool are set to maintenance mode (MAINT).

Investigation and Diagnosis

  1. Query backend state via admin socket:
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | grep -E 'DOWN|MAINT'
  1. Check frontend-to-backend ACL routing matches:
haproxy -c -f /etc/haproxy/haproxy.cfg
  1. Inspect access logs for matching requests:
journalctl -u haproxy -f | grep ' 503 '

Root Cause Identification

Log entry displays <NOSRV> in server column:

http_in api_servers/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- ...

This confirms HAProxy could not identify any operational server in api_servers.

Resolution

  • If servers are marked DOWN, fix underlying health check failures or application crashes.
  • If servers are in maintenance mode, restore them:
echo "enable server api_servers/api1" | sudo socat stdio /run/haproxy/admin.sock

Verification

Send HTTP request through load balancer host:

curl -i http://localhost/api/v1/health

Confirm response code returns 200 OK.

Prevention

Configure backup fallback servers using server app_backup 10.0.1.99:8080 check backup.


Problem: All Backends Marked DOWN (Health Check Failure)

Symptoms

Stats page displays all server rows highlighted red (DOWN). Connections fail despite application servers running normally.

Possible Causes

  • Default option httpchk sends OPTIONS / HTTP/1.1 without a Host header. Modern application frameworks reject headers lacking Host with 400 Bad Request or 404 Not Found.
  • Incorrect HTTP health path configured (e.g., probing / instead of /health).
  • Firewall or host security groups blocking probes originating from HAProxy source IP.

Investigation and Diagnosis

  1. Simulate exact health probe manually from HAProxy host to backend IP:
curl -v -X OPTIONS http://10.0.1.10:8080/
  1. Test probe using explicit Host header and path:
curl -v -H "Host: example.com" http://10.0.1.10:8080/health

Root Cause Identification

If manual curl returns HTTP/1.1 400 Bad Request, backend framework requires explicit Host headers for health checks.

Resolution

Update backend HTTP check definition to explicitly pass Host header:

# HAProxy 2.2+ Modern Syntax
backend app_servers
    option httpchk
    http-check send meth GET uri /health ver HTTP/1.1 hdr Host example.com
    http-check expect status 200
    server app1 10.0.1.10:8080 check

Verification

Query runtime status:

echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | grep app_servers

Confirm status changes from DOWN to UP.

Prevention

Validate health check requirements against application framework routes during deployment setup.


Problem: Health Checks Succeed but Application Fails

Symptoms

HAProxy stats report backend status as green (UP), but users encounter HTTP 500 internal server errors when submitting real application workflows.

Stats UI: /health -> 200 OK (Green / UP)
User Traffic: /api/checkout -> 500 Internal Server Error

Possible Causes

  • Health probe hits a lightweight static stub endpoint (e.g., /health) that returns 200 OK without verifying downstream dependencies like databases or caches.
  • Production endpoints (/api/checkout) trigger broken code paths or failing database transactions while static endpoints remain healthy.

Investigation and Diagnosis

  1. Probe lightweight health endpoint from HAProxy host:
curl -i http://10.0.1.10:8080/health
  1. Probe actual failing application endpoint directly against backend IP:
curl -i http://10.0.1.10:8080/api/checkout
  1. Issue same failing endpoint request through HAProxy load balancer front IP:
curl -i http://lb-frontend/api/checkout

Root Cause Identification

If /health returns 200 OK but /api/checkout returns 500 Internal Server Error directly from the backend IP, the issue resides within application dependencies, not HAProxy configuration.

Resolution

  • Resolve underlying application or database dependency issues on backend servers.
  • Configure deeper health checks that test database read readiness or inspect response bodies using http-check expect string "DB_OK".

Verification

Re-run application endpoint tests through the load balancer:

curl -i http://lb-frontend/api/checkout

Prevention

Avoid implementing superficial health checks that return static responses without verifying core application prerequisites.


Problem: Layer 4 vs Layer 7 Mode Confusion

Symptoms

Host, path, or header-based ACL rules fail to match. Traffic defaults to fallback backend, and HTTP logging attributes are missing.

Possible Causes

Frontend or defaults block is configured with mode tcp. Under Layer 4 TCP mode, HAProxy forwards bytes without parsing HTTP headers, rendering Layer 7 HTTP ACLs ineffective.

Investigation and Diagnosis

  1. Inspect configured modes across stanzas:
grep -E '^\s*mode' /etc/haproxy/haproxy.cfg
  1. Check log file formats. TCP logs display raw byte counts rather than HTTP response codes and request paths.

Root Cause Identification

defaults
    mode tcp # Causes L7 HTTP ACLs to fail silently

frontend http_in
    acl is_api path_beg /api # Silently ignored in TCP mode
    use_backend api_servers if is_api

Resolution

Set explicit mode http on frontends and backends requiring Layer 7 inspection:

frontend http_in
    mode http
    acl is_api path_beg /api
    use_backend api_servers if is_api

Verification

Reload service and execute request matching path rule:

curl -i http://localhost/api/test

Confirm request routes to api_servers backend.

Prevention

Specify mode http explicitly inside frontend stanzas rather than relying entirely on defaults.


Problem: Backend Sees HAProxy IP Instead of Original Client IP

Symptoms

Application backend access logs record HAProxy’s internal IP for all incoming client requests. Rate limiting, geo-location, and IP-based access security break on backend hosts.

Possible Causes

Configuration is missing option forwardfor in HTTP mode, or backend web servers are not configured to process X-Forwarded-For headers.

Investigation and Diagnosis

  1. Inspect HTTP backend headers using local debugging tool or echo server:
curl -i http://lb-host/debug
  1. Check backend configuration for option forwardfor.

Root Cause Identification

Backend request header output lacks X-Forwarded-For: <CLIENT_IP> line.

Resolution

Add option forwardfor to defaults or target backend stanzas:

defaults
    option forwardfor

For Layer 4 TCP proxying (e.g., proxying to NGINX via TCP mode), enable PROXY protocol instead:

backend tcp_out
    mode tcp
    server app1 10.0.1.10:443 send-proxy

Verification

Send test HTTP request through load balancer and inspect backend request headers to confirm X-Forwarded-For header is present.


11. Security and Best Practices

Privileged Process Isolation

Never run HAProxy as root. Drop process execution rights to unprivileged users within the global configuration stanza:

global
    user        haproxy
    group       haproxy
    chroot      /var/lib/haproxy

Hardening TLS Configurations

Enforce strong cipher suites, disable deprecated TLS protocols (SSLv3, TLS 1.0, 1.1), and enable HTTP Strict Transport Security (HSTS):

global
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384

frontend https_in
    bind *:443 ssl crt /etc/haproxy/certs/ strict-sni
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    http-response del-header Server

Administrative Socket Security

Restrict filesystem permissions on the UNIX administrative runtime socket to prevent unauthorized socket injection attacks:

global
    stats socket /run/haproxy/admin.sock mode 660 level admin user haproxy group haproxy

12. Performance and Resource Optimisation

Operating System Kernel Tuning (/etc/sysctl.conf)

High-volume load balancers require Linux kernel network stack adjustments to prevent ephemeral port exhaustion and SYN drop conditions:

# Maximum socket listen backlog
net.core.somaxconn = 65535

# Increase connection tracking limits
net.netfilter.nf_conntrack_max = 1048576

# Expand local ephemeral port range
net.ipv4.ip_local_port_range = 1024 65535

# Enable TCP SYN Cookies against FLOOD attacks
net.ipv4.tcp_syncookies = 1

# Max file descriptors
fs.file-max = 2097152

Apply sysctl changes:

sudo sysctl -p

HAProxy High-Concurrency Tuning

global
    maxconn 100000
    nbthread 4 # Match number of physical CPU cores

defaults
    mode http
    option http-server-close
    option multipkt

13. Backup, Recovery and Maintenance

Essential Files for Backup

To enable complete disaster recovery of an HAProxy node, back up these key paths:

  • Configuration directory: /etc/haproxy/*
  • TLS Certificate files: /etc/haproxy/certs/*
  • Systemd service file: /lib/systemd/system/haproxy.service

Automated Configuration Backup Script

#!/usr/bin/env bash
SET_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/var/backups/haproxy"

mkdir -p "${BACKUP_DIR}"
tar -czf "${BACKUP_DIR}/haproxy_config_${SET_DATE}.tar.gz" \
    /etc/haproxy/ \
    /lib/systemd/system/haproxy.service

# Retain backups for 30 days
find "${BACKUP_DIR}" -type f -name "*.tar.gz" -mtime +30 -delete

Disaster Recovery Restoration Procedure

  1. Reinstall HAProxy binary packages.
  2. Extract backup configuration archive:
tar -xzf /var/backups/haproxy/haproxy_config_20261103_120000.tar.gz -C /
  1. Test syntax:
haproxy -c -f /etc/haproxy/haproxy.cfg
  1. Start service:
systemctl enable --now haproxy

14. Automation and Integration

Ansible Configuration Deployment

An example Ansible playbook task for validating configuration syntax before reloading the service:

- name: Deploy HAProxy Configuration
  hosts: loadbalancers
  befores:
    - name: Ensure certificate store directory exists
      file:
        path: /etc/haproxy/certs
        state: directory
        mode: '0750'
        owner: haproxy
        group: haproxy

  tasks:
    - name: Transfer configuration file
      template:
        src: templates/haproxy.cfg.j2
        dest: /etc/haproxy/haproxy.cfg
        owner: haproxy
        group: haproxy
        mode: '0644'
      validate: 'haproxy -c -f %s'

    - name: Gracefully reload HAProxy service
      systemd:
        name: haproxy
        state: reloaded
        enabled: yes

Prometheus Metrics Integration

HAProxy includes a native Prometheus metric exporter. Enable it directly inside a frontend or listen block:

frontend prometheus_metrics
    bind *:9101
    mode http
    http-request use-service prometheus-exporter if { path /metrics }

15. Practical Labs and Exercises

Lab 1: Basic Dual-Server HTTP Load Balancer Setup

Objective

Configure HAProxy to balance HTTP traffic across two backend servers using the roundrobin algorithm.

Environment Prerequisites

  • 1 HAProxy host (IP: 10.0.1.5)
  • 2 Web server hosts (10.0.1.10, 10.0.1.11) serving port 8080

Instructions

  1. Edit /etc/haproxy/haproxy.cfg and configure the following frontend and backend blocks:
frontend http_in
    bind *:80
    default_backend app_pool

backend app_pool
    balance roundrobin
    option httpchk GET /health
    server node1 10.0.1.10:8080 check
    server node2 10.0.1.11:8080 check
  1. Validate syntax: haproxy -c -f /etc/haproxy/haproxy.cfg.
  2. Reload service: systemctl reload haproxy.

Validation

Run curl [http://10.0.1.5/](http://10.0.1.5/) multiple times consecutively. Verify that response outputs alternate between node1 and node2.


Lab 2: Path-Based ACL Microservice Routing

Objective

Route traffic pointing to /api to an API backend pool, and all remaining traffic to a static web backend pool.

Instructions

  1. Append the following ACL configuration to your frontend:
frontend http_in
    bind *:80
    mode http

    acl is_api path_beg /api

    use_backend api_pool if is_api
    default_backend web_pool

backend api_pool
    mode http
    server api1 10.0.2.10:8080 check

backend web_pool
    mode http
    server web1 10.0.1.10:8080 check
  1. Reload HAProxy.

Validation

  • Execute curl [http://10.0.1.5/api/users](http://10.0.1.5/api/users) -> Confirm routing to api1.
  • Execute curl [http://10.0.1.5/index.html](http://10.0.1.5/index.html) -> Confirm routing to web1.

Lab 3: Troubleshooting 503 Backend Failures

Objective

Investigate and resolve a production outage scenario where HAProxy returns 503 Service Unavailable errors.

Scenario Setup

A newly deployed backend server reports health check failures, marking all nodes DOWN.

Troubleshooting Steps

  1. Execute query against admin runtime socket to list server states:
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | grep DOWN
  1. Inspect log stream for termination codes:
journalctl -u haproxy -f
  1. Test health check URL manually using curl:
curl -v http://10.0.1.10:8080/health
  1. Update configuration to send the correct Host header and expected status code:
http-check send meth GET uri /health ver HTTP/1.1 hdr Host example.com
http-check expect status 200
  1. Reload HAProxy and verify status restores to UP.

16. Quick Reference and Cheat Sheet

Essential Configuration File Paths

Path Purpose
/etc/haproxy/haproxy.cfg Primary operational configuration file.
/run/haproxy/admin.sock Runtime API admin socket file.
/etc/haproxy/certs/ TLS SSL Certificate store directory.

Important Log Locations

Target / Command Purpose
journalctl -u haproxy -f Real-time system log output stream.
/var/log/haproxy.log Dedicated access log file (if configured in rsyslog).

Common Administrative Commands

Operational Task Command
Validate Config Syntax haproxy -c -f /etc/haproxy/haproxy.cfg
Graceful Service Reload systemctl reload haproxy
Inspect Build Capabilities haproxy -vv
Query Runtime Information `echo “show info”
Query Server Health States `echo “show stat”
Drain Server for Maintenance `echo “disable server /”
Enable Drained Server `echo “enable server /”

Troubleshooting Quick Reference

Symptom Primary Diagnostic Command Likely Cause
Service fails to start haproxy -c -f /etc/haproxy/haproxy.cfg Syntax error or bound port collision.
503 Service Unavailable `echo “show stat” socat stdio /run/haproxy/admin.sock`
ACL rules silently ignored grep '^ *mode' /etc/haproxy/haproxy.cfg Frontend running in mode tcp instead of mode http.
Backend logs wrong IP Check defaults block for option forwardfor Missing X-Forwarded-For header injection.

Default Network Ports

Port Protocol Purpose
80 TCP / HTTP Standard HTTP client ingress.
443 TCP / HTTPS Secure TLS/SSL client ingress.
8404 TCP / HTTP Built-in Stats Web Monitoring Interface.
9101 TCP / HTTP Prometheus Metrics Exporter listener endpoint.

17. Further Reading and References

  1. Official HAProxy Documentation Index: haproxy.org/#docs
  2. HAProxy Configuration Directive Reference: haproxy-dconv configuration
  3. HAProxy Starter Guide & Official Tutorial: haproxy.com/blog/haproxy-basic-configuration-tutorial