1. Overview
HAProxy (High Availability Proxy) is an open-source, high-performance TCP and HTTP load balancer and reverse proxy. In production environments, HAProxy sits in front of application servers and distributes incoming network connections according to operator-defined rules. It handles advanced Layer 7 HTTP request routing (host, path, and header-based Access Control Lists), Layer 4 raw TCP connection passthrough, TLS termination, session persistence, and active health monitoring with automatic removal of unhealthy backend servers.
Unlike general-purpose web servers such as NGINX or Apache, HAProxy is purpose-built specifically for proxying, load balancing, and high-concurrency connection routing at scale.
┌─────────────────┐
│ Backend App 1 │
│ 10.0.1.10:8080 │
└─────────────────┘
▲
┌─────────┐ ┌──────────────┐ │
│ Client │ ───► │ HAProxy │ ──────┼─────────────────┐
└─────────┘ │ Load Balancer│ │ │
└──────────────┘ ▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Backend App 2 │ │ Backend App 3 │
│ 10.0.1.11:8080 │ │ 10.0.1.12:8080 │
└─────────────────┘ └─────────────────┘
Problem Solved and Production Value
Modern distributed systems rely on HAProxy to achieve high availability, horizontal scalability, and fault tolerance:
- High Availability: Prevents single-point-of-failure outages by actively checking application health and routing traffic away from failed nodes.
- Horizontal Scaling: Allows operators to add or remove application nodes dynamically behind a single virtual IP or domain.
- Traffic Control: Provides granular inspection of incoming requests to route traffic based on microservice routes, hostnames, client IP ranges, or headers.
- TLS Offloading: Consolidates SSL/TLS management at the edge, freeing backend application servers from CPU-intensive cryptographic handshakes.
HAProxy vs. NGINX Comparison
While NGINX can perform reverse-proxying and basic load balancing, HAProxy is designed primarily for load balancing, deep protocol health checks, and maximum connection throughput. In enterprise architecture, HAProxy is often deployed at the outer network edge as the primary load balancer, forwarding traffic to NGINX instances or specialized application servers behind it.
| Feature | HAProxy | NGINX |
|---|---|---|
| Primary Design Goal | Specialized L4/L7 Load Balancer & Proxy | Web Server, Content Host & Reverse Proxy |
| Static File Serving | No (proxying only) | Excellent (native file caching & static engine) |
| Health Checking | Advanced active probes, custom payload evaluation, status expectation | Basic active checks (mostly commercial NGINX Plus feature) |
| Runtime Administration | Interactive UNIX socket (modify weights, drain servers live) | Configuration reload required (or NGINX Plus API) |
| Connection Density | Extremely high concurrency with minimal CPU/memory footprint | High connection handling with static web serving features |
| Metrics & Diagnostics | Detailed built-in stats interface & rich connection timing logs | Standard access logging (advanced metrics via modules) |
2. Architecture and Mental Model
HAProxy uses a single-threaded, event-driven, non-blocking architecture combined with an epoch-based priority scheduler. It uses low-level epoll (Linux) I/O multiplexing to manage thousands of concurrent connections over a minimal memory footprint.
Understanding HAProxy’s internal request processing lifecycle is critical for debugging routing failures and performance bottlenecks.
Request Lifecycle (Layer 7 HTTP Mode)
When an HTTP request traverses HAProxy, it passes through sequential processing stages:
- Bind: A frontend socket accepts incoming connections on one or more configured
bindaddress:port definitions. - Connection Acceptance & Mode Parsing: HAProxy accepts the socket. In HTTP mode, it parses request headers, URL paths, and cookies.
- ACL Evaluation: ACL rules evaluate request attributes (Host header, path prefix, IP address).
use_backendconditions are evaluated sequentially until the first matching rule triggers. - Backend Selection: The request is assigned to a target backend pool based on ACL matches or falls back to
default_backend. - Server Selection: The backend’s load-balancing algorithm (
balance) selects an available server from the pool. - Health Check Filter: HAProxy skips any servers currently flagged as
DOWNby active health monitoring. - Proxying & Streaming: HAProxy connects to the chosen server, applies header modifications (e.g.,
X-Forwarded-For), streams request/response payloads, and records timing metrics to system logs.
Request Lifecycle Flow
graph TD
Client[Client Connection] --> Bind[1. Bind Socket Entry]
Bind --> ModeCheck{Mode HTTP or TCP?}
ModeCheck -- Mode TCP --> L4Select[L4 Pass-Through Engine]
ModeCheck -- Mode HTTP --> L7Parse[2. Parse HTTP Headers & Path]
L7Parse --> ACLEval[3. ACL Evaluation]
ACLEval --> RuleMatch{use_backend Match?}
RuleMatch -- Yes --> TargetBackend[4. Target Backend Pool]
RuleMatch -- No --> DefaultBackend[4. default_backend Fallback]
L4Select --> AlgSelect[5. Load-Balancing Algorithm]
TargetBackend --> AlgSelect
DefaultBackend --> AlgSelect
AlgSelect --> HealthCheck{6. Server Health UP?}
HealthCheck -- UP --> ProxyData[7. Proxy Data & Header Injection]
HealthCheck -- DOWN --> NextServer[Skip DOWN Server & Pick Next]
NextServer --> HealthCheck
ProxyData --> BackendApp[Backend Application Server]
BackendApp --> LogStats[Record Log Timings & Stats Counters]
3. Core Concepts and Terminology
Understanding these core concepts is essential for configuring and operating HAProxy effectively.
Frontend
The client-facing listener section in haproxy.cfg. Frontends define which network interfaces and ports to listen on (bind), define security constraints, process ACLs, and direct traffic to appropriate backends.
Backend
The server pool configuration section. Backends define the upstream application servers, the load-balancing algorithm, server weights, connection limits, and active health check rules.
Listen
A stanza combining both frontend and backend configurations into a single block. Listen blocks are commonly used for single-purpose setups, Layer 4 TCP proxying, or hosting the internal stats monitoring page.
Mode (mode http vs mode tcp)
mode http: Enables Layer 7 inspection. HAProxy parses HTTP requests, allowing header rewrites, path-based ACL routing, cookie-based sticky sessions, and detailed HTTP access logging.mode tcp: Operates at Layer 4 passthrough. HAProxy proxies raw binary bytes between sockets without inspecting HTTP payloads. Necessary for non-HTTP protocols (databases, SSH) or end-to-end TLS passthrough.
Load Balancing Algorithms (balance)
roundrobin: Cycles sequentially through healthy servers. Adjusts for server weights dynamically. Ideal for short-lived, uniform HTTP requests.leastconn: Directs traffic to the server with the lowest count of active connections. Preferred for long-lived sessions, WebSocket connections, or heavy database queries.source: Hashes the client’s source IP address to pin that IP to a specific backend server. Useful for simple stateful session persistence.uri: Hashes the URL path of the request to maximize cache hit ratios in upstream web proxy farms.
Health Checks
Active background probes configured with the check keyword. HAProxy sends periodic requests (TCP connections or HTTP probes) to each server. Servers failing a defined threshold of consecutive probes are marked DOWN and excluded from load balancing until they pass recovery checks.
Access Control Lists (ACLs)
Condition-testing mechanisms that evaluate request metadata—such as URL paths (path_beg), HTTP Host headers (hdr(host)), query parameters, or client source IPs (src). ACL outcomes trigger routing decisions (use_backend), header manipulations, or request denials.
Client IP Preservation (X-Forwarded-For / PROXY Protocol)
Because HAProxy terminates the client’s TCP socket, backend servers see HAProxy’s IP address as the source by default. Enabling option forwardfor appends the standard X-Forwarded-For HTTP header containing the original client IP. For TCP mode, the PROXY protocol (send-proxy) prepends connection metadata to the raw stream.
4. Installation and Version Management
Prerequisites
- Operating System: Linux (Debian/Ubuntu, RHEL/Rocky Linux/CentOS)
- Privileges: Root or
sudoaccess - Dependencies: Systemd, OpenSSL (
libssl), PCRE libraries
Package Installation
Debian / Ubuntu
sudo apt-get update
sudo apt-get install -y haproxy socat
RHEL / Rocky Linux / AlmaLinux
sudo dnf install -y haproxy socat
Identifying Installed Version and Compiled Capabilities
To inspect build features, SSL engine support, and compiled I/O engines, use the -vv flag:
haproxy -vv
Example Output Breakdown:
HAProxy version 2.8.3-1 2023/09/08 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc -O2
OPTIONS = OPENSSL_WANT_REDUCE_MEMORY=YES USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
Feature support : +EPOLL +SPLICE +OPENSSL +LUA +PCRE +THREAD
+OPENSSL: Confirms native SSL/TLS termination support.+THREAD: Confirms multi-threading capabilities.+EPOLL: Confirms high-performance Linux kernel I/O multiplexing.
System Directory Layout
- Binary Path:
/usr/sbin/haproxy - Main Configuration:
/etc/haproxy/haproxy.cfg - Runtime Socket:
/run/haproxy/admin.sock - Systemd Service:
/lib/systemd/system/haproxy.service
5. Configuration Files and Directory Structure
The primary configuration file for HAProxy is /etc/haproxy/haproxy.cfg.
Configuration Directory Reference
| File / Directory | Purpose | Important Details |
|---|---|---|
/etc/haproxy/haproxy.cfg |
Primary configuration file. | Houses global, defaults, frontend, backend, and listen stanzas. |
/etc/haproxy/conf.d/ |
Supplementary configuration snippets. | Used by modular deployment tools to split frontends/backends into separate files. |
/run/haproxy/admin.sock |
Runtime API socket file. | Enables non-disruptive query and state modifications via socat. |
/etc/haproxy/certs/ |
TLS certificate store. | Contains combined .pem files (SSL Certificate + Private Key + CA Chain). |
/var/log/haproxy.log |
Dedicated log file (if configured in syslog). | Receives structured system statistics and HTTP request access logs. |
Section Hierarchy and Inheritances
global: Configures process-wide security boundaries, system user/group drop privileges, logging endpoints, and socket definitions.defaults: Sets baseline configuration parameters inherited by all subsequentfrontend,backend, andlistenblocks. Explicit settings in specific blocks overridedefaults.frontend: Configures client-facing socket bindings (bind), ACL matching logic, and backend routing directives.backend: Defines target server IP arrays, load balancing rules, active check intervals, and server connection pools.listen: Combines frontend and backend logic into a consolidated configuration block.
Realistic Production Configuration Example
global
log /dev/log local0 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 10000
user haproxy
group haproxy
daemon
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout connect 5s
timeout client 50s
timeout server 50s
timeout tunnel 1h
frontend http_in
bind *:80
bind *:443 ssl crt /etc/haproxy/certs/example.pem
# ACL definitions
acl is_api path_beg /api
acl is_static path_end -i .jpg .png .css .js
# Conditional backend routing
use_backend api_servers if is_api
use_backend static_servers if is_static
default_backend app_servers
backend app_servers
balance roundrobin
option httpchk GET /health
http-check expect status 200
server app1 10.0.1.10:8080 check maxconn 500
server app2 10.0.1.11:8080 check maxconn 500
backend api_servers
balance leastconn
option httpchk GET /api/v1/health
server api1 10.0.2.10:8080 check
server api2 10.0.2.11:8080 check
backend static_servers
balance roundrobin
server static1 10.0.3.10:80 check
listen stats_page
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
Configuration Syntax Validation
Always validate configuration syntax prior to initiating service reloads:
haproxy -c -f /etc/haproxy/haproxy.cfg
Expected Output: Configuration file is valid
6. Service Management and Common Operations
Systemd Service Control
# Check running status, process IDs, and active unit state
systemctl status haproxy
# Start HAProxy service
systemctl start haproxy
# Stop HAProxy service
systemctl stop haproxy
# Gracefully reload configuration (zero downtime)
systemctl reload haproxy
# Hard restart service (drops existing connections)
systemctl restart haproxy
Active Port and Process Inspection
# Inspect process PIDs and parent/child hierarchies
ps aux | grep haproxy
# Verify listening sockets and port bindings
ss -tlnp | grep haproxy
Reload vs. Restart Internal Mechanics
- Graceful Reload (
systemctl reload): The main control daemon parses and validates the disk configuration. If valid, it spawns new worker threads/processes while passing listening socket descriptors via the runtime socket (expose-fd listeners). New connections are immediately routed to the new workers. Old workers finish processing in-flight requests and exit naturally once connections drain. - Restart (
systemctl restart): Immediately kills running worker processes, closes network sockets, and re-executes the service binary. In-flight TCP and HTTP sessions are abruptly terminated, resulting in client connection resets.
7. Common Administrative Tasks and Workflows
Workflow 1: Safely Modifying and Deploying Configurations
- Edit configuration file:
sudo nano /etc/haproxy/haproxy.cfg
- Validate syntax:
haproxy -c -f /etc/haproxy/haproxy.cfg
- Perform graceful reload:
sudo systemctl reload haproxy
- Confirm process status and system logs:
systemctl status haproxy
journalctl -u haproxy -n 20 --no-pager
Workflow 2: Zero-Downtime Server Maintenance (Drain Workflow)
Before taking a backend application server offline for maintenance, drain active sessions gracefully via the runtime admin socket:
- Connect to admin socket and set server state to
DRAIN(disables new sessions while maintaining active connections):
echo "experimental-mode on; set server app_servers/app1 state drain" | sudo socat stdio /run/haproxy/admin.sock
Alternative legacy command:
echo "disable server app_servers/app1" | sudo socat stdio /run/haproxy/admin.sock
- Monitor stats UI or socket output until active connection count (
scur) reaches 0. - Perform server patching or application redeployment.
- Re-enable server traffic:
echo "enable server app_servers/app1" | sudo socat stdio /run/haproxy/admin.sock
Workflow 3: Updating SSL/TLS Certificates Live
HAProxy allows certificate updates without service restarts:
- Combine updated certificate, private key, and intermediate chain:
cat /etc/ssl/example.com.crt /etc/ssl/example.com.key /etc/ssl/chain.crt > /etc/haproxy/certs/example.pem
- Push new certificate payload directly via runtime socket:
echo -e "set ssl cert /etc/haproxy/certs/example.pem <<\n$(cat /etc/haproxy/certs/example.pem)\n" | sudo socat stdio /run/haproxy/admin.sock
echo "commit ssl cert /etc/haproxy/certs/example.pem" | sudo socat stdio /run/haproxy/admin.sock
8. CLI Command Reference and Cheatsheet
Core CLI Executable Commands
| Command | Purpose | Key Flags / Arguments | Example Usage | Operational Context |
|---|---|---|---|---|
haproxy |
Configuration syntax validation. | -c (check syntax) |
-f <file> (config path) | haproxy -c -f /etc/haproxy/haproxy.cfg | Run before any reload to prevent syntax errors from breaking the process. |
| haproxy | Display version details and compile flags. | -v (short version)
-vv (verbose build details) | haproxy -vv | Verifies SSL/TLS, Lua, and epoll engine support. |
| haproxy | Run in debug mode directly in foreground. | -d (debug mode)
-f <file> | haproxy -d -f /etc/haproxy/haproxy.cfg | Prints live connection state machine events directly to stdout. |
Runtime API Management Commands (via socat)
Interactive administration socket commands sent to /run/haproxy/admin.sock:
| Command / Socket String | Purpose | Example | Explanation / Impact |
|---|---|---|---|
show info |
Displays runtime daemon statistics. | `echo “show info” | sudo socat stdio /run/haproxy/admin.sock` |
show stat |
Exports tabular node and pool statistics. | `echo “show stat” | sudo socat stdio /run/haproxy/admin.sock` |
show servers state <backend> |
Dumps detailed state for backend servers. | `echo “show servers state app_servers” | sudo socat stdio /run/haproxy/admin.sock` |
disable server <backend>/<server> |
Places backend server into maintenance mode. | `echo “disable server app_servers/app1” | sudo socat stdio /run/haproxy/admin.sock` |
enable server <backend>/<server> |
Restores maintenance server back to operational pool. | `echo “enable server app_servers/app1” | sudo socat stdio /run/haproxy/admin.sock` |
set server <backend>/<server> weight <N> |
Dynamically alters load-balancing weight. | `echo “set server app_servers/app1 weight 50” | sudo socat stdio /run/haproxy/admin.sock` |
show errors |
Displays detailed protocol error history. | `echo “show errors” | sudo socat stdio /run/haproxy/admin.sock` |
9. Logs, Monitoring and Diagnostics
Syslog Integration
HAProxy logs directly to standard Unix syslog (/dev/log). Under Systemd-based operating systems, logs are consumed by journald.
# Follow HAProxy log entries in real-time
journalctl -u haproxy -f
# Filter log output by error severity level
journalctl -u haproxy -p err..emerg -n 50
Deciphering the Standard HTTP Access Log (httplog)
A standard option httplog log entry provides comprehensive connection metadata:
Oct 24 14:32:10 lb01 haproxy[2041]: 192.168.1.50:52134 [24/Oct/2026:14:32:10.112] http_in app_servers/app1 0/0/1/45/46 200 2412 - - -- 12/4/1/1/0 0/0 "GET /api/v1/users HTTP/1.1"
Field Decomposition Reference
192.168.1.50:52134 ── Client IP & Source Port
[24/Oct/2026:14:32:10.112] ── Log Timestamp
http_in ───────────── Frontend Name
app_servers/app1 ──── Backend Name / Server Choice
0/0/1/45/46 ───────── Session Timings (Tq/Tw/Tc/Tr/Tt)
200 ───────────────── HTTP Response Code
2412 ──────────────── Bytes Transferred to Client
-- ────────────────── Termination State Flags
12/4/1/1/0 ────────── Connection Counters (act/fe/be/srv/retries)
0/0 ───────────────── Queue Lengths (server/backend)
"GET /api/v1/users HTTP/1.1" ── HTTP Request Line
Timing Field Definitions (Tq/Tw/Tc/Tr/Tt)
Tq(Client Request Queue Time): Time spent waiting for the client to submit the complete HTTP request.Tw(Queue Wait Time): Time spent waiting in the backend queue for a free server connection slot.Tc(TCP Connect Time): Time required to establish the TCP connection to the backend server.Tr(Response Header Time): Time taken by the backend server to return the first byte of HTTP headers after receiving the request.Tt(Total Session Time): Total duration of the connection from receipt of first client byte to socket close.
Termination State Field Analysis
The two-character termination code indicates why a TCP or HTTP connection was closed:
| Termination Code | Meaning | Primary Diagnostic Cause |
|---|---|---|
-- |
Normal Termination. | Request completed cleanly without errors. |
cD |
Client Timeout During Data Transfer. | Client connection dropped, network loss, or slow client stream. |
sD |
Server Timeout During Data Transfer. | Backend server crashed or exceeded timeout server limit during payload streaming. |
SC |
Server Closed Connection. | Backend server actively refused, reset (RST), or crashed during socket establishment. |
CD |
Client Aborted Connection. | User closed browser tab, or client curl command was interrupted before completion. |
10. Troubleshooting Guide and Common Failure Scenarios
Problem: HAProxy Won’t Start or Reload
Symptoms
systemctl reload haproxy or systemctl start haproxy returns exit status failure. System logs report configuration parse errors or socket binding failures.
Possible Causes
- Syntax error or unrecognized keyword in
/etc/haproxy/haproxy.cfg. - Target network interface/port already occupied by another service (e.g., NGINX, Apache, or another HAProxy process).
- SSL/TLS certificate path defined in configuration does not exist.
- Duplicate backend or server identifiers.
Investigation and Diagnosis
- Run syntax verification explicitly to print offending line numbers:
haproxy -c -f /etc/haproxy/haproxy.cfg
- Inspect system journal entries for initialization exceptions:
journalctl -u haproxy -n 50 --no-pager
- Check port bindings to see if port 80/443 is already claimed by a rogue process:
ss -tlnp | grep -E ':80|:443'
Root Cause Identification
- If syntax validation throws
[ALERT] 296/141022 (2041) : parsing [/etc/haproxy/haproxy.cfg:42] : unknown keyword, line 42 contains a invalid keyword. - If logs report
cannot bind socket [0.0.0.0:80], another service is listening on port 80.
Resolution
- Correct syntax error on the identified file line.
- Terminate conflicting service or rebind HAProxy to available ports:
systemctl stop nginx
Verification
haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl start haproxy
systemctl status haproxy
Prevention
Integrate haproxy -c checks into CI/CD deployment pipelines before pushing configurations to production nodes.
Problem: Reload Succeeded but Traffic Still Behaves Strangely
Symptoms
Running systemctl reload haproxy returns success, but updated ACL rules, timeouts, or backend modifications do not appear to apply.
Possible Causes
- Long-lived connections delay old worker processes from exiting, leaving active traffic bound to old workers running the old configuration.
- Editing a configuration file different from the one executed by the Systemd unit.
- Runtime socket modifications (
set server weight) were executed live but not saved to/etc/haproxy/haproxy.cfgon disk (a reload restores disk state). - Multiple HAProxy binary instances running concurrently on the host.
Investigation and Diagnosis
- Verify exact configuration file path targeted by systemd:
systemctl cat haproxy | grep ExecStart
- Inspect running processes to check for orphaned worker processes:
ps aux | grep haproxy
- Dump active server states via runtime socket:
echo "show info" | sudo socat stdio /run/haproxy/admin.sock
Root Cause Identification
- If
ps auxreveals worker processes running for hours alongside newly spawned PIDs, old workers are draining long-lived HTTP or WebSocket streams. - If
systemctl cat haproxyshows/etc/haproxy/haproxy-custom.cfg, changes made to/etc/haproxy/haproxy.cfgare ignored.
Resolution
- Ensure all socket-based server weight tweaks are committed to
/etc/haproxy/haproxy.cfgon disk. - Terminate stuck legacy worker processes if drainage takes too long:
kill -9 <OLD_WORKER_PID>
Verification
Execute runtime socket show info query and verify process uptime and build dates match expectations.
Prevention
Set explicit hard limits on worker drainage times in global section: hard-stop-after 5m.
Problem: 503 Service Unavailable Returned to Clients
Symptoms
HTTP clients receive 503 Service Unavailable error pages from HAProxy.
HTTP/1.1 503 Service Unavailable
No server is available to handle this request.
Possible Causes
- All backend servers in target pool are flagged
DOWNby health checks. - ACL rule routes traffic to a backend name that contains a typo or lacks configured server IPs.
- All servers in backend pool are set to maintenance mode (
MAINT).
Investigation and Diagnosis
- Query backend state via admin socket:
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | grep -E 'DOWN|MAINT'
- Check frontend-to-backend ACL routing matches:
haproxy -c -f /etc/haproxy/haproxy.cfg
- Inspect access logs for matching requests:
journalctl -u haproxy -f | grep ' 503 '
Root Cause Identification
Log entry displays <NOSRV> in server column:
http_in api_servers/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- ...
This confirms HAProxy could not identify any operational server in api_servers.
Resolution
- If servers are marked
DOWN, fix underlying health check failures or application crashes. - If servers are in maintenance mode, restore them:
echo "enable server api_servers/api1" | sudo socat stdio /run/haproxy/admin.sock
Verification
Send HTTP request through load balancer host:
curl -i http://localhost/api/v1/health
Confirm response code returns 200 OK.
Prevention
Configure backup fallback servers using server app_backup 10.0.1.99:8080 check backup.
Problem: All Backends Marked DOWN (Health Check Failure)
Symptoms
Stats page displays all server rows highlighted red (DOWN). Connections fail despite application servers running normally.
Possible Causes
- Default
option httpchksendsOPTIONS / HTTP/1.1without aHostheader. Modern application frameworks reject headers lackingHostwith400 Bad Requestor404 Not Found. - Incorrect HTTP health path configured (e.g., probing
/instead of/health). - Firewall or host security groups blocking probes originating from HAProxy source IP.
Investigation and Diagnosis
- Simulate exact health probe manually from HAProxy host to backend IP:
curl -v -X OPTIONS http://10.0.1.10:8080/
- Test probe using explicit Host header and path:
curl -v -H "Host: example.com" http://10.0.1.10:8080/health
Root Cause Identification
If manual curl returns HTTP/1.1 400 Bad Request, backend framework requires explicit Host headers for health checks.
Resolution
Update backend HTTP check definition to explicitly pass Host header:
# HAProxy 2.2+ Modern Syntax
backend app_servers
option httpchk
http-check send meth GET uri /health ver HTTP/1.1 hdr Host example.com
http-check expect status 200
server app1 10.0.1.10:8080 check
Verification
Query runtime status:
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | grep app_servers
Confirm status changes from DOWN to UP.
Prevention
Validate health check requirements against application framework routes during deployment setup.
Problem: Health Checks Succeed but Application Fails
Symptoms
HAProxy stats report backend status as green (UP), but users encounter HTTP 500 internal server errors when submitting real application workflows.
Stats UI: /health -> 200 OK (Green / UP)
User Traffic: /api/checkout -> 500 Internal Server Error
Possible Causes
- Health probe hits a lightweight static stub endpoint (e.g.,
/health) that returns 200 OK without verifying downstream dependencies like databases or caches. - Production endpoints (
/api/checkout) trigger broken code paths or failing database transactions while static endpoints remain healthy.
Investigation and Diagnosis
- Probe lightweight health endpoint from HAProxy host:
curl -i http://10.0.1.10:8080/health
- Probe actual failing application endpoint directly against backend IP:
curl -i http://10.0.1.10:8080/api/checkout
- Issue same failing endpoint request through HAProxy load balancer front IP:
curl -i http://lb-frontend/api/checkout
Root Cause Identification
If /health returns 200 OK but /api/checkout returns 500 Internal Server Error directly from the backend IP, the issue resides within application dependencies, not HAProxy configuration.
Resolution
- Resolve underlying application or database dependency issues on backend servers.
- Configure deeper health checks that test database read readiness or inspect response bodies using
http-check expect string "DB_OK".
Verification
Re-run application endpoint tests through the load balancer:
curl -i http://lb-frontend/api/checkout
Prevention
Avoid implementing superficial health checks that return static responses without verifying core application prerequisites.
Problem: Layer 4 vs Layer 7 Mode Confusion
Symptoms
Host, path, or header-based ACL rules fail to match. Traffic defaults to fallback backend, and HTTP logging attributes are missing.
Possible Causes
Frontend or defaults block is configured with mode tcp. Under Layer 4 TCP mode, HAProxy forwards bytes without parsing HTTP headers, rendering Layer 7 HTTP ACLs ineffective.
Investigation and Diagnosis
- Inspect configured modes across stanzas:
grep -E '^\s*mode' /etc/haproxy/haproxy.cfg
- Check log file formats. TCP logs display raw byte counts rather than HTTP response codes and request paths.
Root Cause Identification
defaults
mode tcp # Causes L7 HTTP ACLs to fail silently
frontend http_in
acl is_api path_beg /api # Silently ignored in TCP mode
use_backend api_servers if is_api
Resolution
Set explicit mode http on frontends and backends requiring Layer 7 inspection:
frontend http_in
mode http
acl is_api path_beg /api
use_backend api_servers if is_api
Verification
Reload service and execute request matching path rule:
curl -i http://localhost/api/test
Confirm request routes to api_servers backend.
Prevention
Specify mode http explicitly inside frontend stanzas rather than relying entirely on defaults.
Problem: Backend Sees HAProxy IP Instead of Original Client IP
Symptoms
Application backend access logs record HAProxy’s internal IP for all incoming client requests. Rate limiting, geo-location, and IP-based access security break on backend hosts.
Possible Causes
Configuration is missing option forwardfor in HTTP mode, or backend web servers are not configured to process X-Forwarded-For headers.
Investigation and Diagnosis
- Inspect HTTP backend headers using local debugging tool or echo server:
curl -i http://lb-host/debug
- Check backend configuration for
option forwardfor.
Root Cause Identification
Backend request header output lacks X-Forwarded-For: <CLIENT_IP> line.
Resolution
Add option forwardfor to defaults or target backend stanzas:
defaults
option forwardfor
For Layer 4 TCP proxying (e.g., proxying to NGINX via TCP mode), enable PROXY protocol instead:
backend tcp_out
mode tcp
server app1 10.0.1.10:443 send-proxy
Verification
Send test HTTP request through load balancer and inspect backend request headers to confirm X-Forwarded-For header is present.
11. Security and Best Practices
Privileged Process Isolation
Never run HAProxy as root. Drop process execution rights to unprivileged users within the global configuration stanza:
global
user haproxy
group haproxy
chroot /var/lib/haproxy
Hardening TLS Configurations
Enforce strong cipher suites, disable deprecated TLS protocols (SSLv3, TLS 1.0, 1.1), and enable HTTP Strict Transport Security (HSTS):
global
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
frontend https_in
bind *:443 ssl crt /etc/haproxy/certs/ strict-sni
http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
http-response del-header Server
Administrative Socket Security
Restrict filesystem permissions on the UNIX administrative runtime socket to prevent unauthorized socket injection attacks:
global
stats socket /run/haproxy/admin.sock mode 660 level admin user haproxy group haproxy
12. Performance and Resource Optimisation
Operating System Kernel Tuning (/etc/sysctl.conf)
High-volume load balancers require Linux kernel network stack adjustments to prevent ephemeral port exhaustion and SYN drop conditions:
# Maximum socket listen backlog
net.core.somaxconn = 65535
# Increase connection tracking limits
net.netfilter.nf_conntrack_max = 1048576
# Expand local ephemeral port range
net.ipv4.ip_local_port_range = 1024 65535
# Enable TCP SYN Cookies against FLOOD attacks
net.ipv4.tcp_syncookies = 1
# Max file descriptors
fs.file-max = 2097152
Apply sysctl changes:
sudo sysctl -p
HAProxy High-Concurrency Tuning
global
maxconn 100000
nbthread 4 # Match number of physical CPU cores
defaults
mode http
option http-server-close
option multipkt
13. Backup, Recovery and Maintenance
Essential Files for Backup
To enable complete disaster recovery of an HAProxy node, back up these key paths:
- Configuration directory:
/etc/haproxy/* - TLS Certificate files:
/etc/haproxy/certs/* - Systemd service file:
/lib/systemd/system/haproxy.service
Automated Configuration Backup Script
#!/usr/bin/env bash
SET_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/var/backups/haproxy"
mkdir -p "${BACKUP_DIR}"
tar -czf "${BACKUP_DIR}/haproxy_config_${SET_DATE}.tar.gz" \
/etc/haproxy/ \
/lib/systemd/system/haproxy.service
# Retain backups for 30 days
find "${BACKUP_DIR}" -type f -name "*.tar.gz" -mtime +30 -delete
Disaster Recovery Restoration Procedure
- Reinstall HAProxy binary packages.
- Extract backup configuration archive:
tar -xzf /var/backups/haproxy/haproxy_config_20261103_120000.tar.gz -C /
- Test syntax:
haproxy -c -f /etc/haproxy/haproxy.cfg
- Start service:
systemctl enable --now haproxy
14. Automation and Integration
Ansible Configuration Deployment
An example Ansible playbook task for validating configuration syntax before reloading the service:
- name: Deploy HAProxy Configuration
hosts: loadbalancers
befores:
- name: Ensure certificate store directory exists
file:
path: /etc/haproxy/certs
state: directory
mode: '0750'
owner: haproxy
group: haproxy
tasks:
- name: Transfer configuration file
template:
src: templates/haproxy.cfg.j2
dest: /etc/haproxy/haproxy.cfg
owner: haproxy
group: haproxy
mode: '0644'
validate: 'haproxy -c -f %s'
- name: Gracefully reload HAProxy service
systemd:
name: haproxy
state: reloaded
enabled: yes
Prometheus Metrics Integration
HAProxy includes a native Prometheus metric exporter. Enable it directly inside a frontend or listen block:
frontend prometheus_metrics
bind *:9101
mode http
http-request use-service prometheus-exporter if { path /metrics }
15. Practical Labs and Exercises
Lab 1: Basic Dual-Server HTTP Load Balancer Setup
Objective
Configure HAProxy to balance HTTP traffic across two backend servers using the roundrobin algorithm.
Environment Prerequisites
- 1 HAProxy host (IP:
10.0.1.5) - 2 Web server hosts (
10.0.1.10,10.0.1.11) serving port 8080
Instructions
- Edit
/etc/haproxy/haproxy.cfgand configure the following frontend and backend blocks:
frontend http_in
bind *:80
default_backend app_pool
backend app_pool
balance roundrobin
option httpchk GET /health
server node1 10.0.1.10:8080 check
server node2 10.0.1.11:8080 check
- Validate syntax:
haproxy -c -f /etc/haproxy/haproxy.cfg. - Reload service:
systemctl reload haproxy.
Validation
Run curl [http://10.0.1.5/](http://10.0.1.5/) multiple times consecutively. Verify that response outputs alternate between node1 and node2.
Lab 2: Path-Based ACL Microservice Routing
Objective
Route traffic pointing to /api to an API backend pool, and all remaining traffic to a static web backend pool.
Instructions
- Append the following ACL configuration to your frontend:
frontend http_in
bind *:80
mode http
acl is_api path_beg /api
use_backend api_pool if is_api
default_backend web_pool
backend api_pool
mode http
server api1 10.0.2.10:8080 check
backend web_pool
mode http
server web1 10.0.1.10:8080 check
- Reload HAProxy.
Validation
- Execute
curl [http://10.0.1.5/api/users](http://10.0.1.5/api/users)-> Confirm routing toapi1. - Execute
curl [http://10.0.1.5/index.html](http://10.0.1.5/index.html)-> Confirm routing toweb1.
Lab 3: Troubleshooting 503 Backend Failures
Objective
Investigate and resolve a production outage scenario where HAProxy returns 503 Service Unavailable errors.
Scenario Setup
A newly deployed backend server reports health check failures, marking all nodes DOWN.
Troubleshooting Steps
- Execute query against admin runtime socket to list server states:
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | grep DOWN
- Inspect log stream for termination codes:
journalctl -u haproxy -f
- Test health check URL manually using
curl:
curl -v http://10.0.1.10:8080/health
- Update configuration to send the correct Host header and expected status code:
http-check send meth GET uri /health ver HTTP/1.1 hdr Host example.com
http-check expect status 200
- Reload HAProxy and verify status restores to
UP.
16. Quick Reference and Cheat Sheet
Essential Configuration File Paths
| Path | Purpose |
|---|---|
/etc/haproxy/haproxy.cfg |
Primary operational configuration file. |
/run/haproxy/admin.sock |
Runtime API admin socket file. |
/etc/haproxy/certs/ |
TLS SSL Certificate store directory. |
Important Log Locations
| Target / Command | Purpose |
|---|---|
journalctl -u haproxy -f |
Real-time system log output stream. |
/var/log/haproxy.log |
Dedicated access log file (if configured in rsyslog). |
Common Administrative Commands
| Operational Task | Command |
|---|---|
| Validate Config Syntax | haproxy -c -f /etc/haproxy/haproxy.cfg |
| Graceful Service Reload | systemctl reload haproxy |
| Inspect Build Capabilities | haproxy -vv |
| Query Runtime Information | `echo “show info” |
| Query Server Health States | `echo “show stat” |
| Drain Server for Maintenance | `echo “disable server |
| Enable Drained Server | `echo “enable server |
Troubleshooting Quick Reference
| Symptom | Primary Diagnostic Command | Likely Cause |
|---|---|---|
| Service fails to start | haproxy -c -f /etc/haproxy/haproxy.cfg |
Syntax error or bound port collision. |
| 503 Service Unavailable | `echo “show stat” | socat stdio /run/haproxy/admin.sock` |
| ACL rules silently ignored | grep '^ *mode' /etc/haproxy/haproxy.cfg |
Frontend running in mode tcp instead of mode http. |
| Backend logs wrong IP | Check defaults block for option forwardfor |
Missing X-Forwarded-For header injection. |
Default Network Ports
| Port | Protocol | Purpose |
|---|---|---|
80 |
TCP / HTTP | Standard HTTP client ingress. |
443 |
TCP / HTTPS | Secure TLS/SSL client ingress. |
8404 |
TCP / HTTP | Built-in Stats Web Monitoring Interface. |
9101 |
TCP / HTTP | Prometheus Metrics Exporter listener endpoint. |
17. Further Reading and References
- Official HAProxy Documentation Index: haproxy.org/#docs
- HAProxy Configuration Directive Reference: haproxy-dconv configuration
- HAProxy Starter Guide & Official Tutorial: haproxy.com/blog/haproxy-basic-configuration-tutorial