1. Overview
Envoy is an open-source, high-performance L4/L7 edge and service proxy designed for cloud-native applications. Developed originally by Lyft, Envoy sits directly in the data path between incoming client requests and backend application services. It handles network traffic by terminating client connections, parsing request protocols (such as HTTP/1.1, HTTP/2, gRPC, and TCP), enforcing routing logic, load-balancing traffic across backend instances, enforcing resilience policies, managing Transport Layer Security (TLS), and emitting telemetry metrics and access logs.
+----------------+ +-------------------+ +-----------------+
| | HTTP | Envoy Proxy | gRPC | |
| Client App / |-------->| (Terminates TLS, |-------->| Upstream Host A |
| External User | HTTPS | Applies Filters | +-----------------+
| | | & Routing) | | Upstream Host B |
+----------------+ +-------------------+ +-----------------+
The Problem Envoy Solves
In modern microservice architectures, infrastructure concerns such as protocol parsing, service discovery, client-side load balancing, TLS encryption, retries, circuit breaking, rate limiting, and distributed tracing are often implemented across multiple application codebases. This leads to fragmented language libraries, inconsistent retry semantics, difficult maintenance, and blind spots in observability.
Envoy abstracts these network mechanisms away from application code. By acting as a universal network proxy, Envoy provides a standard networking layer across all services, regardless of the underlying programming language or runtime environment.
Primary Deployment Models and Use Cases
Envoy is commonly deployed across four primary architectural roles:
- Kubernetes Service Mesh Sidecar: Runs alongside application containers within the same pod (e.g., Istio) to manage inter-service East-West traffic, enforce mutual TLS (mTLS), and handle dynamic discovery.
- Edge Ingress / API Gateway: Functions as the front proxy accepting external North-South client traffic, performing authentication, rate-limiting, path routing, and protocol translation.
- Internal Load Balancer: Provides advanced load balancing (such as gRPC load balancing, ring hash, and least-request routing) with active health checking and outlier detection across internal host clusters.
- Protocol Bridge & Origination: Translates HTTP/1.1 or REST requests into gRPC, originates upstream TLS connections, or proxies raw TCP streams.
Envoy vs. Nginx, HAProxy, and Traefik
Selecting the appropriate reverse proxy depends on configuration models, target operational environments, and control plane requirements:
| Feature / Capability | Envoy Proxy | Nginx | HAProxy | Traefik |
|---|---|---|---|---|
| Primary Design Target | Dynamic cloud-native, microservices, service mesh | Web serving, static content, edge reverse proxy | High-performance L4/L7 load balancing | Dynamic container and Kubernetes ingress |
| Configuration Model | API-driven dynamic control plane (xDS) & static YAML | File-based static configuration files | File-based static configuration files | Dynamic configuration via provider labels/CRDs |
| gRPC & HTTP/2 Support | First-class, bi-directional streaming natively supported | Module-based / HTTP/2 & basic gRPC support | Strong L4/L7 gRPC support | Native HTTP/2 & gRPC support |
| Dynamic Configuration | Complete zero-downtime streaming updates without process reload | Master process reload required for config changes | Runtime socket configuration for endpoints; reload for routes | Automatic reload on container engine events |
| Observability & Telemetry | Granular per-route/cluster Prometheus stats, native tracing | Basic status page; stub_status / extended metrics in Nginx Plus | Detailed stats page and runtime socket statistics | Built-in web dashboard and basic Prometheus metrics |
| Extensibility | C++ native modules, WebAssembly (WASM), and Lua filters | C modules, OpenResty (Lua) | Lua scripts | Go plugins, WebAssembly |
2. Architecture and Mental Model
To effectively operate and troubleshoot Envoy, systems engineers must understand its internal processing pipeline and request execution model.
Incoming Connection
│
▼
┌───────────────┐
│ Listener │ Binds socket (e.g., 0.0.0.0:10000)
└───────┬───────┘
│
▼
┌───────────────┐
│ Filter Chain │ Network Filters (TLS Termination, Byte Inspection)
└───────┬───────┘
│
▼
┌───────────────┐
│ HttpConnMgr │ L7 HTTP Parsing, Header Inspection, Tracing
└───────┬───────┘
│
▼
┌───────────────┐
│ Route Match │ Matches Host, Path, Query Parameters to Target Cluster
└───────┬───────┘
│
▼
┌───────────────┐
│ Cluster │ Selects Upstream Service Pool, Applies LB Policy & Circuit Breakers
└───────┬───────┘
│
▼
┌───────────────┐
│ Endpoint │ Resolves Target IP:Port Host Endpoint
└───────┬───────┘
│
▼
Forward Request to Upstream Host
The Request Lifecycle
- Client Connection Establishment: A client opens a TCP socket to an Envoy Listener listening on a specified network address and port (e.g.,
0.0.0.0:10000). - Filter Chain Execution: The connection passes through a sequence of network (L4) filters configured within the listener. If TLS is configured, the TLS inspector filter terminates TLS or verifies mTLS certificates.
- HTTP Connection Manager (HCM) Parsing: For HTTP/gRPC traffic, the connection is handed off to the
envoy.filters.network.http_connection_managerfilter. The HCM parses HTTP/1.1 or HTTP/2 frames, processes headers, injects tracing contexts, and evaluates L7 HTTP filters (such as authentication or rate limiting). - Route Matching: The HCM evaluates the request’s HTTP
Host/Authorityheaders, path prefix, query parameters, or HTTP headers against a defined Route Configuration. The matching route points to a target Cluster. - Cluster Selection & Load Balancing: Envoy identifies the target cluster representing the group of backend instances. It applies load balancing algorithms (e.g., Round Robin, Least Request, Ring Hash) to select an available Endpoint.
- Resilience & Circuit Breaker Check: Envoy verifies whether cluster-level circuit breaker thresholds (such as pending request limits or maximum connection counts) have been exceeded. It also evaluates whether the selected endpoint is healthy based on active health checks or passive outlier detection.
- Upstream Transmission & Response Routing: Envoy forwards the request to the upstream endpoint IP:port. When the response returns, Envoy passes bytes back through the filter chain to the downstream client, updates internal statistical counters, and emits structured access log records.
Static vs. Dynamic Configuration (xDS Protocol)
Envoy can be configured using a static YAML bootstrap file (envoy.yaml) or dynamically via gRPC/REST APIs known as xDS (x Discovery Service) APIs.
+------------------------+
| Control Plane Server |
| (Istio, Envoy Gateway) |
+-----------+------------+
|
| gRPC Stream (xDS)
v
+---------------------------------------------------------------+
| Envoy Proxy |
| |
| [ LDS ] [ RDS ] [ CDS ] [ EDS ] [ SDS ] |
| Listener --> Route --> Cluster --> Endpoint -> Secret |
| Discovery Discovery Discovery Discovery Discovery|
+---------------------------------------------------------------+
- Static Configuration: Entire configurations (listeners, routes, clusters, endpoints) are specified directly within the local configuration file. Static configurations are simple to test and debug locally, but require process restarts or hot-reloads to update endpoints or routes.
- Dynamic Configuration (xDS): A lightweight static bootstrap configuration instructs Envoy to establish a gRPC stream to an external control plane management server (such as
istiodor an Envoy Gateway controller). The control plane streams incremental configuration updates without dropping connections or restarting the binary: - LDS (Listener Discovery Service): Pushes dynamic updates for listening sockets and filter chains.
- RDS (Route Discovery Service): Pushes dynamic route tables, virtual host domains, and path rules.
- CDS (Cluster Discovery Service): Pushes dynamic definitions of upstream service pools, connection pool settings, and circuit breaking limits.
- EDS (Endpoint Discovery Service): Pushes dynamic lists of operational backend IP addresses and ports. This updates rapidly as pods or hosts autoscale.
- SDS (Secret Discovery Service): Delivers TLS certificates, private keys, and CA verification roots dynamically without writing secrets to disk.
- ADS (Aggregated Discovery Service): Encapsulates all xDS updates over a single unified gRPC multiplexed stream to maintain strict dependency ordering (e.g., ensuring a cluster exists before endpoints reference it).
3. Core Concepts and Terminology
Understanding Envoy’s fundamental abstractions is necessary to design valid configurations and debug runtime issues.
Core Abstractions
- Listener: An operational network socket bound to an IP address and port that accepts incoming connections from downstream clients.
- Filter Chain: A sequential pipeline of network (L4) and HTTP (L7) filters attached to a listener that inspects, modifies, or routes incoming connection bytes.
- HTTP Connection Manager (HCM): The primary network filter (
envoy.filters.network.http_connection_manager) that converts raw stream bytes into HTTP requests and responses, manages header manipulations, and routes traffic. - Route Configuration: A set of routing rules within the HCM that matches incoming requests (by path, host, or header) and directs them to a backend cluster.
- Virtual Host: A sub-component of a route configuration defining domain matching rules (
domains: ["api.example.com"]) and associated route definitions. - Cluster: A logical grouping of upstream backend hosts offering identical service functionality, configured with specific load balancing protocols, health checks, and connection pools.
- Endpoint: A specific IP address and port representing an individual service instance belonging to a cluster.
- Upstream: A backend server or service that receives requests forwarded by Envoy.
- Downstream: A client entity that connects to Envoy to issue requests.
- Circuit Breaker: A protective runtime mechanism that limits connections, pending requests, or retries to an upstream cluster to prevent cascading failures.
- Outlier Detection: Passive health checks where Envoy monitors real application request errors (e.g., 5xx status codes or TCP reset errors) and temporarily ejects failing endpoints from the load balancing pool.
4. Installation and Version Management
Installation Methods
Envoy binaries can be installed directly from distribution repositories, run using official container images, or compiled from source.
Option 1: Official Container Image (Recommended for Production / K8s)
The most common approach in cloud-native deployments is using the official Docker image:
# Pull the official distribution image
docker pull envoyproxy/envoy:v1.31.0
# Run Envoy in static mode with a local configuration
docker run -d --name envoy-proxy \
-p 10000:10000 \
-p 9901:9901 \
-v $(pwd)/envoy.yaml:/etc/envoy/envoy.yaml \
envoyproxy/envoy:v1.31.0
Option 2: Debian / Ubuntu Package Installation
Official apt repositories provided by Tetrate or the Envoy project enable direct binary installation:
# Install prerequisites
sudo apt-get update && sudo apt-get install -y apt-transport-https curl gnupg lsb-release
# Add Envoy repository key and channel
curl -sL 'https://getenvoy.io/GPG-KEY-getenvoy' | sudo gpg --dearmor -o /usr/share/keyrings/getenvoy-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/getenvoy-keyring.gpg] https://deb.dl.getenvoy.io/public/deb/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/getenvoy.list
# Update repository metadata and install Envoy
sudo apt-get update && sudo apt-get install -y getenvoy-envoy
Option 3: RHEL / Fedora Package Installation
On RPM-based Linux systems:
# Install yum-utils
sudo dnf install -y dnf-plugins-core
# Add repository and install Envoy
sudo dnf copr enable -y vbatts/getenvoy
sudo dnf install -y getenvoy-envoy
Version and Setup Verification
Verify binary execution and inspect the compiled build version:
# Check installed binary version
envoy --version
# Validate a static configuration file syntax without executing the daemon
envoy --mode validate -c /etc/envoy/envoy.yaml
Expected output for binary version:
envoy version: d362d2e.../1.31.0/Clean/RELEASE/BoringSSL
Default Directory Layout
| Location | Purpose |
|---|---|
/usr/local/bin/envoy or /usr/bin/envoy |
Main executable binary. |
/etc/envoy/envoy.yaml |
Default bootstrap and static configuration file location. |
/var/log/envoy/ |
Default operational access and system logs directory. |
/etc/envoy/certs/ |
Standard directory for TLS certificates and keys. |
5. Configuration Files and Directory Structure
Envoy uses strict, structured YAML schemas defined by Protocol Buffers (@type URLs).
Complete Configuration Reference Table
| Configuration Object | Purpose | Important Details |
|---|---|---|
admin |
Configures the administrative HTTP endpoint socket. | Default access should be restricted to local loopback (127.0.0.1:9901). |
static_resources |
Contains hardcoded definitions for listeners, clusters, and secrets. | Used for standalone setups or to define control plane connection bootstraps. |
dynamic_resources |
Configures xDS discovery server gRPC connection sources. | Defines lds_config, cds_config, and ads parameters. |
listeners |
Defines incoming ports, TLS settings, and filter chains. | Must define filter configurations with explicit typed protobuf extensions. |
clusters |
Defines backend service pools, protocols, and resilience limits. | Specifies load balancer policies (ROUND_ROBIN, LEAST_REQUEST) and DNS lookup modes. |
Comprehensive Static Configuration Example
Below is a complete, operational static envoy.yaml configuration:
admin:
address:
socket_address:
address: 127.0.0.1
port_value: 9901
static_resources:
listeners:
- name: ingress_listener
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
log_format:
json_format:
start_time: "%START_TIME%"
method: "%REQ(:METHOD)%"
path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
protocol: "%PROTOCOL%"
response_code: "%RESPONSE_CODE%"
response_flags: "%RESPONSE_FLAGS%"
bytes_received: "%BYTES_RECEIVED%"
bytes_sent: "%BYTES_SENT%"
duration: "%DURATION%"
upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
upstream_host: "%UPSTREAM_HOST%"
upstream_cluster: "%UPSTREAM_CLUSTER%"
route_config:
name: local_route
virtual_hosts:
- name: backend_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: service_backend
timeout: 15s
retry_policy:
retry_on: "5xx,connect-failure,refused-stream"
num_retries: 3
per_try_timeout: 3s
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: service_backend
connect_timeout: 5s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: service_backend
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: backend.local
port_value: 8080
health_checks:
- timeout: 2s
interval: 5s
unhealthy_threshold: 3
healthy_threshold: 1
http_health_check:
path: "/healthz"
outlier_detection:
consecutive_5xx: 3
interval: 10s
base_ejection_time: 30s
max_ejection_percent: 50
circuit_breakers:
thresholds:
- priority: DEFAULT
max_connections: 1024
max_pending_requests: 100
max_requests: 1024
max_retries: 3
6. Service Management and Common Operations
Systemd Service Unit
When managed by systemd, Envoy runs as a dedicated system service defined at /etc/systemd/system/envoy.service:
[Unit]
Description=Envoy L4/L7 Proxy and Bus
After=network.target remote-fs.target syslog.target
Documentation=https://www.envoyproxy.io/docs/envoy/latest/
[Service]
Type=simple
User=envoy
Group=envoy
ExecStartPre=/usr/local/bin/envoy --mode validate -c /etc/envoy/envoy.yaml
ExecStart=/usr/local/bin/envoy -c /etc/envoy/envoy.yaml --concurrency auto
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65536
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
Essential Administrative Operations
# Reload systemd manager configuration
sudo systemctl daemon-reload
# Enable service auto-start at system boot
sudo systemctl enable envoy
# Start Envoy service
sudo systemctl start envoy
# Inspect current service status and recent journal logs
sudo systemctl status envoy
# Perform a syntax check on updated configuration files
envoy --mode validate -c /etc/envoy/envoy.yaml
# Check running process details and listening sockets
ss -tlnp | grep envoy
# Dynamically trigger graceful termination via administrative HTTP endpoint
curl -X POST http://127.0.0.1:9901/quitquitquit
Warning: Envoy does not automatically re-read configuration files on
SIGHUPwhen running in dynamic xDS mode. Hot-reloading static YAML files requires starting a new Envoy child process using the--restart-epochflag or sending the/quitquitquitendpoint signal to perform controlled pod replacement.
7. Common Administrative Tasks and Workflows
Task 1: Validating and Applying Static Configuration Changes
- Objective: Modify configuration settings (e.g., timeouts or routes) and deploy them safely.
- Prerequisites: Access to the Envoy binary and
/etc/envoy/envoy.yaml. - Step-by-Step Procedure:
- Edit the configuration file with desired route changes:
sudo nano /etc/envoy/envoy.yaml
- Execute syntax validation before initiating service restarts:
envoy --mode validate -c /etc/envoy/envoy.yaml
- Evaluate validation outcome:
- Expected Result: Output prints
configuration '/etc/envoy/envoy.yaml' OK.
- Restart the service:
sudo systemctl restart envoy
- Verify process initialization and listening sockets:
ss -tlnp | grep envoy
curl -s http://127.0.0.1:9901/server_info | grep state
Task 2: Dynamically Adjusting System Logging Levels
- Objective: Enable debug logging for the HTTP subsystem to trace connection issues without restarting Envoy.
- Prerequisites: HTTP access to the Envoy local administrative endpoint (
127.0.0.1:9901). - Step-by-Step Procedure:
- Inspect current logging subsystem levels:
curl -s http://127.0.0.1:9901/logging
- Change the
httplogging component level todebugdynamically:
curl -X POST "http://127.0.0.1:9901/logging?http=debug"
- Tail system logs to analyze request headers and internal state machine events:
journalctl -u envoy -f --output cat
- Revert logging level back to
infoafter completing diagnostic tasks to avoid log volume degradation:
curl -X POST "http://127.0.0.1:9901/logging?http=info"
Task 3: Inspecting Upstream Host Health and Ejection Status
- Objective: Determine why a backend instance is not receiving load-balanced traffic.
- Step-by-Step Procedure:
- Query the cluster administrative interface:
curl -s http://127.0.0.1:9901/clusters | grep "service_backend"
- Filter for health flag markers:
curl -s http://127.0.0.1:9901/clusters | grep health_flags
- Interpret health flags:
/failed_active_hc: Active HTTP/TCP health checks are failing./failed_outlier_check: Endpoint was ejected due to passive outlier detection (consecutive 5xx errors).
- Temporarily force a local health check failure to test failover routing:
curl -X POST http://127.0.0.1:9901/healthcheck/fail
- Restore operational health state:
curl -X POST http://127.0.0.1:9901/healthcheck/ok
8. CLI Command Reference and Cheatsheet
Process Management & Validation Commands
| Command | Purpose | Important Flags/Arguments | Example | Explanation |
|---|---|---|---|---|
envoy |
Starts the Envoy proxy binary. | -c <path>: Specifies config path. |
--mode validate: Validates syntax only. | envoy -c /etc/envoy/envoy.yaml | Starts Envoy using the designated static configuration file. |
| envoy --mode validate | Validates configuration syntax without starting the server. | -c <path>: Path to target YAML configuration file. | envoy --mode validate -c envoy.yaml | Performs schema checks on configuration files and exits with code 0 if valid. |
| envoy --version | Displays compiled binary build details. | None | envoy --version | Displays version strings, build SHA, and BoringSSL feature flags. |
Administrative HTTP Endpoint Reference ([http://127.0.0.1:9901](http://127.0.0.1:9901))
| Endpoint / Command | Purpose | Important Query Parameters | Example | Explanation |
|---|---|---|---|---|
GET /server_info |
Retrieves proxy uptime, state, and version information. | None | curl -s localhost:9901/server_info |
Inspects binary status, state (LIVE, DRAIN), and execution uptime. |
GET /config_dump |
Displays current active configuration across all xDS layers. | ?include_system_version_parse |
`curl -s localhost:9901/config_dump | jq .` |
GET /clusters |
Displays operational backend pools, endpoint IP addresses, and health flags. | None | curl -s localhost:9901/clusters |
Displays connection counts, active host statuses, and ejection state. |
GET /listeners |
Lists bound active listeners and open network sockets. | None | curl -s localhost:9901/listeners |
Verifies socket bindings and operational listener configurations. |
GET /stats |
Emits internal execution counters and metrics. | ?filter=<regex> |
`curl -s localhost:9901/stats | grep rq_5xx` |
GET /stats/prometheus |
Formats internal metrics into standard Prometheus exposition format. | None | curl -s localhost:9901/stats/prometheus |
Scraping target endpoint for Prometheus metric collectors. |
POST /logging |
Modifies active component logging levels at runtime. | <component>=<level> |
curl -X POST localhost:9901/logging?http=debug |
Dynamically raises or lowers log verbosity without process restarts. |
POST /healthcheck/fail |
Forces active health checks to evaluate as failing. | None | curl -X POST localhost:9901/healthcheck/fail |
Simulates backend degradation for failure testing. |
POST /quitquitquit |
Initiates graceful process shutdown. | None | curl -X POST localhost:9901/quitquitquit |
Drains open connections and stops the proxy process gracefully. |
Kubernetes & Istio Debugging Commands
| Command | Purpose | Example | Explanation |
|---|---|---|---|
kubectl exec |
Queries the sidecar admin interface inside a Kubernetes pod. | kubectl exec POD -c istio-proxy -- curl -s localhost:15000/clusters |
Inspects sidecar proxy runtime cluster statuses. |
istioctl proxy-status |
Displays sync status between control plane (istiod) and proxies. |
istioctl proxy-status |
Identifies stale or unsynced Envoy dynamic configurations across the cluster. |
istioctl analyze |
Scans Kubernetes cluster resources for invalid mesh configurations. | istioctl analyze |
Diagnoses misconfigured VirtualServices, DestinationRules, or gateways. |
9. Logs, Monitoring and Diagnostics
Access Log Response Flags
Envoy access logs include standardized HTTP response flag annotations that help identify the exact source of traffic failures:
| Response Flag | Full Name | Detailed Description | Typical Root Cause |
|---|---|---|---|
UH |
No Healthy Upstream Host | Envoy could not route the request because all upstream hosts in the cluster are marked unhealthy or the pool is empty. | Upstream application pods down, health checks failing, misconfigured ports. |
UF |
Upstream Connection Failure | Envoy failed to establish a L4 TCP connection to the chosen upstream host. | Connection refused, network policy blocking traffic, host down, mTLS mismatch. |
NR |
No Route Configured | Request reached Envoy, but no matching route was found in the virtual host route tables. | Missing route rule, incorrect Host header, missing VirtualService binding. |
UO |
Upstream Overflow | Request was rejected because backend circuit breaker capacity thresholds were exceeded. | Connection pool exhausted, high pending request counts, retry storm. |
DC |
Downstream Connection Termination | The downstream client terminated the connection before Envoy could send a response. | Client side timeout, network drop, canceled client requests. |
UT |
Upstream Request Timeout | Upstream backend host exceeded the per-route response timeout threshold. | Application process hanging, database query slowdowns, deadlocks. |
UO / RL |
Rate Limited | Request was rejected by the local or global rate limiting filter. | Client exceeding allocated request rate quotas. |
Core Promotional & Prometheus Metrics
Envoy tracks detailed metrics across three primary scopes: listener, cluster, and control_plane:
# Filter for 5xx responses and connection failures
curl -s http://127.0.0.1:9901/stats | grep -E 'upstream_rq_|cx_|rq_5xx|rq_4xx'
Key metric names to monitor:
envoy_cluster_upstream_cx_active: Number of currently active connections to upstream hosts.envoy_cluster_upstream_cx_connect_fail: Total TCP connection establishment failures to upstream backends.envoy_cluster_upstream_rq_timeout: Count of requests that timed out waiting for an upstream response.envoy_cluster_upstream_rq_pending_overflow: Count of requests rejected due to circuit breaker pending request limits.envoy_cluster_ssl_handshake_failed: Total count of TLS handshake failures when connecting to backends.
10. Troubleshooting Guide and Common Failure Scenarios
Scenario 1: 503 Service Unavailable / No Healthy Upstream (UH)
Symptoms
- Client receives HTTP
503 Service Unavailable. - Access logs show response flag
UH(No Healthy Upstream). - Application metrics report elevated 5xx error rates.
Possible Causes
- All backend host pods/VMs are offline or non-functional.
- Active health checks are failing due to misconfigured paths, ports, or non-200 responses.
- Dynamic Endpoint Discovery (EDS) is not delivering endpoints to Envoy.
Investigation and Diagnosis
- Query the cluster state via the admin endpoint:
curl -s http://127.0.0.1:9901/clusters | grep -A 10 "service_backend"
- Check host health status flags:
curl -s http://127.0.0.1:9901/clusters | grep health_flags
- *If output shows
/failed_active_hc*: Active health check probes are failing. - If output shows empty endpoint lists: Endpoint registration issue.
- Validate active endpoint targets directly using container network tools:
curl -v http://<endpoint-ip>:<port>/healthz
Root Cause Identification
If curl directly to the backend IP returns HTTP 404 or Connection Refused on /healthz, the backend application path or port configuration is mismatched with Envoy’s health check definition.
Resolution
Update the cluster health check path or port in envoy.yaml to match the application specification:
health_checks:
- timeout: 2s
interval: 5s
http_health_check:
path: "/actuator/health" # Fixed path matching backend app
Verification
Confirm host recovery status:
curl -s http://127.0.0.1:9901/clusters | grep "healthy"
Ensure access logs no longer contain the UH flag.
Prevention
Configure active health check initial delays and use proper readiness probes during container deployments.
Scenario 2: Upstream Connection Failure (UF)
Symptoms
- Client receives HTTP
503 Service Unavailableor502 Bad Gateway. - Access log entries display the
UFresponse flag.
Possible Causes
- The backend application service is down or not listening on the expected port.
- Host firewall settings (iptables/nftables) or Kubernetes NetworkPolicies are blocking sidecar egress.
- Protocol/TLS mismatch (e.g., Envoy attempts plaintext HTTP to an HTTPS backend socket).
Investigation and Diagnosis
- Check connection failure counters via the stats endpoint:
curl -s http://127.0.0.1:9901/stats | grep "upstream_cx_connect_fail"
- Test network connectivity directly from inside the Envoy proxy container:
kubectl exec -it <pod-name> -c istio-proxy -- nc -zv <backend-ip> 8080
- Inspect active configuration definitions to verify protocol settings:
curl -s http://127.0.0.1:9901/config_dump | jq '.configs[] | select(.["@type"]=="type.googleapis.com/envoy.admin.v3.ClustersConfigDump")'
Resolution
- Correct destination ports or protocol settings in the cluster configuration.
- Update container NetworkPolicy definitions to permit network traffic between Envoy and the targeted port.
Verification
Re-test TCP connection establishment and issue standard client requests:
curl -i http://localhost:10000/api/v1/resource
Scenario 3: 404 Not Found / No Route Configured (NR)
Symptoms
- Downstream requests return HTTP
404 Not Found. - Access logs show response flag
NR.
Possible Causes
- The incoming request’s HTTP
Hostheader does not match any configureddomainsarray entries in the route table. - Path prefix routing rules do not match the requested path string.
- In service mesh environments, VirtualService bindings are misconfigured or bound to the wrong Gateway.
Investigation and Diagnosis
- Dump dynamic route configurations:
curl -s http://127.0.0.1:9901/config_dump | jq '.configs[] | select(.["@type"]=="type.googleapis.com/envoy.admin.v3.RoutesConfigDump")'
- Compare the requested Host header and path against active VirtualHost configuration domains:
curl -v -H "Host: misconfigured.domain.com" http://localhost:10000/path
Resolution
Ensure the VirtualHost configuration contains valid matching domain wildcard strings or explicit host identifiers:
virtual_hosts:
- name: backend
domains: ["api.example.com", "api.example.internal"] # Ensure Host header matches
routes:
- match:
prefix: "/v1"
route:
cluster: service_backend
Scenario 4: Upstream Circuit Breaker Overflow (UO)
Symptoms
- Downstream client requests fail with HTTP
503 Service Unavailable. - Access logs show response flag
UO. - Metric
upstream_rq_pending_overflowincrements rapidly.
Possible Causes
- Upstream backend slowdown causing requests to queue up past Envoy’s
max_pending_requeststhreshold. - Extreme traffic spikes exceeding configured connection pool limits (
max_connections). - Aggressive retry policies causing retry storms.
Investigation and Diagnosis
- Check active circuit breaker overflow metrics:
curl -s http://127.0.0.1:9901/stats | grep -E 'pending_overflow|retry_overflow'
- Analyze backend response latencies to confirm whether upstream services are overloaded.
Resolution
Adjust cluster circuit breaker capacity thresholds to handle expected spike volume:
circuit_breakers:
thresholds:
- priority: DEFAULT
max_connections: 4096 # Raised from default limits
max_pending_requests: 1024 # Raised to handle queue spikes
max_requests: 4096
11. Security and Best Practices
Securing the Administrative API
The Envoy administrative interface allows dumping full configuration state (including secrets) and initiating process shutdowns.
Critical Operational Guardrail: Never expose the administrative port publicly (
0.0.0.0:9901) without strict external network authentication.
Best practice configuration:
admin:
address:
socket_address:
address: 127.0.0.1 # Bind exclusively to local loopback interface
port_value: 9901
If remote administrative access is required, access it securely using SSH tunneling or Kubernetes port-forwarding:
kubectl port-forward pod/<pod-name> 9901:9901
Least-Privilege Process Execution
Run Envoy as an unprivileged dedicated system user (envoy:envoy) rather than root:
# Verify process ownership
ps aux | grep envoy
TLS and Secret Management via SDS
Avoid writing static TLS certificates and private key files directly to disk configurations. Use SDS (Secret Discovery Service) integrations to fetch and rotate TLS credentials dynamically from memory without restarting the proxy process.
12. Performance and Resource Optimisation
Multi-Threading and Concurrency Controls
Envoy uses an event-driven single-threaded worker model. Worker thread scaling is controlled using the --concurrency CLI option:
# Launch Envoy allocating one worker thread per available CPU core
envoy -c /etc/envoy/envoy.yaml --concurrency auto
# Restrict Envoy execution to explicit thread allocations on shared hosts
envoy -c /etc/envoy/envoy.yaml --concurrency 4
Memory Footprint and Log Optimization
- Disable Verbose Access Logging in Production: High-throughput edge proxies emitting full text logs on every request introduce substantial disk I/O and CPU overhead. Use structured JSON formatting with sampled logging streams or emit metrics directly to Prometheus.
- Connection Pooling: Configure
max_requests_per_connectionto enable upstream HTTP/2 connection reuse and reduce TCP/TLS handshake latency.
13. Backup, Recovery and Maintenance
Backing Up Active Dynamic Configurations
Because Envoy proxies operating in dynamic xDS mode receive configurations over streaming gRPC, the static configuration file contains minimal detail. To back up active operational runtime configurations, dump the live state via the admin API:
# Dump and save complete dynamic runtime state to JSON
curl -s http://127.0.0.1:9901/config_dump > /var/backups/envoy_config_dump_$(date +%F).json
Graceful Draining in Kubernetes Workloads
When updating or terminating Envoy proxy pods, ensure in-flight requests complete cleanly before process shutdown:
- Send a pre-stop drain signal via the admin API:
curl -X POST http://127.0.0.1:9901/healthcheck/fail
- Configure a graceful termination delay (
preStophook) in Kubernetes manifests to give active connections time to finish before sendingSIGKILL.
14. Automation and Integration
CI/CD Pipeline Configuration Validation
Integrate configuration validation into automated CI/CD deployment pipelines to catch syntax errors prior to deployment:
# Example GitHub Actions pipeline validation step
jobs:
validate-envoy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Validate Envoy Configuration
run: |
docker run --rm -v ${{ github.workspace }}/envoy.yaml:/etc/envoy/envoy.yaml \
envoyproxy/envoy:v1.31.0 --mode validate -c /etc/envoy/envoy.yaml
Scraping Telemetry Metrics with Prometheus
To collect operational telemetry metrics, add the following target definition to your prometheus.yml server configuration:
scrape_configs:
- job_name: 'envoy_proxies'
metrics_path: '/stats/prometheus'
static_configs:
- targets: ['10.0.1.50:9901', '10.0.1.51:9901']
15. Practical Labs and Exercises
Lab 1 (Beginner): Deploying and Validating a Static Reverse Proxy
- Objective: Deploy a static Envoy proxy that routes incoming traffic on port 10000 to an external upstream service, and inspect its runtime stats.
- Environment: Local Linux workstation with Docker installed.
Step 1: Create a minimal static configuration file (lab1-envoy.yaml)
admin:
address:
socket_address: { address: 127.0.0.1, port_value: 9901 }
static_resources:
listeners:
- name: lab1_listener
address:
socket_address: { address: 0.0.0.0, port_value: 10000 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: lab1_http
route_config:
virtual_hosts:
- name: echo_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route: { host_rewrite_literal: "postman-echo.com", cluster: echo_cluster }
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: echo_cluster
connect_timeout: 5s
type: LOGICAL_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: echo_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: postman-echo.com, port_value: 80 }
Step 2: Validate the configuration syntax
docker run --rm -v $(pwd)/lab1-envoy.yaml:/etc/envoy/envoy.yaml envoyproxy/envoy:v1.31.0 --mode validate -c /etc/envoy/envoy.yaml
Step 3: Run the proxy and verify traffic routing
docker run -d --name lab1-envoy -p 10000:10000 -p 9901:9901 -v $(pwd)/lab1-envoy.yaml:/etc/envoy/envoy.yaml envoyproxy/envoy:v1.31.0
curl -i http://localhost:10000/get
Step 4: Verify metrics updates
curl -s http://127.0.0.1:9901/stats | grep "lab1_http.downstream_rq_2xx"
Lab 2 (Intermediate): Troubleshooting an Unhealthy Upstream Failure (UH)
- Objective: Diagnose and resolve an HTTP 503
UHfailure using administrative tools. - Environment: Linux environment with
dockerandcurl.
Step 1: Deploy a proxy misconfigured to point to a nonexistent health path
Deploy an Envoy proxy configured to perform HTTP health checks against /nonexistent-health on a test backend.
Step 2: Observe the failure
curl -i http://localhost:10000/api
Expected Output: HTTP/1.1 503 Service Unavailable
Step 3: Diagnose using the admin interface
# Step A: Inspect access flags or query clusters
curl -s http://127.0.0.1:9901/clusters | grep health_flags
# Step B: Locate the failed health check marker
# Output confirms: /failed_active_hc
Step 4: Resolve the issue
Update the health check path in the cluster definition to / and reload the configuration. Verify host status returns to healthy.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| File / Location | Operational Purpose |
|---|---|
/etc/envoy/envoy.yaml |
Primary default static bootstrap configuration file. |
/etc/systemd/system/envoy.service |
Systemd service unit definition file. |
/var/log/envoy/ |
Standard directory for Envoy proxy operational access logs. |
Key Administrative Endpoints ([http://127.0.0.1:9901](http://127.0.0.1:9901))
| Path | Purpose |
|---|---|
/server_info |
Shows proxy state, uptime, and build details. |
/config_dump |
Displays full active configuration across dynamic xDS layers. |
/clusters |
Lists cluster health status, host IP endpoints, and flags. |
/stats/prometheus |
Exposes performance metrics formatted for Prometheus scraping. |
/logging?http=debug |
Dynamically updates component log levels without process restarts. |
Essential CLI Commands
| Operational Task | Command Command String |
|---|---|
| Validate configuration syntax | envoy --mode validate -c /etc/envoy/envoy.yaml |
| Start Envoy with configuration file | envoy -c /etc/envoy/envoy.yaml |
| Check binary build version | envoy --version |
| Verify open listening ports | `ss -tlnp |
| Initiate graceful process termination | curl -X POST [http://127.0.0.1:9901/quitquitquit](http://127.0.0.1:9901/quitquitquit) |
Access Log Response Flags Quick Reference
| Flag | Meaning | Primary Action Item |
|---|---|---|
UH |
No Healthy Upstream | Check backend process status and verify health check paths. |
UF |
Upstream Connection Failed | Check target port, firewall rule, and mTLS configuration. |
NR |
No Route Matched | Verify request Host header and path prefix routing rules. |
UO |
Upstream Overflow | Increase circuit breaker max connection/pending request thresholds. |
DC |
Downstream Connection Terminated | Check client-side application request timeouts. |
Default Ports and Protocols
| Port | Protocol | Usage / Description |
|---|---|---|
9901 |
HTTP | Local Envoy administrative management interface. |
15000 |
HTTP | Default admin interface port inside Istio sidecars. |
15001 |
TCP | Istio outbound sidecar traffic capture listener port. |
15006 |
TCP | Istio inbound sidecar traffic capture listener port. |
10000 |
HTTP/TCP | Standard default application proxy listener port. |
17. Further Reading and References
- Official Envoy Proxy Documentation: https://www.envoyproxy.io/docs
- Envoy Configuration Architecture Overview: https://www.envoyproxy.io/docs/envoy/latest/configuration/configuration
- Envoy Admin Interface Documentation: https://www.envoyproxy.io/docs/envoy/latest/operations/admin
- xDS Transport Protocol Specification: https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol