Domain 7Web & Application LayerINTERMEDIATE

Envoy Proxy Operational Handbook: Architecture, Configuration, xDS, and Troubleshooting

A comprehensive guide to Envoy Proxy covering data path architecture, filter chains, xDS dynamic discovery, administrative tools, troubleshooting workflows, and operational best practices.

Domain: Domain 7: Web & Application Layer•Updated: 2026-11-03
#linux#sysadmin#envoy#networking#sre

1. Overview

Envoy is an open-source, high-performance L4/L7 edge and service proxy designed for cloud-native applications. Developed originally by Lyft, Envoy sits directly in the data path between incoming client requests and backend application services. It handles network traffic by terminating client connections, parsing request protocols (such as HTTP/1.1, HTTP/2, gRPC, and TCP), enforcing routing logic, load-balancing traffic across backend instances, enforcing resilience policies, managing Transport Layer Security (TLS), and emitting telemetry metrics and access logs.

  +----------------+         +-------------------+         +-----------------+
  |                |  HTTP   |    Envoy Proxy    |  gRPC   |                 |
  |  Client App /  |-------->| (Terminates TLS,  |-------->| Upstream Host A |
  | External User  |  HTTPS  |   Applies Filters |         +-----------------+
  |                |         |    & Routing)     |         | Upstream Host B |
  +----------------+         +-------------------+         +-----------------+

The Problem Envoy Solves

In modern microservice architectures, infrastructure concerns such as protocol parsing, service discovery, client-side load balancing, TLS encryption, retries, circuit breaking, rate limiting, and distributed tracing are often implemented across multiple application codebases. This leads to fragmented language libraries, inconsistent retry semantics, difficult maintenance, and blind spots in observability.

Envoy abstracts these network mechanisms away from application code. By acting as a universal network proxy, Envoy provides a standard networking layer across all services, regardless of the underlying programming language or runtime environment.

Primary Deployment Models and Use Cases

Envoy is commonly deployed across four primary architectural roles:

  1. Kubernetes Service Mesh Sidecar: Runs alongside application containers within the same pod (e.g., Istio) to manage inter-service East-West traffic, enforce mutual TLS (mTLS), and handle dynamic discovery.
  2. Edge Ingress / API Gateway: Functions as the front proxy accepting external North-South client traffic, performing authentication, rate-limiting, path routing, and protocol translation.
  3. Internal Load Balancer: Provides advanced load balancing (such as gRPC load balancing, ring hash, and least-request routing) with active health checking and outlier detection across internal host clusters.
  4. Protocol Bridge & Origination: Translates HTTP/1.1 or REST requests into gRPC, originates upstream TLS connections, or proxies raw TCP streams.

Envoy vs. Nginx, HAProxy, and Traefik

Selecting the appropriate reverse proxy depends on configuration models, target operational environments, and control plane requirements:

Feature / Capability Envoy Proxy Nginx HAProxy Traefik
Primary Design Target Dynamic cloud-native, microservices, service mesh Web serving, static content, edge reverse proxy High-performance L4/L7 load balancing Dynamic container and Kubernetes ingress
Configuration Model API-driven dynamic control plane (xDS) & static YAML File-based static configuration files File-based static configuration files Dynamic configuration via provider labels/CRDs
gRPC & HTTP/2 Support First-class, bi-directional streaming natively supported Module-based / HTTP/2 & basic gRPC support Strong L4/L7 gRPC support Native HTTP/2 & gRPC support
Dynamic Configuration Complete zero-downtime streaming updates without process reload Master process reload required for config changes Runtime socket configuration for endpoints; reload for routes Automatic reload on container engine events
Observability & Telemetry Granular per-route/cluster Prometheus stats, native tracing Basic status page; stub_status / extended metrics in Nginx Plus Detailed stats page and runtime socket statistics Built-in web dashboard and basic Prometheus metrics
Extensibility C++ native modules, WebAssembly (WASM), and Lua filters C modules, OpenResty (Lua) Lua scripts Go plugins, WebAssembly

2. Architecture and Mental Model

To effectively operate and troubleshoot Envoy, systems engineers must understand its internal processing pipeline and request execution model.

Incoming Connection 
        │
        ▼
┌───────────────┐
│   Listener    │  Binds socket (e.g., 0.0.0.0:10000)
└───────┬───────┘
        │
        ▼
┌───────────────┐
│ Filter Chain  │  Network Filters (TLS Termination, Byte Inspection)
└───────┬───────┘
        │
        ▼
┌───────────────┐
│ HttpConnMgr   │  L7 HTTP Parsing, Header Inspection, Tracing
└───────┬───────┘
        │
        ▼
┌───────────────┐
│ Route Match   │  Matches Host, Path, Query Parameters to Target Cluster
└───────┬───────┘
        │
        ▼
┌───────────────┐
│    Cluster    │  Selects Upstream Service Pool, Applies LB Policy & Circuit Breakers
└───────┬───────┘
        │
        ▼
┌───────────────┐
│   Endpoint    │  Resolves Target IP:Port Host Endpoint
└───────┬───────┘
        │
        ▼
Forward Request to Upstream Host

The Request Lifecycle

  1. Client Connection Establishment: A client opens a TCP socket to an Envoy Listener listening on a specified network address and port (e.g., 0.0.0.0:10000).
  2. Filter Chain Execution: The connection passes through a sequence of network (L4) filters configured within the listener. If TLS is configured, the TLS inspector filter terminates TLS or verifies mTLS certificates.
  3. HTTP Connection Manager (HCM) Parsing: For HTTP/gRPC traffic, the connection is handed off to the envoy.filters.network.http_connection_manager filter. The HCM parses HTTP/1.1 or HTTP/2 frames, processes headers, injects tracing contexts, and evaluates L7 HTTP filters (such as authentication or rate limiting).
  4. Route Matching: The HCM evaluates the request’s HTTP Host/Authority headers, path prefix, query parameters, or HTTP headers against a defined Route Configuration. The matching route points to a target Cluster.
  5. Cluster Selection & Load Balancing: Envoy identifies the target cluster representing the group of backend instances. It applies load balancing algorithms (e.g., Round Robin, Least Request, Ring Hash) to select an available Endpoint.
  6. Resilience & Circuit Breaker Check: Envoy verifies whether cluster-level circuit breaker thresholds (such as pending request limits or maximum connection counts) have been exceeded. It also evaluates whether the selected endpoint is healthy based on active health checks or passive outlier detection.
  7. Upstream Transmission & Response Routing: Envoy forwards the request to the upstream endpoint IP:port. When the response returns, Envoy passes bytes back through the filter chain to the downstream client, updates internal statistical counters, and emits structured access log records.

Static vs. Dynamic Configuration (xDS Protocol)

Envoy can be configured using a static YAML bootstrap file (envoy.yaml) or dynamically via gRPC/REST APIs known as xDS (x Discovery Service) APIs.

                   +------------------------+
                   |  Control Plane Server  |
                   | (Istio, Envoy Gateway) |
                   +-----------+------------+
                               |
                               | gRPC Stream (xDS)
                               v
+---------------------------------------------------------------+
|                         Envoy Proxy                           |
|                                                               |
|  [ LDS ]       [ RDS ]       [ CDS ]       [ EDS ]    [ SDS ] |
| Listener  -->  Route   -->  Cluster --> Endpoint -> Secret  |
| Discovery     Discovery    Discovery   Discovery   Discovery|
+---------------------------------------------------------------+
  • Static Configuration: Entire configurations (listeners, routes, clusters, endpoints) are specified directly within the local configuration file. Static configurations are simple to test and debug locally, but require process restarts or hot-reloads to update endpoints or routes.
  • Dynamic Configuration (xDS): A lightweight static bootstrap configuration instructs Envoy to establish a gRPC stream to an external control plane management server (such as istiod or an Envoy Gateway controller). The control plane streams incremental configuration updates without dropping connections or restarting the binary:
  • LDS (Listener Discovery Service): Pushes dynamic updates for listening sockets and filter chains.
  • RDS (Route Discovery Service): Pushes dynamic route tables, virtual host domains, and path rules.
  • CDS (Cluster Discovery Service): Pushes dynamic definitions of upstream service pools, connection pool settings, and circuit breaking limits.
  • EDS (Endpoint Discovery Service): Pushes dynamic lists of operational backend IP addresses and ports. This updates rapidly as pods or hosts autoscale.
  • SDS (Secret Discovery Service): Delivers TLS certificates, private keys, and CA verification roots dynamically without writing secrets to disk.
  • ADS (Aggregated Discovery Service): Encapsulates all xDS updates over a single unified gRPC multiplexed stream to maintain strict dependency ordering (e.g., ensuring a cluster exists before endpoints reference it).

3. Core Concepts and Terminology

Understanding Envoy’s fundamental abstractions is necessary to design valid configurations and debug runtime issues.

Core Abstractions

  • Listener: An operational network socket bound to an IP address and port that accepts incoming connections from downstream clients.
  • Filter Chain: A sequential pipeline of network (L4) and HTTP (L7) filters attached to a listener that inspects, modifies, or routes incoming connection bytes.
  • HTTP Connection Manager (HCM): The primary network filter (envoy.filters.network.http_connection_manager) that converts raw stream bytes into HTTP requests and responses, manages header manipulations, and routes traffic.
  • Route Configuration: A set of routing rules within the HCM that matches incoming requests (by path, host, or header) and directs them to a backend cluster.
  • Virtual Host: A sub-component of a route configuration defining domain matching rules (domains: ["api.example.com"]) and associated route definitions.
  • Cluster: A logical grouping of upstream backend hosts offering identical service functionality, configured with specific load balancing protocols, health checks, and connection pools.
  • Endpoint: A specific IP address and port representing an individual service instance belonging to a cluster.
  • Upstream: A backend server or service that receives requests forwarded by Envoy.
  • Downstream: A client entity that connects to Envoy to issue requests.
  • Circuit Breaker: A protective runtime mechanism that limits connections, pending requests, or retries to an upstream cluster to prevent cascading failures.
  • Outlier Detection: Passive health checks where Envoy monitors real application request errors (e.g., 5xx status codes or TCP reset errors) and temporarily ejects failing endpoints from the load balancing pool.

4. Installation and Version Management

Installation Methods

Envoy binaries can be installed directly from distribution repositories, run using official container images, or compiled from source.

The most common approach in cloud-native deployments is using the official Docker image:

# Pull the official distribution image
docker pull envoyproxy/envoy:v1.31.0

# Run Envoy in static mode with a local configuration
docker run -d --name envoy-proxy \
  -p 10000:10000 \
  -p 9901:9901 \
  -v $(pwd)/envoy.yaml:/etc/envoy/envoy.yaml \
  envoyproxy/envoy:v1.31.0

Option 2: Debian / Ubuntu Package Installation

Official apt repositories provided by Tetrate or the Envoy project enable direct binary installation:

# Install prerequisites
sudo apt-get update && sudo apt-get install -y apt-transport-https curl gnupg lsb-release

# Add Envoy repository key and channel
curl -sL 'https://getenvoy.io/GPG-KEY-getenvoy' | sudo gpg --dearmor -o /usr/share/keyrings/getenvoy-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/getenvoy-keyring.gpg] https://deb.dl.getenvoy.io/public/deb/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/getenvoy.list

# Update repository metadata and install Envoy
sudo apt-get update && sudo apt-get install -y getenvoy-envoy

Option 3: RHEL / Fedora Package Installation

On RPM-based Linux systems:

# Install yum-utils
sudo dnf install -y dnf-plugins-core

# Add repository and install Envoy
sudo dnf copr enable -y vbatts/getenvoy
sudo dnf install -y getenvoy-envoy

Version and Setup Verification

Verify binary execution and inspect the compiled build version:

# Check installed binary version
envoy --version

# Validate a static configuration file syntax without executing the daemon
envoy --mode validate -c /etc/envoy/envoy.yaml

Expected output for binary version:

envoy  version: d362d2e.../1.31.0/Clean/RELEASE/BoringSSL

Default Directory Layout

Location Purpose
/usr/local/bin/envoy or /usr/bin/envoy Main executable binary.
/etc/envoy/envoy.yaml Default bootstrap and static configuration file location.
/var/log/envoy/ Default operational access and system logs directory.
/etc/envoy/certs/ Standard directory for TLS certificates and keys.

5. Configuration Files and Directory Structure

Envoy uses strict, structured YAML schemas defined by Protocol Buffers (@type URLs).

Complete Configuration Reference Table

Configuration Object Purpose Important Details
admin Configures the administrative HTTP endpoint socket. Default access should be restricted to local loopback (127.0.0.1:9901).
static_resources Contains hardcoded definitions for listeners, clusters, and secrets. Used for standalone setups or to define control plane connection bootstraps.
dynamic_resources Configures xDS discovery server gRPC connection sources. Defines lds_config, cds_config, and ads parameters.
listeners Defines incoming ports, TLS settings, and filter chains. Must define filter configurations with explicit typed protobuf extensions.
clusters Defines backend service pools, protocols, and resilience limits. Specifies load balancer policies (ROUND_ROBIN, LEAST_REQUEST) and DNS lookup modes.

Comprehensive Static Configuration Example

Below is a complete, operational static envoy.yaml configuration:

admin:
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 9901

static_resources:
  listeners:
  - name: ingress_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              log_format:
                json_format:
                  start_time: "%START_TIME%"
                  method: "%REQ(:METHOD)%"
                  path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
                  protocol: "%PROTOCOL%"
                  response_code: "%RESPONSE_CODE%"
                  response_flags: "%RESPONSE_FLAGS%"
                  bytes_received: "%BYTES_RECEIVED%"
                  bytes_sent: "%BYTES_SENT%"
                  duration: "%DURATION%"
                  upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
                  upstream_host: "%UPSTREAM_HOST%"
                  upstream_cluster: "%UPSTREAM_CLUSTER%"
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: service_backend
                  timeout: 15s
                  retry_policy:
                    retry_on: "5xx,connect-failure,refused-stream"
                    num_retries: 3
                    per_try_timeout: 3s
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
  - name: service_backend
    connect_timeout: 5s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    dns_lookup_family: V4_ONLY
    load_assignment:
      cluster_name: service_backend
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend.local
                port_value: 8080
    health_checks:
    - timeout: 2s
      interval: 5s
      unhealthy_threshold: 3
      healthy_threshold: 1
      http_health_check:
        path: "/healthz"
    outlier_detection:
      consecutive_5xx: 3
      interval: 10s
      base_ejection_time: 30s
      max_ejection_percent: 50
    circuit_breakers:
      thresholds:
      - priority: DEFAULT
        max_connections: 1024
        max_pending_requests: 100
        max_requests: 1024
        max_retries: 3

6. Service Management and Common Operations

Systemd Service Unit

When managed by systemd, Envoy runs as a dedicated system service defined at /etc/systemd/system/envoy.service:

[Unit]
Description=Envoy L4/L7 Proxy and Bus
After=network.target remote-fs.target syslog.target
Documentation=https://www.envoyproxy.io/docs/envoy/latest/

[Service]
Type=simple
User=envoy
Group=envoy
ExecStartPre=/usr/local/bin/envoy --mode validate -c /etc/envoy/envoy.yaml
ExecStart=/usr/local/bin/envoy -c /etc/envoy/envoy.yaml --concurrency auto
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65536
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Essential Administrative Operations

# Reload systemd manager configuration
sudo systemctl daemon-reload

# Enable service auto-start at system boot
sudo systemctl enable envoy

# Start Envoy service
sudo systemctl start envoy

# Inspect current service status and recent journal logs
sudo systemctl status envoy

# Perform a syntax check on updated configuration files
envoy --mode validate -c /etc/envoy/envoy.yaml

# Check running process details and listening sockets
ss -tlnp | grep envoy

# Dynamically trigger graceful termination via administrative HTTP endpoint
curl -X POST http://127.0.0.1:9901/quitquitquit

Warning: Envoy does not automatically re-read configuration files on SIGHUP when running in dynamic xDS mode. Hot-reloading static YAML files requires starting a new Envoy child process using the --restart-epoch flag or sending the /quitquitquit endpoint signal to perform controlled pod replacement.


7. Common Administrative Tasks and Workflows

Task 1: Validating and Applying Static Configuration Changes

  • Objective: Modify configuration settings (e.g., timeouts or routes) and deploy them safely.
  • Prerequisites: Access to the Envoy binary and /etc/envoy/envoy.yaml.
  • Step-by-Step Procedure:
  1. Edit the configuration file with desired route changes:
sudo nano /etc/envoy/envoy.yaml
  1. Execute syntax validation before initiating service restarts:
envoy --mode validate -c /etc/envoy/envoy.yaml
  1. Evaluate validation outcome:
  • Expected Result: Output prints configuration '/etc/envoy/envoy.yaml' OK.
  1. Restart the service:
sudo systemctl restart envoy
  1. Verify process initialization and listening sockets:
ss -tlnp | grep envoy
curl -s http://127.0.0.1:9901/server_info | grep state

Task 2: Dynamically Adjusting System Logging Levels

  • Objective: Enable debug logging for the HTTP subsystem to trace connection issues without restarting Envoy.
  • Prerequisites: HTTP access to the Envoy local administrative endpoint (127.0.0.1:9901).
  • Step-by-Step Procedure:
  1. Inspect current logging subsystem levels:
curl -s http://127.0.0.1:9901/logging
  1. Change the http logging component level to debug dynamically:
curl -X POST "http://127.0.0.1:9901/logging?http=debug"
  1. Tail system logs to analyze request headers and internal state machine events:
journalctl -u envoy -f --output cat
  1. Revert logging level back to info after completing diagnostic tasks to avoid log volume degradation:
curl -X POST "http://127.0.0.1:9901/logging?http=info"

Task 3: Inspecting Upstream Host Health and Ejection Status

  • Objective: Determine why a backend instance is not receiving load-balanced traffic.
  • Step-by-Step Procedure:
  1. Query the cluster administrative interface:
curl -s http://127.0.0.1:9901/clusters | grep "service_backend"
  1. Filter for health flag markers:
curl -s http://127.0.0.1:9901/clusters | grep health_flags
  1. Interpret health flags:
  • /failed_active_hc: Active HTTP/TCP health checks are failing.
  • /failed_outlier_check: Endpoint was ejected due to passive outlier detection (consecutive 5xx errors).
  1. Temporarily force a local health check failure to test failover routing:
curl -X POST http://127.0.0.1:9901/healthcheck/fail
  1. Restore operational health state:
curl -X POST http://127.0.0.1:9901/healthcheck/ok

8. CLI Command Reference and Cheatsheet

Process Management & Validation Commands

Command Purpose Important Flags/Arguments Example Explanation
envoy Starts the Envoy proxy binary. -c <path>: Specifies config path.


--mode validate: Validates syntax only. | envoy -c /etc/envoy/envoy.yaml | Starts Envoy using the designated static configuration file. | | envoy --mode validate | Validates configuration syntax without starting the server. | -c <path>: Path to target YAML configuration file. | envoy --mode validate -c envoy.yaml | Performs schema checks on configuration files and exits with code 0 if valid. | | envoy --version | Displays compiled binary build details. | None | envoy --version | Displays version strings, build SHA, and BoringSSL feature flags. |


Administrative HTTP Endpoint Reference ([http://127.0.0.1:9901](http://127.0.0.1:9901))

Endpoint / Command Purpose Important Query Parameters Example Explanation
GET /server_info Retrieves proxy uptime, state, and version information. None curl -s localhost:9901/server_info Inspects binary status, state (LIVE, DRAIN), and execution uptime.
GET /config_dump Displays current active configuration across all xDS layers. ?include_system_version_parse `curl -s localhost:9901/config_dump jq .`
GET /clusters Displays operational backend pools, endpoint IP addresses, and health flags. None curl -s localhost:9901/clusters Displays connection counts, active host statuses, and ejection state.
GET /listeners Lists bound active listeners and open network sockets. None curl -s localhost:9901/listeners Verifies socket bindings and operational listener configurations.
GET /stats Emits internal execution counters and metrics. ?filter=<regex> `curl -s localhost:9901/stats grep rq_5xx`
GET /stats/prometheus Formats internal metrics into standard Prometheus exposition format. None curl -s localhost:9901/stats/prometheus Scraping target endpoint for Prometheus metric collectors.
POST /logging Modifies active component logging levels at runtime. <component>=<level> curl -X POST localhost:9901/logging?http=debug Dynamically raises or lowers log verbosity without process restarts.
POST /healthcheck/fail Forces active health checks to evaluate as failing. None curl -X POST localhost:9901/healthcheck/fail Simulates backend degradation for failure testing.
POST /quitquitquit Initiates graceful process shutdown. None curl -X POST localhost:9901/quitquitquit Drains open connections and stops the proxy process gracefully.

Kubernetes & Istio Debugging Commands

Command Purpose Example Explanation
kubectl exec Queries the sidecar admin interface inside a Kubernetes pod. kubectl exec POD -c istio-proxy -- curl -s localhost:15000/clusters Inspects sidecar proxy runtime cluster statuses.
istioctl proxy-status Displays sync status between control plane (istiod) and proxies. istioctl proxy-status Identifies stale or unsynced Envoy dynamic configurations across the cluster.
istioctl analyze Scans Kubernetes cluster resources for invalid mesh configurations. istioctl analyze Diagnoses misconfigured VirtualServices, DestinationRules, or gateways.

9. Logs, Monitoring and Diagnostics

Access Log Response Flags

Envoy access logs include standardized HTTP response flag annotations that help identify the exact source of traffic failures:

Response Flag Full Name Detailed Description Typical Root Cause
UH No Healthy Upstream Host Envoy could not route the request because all upstream hosts in the cluster are marked unhealthy or the pool is empty. Upstream application pods down, health checks failing, misconfigured ports.
UF Upstream Connection Failure Envoy failed to establish a L4 TCP connection to the chosen upstream host. Connection refused, network policy blocking traffic, host down, mTLS mismatch.
NR No Route Configured Request reached Envoy, but no matching route was found in the virtual host route tables. Missing route rule, incorrect Host header, missing VirtualService binding.
UO Upstream Overflow Request was rejected because backend circuit breaker capacity thresholds were exceeded. Connection pool exhausted, high pending request counts, retry storm.
DC Downstream Connection Termination The downstream client terminated the connection before Envoy could send a response. Client side timeout, network drop, canceled client requests.
UT Upstream Request Timeout Upstream backend host exceeded the per-route response timeout threshold. Application process hanging, database query slowdowns, deadlocks.
UO / RL Rate Limited Request was rejected by the local or global rate limiting filter. Client exceeding allocated request rate quotas.

Core Promotional & Prometheus Metrics

Envoy tracks detailed metrics across three primary scopes: listener, cluster, and control_plane:

# Filter for 5xx responses and connection failures
curl -s http://127.0.0.1:9901/stats | grep -E 'upstream_rq_|cx_|rq_5xx|rq_4xx'

Key metric names to monitor:

  • envoy_cluster_upstream_cx_active: Number of currently active connections to upstream hosts.
  • envoy_cluster_upstream_cx_connect_fail: Total TCP connection establishment failures to upstream backends.
  • envoy_cluster_upstream_rq_timeout: Count of requests that timed out waiting for an upstream response.
  • envoy_cluster_upstream_rq_pending_overflow: Count of requests rejected due to circuit breaker pending request limits.
  • envoy_cluster_ssl_handshake_failed: Total count of TLS handshake failures when connecting to backends.

10. Troubleshooting Guide and Common Failure Scenarios

Scenario 1: 503 Service Unavailable / No Healthy Upstream (UH)

Symptoms

  • Client receives HTTP 503 Service Unavailable.
  • Access logs show response flag UH (No Healthy Upstream).
  • Application metrics report elevated 5xx error rates.

Possible Causes

  1. All backend host pods/VMs are offline or non-functional.
  2. Active health checks are failing due to misconfigured paths, ports, or non-200 responses.
  3. Dynamic Endpoint Discovery (EDS) is not delivering endpoints to Envoy.

Investigation and Diagnosis

  1. Query the cluster state via the admin endpoint:
curl -s http://127.0.0.1:9901/clusters | grep -A 10 "service_backend"
  1. Check host health status flags:
curl -s http://127.0.0.1:9901/clusters | grep health_flags
  • *If output shows /failed_active_hc*: Active health check probes are failing.
  • If output shows empty endpoint lists: Endpoint registration issue.
  1. Validate active endpoint targets directly using container network tools:
curl -v http://<endpoint-ip>:<port>/healthz

Root Cause Identification

If curl directly to the backend IP returns HTTP 404 or Connection Refused on /healthz, the backend application path or port configuration is mismatched with Envoy’s health check definition.

Resolution

Update the cluster health check path or port in envoy.yaml to match the application specification:

health_checks:
- timeout: 2s
  interval: 5s
  http_health_check:
    path: "/actuator/health"  # Fixed path matching backend app

Verification

Confirm host recovery status:

curl -s http://127.0.0.1:9901/clusters | grep "healthy"

Ensure access logs no longer contain the UH flag.

Prevention

Configure active health check initial delays and use proper readiness probes during container deployments.


Scenario 2: Upstream Connection Failure (UF)

Symptoms

  • Client receives HTTP 503 Service Unavailable or 502 Bad Gateway.
  • Access log entries display the UF response flag.

Possible Causes

  1. The backend application service is down or not listening on the expected port.
  2. Host firewall settings (iptables/nftables) or Kubernetes NetworkPolicies are blocking sidecar egress.
  3. Protocol/TLS mismatch (e.g., Envoy attempts plaintext HTTP to an HTTPS backend socket).

Investigation and Diagnosis

  1. Check connection failure counters via the stats endpoint:
curl -s http://127.0.0.1:9901/stats | grep "upstream_cx_connect_fail"
  1. Test network connectivity directly from inside the Envoy proxy container:
kubectl exec -it <pod-name> -c istio-proxy -- nc -zv <backend-ip> 8080
  1. Inspect active configuration definitions to verify protocol settings:
curl -s http://127.0.0.1:9901/config_dump | jq '.configs[] | select(.["@type"]=="type.googleapis.com/envoy.admin.v3.ClustersConfigDump")'

Resolution

  • Correct destination ports or protocol settings in the cluster configuration.
  • Update container NetworkPolicy definitions to permit network traffic between Envoy and the targeted port.

Verification

Re-test TCP connection establishment and issue standard client requests:

curl -i http://localhost:10000/api/v1/resource

Scenario 3: 404 Not Found / No Route Configured (NR)

Symptoms

  • Downstream requests return HTTP 404 Not Found.
  • Access logs show response flag NR.

Possible Causes

  1. The incoming request’s HTTP Host header does not match any configured domains array entries in the route table.
  2. Path prefix routing rules do not match the requested path string.
  3. In service mesh environments, VirtualService bindings are misconfigured or bound to the wrong Gateway.

Investigation and Diagnosis

  1. Dump dynamic route configurations:
curl -s http://127.0.0.1:9901/config_dump | jq '.configs[] | select(.["@type"]=="type.googleapis.com/envoy.admin.v3.RoutesConfigDump")'
  1. Compare the requested Host header and path against active VirtualHost configuration domains:
curl -v -H "Host: misconfigured.domain.com" http://localhost:10000/path

Resolution

Ensure the VirtualHost configuration contains valid matching domain wildcard strings or explicit host identifiers:

virtual_hosts:
- name: backend
  domains: ["api.example.com", "api.example.internal"]  # Ensure Host header matches
  routes:
  - match:
      prefix: "/v1"
    route:
      cluster: service_backend

Scenario 4: Upstream Circuit Breaker Overflow (UO)

Symptoms

  • Downstream client requests fail with HTTP 503 Service Unavailable.
  • Access logs show response flag UO.
  • Metric upstream_rq_pending_overflow increments rapidly.

Possible Causes

  1. Upstream backend slowdown causing requests to queue up past Envoy’s max_pending_requests threshold.
  2. Extreme traffic spikes exceeding configured connection pool limits (max_connections).
  3. Aggressive retry policies causing retry storms.

Investigation and Diagnosis

  1. Check active circuit breaker overflow metrics:
curl -s http://127.0.0.1:9901/stats | grep -E 'pending_overflow|retry_overflow'
  1. Analyze backend response latencies to confirm whether upstream services are overloaded.

Resolution

Adjust cluster circuit breaker capacity thresholds to handle expected spike volume:

circuit_breakers:
  thresholds:
  - priority: DEFAULT
    max_connections: 4096       # Raised from default limits
    max_pending_requests: 1024  # Raised to handle queue spikes
    max_requests: 4096

11. Security and Best Practices

Securing the Administrative API

The Envoy administrative interface allows dumping full configuration state (including secrets) and initiating process shutdowns.

Critical Operational Guardrail: Never expose the administrative port publicly (0.0.0.0:9901) without strict external network authentication.

Best practice configuration:

admin:
  address:
    socket_address:
      address: 127.0.0.1  # Bind exclusively to local loopback interface
      port_value: 9901

If remote administrative access is required, access it securely using SSH tunneling or Kubernetes port-forwarding:

kubectl port-forward pod/<pod-name> 9901:9901

Least-Privilege Process Execution

Run Envoy as an unprivileged dedicated system user (envoy:envoy) rather than root:

# Verify process ownership
ps aux | grep envoy

TLS and Secret Management via SDS

Avoid writing static TLS certificates and private key files directly to disk configurations. Use SDS (Secret Discovery Service) integrations to fetch and rotate TLS credentials dynamically from memory without restarting the proxy process.


12. Performance and Resource Optimisation

Multi-Threading and Concurrency Controls

Envoy uses an event-driven single-threaded worker model. Worker thread scaling is controlled using the --concurrency CLI option:

# Launch Envoy allocating one worker thread per available CPU core
envoy -c /etc/envoy/envoy.yaml --concurrency auto

# Restrict Envoy execution to explicit thread allocations on shared hosts
envoy -c /etc/envoy/envoy.yaml --concurrency 4

Memory Footprint and Log Optimization

  • Disable Verbose Access Logging in Production: High-throughput edge proxies emitting full text logs on every request introduce substantial disk I/O and CPU overhead. Use structured JSON formatting with sampled logging streams or emit metrics directly to Prometheus.
  • Connection Pooling: Configure max_requests_per_connection to enable upstream HTTP/2 connection reuse and reduce TCP/TLS handshake latency.

13. Backup, Recovery and Maintenance

Backing Up Active Dynamic Configurations

Because Envoy proxies operating in dynamic xDS mode receive configurations over streaming gRPC, the static configuration file contains minimal detail. To back up active operational runtime configurations, dump the live state via the admin API:

# Dump and save complete dynamic runtime state to JSON
curl -s http://127.0.0.1:9901/config_dump > /var/backups/envoy_config_dump_$(date +%F).json

Graceful Draining in Kubernetes Workloads

When updating or terminating Envoy proxy pods, ensure in-flight requests complete cleanly before process shutdown:

  1. Send a pre-stop drain signal via the admin API:
curl -X POST http://127.0.0.1:9901/healthcheck/fail
  1. Configure a graceful termination delay (preStop hook) in Kubernetes manifests to give active connections time to finish before sending SIGKILL.

14. Automation and Integration

CI/CD Pipeline Configuration Validation

Integrate configuration validation into automated CI/CD deployment pipelines to catch syntax errors prior to deployment:

# Example GitHub Actions pipeline validation step
jobs:
  validate-envoy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Validate Envoy Configuration
      run: |
        docker run --rm -v ${{ github.workspace }}/envoy.yaml:/etc/envoy/envoy.yaml \
          envoyproxy/envoy:v1.31.0 --mode validate -c /etc/envoy/envoy.yaml

Scraping Telemetry Metrics with Prometheus

To collect operational telemetry metrics, add the following target definition to your prometheus.yml server configuration:

scrape_configs:
  - job_name: 'envoy_proxies'
    metrics_path: '/stats/prometheus'
    static_configs:
      - targets: ['10.0.1.50:9901', '10.0.1.51:9901']

15. Practical Labs and Exercises

Lab 1 (Beginner): Deploying and Validating a Static Reverse Proxy

  • Objective: Deploy a static Envoy proxy that routes incoming traffic on port 10000 to an external upstream service, and inspect its runtime stats.
  • Environment: Local Linux workstation with Docker installed.

Step 1: Create a minimal static configuration file (lab1-envoy.yaml)

admin:
  address:
    socket_address: { address: 127.0.0.1, port_value: 9901 }
static_resources:
  listeners:
  - name: lab1_listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 10000 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: lab1_http
          route_config:
            virtual_hosts:
            - name: echo_service
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route: { host_rewrite_literal: "postman-echo.com", cluster: echo_cluster }
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: echo_cluster
    connect_timeout: 5s
    type: LOGICAL_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: echo_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: postman-echo.com, port_value: 80 }

Step 2: Validate the configuration syntax

docker run --rm -v $(pwd)/lab1-envoy.yaml:/etc/envoy/envoy.yaml envoyproxy/envoy:v1.31.0 --mode validate -c /etc/envoy/envoy.yaml

Step 3: Run the proxy and verify traffic routing

docker run -d --name lab1-envoy -p 10000:10000 -p 9901:9901 -v $(pwd)/lab1-envoy.yaml:/etc/envoy/envoy.yaml envoyproxy/envoy:v1.31.0
curl -i http://localhost:10000/get

Step 4: Verify metrics updates

curl -s http://127.0.0.1:9901/stats | grep "lab1_http.downstream_rq_2xx"

Lab 2 (Intermediate): Troubleshooting an Unhealthy Upstream Failure (UH)

  • Objective: Diagnose and resolve an HTTP 503 UH failure using administrative tools.
  • Environment: Linux environment with docker and curl.

Step 1: Deploy a proxy misconfigured to point to a nonexistent health path

Deploy an Envoy proxy configured to perform HTTP health checks against /nonexistent-health on a test backend.

Step 2: Observe the failure

curl -i http://localhost:10000/api

Expected Output: HTTP/1.1 503 Service Unavailable

Step 3: Diagnose using the admin interface

# Step A: Inspect access flags or query clusters
curl -s http://127.0.0.1:9901/clusters | grep health_flags

# Step B: Locate the failed health check marker
# Output confirms: /failed_active_hc

Step 4: Resolve the issue

Update the health check path in the cluster definition to / and reload the configuration. Verify host status returns to healthy.


16. Quick Reference and Cheat Sheet

Important Configuration Files

File / Location Operational Purpose
/etc/envoy/envoy.yaml Primary default static bootstrap configuration file.
/etc/systemd/system/envoy.service Systemd service unit definition file.
/var/log/envoy/ Standard directory for Envoy proxy operational access logs.

Key Administrative Endpoints ([http://127.0.0.1:9901](http://127.0.0.1:9901))

Path Purpose
/server_info Shows proxy state, uptime, and build details.
/config_dump Displays full active configuration across dynamic xDS layers.
/clusters Lists cluster health status, host IP endpoints, and flags.
/stats/prometheus Exposes performance metrics formatted for Prometheus scraping.
/logging?http=debug Dynamically updates component log levels without process restarts.

Essential CLI Commands

Operational Task Command Command String
Validate configuration syntax envoy --mode validate -c /etc/envoy/envoy.yaml
Start Envoy with configuration file envoy -c /etc/envoy/envoy.yaml
Check binary build version envoy --version
Verify open listening ports `ss -tlnp
Initiate graceful process termination curl -X POST [http://127.0.0.1:9901/quitquitquit](http://127.0.0.1:9901/quitquitquit)

Access Log Response Flags Quick Reference

Flag Meaning Primary Action Item
UH No Healthy Upstream Check backend process status and verify health check paths.
UF Upstream Connection Failed Check target port, firewall rule, and mTLS configuration.
NR No Route Matched Verify request Host header and path prefix routing rules.
UO Upstream Overflow Increase circuit breaker max connection/pending request thresholds.
DC Downstream Connection Terminated Check client-side application request timeouts.

Default Ports and Protocols

Port Protocol Usage / Description
9901 HTTP Local Envoy administrative management interface.
15000 HTTP Default admin interface port inside Istio sidecars.
15001 TCP Istio outbound sidecar traffic capture listener port.
15006 TCP Istio inbound sidecar traffic capture listener port.
10000 HTTP/TCP Standard default application proxy listener port.

17. Further Reading and References

  1. Official Envoy Proxy Documentation: https://www.envoyproxy.io/docs
  2. Envoy Configuration Architecture Overview: https://www.envoyproxy.io/docs/envoy/latest/configuration/configuration
  3. Envoy Admin Interface Documentation: https://www.envoyproxy.io/docs/envoy/latest/operations/admin
  4. xDS Transport Protocol Specification: https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol