Domain 4Kubernetes EngineeringINTERMEDIATE

etcd Administration, Operations, and Troubleshooting Guide for Linux SREs

A comprehensive operational handbook for deploying, maintaining, backing up, and troubleshooting etcd clusters in production and Kubernetes control-plane environments.

Domain: Domain 4: Kubernetes Cluster Engineering•Updated: 2026-11-03
#linux#sysadmin#etcd#kubernetes

1. Overview

etcd is a strongly consistent, distributed key-value store engineered primarily for shared configuration, service discovery, distributed locking, and critical cluster state management. Written in Go, it implements the Raft consensus algorithm to guarantee linearizable reads and atomic writes across a cluster of nodes.

Unlike relational databases (such as PostgreSQL or MySQL) or high-throughput NoSQL document databases (such as MongoDB), etcd is intentionally optimized for small, high-reliability read and write operations rather than large payload storage, complex JOIN queries, or analytical workloads.

+-----------------------------------------------------------------------+
|                         Key Characteristics                           |
+------------------------------------+----------------------------------+
| Strongly Consistent                | Raft consensus majority commits  |
| Key-Value Storage                  | Flat byte-array keys & values    |
| Event Notification                 | HTTP/2 gRPC streaming Watches    |
| Operational Scope                  | Metadata & cluster state storage |
+------------------------------------+----------------------------------+

Primary Use Cases

  • Kubernetes Control Plane State Store: etcd serves as the sole persistent data backend for the Kubernetes API server. All cluster objects—Pods, Services, ConfigMaps, Secrets, Custom Resource Definitions (CRDs), and state transitions—are stored in etcd.
  • Service Discovery: Dynamic registration and continuous health tracking of endpoints across distributed microservices.
  • Distributed Locking & Leader Election: Coordinating exclusive work and single-leader operations across clustered application instances.
  • Shared Configuration Management: Storing global settings that require strict synchronization and immediate notifications upon modification across fleet servers.

Role in Kubernetes and Production Environments

In Kubernetes, when an administrator executes kubectl apply -f deployment.yaml, the kube-apiserver validates the object definition and persists it directly into etcd. Control plane controllers (such as the Deployment Controller and Scheduler) monitor etcd via gRPC watches managed by the API server to trigger state reconciliation.

 [kubectl / Client]
        |
        v
 [kube-apiserver] <---- gRPC / TLS (2379) ----> [ etcd Cluster ]
        |                                             |
  (Watches State)                             (Raft Consensus)
        |                                             |
 [Controllers / Kubelet] <----------------------------+

If etcd becomes unavailable or experiences data corruption, the Kubernetes control plane loses its ability to process mutations. While existing container workloads running on Kubelets continue executing on worker nodes, scheduling new pods, auto-scaling, node failure recovery, and configuration updates halt completely.

In managed Kubernetes offerings (such as Amazon EKS, Google GKE, or Azure AKS), the cloud provider manages etcd operational complexity. However, in self-managed bare-metal clusters, on-premises environments, or clusters provisioned via kubeadm, Linux System Administrators and Site Reliability Engineers (SREs) are directly responsible for etcd health, storage management, backups, and disaster recovery.


2. Architecture and Mental Model

To effectively operate and troubleshoot etcd, SREs must understand its internal layered architecture, request lifecycle, and consensus mechanisms.

Architecture Diagram

graph TD
    subgraph Client Layer
        C1[etcdctl CLI / Application Client]
        C2[kube-apiserver]
    end

    subgraph etcd Node 1 - LEADER
        API1[gRPC Server / API Layer :2379]
        Raft1[Raft Consensus Engine :2380]
        MVCC1[MVCC Storage Module]
        BTree1[In-Memory B-Tree Index]
        Bolt1[(BoltDB On-Disk Store)]
    end

    subgraph etcd Node 2 - FOLLOWER
        API2[gRPC Server / API Layer :2379]
        Raft2[Raft Consensus Engine :2380]
        MVCC2[MVCC Storage Module]
        BTree2[In-Memory B-Tree Index]
        Bolt2[(BoltDB On-Disk Store)]
    end

    subgraph etcd Node 3 - FOLLOWER
        API3[gRPC Server / API Layer :2379]
        Raft3[Raft Consensus Engine :2380]
        MVCC3[MVCC Storage Module]
        BTree3[In-Memory B-Tree Index]
        Bolt3[(BoltDB On-Disk Store)]
    end

    C1 -->|gRPC / TLS| API1
    C2 -->|gRPC / TLS| API1

    API1 --> Raft1
    Raft1 -->|Raft AppendEntries| Raft2
    Raft1 -->|Raft AppendEntries| Raft3

    Raft1 --> MVCC1
    Raft2 --> MVCC2
    Raft3 --> MVCC3

    MVCC1 --> BTree1
    MVCC1 --> Bolt1
    MVCC2 --> BTree2
    MVCC2 --> Bolt2
    MVCC3 --> BTree3
    MVCC3 --> Bolt3

Internal Components and Lifecycle

  1. gRPC Server & API Layer (Port 2379): Handles client communication over HTTP/2 using gRPC. Supports authentication, role-based access control (RBAC), and connection multiplexing.
  2. Raft Consensus Module (Port 2380): Enforces consensus among cluster members. Handles leader elections, heartbeat distribution, and log entry replication.
  3. Multi-Version Concurrency Control (MVCC) Layer: Implements immutability. Modifications do not overwrite existing key-value pairs; instead, every write creates a new revision of the key.
  4. In-Memory B-Tree Index: Maintains key-to-revision mappings in RAM to allow $O(\log N)$ key lookups to locate disk offsets.
  5. BoltDB (bbolt) Storage Backend: Embedded key-value database providing persistent copy-on-write B+ tree storage on disk (--data-dir).

Request Lifecycle (Write vs. Read)

Write Operation (PUT Request)

  1. gRPC Entry: Client sends a PUT request to port 2379 of any cluster member. If received by a follower, the follower forwards the request to the leader.
  2. Raft Log Proposal: The leader assigns a new global revision number, appends the mutation to its local Raft log, and broadcasts an AppendEntries RPC to all follower nodes over port 2380.
  3. Quorum Replication: Each follower appends the entry to its Write-Ahead Log (WAL) on disk and responds with an ACK.
  4. Commit & Apply: Once a majority (quorum) of nodes ACK the entry, the leader marks the log entry as committed. It applies the change to its MVCC storage and BoltDB backend.
  5. Response & Watch Broadcast: The leader returns success to the client and streams the mutation event to active gRPC Watch clients matching the key prefix.

Read Operation (GET Request)

  • Linearizable Reads (Default): The request goes to the leader. The leader verifies its status with a majority of followers via a lightweight read-index check before reading from MVCC/BoltDB, ensuring stale data is never returned.
  • Serializable Reads: Served directly from any member’s local MVCC storage without contacting the leader. Offers lower latency but may return slightly stale data if the local member lags in Raft log replication.

Quorum Math and Node Counts

etcd requires a majority of operational nodes to maintain quorum:

$$\text{Quorum} = \left\lfloor \frac{N}{2} \right\rfloor + 1$$

Cluster Size ($N$) Quorum Needed Fault Tolerance (Max Failure)
1 1 0 nodes
3 2 1 node
5 3 2 nodes
7 4 3 nodes

Operational Insight: Always maintain an odd number of members (3 or 5). Adding a 4th node to a 3-node cluster increases the required quorum from 2 to 3 without increasing failure tolerance (both can only survive 1 failure).


3. Core Concepts and Terminology

Understanding etcd requires familiarity with several core primitives:

Keys and Values

Keys and values are stored as raw byte sequences. This design allows etcd to handle arbitrary binary structures or UTF-8 strings. Key paths in Kubernetes follow hierarchical, UNIX-like URI conventions (e.g., /registry/pods/default/nginx).

Monotonic Revision

A 64-bit cluster-wide global counter. Every write operation (PUT or DELETE) increments the revision counter globally. Revisions represent logical time in etcd and provide consistent point-in-time snapshot isolation.

Leases

Leases are TTL (Time-To-Live) timers attached to keys. When a lease expires, all keys bound to that lease are automatically deleted.

  • Use Case: Kubernetes node heartbeats (/registry/leases/kube-node-lease/<node-name>) use leases. If a node stops renewing its lease, its status changes to NotReady.

Watches

Watches allow clients to subscribe to key changes asynchronously. Unlike traditional HTTP polling, etcd watches establish persistent gRPC streams starting from a specified revision (--rev), receiving real-time key mutations without polling overhead.

Transactions (txn)

Atomic Compare-And-Swap (CAS) blocks executing conditionally in a single execution step:

IF (revision("foo") = 10 AND value("bar") = "active")
THEN PUT "foo" "updated"
ELSE GET "foo"

This construct prevents race conditions during concurrent controller operations.


4. Installation and Version Management

System Prerequisites

  • Storage: Low-latency SSDs or NVMe drives. Disk fdatasync latency must remain under 10 ms (ideally $<1\text{ ms}$).
  • Network: Dedicated sub-millisecond LAN interconnects between cluster members.
  • Time Synchronization: Synchronized system clocks (via chrony or systemd-timesyncd) across all nodes.

Installation Methods

Method A: Native Linux Package Manager (Debian/Ubuntu/RHEL)

# Debian / Ubuntu
sudo apt-get update && sudo apt-get install -y etcd-server etcd-client

# RHEL / Rocky Linux / Fedora
sudo dnf install -y etcd
ETCD_VER=v3.5.12
DOWNLOAD_URL=https://github.com/etcd-io/etcd/releases/download

# Download official release tarball
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

# Extract archives
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/
sudo mv /tmp/etcd-${ETCD_VER}-linux-amd64/etcd* /usr/local/bin/

# Verify binary installation
etcd --version
etcdctl version

Method C: Kubernetes Static Pods (kubeadm)

In kubeadm-provisioned clusters, etcd runs as a static pod controlled by kubelet using the container image defined in /etc/kubernetes/manifests/etcd.yaml.

Version Identification and Verification

# Check client and API protocol version
export ETCDCTL_API=3
etcdctl version

Output:

etcdctl version: 3.5.12
API version: 3.5

5. Configuration Files and Directory Structure

File / Directory Default Path / Location Purpose & Administrative Details
Main Configuration File /etc/etcd/etcd.conf.yml YAML configuration controlling network listeners, paths, TLS certs, and Raft clustering.
Data Directory /var/lib/etcd/ Contains the Write-Ahead Log (wal/) and BoltDB backend store (member/snap/db).
Systemd Unit File /etc/systemd/system/etcd.service Defines unit initialization parameters, resource limits, and service restart directives.
TLS Certificate Directory /etc/etcd/pki/ or /etc/kubernetes/pki/etcd/ Holds server, peer, and CA certificates for client and inter-node TLS authentication.
Static Pod Manifest /etc/kubernetes/manifests/etcd.yaml Configuration file evaluated by Kubelet on Kubernetes control-plane nodes.

Production YAML Configuration Example (/etc/etcd/etcd.conf.yml)

name: 'etcd-node-01'
data-dir: '/var/lib/etcd'
listen-peer-urls: 'https://192.168.1.10:2380'
listen-client-urls: 'https://192.168.1.10:2379,https://127.0.0.1:2379'
initial-advertise-peer-urls: 'https://192.168.1.10:2380'
advertise-client-urls: 'https://192.168.1.10:2379'

initial-cluster: 'etcd-node-01=https://192.168.1.10:2380,etcd-node-02=https://192.168.1.11:2380,etcd-node-03=https://192.168.1.12:2380'
initial-cluster-token: 'etcd-cluster-token-prod'
initial-cluster-state: 'new'

# Storage & Maintenance limits
quota-backend-bytes: 8589934592 # 8 GB backend quota
auto-compaction-retention: '1h'
auto-compaction-mode: 'periodic'

# Security (Mutual TLS)
client-transport-security:
  cert-file: '/etc/etcd/pki/server.crt'
  key-file: '/etc/etcd/pki/server.key'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/pki/ca.crt'

peer-transport-security:
  cert-file: '/etc/etcd/pki/peer.crt'
  key-file: '/etc/etcd/pki/peer.key'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/pki/ca.crt'

Security and Directory Ownership

The etcd data directory contains unencrypted master keys and system state. Ensure restricted filesystem access:

sudo groupadd --system etcd
sudo useradd --system -g etcd -s /sbin/nologin etcd
sudo chown -R etcd:etcd /var/lib/etcd /etc/etcd
sudo chmod 700 /var/lib/etcd

6. Service Management and Common Operations

Managing Standalone etcd via Systemd

# Enable service to auto-start on boot
sudo systemctl enable etcd

# Start etcd service
sudo systemctl start etcd

# Verify process status
sudo systemctl status etcd

# Reload unit file changes without dropping connections
sudo systemctl reload etcd

# Gracefully restart etcd
sudo systemctl restart etcd

Managing Kubernetes Stacked etcd (Static Pods)

In kubeadm environments, etcd runs as containerized static pods managed by kubelet.

# Find active etcd pod using container runtime CLI
crictl pods --name etcd

# Inspect etcd container logs directly
crictl logs $(crictl ps --name etcd -q)

# Alternative for Docker environments
docker ps | grep etcd

Listening Port and Process Verification

# Verify client API listener on port 2379
ss -tlnp | grep 2379

# Verify Raft peer interconnect listener on port 2380
ss -tlnp | grep 2380

Expected output:

LISTEN 0      4096     192.168.1.10:2379      0.0.0.0:*    users:(("etcd",pid=1420,fd=7))
LISTEN 0      4096     192.168.1.10:2380      0.0.0.0:*    users:(("etcd",pid=1420,fd=8))

7. Common Administrative Tasks and Workflows

Workflow A: Adding a New Node to a Cluster

Adding a member requires a strict two-step process: first registering the member in the Raft topology via CLI, then starting the new daemon.

  1. Register the new member:
etcdctl member add etcd-node-04 --peer-urls=https://192.168.1.13:2380
  1. **Configure the new node’s /etc/etcd/etcd.conf.yml**:
  • Set initial-cluster-state: 'existing'.
  • Include the full updated member string in initial-cluster.
  1. Start the new node process:
sudo systemctl start etcd

Workflow B: Removing a Failed Node

# 1. Identify member ID
etcdctl member list -w table

# 2. Remove node from Raft consensus
etcdctl member remove 8e9e05c52164694d

Warning: Never remove members if doing so drops active operational nodes below quorum.

Workflow C: Compaction and Defragmentation

Without manual or automated maintenance, MVCC retains every historical key mutation indefinitely, eventually exceeding disk backend quotas.

  Compaction: Marks historical revisions as obsolete inside the database.
              (Frees logical space, but disk file size DOES NOT shrink).
                                   │
                                   ▼
Defragmentation: Reclaims empty pages inside BoltDB files.
                 (Shrinks physical disk file size on disk).
# Step 1: Discover current global revision
REV=$(etcdctl endpoint status -w json | jq '.[0].Header.revision')

# Step 2: Compact historical revisions prior to current revision
etcdctl compact $REV

# Step 3: Reclaim physical disk space across all cluster members
etcdctl defrag --cluster

# Step 4: Disarm NOSPACE alarm if previously triggered
etcdctl alarm disarm

8. CLI Command Reference and Cheatsheet

Always enforce ETCDCTL_API=3 when interacting with modern etcd clusters.

Environment Variables Configuration

Set these shell environment variables on client nodes to avoid repeatedly typing flags:

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS="https://192.168.1.10:2379,https://192.168.1.11:2379,https://192.168.1.12:2379"
export ETCDCTL_CACERT="/etc/etcd/pki/ca.crt"
export ETCDCTL_CERT="/etc/etcd/pki/server.crt"
export ETCDCTL_KEY="/etc/etcd/pki/server.key"

Key-Value Operations (etcdctl)

Task Command Flags / Options Example Explanation
Write Key etcdctl put --lease=<ID> etcdctl put /config/app "v2.1" Stores value at key path.
Read Key etcdctl get --prefix, -w json etcdctl get /config/app Retrieves value for specified key.
Prefix Scan etcdctl get --prefix, --keys-only etcdctl get /registry/pods --prefix Lists all keys matching hierarchical path.
Delete Key etcdctl del --prefix etcdctl del /config/app Removes key or prefix tree.
Stream Key etcdctl watch --prefix, --rev=<REV> etcdctl watch /registry/pods --prefix Opens real-time change stream.

Health and Cluster Management

Task Command Flags / Options Example Explanation
Member Health etcdctl endpoint health --cluster etcdctl endpoint health --cluster Checks individual node gRPC accessibility and consensus capabilities.
Status Table etcdctl endpoint status -w table, -w json etcdctl endpoint status -w table Displays leader ID, Raft term, DB size, and revision.
List Members etcdctl member list -w table etcdctl member list -w table Lists cluster node IDs, peer URLs, and client URLs.
List Alarms etcdctl alarm list None etcdctl alarm list Queries triggered system alarms (NOSPACE, CORRUPT).
Disarm Alarm etcdctl alarm disarm None etcdctl alarm disarm Clears active maintenance alarm flags.

9. Logs, Monitoring and Diagnostics

Log Inspection Methods

Systemd Journal (Standalone Deployments)

# Follow live etcd logs
sudo journalctl -u etcd -f -o cat

# Filter logs for warnings and errors since boot
sudo journalctl -u etcd -p err..warning -b

Container Runtime Logs (Kubernetes Static Pods)

# Fetch last 100 log lines from etcd container
crictl logs --tail=100 $(crictl ps --name etcd -q)

Critical Log Indicators to Monitor

  • raft.node: <ID> changed leader: Signals a Raft election event. High frequencies indicate network instability or I/O stalls.
  • apply entries took too long: Indicates slow disk writes during WAL fsync or BoltDB commits.
  • database space exceeded: Indicates backend storage quota has been breached (NOSPACE alarm active).
  • failed to send message ... wall clock difference: Indicates NTP time drift across cluster nodes.

Diagnostic Commands for Storage and Network

# 1. Inspect physical disk space on data partition
df -h /var/lib/etcd

# 2. Extract database size across endpoints in bytes
etcdctl endpoint status -w json | jq '.[] | {endpoint: .Endpoint, dbSize: .Status.dbSize}'

# 3. Test disk sync performance with fio (Must show fdatasync <= 10ms)
fio --name=bench-etcd --directory=/var/lib/etcd --size=100m --rw=write --bs=4k --fdatasync=1 --iodepth=1

10. Troubleshooting Guide and Common Failure Scenarios

Scenario 1: Connection Refused / Endpoint Unhealthy

Symptoms

  • etcdctl endpoint health returns connection refused or context deadline exceeded.
  • In Kubernetes, kubectl operations fail with Error from server (InternalError): net/http: TLS handshake timeout.

Possible Causes

  1. etcd daemon process is crashed or stopped.
  2. Network firewall blocking port 2379.
  3. Client targeting wrong ETCDCTL_ENDPOINTS IP or port.

Investigation and Diagnosis

  1. Check process status:
systemctl status etcd
# Or for static pods:
crictl ps --name etcd
  1. Verify local socket binding:
ss -tlnp | grep 2379
  1. Test TCP network connectivity:
nc -zvw3 192.168.1.10 2379

Root Cause Identification

If ss shows no socket open, the service process failed during initialization. If socket is open locally but remote nc fails, firewall or network routing is blocking traffic.

Resolution

  • Restart etcd service (systemctl restart etcd).
  • Update firewall rules: sudo ufw allow 2379/tcp or sudo firewall-cmd --add-port=2379/tcp --permanent && sudo firewall-cmd --reload.

Verification

etcdctl endpoint health

Prevention

Implement automated service health checks and automated restart policies in systemd.


Scenario 2: No Quorum / Cluster Unavailable

Symptoms

  • etcdctl operations hang indefinitely or return etcdserver: request timed out.
  • kubectl commands fail across the board.
  • Kubernetes API server logs show repeated context timeouts communicating with backend etcd.

Possible Causes

  • A majority of cluster members are offline or network-isolated (e.g., 2 out of 3 nodes down).
  • Network partition isolating the leader.

Investigation and Diagnosis

  1. Check member status across all endpoints:
etcdctl --endpoints=https://192.168.1.10:2379,https://192.168.1.11:2379,https://192.168.1.12:2379 endpoint health
  1. Inspect member list:
etcdctl member list -w table

Root Cause Identification

Compare operational node count against minimum quorum formula ($Q = \lfloor N/2 \rfloor + 1$). If active node count $< Q$, quorum is lost.

Resolution

  1. Bring offline nodes back online immediately.
  2. If nodes are permanently destroyed, force cluster reconfiguration by starting an isolated surviving member with --force-new-cluster (Last Resort Disaster Recovery).

Warning: --force-new-cluster creates a single-node cluster and resets Raft metadata. Use only when lost nodes cannot be recovered.

Verification

etcdctl endpoint status -w table

Prevention

Deploy member nodes across distinct physical hardware racks or availability zones. Maintain 5-node clusters for critical production workloads to survive up to 2 simultaneous node failures.


Scenario 3: Frequent Leader Elections & High Latency

Symptoms

  • Logs show frequent raft.node changed leader entries.
  • etcdctl endpoint status shows changing Raft Terms (RAFT TERM).
  • Severe latency spikes in Kubernetes API responses.

Possible Causes

  1. High disk write latency (slow disk fsync) on WAL store.
  2. CPU starvation on etcd control plane host.
  3. Inter-node network latency/packet loss on peer port 2380.

Investigation and Diagnosis

  1. Check disk I/O metrics using iostat:
iostat -xz 1 10
  1. Query server latency metrics:
etcdctl endpoint status -w table
  1. Check system resource usage via top or htop.

Root Cause Identification

High %util or extended await times on the storage disk hosting --data-dir indicate storage bottleneck.

Resolution

  • Move --data-dir to high-speed NVMe SSD storage.
  • Isolate etcd workload; avoid running disk-heavy application pods on control-plane nodes.

Verification

Monitor logs to confirm stable leader elections without term increments.

Prevention

Dedicate high-performance SSD storage exclusively for etcd data paths (/var/lib/etcd).


Scenario 4: NOSPACE Alarm / Database Size Limit

Symptoms

  • Writes fail with etcdserver: mvcc: database space exceeded.
  • etcdctl alarm list returns memberID:... alarm:NOSPACE.

Possible Causes

BoltDB backend storage has reached the configured disk quota (--quota-backend-bytes, default 2GB if unspecified).

Investigation and Diagnosis

  1. List active alarms:
etcdctl alarm list
  1. Inspect current database size:
etcdctl endpoint status -w table

Root Cause Identification

If DB SIZE is near or equal to the backend quota limit, etcd triggers the NOSPACE alarm and blocks incoming writes.

Resolution

# 1. Get current revision number
REV=$(etcdctl endpoint status -w json | jq '.[0].Header.revision')

# 2. Compact old revisions
etcdctl compact $REV

# 3. Defragment backend storage to reclaim physical space
etcdctl defrag --cluster

# 4. Disarm active NOSPACE alarm
etcdctl alarm disarm

Verification

etcdctl put test-key test-value
etcdctl del test-key

Prevention

Set --quota-backend-bytes=8589934592 (8 GB) in configuration and configure automated periodic compaction (--auto-compaction-retention=1h).


Scenario 5: CORRUPT Alarm / Data Corruption

Symptoms

  • Logs show fatal error: corrupt database or hash mismatch.
  • etcdctl alarm list displays alarm:CORRUPT.

Possible Causes

Unscheduled node power failure, underlying disk corruption, or hardware memory failures causing inconsistent byte states on disk.

Investigation and Diagnosis

  1. Read active alarms:
etcdctl alarm list
  1. Inspect log errors:
journalctl -u etcd -e

Root Cause Identification

Logged checksum errors or failed start sequences due to invalid byte markers in the WAL directory.

Resolution

  1. Stop the corrupted member service:
sudo systemctl stop etcd
  1. Delete corrupted node data directory:
sudo rm -rf /var/lib/etcd/member
  1. Restart the service to force synchronization from healthy cluster members:
sudo systemctl start etcd
  1. If all members are corrupted, perform a full disaster recovery restore from a snapshot.

Verification

etcdctl endpoint health --cluster

Prevention

Use ECC RAM, enterprise SSDs with power-loss protection (PLP), and schedule automated daily snapshot backups.


Scenario 6: TLS / Certificate Expiration Errors

Symptoms

  • etcdctl fails with x509: certificate has expired or is not yet valid or x509: certificate signed by unknown authority.
  • Kubernetes API server fails to initialize.

Possible Causes

  1. CA, server, or client TLS certificates have expired.
  2. Mismatched trust certificates passed in flags.

Investigation and Diagnosis

Inspect certificate expiration dates using OpenSSL:

openssl x509 -in /etc/etcd/pki/server.crt -text -noout | grep -A 2 "Validity"

Root Cause Identification

Not After field shows a past timestamp.

Resolution

  1. Regenerate expired certificates using your PKI tool (e.g., kubeadm certs renew etcd-server).
  2. Update cert files in /etc/etcd/pki/.
  3. Restart etcd service.

Verification

etcdctl endpoint health

Prevention

Implement automated certificate monitoring metrics and automated rotation pipelines.


Scenario 7: Kubernetes API Server Cannot Reach etcd

Symptoms

  • kubectl commands time out with service unavailable.
  • kube-apiserver static pod repeatedly restarts.

Possible Causes

  1. Incorrect endpoint flag (--etcd-servers) in API server manifest.
  2. Missing or incorrect client TLS certificates passed to kube-apiserver.

Investigation and Diagnosis

Inspect API server configuration manifest:

grep etcd /etc/kubernetes/manifests/kube-apiserver.yaml

Examine API server log output:

crictl logs $(crictl ps --name kube-apiserver -q) | grep -i etcd

Root Cause Identification

API server logs contain connection refused or x509: certificate invalid matching specified etcd target endpoints.

Resolution

Correct parameters in /etc/kubernetes/manifests/kube-apiserver.yaml:

  • Adjust --etcd-servers host endpoints.
  • Confirm validity of --etcd-cafile, --etcd-certfile, and --etcd-keyfile paths.

Verification

Verify API server pod reaches healthy running state:

crictl pods --name kube-apiserver

Prevention

Avoid manually editing static manifest paths without using configuration management automation (e.g., Ansible/Helm).


Scenario 8: Member Failed to Join Cluster

Symptoms

  • New etcd node crashes upon initial startup.
  • Logs report publish error: member count mismatch or conflicting cluster ID.

Possible Causes

  1. Starting node without pre-registering it via etcdctl member add.
  2. Specifying --initial-cluster-state=new instead of existing.
  3. Firewall blocking peer interconnect port 2380.

Investigation and Diagnosis

Check join startup flags in systemd logs:

journalctl -u etcd -n 50

Root Cause Identification

Log entries report cluster ID mismatch or connection timeouts to peer node URLs on port 2380.

Resolution

  1. Verify peer port reachability (nc -zvw3 peer-ip 2380).
  2. Ensure node was added to cluster topology via etcdctl member add.
  3. Set initial-cluster-state: 'existing' in configuration file.
  4. Wipe stale data dir (rm -rf /var/lib/etcd/*) before starting.

Verification

etcdctl member list -w table

Prevention

Strictly adhere to the member addition sequence: API registration first, config update second, process launch third.


11. Security and Best Practices

  • Enforce Mutual TLS (mTLS): Never run etcd unencrypted in production. Use separate CA chains for client API traffic (2379) and peer inter-node communication (2380).
  • Network Isolation: Restrict port 2379 access exclusively to API servers. Block external perimeter access to port 2380 (Raft consensus) using host-level firewalls.
  • Implement Role-Based Access Control (RBAC): When used outside Kubernetes, enable authentication (etcdctl auth enable) and create limited scope roles.
  • Non-Root Execution: Run the etcd process under a dedicated, unprivileged system user (etcd) with strict directory ownership permissions (0700) on /var/lib/etcd.

12. Performance and Resource Optimisation

  • Dedicated High-Speed SSD Storage: etcd is sensitive to disk write latency due to continuous WAL synchronization (fdatasync). Avoid sharing etcd disks with IO-heavy container workloads.
  • Memory Sizing: Allocate adequate RAM to hold the active B-Tree key index in memory (minimum 8 GB RAM recommended for production Kubernetes control planes).
  • Automate Compaction Schedule: Prevent uncontrolled history growth by enabling automatic retention:
auto-compaction-retention: '1h'
auto-compaction-mode: 'periodic'
  • Backend Storage Quota: Explicitly set maximum database limit to prevent unexpected disk exhaustion crashes:
quota-backend-bytes: 8589934592 # 8GB

13. Backup, Recovery and Maintenance

Replication is not a backup mechanism; corrupted data or unintended drop commands will instantly replicate across all nodes. regular external snapshot backups are required.

Saving a Snapshot Backup

# Save point-in-time snapshot file
etcdctl snapshot save /backup/etcd-snapshot-$(date +%F).db

# Verify snapshot integrity
etcdctl snapshot status /backup/etcd-snapshot-$(date +%F).db -w table

Expected validation output:

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 9f8a32b1 |    45210 |       3201 |     45 MB  |
+----------+----------+------------+------------+

Restoring a Cluster from a Snapshot

To restore a failed cluster, restore the snapshot into a new data directory on each member node before initializing the process.

# 1. Stop etcd process on all nodes
sudo systemctl stop etcd

# 2. Extract and restore snapshot into a fresh directory path
etcdctl snapshot restore /backup/etcd-snapshot-2026-11-03.db \
  --name etcd-node-01 \
  --data-dir /var/lib/etcd-restored \
  --initial-cluster etcd-node-01=https://192.168.1.10:2380 \
  --initial-cluster-token etcd-cluster-restored \
  --initial-advertise-peer-urls https://192.168.1.10:2380

# 3. Update configuration file data-dir to point to /var/lib/etcd-restored
sudo sed -i 's|/var/lib/etcd|/var/lib/etcd-restored|g' /etc/etcd/etcd.conf.yml

# 4. Start etcd daemon
sudo systemctl start etcd

14. Automation and Integration

Automated Systemd Backup Timer

Create an automated backup task using systemd timers:

/etc/systemd/system/etcd-backup.service

[Unit]
Description=etcd Automated Snapshot Backup
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/etcdctl snapshot save /var/backups/etcd/etcd-snap-%U.db
Environment="ETCDCTL_API=3"
Environment="ETCDCTL_ENDPOINTS=https://127.0.0.1:2379"
Environment="ETCDCTL_CACERT=/etc/etcd/pki/ca.crt"
Environment="ETCDCTL_CERT=/etc/etcd/pki/server.crt"
Environment="ETCDCTL_KEY=/etc/etcd/pki/server.key"

/etc/systemd/system/etcd-backup.timer

[Unit]
Description=Run etcd backup daily at midnight

[Timer]
OnCalendar=*-*-* 00:00:00
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer:

sudo systemctl daemon-reload
sudo systemctl enable --now etcd-backup.timer

15. Practical Labs and Exercises

Lab 1: Cluster Health and Key Manipulation (Beginner)

Objective

Learn basic key write/read operations and cluster health verification.

Procedure

  1. Confirm API version:
export ETCDCTL_API=3
  1. Verify endpoints health:
etcdctl endpoint health
  1. Write test key:
etcdctl put /test/key "Hello SRE"
  1. Read key:
etcdctl get /test/key
  1. Clean up:
etcdctl del /test/key

Lab 2: Backup and Disaster Recovery Simulation (Intermediate)

Objective

Simulate a catastrophic database corruption and recover operations via snapshot restore.

Procedure

  1. Create a test key: etcdctl put /production/state "operational".
  2. Take a snapshot backup: etcdctl snapshot save /tmp/backup.db.
  3. Simulate disaster: Stop etcd service (sudo systemctl stop etcd) and clear data folder (sudo rm -rf /var/lib/etcd/*).
  4. Restore snapshot:
etcdctl snapshot restore /tmp/backup.db --data-dir=/var/lib/etcd
  1. Fix permissions: sudo chown -R etcd:etcd /var/lib/etcd
  2. Start etcd service: sudo systemctl start etcd
  3. Verify state recovery: etcdctl get /production/state (Must return "operational").

Lab 3: Troubleshooting a NOSPACE Storage Alarm (Advanced SRE)

Objective

Diagnose a write-blocked etcd cluster, compact historical revisions, defragment storage, and clear active alarms.

Procedure

  1. Trigger alarm manually (or simulate full storage condition):
etcdctl alarm disarm # ensure clean state first
  1. Query active alarms:
etcdctl alarm list
  1. Read target revision:
REV=$(etcdctl endpoint status -w json | jq '.[0].Header.revision')
  1. Perform compaction:
etcdctl compact $REV
  1. Run defragmentation across endpoints:
etcdctl defrag --cluster
  1. Clear system alarm state:
etcdctl alarm disarm
  1. Verify write functionality restored:
etcdctl put /health/check "ok"

16. Quick Reference and Cheat Sheet

Important Configuration Files

File Path Purpose
/etc/etcd/etcd.conf.yml Primary standalone YAML configuration file.
/etc/kubernetes/manifests/etcd.yaml Kubernetes static pod spec definition.
/var/lib/etcd/ Physical WAL and snapshot database directory.

Important Log Locations

Log Target Command
Standalone Service sudo journalctl -u etcd -f
Kubernetes Static Pod crictl logs $(crictl ps --name etcd -q)

Common Operational Commands

Operation Command
Check Health etcdctl endpoint health --cluster
Status & DB Size etcdctl endpoint status -w table
Save Snapshot etcdctl snapshot save /backup/snap.db
Defragment Cluster etcdctl defrag --cluster
Disarm Alarm etcdctl alarm disarm

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command Primary Cause
Connection Refused `ss -tlnp grep 2379`
Cluster Unavailable etcdctl member list Lost Raft consensus quorum (majority nodes down).
NOSPACE Error etcdctl alarm list Backend disk storage exceeded quota limit.
Cert Expired openssl x509 -in server.crt -noout -dates Expired PKI TLS server or CA certificate.

Important Ports and Protocols

Port Protocol Purpose
2379 TCP / gRPC Client API communications (etcdctl, kube-apiserver).
2380 TCP / HTTP Inter-node peer Raft log replication and consensus.

17. Further Reading and References

  1. Official etcd Documentation: etcd.io/docs
  2. etcd Operational Guide: etcd.io operations guide
  3. Kubernetes etcd Configuration and Maintenance Guide: kubernetes.io - Operating etcd for Kubernetes
  4. Interactive Visualization of the Raft Consensus Algorithm: raft.github.io