1. Overview
etcd is a strongly consistent, distributed key-value store engineered primarily for shared configuration, service discovery, distributed locking, and critical cluster state management. Written in Go, it implements the Raft consensus algorithm to guarantee linearizable reads and atomic writes across a cluster of nodes.
Unlike relational databases (such as PostgreSQL or MySQL) or high-throughput NoSQL document databases (such as MongoDB), etcd is intentionally optimized for small, high-reliability read and write operations rather than large payload storage, complex JOIN queries, or analytical workloads.
+-----------------------------------------------------------------------+
| Key Characteristics |
+------------------------------------+----------------------------------+
| Strongly Consistent | Raft consensus majority commits |
| Key-Value Storage | Flat byte-array keys & values |
| Event Notification | HTTP/2 gRPC streaming Watches |
| Operational Scope | Metadata & cluster state storage |
+------------------------------------+----------------------------------+
Primary Use Cases
- Kubernetes Control Plane State Store: etcd serves as the sole persistent data backend for the Kubernetes API server. All cluster objects—Pods, Services, ConfigMaps, Secrets, Custom Resource Definitions (CRDs), and state transitions—are stored in etcd.
- Service Discovery: Dynamic registration and continuous health tracking of endpoints across distributed microservices.
- Distributed Locking & Leader Election: Coordinating exclusive work and single-leader operations across clustered application instances.
- Shared Configuration Management: Storing global settings that require strict synchronization and immediate notifications upon modification across fleet servers.
Role in Kubernetes and Production Environments
In Kubernetes, when an administrator executes kubectl apply -f deployment.yaml, the kube-apiserver validates the object definition and persists it directly into etcd. Control plane controllers (such as the Deployment Controller and Scheduler) monitor etcd via gRPC watches managed by the API server to trigger state reconciliation.
[kubectl / Client]
|
v
[kube-apiserver] <---- gRPC / TLS (2379) ----> [ etcd Cluster ]
| |
(Watches State) (Raft Consensus)
| |
[Controllers / Kubelet] <----------------------------+
If etcd becomes unavailable or experiences data corruption, the Kubernetes control plane loses its ability to process mutations. While existing container workloads running on Kubelets continue executing on worker nodes, scheduling new pods, auto-scaling, node failure recovery, and configuration updates halt completely.
In managed Kubernetes offerings (such as Amazon EKS, Google GKE, or Azure AKS), the cloud provider manages etcd operational complexity. However, in self-managed bare-metal clusters, on-premises environments, or clusters provisioned via kubeadm, Linux System Administrators and Site Reliability Engineers (SREs) are directly responsible for etcd health, storage management, backups, and disaster recovery.
2. Architecture and Mental Model
To effectively operate and troubleshoot etcd, SREs must understand its internal layered architecture, request lifecycle, and consensus mechanisms.
Architecture Diagram
graph TD
subgraph Client Layer
C1[etcdctl CLI / Application Client]
C2[kube-apiserver]
end
subgraph etcd Node 1 - LEADER
API1[gRPC Server / API Layer :2379]
Raft1[Raft Consensus Engine :2380]
MVCC1[MVCC Storage Module]
BTree1[In-Memory B-Tree Index]
Bolt1[(BoltDB On-Disk Store)]
end
subgraph etcd Node 2 - FOLLOWER
API2[gRPC Server / API Layer :2379]
Raft2[Raft Consensus Engine :2380]
MVCC2[MVCC Storage Module]
BTree2[In-Memory B-Tree Index]
Bolt2[(BoltDB On-Disk Store)]
end
subgraph etcd Node 3 - FOLLOWER
API3[gRPC Server / API Layer :2379]
Raft3[Raft Consensus Engine :2380]
MVCC3[MVCC Storage Module]
BTree3[In-Memory B-Tree Index]
Bolt3[(BoltDB On-Disk Store)]
end
C1 -->|gRPC / TLS| API1
C2 -->|gRPC / TLS| API1
API1 --> Raft1
Raft1 -->|Raft AppendEntries| Raft2
Raft1 -->|Raft AppendEntries| Raft3
Raft1 --> MVCC1
Raft2 --> MVCC2
Raft3 --> MVCC3
MVCC1 --> BTree1
MVCC1 --> Bolt1
MVCC2 --> BTree2
MVCC2 --> Bolt2
MVCC3 --> BTree3
MVCC3 --> Bolt3
Internal Components and Lifecycle
- gRPC Server & API Layer (Port 2379): Handles client communication over HTTP/2 using gRPC. Supports authentication, role-based access control (RBAC), and connection multiplexing.
- Raft Consensus Module (Port 2380): Enforces consensus among cluster members. Handles leader elections, heartbeat distribution, and log entry replication.
- Multi-Version Concurrency Control (MVCC) Layer: Implements immutability. Modifications do not overwrite existing key-value pairs; instead, every write creates a new revision of the key.
- In-Memory B-Tree Index: Maintains key-to-revision mappings in RAM to allow $O(\log N)$ key lookups to locate disk offsets.
- BoltDB (bbolt) Storage Backend: Embedded key-value database providing persistent copy-on-write B+ tree storage on disk (
--data-dir).
Request Lifecycle (Write vs. Read)
Write Operation (PUT Request)
- gRPC Entry: Client sends a
PUTrequest to port 2379 of any cluster member. If received by a follower, the follower forwards the request to the leader. - Raft Log Proposal: The leader assigns a new global revision number, appends the mutation to its local Raft log, and broadcasts an
AppendEntriesRPC to all follower nodes over port 2380. - Quorum Replication: Each follower appends the entry to its Write-Ahead Log (WAL) on disk and responds with an ACK.
- Commit & Apply: Once a majority (quorum) of nodes ACK the entry, the leader marks the log entry as committed. It applies the change to its MVCC storage and BoltDB backend.
- Response & Watch Broadcast: The leader returns success to the client and streams the mutation event to active gRPC Watch clients matching the key prefix.
Read Operation (GET Request)
- Linearizable Reads (Default): The request goes to the leader. The leader verifies its status with a majority of followers via a lightweight read-index check before reading from MVCC/BoltDB, ensuring stale data is never returned.
- Serializable Reads: Served directly from any member’s local MVCC storage without contacting the leader. Offers lower latency but may return slightly stale data if the local member lags in Raft log replication.
Quorum Math and Node Counts
etcd requires a majority of operational nodes to maintain quorum:
$$\text{Quorum} = \left\lfloor \frac{N}{2} \right\rfloor + 1$$
| Cluster Size ($N$) | Quorum Needed | Fault Tolerance (Max Failure) |
|---|---|---|
| 1 | 1 | 0 nodes |
| 3 | 2 | 1 node |
| 5 | 3 | 2 nodes |
| 7 | 4 | 3 nodes |
Operational Insight: Always maintain an odd number of members (3 or 5). Adding a 4th node to a 3-node cluster increases the required quorum from 2 to 3 without increasing failure tolerance (both can only survive 1 failure).
3. Core Concepts and Terminology
Understanding etcd requires familiarity with several core primitives:
Keys and Values
Keys and values are stored as raw byte sequences. This design allows etcd to handle arbitrary binary structures or UTF-8 strings. Key paths in Kubernetes follow hierarchical, UNIX-like URI conventions (e.g., /registry/pods/default/nginx).
Monotonic Revision
A 64-bit cluster-wide global counter. Every write operation (PUT or DELETE) increments the revision counter globally. Revisions represent logical time in etcd and provide consistent point-in-time snapshot isolation.
Leases
Leases are TTL (Time-To-Live) timers attached to keys. When a lease expires, all keys bound to that lease are automatically deleted.
- Use Case: Kubernetes node heartbeats (
/registry/leases/kube-node-lease/<node-name>) use leases. If a node stops renewing its lease, its status changes toNotReady.
Watches
Watches allow clients to subscribe to key changes asynchronously. Unlike traditional HTTP polling, etcd watches establish persistent gRPC streams starting from a specified revision (--rev), receiving real-time key mutations without polling overhead.
Transactions (txn)
Atomic Compare-And-Swap (CAS) blocks executing conditionally in a single execution step:
IF (revision("foo") = 10 AND value("bar") = "active")
THEN PUT "foo" "updated"
ELSE GET "foo"
This construct prevents race conditions during concurrent controller operations.
4. Installation and Version Management
System Prerequisites
- Storage: Low-latency SSDs or NVMe drives. Disk
fdatasynclatency must remain under 10 ms (ideally $<1\text{ ms}$). - Network: Dedicated sub-millisecond LAN interconnects between cluster members.
- Time Synchronization: Synchronized system clocks (via
chronyorsystemd-timesyncd) across all nodes.
Installation Methods
Method A: Native Linux Package Manager (Debian/Ubuntu/RHEL)
# Debian / Ubuntu
sudo apt-get update && sudo apt-get install -y etcd-server etcd-client
# RHEL / Rocky Linux / Fedora
sudo dnf install -y etcd
Method B: Manual Binary Deployment (Production Recommended)
ETCD_VER=v3.5.12
DOWNLOAD_URL=https://github.com/etcd-io/etcd/releases/download
# Download official release tarball
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
# Extract archives
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/
sudo mv /tmp/etcd-${ETCD_VER}-linux-amd64/etcd* /usr/local/bin/
# Verify binary installation
etcd --version
etcdctl version
Method C: Kubernetes Static Pods (kubeadm)
In kubeadm-provisioned clusters, etcd runs as a static pod controlled by kubelet using the container image defined in /etc/kubernetes/manifests/etcd.yaml.
Version Identification and Verification
# Check client and API protocol version
export ETCDCTL_API=3
etcdctl version
Output:
etcdctl version: 3.5.12
API version: 3.5
5. Configuration Files and Directory Structure
| File / Directory | Default Path / Location | Purpose & Administrative Details |
|---|---|---|
| Main Configuration File | /etc/etcd/etcd.conf.yml |
YAML configuration controlling network listeners, paths, TLS certs, and Raft clustering. |
| Data Directory | /var/lib/etcd/ |
Contains the Write-Ahead Log (wal/) and BoltDB backend store (member/snap/db). |
| Systemd Unit File | /etc/systemd/system/etcd.service |
Defines unit initialization parameters, resource limits, and service restart directives. |
| TLS Certificate Directory | /etc/etcd/pki/ or /etc/kubernetes/pki/etcd/ |
Holds server, peer, and CA certificates for client and inter-node TLS authentication. |
| Static Pod Manifest | /etc/kubernetes/manifests/etcd.yaml |
Configuration file evaluated by Kubelet on Kubernetes control-plane nodes. |
Production YAML Configuration Example (/etc/etcd/etcd.conf.yml)
name: 'etcd-node-01'
data-dir: '/var/lib/etcd'
listen-peer-urls: 'https://192.168.1.10:2380'
listen-client-urls: 'https://192.168.1.10:2379,https://127.0.0.1:2379'
initial-advertise-peer-urls: 'https://192.168.1.10:2380'
advertise-client-urls: 'https://192.168.1.10:2379'
initial-cluster: 'etcd-node-01=https://192.168.1.10:2380,etcd-node-02=https://192.168.1.11:2380,etcd-node-03=https://192.168.1.12:2380'
initial-cluster-token: 'etcd-cluster-token-prod'
initial-cluster-state: 'new'
# Storage & Maintenance limits
quota-backend-bytes: 8589934592 # 8 GB backend quota
auto-compaction-retention: '1h'
auto-compaction-mode: 'periodic'
# Security (Mutual TLS)
client-transport-security:
cert-file: '/etc/etcd/pki/server.crt'
key-file: '/etc/etcd/pki/server.key'
client-cert-auth: true
trusted-ca-file: '/etc/etcd/pki/ca.crt'
peer-transport-security:
cert-file: '/etc/etcd/pki/peer.crt'
key-file: '/etc/etcd/pki/peer.key'
client-cert-auth: true
trusted-ca-file: '/etc/etcd/pki/ca.crt'
Security and Directory Ownership
The etcd data directory contains unencrypted master keys and system state. Ensure restricted filesystem access:
sudo groupadd --system etcd
sudo useradd --system -g etcd -s /sbin/nologin etcd
sudo chown -R etcd:etcd /var/lib/etcd /etc/etcd
sudo chmod 700 /var/lib/etcd
6. Service Management and Common Operations
Managing Standalone etcd via Systemd
# Enable service to auto-start on boot
sudo systemctl enable etcd
# Start etcd service
sudo systemctl start etcd
# Verify process status
sudo systemctl status etcd
# Reload unit file changes without dropping connections
sudo systemctl reload etcd
# Gracefully restart etcd
sudo systemctl restart etcd
Managing Kubernetes Stacked etcd (Static Pods)
In kubeadm environments, etcd runs as containerized static pods managed by kubelet.
# Find active etcd pod using container runtime CLI
crictl pods --name etcd
# Inspect etcd container logs directly
crictl logs $(crictl ps --name etcd -q)
# Alternative for Docker environments
docker ps | grep etcd
Listening Port and Process Verification
# Verify client API listener on port 2379
ss -tlnp | grep 2379
# Verify Raft peer interconnect listener on port 2380
ss -tlnp | grep 2380
Expected output:
LISTEN 0 4096 192.168.1.10:2379 0.0.0.0:* users:(("etcd",pid=1420,fd=7))
LISTEN 0 4096 192.168.1.10:2380 0.0.0.0:* users:(("etcd",pid=1420,fd=8))
7. Common Administrative Tasks and Workflows
Workflow A: Adding a New Node to a Cluster
Adding a member requires a strict two-step process: first registering the member in the Raft topology via CLI, then starting the new daemon.
- Register the new member:
etcdctl member add etcd-node-04 --peer-urls=https://192.168.1.13:2380
- **Configure the new node’s
/etc/etcd/etcd.conf.yml**:
- Set
initial-cluster-state: 'existing'. - Include the full updated member string in
initial-cluster.
- Start the new node process:
sudo systemctl start etcd
Workflow B: Removing a Failed Node
# 1. Identify member ID
etcdctl member list -w table
# 2. Remove node from Raft consensus
etcdctl member remove 8e9e05c52164694d
Warning: Never remove members if doing so drops active operational nodes below quorum.
Workflow C: Compaction and Defragmentation
Without manual or automated maintenance, MVCC retains every historical key mutation indefinitely, eventually exceeding disk backend quotas.
Compaction: Marks historical revisions as obsolete inside the database.
(Frees logical space, but disk file size DOES NOT shrink).
│
▼
Defragmentation: Reclaims empty pages inside BoltDB files.
(Shrinks physical disk file size on disk).
# Step 1: Discover current global revision
REV=$(etcdctl endpoint status -w json | jq '.[0].Header.revision')
# Step 2: Compact historical revisions prior to current revision
etcdctl compact $REV
# Step 3: Reclaim physical disk space across all cluster members
etcdctl defrag --cluster
# Step 4: Disarm NOSPACE alarm if previously triggered
etcdctl alarm disarm
8. CLI Command Reference and Cheatsheet
Always enforce ETCDCTL_API=3 when interacting with modern etcd clusters.
Environment Variables Configuration
Set these shell environment variables on client nodes to avoid repeatedly typing flags:
export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS="https://192.168.1.10:2379,https://192.168.1.11:2379,https://192.168.1.12:2379"
export ETCDCTL_CACERT="/etc/etcd/pki/ca.crt"
export ETCDCTL_CERT="/etc/etcd/pki/server.crt"
export ETCDCTL_KEY="/etc/etcd/pki/server.key"
Key-Value Operations (etcdctl)
| Task | Command | Flags / Options | Example | Explanation |
|---|---|---|---|---|
| Write Key | etcdctl put |
--lease=<ID> |
etcdctl put /config/app "v2.1" |
Stores value at key path. |
| Read Key | etcdctl get |
--prefix, -w json |
etcdctl get /config/app |
Retrieves value for specified key. |
| Prefix Scan | etcdctl get |
--prefix, --keys-only |
etcdctl get /registry/pods --prefix |
Lists all keys matching hierarchical path. |
| Delete Key | etcdctl del |
--prefix |
etcdctl del /config/app |
Removes key or prefix tree. |
| Stream Key | etcdctl watch |
--prefix, --rev=<REV> |
etcdctl watch /registry/pods --prefix |
Opens real-time change stream. |
Health and Cluster Management
| Task | Command | Flags / Options | Example | Explanation |
|---|---|---|---|---|
| Member Health | etcdctl endpoint health |
--cluster |
etcdctl endpoint health --cluster |
Checks individual node gRPC accessibility and consensus capabilities. |
| Status Table | etcdctl endpoint status |
-w table, -w json |
etcdctl endpoint status -w table |
Displays leader ID, Raft term, DB size, and revision. |
| List Members | etcdctl member list |
-w table |
etcdctl member list -w table |
Lists cluster node IDs, peer URLs, and client URLs. |
| List Alarms | etcdctl alarm list |
None | etcdctl alarm list |
Queries triggered system alarms (NOSPACE, CORRUPT). |
| Disarm Alarm | etcdctl alarm disarm |
None | etcdctl alarm disarm |
Clears active maintenance alarm flags. |
9. Logs, Monitoring and Diagnostics
Log Inspection Methods
Systemd Journal (Standalone Deployments)
# Follow live etcd logs
sudo journalctl -u etcd -f -o cat
# Filter logs for warnings and errors since boot
sudo journalctl -u etcd -p err..warning -b
Container Runtime Logs (Kubernetes Static Pods)
# Fetch last 100 log lines from etcd container
crictl logs --tail=100 $(crictl ps --name etcd -q)
Critical Log Indicators to Monitor
raft.node: <ID> changed leader: Signals a Raft election event. High frequencies indicate network instability or I/O stalls.apply entries took too long: Indicates slow disk writes during WAL fsync or BoltDB commits.database space exceeded: Indicates backend storage quota has been breached (NOSPACEalarm active).failed to send message ... wall clock difference: Indicates NTP time drift across cluster nodes.
Diagnostic Commands for Storage and Network
# 1. Inspect physical disk space on data partition
df -h /var/lib/etcd
# 2. Extract database size across endpoints in bytes
etcdctl endpoint status -w json | jq '.[] | {endpoint: .Endpoint, dbSize: .Status.dbSize}'
# 3. Test disk sync performance with fio (Must show fdatasync <= 10ms)
fio --name=bench-etcd --directory=/var/lib/etcd --size=100m --rw=write --bs=4k --fdatasync=1 --iodepth=1
10. Troubleshooting Guide and Common Failure Scenarios
Scenario 1: Connection Refused / Endpoint Unhealthy
Symptoms
etcdctl endpoint healthreturnsconnection refusedorcontext deadline exceeded.- In Kubernetes,
kubectloperations fail withError from server (InternalError): net/http: TLS handshake timeout.
Possible Causes
etcddaemon process is crashed or stopped.- Network firewall blocking port 2379.
- Client targeting wrong
ETCDCTL_ENDPOINTSIP or port.
Investigation and Diagnosis
- Check process status:
systemctl status etcd
# Or for static pods:
crictl ps --name etcd
- Verify local socket binding:
ss -tlnp | grep 2379
- Test TCP network connectivity:
nc -zvw3 192.168.1.10 2379
Root Cause Identification
If ss shows no socket open, the service process failed during initialization. If socket is open locally but remote nc fails, firewall or network routing is blocking traffic.
Resolution
- Restart etcd service (
systemctl restart etcd). - Update firewall rules:
sudo ufw allow 2379/tcporsudo firewall-cmd --add-port=2379/tcp --permanent && sudo firewall-cmd --reload.
Verification
etcdctl endpoint health
Prevention
Implement automated service health checks and automated restart policies in systemd.
Scenario 2: No Quorum / Cluster Unavailable
Symptoms
etcdctloperations hang indefinitely or returnetcdserver: request timed out.kubectlcommands fail across the board.- Kubernetes API server logs show repeated context timeouts communicating with backend etcd.
Possible Causes
- A majority of cluster members are offline or network-isolated (e.g., 2 out of 3 nodes down).
- Network partition isolating the leader.
Investigation and Diagnosis
- Check member status across all endpoints:
etcdctl --endpoints=https://192.168.1.10:2379,https://192.168.1.11:2379,https://192.168.1.12:2379 endpoint health
- Inspect member list:
etcdctl member list -w table
Root Cause Identification
Compare operational node count against minimum quorum formula ($Q = \lfloor N/2 \rfloor + 1$). If active node count $< Q$, quorum is lost.
Resolution
- Bring offline nodes back online immediately.
- If nodes are permanently destroyed, force cluster reconfiguration by starting an isolated surviving member with
--force-new-cluster(Last Resort Disaster Recovery).
Warning:
--force-new-clustercreates a single-node cluster and resets Raft metadata. Use only when lost nodes cannot be recovered.
Verification
etcdctl endpoint status -w table
Prevention
Deploy member nodes across distinct physical hardware racks or availability zones. Maintain 5-node clusters for critical production workloads to survive up to 2 simultaneous node failures.
Scenario 3: Frequent Leader Elections & High Latency
Symptoms
- Logs show frequent
raft.node changed leaderentries. etcdctl endpoint statusshows changing Raft Terms (RAFT TERM).- Severe latency spikes in Kubernetes API responses.
Possible Causes
- High disk write latency (slow disk fsync) on WAL store.
- CPU starvation on etcd control plane host.
- Inter-node network latency/packet loss on peer port 2380.
Investigation and Diagnosis
- Check disk I/O metrics using
iostat:
iostat -xz 1 10
- Query server latency metrics:
etcdctl endpoint status -w table
- Check system resource usage via
toporhtop.
Root Cause Identification
High %util or extended await times on the storage disk hosting --data-dir indicate storage bottleneck.
Resolution
- Move
--data-dirto high-speed NVMe SSD storage. - Isolate etcd workload; avoid running disk-heavy application pods on control-plane nodes.
Verification
Monitor logs to confirm stable leader elections without term increments.
Prevention
Dedicate high-performance SSD storage exclusively for etcd data paths (/var/lib/etcd).
Scenario 4: NOSPACE Alarm / Database Size Limit
Symptoms
- Writes fail with
etcdserver: mvcc: database space exceeded. etcdctl alarm listreturnsmemberID:... alarm:NOSPACE.
Possible Causes
BoltDB backend storage has reached the configured disk quota (--quota-backend-bytes, default 2GB if unspecified).
Investigation and Diagnosis
- List active alarms:
etcdctl alarm list
- Inspect current database size:
etcdctl endpoint status -w table
Root Cause Identification
If DB SIZE is near or equal to the backend quota limit, etcd triggers the NOSPACE alarm and blocks incoming writes.
Resolution
# 1. Get current revision number
REV=$(etcdctl endpoint status -w json | jq '.[0].Header.revision')
# 2. Compact old revisions
etcdctl compact $REV
# 3. Defragment backend storage to reclaim physical space
etcdctl defrag --cluster
# 4. Disarm active NOSPACE alarm
etcdctl alarm disarm
Verification
etcdctl put test-key test-value
etcdctl del test-key
Prevention
Set --quota-backend-bytes=8589934592 (8 GB) in configuration and configure automated periodic compaction (--auto-compaction-retention=1h).
Scenario 5: CORRUPT Alarm / Data Corruption
Symptoms
- Logs show
fatal error: corrupt databaseorhash mismatch. etcdctl alarm listdisplaysalarm:CORRUPT.
Possible Causes
Unscheduled node power failure, underlying disk corruption, or hardware memory failures causing inconsistent byte states on disk.
Investigation and Diagnosis
- Read active alarms:
etcdctl alarm list
- Inspect log errors:
journalctl -u etcd -e
Root Cause Identification
Logged checksum errors or failed start sequences due to invalid byte markers in the WAL directory.
Resolution
- Stop the corrupted member service:
sudo systemctl stop etcd
- Delete corrupted node data directory:
sudo rm -rf /var/lib/etcd/member
- Restart the service to force synchronization from healthy cluster members:
sudo systemctl start etcd
- If all members are corrupted, perform a full disaster recovery restore from a snapshot.
Verification
etcdctl endpoint health --cluster
Prevention
Use ECC RAM, enterprise SSDs with power-loss protection (PLP), and schedule automated daily snapshot backups.
Scenario 6: TLS / Certificate Expiration Errors
Symptoms
etcdctlfails withx509: certificate has expired or is not yet validorx509: certificate signed by unknown authority.- Kubernetes API server fails to initialize.
Possible Causes
- CA, server, or client TLS certificates have expired.
- Mismatched trust certificates passed in flags.
Investigation and Diagnosis
Inspect certificate expiration dates using OpenSSL:
openssl x509 -in /etc/etcd/pki/server.crt -text -noout | grep -A 2 "Validity"
Root Cause Identification
Not After field shows a past timestamp.
Resolution
- Regenerate expired certificates using your PKI tool (e.g.,
kubeadm certs renew etcd-server). - Update cert files in
/etc/etcd/pki/. - Restart etcd service.
Verification
etcdctl endpoint health
Prevention
Implement automated certificate monitoring metrics and automated rotation pipelines.
Scenario 7: Kubernetes API Server Cannot Reach etcd
Symptoms
kubectlcommands time out withservice unavailable.kube-apiserverstatic pod repeatedly restarts.
Possible Causes
- Incorrect endpoint flag (
--etcd-servers) in API server manifest. - Missing or incorrect client TLS certificates passed to
kube-apiserver.
Investigation and Diagnosis
Inspect API server configuration manifest:
grep etcd /etc/kubernetes/manifests/kube-apiserver.yaml
Examine API server log output:
crictl logs $(crictl ps --name kube-apiserver -q) | grep -i etcd
Root Cause Identification
API server logs contain connection refused or x509: certificate invalid matching specified etcd target endpoints.
Resolution
Correct parameters in /etc/kubernetes/manifests/kube-apiserver.yaml:
- Adjust
--etcd-servershost endpoints. - Confirm validity of
--etcd-cafile,--etcd-certfile, and--etcd-keyfilepaths.
Verification
Verify API server pod reaches healthy running state:
crictl pods --name kube-apiserver
Prevention
Avoid manually editing static manifest paths without using configuration management automation (e.g., Ansible/Helm).
Scenario 8: Member Failed to Join Cluster
Symptoms
- New etcd node crashes upon initial startup.
- Logs report
publish error: member count mismatchorconflicting cluster ID.
Possible Causes
- Starting node without pre-registering it via
etcdctl member add. - Specifying
--initial-cluster-state=newinstead ofexisting. - Firewall blocking peer interconnect port 2380.
Investigation and Diagnosis
Check join startup flags in systemd logs:
journalctl -u etcd -n 50
Root Cause Identification
Log entries report cluster ID mismatch or connection timeouts to peer node URLs on port 2380.
Resolution
- Verify peer port reachability (
nc -zvw3 peer-ip 2380). - Ensure node was added to cluster topology via
etcdctl member add. - Set
initial-cluster-state: 'existing'in configuration file. - Wipe stale data dir (
rm -rf /var/lib/etcd/*) before starting.
Verification
etcdctl member list -w table
Prevention
Strictly adhere to the member addition sequence: API registration first, config update second, process launch third.
11. Security and Best Practices
- Enforce Mutual TLS (mTLS): Never run etcd unencrypted in production. Use separate CA chains for client API traffic (2379) and peer inter-node communication (2380).
- Network Isolation: Restrict port 2379 access exclusively to API servers. Block external perimeter access to port 2380 (Raft consensus) using host-level firewalls.
- Implement Role-Based Access Control (RBAC): When used outside Kubernetes, enable authentication (
etcdctl auth enable) and create limited scope roles. - Non-Root Execution: Run the etcd process under a dedicated, unprivileged system user (
etcd) with strict directory ownership permissions (0700) on/var/lib/etcd.
12. Performance and Resource Optimisation
- Dedicated High-Speed SSD Storage: etcd is sensitive to disk write latency due to continuous WAL synchronization (
fdatasync). Avoid sharing etcd disks with IO-heavy container workloads. - Memory Sizing: Allocate adequate RAM to hold the active B-Tree key index in memory (minimum 8 GB RAM recommended for production Kubernetes control planes).
- Automate Compaction Schedule: Prevent uncontrolled history growth by enabling automatic retention:
auto-compaction-retention: '1h'
auto-compaction-mode: 'periodic'
- Backend Storage Quota: Explicitly set maximum database limit to prevent unexpected disk exhaustion crashes:
quota-backend-bytes: 8589934592 # 8GB
13. Backup, Recovery and Maintenance
Replication is not a backup mechanism; corrupted data or unintended drop commands will instantly replicate across all nodes. regular external snapshot backups are required.
Saving a Snapshot Backup
# Save point-in-time snapshot file
etcdctl snapshot save /backup/etcd-snapshot-$(date +%F).db
# Verify snapshot integrity
etcdctl snapshot status /backup/etcd-snapshot-$(date +%F).db -w table
Expected validation output:
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 9f8a32b1 | 45210 | 3201 | 45 MB |
+----------+----------+------------+------------+
Restoring a Cluster from a Snapshot
To restore a failed cluster, restore the snapshot into a new data directory on each member node before initializing the process.
# 1. Stop etcd process on all nodes
sudo systemctl stop etcd
# 2. Extract and restore snapshot into a fresh directory path
etcdctl snapshot restore /backup/etcd-snapshot-2026-11-03.db \
--name etcd-node-01 \
--data-dir /var/lib/etcd-restored \
--initial-cluster etcd-node-01=https://192.168.1.10:2380 \
--initial-cluster-token etcd-cluster-restored \
--initial-advertise-peer-urls https://192.168.1.10:2380
# 3. Update configuration file data-dir to point to /var/lib/etcd-restored
sudo sed -i 's|/var/lib/etcd|/var/lib/etcd-restored|g' /etc/etcd/etcd.conf.yml
# 4. Start etcd daemon
sudo systemctl start etcd
14. Automation and Integration
Automated Systemd Backup Timer
Create an automated backup task using systemd timers:
/etc/systemd/system/etcd-backup.service
[Unit]
Description=etcd Automated Snapshot Backup
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/etcdctl snapshot save /var/backups/etcd/etcd-snap-%U.db
Environment="ETCDCTL_API=3"
Environment="ETCDCTL_ENDPOINTS=https://127.0.0.1:2379"
Environment="ETCDCTL_CACERT=/etc/etcd/pki/ca.crt"
Environment="ETCDCTL_CERT=/etc/etcd/pki/server.crt"
Environment="ETCDCTL_KEY=/etc/etcd/pki/server.key"
/etc/systemd/system/etcd-backup.timer
[Unit]
Description=Run etcd backup daily at midnight
[Timer]
OnCalendar=*-*-* 00:00:00
Persistent=true
[Install]
WantedBy=timers.target
Enable the timer:
sudo systemctl daemon-reload
sudo systemctl enable --now etcd-backup.timer
15. Practical Labs and Exercises
Lab 1: Cluster Health and Key Manipulation (Beginner)
Objective
Learn basic key write/read operations and cluster health verification.
Procedure
- Confirm API version:
export ETCDCTL_API=3
- Verify endpoints health:
etcdctl endpoint health
- Write test key:
etcdctl put /test/key "Hello SRE"
- Read key:
etcdctl get /test/key
- Clean up:
etcdctl del /test/key
Lab 2: Backup and Disaster Recovery Simulation (Intermediate)
Objective
Simulate a catastrophic database corruption and recover operations via snapshot restore.
Procedure
- Create a test key:
etcdctl put /production/state "operational". - Take a snapshot backup:
etcdctl snapshot save /tmp/backup.db. - Simulate disaster: Stop etcd service (
sudo systemctl stop etcd) and clear data folder (sudo rm -rf /var/lib/etcd/*). - Restore snapshot:
etcdctl snapshot restore /tmp/backup.db --data-dir=/var/lib/etcd
- Fix permissions:
sudo chown -R etcd:etcd /var/lib/etcd - Start etcd service:
sudo systemctl start etcd - Verify state recovery:
etcdctl get /production/state(Must return"operational").
Lab 3: Troubleshooting a NOSPACE Storage Alarm (Advanced SRE)
Objective
Diagnose a write-blocked etcd cluster, compact historical revisions, defragment storage, and clear active alarms.
Procedure
- Trigger alarm manually (or simulate full storage condition):
etcdctl alarm disarm # ensure clean state first
- Query active alarms:
etcdctl alarm list
- Read target revision:
REV=$(etcdctl endpoint status -w json | jq '.[0].Header.revision')
- Perform compaction:
etcdctl compact $REV
- Run defragmentation across endpoints:
etcdctl defrag --cluster
- Clear system alarm state:
etcdctl alarm disarm
- Verify write functionality restored:
etcdctl put /health/check "ok"
16. Quick Reference and Cheat Sheet
Important Configuration Files
| File Path | Purpose |
|---|---|
/etc/etcd/etcd.conf.yml |
Primary standalone YAML configuration file. |
/etc/kubernetes/manifests/etcd.yaml |
Kubernetes static pod spec definition. |
/var/lib/etcd/ |
Physical WAL and snapshot database directory. |
Important Log Locations
| Log Target | Command |
|---|---|
| Standalone Service | sudo journalctl -u etcd -f |
| Kubernetes Static Pod | crictl logs $(crictl ps --name etcd -q) |
Common Operational Commands
| Operation | Command |
|---|---|
| Check Health | etcdctl endpoint health --cluster |
| Status & DB Size | etcdctl endpoint status -w table |
| Save Snapshot | etcdctl snapshot save /backup/snap.db |
| Defragment Cluster | etcdctl defrag --cluster |
| Disarm Alarm | etcdctl alarm disarm |
Troubleshooting Quick Reference
| Symptom | Initial Diagnostic Command | Primary Cause |
|---|---|---|
| Connection Refused | `ss -tlnp | grep 2379` |
| Cluster Unavailable | etcdctl member list |
Lost Raft consensus quorum (majority nodes down). |
| NOSPACE Error | etcdctl alarm list |
Backend disk storage exceeded quota limit. |
| Cert Expired | openssl x509 -in server.crt -noout -dates |
Expired PKI TLS server or CA certificate. |
Important Ports and Protocols
| Port | Protocol | Purpose |
|---|---|---|
| 2379 | TCP / gRPC | Client API communications (etcdctl, kube-apiserver). |
| 2380 | TCP / HTTP | Inter-node peer Raft log replication and consensus. |
17. Further Reading and References
- Official etcd Documentation: etcd.io/docs
- etcd Operational Guide: etcd.io operations guide
- Kubernetes etcd Configuration and Maintenance Guide: kubernetes.io - Operating etcd for Kubernetes
- Interactive Visualization of the Raft Consensus Algorithm: raft.github.io