Domain 9Security & PKIINTERMEDIATE

HashiCorp Vault Production Handbook: Architecture, Operations, and Troubleshooting

A comprehensive SysAdmin and SRE operational guide to HashiCorp Vault, covering internal architecture, Raft storage, secrets engines, ACL policies, CLI workflows, production troubleshooting, and SRE best practices.

Domain: Domain 9: Security & PKI•Updated: 2026-11-03
#linux#sysadmin#vault#security#devops#sre

1. Overview

HashiCorp Vault is an enterprise-grade secrets management system designed to secure, store, and tightly control access to tokens, passwords, certificates, encryption keys, and sensitive application credentials. In modern production environments, traditional methods of secrets delivery—such as hardcoding plain-text API keys in repositories, injecting raw variables into .env files, or relying on unencrypted base64 Kubernetes Secrets—introduce significant security risks. These static secrets lack automated rotation, generate poor audit trails, and frequently leak during security breaches.

Vault addresses these challenges by acting as a central control plane for identity-based security. Rather than distributing long-lived credentials across environments, applications and operational workflows fetch dynamic or versioned secrets at runtime via secure API calls. Vault encrypts all sensitive data at rest, logs every API request to strict audit backends, enforces granular Access Control List (ACL) policies, and attaches time-to-live (TTL) leases to credentials so they expire automatically.

+-----------------------------------------------------------------------+
|                         APPLICATION WORKLOAD                          |
|         (Kubernetes Pod / CI Pipeline / Microservice / VM)           |
+-----------------------------------------------------------------------+
                                    |
          1. Authenticate           | 2. Identity Token & Policy
         (AppRole / JWT / K8s)      v
+-----------------------------------------------------------------------+
|                            HASHICORP VAULT                            |
|                                                                       |
|  +---------------------+  +--------------------+  +----------------+  |
|  | Authentication Layer |  | Security Barrier   |  | Audit Engine   |  |
|  +---------------------+  +--------------------+  +----------------+  |
|                                                                       |
|  +-----------------------------------------------------------------+  |
|  |                        SECRETS ENGINES                          |  |
|  |  +------------+  +--------------+  +-----------+  +----------+  |  |
|  |  | KV v1 / v2 |  | Dynamic DB   |  | PKI / CA  |  | Transit  |  |  |
|  |  +------------+  +--------------+  +-----------+  +----------+  |  |
|  +-----------------------------------------------------------------+  |
+-----------------------------------------------------------------------+
                                    |
          3. Encrypted Data         | (Internal Raft Storage)
          Storage & Retrieval       v
+-----------------------------------------------------------------------+
|                     ENCRYPTED STORAGE BACKEND                         |
+-----------------------------------------------------------------------+

Common Production Use Cases

Vault functions as a multi-purpose security platform across infrastructure ecosystems:

  • Static Application Secrets: Versioned, audited key/value storage (KV v2) for API keys, database URLs, and third-party tokens.
  • Dynamic Database Credentials: On-demand generation of short-lived database users for PostgreSQL, MySQL, or MongoDB that automatically expire when their lease ends.
  • TLS / Internal PKI: Built-in Certificate Authority (CA) capable of generating and renewing X.509 TLS certificates dynamically for internal services.
  • Encryption as a Service (Transit): Offloads encryption and decryption of application data to Vault without exposing master key material to application memory.
  • Cloud IAM Credentials: Dynamically provisions temporary AWS IAM user keys or GCP access tokens scoped to short-lived automation tasks.
  • CI/CD & Automation Pipelines: Enables Jenkins, GitHub Actions, or GitLab CI to authenticate via AppRole or JWT/OIDC and retrieve ephemeral credentials during deployments.
  • Kubernetes Integration: Native secret injection into application pods using sidecar containers (Vault Agent) or the Vault Container Storage Interface (CSI) driver.
  • SSH Access Control: Issues one-time passwords (OTP) or short-lived signed SSH certificates to engineers, eliminating static authorization keys on servers.
  • Secure Secret Handoff: Response wrapping delivers sensitive responses inside single-use tokens, preventing plaintext exposures across chat platforms or ticket systems.

Architectural Comparison: Vault vs. Traditional Solutions

Feature / Capability Plaintext / .env Files Native Kubernetes Secrets HashiCorp Vault
Storage Encryption None (Plaintext on disk) Base64 encoded (Plaintext unless KMS-at-rest configured) AES-256 GCM encrypted at rest behind a cryptographic barrier
Credential Lifetime Static / Long-lived Static / Long-lived Ephemeral / Dynamic (TTL & automatic revocation)
Access Control File permissions Kubernetes RBAC (Namespace-scoped) Fine-grained path/capability ACL policies
Audit Logging None Kubernetes API audit logs (generic) Detailed secret-level request & response audit trails
Secret Rotation Manual script / Redeployment Manual / Complex custom controllers Automatic engine-driven dynamic rotation

2. Architecture and Mental Model

To effectively operate and troubleshoot Vault, SREs must understand its internal cryptographic lifecycle, core subsystem router, and storage architecture.

The Cryptographic Barrier and Sealed State

All data stored inside Vault passes through a strict Cryptographic Barrier. When Vault is stopped or restarted, it enters a Sealed state. In this state, Vault’s storage engine remains accessible on disk, but the internal Master Key and Encryption Key are purged from memory.

                  +-----------------------------------+
                  |      UNSEAL PROCEDURE STARTED     |
                  +-----------------------------------+
                                    |
                    Submit Key Shards (Shamir Split)
                                    v
                  +-----------------------------------+
                  | Reconstruct Master Encryption Key |
                  +-----------------------------------+
                                    |
                     Decrypt Storage Barrier Keys
                                    v
                  +-----------------------------------+
                  |  VAULT UNSEALED (API Ready 200)   |
                  +-----------------------------------+
  1. Sealed State: Vault cannot read or write data. All external API calls return an HTTP 503 Service Unavailable error.
  2. Unsealing: To transition to an unsealed state, operators must present key shares reconstructed via Shamir’s Secret Sharing algorithm, or rely on Cloud Auto-Unseal (utilizing AWS KMS, GCP KMS, Azure Key Vault, or Transit Vault).
  3. In-Memory Operation: Once unsealed, the decrypted master keys reside exclusively in system RAM. Vault uses memory-locking (mlock system call) to prevent memory pages containing keys from swapping to physical disk.

Core Subsystems and Architecture Flow

graph TD
    Client[Client / Application / CLI] -->|1. Authenticate| AuthEngine[Auth Method: AppRole / K8s / OIDC]
    AuthEngine -->|2. Generate Token & ACL Policy| TokenStore[Token Store & Lease Manager]
    Client -->|3. Secret Read Request + Token| Barrier[Cryptographic Barrier]
    Barrier -->|4. Validate Token & Path ACL| PolicyEngine[Policy Engine / ACL Check]
    PolicyEngine -->|5. Forward Allowed Request| EngineRouter[Secrets Engine Router]
    EngineRouter -->|6. Execute Read / Mint Dynamic Cred| KVEngine[Secrets Engine: KV v2 / DB / PKI]
    KVEngine -->|7. Persist / Fetch Encrypted Data| RaftStorage[(Integrated Storage: Raft Consensus)]
    Barrier -->|8. Audit Log Request & Response| AuditDevice[Audit Log File / Syslog]
    KVEngine -->|9. Decrypt & Return Payload + TTL Lease| Client

High Availability and Raft Consensus

Modern Vault installations utilize Integrated Storage (Raft) as their primary production backend. Raft removes external dependencies like Consul by embedding a distributed consensus protocol directly into the Vault binary.

  • Leader Node: A single Vault instance acts as the active leader, handling all write operations, engine state updates, and lease generation.
  • Follower / Standby Nodes: Remaining nodes maintain replicated state logs via mTLS over port 8201. If client requests hit a standby node, the node forwards the request internally to the active leader or returns a redirect.
  • Raft Quorum: A cluster of $N$ nodes tolerates up to $(N-1)/2$ failures. Production clusters typically run 3 or 5 nodes across distinct availability zones.

3. Core Concepts and Terminology

  • Mount Path: The URI prefix where a secrets engine or auth method is enabled (e.g., secret/, database/, auth/approle/). Paths segment Vault’s internal routing table.
  • Secrets Engine: A Vault plugin responsible for managing, storing, or dynamically generating credentials. Examples include kv-v2, database, pki, transit, and aws.
  • Auth Method: A plugin that verifies client identity (human or machine) and converts that identity into a Vault token bound to specific ACL policies.
  • Policy: An HCL (HashiCorp Configuration Language) document defining path-level authorization permissions using explicit capabilities (create, read, update, delete, list, sudo).
  • Token: A cryptographic bearer string used by clients to authenticate API requests. Tokens are attached to policies, bound to TTLs, and organized in parent-child hierarchies.
  • Lease: A temporal metadata wrapper assigned to dynamic secrets and renewable tokens. When a lease’s TTL expires without renewal, Vault revokes the credential automatically.
  • Response Wrapping: A temporary security mechanism where Vault wraps a response payload inside a short-lived single-use token (hvs.CAES...). The intended recipient unwraps the payload, ensuring intermediate transport channels cannot read the secret.
  • Audit Device: An enabled logging destination (file, syslog, or socket) that records a JSON entry for every incoming API request and outgoing response.

4. Installation and Version Management

Hardware Sizing Guidelines

Production Vault clusters running Integrated Storage (Raft) should adhere to standard resource allocations:

Sizing Tier CPU Cores Memory (RAM) Storage Capacity Storage Performance Use Case
Small Cluster 2 - 4 Cores 8 - 16 GB 100+ GB NVMe/SSD 3,000+ IOPS Small production, staging, test environments
Large Cluster 4 - 8 Cores 32 - 64 GB 200+ GB NVMe/SSD 10,000+ IOPS High-transaction production workloads

Package Installation

HashiCorp provides official Linux packages containing the vault binary, default configuration files, and systemd service units.

Debian / Ubuntu Installation

# Install GPG key and repository manager
sudo apt-get update && sudo apt-get install -y gpg coreutils curl lsb-release

# Add HashiCorp GPG key
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

# Add official repository
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

# Update index and install Vault
sudo apt-get update && sudo apt-get install -y vault

RHEL / CentOS / Rocky Linux Installation

# Install yum-utils
sudo yum install -y yum-utils

# Add HashiCorp official repository
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo

# Install Vault package
sudo yum -y install vault

Binary Capability & System Privileges

Vault requires memory locking (mlock) to prevent sensitive keys from swapping to disk. The official package configures Linux capabilities on the binary:

# Verify capability assignment on the binary
sudo getcap /usr/bin/vault
# Expected output: /usr/bin/vault cap_ipc_lock=+ep

# If missing, explicitly set the capability
sudo setcap cap_ipc_lock=+ep /usr/bin/vault

5. Configuration Files and Directory Structure

Directory and Layout Reference

File / Directory Owner : Group Permissions Purpose
/etc/vault.d/vault.hcl vault:vault 0640 Primary server configuration file.
/etc/vault.d/vault.hclic vault:vault 0640 Enterprise license file (Enterprise editions only).
/opt/vault/data/ vault:vault 0700 Storage data directory for Raft consensus engine.
/opt/vault/tls/ vault:vault 0700 Server TLS certificates, private keys, and CA bundles.
/var/log/vault/ vault:vault 0750 Vault server logs and plaintext/JSON audit trail files.
/lib/systemd/system/vault.service root:root 0644 Systemd service management unit file.

Production Configuration Example (/etc/vault.d/vault.hcl)

# Server Operational Settings
ui            = true
disable_cache = false
mlock         = true
default_lease_ttl = "168h"
max_lease_ttl     = "720h"

# Storage Backend configuration using Integrated Storage (Raft)
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-node-01"

  retry_join {
    leader_api_addr = "https://192.168.1.10:8200"
  }
  retry_join {
    leader_api_addr = "https://192.168.1.11:8200"
  }
  retry_join {
    leader_api_addr = "https://192.168.1.12:8200"
  }
}

# Network Listener Configuration
listener "tcp" {
  address       = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file = "/opt/vault/tls/vault-cert.pem"
  tls_key_file  = "/opt/vault/tls/vault-key.pem"
  tls_client_ca_file = "/opt/vault/tls/vault-ca.pem"
  tls_disable   = false
}

# API and Cluster Cluster Communication Endpoints
api_addr     = "https://192.168.1.10:8200"
cluster_addr = "https://192.168.1.10:8201"

Environment Variables

Operators configure client interactions via environment variables:

export VAULT_ADDR="https://127.0.0.1:8200"     # Vault server target endpoint
export VAULT_TOKEN="hvs.CAES..."                # Active authentication token
export VAULT_CACERT="/opt/vault/tls/vault-ca.pem" # Custom CA certificate path
# export VAULT_SKIP_VERIFY="true"               # Strictly for local dev/testing

Configuration Validation

Always validate syntax before applying changes or triggering restarts:

# Validate configuration file syntax
vault server -config=/etc/vault.d/vault.hcl -validate

6. Service Management and Common Operations

Systemd Operations

# Enable Vault service at system boot
sudo systemctl enable vault

# Start Vault server daemon
sudo systemctl start vault

# Check service runtime status
sudo systemctl status vault

# Reload configuration without restarting (SIGHUP)
sudo systemctl reload vault

# Restart service (triggers Sealed state)
sudo systemctl restart vault

Validating Service Sockets and Health

# Verify listening ports (8200 for API/UI, 8201 for Raft mTLS)
sudo ss -tlnp | grep 820

# Query unauthenticated health endpoint
curl -s $VAULT_ADDR/v1/sys/health | jq .

7. Common Administrative Tasks and Workflows

Workflow 1: Initializing and Unsealing a Cluster

# 1. Initialize cluster with 5 key shares and a threshold of 3
export VAULT_ADDR="https://127.0.0.1:8200"
vault operator init -key-shares=5 -key-threshold=3 > /root/vault-init.txt
sudo chmod 0600 /root/vault-init.txt

# 2. Check cluster status (Output will show Sealed: true)
vault status

# 3. Supply 3 distinct unseal keys sequentially
vault operator unseal <Unseal-Key-1>
vault operator unseal <Unseal-Key-2>
vault operator unseal <Unseal-Key-3>

# 4. Authenticate using the root token
vault login <Initial-Root-Token>

Workflow 2: Enabling KV v2 Secrets Engine & Reading Secrets

# 1. Enable KV v2 secrets engine at custom path 'secret/'
vault secrets enable -path=secret kv-v2

# 2. Write key/value secret data
vault kv put secret/myapp/config db_user="app_prod" db_pass="SuperSecretPassword123!"

# 3. Read secret values
vault kv get secret/myapp/config

# 4. Inspect secret metadata and versions
vault kv metadata get secret/myapp/config

Workflow 3: Writing Least-Privilege Policy & AppRole Authentication

# 1. Define HCL policy file
cat <<EOF > /tmp/myapp-policy.hcl
path "secret/data/myapp/*" {
  capabilities = ["read", "list"]
}
EOF

# 2. Write policy to Vault
vault policy write myapp-read /tmp/myapp-policy.hcl

# 3. Enable AppRole authentication
vault auth enable approle

# 4. Create AppRole bound to policy
vault write auth/approle/role/myapp-role \
    secret_id_ttl="24h" \
    token_num_uses=10 \
    token_ttl="1h" \
    token_max_ttl="4h" \
    token_policies="myapp-read"

# 5. Fetch Role ID and generate Secret ID for client application
vault read auth/approle/role/myapp-role/role-id -format=json | jq -r .data.role_id
vault write -f auth/approle/role/myapp-role/secret-id -format=json | jq -r .data.secret_id

8. CLI Command Reference and Cheatsheet

Environment & System Inspection

Command Purpose Key Flags / Arguments Practical Example
vault status Displays seal state, HA status, and Raft info. -format=json vault status -format=json
vault version Prints Vault CLI and server version info. N/A vault version
vault login Authenticates CLI session. -method=<method> vault login -method=userpass username=admin

Secrets Engine Administration (KV v2)

Command Purpose Key Flags / Arguments Practical Example
vault secrets enable Mounts a secrets engine. -path=<mount> vault secrets enable -path=secret kv-v2
vault kv put Writes key-value data. key=value vault kv put secret/db pass="s3cr3t"
vault kv get Reads secret values. -field=<key>, -version=<num> vault kv get -field=pass secret/db
vault kv list Lists keys under path. N/A vault kv list secret/
vault kv metadata get Shows versions and metadata. N/A vault kv metadata get secret/db
vault kv destroy Permanently erases secret version -versions=<list> vault kv destroy -versions=1 secret/db

Critical Note on KV Paths: For KV v2 engines, the underlying API path automatically injects data/ for data operations (e.g., CLI path secret/myapp maps internally to API path secret/data/myapp). ACL policies must explicitly specify secret/data/... when defining rules for KV v2.

Auth Methods & Policy Management

Command Purpose Key Flags / Arguments Practical Example
vault auth enable Enables authentication method. -path=<mount> vault auth enable approle
vault policy write Creates or updates ACL policy. ` <file - >`
vault token lookup Inspects token metadata. <token> vault token lookup
vault token capabilities Tests policy permissions for path. <path> vault token capabilities secret/data/app

Cluster & Raft Storage Operations

Command Purpose Key Flags / Arguments Practical Example
vault operator raft list-peers Shows Raft cluster peers & state. N/A vault operator raft list-peers
vault operator raft snapshot save Saves binary Raft backup snapshot. <path/to/file> vault operator raft snapshot save /backup/vault.snap
vault operator raft snapshot restore Restores cluster state from snapshot -force vault operator raft snapshot restore -force /backup/vault.snap

9. Logs, Monitoring and Diagnostics

Audit Logging Setup

Audit logs record every incoming request and outgoing response. Enable a file audit device:

# Enable audit logging to file
vault audit enable file file_path=/var/log/vault/audit.log

# Inspect audit log tail
sudo tail -f /var/log/vault/audit.log | jq .

Journalctl and Server Logs

# Follow systemd journal logs for Vault daemon
sudo journalctl -u vault -f --no-pager

HTTP Health Check Endpoint Reference

The /v1/sys/health endpoint provides raw health status for load balancers and monitoring agents:

curl -s $VAULT_ADDR/v1/sys/health | jq .
HTTP Status Code Meaning / Operational State Load Balancer Action
200 OK Initialized, unsealed, and currently active leader node. Pass traffic.
429 Too Many Requests Unsealed, healthy standby follower node. Remove from active pool (or forward).
472 Custom Code Performance standby or DR secondary node. Do not route active writes.
501 Not Implemented Vault server is not initialized. Alert operator.
503 Service Unavailable Vault is sealed or unreachable. Immediate high-severity alert.

10. Troubleshooting Guide and Common Failure Scenarios

Problem 1: Vault Cluster is Sealed (503 Service Unavailable)

Symptoms

Applications fail with connection or HTTP 503 errors. Running vault status returns:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          true
Total Shares    5
Threshold       3

Possible Causes

  1. The Vault daemon was restarted manually or crashed due to Out-Of-Memory (OOM).
  2. The host server rebooted.
  3. Auto-unseal backend (AWS KMS / Transit) failed IAM authentication or network checks.

Investigation and Diagnosis

  1. Inspect systemd status: sudo systemctl status vault.
  2. Check system logs for crash context: sudo journalctl -u vault -n 50 --no-pager.
  3. Query health endpoint: curl -s $VAULT_ADDR/v1/sys/health.

Root Cause Identification

Confirm if Sealed: true is present. If using Auto-unseal, check if cloud credentials or KMS network endpoints timed out during startup.

Resolution

For manual Shamir setups, submit threshold key shards:

vault operator unseal <Key-Shard-1>
vault operator unseal <Key-Shard-2>
vault operator unseal <Key-Shard-3>

For Auto-unseal, fix KMS network routes or cloud IAM roles, then restart the service: sudo systemctl restart vault.

Verification

Run vault status and confirm Sealed: false.

Prevention

Implement Auto-unseal via cloud KMS to ensure automatic recovery after host reboot.


Problem 2: Access Denied / Permission Denied on Secret Operations

Symptoms

CLI returns Error reading secret/myapp/config: Error making API request. Code: 403. Errors: * 1 error occurred: * permission denied.

Possible Causes

  1. Token lacks required ACL policy permissions.
  2. Incorrect path syntax for KV v2 (missing or erroneous data/ prefix in policy vs CLI).
  3. Expired authentication token.

Investigation and Diagnosis

  1. Inspect current token details: vault token lookup.
  2. Check capabilities for target path: vault token capabilities secret/data/myapp/config.

Root Cause Identification

If capabilities return deny, the policy attached to the token does not match the target API path.

Resolution

Update policy to explicitly reference KV v2 path structures:

# Fix: Ensure path uses 'data/' for KV v2
path "secret/data/myapp/*" {
  capabilities = ["read", "list"]
}

Apply updated policy: vault policy write myapp-policy /tmp/myapp-policy.hcl.

Verification

Re-test capabilities: vault token capabilities secret/data/myapp/config (Should return read, list).

Prevention

Standardize policy path naming conventions across KV v1 and KV v2 engines.


Problem 3: Client Token Expired or Invalid

Symptoms

Requests fail with * invalid token or permission denied.

Possible Causes

  1. Token reached its max TTL limit without renewal.
  2. Static token hardcoded into deployment scripts.

Investigation and Diagnosis

Check token expiration:

vault token lookup

Root Cause Identification

If expire_time has passed or ttl is 0, the token has expired and cannot perform operations.

Resolution

If renewable and within max TTL:

vault token renew

If completely expired, re-authenticate using assigned auth method (e.g., AppRole or Kubernetes SA).

Verification

Verify token validity: vault token lookup.

Prevention

Avoid static tokens in production applications; use dynamic authentication methods (AppRole, Kubernetes Auth) that execute automatic re-authentication routines.


Problem 4: Connection Refused or TLS Handshake Errors

Symptoms

CLI returns Error checking status: Get "[https://127.0.0.1:8200/v1/sys/seal-status](https://127.0.0.1:8200/v1/sys/seal-status)": dial tcp 127.0.0.1:8200: connect: connection refused or TLS certificate validation errors.

Possible Causes

  1. Vault daemon is not running.
  2. Listening port 8200 blocked by firewall (ufw, iptables, security groups).
  3. VAULT_ADDR protocol mismatch (http:// instead of https://).
  4. Self-signed TLS certificate missing local CA trust.

Investigation and Diagnosis

  1. Check process status: sudo systemctl status vault.
  2. Inspect network listener sockets: sudo ss -tlnp | grep 8200.
  3. Test raw HTTP endpoint: curl -v $VAULT_ADDR/v1/sys/health.

Root Cause Identification

If ss shows no socket, Vault daemon failed to bind. If curl fails with SSL error, local trust store is missing VAULT_CACERT.

Resolution

Fix VAULT_ADDR and specify CA certificate explicitly:

export VAULT_ADDR="https://vault.example.com:8200"
export VAULT_CACERT="/etc/vault.d/vault-ca.pem"

Verification

Run vault status successfully.

Prevention

Automate certificate deployment across cluster infrastructure using configuration management tools.


Problem 5: Secret Not Found (404 Error)

Symptoms

CLI or API returns Error reading secret/myapp/foo: Error making API request. Code: 404.

Possible Causes

  1. Secret path contains typos.
  2. Secret version was soft-deleted or permanently destroyed.
  3. Mount path does not exist.

Investigation and Diagnosis

  1. List enabled mounts: vault secrets list.
  2. Check secret metadata: vault kv metadata get secret/myapp/foo.

Root Cause Identification

Review output from metadata get to see if current version is marked destroyed: true or deletion_time is set.

Resolution

If soft-deleted, restore version:

vault kv undelete -versions=2 secret/myapp/foo

Verification

Read secret: vault kv get secret/myapp/foo.

Prevention

Enforce soft-deletion routines in staging environments before authorizing destruction commands.


Problem 6: Standby Node Read Failure (429 HTTP Response)

Symptoms

API requests routed to non-leader cluster nodes fail or fail to process writes.

Possible Causes

Load balancer misconfiguration routing active traffic to standby Raft followers.

Investigation and Diagnosis

Inspect node HA status:

vault status

Check HTTP status code:

curl -s -i $VAULT_ADDR/v1/sys/health

Root Cause Identification

If HTTP response returns 429 and vault status indicates HA Mode: standby, the node is a follower.

Resolution

Reconfigure load balancer health check paths to query /v1/sys/health so standby nodes are excluded from active write traffic pools.

Verification

Confirm active traffic routes exclusively to the leader returning HTTP 200.

Prevention

Utilize active/standby routing pools on ingress controllers and load balancers.


Problem 7: Kubernetes Vault Agent Injector Sidecar Failure

Symptoms

Application pods stuck in Init:0/1 or CreateContainerConfigError state.

Possible Causes

  1. Pod service account not bound to Vault Kubernetes auth role.
  2. Vault Agent injector service unable to connect to VAULT_ADDR.

Investigation and Diagnosis

  1. Inspect pod events: kubectl describe pod <pod-name>.
  2. View init container logs: kubectl logs <pod-name> -c vault-agent-init.

Root Cause Identification

Log output shows permission denied or connection refused during Kubernetes auth login token exchange.

Resolution

Verify Vault Kubernetes auth configuration:

vault write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc:443"

Verification

Restart pod deployment: kubectl rollout restart deployment/<app-name>.

Prevention

Automate Kubernetes auth role definitions alongside service account creation via IaC.


Problem 8: Database Secrets Engine Dynamic Credential Failure

Symptoms

API call vault read database/creds/my-role fails.

Possible Causes

  1. Vault administrative DB user lacks permissions to execute CREATE USER on target database engine.
  2. Database connection string misconfigured inside Vault.

Investigation and Diagnosis

Test database connection configuration:

vault read database/config/mydb

Root Cause Identification

Vault logs indicate sql error GRANT/CREATE permission denied on database instance.

Resolution

Grant administrative privileges to Vault’s management user inside PostgreSQL/MySQL.

Verification

Generate dynamic credentials: vault read database/creds/my-role.

Prevention

Audit database user permissions during initial database engine configuration.


Problem 9: Audit Log Disk Full / Storage Exhaustion

Symptoms

Vault stops responding to write operations; API calls hang or time out.

Possible Causes

Audit log files filled root partition /var/log/vault/. Vault strictly blocks operations if an active audit device cannot write log output.

Investigation and Diagnosis

Check disk space utilization:

df -h /var/log/vault/

Root Cause Identification

Partition shows 100% usage.

Resolution

Rotate or compress old audit log files, or expand disk volume:

sudo logrotate -f /etc/logrotate.d/vault

Verification

Check storage space: df -h /var/log/vault/.

Prevention

Configure log rotation policies (logrotate) and allocate separate physical volumes for audit logging directories.


11. Security and Best Practices

  1. System Memory Locking (mlock): Ensure mlock = true is set in configuration to prevent sensitive data from swapping to swap files.
  2. Revoke Root Tokens Post-Bootstrap: Never use initial root tokens for production microservices. Create fine-grained AppRole/Kubernetes auth roles and revoke root tokens immediately after setting up cluster administration.
  3. Enforce TLS Everything: Always run Vault behind valid TLS certificates. Disable VAULT_SKIP_VERIFY in all environments outside local sandbox development.
  4. Audit Logging Redundancy: Enable at least two independent audit backends (e.g., file and syslog) to prevent single-point logging failures from freezing API traffic.
  5. Least-Privilege Policies: Restrict capabilities to specific paths. Avoid broad wildcard policy definitions like path "*".

12. Performance and Resource Optimisation

  • Disk Subsystem IOPS: Ensure Raft storage backend volumes run on high-performance NVMe SSDs capable of delivering >3,000 IOPS for small clusters or >10,000 IOPS for large deployments. Slower disks create latency bottlenecks during Raft log entries.
  • Separation of Audit Storage: Store audit log files on a physically separate disk volume from the Raft storage directory (/opt/vault/data) to prevent high I/O write contention.
  • Network Latency Thresholds: Maintain cross-node network latency under 8ms across availability zones to avoid Raft consensus timeouts and election flaps.
  • Lease Cleanup: Set reasonable default_lease_ttl and max_lease_ttl values to prevent storage state bloat caused by unrevoked dynamic credentials.

13. Backup, Recovery and Maintenance

Raft Snapshot Automated Backups

Run periodic automated snapshot backups:

# Take a snapshot backup of integrated Raft storage
vault operator raft snapshot save /backup/vault-$(date +%F).snap

Important: Raft storage snapshots contain encrypted Vault data. Restoring a snapshot requires access to the original unseal keys or auto-unseal KMS key used when the backup was created.

Snapshot Restoration Procedure

# Restore cluster state from backup snapshot (Disruptive Operation)
vault operator raft snapshot restore -force /backup/vault-2026-11-03.snap

14. Automation and Integration

Terraform Vault Provider Example

provider "vault" {
  address = "https://vault.example.com:8200"
}

resource "vault_mount" "kvv2" {
  path        = "secret"
  type        = "kv"
  options     = { version = "2" }
  description = "Production KV v2 Secrets Store"
}

resource "vault_policy" "reader" {
  name   = "myapp-reader"
  policy = <<EOT
path "secret/data/myapp/*" {
  capabilities = ["read"]
}
EOT
}

CI/CD Authentication (GitHub Actions / AppRole)

jobs:
  fetch-secrets:
    runs-on: ubuntu-latest
    steps:
      - name: Fetch Secret from Vault
        env:
          VAULT_ADDR: 'https://vault.example.com:8200'
          ROLE_ID: ${{ secrets.VAULT_ROLE_ID }}
          SECRET_ID: ${{ secrets.VAULT_SECRET_ID }}
        run: |
          VAULT_TOKEN=$(curl -s --request POST \
            --data "{\"role_id\": \"$ROLE_ID\", \"secret_id\": \"$SECRET_ID\"}" \
            $VAULT_ADDR/v1/auth/approle/login | jq -r .auth.client_token)

          DB_PASS=$(curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
            $VAULT_ADDR/v1/secret/data/myapp/config | jq -r .data.data.db_pass)

          echo "DB_PASS=$DB_PASS" >> $GITHUB_ENV

15. Practical Labs and Exercises

Lab 1 (Beginner): Local Initialization and KV Engine Setup

  1. Launch a local dev Vault server or initialize a new node:
vault operator init -key-shares=1 -key-threshold=1
  1. Unseal the instance using the generated key shard.
  2. Enable KV v2 engine at secret/ and store a key/value pair.
  3. Read the secret back using -field output formatting.

Lab 2 (Intermediate): AppRole Auth & Policy Scoping

  1. Create a policy file webapp.hcl granting read capability on secret/data/webapp/*.
  2. Enable approle auth and attach the policy to webapp-role.
  3. Mint a Role ID and Secret ID.
  4. Authenticate to Vault via curl using the credentials and verify that secret reading succeeds.

Lab 3 (Advanced Troubleshooting): Sealed State Recovery & Audit Bottleneck

  1. Force-seal your local test cluster: vault operator seal.
  2. Observe API failure responses (HTTP 503).
  3. Unseal the cluster using required key shards.
  4. Enable file audit logging to /var/log/vault/audit.log and monitor incoming API calls in real time using tail -f.

16. Quick Reference and Cheat Sheet

Important Configuration Files

File / Directory Purpose
/etc/vault.d/vault.hcl Primary server configuration file.
/opt/vault/data/ Raft storage data directory.
/opt/vault/tls/ Server TLS certificates and keys.

Important Log Locations

Log Path Purpose
/var/log/vault/audit.log Detailed API audit device event stream.
journalctl -u vault Systemd server operational logs.

Common Administrative Commands

Task Command
Check Status vault status
Unseal Cluster vault operator unseal <key-shard>
Write KV Secret vault kv put secret/myapp key=val
Read KV Secret vault kv get secret/myapp
Save Backup vault operator raft snapshot save backup.snap

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command Primary Fix Action
HTTP 503 Sealed vault status Provide unseal key shards via vault operator unseal
HTTP 403 Forbidden vault token capabilities <path> Fix policy path rules (include data/ for KV v2)
HTTP 429 Standby curl -s $VAULT_ADDR/v1/sys/health Re-route client traffic to active leader node
Connection Refused `sudo ss -tlnp grep 8200`

Ports and Protocols

Port Protocol Purpose
8200 TCP / HTTP(S) Vault API, CLI commands, and UI interface.
8201 TCP / mTLS Raft cluster consensus gossip & request forwarding.

17. Further Reading and References

  1. HashiCorp Vault Official Documentation: https://developer.hashicorp.com/vault/docs
  2. HashiCorp Vault Raft Deployment Guide: https://developer.hashicorp.com/vault/tutorials/day-one-raft/raft-deployment-guide
  3. Vault Reference Architecture: https://developer.hashicorp.com/vault/tutorials/day-one-raft/raft-reference-architecture
  4. SadServers HashiCorp Vault Scenario Labs: https://sadservers.com/labs/hashicorp-vault/cheatsheet