1. Overview
Deploying and operating production applications on Kubernetes requires coordinating a complex ecosystem of API objects. A single application stack typically requires a Kubernetes Deployment or StatefulSet for workload management, a Service for network routing, ConfigMaps and Secrets for configuration, an Ingress object for external routing, and ServiceAccount, Role, and RoleBinding objects for Role-Based Access Control (RBAC). Managing these manifests as raw, disconnected YAML files quickly becomes error-prone, leading to configuration drift, complex deployment pipelines, and fragile rollback procedures.
Helm solves this operational challenge by acting as the package manager for Kubernetes. Helm bundles related Kubernetes manifests into a single, versioned unit of software known as a chart. A chart combines Go-templated YAML files with default configuration parameters stored in a values.yaml file.
+-------------------------------------------------------------------+
| HELM CHART |
| |
| +--------------------+ +------------------+ +---------------+ |
| | Chart.yaml | | values.yaml | | templates/ | |
| | Metadata & Version | | Defaults Config | | Deployment.yaml |
| +--------------------+ +------------------+ | Service.yaml | |
| | Ingress.yaml | |
| +---------------+ |
+-------------------------------------------------------------------+
|
v (helm install / upgrade)
+-------------------------------------------------------------------+
| KUBERNETES RELEASE |
| |
| [ Deployment ] [ Service ] [ ConfigMap ] [ Secret ] |
+-------------------------------------------------------------------+
Core Problems Solved by Helm
- Parameterization & Reusability: Instead of duplicating raw YAML manifests across development, staging, and production clusters, Helm allows administrators to parameterize values (e.g., replica counts, image tags, resource requests) and apply environment-specific configuration files.
- Atomic Release Management: Helm installs or upgrades an entire application stack with a single command, automatically managing standard labels and metadata across all instantiated objects.
- Revision Tracking & Rollbacks: Every modification to an application creates an incremental release revision. If an upgrade fails or introduces bugs, Helm provides single-command rollbacks to any previous working state.
- Dependency Orchestration: Applications that depend on supporting infrastructure (such as databases or caching layers) can declare subchart dependencies, allowing Helm to resolve, fetch, and deploy the full application hierarchy in sequence.
Comparison: Raw Manifests vs. Kustomize vs. Helm
| Feature | Raw kubectl Manifests |
Kustomize | Helm 3 |
|---|---|---|---|
| Templating Mechanism | None (Static files) | Overlay / Patching (Template-free) | Go Templating (.Values, .Release) |
| State & Release Tracking | None (Cluster objects only) | None (Cluster objects only) | Native Release Secrets in cluster |
| Rollback Mechanism | Manual kubectl rollout per workload |
Manual patch re-application | Single-command helm rollback |
| Package Distribution | Git repos / File archives | Git repos | HTTP Repositories & OCI Registries |
| Lifecycle Hooks | None | Limited | Native lifecycle hooks (pre-install, etc.) |
2. Architecture and Mental Model
To effectively administer and troubleshoot Helm, SREs must understand the architectural transition from Helm 2 to Helm 3 and how Helm interacts with the Kubernetes control plane.
Helm 2 vs. Helm 3 Architecture
In Helm 2, the system relied on an in-cluster server component named Tiller. Tiller ran inside the Kubernetes cluster with elevated privileges (often cluster-admin), acting as an intermediary that accepted commands from the local Helm CLI, rendered templates, and applied them directly to the API server. This introduced severe security risks and complex RBAC delegation issues.
Helm 3 completely eliminated Tiller. Helm 3 operates strictly as a client-side binary. It renders templates locally on the administrator’s workstation or CI/CD worker node and communicates directly with the Kubernetes API server using the active credentials in the user’s kubeconfig file.
+-----------------------------------------------------------------------+
| LOCAL WORKSTATION / CI |
| |
| +------------------+ +-------------------+ +--------------+ |
| | Chart Templates| + | User values.yaml | --> | Helm 3 CLI | |
| +------------------+ +-------------------+ +--------------+ |
+-----------------------------------------------------------|-----------+
|
Direct KubeAPI Call | (Using user's kubeconfig context)
v
+-----------------------------------------------------------------------+
| KUBERNETES CLUSTER |
| |
| +-----------------------------------------------------------------+ |
| | Kubernetes API Server | |
| +-----------------------------------------------------------------+ |
| | |
| +----------------------+----------------------+ |
| | | |
| v v |
| +-------------------+ +-------------------+ |
| | Application Stack | | Release Storage | |
| | (Deployment, Pod, | | (Kubernetes Secret| |
| | Service, Ingress) | | `owner=helm`) | |
| +-------------------+ +-------------------+ |
+-----------------------------------------------------------------------+
Components and Responsibilities
- Helm Client (CLI): A local command-line binary that manages chart repositories, builds packages, merges configurations, parses Go templates into raw YAML, and invokes API operations.
- Kubernetes API Server: The cluster endpoint that receives rendered manifests via standard HTTP/REST requests and enforces cluster RBAC.
- Release Storage Backend: Helm 3 stores release metadata directly within the Kubernetes cluster as
Secrets(default) orConfigMapsin the namespace where the release is installed. These secrets carry the labelowner=helmand are named following the patternsh.helm.release.v1.<RELEASE_NAME>.v<REVISION>.
Detailed Install and Upgrade Request Lifecycle
When an administrator executes helm install or helm upgrade, Helm processes the request through four major lifecycle phases:
graph TD
A[Execute helm install/upgrade] --> B[1. Render Phase]
B --> B1[Load Chart.yaml & templates/]
B --> B2[Merge values.yaml + -f files + --set overrides]
B2 --> B3[Evaluate Go Templates]
B3 --> C[2. Hook Evaluation & Execution]
C --> C1{Pre-install/upgrade hooks?}
C1 -- Yes --> C2[Execute Hook Jobs/Pods & Wait]
C1 -- No --> D[3. Apply Phase]
C2 -- Success --> D
D --> D1[Transmit Rendered Manifests to Kube API]
D1 --> D2[Perform 3-Way Strategic Merge Patch / Create]
D2 --> E[4. Record Phase]
E --> E1[Gzip & Base64 Encode Release State]
E1 --> E2[Store as Secret in Target Namespace]
- Render Phase: Helm loads the target chart, merges default
values.yamlwith user-supplied override files (-f) and inline flags (--set), and runs the combined data context through the Go template engine to generate standard Kubernetes YAML manifests. - Hook Phase: If the chart contains resources annotated with lifecycle hooks (e.g.,
helm.sh/hook: pre-install), Helm submits those hook resources (typically KubernetesJobs) to the API server and waits for their successful execution before proceeding. - Apply Phase: Helm transmits the final rendered manifests to the Kubernetes API server. For upgrades, Helm performs a 3-way strategic merge patch comparing the current state, desired state, and previous manifest state.
- Record Phase: Upon successful API acceptance, Helm serializes the complete release metadata, chart contents, and effective values into a gzipped, base64-encoded secret, recording a new release revision in the target namespace.
3. Core Concepts and Terminology
Understanding Helm’s domain vocabulary is essential for effective administration.
+------------------------------------------------------------------+
| HELM GLOSSARY |
| |
| CHART --> Packaging blueprint containing templates. |
| VALUES --> Configuration parameters fed into templates. |
| RELEASE --> Running instance of a chart in a namespace. |
| REVISION --> Immutable numerical snapshot of a release. |
| REPOSITORY --> Remote HTTP or OCI registry hosting charts. |
+------------------------------------------------------------------+
Core Terms
- Chart: An organized directory of files (or a gzipped
.tgzarchive) containing template files, default values, and metadata (Chart.yaml) that define a set of Kubernetes resources. - Release: A named, deployed instance of a chart running within a specific Kubernetes namespace. A single chart (e.g.,
nginx-ingress) can be installed multiple times in the same cluster under different release names (e.g.,ingress-publicandingress-internal). - Revision: A numbered, immutable snapshot representing a specific change state of a release. Every
helm install,helm upgrade, orhelm rollbackincrements the revision counter ($1, 2, 3 \dots$). - Values: Parameter inputs used to customize the templates within a chart. Defaults are defined in
values.yaml, which can be overridden at runtime using custom value files (-f) or command-line parameters (--set). - Repository: A central server location hosting packaged charts. Repositories can be traditional HTTP web servers hosting an
index.yamlfile or modern Open Container Initiative (OCI) registries. - Subchart / Dependency: A chart nested inside another chart’s
charts/directory or declared inChart.yaml. Subcharts allow complex application stacks (e.g., a web app requiring PostgreSQL and Redis) to be managed as a unified chart. - Hooks: Annotations attached to resources (such as
JobsorPods) that instruct Helm to execute them at specific points during a release lifecycle (e.g., database schema migrations before updating web deployment pods).
4. Installation and Version Management
Prerequisites and Dependencies
- Operating System: Linux (Ubuntu, Debian, RHEL, Rocky Linux, SLES), macOS, or Windows.
- Access Tools:
kubectlinstalled and configured with access to a Kubernetes cluster. - Authentication: A valid
kubeconfigfile located at~/.kube/config(or referenced via$KUBECONFIG) with sufficient RBAC privileges to create objects in target namespaces.
Installation Methods
Option 1: Installing via Binary Release (Recommended for Linux SysAdmins)
To install a specific version of Helm 3 directly from GitHub releases:
# Set desired Helm version
HELM_VERSION="v3.16.2"
# Download binary archive
curl -fsSL -o helm.tar.gz "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz"
# Unpack archive
tar -zxvf helm.tar.gz
# Move binary to path
sudo mv linux-amd64/helm /usr/local/bin/helm
# Clean up
rm -rf linux-amd64 helm.tar.gz
Option 2: Installing via Official Script
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
Option 3: Package Managers
- Debian/Ubuntu:
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
- RHEL / Rocky Linux / Fedora:
sudo dnf install helm
Verifying Installation and Identifying Paths
Run the version command to verify installation:
helm version
Output:
version.BuildInfo{Version:"v3.16.2", GitCommit:"1365421ee17c5b6ed5009108e73f2a5aa0277ac7", GitTreeState:"clean", GoVersion:"go1.22.7"}
Client Configuration File and Directory Locations
| Directory / File | Path | Purpose |
|---|---|---|
| Cache Directory | ~/.cache/helm/ |
Stores downloaded chart repositories, cached index.yaml files, and unpacked OCI artifacts. |
| Config Directory | ~/.config/helm/ |
Contains global configuration files, including repository indices (repositories.yaml). |
| Data / Plugins Directory | ~/.local/share/helm/ or ~/.data/helm/ |
Stores installed Helm CLI plugins (e.g., helm-diff, helm-unittest). |
| Binary Location | /usr/local/bin/helm or /usr/bin/helm |
System-wide CLI executable. |
5. Configuration Files and Directory Structure
Helm Chart Directory Anatomy
A standard Helm chart follows a strict layout structure. Run helm create mychart to generate an initial scaffold:
mychart/
├── Chart.yaml # Metadata: chart version, app version, dependencies
├── values.yaml # Default configuration values for templates
├── values.schema.json # Optional: JSON Schema to validate values.yaml structure
├── charts/ # Directory containing subchart dependencies
├── crds/ # Custom Resource Definitions installed prior to templates
├── templates/ # Directory of Go-templated Kubernetes manifest files
│ ├── NOTES.txt # Post-installation instructions displayed to user
│ ├── _helpers.tpl # Reusable Go template snippets and named templates
│ ├── deployment.yaml # Kubernetes Deployment manifest template
│ ├── service.yaml # Kubernetes Service manifest template
│ ├── ingress.yaml # Kubernetes Ingress manifest template
│ └── tests/ # Chart test pods run via `helm test`
└── .helmignore # Patterns to ignore when packaging the chart
Key Configuration Files Explained
1. Chart.yaml Reference
The Chart.yaml file contains the top-level metadata for the chart:
apiVersion: v2
name: myapp
description: Production-grade Web Application Helm Chart
type: application
version: 1.2.0 # Chart semantic version (incremented on chart changes)
appVersion: "2.1.0" # Version of the underlying application workload
dependencies: # External subchart dependencies
- name: postgresql
version: 12.5.2
repository: "https://charts.bitnami.com/bitnami"
condition: postgresql.enabled
2. values.yaml Reference
The values.yaml file specifies default parameter values injected into the templates/ directory during rendering:
# values.yaml sample
replicaCount: 3
image:
repository: myapp
tag: "2.1.0"
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 80
resources:
limits:
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
3. Go Template Syntax in templates/
In manifest templates under templates/, parameters are accessed using dot notation wrapped in Go template delimiters ({{ .Values... }} or {{ .Release... }}):
# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-myapp
labels:
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- containerPort: {{ .Values.service.port }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
Configuration Precedence and Overrides
When Helm renders a release, configuration sources are merged according to a strict priority hierarchy. Higher-priority sources completely override matching keys from lower-priority sources:
LOWEST PRIORITY HIGHEST PRIORITY
[ Chart defaults (values.yaml) ] --> [ -f File 1 ] --> [ -f File 2 ] --> [ --set key=val ]
- **Chart
values.yaml**: Default values compiled inside the chart package. - First Values File (
-f values-base.yaml): User-supplied values file. - Subsequent Values Files (
-f values-prod.yaml): Evaluated left-to-right; matching keys in later files override earlier ones. - Inline Set Flags (
--set key=valueor--set-file): Takes ultimate precedence over all files.
6. Service Management and Common Operations
Because Helm 3 is a client-side utility, operational administration revolves around managing releases within Kubernetes namespaces rather than managing systemd daemons.
Managing Releases
The core lifecycle commands for Helm releases are detailed below:
# 1. Installing a release
helm install my-web ./mychart -n production --create-namespace
# 2. Idempotent installation or upgrade (Recommended for CI/CD)
helm upgrade --install my-web ./mychart -f values-prod.yaml -n production
# 3. Checking release status
helm status my-web -n production
# 4. Listing revisions in release history
helm history my-web -n production
# 5. Rolling back to a previous revision
helm rollback my-web 2 -n production
# 6. Uninstalling a release
helm uninstall my-web -n production
Distinguishing Operational Actions
| Operation | Action Taken | Operational Impact | SRE Recommendation |
|---|---|---|---|
helm upgrade |
Renders new manifest, compares against release state, applies patch to cluster objects, creates new revision ($N+1$). | Modifies running workloads in-place. | Always run with --dry-run=server or execute helm template first. |
helm rollback |
Takes manifest state from a prior historical revision ($M$) and applies it as a brand-new revision ($N+1$). | Immediately restores previous configuration and image tags. | Primary recovery mechanism when upgrades introduce application regressions. |
helm uninstall |
Queries stored release secrets, deletes all tracked Kubernetes objects from the cluster, and purges release metadata. | Removes workloads, services, and associated objects. | Objects marked with policy keep (such as PVCs) or unmanaged CRDs will remain. |
7. Common Administrative Tasks and Workflows
Workflow 1: Multi-Environment Deployment Pipeline
In production environments, SREs maintain distinct configuration files for each environment (values-staging.yaml, values-prod.yaml) while deploying the exact same immutable chart version.
# Deploy to Staging
helm upgrade --install payment-api ./charts/payment-service \
--namespace staging \
--create-namespace \
-f ./environments/values-staging.yaml
# Deploy to Production with set override for replicas
helm upgrade --install payment-api ./charts/payment-service \
--namespace production \
--create-namespace \
-f ./environments/values-prod.yaml \
--set replicaCount=5
Workflow 2: Safe Preview and Verification (Dry-Run Inspection)
Before applying upgrades to critical production environments, inspect the exact rendered output to prevent unintended changes.
# Client-side template render (does not talk to Kube API)
helm template payment-api ./charts/payment-service -f values-prod.yaml > rendered-output.yaml
# Server-side dry-run (validates rendered manifests against live API schemas and admission controllers)
helm upgrade --install payment-api ./charts/payment-service \
-f values-prod.yaml \
--dry-run=server \
-n production
Workflow 3: Subchart and Dependency Management
When a chart depends on third-party subcharts (e.g., Bitnami PostgreSQL):
# 1. Declare dependencies in Chart.yaml
# 2. Fetch and vendor dependencies into the charts/ directory
helm dependency update ./mychart
# 3. Verify fetched subchart archives (.tgz) inside charts/
ls -l ./mychart/charts/
*Overriding subchart values in parent values.yaml*:
Subchart values are overridden by nesting parameters under the subchart name key:
# Parent values.yaml
replicaCount: 2
# Subchart override key (matches subchart name in Chart.yaml)
postgresql:
auth:
database: app_db
username: app_user
primary:
resources:
limits:
memory: 512Mi
Workflow 4: Working with OCI Registries
Modern Helm charts are distributed using OCI compliant container registries (e.g., Harbor, GHCR, AWS ECR, GCP Artifact Registry).
# 1. Authenticate with the OCI registry
helm registry login registry.example.com -u sysadmin --password-stdin < /etc/ci-tokens/registry.key
# 2. Package local chart into tarball archive
helm package ./mychart
# Output: mychart-1.2.0.tgz
# 3. Push chart package to OCI registry
helm push mychart-1.2.0.tgz oci://registry.example.com/charts/infrastructure
# 4. Install chart directly from OCI URL
helm install my-app oci://registry.example.com/charts/infrastructure/mychart --version 1.2.0 -n production
8. CLI Command Reference and Cheatsheet
Repository and Registry Operations
| Command | Purpose | Important Flags / Arguments | Example | Explanation |
|---|---|---|---|---|
helm repo add |
Add remote HTTP chart repository. | [NAME] [URL] |
helm repo add bitnami [https://charts.bitnami.com/bitnami](https://charts.bitnami.com/bitnami) |
Registers a classic HTTP repository locally. |
helm repo update |
Fetch latest repository index files. | None | helm repo update |
Refreshes local repository cache to find new chart versions. |
helm search repo |
Search local repository index for keyword. | [KEYWORD] |
helm search repo nginx |
Searches cached repositories for available charts. |
helm search hub |
Search Artifact Hub public catalog. | [KEYWORD] |
helm search hub redis |
Searches global web index (artifacthub.io). |
helm registry login |
Authenticate with OCI container registry. | -u, --password-stdin |
helm registry login registry.example.com |
Logs into OCI registry for push/pull operations. |
Release Lifecycle Operations
| Command | Purpose | Important Flags / Arguments | Example | Explanation |
|---|---|---|---|---|
helm install |
Install a chart into cluster. | [RELEASE] [CHART], -n, --values, --set |
helm install web bitnami/nginx -n prod |
Installs a new release; fails if release exists. |
helm upgrade |
Upgrade an existing release. | --install, -f, --set, --wait |
helm upgrade --install web bitnami/nginx -n prod |
Idempotent command; installs if missing, upgrades if present. |
helm rollback |
Revert release to a revision. | [RELEASE] [REVISION], -n |
helm rollback web 2 -n prod |
Restores manifests from target historical revision. |
helm uninstall |
Uninstall release & remove objects. | [RELEASE], -n, --keep-history |
helm uninstall web -n prod |
Removes release components from target namespace. |
Release State and Inspection
| Command | Purpose | Important Flags / Arguments | Example | Explanation |
|---|---|---|---|---|
helm list |
List releases in namespace. | -n, -A (all namespaces), -a (all states) |
helm list -A |
Displays release names, status, revisions, and app versions. |
helm status |
Show current release status. | [RELEASE], -n |
helm status web -n prod |
Prints release notes, last deployed time, and status. |
helm history |
List historical release revisions. | [RELEASE], -n |
helm history web -n prod |
Shows table of all revisions with status and descriptions. |
helm get values |
Show values applied to release. | [RELEASE], -n, --all |
helm get values web -n prod |
Prints user-supplied override values for the release. |
helm get manifest |
Retrieve deployed YAML manifests. | [RELEASE], -n |
helm get manifest web -n prod |
Dumps rendered manifests stored in the cluster release secret. |
Chart Development and Inspection
| Command | Purpose | Important Flags / Arguments | Example | Explanation |
|---|---|---|---|---|
helm create |
Scaffold new local chart directory. | [NAME] |
helm create mychart |
Generates directory layout with boilerplate templates. |
helm lint |
Check chart for syntax/style issues. | [PATH] |
helm lint ./mychart |
Validates YAML syntax and required fields in Chart.yaml. |
helm template |
Render templates locally. | [RELEASE] [PATH], -f, --debug |
helm template test ./mychart --debug |
Prints raw rendered YAML to stdout without contacting cluster. |
helm package |
Package chart into tar archive. | [PATH] |
helm package ./mychart |
Creates a compressed .tgz file named <name>-<version>.tgz. |
helm test |
Execute test hooks in chart. | [RELEASE], -n |
helm test web -n prod |
Runs test pods defined in chart to verify functionality. |
Kubectl and Diagnostic Helper Commands
# Query pods associated with a Helm release using standard recommended labels
kubectl get pods -n production -l app.kubernetes.io/instance=web
# Stream logs across all pods owned by a Helm release (requires stern krew plugin)
stern -l app.kubernetes.io/instance=web -n production
# View object ownership dependency tree for a deployment (requires kubectl-tree plugin)
kubectl tree deploy -l app.kubernetes.io/instance=web -n production
9. Logs, Monitoring and Diagnostics
Locating Logs and Diagnostic Data
Helm itself is stateless and client-side; it does not generate centralized system log files on server nodes. Diagnostic information exists in three locations:
- CLI Debug Output: Executing commands with the
--debugflag forces Helm to output verbose internal operations, Go template stack traces, and raw Kube API HTTP requests/responses. - Kubernetes Release Secrets: Stored in the target namespace as gzipped base64 data.
- Workload & Hook Pod Logs: Diagnostic data for failing installs/upgrades resides within the execution logs of Kubernetes Pods and Jobs.
# Inspecting Helm release storage secrets directly
kubectl get secrets -n production -l owner=helm
Output:
NAME TYPE DATA AGE
sh.helm.release.v1.my-web.v1 helm.sh/release.v1 1 5d2h
sh.helm.release.v1.my-web.v2 helm.sh/release.v1 1 12h
Decoding Release Metadata Secret Content
If a release becomes corrupted and Helm CLI cannot parse it, inspect raw release data manually:
kubectl get secret sh.helm.release.v1.my-web.v2 -n production -o jsonpath='{.data.release}' | base64 -d | base64 -d | gunzip | jq .
Systemic Diagnostics Workflow
+-----------------------------------------------------------------------+
| HELM DIAGNOSTIC WORKFLOW |
| |
| STEP 1: Verify Release State |
| $ helm list -n NS |
| $ helm status RELEASE -n NS |
| $ helm history RELEASE -n NS |
| |
| STEP 2: Inspect Effective Manifests & Values |
| $ helm get manifest RELEASE -n NS | less |
| $ helm get values RELEASE -n NS |
| |
| STEP 3: Inspect Cluster Workload Health |
| $ kubectl get pods -n NS -l app.kubernetes.io/instance=RELEASE |
| $ kubectl describe pod -n NS -l app.kubernetes.io/instance=RELEASE |
| $ stern -l app.kubernetes.io/instance=RELEASE -n NS |
+-----------------------------------------------------------------------+
10. Troubleshooting Guide and Common Failure Scenarios
Problem 1: Install or Upgrade Failed (Exit Non-Zero)
Symptoms
helm install or helm upgrade exits with a non-zero status code. The terminal displays errors such as Error: UPGRADE FAILED: timed out waiting for the condition or Error: release failed: deployment failed.
Error: UPGRADE FAILED: release payment-api failed and has been rolled back: timed out waiting for the condition
Possible Causes
- Workload CrashLoopBackOff: Application pods failed startup or readiness probes during installation.
- Insufficient Resource Quotas: Cluster lacks CPU, Memory, or PersistentVolume capacity to schedule pods.
- Image Pull Errors: Invalid container registry image path, missing image tag, or missing
imagePullSecrets.
Investigation and Diagnosis
- Check the recorded release status and revision history:
helm status payment-api -n production
helm history payment-api -n production
- Retrieve rendered manifests deployed to the cluster to confirm image tags and resource specs:
helm get manifest payment-api -n production | grep -E "image:|memory:|cpu:"
- Query all pods associated with the Helm release instance label:
kubectl get pods -n production -l app.kubernetes.io/instance=payment-api
- Describe failing pods to review Kubernetes event messages:
kubectl describe pod -n production -l app.kubernetes.io/instance=payment-api
- Tail pod application logs:
stern -l app.kubernetes.io/instance=payment-api -n production
Root Cause Identification
- If
kubectl describe poddisplaysImagePullBackOff, the registry path or credential is missing. - If pods show
CrashLoopBackOff, inspect stdout logs viasternto diagnose application code startup errors.
Resolution
Fix the underlying configuration file or container image tag, then re-run the idempotent upgrade command:
helm upgrade --install payment-api ./charts/payment-service -f values-prod.yaml -n production
Verification
Confirm that the release status is deployed and all pods report Running and Ready status:
helm status payment-api -n production
kubectl get pods -n production -l app.kubernetes.io/instance=payment-api
Prevention
- Always test chart renders using
helm template --debugbefore executing deployments. - Include standard readiness and liveness probes with appropriate delay parameters in templates.
Problem 2: Release Stuck in pending-install or pending-upgrade
Symptoms
Subsequent execution of helm upgrade fails with Error: another operation (install/upgrade/rollback) is in progress. helm list shows status pending-install or pending-upgrade.
Error: HELM UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Possible Causes
- Interrupted CLI Process: The
helmcommand-line process was terminated abruptly (e.g., CI/CD pipeline timeout,SIGKILL, or network loss) during execution. - Hanging Lifecycle Hook: A pre-install or pre-upgrade hook Job became stuck or blocked indefinitely, preventing Helm from finalizing the secret record.
Investigation and Diagnosis
- Inspect release history to identify the stuck revision number:
helm history payment-api -n production
- Check for hanging hook pods or jobs:
kubectl get pods,jobs -n production -l helm.sh/hook
- Check release secrets directly in the namespace:
kubectl get secret -n production -l owner=helm
Root Cause Identification
If helm history shows the latest revision in pending-upgrade and no local CLI process is running, Helm’s release secret state lock was left unreleased.
Resolution
- Method 1: Manual Rollback (Preferred): Roll back to the last known good revision:
helm rollback payment-api 3 -n production
- Method 2: Deleting Stuck Secret (Last Resort):
If
helm rollbackrefuses to run due to the pending lock, remove the pending revision’s secret directly:
# Find stuck revision secret (e.g., revision 4)
kubectl delete secret sh.helm.release.v1.payment-api.v4 -n production
# Retry upgrade command
helm upgrade --install payment-api ./charts/payment-service -f values-prod.yaml -n production
Verification
Run helm list -n production and ensure status has changed from pending-upgrade to deployed.
Prevention
- Implement timeouts in CI/CD pipelines using
--timeout 10m0sflag. - Ensure hook Jobs contain
activeDeadlineSecondsin their pod specification so Kubernetes terminates stuck hook jobs automatically.
Problem 3: Unexpected Configuration Applied After Upgrade (Values Precedence Issue)
Symptoms
An application behaves incorrectly after an upgrade (e.g., wrong database endpoint or replica count) despite passing a custom values file.
Possible Causes
- Values Precedence Misunderstanding: Values passed via
--setor secondary-ffiles overridden the intended configuration file. - Key Typo: A key in
values.yamlwas misspelled or placed at the wrong indentation level, causing the template to fall back to defaults.
Investigation and Diagnosis
- Export the effective values recorded in the active release revision:
helm get values payment-api -n production
- Compare the effective values against the raw source file:
diff -u <(helm get values payment-api -n production) values-prod.yaml
Root Cause Identification
If helm get values shows parameters matching --set inputs rather than values-prod.yaml, the command-line order allowed --set to override file settings.
Resolution
- Correct the execution command, ensuring the intended values file is placed last or inline
--setflags are corrected. - Roll back immediately if production is impacted:
helm rollback payment-api PREVIOUS_REV -n production
- Re-apply using the proper merge hierarchy:
helm upgrade payment-api ./charts/payment-service -f values-prod.yaml -n production
Verification
Run helm get values payment-api -n production and verify all key-value pairs match expectations.
Prevention
Maintain environment-specific settings inside dedicated, version-controlled values files (values-staging.yaml, values-prod.yaml) rather than passing long strings of --set flags in CI scripts.
Problem 4: Chart Not Found or Repository Synchronization Failure
Symptoms
Executing helm install returns Error: chart "nginx" not found in bitnami repository or Error: failed to download "oci://...".
Error: chart "payment-service" matching 1.2.0 not found in myrepo repository (have targets updated?)
Possible Causes
- Stale Local Repository Cache: The local repository index is out of date.
- Missing Version Tag in OCI Registry: The specified version does not exist in the OCI container registry.
- Air-Gapped Network Isolation: The cluster or workstation cannot reach external HTTP chart repositories.
Investigation and Diagnosis
- Search local repository cache for available versions:
helm search repo payment-service --versions
- Verify repository configuration list:
helm repo list
Resolution
- For Standard Repositories: Update repository index cache:
helm repo update
- For OCI Repositories: Verify OCI path syntax and explicitly append the required
--versionflag:
helm install my-app oci://registry.example.com/charts/payment-service --version 1.2.0 -n production
- For Air-Gapped Environments: Download or package the chart locally into a
.tgzarchive and install from the local tarball file path:
helm install payment-api ./payment-service-1.2.0.tgz -n production
Verification
Run helm search repo <chart-name> to verify that the chart version is indexed and installable.
Problem 5: Template Rendering Error (Nil Pointer, YAML Syntax, Indentation Failure)
Symptoms
Executing helm install or helm template outputs Go template execution errors or YAML parsing errors:
Error: YAML parse error on payment-service/templates/deployment.yaml: error converting YAML to JSON: yaml: line 18: mapping values are not allowed in this context
or
Error: template: payment-service/templates/deployment.yaml:24:14: executing "payment-service/templates/deployment.yaml" at <.Values.database.port>: nil pointer evaluating interface {}
Possible Causes
- Nil Pointer Dereference: The template references
.Values.database.port, butdatabaseis undefined invalues.yaml. - Indentation Misalignment: Go template helper pipelines (e.g.,
nindent) injected YAML blocks at incorrect column offsets.
Investigation and Diagnosis
Execute helm template with --debug flag to display rendered YAML up to the failure point:
helm template test-release ./charts/payment-service -f values-prod.yaml --debug
Root Cause Identification
The --debug output displays the line number where rendering stopped. Inspect the generated output immediately above the error line to locate malformed YAML or missing key references.
Resolution
- Fix missing keys in
values.yamlor wrap template access in conditional checks:
{{- if .Values.database }}
port: {{ .Values.database.port }}
{{- end }}
- Run
helm lintto validate chart syntax:
helm lint ./charts/payment-service
Verification
Ensure helm template ./charts/payment-service renders cleanly without error messages.
Problem 6: RBAC / Forbidden Errors During Install or Upgrade
Symptoms
Helm returns API authorization errors during execution:
Error: INSTALL FAILED: deployments.apps is forbidden: User "ci-service-account" cannot create resource "deployments" in API group "apps" in the namespace "production"
Possible Causes
The user identity or ServiceAccount configured in kubeconfig lacks RBAC permissions to manage specific Kubernetes resource types (Deployments, CRDs, Roles, Ingress) within the target namespace.
Investigation and Diagnosis
Test API permissions directly using kubectl auth can-i:
# Check permission to create deployments in target namespace
kubectl auth can-i create deployments -n production --as=system:serviceaccount:ci-cd:helm-runner
# Check permission to create CRDs cluster-wide
kubectl auth can-i create crds --as=system:serviceaccount:ci-cd:helm-runner
Resolution
Bind an appropriate Role or ClusterRole to the executing ServiceAccount or user identity via RoleBinding or ClusterRoleBinding:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: helm-runner-binding
namespace: production
subjects:
- kind: ServiceAccount
name: helm-runner
namespace: ci-cd
roleRef:
kind: ClusterRole
name: admin
apiGroup: rbac.authorization.k8s.io
Verification
Re-run kubectl auth can-i create deployments -n production, verifying that the output prints yes.
Problem 7: Pre-install / Post-upgrade Lifecycle Hook Failure
Symptoms
Deployment hangs or fails during execution, displaying Error: Hook pre-install payment-service/templates/job-db-migrate.yaml failed: Job failed.
Possible Causes
The Kubernetes Job or Pod generated by the hook template failed during execution (e.g., database connection failure during schema migration).
Investigation and Diagnosis
- Query hook jobs and pods in the target namespace:
kubectl get jobs,pods -n production -l helm.sh/hook
- Inspect logs of the failing hook job pod:
kubectl logs job/payment-api-db-migrate -n production
Resolution
- Resolve the root cause in the hook script or container payload (e.g., fix database credentials).
- Manually delete the failed hook job so Helm can recreate it on retry:
kubectl delete job payment-api-db-migrate -n production
- Re-run
helm upgrade --install. - Emergency Bypass: If a non-critical hook blocks production recovery, bypass hooks temporarily using
--no-hooks(use with extreme caution):
helm upgrade payment-api ./charts/payment-service --no-hooks -n production
Verification
Confirm that hook pods exit with status Completed (Exit Code 0) and the release updates to deployed.
Problem 8: Resources Remaining in Cluster After Uninstall
Symptoms
Executing helm uninstall payment-api -n production succeeds, but kubectl get pvc or kubectl get crd still lists resources previously created by the release.
Possible Causes
- Resource Policy Annotation: Resources annotated with
helm.sh/resource-policy: keepare explicitly ignored by Helm during uninstallation. - Custom Resource Definitions (CRDs): Helm intentionally does not automatically delete CRDs installed from the
crds/directory to prevent accidental data loss across shared cluster workloads.
Investigation and Diagnosis
Check object annotations for resource keep policies:
kubectl get pvc -n production -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.helm\.sh/resource-policy}{"\n"}{end}'
Resolution
If the remaining PVCs or CRDs are no longer required, delete them manually via kubectl:
# Delete orphaned PersistentVolumeClaims
kubectl delete pvc -l app.kubernetes.io/instance=payment-api -n production
# Delete custom resource definitions if safe
kubectl delete crd paymentcustomresources.example.com
Prevention
Document resource cleanup policies in operational runbooks so SREs know which storage volumes require manual lifecycle management.
Problem 9: Pods Healthy and Running but Application Inaccessible
Symptoms
helm status reports deployed and all pods show Running 1/1 status. However, HTTP traffic to the application endpoint fails with 502 Bad Gateway or Connection Refused.
Possible Causes
This is an application routing or environment variable configuration error rather than a Helm release failure. Common issues include incorrect Service target ports, missing ConfigMap values, or misconfigured Ingress controllers.
Investigation and Diagnosis
- Inspect deployed values stored in the release record:
helm get values payment-api -n production
- Verify environment variables and mounted volume files inside the running container:
kubectl exec -it deployment/payment-api -n production -- env
- Test direct container access by forwarding pod ports locally:
kubectl port-forward pod/payment-api-7b89d-x4z2 8080:8080 -n production
Resolution
- If port-forwarding succeeds, the failure lies in
Serviceport mapping orIngressrouting rules. Alignservice.portandtargetPortparameters invalues.yaml. - Upgrade the release with corrected routing values:
helm upgrade payment-api ./charts/payment-service -f values-prod.yaml -n production
Verification
Send an HTTP test request to the ingress URL or Service IP and confirm 200 OK response.
11. Security and Best Practices
1. Managing Sensitive Data in Helm Values
Never commit plaintext passwords, API tokens, or TLS private keys into Git repositories inside values.yaml files.
- Sealed Secrets: Encrypt secrets using Bitnami
SealedSecretsso encrypted values can be safely stored in Git and decrypted only by the cluster controller. - **SOPS &
helm-secrets**: Use Mozilla SOPS withhelm-secretsplugin to decrypt encrypted values files transparently during CI/CD execution:
helm secrets upgrade --install payment-api ./charts/payment-service -f secrets.prod.yaml -n production
- External Secrets Operator: Reference external secret providers (AWS Secrets Manager, HashiCorp Vault) using
ExternalSecretmanifests inside the chart templates.
2. Enforcing Least-Privilege RBAC for CI/CD Automated Pipelines
Avoid executing Helm pipelines using global cluster-admin access. Grant CI/CD ServiceAccounts namespace-scoped Roles restricted to the exact API groups required by the chart (Deployments, Services, ConfigMaps, Ingress).
3. Securing Helm Release Metadata Secrets
Because Helm 3 stores release history as Kubernetes Secrets labeled with owner=helm, any user with get and list permissions on secrets in a namespace can read raw release manifests and effective values. Restrict access using Kubernetes RBAC rules:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: prevent-release-secrets-access
namespace: production
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
resourceNames: [] # Restrict secret access explicitly
4. Production Chart Management Best Practices
- Version Pinning: Always pin explicit chart versions (
--version 1.2.3) in production deployment scripts. Never rely on floating tags or unpinned release installations. - Chart Provenance & Integrity: Sign charts using GPG keys during packaging (
helm package --sign) and enforce signature validation during installation using--verify. - Private OCI Registries: Store internal enterprise charts in private OCI container registries with vulnerability scanning enabled.
12. Performance and Resource Optimisation
Managing Release Secret Accumulation
By default, Helm retains historical revision records indefinitely. Over months of automated deployment, hundreds of release secrets (sh.helm.release.v1.app.v1 … v300) accumulate in Kubernetes namespaces, increasing API server memory usage and slowing down namespace operations.
Optimization: Limit maximum recorded release history using --history-max during upgrades:
helm upgrade --install payment-api ./charts/payment-service \
-f values-prod.yaml \
--history-max 10 \
-n production
Network Bandwidth: OCI Registries vs. Legacy Repositories
Legacy HTTP chart repositories rely on a single monolithic index.yaml file listing all available chart packages. For large enterprise repositories containing thousands of chart versions, downloading index.yaml consumes megabytes of bandwidth per client update.
Switching to OCI Registries completely eliminates index.yaml overhead. Clients pull only the exact chart version package required (oci://[registry.example.com/charts/myapp:1.2.0](https://registry.example.com/charts/myapp:1.2.0)), dramatically reducing bandwidth consumption and accelerating CI/CD pipeline execution times.
13. Backup, Recovery and Maintenance
Backing Up Helm Release State
Because Helm stores release state as standard Kubernetes Secrets in the application’s namespace, backing up release metadata requires exporting the matching secret objects:
# Export all Helm release secrets across the cluster
kubectl get secrets -A -l owner=helm -o yaml > helm-releases-backup.yaml
Recovering a Corrupted Release Metadata Secret
If a release secret is accidentally deleted or corrupted, Helm lose tracking of that application instance. To recover:
- Extract the manifest from the latest working backup or render the current manifest manually:
helm template payment-api ./charts/payment-service -f values-prod.yaml -n production > current-manifest.yaml
- Re-import the release state without modifying live cluster resources using
--forceor standard idempotent upgrade:
helm upgrade --install payment-api ./charts/payment-service -f values-prod.yaml -n production
Maintaining Chart Repositories
Periodically purge obsolete chart packages from local caches and remote OCI registries to reclaim storage space:
# Clear local Helm repository cache
helm repo remove bitnami
rm -rf ~/.cache/helm/repository/*
14. Automation and Integration
CI/CD Pipeline Automation (GitLab CI Example)
An idempotent, robust automated pipeline step for deploying Helm charts to Kubernetes:
# .gitlab-ci.yml
deploy_production:
stage: deploy
image: alpine/helm:3.16.2
script:
- helm registry login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
- helm lint ./charts/payment-service
- helm upgrade --install payment-api oci://$CI_REGISTRY/charts/payment-service \
--version $CI_COMMIT_TAG \
--namespace production \
--create-namespace \
-f ./environments/values-prod.yaml \
--history-max 10 \
--atomic \
--timeout 5m0s
only:
- tags
GitOps Integration (Argo CD / Flux)
In modern SRE GitOps workflows, Helm charts are rendered and managed declaratively by GitOps controllers running inside the cluster.
Example Argo CD Application manifest referencing a Helm chart:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: payment-api-production
namespace: argocd
spec:
project: default
source:
repoURL: 'https://charts.bitnami.com/bitnami'
chart: nginx
targetRevision: 15.4.0
helm:
valueFiles:
- values-production.yaml
destination:
server: 'https://kubernetes.default.svc'
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
15. Practical Labs and Exercises
Lab 1 (Beginner): Deploying and Customizing an Application with Helm
Objective
Install an NGINX web server release using Helm, customize its configuration using an override file, and verify workload operation.
Environment & Prerequisites
- Access to a running Kubernetes cluster (Minikube, Kind, or cloud cluster).
- Local Helm 3 CLI installed.
Instructions
- Add the Bitnami public chart repository:
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
- Create a custom values file named
lab1-values.yaml:
replicaCount: 2
service:
type: ClusterIP
- Deploy the release named
my-nginxinto namespacelab1:
helm install my-nginx bitnami/nginx -f lab1-values.yaml -n lab1 --create-namespace
- Verify release status and running pods:
helm status my-nginx -n lab1
kubectl get pods -n lab1 -l app.kubernetes.io/instance=my-nginx
Validation
Verify that exactly two NGINX pods are running in the lab1 namespace.
Lab 2 (Intermediate): Building a Multi-Tier Chart with Subchart Dependencies
Objective
Create a custom application chart (web-app), declare a PostgreSQL subchart dependency, and configure subchart values from the parent chart.
Instructions
- Scaffold a new chart:
helm create web-app
- Edit
web-app/Chart.yamlto declare PostgreSQL as a dependency:
dependencies:
- name: postgresql
version: 12.5.2
repository: https://charts.bitnami.com/bitnami
- Fetch the subchart dependency:
helm dependency update ./web-app
- Update
web-app/values.yamlto configure the subchart database parameters:
postgresql:
auth:
database: user_data
username: db_admin
- Perform a dry-run render to verify that both web application and database manifests render cleanly:
helm template test-stack ./web-app --debug
Validation
Inspect stdout to confirm that both Deployment (web-app) and StatefulSet (postgresql) manifests are generated in the output stream.
Lab 3 (Troubleshooting): Recovering a Stuck Release
Objective
Diagnose a release stuck in pending-upgrade state caused by an interrupted deployment and recover the release safely.
Instructions
- Inspect the stuck release status:
helm list -n production
helm history payment-api -n production
- Observe that revision 4 is stuck in
pending-upgrade. - Identify the release secrets stored in the namespace:
kubectl get secret -n production -l owner=helm
- Delete the pending secret record (revision 4):
kubectl delete secret sh.helm.release.v1.payment-api.v4 -n production
- Roll back release state to revision 3:
helm rollback payment-api 3 -n production
Validation
Run helm status payment-api -n production and confirm status shows deployed at revision 5.
16. Quick Reference and Cheat Sheet
Important Configuration Files & Locations
| Path | Purpose |
|---|---|
Chart.yaml |
Main chart metadata, semantic version, app version, and subchart dependencies. |
values.yaml |
Default configuration parameter values for templates. |
templates/ |
Directory containing Go-templated Kubernetes manifest files. |
templates/NOTES.txt |
Post-installation operational instructions displayed to stdout. |
~/.config/helm/repositories.yaml |
Local index of configured HTTP chart repositories. |
Release Secret Naming Format
| Pattern | Example | Purpose |
|---|---|---|
sh.helm.release.v1.<RELEASE>.<REVISION> |
sh.helm.release.v1.payment-api.v2 |
In-cluster storage secret containing gzipped release metadata. |
Common Administrative Commands
| Task | Command |
|---|---|
| Add Repository | helm repo add [NAME] [URL] |
| Update Repository Index | helm repo update |
| Install Release | helm install [RELEASE] [CHART] -n [NS] --create-namespace |
| Idempotent Upgrade/Install | helm upgrade --install [RELEASE] [CHART] -f values.yaml -n [NS] |
| Set Inline Value Override | helm upgrade [RELEASE] [CHART] --set key=value -n [NS] |
| Rollback to Revision | helm rollback [RELEASE] [REVISION] -n [NS] |
| Uninstall Release | helm uninstall [RELEASE] -n [NS] |
| List Releases Across All Namespaces | helm list -A |
| Inspect Release History | helm history [RELEASE] -n [NS] |
| Display Deployed Values | helm get values [RELEASE] -n [NS] |
| Render Templates Locally | helm template [RELEASE] [CHART-PATH] -f values.yaml --debug |
Troubleshooting Quick Matrix
| Symptom | Initial Investigation Step | Primary Remediation |
|---|---|---|
| Upgrade Failed / Timeout | kubectl get pods -n NS -l app.kubernetes.io/instance=RELEASE |
Fix image tag/config in values.yaml & re-run helm upgrade. |
**Stuck in pending-upgrade** |
kubectl get secret -n NS -l owner=helm |
Delete stuck revision secret & execute helm rollback. |
| Template Syntax Error | helm template RELEASE CHART -f values.yaml --debug |
Fix indentation or nil pointer keys in template. |
| RBAC / Forbidden Error | kubectl auth can-i create deployments -n NS |
Grant RoleBinding permissions to executing ServiceAccount. |
| Hook Job Failed | kubectl logs job/HOOK_JOB -n NS |
Fix underlying hook script; delete failed job & retry. |
| Chart Not Found | helm repo update |
Sync repo index or append --version to OCI path. |
17. Further Reading and References
- Official Helm Documentation: helm.sh/docs
- Helm Charts Guide: helm.sh - charts
- Helm Values Files Guide: helm.sh - values
- Artifact Hub Public Chart Registry: artifacthub.io
- Kubernetes Workload & Logging Reference: Kubernetes Documentation