Domain 4Kubernetes EngineeringINTERMEDIATE

Mastering Helm 3: The Complete Kubernetes Package Manager Guide for SysAdmins and SREs

A comprehensive operational guide and SRE handbook for Helm 3, covering architecture, chart development, release management, CLI diagnostics, troubleshooting, and production best practices.

Domain: Domain 4: Kubernetes Cluster Engineering•Updated: 2026-11-03
#linux#sysadmin#kubernetes#helm#sre

1. Overview

Deploying and operating production applications on Kubernetes requires coordinating a complex ecosystem of API objects. A single application stack typically requires a Kubernetes Deployment or StatefulSet for workload management, a Service for network routing, ConfigMaps and Secrets for configuration, an Ingress object for external routing, and ServiceAccount, Role, and RoleBinding objects for Role-Based Access Control (RBAC). Managing these manifests as raw, disconnected YAML files quickly becomes error-prone, leading to configuration drift, complex deployment pipelines, and fragile rollback procedures.

Helm solves this operational challenge by acting as the package manager for Kubernetes. Helm bundles related Kubernetes manifests into a single, versioned unit of software known as a chart. A chart combines Go-templated YAML files with default configuration parameters stored in a values.yaml file.

+-------------------------------------------------------------------+
|                            HELM CHART                             |
|                                                                   |
|  +--------------------+  +------------------+  +---------------+  |
|  |    Chart.yaml      |  |   values.yaml    |  |  templates/   |  |
|  | Metadata & Version |  | Defaults Config  |  | Deployment.yaml |
|  +--------------------+  +------------------+  | Service.yaml  |  |
|                                                | Ingress.yaml  |  |
|                                                +---------------+  |
+-------------------------------------------------------------------+
                                  |
                                  v  (helm install / upgrade)
+-------------------------------------------------------------------+
|                        KUBERNETES RELEASE                         |
|                                                                   |
|  [ Deployment ]      [ Service ]      [ ConfigMap ]   [ Secret ]  |
+-------------------------------------------------------------------+

Core Problems Solved by Helm

  • Parameterization & Reusability: Instead of duplicating raw YAML manifests across development, staging, and production clusters, Helm allows administrators to parameterize values (e.g., replica counts, image tags, resource requests) and apply environment-specific configuration files.
  • Atomic Release Management: Helm installs or upgrades an entire application stack with a single command, automatically managing standard labels and metadata across all instantiated objects.
  • Revision Tracking & Rollbacks: Every modification to an application creates an incremental release revision. If an upgrade fails or introduces bugs, Helm provides single-command rollbacks to any previous working state.
  • Dependency Orchestration: Applications that depend on supporting infrastructure (such as databases or caching layers) can declare subchart dependencies, allowing Helm to resolve, fetch, and deploy the full application hierarchy in sequence.

Comparison: Raw Manifests vs. Kustomize vs. Helm

Feature Raw kubectl Manifests Kustomize Helm 3
Templating Mechanism None (Static files) Overlay / Patching (Template-free) Go Templating (.Values, .Release)
State & Release Tracking None (Cluster objects only) None (Cluster objects only) Native Release Secrets in cluster
Rollback Mechanism Manual kubectl rollout per workload Manual patch re-application Single-command helm rollback
Package Distribution Git repos / File archives Git repos HTTP Repositories & OCI Registries
Lifecycle Hooks None Limited Native lifecycle hooks (pre-install, etc.)

2. Architecture and Mental Model

To effectively administer and troubleshoot Helm, SREs must understand the architectural transition from Helm 2 to Helm 3 and how Helm interacts with the Kubernetes control plane.

Helm 2 vs. Helm 3 Architecture

In Helm 2, the system relied on an in-cluster server component named Tiller. Tiller ran inside the Kubernetes cluster with elevated privileges (often cluster-admin), acting as an intermediary that accepted commands from the local Helm CLI, rendered templates, and applied them directly to the API server. This introduced severe security risks and complex RBAC delegation issues.

Helm 3 completely eliminated Tiller. Helm 3 operates strictly as a client-side binary. It renders templates locally on the administrator’s workstation or CI/CD worker node and communicates directly with the Kubernetes API server using the active credentials in the user’s kubeconfig file.

+-----------------------------------------------------------------------+
|                             LOCAL WORKSTATION / CI                    |
|                                                                       |
|  +------------------+     +-------------------+     +--------------+  |
|  |   Chart Templates| +   | User values.yaml  | --> |  Helm 3 CLI  |  |
|  +------------------+     +-------------------+     +--------------+  |
+-----------------------------------------------------------|-----------+
                                                            |
                                        Direct KubeAPI Call | (Using user's kubeconfig context)
                                                            v
+-----------------------------------------------------------------------+
|                          KUBERNETES CLUSTER                           |
|                                                                       |
|  +-----------------------------------------------------------------+  |
|  |                      Kubernetes API Server                      |  |
|  +-----------------------------------------------------------------+  |
|                                   |                                   |
|            +----------------------+----------------------+            |
|            |                                             |            |
|            v                                             v            |
|  +-------------------+                         +-------------------+  |
|  | Application Stack |                         |   Release Storage |  |
|  | (Deployment, Pod, |                         | (Kubernetes Secret|  |
|  | Service, Ingress) |                         |  `owner=helm`)    |  |
|  +-------------------+                         +-------------------+  |
+-----------------------------------------------------------------------+

Components and Responsibilities

  1. Helm Client (CLI): A local command-line binary that manages chart repositories, builds packages, merges configurations, parses Go templates into raw YAML, and invokes API operations.
  2. Kubernetes API Server: The cluster endpoint that receives rendered manifests via standard HTTP/REST requests and enforces cluster RBAC.
  3. Release Storage Backend: Helm 3 stores release metadata directly within the Kubernetes cluster as Secrets (default) or ConfigMaps in the namespace where the release is installed. These secrets carry the label owner=helm and are named following the pattern sh.helm.release.v1.<RELEASE_NAME>.v<REVISION>.

Detailed Install and Upgrade Request Lifecycle

When an administrator executes helm install or helm upgrade, Helm processes the request through four major lifecycle phases:

graph TD
    A[Execute helm install/upgrade] --> B[1. Render Phase]
    B --> B1[Load Chart.yaml & templates/]
    B --> B2[Merge values.yaml + -f files + --set overrides]
    B2 --> B3[Evaluate Go Templates]
    B3 --> C[2. Hook Evaluation & Execution]
    C --> C1{Pre-install/upgrade hooks?}
    C1 -- Yes --> C2[Execute Hook Jobs/Pods & Wait]
    C1 -- No --> D[3. Apply Phase]
    C2 -- Success --> D
    D --> D1[Transmit Rendered Manifests to Kube API]
    D1 --> D2[Perform 3-Way Strategic Merge Patch / Create]
    D2 --> E[4. Record Phase]
    E --> E1[Gzip & Base64 Encode Release State]
    E1 --> E2[Store as Secret in Target Namespace]
  1. Render Phase: Helm loads the target chart, merges default values.yaml with user-supplied override files (-f) and inline flags (--set), and runs the combined data context through the Go template engine to generate standard Kubernetes YAML manifests.
  2. Hook Phase: If the chart contains resources annotated with lifecycle hooks (e.g., helm.sh/hook: pre-install), Helm submits those hook resources (typically Kubernetes Jobs) to the API server and waits for their successful execution before proceeding.
  3. Apply Phase: Helm transmits the final rendered manifests to the Kubernetes API server. For upgrades, Helm performs a 3-way strategic merge patch comparing the current state, desired state, and previous manifest state.
  4. Record Phase: Upon successful API acceptance, Helm serializes the complete release metadata, chart contents, and effective values into a gzipped, base64-encoded secret, recording a new release revision in the target namespace.

3. Core Concepts and Terminology

Understanding Helm’s domain vocabulary is essential for effective administration.

+------------------------------------------------------------------+
|                          HELM GLOSSARY                           |
|                                                                  |
|  CHART       --> Packaging blueprint containing templates.       |
|  VALUES      --> Configuration parameters fed into templates.    |
|  RELEASE     --> Running instance of a chart in a namespace.     |
|  REVISION    --> Immutable numerical snapshot of a release.      |
|  REPOSITORY  --> Remote HTTP or OCI registry hosting charts.     |
+------------------------------------------------------------------+

Core Terms

  • Chart: An organized directory of files (or a gzipped .tgz archive) containing template files, default values, and metadata (Chart.yaml) that define a set of Kubernetes resources.
  • Release: A named, deployed instance of a chart running within a specific Kubernetes namespace. A single chart (e.g., nginx-ingress) can be installed multiple times in the same cluster under different release names (e.g., ingress-public and ingress-internal).
  • Revision: A numbered, immutable snapshot representing a specific change state of a release. Every helm install, helm upgrade, or helm rollback increments the revision counter ($1, 2, 3 \dots$).
  • Values: Parameter inputs used to customize the templates within a chart. Defaults are defined in values.yaml, which can be overridden at runtime using custom value files (-f) or command-line parameters (--set).
  • Repository: A central server location hosting packaged charts. Repositories can be traditional HTTP web servers hosting an index.yaml file or modern Open Container Initiative (OCI) registries.
  • Subchart / Dependency: A chart nested inside another chart’s charts/ directory or declared in Chart.yaml. Subcharts allow complex application stacks (e.g., a web app requiring PostgreSQL and Redis) to be managed as a unified chart.
  • Hooks: Annotations attached to resources (such as Jobs or Pods) that instruct Helm to execute them at specific points during a release lifecycle (e.g., database schema migrations before updating web deployment pods).

4. Installation and Version Management

Prerequisites and Dependencies

  • Operating System: Linux (Ubuntu, Debian, RHEL, Rocky Linux, SLES), macOS, or Windows.
  • Access Tools: kubectl installed and configured with access to a Kubernetes cluster.
  • Authentication: A valid kubeconfig file located at ~/.kube/config (or referenced via $KUBECONFIG) with sufficient RBAC privileges to create objects in target namespaces.

Installation Methods

To install a specific version of Helm 3 directly from GitHub releases:

# Set desired Helm version
HELM_VERSION="v3.16.2"

# Download binary archive
curl -fsSL -o helm.tar.gz "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz"

# Unpack archive
tar -zxvf helm.tar.gz

# Move binary to path
sudo mv linux-amd64/helm /usr/local/bin/helm

# Clean up
rm -rf linux-amd64 helm.tar.gz

Option 2: Installing via Official Script

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Option 3: Package Managers

  • Debian/Ubuntu:
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
  • RHEL / Rocky Linux / Fedora:
sudo dnf install helm

Verifying Installation and Identifying Paths

Run the version command to verify installation:

helm version

Output:

version.BuildInfo{Version:"v3.16.2", GitCommit:"1365421ee17c5b6ed5009108e73f2a5aa0277ac7", GitTreeState:"clean", GoVersion:"go1.22.7"}

Client Configuration File and Directory Locations

Directory / File Path Purpose
Cache Directory ~/.cache/helm/ Stores downloaded chart repositories, cached index.yaml files, and unpacked OCI artifacts.
Config Directory ~/.config/helm/ Contains global configuration files, including repository indices (repositories.yaml).
Data / Plugins Directory ~/.local/share/helm/ or ~/.data/helm/ Stores installed Helm CLI plugins (e.g., helm-diff, helm-unittest).
Binary Location /usr/local/bin/helm or /usr/bin/helm System-wide CLI executable.

5. Configuration Files and Directory Structure

Helm Chart Directory Anatomy

A standard Helm chart follows a strict layout structure. Run helm create mychart to generate an initial scaffold:

mychart/
├── Chart.yaml          # Metadata: chart version, app version, dependencies
├── values.yaml         # Default configuration values for templates
├── values.schema.json  # Optional: JSON Schema to validate values.yaml structure
├── charts/             # Directory containing subchart dependencies
├── crds/               # Custom Resource Definitions installed prior to templates
├── templates/          # Directory of Go-templated Kubernetes manifest files
│   ├── NOTES.txt       # Post-installation instructions displayed to user
│   ├── _helpers.tpl    # Reusable Go template snippets and named templates
│   ├── deployment.yaml # Kubernetes Deployment manifest template
│   ├── service.yaml    # Kubernetes Service manifest template
│   ├── ingress.yaml    # Kubernetes Ingress manifest template
│   └── tests/          # Chart test pods run via `helm test`
└── .helmignore         # Patterns to ignore when packaging the chart

Key Configuration Files Explained

1. Chart.yaml Reference

The Chart.yaml file contains the top-level metadata for the chart:

apiVersion: v2
name: myapp
description: Production-grade Web Application Helm Chart
type: application
version: 1.2.0        # Chart semantic version (incremented on chart changes)
appVersion: "2.1.0"   # Version of the underlying application workload
dependencies:         # External subchart dependencies
  - name: postgresql
    version: 12.5.2
    repository: "https://charts.bitnami.com/bitnami"
    condition: postgresql.enabled

2. values.yaml Reference

The values.yaml file specifies default parameter values injected into the templates/ directory during rendering:

# values.yaml sample
replicaCount: 3

image:
  repository: myapp
  tag: "2.1.0"
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80

resources:
  limits:
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi

3. Go Template Syntax in templates/

In manifest templates under templates/, parameters are accessed using dot notation wrapped in Go template delimiters ({{ .Values... }} or {{ .Release... }}):

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-myapp
  labels:
    app.kubernetes.io/name: {{ .Chart.Name }}
    app.kubernetes.io/instance: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: {{ .Release.Name }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: {{ .Values.service.port }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

Configuration Precedence and Overrides

When Helm renders a release, configuration sources are merged according to a strict priority hierarchy. Higher-priority sources completely override matching keys from lower-priority sources:

LOWEST PRIORITY                                                         HIGHEST PRIORITY
[ Chart defaults (values.yaml) ] --> [ -f File 1 ] --> [ -f File 2 ] --> [ --set key=val ]
  1. **Chart values.yaml**: Default values compiled inside the chart package.
  2. First Values File (-f values-base.yaml): User-supplied values file.
  3. Subsequent Values Files (-f values-prod.yaml): Evaluated left-to-right; matching keys in later files override earlier ones.
  4. Inline Set Flags (--set key=value or --set-file): Takes ultimate precedence over all files.

6. Service Management and Common Operations

Because Helm 3 is a client-side utility, operational administration revolves around managing releases within Kubernetes namespaces rather than managing systemd daemons.

Managing Releases

The core lifecycle commands for Helm releases are detailed below:

# 1. Installing a release
helm install my-web ./mychart -n production --create-namespace

# 2. Idempotent installation or upgrade (Recommended for CI/CD)
helm upgrade --install my-web ./mychart -f values-prod.yaml -n production

# 3. Checking release status
helm status my-web -n production

# 4. Listing revisions in release history
helm history my-web -n production

# 5. Rolling back to a previous revision
helm rollback my-web 2 -n production

# 6. Uninstalling a release
helm uninstall my-web -n production

Distinguishing Operational Actions

Operation Action Taken Operational Impact SRE Recommendation
helm upgrade Renders new manifest, compares against release state, applies patch to cluster objects, creates new revision ($N+1$). Modifies running workloads in-place. Always run with --dry-run=server or execute helm template first.
helm rollback Takes manifest state from a prior historical revision ($M$) and applies it as a brand-new revision ($N+1$). Immediately restores previous configuration and image tags. Primary recovery mechanism when upgrades introduce application regressions.
helm uninstall Queries stored release secrets, deletes all tracked Kubernetes objects from the cluster, and purges release metadata. Removes workloads, services, and associated objects. Objects marked with policy keep (such as PVCs) or unmanaged CRDs will remain.

7. Common Administrative Tasks and Workflows

Workflow 1: Multi-Environment Deployment Pipeline

In production environments, SREs maintain distinct configuration files for each environment (values-staging.yaml, values-prod.yaml) while deploying the exact same immutable chart version.

# Deploy to Staging
helm upgrade --install payment-api ./charts/payment-service \
  --namespace staging \
  --create-namespace \
  -f ./environments/values-staging.yaml

# Deploy to Production with set override for replicas
helm upgrade --install payment-api ./charts/payment-service \
  --namespace production \
  --create-namespace \
  -f ./environments/values-prod.yaml \
  --set replicaCount=5

Workflow 2: Safe Preview and Verification (Dry-Run Inspection)

Before applying upgrades to critical production environments, inspect the exact rendered output to prevent unintended changes.

# Client-side template render (does not talk to Kube API)
helm template payment-api ./charts/payment-service -f values-prod.yaml > rendered-output.yaml

# Server-side dry-run (validates rendered manifests against live API schemas and admission controllers)
helm upgrade --install payment-api ./charts/payment-service \
  -f values-prod.yaml \
  --dry-run=server \
  -n production

Workflow 3: Subchart and Dependency Management

When a chart depends on third-party subcharts (e.g., Bitnami PostgreSQL):

# 1. Declare dependencies in Chart.yaml
# 2. Fetch and vendor dependencies into the charts/ directory
helm dependency update ./mychart

# 3. Verify fetched subchart archives (.tgz) inside charts/
ls -l ./mychart/charts/

*Overriding subchart values in parent values.yaml*: Subchart values are overridden by nesting parameters under the subchart name key:

# Parent values.yaml
replicaCount: 2

# Subchart override key (matches subchart name in Chart.yaml)
postgresql:
  auth:
    database: app_db
    username: app_user
  primary:
    resources:
      limits:
        memory: 512Mi

Workflow 4: Working with OCI Registries

Modern Helm charts are distributed using OCI compliant container registries (e.g., Harbor, GHCR, AWS ECR, GCP Artifact Registry).

# 1. Authenticate with the OCI registry
helm registry login registry.example.com -u sysadmin --password-stdin < /etc/ci-tokens/registry.key

# 2. Package local chart into tarball archive
helm package ./mychart
# Output: mychart-1.2.0.tgz

# 3. Push chart package to OCI registry
helm push mychart-1.2.0.tgz oci://registry.example.com/charts/infrastructure

# 4. Install chart directly from OCI URL
helm install my-app oci://registry.example.com/charts/infrastructure/mychart --version 1.2.0 -n production

8. CLI Command Reference and Cheatsheet

Repository and Registry Operations

Command Purpose Important Flags / Arguments Example Explanation
helm repo add Add remote HTTP chart repository. [NAME] [URL] helm repo add bitnami [https://charts.bitnami.com/bitnami](https://charts.bitnami.com/bitnami) Registers a classic HTTP repository locally.
helm repo update Fetch latest repository index files. None helm repo update Refreshes local repository cache to find new chart versions.
helm search repo Search local repository index for keyword. [KEYWORD] helm search repo nginx Searches cached repositories for available charts.
helm search hub Search Artifact Hub public catalog. [KEYWORD] helm search hub redis Searches global web index (artifacthub.io).
helm registry login Authenticate with OCI container registry. -u, --password-stdin helm registry login registry.example.com Logs into OCI registry for push/pull operations.

Release Lifecycle Operations

Command Purpose Important Flags / Arguments Example Explanation
helm install Install a chart into cluster. [RELEASE] [CHART], -n, --values, --set helm install web bitnami/nginx -n prod Installs a new release; fails if release exists.
helm upgrade Upgrade an existing release. --install, -f, --set, --wait helm upgrade --install web bitnami/nginx -n prod Idempotent command; installs if missing, upgrades if present.
helm rollback Revert release to a revision. [RELEASE] [REVISION], -n helm rollback web 2 -n prod Restores manifests from target historical revision.
helm uninstall Uninstall release & remove objects. [RELEASE], -n, --keep-history helm uninstall web -n prod Removes release components from target namespace.

Release State and Inspection

Command Purpose Important Flags / Arguments Example Explanation
helm list List releases in namespace. -n, -A (all namespaces), -a (all states) helm list -A Displays release names, status, revisions, and app versions.
helm status Show current release status. [RELEASE], -n helm status web -n prod Prints release notes, last deployed time, and status.
helm history List historical release revisions. [RELEASE], -n helm history web -n prod Shows table of all revisions with status and descriptions.
helm get values Show values applied to release. [RELEASE], -n, --all helm get values web -n prod Prints user-supplied override values for the release.
helm get manifest Retrieve deployed YAML manifests. [RELEASE], -n helm get manifest web -n prod Dumps rendered manifests stored in the cluster release secret.

Chart Development and Inspection

Command Purpose Important Flags / Arguments Example Explanation
helm create Scaffold new local chart directory. [NAME] helm create mychart Generates directory layout with boilerplate templates.
helm lint Check chart for syntax/style issues. [PATH] helm lint ./mychart Validates YAML syntax and required fields in Chart.yaml.
helm template Render templates locally. [RELEASE] [PATH], -f, --debug helm template test ./mychart --debug Prints raw rendered YAML to stdout without contacting cluster.
helm package Package chart into tar archive. [PATH] helm package ./mychart Creates a compressed .tgz file named <name>-<version>.tgz.
helm test Execute test hooks in chart. [RELEASE], -n helm test web -n prod Runs test pods defined in chart to verify functionality.

Kubectl and Diagnostic Helper Commands

# Query pods associated with a Helm release using standard recommended labels
kubectl get pods -n production -l app.kubernetes.io/instance=web

# Stream logs across all pods owned by a Helm release (requires stern krew plugin)
stern -l app.kubernetes.io/instance=web -n production

# View object ownership dependency tree for a deployment (requires kubectl-tree plugin)
kubectl tree deploy -l app.kubernetes.io/instance=web -n production

9. Logs, Monitoring and Diagnostics

Locating Logs and Diagnostic Data

Helm itself is stateless and client-side; it does not generate centralized system log files on server nodes. Diagnostic information exists in three locations:

  1. CLI Debug Output: Executing commands with the --debug flag forces Helm to output verbose internal operations, Go template stack traces, and raw Kube API HTTP requests/responses.
  2. Kubernetes Release Secrets: Stored in the target namespace as gzipped base64 data.
  3. Workload & Hook Pod Logs: Diagnostic data for failing installs/upgrades resides within the execution logs of Kubernetes Pods and Jobs.
# Inspecting Helm release storage secrets directly
kubectl get secrets -n production -l owner=helm

Output:

NAME                               TYPE                 DATA   AGE
sh.helm.release.v1.my-web.v1       helm.sh/release.v1   1      5d2h
sh.helm.release.v1.my-web.v2       helm.sh/release.v1   1      12h

Decoding Release Metadata Secret Content

If a release becomes corrupted and Helm CLI cannot parse it, inspect raw release data manually:

kubectl get secret sh.helm.release.v1.my-web.v2 -n production -o jsonpath='{.data.release}' | base64 -d | base64 -d | gunzip | jq .

Systemic Diagnostics Workflow

+-----------------------------------------------------------------------+
|                    HELM DIAGNOSTIC WORKFLOW                           |
|                                                                       |
|  STEP 1: Verify Release State                                         |
|  $ helm list -n NS                                                    |
|  $ helm status RELEASE -n NS                                          |
|  $ helm history RELEASE -n NS                                         |
|                                                                       |
|  STEP 2: Inspect Effective Manifests & Values                         |
|  $ helm get manifest RELEASE -n NS | less                             |
|  $ helm get values RELEASE -n NS                                      |
|                                                                       |
|  STEP 3: Inspect Cluster Workload Health                              |
|  $ kubectl get pods -n NS -l app.kubernetes.io/instance=RELEASE       |
|  $ kubectl describe pod -n NS -l app.kubernetes.io/instance=RELEASE   |
|  $ stern -l app.kubernetes.io/instance=RELEASE -n NS                  |
+-----------------------------------------------------------------------+

10. Troubleshooting Guide and Common Failure Scenarios

Problem 1: Install or Upgrade Failed (Exit Non-Zero)

Symptoms

helm install or helm upgrade exits with a non-zero status code. The terminal displays errors such as Error: UPGRADE FAILED: timed out waiting for the condition or Error: release failed: deployment failed.

Error: UPGRADE FAILED: release payment-api failed and has been rolled back: timed out waiting for the condition

Possible Causes

  1. Workload CrashLoopBackOff: Application pods failed startup or readiness probes during installation.
  2. Insufficient Resource Quotas: Cluster lacks CPU, Memory, or PersistentVolume capacity to schedule pods.
  3. Image Pull Errors: Invalid container registry image path, missing image tag, or missing imagePullSecrets.

Investigation and Diagnosis

  1. Check the recorded release status and revision history:
helm status payment-api -n production
helm history payment-api -n production
  1. Retrieve rendered manifests deployed to the cluster to confirm image tags and resource specs:
helm get manifest payment-api -n production | grep -E "image:|memory:|cpu:"
  1. Query all pods associated with the Helm release instance label:
kubectl get pods -n production -l app.kubernetes.io/instance=payment-api
  1. Describe failing pods to review Kubernetes event messages:
kubectl describe pod -n production -l app.kubernetes.io/instance=payment-api
  1. Tail pod application logs:
stern -l app.kubernetes.io/instance=payment-api -n production

Root Cause Identification

  • If kubectl describe pod displays ImagePullBackOff, the registry path or credential is missing.
  • If pods show CrashLoopBackOff, inspect stdout logs via stern to diagnose application code startup errors.

Resolution

Fix the underlying configuration file or container image tag, then re-run the idempotent upgrade command:

helm upgrade --install payment-api ./charts/payment-service -f values-prod.yaml -n production

Verification

Confirm that the release status is deployed and all pods report Running and Ready status:

helm status payment-api -n production
kubectl get pods -n production -l app.kubernetes.io/instance=payment-api

Prevention

  • Always test chart renders using helm template --debug before executing deployments.
  • Include standard readiness and liveness probes with appropriate delay parameters in templates.

Problem 2: Release Stuck in pending-install or pending-upgrade

Symptoms

Subsequent execution of helm upgrade fails with Error: another operation (install/upgrade/rollback) is in progress. helm list shows status pending-install or pending-upgrade.

Error: HELM UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Possible Causes

  1. Interrupted CLI Process: The helm command-line process was terminated abruptly (e.g., CI/CD pipeline timeout, SIGKILL, or network loss) during execution.
  2. Hanging Lifecycle Hook: A pre-install or pre-upgrade hook Job became stuck or blocked indefinitely, preventing Helm from finalizing the secret record.

Investigation and Diagnosis

  1. Inspect release history to identify the stuck revision number:
helm history payment-api -n production
  1. Check for hanging hook pods or jobs:
kubectl get pods,jobs -n production -l helm.sh/hook
  1. Check release secrets directly in the namespace:
kubectl get secret -n production -l owner=helm

Root Cause Identification

If helm history shows the latest revision in pending-upgrade and no local CLI process is running, Helm’s release secret state lock was left unreleased.

Resolution

  • Method 1: Manual Rollback (Preferred): Roll back to the last known good revision:
helm rollback payment-api 3 -n production
  • Method 2: Deleting Stuck Secret (Last Resort): If helm rollback refuses to run due to the pending lock, remove the pending revision’s secret directly:
# Find stuck revision secret (e.g., revision 4)
kubectl delete secret sh.helm.release.v1.payment-api.v4 -n production

# Retry upgrade command
helm upgrade --install payment-api ./charts/payment-service -f values-prod.yaml -n production

Verification

Run helm list -n production and ensure status has changed from pending-upgrade to deployed.

Prevention

  • Implement timeouts in CI/CD pipelines using --timeout 10m0s flag.
  • Ensure hook Jobs contain activeDeadlineSeconds in their pod specification so Kubernetes terminates stuck hook jobs automatically.

Problem 3: Unexpected Configuration Applied After Upgrade (Values Precedence Issue)

Symptoms

An application behaves incorrectly after an upgrade (e.g., wrong database endpoint or replica count) despite passing a custom values file.

Possible Causes

  1. Values Precedence Misunderstanding: Values passed via --set or secondary -f files overridden the intended configuration file.
  2. Key Typo: A key in values.yaml was misspelled or placed at the wrong indentation level, causing the template to fall back to defaults.

Investigation and Diagnosis

  1. Export the effective values recorded in the active release revision:
helm get values payment-api -n production
  1. Compare the effective values against the raw source file:
diff -u <(helm get values payment-api -n production) values-prod.yaml

Root Cause Identification

If helm get values shows parameters matching --set inputs rather than values-prod.yaml, the command-line order allowed --set to override file settings.

Resolution

  1. Correct the execution command, ensuring the intended values file is placed last or inline --set flags are corrected.
  2. Roll back immediately if production is impacted:
helm rollback payment-api PREVIOUS_REV -n production
  1. Re-apply using the proper merge hierarchy:
helm upgrade payment-api ./charts/payment-service -f values-prod.yaml -n production

Verification

Run helm get values payment-api -n production and verify all key-value pairs match expectations.

Prevention

Maintain environment-specific settings inside dedicated, version-controlled values files (values-staging.yaml, values-prod.yaml) rather than passing long strings of --set flags in CI scripts.


Problem 4: Chart Not Found or Repository Synchronization Failure

Symptoms

Executing helm install returns Error: chart "nginx" not found in bitnami repository or Error: failed to download "oci://...".

Error: chart "payment-service" matching 1.2.0 not found in myrepo repository (have targets updated?)

Possible Causes

  1. Stale Local Repository Cache: The local repository index is out of date.
  2. Missing Version Tag in OCI Registry: The specified version does not exist in the OCI container registry.
  3. Air-Gapped Network Isolation: The cluster or workstation cannot reach external HTTP chart repositories.

Investigation and Diagnosis

  1. Search local repository cache for available versions:
helm search repo payment-service --versions
  1. Verify repository configuration list:
helm repo list

Resolution

  • For Standard Repositories: Update repository index cache:
helm repo update
  • For OCI Repositories: Verify OCI path syntax and explicitly append the required --version flag:
helm install my-app oci://registry.example.com/charts/payment-service --version 1.2.0 -n production
  • For Air-Gapped Environments: Download or package the chart locally into a .tgz archive and install from the local tarball file path:
helm install payment-api ./payment-service-1.2.0.tgz -n production

Verification

Run helm search repo <chart-name> to verify that the chart version is indexed and installable.


Problem 5: Template Rendering Error (Nil Pointer, YAML Syntax, Indentation Failure)

Symptoms

Executing helm install or helm template outputs Go template execution errors or YAML parsing errors:

Error: YAML parse error on payment-service/templates/deployment.yaml: error converting YAML to JSON: yaml: line 18: mapping values are not allowed in this context

or

Error: template: payment-service/templates/deployment.yaml:24:14: executing "payment-service/templates/deployment.yaml" at <.Values.database.port>: nil pointer evaluating interface {}

Possible Causes

  1. Nil Pointer Dereference: The template references .Values.database.port, but database is undefined in values.yaml.
  2. Indentation Misalignment: Go template helper pipelines (e.g., nindent) injected YAML blocks at incorrect column offsets.

Investigation and Diagnosis

Execute helm template with --debug flag to display rendered YAML up to the failure point:

helm template test-release ./charts/payment-service -f values-prod.yaml --debug

Root Cause Identification

The --debug output displays the line number where rendering stopped. Inspect the generated output immediately above the error line to locate malformed YAML or missing key references.

Resolution

  1. Fix missing keys in values.yaml or wrap template access in conditional checks:
{{- if .Values.database }}
port: {{ .Values.database.port }}
{{- end }}
  1. Run helm lint to validate chart syntax:
helm lint ./charts/payment-service

Verification

Ensure helm template ./charts/payment-service renders cleanly without error messages.


Problem 6: RBAC / Forbidden Errors During Install or Upgrade

Symptoms

Helm returns API authorization errors during execution:

Error: INSTALL FAILED: deployments.apps is forbidden: User "ci-service-account" cannot create resource "deployments" in API group "apps" in the namespace "production"

Possible Causes

The user identity or ServiceAccount configured in kubeconfig lacks RBAC permissions to manage specific Kubernetes resource types (Deployments, CRDs, Roles, Ingress) within the target namespace.

Investigation and Diagnosis

Test API permissions directly using kubectl auth can-i:

# Check permission to create deployments in target namespace
kubectl auth can-i create deployments -n production --as=system:serviceaccount:ci-cd:helm-runner

# Check permission to create CRDs cluster-wide
kubectl auth can-i create crds --as=system:serviceaccount:ci-cd:helm-runner

Resolution

Bind an appropriate Role or ClusterRole to the executing ServiceAccount or user identity via RoleBinding or ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: helm-runner-binding
  namespace: production
subjects:
  - kind: ServiceAccount
    name: helm-runner
    namespace: ci-cd
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: rbac.authorization.k8s.io

Verification

Re-run kubectl auth can-i create deployments -n production, verifying that the output prints yes.


Problem 7: Pre-install / Post-upgrade Lifecycle Hook Failure

Symptoms

Deployment hangs or fails during execution, displaying Error: Hook pre-install payment-service/templates/job-db-migrate.yaml failed: Job failed.

Possible Causes

The Kubernetes Job or Pod generated by the hook template failed during execution (e.g., database connection failure during schema migration).

Investigation and Diagnosis

  1. Query hook jobs and pods in the target namespace:
kubectl get jobs,pods -n production -l helm.sh/hook
  1. Inspect logs of the failing hook job pod:
kubectl logs job/payment-api-db-migrate -n production

Resolution

  1. Resolve the root cause in the hook script or container payload (e.g., fix database credentials).
  2. Manually delete the failed hook job so Helm can recreate it on retry:
kubectl delete job payment-api-db-migrate -n production
  1. Re-run helm upgrade --install.
  2. Emergency Bypass: If a non-critical hook blocks production recovery, bypass hooks temporarily using --no-hooks (use with extreme caution):
helm upgrade payment-api ./charts/payment-service --no-hooks -n production

Verification

Confirm that hook pods exit with status Completed (Exit Code 0) and the release updates to deployed.


Problem 8: Resources Remaining in Cluster After Uninstall

Symptoms

Executing helm uninstall payment-api -n production succeeds, but kubectl get pvc or kubectl get crd still lists resources previously created by the release.

Possible Causes

  1. Resource Policy Annotation: Resources annotated with helm.sh/resource-policy: keep are explicitly ignored by Helm during uninstallation.
  2. Custom Resource Definitions (CRDs): Helm intentionally does not automatically delete CRDs installed from the crds/ directory to prevent accidental data loss across shared cluster workloads.

Investigation and Diagnosis

Check object annotations for resource keep policies:

kubectl get pvc -n production -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.helm\.sh/resource-policy}{"\n"}{end}'

Resolution

If the remaining PVCs or CRDs are no longer required, delete them manually via kubectl:

# Delete orphaned PersistentVolumeClaims
kubectl delete pvc -l app.kubernetes.io/instance=payment-api -n production

# Delete custom resource definitions if safe
kubectl delete crd paymentcustomresources.example.com

Prevention

Document resource cleanup policies in operational runbooks so SREs know which storage volumes require manual lifecycle management.


Problem 9: Pods Healthy and Running but Application Inaccessible

Symptoms

helm status reports deployed and all pods show Running 1/1 status. However, HTTP traffic to the application endpoint fails with 502 Bad Gateway or Connection Refused.

Possible Causes

This is an application routing or environment variable configuration error rather than a Helm release failure. Common issues include incorrect Service target ports, missing ConfigMap values, or misconfigured Ingress controllers.

Investigation and Diagnosis

  1. Inspect deployed values stored in the release record:
helm get values payment-api -n production
  1. Verify environment variables and mounted volume files inside the running container:
kubectl exec -it deployment/payment-api -n production -- env
  1. Test direct container access by forwarding pod ports locally:
kubectl port-forward pod/payment-api-7b89d-x4z2 8080:8080 -n production

Resolution

  1. If port-forwarding succeeds, the failure lies in Service port mapping or Ingress routing rules. Align service.port and targetPort parameters in values.yaml.
  2. Upgrade the release with corrected routing values:
helm upgrade payment-api ./charts/payment-service -f values-prod.yaml -n production

Verification

Send an HTTP test request to the ingress URL or Service IP and confirm 200 OK response.


11. Security and Best Practices

1. Managing Sensitive Data in Helm Values

Never commit plaintext passwords, API tokens, or TLS private keys into Git repositories inside values.yaml files.

  • Sealed Secrets: Encrypt secrets using Bitnami SealedSecrets so encrypted values can be safely stored in Git and decrypted only by the cluster controller.
  • **SOPS & helm-secrets**: Use Mozilla SOPS with helm-secrets plugin to decrypt encrypted values files transparently during CI/CD execution:
helm secrets upgrade --install payment-api ./charts/payment-service -f secrets.prod.yaml -n production
  • External Secrets Operator: Reference external secret providers (AWS Secrets Manager, HashiCorp Vault) using ExternalSecret manifests inside the chart templates.

2. Enforcing Least-Privilege RBAC for CI/CD Automated Pipelines

Avoid executing Helm pipelines using global cluster-admin access. Grant CI/CD ServiceAccounts namespace-scoped Roles restricted to the exact API groups required by the chart (Deployments, Services, ConfigMaps, Ingress).

3. Securing Helm Release Metadata Secrets

Because Helm 3 stores release history as Kubernetes Secrets labeled with owner=helm, any user with get and list permissions on secrets in a namespace can read raw release manifests and effective values. Restrict access using Kubernetes RBAC rules:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prevent-release-secrets-access
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
    resourceNames: [] # Restrict secret access explicitly

4. Production Chart Management Best Practices

  • Version Pinning: Always pin explicit chart versions (--version 1.2.3) in production deployment scripts. Never rely on floating tags or unpinned release installations.
  • Chart Provenance & Integrity: Sign charts using GPG keys during packaging (helm package --sign) and enforce signature validation during installation using --verify.
  • Private OCI Registries: Store internal enterprise charts in private OCI container registries with vulnerability scanning enabled.

12. Performance and Resource Optimisation

Managing Release Secret Accumulation

By default, Helm retains historical revision records indefinitely. Over months of automated deployment, hundreds of release secrets (sh.helm.release.v1.app.v1 … v300) accumulate in Kubernetes namespaces, increasing API server memory usage and slowing down namespace operations.

Optimization: Limit maximum recorded release history using --history-max during upgrades:

helm upgrade --install payment-api ./charts/payment-service \
  -f values-prod.yaml \
  --history-max 10 \
  -n production

Network Bandwidth: OCI Registries vs. Legacy Repositories

Legacy HTTP chart repositories rely on a single monolithic index.yaml file listing all available chart packages. For large enterprise repositories containing thousands of chart versions, downloading index.yaml consumes megabytes of bandwidth per client update.

Switching to OCI Registries completely eliminates index.yaml overhead. Clients pull only the exact chart version package required (oci://[registry.example.com/charts/myapp:1.2.0](https://registry.example.com/charts/myapp:1.2.0)), dramatically reducing bandwidth consumption and accelerating CI/CD pipeline execution times.


13. Backup, Recovery and Maintenance

Backing Up Helm Release State

Because Helm stores release state as standard Kubernetes Secrets in the application’s namespace, backing up release metadata requires exporting the matching secret objects:

# Export all Helm release secrets across the cluster
kubectl get secrets -A -l owner=helm -o yaml > helm-releases-backup.yaml

Recovering a Corrupted Release Metadata Secret

If a release secret is accidentally deleted or corrupted, Helm lose tracking of that application instance. To recover:

  1. Extract the manifest from the latest working backup or render the current manifest manually:
helm template payment-api ./charts/payment-service -f values-prod.yaml -n production > current-manifest.yaml
  1. Re-import the release state without modifying live cluster resources using --force or standard idempotent upgrade:
helm upgrade --install payment-api ./charts/payment-service -f values-prod.yaml -n production

Maintaining Chart Repositories

Periodically purge obsolete chart packages from local caches and remote OCI registries to reclaim storage space:

# Clear local Helm repository cache
helm repo remove bitnami
rm -rf ~/.cache/helm/repository/*

14. Automation and Integration

CI/CD Pipeline Automation (GitLab CI Example)

An idempotent, robust automated pipeline step for deploying Helm charts to Kubernetes:

# .gitlab-ci.yml
deploy_production:
  stage: deploy
  image: alpine/helm:3.16.2
  script:
    - helm registry login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
    - helm lint ./charts/payment-service
    - helm upgrade --install payment-api oci://$CI_REGISTRY/charts/payment-service \
        --version $CI_COMMIT_TAG \
        --namespace production \
        --create-namespace \
        -f ./environments/values-prod.yaml \
        --history-max 10 \
        --atomic \
        --timeout 5m0s
  only:
    - tags

GitOps Integration (Argo CD / Flux)

In modern SRE GitOps workflows, Helm charts are rendered and managed declaratively by GitOps controllers running inside the cluster.

Example Argo CD Application manifest referencing a Helm chart:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-api-production
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://charts.bitnami.com/bitnami'
    chart: nginx
    targetRevision: 15.4.0
    helm:
      valueFiles:
        - values-production.yaml
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

15. Practical Labs and Exercises

Lab 1 (Beginner): Deploying and Customizing an Application with Helm

Objective

Install an NGINX web server release using Helm, customize its configuration using an override file, and verify workload operation.

Environment & Prerequisites

  • Access to a running Kubernetes cluster (Minikube, Kind, or cloud cluster).
  • Local Helm 3 CLI installed.

Instructions

  1. Add the Bitnami public chart repository:
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
  1. Create a custom values file named lab1-values.yaml:
replicaCount: 2
service:
  type: ClusterIP
  1. Deploy the release named my-nginx into namespace lab1:
helm install my-nginx bitnami/nginx -f lab1-values.yaml -n lab1 --create-namespace
  1. Verify release status and running pods:
helm status my-nginx -n lab1
kubectl get pods -n lab1 -l app.kubernetes.io/instance=my-nginx

Validation

Verify that exactly two NGINX pods are running in the lab1 namespace.


Lab 2 (Intermediate): Building a Multi-Tier Chart with Subchart Dependencies

Objective

Create a custom application chart (web-app), declare a PostgreSQL subchart dependency, and configure subchart values from the parent chart.

Instructions

  1. Scaffold a new chart:
helm create web-app
  1. Edit web-app/Chart.yaml to declare PostgreSQL as a dependency:
dependencies:
  - name: postgresql
    version: 12.5.2
    repository: https://charts.bitnami.com/bitnami
  1. Fetch the subchart dependency:
helm dependency update ./web-app
  1. Update web-app/values.yaml to configure the subchart database parameters:
postgresql:
  auth:
    database: user_data
    username: db_admin
  1. Perform a dry-run render to verify that both web application and database manifests render cleanly:
helm template test-stack ./web-app --debug

Validation

Inspect stdout to confirm that both Deployment (web-app) and StatefulSet (postgresql) manifests are generated in the output stream.


Lab 3 (Troubleshooting): Recovering a Stuck Release

Objective

Diagnose a release stuck in pending-upgrade state caused by an interrupted deployment and recover the release safely.

Instructions

  1. Inspect the stuck release status:
helm list -n production
helm history payment-api -n production
  1. Observe that revision 4 is stuck in pending-upgrade.
  2. Identify the release secrets stored in the namespace:
kubectl get secret -n production -l owner=helm
  1. Delete the pending secret record (revision 4):
kubectl delete secret sh.helm.release.v1.payment-api.v4 -n production
  1. Roll back release state to revision 3:
helm rollback payment-api 3 -n production

Validation

Run helm status payment-api -n production and confirm status shows deployed at revision 5.


16. Quick Reference and Cheat Sheet

Important Configuration Files & Locations

Path Purpose
Chart.yaml Main chart metadata, semantic version, app version, and subchart dependencies.
values.yaml Default configuration parameter values for templates.
templates/ Directory containing Go-templated Kubernetes manifest files.
templates/NOTES.txt Post-installation operational instructions displayed to stdout.
~/.config/helm/repositories.yaml Local index of configured HTTP chart repositories.

Release Secret Naming Format

Pattern Example Purpose
sh.helm.release.v1.<RELEASE>.<REVISION> sh.helm.release.v1.payment-api.v2 In-cluster storage secret containing gzipped release metadata.

Common Administrative Commands

Task Command
Add Repository helm repo add [NAME] [URL]
Update Repository Index helm repo update
Install Release helm install [RELEASE] [CHART] -n [NS] --create-namespace
Idempotent Upgrade/Install helm upgrade --install [RELEASE] [CHART] -f values.yaml -n [NS]
Set Inline Value Override helm upgrade [RELEASE] [CHART] --set key=value -n [NS]
Rollback to Revision helm rollback [RELEASE] [REVISION] -n [NS]
Uninstall Release helm uninstall [RELEASE] -n [NS]
List Releases Across All Namespaces helm list -A
Inspect Release History helm history [RELEASE] -n [NS]
Display Deployed Values helm get values [RELEASE] -n [NS]
Render Templates Locally helm template [RELEASE] [CHART-PATH] -f values.yaml --debug

Troubleshooting Quick Matrix

Symptom Initial Investigation Step Primary Remediation
Upgrade Failed / Timeout kubectl get pods -n NS -l app.kubernetes.io/instance=RELEASE Fix image tag/config in values.yaml & re-run helm upgrade.
**Stuck in pending-upgrade** kubectl get secret -n NS -l owner=helm Delete stuck revision secret & execute helm rollback.
Template Syntax Error helm template RELEASE CHART -f values.yaml --debug Fix indentation or nil pointer keys in template.
RBAC / Forbidden Error kubectl auth can-i create deployments -n NS Grant RoleBinding permissions to executing ServiceAccount.
Hook Job Failed kubectl logs job/HOOK_JOB -n NS Fix underlying hook script; delete failed job & retry.
Chart Not Found helm repo update Sync repo index or append --version to OCI path.

17. Further Reading and References

  1. Official Helm Documentation: helm.sh/docs
  2. Helm Charts Guide: helm.sh - charts
  3. Helm Values Files Guide: helm.sh - values
  4. Artifact Hub Public Chart Registry: artifacthub.io
  5. Kubernetes Workload & Logging Reference: Kubernetes Documentation