Domain 4Kubernetes EngineeringINTERMEDIATE

Kubernetes Operational Handbook: Architecture, Administration, and Production Troubleshooting

A comprehensive guide to Kubernetes architecture, core workload objects, CLI tools like kubectl and krew, system maintenance, and systematic production troubleshooting.

Domain: Domain 4: Kubernetes Cluster Engineering•Updated: 2026-11-03
#linux#sysadmin#kubernetes#sre#containers#devops

1. Overview

Kubernetes (often abbreviated as K8s) is an open-source container orchestration platform designed to automate the deployment, scaling, management, and operational lifecycle of containerized applications.

In modern infrastructure, managing hundreds of isolated containers across bare-metal servers or virtual machines creates significant operational complexity. Manually tracking container placement, handling hardware failures, updating software without downtime, and balancing network traffic across nodes is error-prone and unscalable. Kubernetes solves this problem by abstracting individual physical or virtual machines into a single unified pool of compute resources.

Instead of manually starting containers on specific host machines, system administrators and Site Reliability Engineers (SREs) declare the desired state of their infrastructure and workloads using declarative configuration files (typically written in YAML). Kubernetes control loops continuously observe the actual state of the cluster and make automated adjustments to reconcile the actual state back to the defined desired state.

+-----------------------------------------------------------------------+
|                         Kubernetes Cluster                            |
|                                                                       |
|   +-----------------------+               +-----------------------+   |
|   |   Worker Node A       |               |   Worker Node B       |   |
|   |  +-----------------+  |               |  +-----------------+  |   |
|   |  | Pod (App v1.0)  |  |               |  | Pod (App v1.0)  |  |   |
|   |  +-----------------+  |               |  +-----------------+  |   |
|   +-----------------------+               +-----------------------+   |
|               ^                                       ^               |
+---------------|---------------------------------------|---------------+
                | Automated Reconciliation Loop         |
                +---------------------------------------+
                                    |
                    +-------------------------------+
                    |     Desired State: 2 Replicas |
                    +-------------------------------+

Common Use Cases

  • Microservices Hosting: Running decoupled, independently scalable software microservices with internal service discovery.
  • Stateless and Stateful Applications: Managing both ephemeral web applications and persistent storage systems like databases.
  • Automated Batch Jobs: Executing scheduled cron jobs and batch data processing pipelines.
  • Hybrid & Multi-Cloud Deployments: Standardizing operational primitives across on-premises data centers and public cloud providers (such as AWS EKS, Google GKE, and Azure AKS).

Advantages and Limitations

Category Description / Details
Key Advantages High availability, auto-scaling, self-healing container restarts, declarative rollouts and rollbacks, automated service discovery, load balancing, and resource optimization.
Limitations Steep learning curve, operational complexity for self-managed control planes, network overhead, and potential resource waste if limits/requests are misconfigured.

Kubernetes vs. Alternatives

  • Docker Compose: Ideal for single-host multi-container development environments, but lacks native multi-node clustering, auto-scaling, and advanced self-healing.
  • HashiCorp Nomad: A lightweight workload orchestrator for containers and non-containerized applications; simpler to operate than Kubernetes, but has a smaller ecosystem and fewer built-in cloud-native extensions.

Operational Context

In an SRE environment, engineers rarely think about which specific virtual machine hosts a running process. Instead, operations revolve around Kubernetes primitives such as Pods, Deployments, Services, and Ingress controllers.


2. Architecture and Mental Model

To effectively operate and debug a Kubernetes cluster, administrators must understand its split Control Plane and Worker Node architecture.

graph TB
    subgraph Control_Plane["Control Plane (Master Nodes)"]
        APISERVER["kube-apiserver"]
        ETCD["etcd (Key-Value Store)"]
        SCHEDULER["kube-scheduler"]
        CM["kube-controller-manager"]
        CCM["cloud-controller-manager"]

        APISERVER <--> ETCD
        SCHEDULER <--> APISERVER
        CM <--> APISERVER
        CCM <--> APISERVER
    end

    subgraph Worker_Node_1["Worker Node 1"]
        KUBELET1["kubelet"]
        KPROXY1["kube-proxy"]
        CRUN1["Container Runtime (containerd)"]
        POD1["Pod A"]
        POD2["Pod B"]

        KUBELET1 <--> APISERVER
        KUBELET1 --> CRUN1
        CRUN1 --> POD1
        CRUN1 --> POD2
        KPROXY1 <--> APISERVER
    end

    subgraph Worker_Node_2["Worker Node 2"]
        KUBELET2["kubelet"]
        KPROXY2["kube-proxy"]
        CRUN2["Container Runtime (containerd)"]
        POD3["Pod C"]

        KUBELET2 <--> APISERVER
        KUBELET2 --> CRUN2
        CRUN2 --> POD3
        KPROXY2 <--> APISERVER
    end

    KUBECTL["kubectl CLI"] --> APISERVER

Control Plane Components

The control plane makes global decisions about the cluster (e.g., scheduling pods), detects cluster events, and responds to changes.

  • kube-apiserver: The REST API gateway and central communication hub for the entire cluster. All CLI client tools (kubectl), node components, and control plane services communicate exclusively through the API server over HTTPS.
  • etcd: A consistent, highly available distributed key-value store that acts as the single source of truth for all cluster state, object definitions, and metadata.
  • kube-scheduler: Inspects newly created Pods that have no assigned worker node and selects the most optimal node for them based on resource requests, hardware constraints, node affinity, taints, and tolerations.
  • kube-controller-manager: Runs daemon controllers (such as DeploymentController, NodeController, ReplicaSetController) in a single process to continuously reconcile actual state with desired state.
  • cloud-controller-manager: Integrates underlying cloud provider APIs (AWS, GCP, Azure) to provision resources like cloud load balancers, node IP routing, and persistent storage volumes.

Worker Node Components

Worker nodes execute actual application workloads within containers.

  • kubelet: An agent running on every node that watches the API server for PodSpecs assigned to its node and ensures that the corresponding containers are running and healthy.
  • kube-proxy: Maintains network rules on each host system to implement Kubernetes Service abstractions, performing IP forwarding and load balancing via Linux iptables or IPVS.
  • Container Runtime: The underlying engine responsible for running containers (e.g., containerd or CRI-O, communicating via the Container Runtime Interface / CRI).

kubectl Interaction Lifecycle

When an administrator executes a command such as kubectl apply -f deployment.yaml:

sequenceDiagram
    autonumber
    actor Admin as SysAdmin / kubectl
    participant API as kube-apiserver
    participant ETCD as etcd Store
    participant CTRL as kube-controller-manager
    participant SCHED as kube-scheduler
    participant KUBELET as Worker kubelet

    Admin->>API: 1. Submit YAML over HTTPS (~/.kube/config)
    API->>API: 2. Authenticate, Authorize (RBAC) & Validate Schema
    API->>ETCD: 3. Persist Object Data
    ETCD-->>API: 4. Data Saved
    CTRL->>API: 5. Watcher detects missing Pods (ReplicaSet)
    CTRL->>API: 6. Request creation of new Pod spec
    SCHED->>API: 7. Watcher detects unscheduled Pod
    SCHED->>API: 8. Assigns Pod to Worker Node
    API->>ETCD: 9. Update Pod binding in etcd
    KUBELET->>API: 10. Watcher detects Pod assigned to local node
    KUBELET->>KUBELET: 11. Instruct container runtime to pull image & start containers
    KUBELET->>API: 12. Report Pod status back (Running)
  1. Config & Request: kubectl reads authentication credentials and API endpoints from ~/.kube/config and sends an HTTPS REST request to kube-apiserver.
  2. Admission & Validation: kube-apiserver authenticates the request, verifies RBAC authorization, passes it through admission webhooks (mutating and validating), and validates the schema.
  3. Persist: The API server writes the validated resource definition into etcd.
  4. Reconcile: The relevant controller in kube-controller-manager notices the change via an active API watch stream and creates missing underlying resources (e.g., ReplicaSet creates Pod specs).
  5. Schedule: kube-scheduler observes unscheduled Pods and assigns them to an appropriate node.
  6. Execution: The kubelet on the target worker node detects the scheduled Pod, instructs the local Container Runtime (e.g., containerd) to pull the container image and create network namespaces, and periodically updates status back to kube-apiserver.

Managed vs Self-Managed: In managed offerings (AWS EKS, Google GKE, Azure AKS), cloud providers manage and hide the control plane components (etcd, kube-apiserver, etc.). However, SREs still manage and debug workloads on worker nodes using identical kubectl workflows.


3. Core Concepts and Terminology

Workload Objects

  • Pod: The smallest and basic deployable computing unit in Kubernetes. A Pod wraps one or more tightly coupled containers that share a network namespace (same IP address and port space), IPC namespace, and storage volumes.
  • Deployment: Declarative controller used for running stateless applications. Manages underlying ReplicaSets to provide declarative rolling updates, instant rollbacks, and replica scaling.
  • StatefulSet: Used for stateful applications (such as databases or message queues) requiring stable network identifiers (e.g., pod-0, pod-1), persistent storage bindings, and ordered deployments.
  • DaemonSet: Guarantees that a copy of a specified Pod runs on all (or selected) worker nodes. Commonly used for node-level agents, log collectors (e.g., Fluentbit), and network monitoring tools.
  • Job & CronJob: A Job creates Pods that run until a specific task completes successfully. A CronJob runs batch tasks on a periodic schedule using standard cron syntax.

Networking and Exposure Primitives

+-------------------------------------------------------------------------+
|                              Ingress                                    |
|                       (HTTP Routing: app.com)                           |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
|                              Service                                    |
|                  (Stable Virtual IP / ClusterIP)                        |
+-------------------------------------------------------------------------+
                                    |
             +----------------------+----------------------+
             |                                             |
             v                                             v
+--------------------------+                 +--------------------------+
|          Pod 1           |                 |          Pod 2           |
|  (IP: 10.244.1.15:8080)  |                 |  (IP: 10.244.2.22:8080)  |
+--------------------------+                 +--------------------------+
  • Service: An abstract logical layer that defines a policy-based policy and access mechanism for a set of ephemeral Pods. Provides a stable virtual IP (ClusterIP) and DNS entry.

  • ClusterIP: Default type. Exposes the Service on an internal IP inside the cluster.

  • NodePort: Exposes the Service on each worker node’s IP at a static port (typically 30000-32767).

  • LoadBalancer: Provisions an external, vendor-managed cloud load balancer that routes directly to NodePorts or Pod IPs.

  • Ingress: An API object managing external HTTP/S routing into cluster services based on path or host rules. Requires an active Ingress Controller (such as NGINX or Traefik) to execute the routing.

  • NetworkPolicy: Acts as an internal firewall between Pods, controlling layer 3/4 network traffic based on labels and CIDR blocks. Must be supported by the cluster Container Network Interface (CNI) plugin (e.g., Calico, Cilium).

  • Cluster DNS: Internal DNS resolving service names into IP addresses following the standard format: <service-name>.<namespace>.svc.cluster.local.

Configuration and Isolation

  • ConfigMap: Stores non-sensitive configuration parameters as key-value pairs or complete configuration files. Mounted into Pods as environment variables or volume files.
  • Secret: Holds sensitive data such as API keys, passwords, and SSH keys. Stored as base64-encoded strings (note: base64 is encoding, not encryption; requires encryption-at-rest configuration).
  • Namespace: Virtual clusters providing logical isolation for resources, resource quotas, and team environments (e.g., default, kube-system, production, staging).
  • Role-Based Access Control (RBAC): Restricts operations within the cluster.
  • Role / ClusterRole: Defines API permissions (verbs like get, list, create, delete on resources) within a namespace or cluster-wide.
  • RoleBinding / ClusterRoleBinding: Grants defined roles to users, groups, or ServiceAccounts.

4. Installation and Version Management

Prerequisites

Before setting up a node with container components:

  1. Disable Linux swap space to ensure deterministic memory allocation for kubelet:
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
  1. Enable kernel module loading and IP forwarding required for pod networking:
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

Package Manager Installation (Debian/Ubuntu Example)

To install core control components manually via apt:

# Install dependencies
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg

# Add Kubernetes GPG signing key and repository
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

# Install kubelet, kubeadm, and kubectl
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Local Development and Lightweight Distros

  • minikube / kind (Kubernetes in Docker): Runs complete multi-node clusters inside local Docker containers for rapid testing.
  • k3s: Lightweight single-binary Kubernetes distribution optimized for edge environments and lab nodes.

Version Management and Verification

Check client and cluster server versions:

# Inspect local kubectl version and remote API server version
kubectl version --client -o yaml
kubectl version -o yaml

Identify file system locations:

  • Binaries: /usr/bin/kubelet, /usr/bin/kubectl, /usr/bin/kubeadm
  • Configuration: /etc/kubernetes/
  • Kubelet state: /var/lib/kubelet/

5. Configuration Files and Directory Structure

Important System Files and Directories

File / Directory Purpose Important Details
~/.kube/config Client configuration file. Contains clusters, server URLs, user credentials, contexts, and default namespaces.
/etc/kubernetes/admin.conf Admin kubeconfig generated by kubeadm. Grants root-level cluster administrative access.
/etc/kubernetes/manifests/ Static Pod manifest directory. Pod YAML files placed here are directly supervised and executed by kubelet (e.g., control plane pods).
/etc/kubernetes/kubelet.conf Kubelet credentials and API endpoint reference. Used by kubelet to authenticate against kube-apiserver.
/var/lib/kubelet/ Kubelet internal working directory. Stores state, dynamic plugins, pod volume mounts, and secret keys.
/var/lib/etcd/ Database store directory for etcd. Holds persistent cluster state data; must be backed up frequently.
/var/log/pods/ & /var/log/containers/ Node-level log symlinks. Raw container output logs collected from standard output/error streams.

Structure of ~/.kube/config

apiVersion: v1
kind: Config
current-context: prod-cluster-admin
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTi...
    server: https://192.168.1.100:6443
  name: prod-cluster
users:
- name: admin-user
  user:
    client-certificate-data: LS0tLS1CRUdJTi...
    client-key-data: LS0tLS1CRUdJTi...
contexts:
- context:
    cluster: prod-cluster
    namespace: production
    user: admin-user
  name: prod-cluster-admin

Workload Manifest Structure (Deployment Example)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: production
  labels:
    app: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

To validate configuration syntax before applying:

kubectl apply --dry-run=client -f deployment.yaml

6. Service Management and Common Operations

Managing Node System Daemons

On worker and control plane nodes, daemon status is managed using standard systemctl commands:

# Inspect status of kubelet agent
systemctl status kubelet

# Inspect status of container runtime daemon
systemctl status containerd

# Restart node agent after configuration modifications
sudo systemctl restart kubelet

Verifying Node & Cluster Status

# List all nodes in the cluster with status and IP details
kubectl get nodes -o wide

# Check API Server system readiness and health endpoints
kubectl get --raw /healthz
kubectl get --raw /readyz

Workload Management Operations

Operation Command Example Explanation
Deploy / Update kubectl apply -f manifest.yaml Declaratively updates or creates resources defined in the YAML manifest.
Scale Workload kubectl scale deploy/web-app --replicas=5 -n prod Dynamically increases or decreases the running Pod count.
Rolling Update kubectl set image deploy/web-app nginx=nginx:1.26 -n prod Triggers a zero-downtime progressive Pod replacement update.
Rollout Status kubectl rollout status deploy/web-app -n prod Monitors progress of deployment rollout.
Rollback kubectl rollout undo deploy/web-app -n prod Instantly reverts to the previous ReplicaSet revision.

Difference between Reload vs. Restart: Modifying a Deployment spec updates the Pod template spec, causing a progressive rolling restart where new Pods start before old ones terminate. Updating a ConfigMap does not automatically restart Pods unless a tool (like Reloader) is used or a manual restart is issued: kubectl rollout restart deployment/web-app -n prod


7. Common Administrative Tasks and Workflows

Workflow 1: Switching Contexts and Default Namespaces

SREs routinely work across multiple environments (e.g., development, staging, production).

# List all available contexts in ~/.kube/config
kubectl config get-contexts

# Switch to production cluster context
kubectl config use-context prod-cluster-admin

# Change default namespace for subsequent commands in current context
kubectl config set-context --current --namespace=production

Workflow 2: Creating ConfigMaps and Secrets

# Create a ConfigMap from command line key-value pairs
kubectl create configmap app-config --from-literal=DB_HOST=10.0.0.5 --from-literal=DB_PORT=5432 -n production

# Create a Secret from a file containing sensitive passwords
kubectl create secret generic app-db-pass --from-file=api-key=./secret-key.txt -n production

Workflow 3: Draining a Node for System Maintenance

When performing host-level updates or kernel upgrades, safely evacuate workloads off the target node:

# 1. Mark node as unschedulable to prevent new pods from arriving
kubectl cordon node-worker-02

# 2. Evict running pods safely (ignoring DaemonSets and keeping local data limits)
kubectl drain node-worker-02 --ignore-daemonsets --delete-emptydir-data

# 3. Perform node hardware or OS maintenance...

# 4. Uncordon node to restore normal scheduling
kubectl uncordon node-worker-02

8. CLI Command Reference and Cheatsheet

Standard kubectl Commands

Command Purpose Important Flags/Arguments Example Explanation
kubectl get List resources. -n <ns>, -A (all namespaces), -o wide, -o yaml kubectl get pods -n production -o wide Fetches Pod list with node assignment and IPs.
kubectl describe Detailed inspection. Target resource and name kubectl describe pod web-app-x821 -n production Shows state, conditions, mounts, and recent lifecycle events.
kubectl logs Fetch pod logs. -f (tail), -c <container>, --previous kubectl logs web-app-x821 -c nginx -p -n production Shows logs of previous crashed container instance.
kubectl exec Execute command in container. -it (interactive tty), -c <container> kubectl exec -it web-app-x821 -n production -- sh Spawns an interactive shell inside container.
kubectl port-forward Local port forwarding. <local-port>:<remote-port> kubectl port-forward svc/db-service 5432:5432 -n production Opens local network access directly to cluster service.
kubectl auth can-i Test RBAC permissions. VERB RESOURCE, --as=<user> kubectl auth can-i create pods --as=jane -n production Checks if specified user has RBAC privileges.
kubectl top Resource metrics. pods, nodes kubectl top pods -n production Shows real-time CPU and memory usage (requires Metrics Server).

Helper Tooling (krew and Essential Plugins)

krew is the official plugin manager for kubectl.

# Install krew plugins
kubectl krew install stern ctx ns tree view-secret neat popeye
                     +----------------------------------+
                     |         kubectl krew             |
                     |        (Plugin Manager)          |
                     +----------------------------------+
                                      |
     +------------+------------+------+------+------------+------------+
     |            |            |             |            |            |
     v            v            v             v            v            v
+--------+   +--------+   +--------+    +--------+   +--------+   +--------+
| stern  |   |  ctx   |   |   ns   |    |  tree  |   | view-  |   |  neat  |
| (Log   |   | (Kube  |   | (Kube  |    | (Owner |   | secret |   | (YAML  |
| Tail)  |   | Context|   | Namesp)|    | Hier)  |   | Decode)|   | Clean) |
+--------+   +--------+   +--------+    +--------+   +--------+   +--------+
Plugin Purpose Example Command Operational Value
stern Multi-pod / multi-container log tailing. stern -l app=web-app -n production --since 10m Streams logs across dynamically created Pods using regular expressions.
**ctx / ns** Fast context/namespace switching. kubectl ctx prod-cluster / kubectl ns production Eliminates long context switching syntax.
tree Visualize object ownership hierarchies. kubectl tree deploy/web-app -n production Displays Deployment -> ReplicaSet -> Pod relationships visually.
view-secret Decode Secret objects inline. kubectl view-secret app-db-pass -n production Automatically decodes base64 strings directly on CLI without manual base64 -d piping.
neat Clean generated clutter from YAML output. `kubectl get deploy web-app -o yaml kubectl neat`
popeye Cluster hygiene and security scanner. kubectl popeye -n production Scans resources for misconfigurations, missing limits, or dead links.

9. Logs, Monitoring and Diagnostics

System Log Inspection

When debugging node-level issues (such as kubelet failures or Container Runtime errors):

# Inspect kubelet service logs on host
journalctl -u kubelet -f --no-pager

# Inspect containerd runtime logs
journalctl -u containerd -n 100

Pod Log Inspection

# Stream current logs from container
kubectl logs -f pod/web-app-x821 -c nginx -n production

# Inspect logs of container instance prior to its last restart (Crucial for crashes!)
kubectl logs pod/web-app-x821 -c nginx --previous -n production

Raw Cluster API Diagnostics (kubectl proxy)

The kubectl proxy command establishes a local HTTP proxy server to safely access the Kubernetes REST API server using local credentials:

# Start proxy background job listening on localhost:8001
kubectl proxy --port=8001 &

# Query cluster health status endpoints
curl http://localhost:8001/healthz
curl http://localhost:8001/livez
curl http://localhost:8001/readyz

# Retrieve raw cluster node metrics or object lists
curl http://localhost:8001/api/v1/nodes

10. Troubleshooting Guide and Common Failure Scenarios

Systematic Investigation Flowchart

graph TD
    A[Reported Failure / Issue] --> B{Service / App Accessible?}
    B -- No --> C[Check Pod Status: kubectl get pods -n NS]
    
    C --> D{Pod Status?}
    D -- Pending --> E[Scenario 1: Pending]
    D -- ImagePullBackOff --> F[Scenario 2: ImagePullBackOff]
    D -- CrashLoopBackOff --> G[Scenario 3: CrashLoopBackOff]
    D -- OOMKilled --> H[Scenario 4: OOMKilled]
    D -- Running --> I{Endpoints Present?}
    
    I -- No --> J[Scenario 5: Service Routing Failure]
    I -- Yes --> K[Check NetworkPolicy / Ingress Config]

    B -- Auth Error --> L[Scenario 6: Forbidden / RBAC Error]
    
    A --> M{Node Status / API Slow?}
    M -- Node NotReady --> N[Scenario 7: Node NotReady]
    M -- API Timeout --> O[Scenario 8: API Server / etcd Degradation]

Scenario 1: Pod Stuck in Pending Status

Symptoms

Pod remains in state Pending indefinitely. No container starts.

Possible Causes

  • Insufficient CPU or Memory allocatable across all worker nodes.
  • Unsatisfied nodeSelector, nodeAffinity, or volume bindings (PersistentVolumeClaim not bound).
  • Node taints present without matching Pod tolerations.

Investigation and Diagnosis

  1. Run kubectl describe pod <POD_NAME> -n <NS> and read the Events section at the bottom:
kubectl describe pod web-pending-7d6f -n production

Look for message: 0/4 nodes are available: 4 Insufficient memory. or node(s) had untolerated taint. 2. Inspect node capacity and resource allocatable allocations:

kubectl describe nodes | grep -A 8 "Allocated resources"

Root Cause Identification

If events indicate scheduling failure due to resources, compare total requested CPU/Memory against available node capacity.

Resolution

  • Adjust Pod resource requests down in the deployment manifest if overestimated.
  • Scale up worker nodes in the cluster.
  • Add matching tolerations to Pod spec if node taints are expected.

Verification

Confirm Pod switches state to ContainerCreating and then Running:

kubectl get pod web-pending-7d6f -n production -w

Prevention

Implement ResourceQuotas and set baseline resource requests matching actual application utilization.


Scenario 2: ImagePullBackOff / ErrImagePull

Symptoms

Pod status displays ImagePullBackOff or ErrImagePull.

NAME             READY   STATUS             RESTARTS   AGE
web-app-564ab    0/1     ImagePullBackOff   0          2m

Possible Causes

  • Container image tag does not exist or was misspelled.
  • Private container registry authentication missing (imagePullSecrets omitted).
  • Registry rate-limiting or network unreachable from worker node.

Investigation and Diagnosis

  1. Inspect detailed image pull failure events:
kubectl describe pod web-app-564ab -n production | grep -A 5 Events

Example message: Failed to pull image "[registry.company.com/app:v2.1](https://registry.company.com/app:v2.1)": rpc error: code = Unknown desc = Error response from daemon: pull access denied. 2. Verify image name validity manually:

docker pull registry.company.com/app:v2.1

Resolution

  • Correct image tag typo in Deployment spec.
  • Create registry credential secret and attach to service account or pod spec:
kubectl create secret docker-registry regcred \
  --docker-server=registry.company.com \
  --docker-username=dbadmin \
  --docker-password=secretpass -n production

Add to pod spec:

spec:
  imagePullSecrets:
  - name: regcred

Verification

Verify image is pulled successfully and pod enters Running state.


Scenario 3: CrashLoopBackOff

Symptoms

Pod starts, crashes, and continuously attempts restarting with exponential backoff delay. Restarts count keeps incrementing.

Possible Causes

  • Application startup code exception (missing environment variable, configuration file missing).
  • Application failed health check probe (Liveness/Readiness).
  • Container entrypoint command invalid or target port already bound.

Investigation and Diagnosis

  1. Check logs from the crashed instance immediately prior to restart:
kubectl logs web-app-crashed-9812 -n production --previous
  1. Inspect exit code details:
kubectl describe pod web-app-crashed-9812 -n production | grep -A 4 "Last State"

Exit Code 1: General app failure/uncaught exception. Exit Code 137: Killed by OS due to Out-Of-Memory (OOM). Exit Code 127: Entrypoint command or binary not found.

Resolution

  • Fix underlying code issue or supply missing ConfigMap/Secret environment variables.
  • Update container command line parameters in manifest.

Verification

Check restart count stabilizes and pod health remains READY 1/1.


Scenario 4: Pod Terminated due to OOMKilled

Symptoms

Container crashes abruptly. kubectl describe pod shows Reason: OOMKilled and Exit Code: 137.

Possible Causes

  • Container memory consumption exceeded its configured spec.containers[].resources.limits.memory.
  • Application contains a memory leak.

Investigation and Diagnosis

  1. Confirm OOM termination event:
kubectl describe pod app-mem-6712 -n production | grep -E "OOMKilled|Exit Code"
  1. Monitor memory trend prior to crash using metrics:
kubectl top pod app-mem-6712 -n production

Resolution

  • Increase memory resource limits in manifest:
resources:
  limits:
    memory: "512Mi" # Upgraded from 256Mi
  • Profile application code to patch memory leaks.

Verification

Verify Pod running stable without crashing under load.


Scenario 5: Service Routing / Endpoints Not Reachable

Symptoms

Requests routed to internal Service IP or DNS name fail or connection times out.

Possible Causes

  • Service selector labels do not match target Pod labels (causing empty Endpoints list).
  • Application container not listening on target containerPort.
  • NetworkPolicy blocking inter-pod communication.

Investigation and Diagnosis

  1. Verify if Service has active Pod IP endpoints attached:
kubectl get endpoints web-service -n production

If ENDPOINTS displays <none>, label selector mismatch exists. 2. Verify Pod labels against Service selector:

kubectl get svc web-service -n production -o yaml | grep -A 4 selector
kubectl get pods -n production --show-labels
  1. Test connectivity from inside pod network using a netshoot/busybox ephemeral pod:
kubectl run tmp-net-test --rm -it --image=nicolaka/netshoot -n production -- curl -v http://web-service:80

Resolution

  • Align spec.selector in Service manifest with target Pod metadata.labels.
  • Adjust NetworkPolicy rules to permit ingress traffic on specified port.

Verification

kubectl get endpoints displays valid pod IP list and test curl returns HTTP 200.


Scenario 6: Forbidden / RBAC Authorization Errors

Symptoms

Client tool, service account, or application returns API error: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:..." cannot list resource....

Possible Causes

  • Missing Role / ClusterRole permissions.
  • ServiceAccount missing appropriate RoleBinding.
  • Command issued against wrong namespace context.

Investigation and Diagnosis

  1. Test RBAC permissions using can-i:
kubectl auth can-i list pods --as=system:serviceaccount:production:my-sa -n production
  1. Verify active namespace setting:
kubectl config view --minify | grep namespace

Resolution

Create or update RoleBinding binding the required permissions to the ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-read-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: my-sa
  namespace: production
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Verification

Re-run kubectl auth can-i to confirm command returns yes.


Scenario 7: Node Status NotReady

Symptoms

Worker node status transitions to NotReady in kubectl get nodes. Pods on the node are marked for eviction after timeout.

NAME            STATUS     ROLES    AGE   VERSION
node-worker-01  NotReady   <none>   45d   v1.30.1

Possible Causes

  • kubelet daemon service stopped or crashed on host.
  • Node experiencing severe disk pressure (DiskPressure), memory pressure, or out of PIDs.
  • Container runtime service (containerd) dead or unresponsive.
  • Network partition between worker node and control plane.

Investigation and Diagnosis

  1. Inspect Node conditions:
kubectl describe node node-worker-01 | grep -A 10 Conditions
  1. Log into target node via SSH and verify daemons:
systemctl status kubelet containerd
journalctl -u kubelet -n 50 --no-pager
  1. Check host disk space:
df -h /var/lib/containerd /var/lib/kubelet

Resolution

  • Restart failed node services: sudo systemctl restart containerd kubelet.
  • Prune unused container images if disk pressure exists: sudo crictl rmi --prune.

Verification

Confirm node transitions back to Ready status.


Scenario 8: API Server Slow or kubectl Timeouts (etcd Bottleneck)

Symptoms

kubectl operations fail with client timeout errors. API server responses are severely delayed.

Possible Causes

  • etcd disk write latency degradation or high memory utilization.
  • etcd cluster lost quorum (in multi-master setup).
  • Excessive watch requests or unindexed API queries flooding kube-apiserver.

Investigation and Diagnosis

  1. Check overall control plane API health:
kubectl get --raw /healthz
  1. Check etcd cluster health metrics (on control plane nodes running etcdctl):
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  endpoint health

Resolution

  • Store etcd directory /var/lib/etcd on fast dedicated NVMe SSD disks to minimize write latency.
  • Defragment etcd database and restore quorum if node lost.

Verification

API response times return to normal sub-second latencies.


Scenario 9: Helm Release Failed or Stuck

Symptoms

Helm deployments hang in pending-upgrade or fail mid-deployment, leaving half-applied resources.

Possible Causes

  • Pre-install or post-upgrade hooks failed to execute.
  • Resource validation failure during manifest application.

Investigation and Diagnosis

  1. Check release status: helm status <RELEASE_NAME> -n <NS>
  2. Inspect underlying failed pod objects created by Helm using standard kubectl commands:
kubectl get pods -n <NS> --selector=app.kubernetes.io/instance=<RELEASE_NAME>

Resolution

Roll back to last known healthy release version:

helm rollback <RELEASE_NAME> <REVISION_NUMBER> -n <NS>

11. Security and Best Practices

Least Privilege RBAC Management

Avoid using cluster-admin for daily administration. Create granular roles scoped strictly to relevant namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: developer-role
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update"]

Workload Isolation and Security Contexts

Enforce pod security standards to block running applications as root:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    readOnlyRootFilesystem: true
    capabilities:
      drop:
      - ALL

Network Policy Hardening

By default, all Pods in a Kubernetes cluster can talk to each other. Implement default-deny NetworkPolicies to isolate components:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Secrets Security

  • Do not store raw secrets in Git repositories. Use tools like SOPS or Sealed Secrets.
  • Enable KMS (Key Management Service) encryption at rest in kube-apiserver so secrets saved in etcd are strongly encrypted.

12. Performance and Resource Optimisation

Sizing Requests and Limits

Setting CPU and Memory requests and limits correctly is crucial for performance stability:

resources:
  requests:    # Used by scheduler to pick node
    cpu: "250m"
    memory: "256Mi"
  limits:      # Upper threshold; enforced by kernel cgroups
    cpu: "500m"
    memory: "512Mi"
  • CPU Requests: Used by kube-scheduler to place pods. CPU limits use CFS quotas; exceeding CPU limits causes throttling (slowness), but does not terminate the pod.
  • Memory Limits: Hard ceiling. Exceeding memory limits triggers kernel Out-Of-Memory (OOM) killer immediately, terminating the pod (OOMKilled).

Node Allocatable Architecture

Worker nodes reserve capacity for system daemons and kernel tasks:

$$\text{Node Total Capacity} = \text{Allocatable} + \text{kube-reserved} + \text{system-reserved} + \text{eviction-threshold}$$

+-------------------------------------------------------------------+
|                        Node Total Capacity                        |
+-------------------+--------------------+------------------+-------+
|  Allocatable      |  kube-reserved     | system-reserved  | Evict |
|  (Pods run here)  |  (kubelet, CRI)    | (sshd, systemd)  | Thresh|
+-------------------+--------------------+------------------+-------+

13. Backup, Recovery and Maintenance

etcd Backup Snapshot Procedure

The entire cluster state lives inside etcd. Create regular snapshot backups before major upgrades:

# Save etcd snapshot
sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  snapshot save /var/backups/etcd-snapshot-$(date +%Y%m%d).db

# Verify snapshot integrity
sudo ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd-snapshot-$(date +%Y%m%d).db -w table

Restoring etcd State

To restore cluster state from a snapshot:

sudo ETCDCTL_API=3 etcdctl snapshot restore /var/backups/etcd-snapshot-20261103.db \
  --data-dir=/var/lib/etcd-restored

Update /etc/kubernetes/manifests/etcd.yaml host path to point to /var/lib/etcd-restored.


14. Automation and Integration

Package Management with Helm

Helm packages related Kubernetes manifests into reusable, versioned charts.

# Add third-party helm repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# Install chart release into dedicated namespace
helm install ingress-controller ingress-nginx/ingress-nginx --namespace ingress-system --create-namespace

Continuous Deployment via GitOps (ArgoCD Concept)

Rather than manually running kubectl apply, SRE teams store declarative Kubernetes manifests in a Git repository. GitOps controllers (like ArgoCD or Flux) watch the Git repo and automatically reconcile live cluster state with Git commits.

+------------------+         Git Commit         +-----------------------+
|  Git Repository  | ------------------------>  |  ArgoCD Controller    |
| (YAML Manifests) |                            | (Runs inside Cluster) |
+------------------+                            +-----------------------+
                                                            |
                                                            v Auto-Sync
                                                +-----------------------+
                                                |   Kubernetes API      |
                                                +-----------------------+

15. Practical Labs and Exercises

Lab 1: Deploying a Scalable Web Application (Beginner)

Objective

Deploy a 3-replica NGINX web application and expose it internally via a ClusterIP service.

Instructions

  1. Save the following code as lab1-web.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lab1-nginx
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: lab1-nginx
  template:
    metadata:
      labels:
        app: lab1-nginx
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: lab1-service
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: lab1-nginx
  ports:
  - port: 80
    targetPort: 80
  1. Apply the manifest: kubectl apply -f lab1-web.yaml

Validation

Check pod statuses and test internal service resolution:

kubectl get pods -l app=lab1-nginx -o wide
kubectl run test-pod --rm -it --image=busybox -- wget -qO- http://lab1-service

Lab 2: Troubleshooting Broken Image & Missing Config (Intermediate)

Objective

Diagnose and fix a deployment stuck in ImagePullBackOff and CrashLoopBackOff.

Instructions

  1. Apply broken manifest lab2-broken.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broken-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: broken-app
  template:
    metadata:
      labels:
        app: broken-app
    spec:
      containers:
      - name: app
        image: nginx:nonexistent-tag-999
  1. Run kubectl get pods and observe status (ImagePullBackOff).
  2. Run kubectl describe pod -l app=broken-app to read event failure reasons.
  3. Fix the manifest by updating tag to nginx:latest and re-apply.

Validation

Confirm the Pod reaches state READY 1/1.


Lab 3: Deep Connectivity Debugging with Ephemeral Debug Containers (Advanced)

Objective

Debug application environment and test network routing inside a distroless app container that has no shell using kubectl debug.

Instructions

  1. Attach an ephemeral container equipped with debugging tools to a running pod:
kubectl debug pod/lab1-nginx-7d84b986b-x291 -it --image=nicolaka/netshoot --target=nginx
  1. Inside the debug prompt, execute network probes:
netstat -tulpn
dig lab1-service.default.svc.cluster.local
  1. Exit shell (exit).

16. Quick Reference and Cheat Sheet

Important Configuration Files

File / Location Purpose
~/.kube/config Main local client configuration for cluster access.
/etc/kubernetes/manifests/ Host static Pod manifests (Control plane daemons).
/etc/kubernetes/admin.conf Cluster root administrative credentials.
/var/lib/kubelet/ Internal runtime directory for Kubelet state data.

Important Log Locations

Log Target Command / Path Purpose
kubelet service journalctl -u kubelet -f Node agent operational logs.
containerd runtime journalctl -u containerd -f Container lifecycle & runtime errors.
Pod container logs kubectl logs <POD> -c <CONTAINER> -n <NS> Application stdout/stderr output.
Crashed pod logs kubectl logs <POD> --previous -n <NS> Last log stream prior to container exit.

Common Administrative Commands

Task Command
List all Pods across namespaces kubectl get pods -A -o wide
Switch cluster context kubectl config use-context <CONTEXT_NAME>
Change default namespace kubectl config set-context --current --namespace=<NS>
Stream logs matching label stern -l app=<LABEL> -n <NS>
Decode Secret value kubectl view-secret secret/<SECRET_NAME> -n <NS>
Drain node for maintenance kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data
Execute temp debug container kubectl run tmp-shell --rm -it --image=nicolaka/netshoot -- bash

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command Primary Suspect
Pending kubectl describe pod <POD> -n <NS> Insufficient CPU/Memory, Taints/Tolerations.
ImagePullBackOff kubectl describe pod <POD> -n <NS> Misspelled image tag, missing secret.
CrashLoopBackOff kubectl logs <POD> --previous -n <NS> App crash, missing env var/config.
OOMKilled kubectl describe pod <POD> -n <NS> Memory limit set too low.
Service Unreachable kubectl get endpoints <SVC> -n <NS> Label selector mismatch.
Forbidden (403) kubectl auth can-i <VERB> <RESOURCE> Missing RBAC RoleBinding.
**Node NotReady** journalctl -u kubelet on worker node kubelet service or container runtime down.

Ports and Protocols

Port Protocol Component / Purpose
6443 TCP kube-apiserver HTTPS endpoint.
2379 - 2380 TCP etcd client/peer communication.
10250 TCP kubelet API (Control Plane to Worker).
30000 - 32767 TCP/UDP NodePort Service default exposure range.

17. Further Reading and References

  1. Kubernetes Official Documentation: https://kubernetes.io/docs
  2. Kubernetes Concepts Guide: https://kubernetes.io/docs/concepts
  3. kubectl CLI Command Reference: https://kubernetes.io/docs/reference/kubectl/
  4. Krew Plugin Index Repository: https://krew.sigs.k8s.io/plugins/
  5. SadServers Hands-On Debugging Playgrounds: https://sadservers.com