1. Overview
Kubernetes (often abbreviated as K8s) is an open-source container orchestration platform designed to automate the deployment, scaling, management, and operational lifecycle of containerized applications.
In modern infrastructure, managing hundreds of isolated containers across bare-metal servers or virtual machines creates significant operational complexity. Manually tracking container placement, handling hardware failures, updating software without downtime, and balancing network traffic across nodes is error-prone and unscalable. Kubernetes solves this problem by abstracting individual physical or virtual machines into a single unified pool of compute resources.
Instead of manually starting containers on specific host machines, system administrators and Site Reliability Engineers (SREs) declare the desired state of their infrastructure and workloads using declarative configuration files (typically written in YAML). Kubernetes control loops continuously observe the actual state of the cluster and make automated adjustments to reconcile the actual state back to the defined desired state.
+-----------------------------------------------------------------------+
| Kubernetes Cluster |
| |
| +-----------------------+ +-----------------------+ |
| | Worker Node A | | Worker Node B | |
| | +-----------------+ | | +-----------------+ | |
| | | Pod (App v1.0) | | | | Pod (App v1.0) | | |
| | +-----------------+ | | +-----------------+ | |
| +-----------------------+ +-----------------------+ |
| ^ ^ |
+---------------|---------------------------------------|---------------+
| Automated Reconciliation Loop |
+---------------------------------------+
|
+-------------------------------+
| Desired State: 2 Replicas |
+-------------------------------+
Common Use Cases
- Microservices Hosting: Running decoupled, independently scalable software microservices with internal service discovery.
- Stateless and Stateful Applications: Managing both ephemeral web applications and persistent storage systems like databases.
- Automated Batch Jobs: Executing scheduled cron jobs and batch data processing pipelines.
- Hybrid & Multi-Cloud Deployments: Standardizing operational primitives across on-premises data centers and public cloud providers (such as AWS EKS, Google GKE, and Azure AKS).
Advantages and Limitations
| Category | Description / Details |
|---|---|
| Key Advantages | High availability, auto-scaling, self-healing container restarts, declarative rollouts and rollbacks, automated service discovery, load balancing, and resource optimization. |
| Limitations | Steep learning curve, operational complexity for self-managed control planes, network overhead, and potential resource waste if limits/requests are misconfigured. |
Kubernetes vs. Alternatives
- Docker Compose: Ideal for single-host multi-container development environments, but lacks native multi-node clustering, auto-scaling, and advanced self-healing.
- HashiCorp Nomad: A lightweight workload orchestrator for containers and non-containerized applications; simpler to operate than Kubernetes, but has a smaller ecosystem and fewer built-in cloud-native extensions.
Operational Context
In an SRE environment, engineers rarely think about which specific virtual machine hosts a running process. Instead, operations revolve around Kubernetes primitives such as Pods, Deployments, Services, and Ingress controllers.
2. Architecture and Mental Model
To effectively operate and debug a Kubernetes cluster, administrators must understand its split Control Plane and Worker Node architecture.
graph TB
subgraph Control_Plane["Control Plane (Master Nodes)"]
APISERVER["kube-apiserver"]
ETCD["etcd (Key-Value Store)"]
SCHEDULER["kube-scheduler"]
CM["kube-controller-manager"]
CCM["cloud-controller-manager"]
APISERVER <--> ETCD
SCHEDULER <--> APISERVER
CM <--> APISERVER
CCM <--> APISERVER
end
subgraph Worker_Node_1["Worker Node 1"]
KUBELET1["kubelet"]
KPROXY1["kube-proxy"]
CRUN1["Container Runtime (containerd)"]
POD1["Pod A"]
POD2["Pod B"]
KUBELET1 <--> APISERVER
KUBELET1 --> CRUN1
CRUN1 --> POD1
CRUN1 --> POD2
KPROXY1 <--> APISERVER
end
subgraph Worker_Node_2["Worker Node 2"]
KUBELET2["kubelet"]
KPROXY2["kube-proxy"]
CRUN2["Container Runtime (containerd)"]
POD3["Pod C"]
KUBELET2 <--> APISERVER
KUBELET2 --> CRUN2
CRUN2 --> POD3
KPROXY2 <--> APISERVER
end
KUBECTL["kubectl CLI"] --> APISERVER
Control Plane Components
The control plane makes global decisions about the cluster (e.g., scheduling pods), detects cluster events, and responds to changes.
kube-apiserver: The REST API gateway and central communication hub for the entire cluster. All CLI client tools (kubectl), node components, and control plane services communicate exclusively through the API server over HTTPS.etcd: A consistent, highly available distributed key-value store that acts as the single source of truth for all cluster state, object definitions, and metadata.kube-scheduler: Inspects newly created Pods that have no assigned worker node and selects the most optimal node for them based on resource requests, hardware constraints, node affinity, taints, and tolerations.kube-controller-manager: Runs daemon controllers (such as DeploymentController, NodeController, ReplicaSetController) in a single process to continuously reconcile actual state with desired state.cloud-controller-manager: Integrates underlying cloud provider APIs (AWS, GCP, Azure) to provision resources like cloud load balancers, node IP routing, and persistent storage volumes.
Worker Node Components
Worker nodes execute actual application workloads within containers.
kubelet: An agent running on every node that watches the API server for PodSpecs assigned to its node and ensures that the corresponding containers are running and healthy.kube-proxy: Maintains network rules on each host system to implement KubernetesServiceabstractions, performing IP forwarding and load balancing via Linux iptables or IPVS.- Container Runtime: The underlying engine responsible for running containers (e.g.,
containerdorCRI-O, communicating via the Container Runtime Interface / CRI).
kubectl Interaction Lifecycle
When an administrator executes a command such as kubectl apply -f deployment.yaml:
sequenceDiagram
autonumber
actor Admin as SysAdmin / kubectl
participant API as kube-apiserver
participant ETCD as etcd Store
participant CTRL as kube-controller-manager
participant SCHED as kube-scheduler
participant KUBELET as Worker kubelet
Admin->>API: 1. Submit YAML over HTTPS (~/.kube/config)
API->>API: 2. Authenticate, Authorize (RBAC) & Validate Schema
API->>ETCD: 3. Persist Object Data
ETCD-->>API: 4. Data Saved
CTRL->>API: 5. Watcher detects missing Pods (ReplicaSet)
CTRL->>API: 6. Request creation of new Pod spec
SCHED->>API: 7. Watcher detects unscheduled Pod
SCHED->>API: 8. Assigns Pod to Worker Node
API->>ETCD: 9. Update Pod binding in etcd
KUBELET->>API: 10. Watcher detects Pod assigned to local node
KUBELET->>KUBELET: 11. Instruct container runtime to pull image & start containers
KUBELET->>API: 12. Report Pod status back (Running)
- Config & Request:
kubectlreads authentication credentials and API endpoints from~/.kube/configand sends an HTTPS REST request tokube-apiserver. - Admission & Validation:
kube-apiserverauthenticates the request, verifies RBAC authorization, passes it through admission webhooks (mutating and validating), and validates the schema. - Persist: The API server writes the validated resource definition into
etcd. - Reconcile: The relevant controller in
kube-controller-managernotices the change via an active API watch stream and creates missing underlying resources (e.g., ReplicaSet creates Pod specs). - Schedule:
kube-schedulerobserves unscheduled Pods and assigns them to an appropriate node. - Execution: The
kubeleton the target worker node detects the scheduled Pod, instructs the local Container Runtime (e.g.,containerd) to pull the container image and create network namespaces, and periodically updates status back tokube-apiserver.
Managed vs Self-Managed: In managed offerings (AWS EKS, Google GKE, Azure AKS), cloud providers manage and hide the control plane components (
etcd,kube-apiserver, etc.). However, SREs still manage and debug workloads on worker nodes using identicalkubectlworkflows.
3. Core Concepts and Terminology
Workload Objects
- Pod: The smallest and basic deployable computing unit in Kubernetes. A Pod wraps one or more tightly coupled containers that share a network namespace (same IP address and port space), IPC namespace, and storage volumes.
- Deployment: Declarative controller used for running stateless applications. Manages underlying
ReplicaSetsto provide declarative rolling updates, instant rollbacks, and replica scaling. - StatefulSet: Used for stateful applications (such as databases or message queues) requiring stable network identifiers (e.g.,
pod-0,pod-1), persistent storage bindings, and ordered deployments. - DaemonSet: Guarantees that a copy of a specified Pod runs on all (or selected) worker nodes. Commonly used for node-level agents, log collectors (e.g., Fluentbit), and network monitoring tools.
- Job & CronJob: A
Jobcreates Pods that run until a specific task completes successfully. ACronJobruns batch tasks on a periodic schedule using standard cron syntax.
Networking and Exposure Primitives
+-------------------------------------------------------------------------+
| Ingress |
| (HTTP Routing: app.com) |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Service |
| (Stable Virtual IP / ClusterIP) |
+-------------------------------------------------------------------------+
|
+----------------------+----------------------+
| |
v v
+--------------------------+ +--------------------------+
| Pod 1 | | Pod 2 |
| (IP: 10.244.1.15:8080) | | (IP: 10.244.2.22:8080) |
+--------------------------+ +--------------------------+
-
Service: An abstract logical layer that defines a policy-based policy and access mechanism for a set of ephemeral Pods. Provides a stable virtual IP (ClusterIP) and DNS entry.
-
ClusterIP: Default type. Exposes the Service on an internal IP inside the cluster. -
NodePort: Exposes the Service on each worker node’s IP at a static port (typically30000-32767). -
LoadBalancer: Provisions an external, vendor-managed cloud load balancer that routes directly to NodePorts or Pod IPs. -
Ingress: An API object managing external HTTP/S routing into cluster services based on path or host rules. Requires an active Ingress Controller (such as NGINX or Traefik) to execute the routing.
-
NetworkPolicy: Acts as an internal firewall between Pods, controlling layer 3/4 network traffic based on labels and CIDR blocks. Must be supported by the cluster Container Network Interface (CNI) plugin (e.g., Calico, Cilium).
-
Cluster DNS: Internal DNS resolving service names into IP addresses following the standard format:
<service-name>.<namespace>.svc.cluster.local.
Configuration and Isolation
- ConfigMap: Stores non-sensitive configuration parameters as key-value pairs or complete configuration files. Mounted into Pods as environment variables or volume files.
- Secret: Holds sensitive data such as API keys, passwords, and SSH keys. Stored as base64-encoded strings (note: base64 is encoding, not encryption; requires encryption-at-rest configuration).
- Namespace: Virtual clusters providing logical isolation for resources, resource quotas, and team environments (e.g.,
default,kube-system,production,staging). - Role-Based Access Control (RBAC): Restricts operations within the cluster.
Role/ClusterRole: Defines API permissions (verbs likeget,list,create,deleteon resources) within a namespace or cluster-wide.RoleBinding/ClusterRoleBinding: Grants defined roles to users, groups, orServiceAccounts.
4. Installation and Version Management
Prerequisites
Before setting up a node with container components:
- Disable Linux swap space to ensure deterministic memory allocation for
kubelet:
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
- Enable kernel module loading and IP forwarding required for pod networking:
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
Package Manager Installation (Debian/Ubuntu Example)
To install core control components manually via apt:
# Install dependencies
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
# Add Kubernetes GPG signing key and repository
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
# Install kubelet, kubeadm, and kubectl
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
Local Development and Lightweight Distros
minikube/kind(Kubernetes in Docker): Runs complete multi-node clusters inside local Docker containers for rapid testing.k3s: Lightweight single-binary Kubernetes distribution optimized for edge environments and lab nodes.
Version Management and Verification
Check client and cluster server versions:
# Inspect local kubectl version and remote API server version
kubectl version --client -o yaml
kubectl version -o yaml
Identify file system locations:
- Binaries:
/usr/bin/kubelet,/usr/bin/kubectl,/usr/bin/kubeadm - Configuration:
/etc/kubernetes/ - Kubelet state:
/var/lib/kubelet/
5. Configuration Files and Directory Structure
Important System Files and Directories
| File / Directory | Purpose | Important Details |
|---|---|---|
~/.kube/config |
Client configuration file. | Contains clusters, server URLs, user credentials, contexts, and default namespaces. |
/etc/kubernetes/admin.conf |
Admin kubeconfig generated by kubeadm. |
Grants root-level cluster administrative access. |
/etc/kubernetes/manifests/ |
Static Pod manifest directory. | Pod YAML files placed here are directly supervised and executed by kubelet (e.g., control plane pods). |
/etc/kubernetes/kubelet.conf |
Kubelet credentials and API endpoint reference. | Used by kubelet to authenticate against kube-apiserver. |
/var/lib/kubelet/ |
Kubelet internal working directory. | Stores state, dynamic plugins, pod volume mounts, and secret keys. |
/var/lib/etcd/ |
Database store directory for etcd. |
Holds persistent cluster state data; must be backed up frequently. |
/var/log/pods/ & /var/log/containers/ |
Node-level log symlinks. | Raw container output logs collected from standard output/error streams. |
Structure of ~/.kube/config
apiVersion: v1
kind: Config
current-context: prod-cluster-admin
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTi...
server: https://192.168.1.100:6443
name: prod-cluster
users:
- name: admin-user
user:
client-certificate-data: LS0tLS1CRUdJTi...
client-key-data: LS0tLS1CRUdJTi...
contexts:
- context:
cluster: prod-cluster
namespace: production
user: admin-user
name: prod-cluster-admin
Workload Manifest Structure (Deployment Example)
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
namespace: production
labels:
app: web-app
spec:
replicas: 3
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: nginx
image: nginx:1.25
ports:
- containerPort: 80
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
To validate configuration syntax before applying:
kubectl apply --dry-run=client -f deployment.yaml
6. Service Management and Common Operations
Managing Node System Daemons
On worker and control plane nodes, daemon status is managed using standard systemctl commands:
# Inspect status of kubelet agent
systemctl status kubelet
# Inspect status of container runtime daemon
systemctl status containerd
# Restart node agent after configuration modifications
sudo systemctl restart kubelet
Verifying Node & Cluster Status
# List all nodes in the cluster with status and IP details
kubectl get nodes -o wide
# Check API Server system readiness and health endpoints
kubectl get --raw /healthz
kubectl get --raw /readyz
Workload Management Operations
| Operation | Command Example | Explanation |
|---|---|---|
| Deploy / Update | kubectl apply -f manifest.yaml |
Declaratively updates or creates resources defined in the YAML manifest. |
| Scale Workload | kubectl scale deploy/web-app --replicas=5 -n prod |
Dynamically increases or decreases the running Pod count. |
| Rolling Update | kubectl set image deploy/web-app nginx=nginx:1.26 -n prod |
Triggers a zero-downtime progressive Pod replacement update. |
| Rollout Status | kubectl rollout status deploy/web-app -n prod |
Monitors progress of deployment rollout. |
| Rollback | kubectl rollout undo deploy/web-app -n prod |
Instantly reverts to the previous ReplicaSet revision. |
Difference between Reload vs. Restart: Modifying a Deployment spec updates the Pod template spec, causing a progressive rolling restart where new Pods start before old ones terminate. Updating a ConfigMap does not automatically restart Pods unless a tool (like Reloader) is used or a manual restart is issued:
kubectl rollout restart deployment/web-app -n prod
7. Common Administrative Tasks and Workflows
Workflow 1: Switching Contexts and Default Namespaces
SREs routinely work across multiple environments (e.g., development, staging, production).
# List all available contexts in ~/.kube/config
kubectl config get-contexts
# Switch to production cluster context
kubectl config use-context prod-cluster-admin
# Change default namespace for subsequent commands in current context
kubectl config set-context --current --namespace=production
Workflow 2: Creating ConfigMaps and Secrets
# Create a ConfigMap from command line key-value pairs
kubectl create configmap app-config --from-literal=DB_HOST=10.0.0.5 --from-literal=DB_PORT=5432 -n production
# Create a Secret from a file containing sensitive passwords
kubectl create secret generic app-db-pass --from-file=api-key=./secret-key.txt -n production
Workflow 3: Draining a Node for System Maintenance
When performing host-level updates or kernel upgrades, safely evacuate workloads off the target node:
# 1. Mark node as unschedulable to prevent new pods from arriving
kubectl cordon node-worker-02
# 2. Evict running pods safely (ignoring DaemonSets and keeping local data limits)
kubectl drain node-worker-02 --ignore-daemonsets --delete-emptydir-data
# 3. Perform node hardware or OS maintenance...
# 4. Uncordon node to restore normal scheduling
kubectl uncordon node-worker-02
8. CLI Command Reference and Cheatsheet
Standard kubectl Commands
| Command | Purpose | Important Flags/Arguments | Example | Explanation |
|---|---|---|---|---|
kubectl get |
List resources. | -n <ns>, -A (all namespaces), -o wide, -o yaml |
kubectl get pods -n production -o wide |
Fetches Pod list with node assignment and IPs. |
kubectl describe |
Detailed inspection. | Target resource and name | kubectl describe pod web-app-x821 -n production |
Shows state, conditions, mounts, and recent lifecycle events. |
kubectl logs |
Fetch pod logs. | -f (tail), -c <container>, --previous |
kubectl logs web-app-x821 -c nginx -p -n production |
Shows logs of previous crashed container instance. |
kubectl exec |
Execute command in container. | -it (interactive tty), -c <container> |
kubectl exec -it web-app-x821 -n production -- sh |
Spawns an interactive shell inside container. |
kubectl port-forward |
Local port forwarding. | <local-port>:<remote-port> |
kubectl port-forward svc/db-service 5432:5432 -n production |
Opens local network access directly to cluster service. |
kubectl auth can-i |
Test RBAC permissions. | VERB RESOURCE, --as=<user> |
kubectl auth can-i create pods --as=jane -n production |
Checks if specified user has RBAC privileges. |
kubectl top |
Resource metrics. | pods, nodes |
kubectl top pods -n production |
Shows real-time CPU and memory usage (requires Metrics Server). |
Helper Tooling (krew and Essential Plugins)
krew is the official plugin manager for kubectl.
# Install krew plugins
kubectl krew install stern ctx ns tree view-secret neat popeye
+----------------------------------+
| kubectl krew |
| (Plugin Manager) |
+----------------------------------+
|
+------------+------------+------+------+------------+------------+
| | | | | |
v v v v v v
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
| stern | | ctx | | ns | | tree | | view- | | neat |
| (Log | | (Kube | | (Kube | | (Owner | | secret | | (YAML |
| Tail) | | Context| | Namesp)| | Hier) | | Decode)| | Clean) |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
| Plugin | Purpose | Example Command | Operational Value |
|---|---|---|---|
stern |
Multi-pod / multi-container log tailing. | stern -l app=web-app -n production --since 10m |
Streams logs across dynamically created Pods using regular expressions. |
**ctx / ns** |
Fast context/namespace switching. | kubectl ctx prod-cluster / kubectl ns production |
Eliminates long context switching syntax. |
tree |
Visualize object ownership hierarchies. | kubectl tree deploy/web-app -n production |
Displays Deployment -> ReplicaSet -> Pod relationships visually. |
view-secret |
Decode Secret objects inline. | kubectl view-secret app-db-pass -n production |
Automatically decodes base64 strings directly on CLI without manual base64 -d piping. |
neat |
Clean generated clutter from YAML output. | `kubectl get deploy web-app -o yaml | kubectl neat` |
popeye |
Cluster hygiene and security scanner. | kubectl popeye -n production |
Scans resources for misconfigurations, missing limits, or dead links. |
9. Logs, Monitoring and Diagnostics
System Log Inspection
When debugging node-level issues (such as kubelet failures or Container Runtime errors):
# Inspect kubelet service logs on host
journalctl -u kubelet -f --no-pager
# Inspect containerd runtime logs
journalctl -u containerd -n 100
Pod Log Inspection
# Stream current logs from container
kubectl logs -f pod/web-app-x821 -c nginx -n production
# Inspect logs of container instance prior to its last restart (Crucial for crashes!)
kubectl logs pod/web-app-x821 -c nginx --previous -n production
Raw Cluster API Diagnostics (kubectl proxy)
The kubectl proxy command establishes a local HTTP proxy server to safely access the Kubernetes REST API server using local credentials:
# Start proxy background job listening on localhost:8001
kubectl proxy --port=8001 &
# Query cluster health status endpoints
curl http://localhost:8001/healthz
curl http://localhost:8001/livez
curl http://localhost:8001/readyz
# Retrieve raw cluster node metrics or object lists
curl http://localhost:8001/api/v1/nodes
10. Troubleshooting Guide and Common Failure Scenarios
Systematic Investigation Flowchart
graph TD
A[Reported Failure / Issue] --> B{Service / App Accessible?}
B -- No --> C[Check Pod Status: kubectl get pods -n NS]
C --> D{Pod Status?}
D -- Pending --> E[Scenario 1: Pending]
D -- ImagePullBackOff --> F[Scenario 2: ImagePullBackOff]
D -- CrashLoopBackOff --> G[Scenario 3: CrashLoopBackOff]
D -- OOMKilled --> H[Scenario 4: OOMKilled]
D -- Running --> I{Endpoints Present?}
I -- No --> J[Scenario 5: Service Routing Failure]
I -- Yes --> K[Check NetworkPolicy / Ingress Config]
B -- Auth Error --> L[Scenario 6: Forbidden / RBAC Error]
A --> M{Node Status / API Slow?}
M -- Node NotReady --> N[Scenario 7: Node NotReady]
M -- API Timeout --> O[Scenario 8: API Server / etcd Degradation]
Scenario 1: Pod Stuck in Pending Status
Symptoms
Pod remains in state Pending indefinitely. No container starts.
Possible Causes
- Insufficient CPU or Memory allocatable across all worker nodes.
- Unsatisfied
nodeSelector,nodeAffinity, or volume bindings (PersistentVolumeClaimnot bound). - Node taints present without matching Pod tolerations.
Investigation and Diagnosis
- Run
kubectl describe pod <POD_NAME> -n <NS>and read the Events section at the bottom:
kubectl describe pod web-pending-7d6f -n production
Look for message: 0/4 nodes are available: 4 Insufficient memory. or node(s) had untolerated taint.
2. Inspect node capacity and resource allocatable allocations:
kubectl describe nodes | grep -A 8 "Allocated resources"
Root Cause Identification
If events indicate scheduling failure due to resources, compare total requested CPU/Memory against available node capacity.
Resolution
- Adjust Pod resource requests down in the deployment manifest if overestimated.
- Scale up worker nodes in the cluster.
- Add matching tolerations to Pod spec if node taints are expected.
Verification
Confirm Pod switches state to ContainerCreating and then Running:
kubectl get pod web-pending-7d6f -n production -w
Prevention
Implement ResourceQuotas and set baseline resource requests matching actual application utilization.
Scenario 2: ImagePullBackOff / ErrImagePull
Symptoms
Pod status displays ImagePullBackOff or ErrImagePull.
NAME READY STATUS RESTARTS AGE
web-app-564ab 0/1 ImagePullBackOff 0 2m
Possible Causes
- Container image tag does not exist or was misspelled.
- Private container registry authentication missing (
imagePullSecretsomitted). - Registry rate-limiting or network unreachable from worker node.
Investigation and Diagnosis
- Inspect detailed image pull failure events:
kubectl describe pod web-app-564ab -n production | grep -A 5 Events
Example message: Failed to pull image "[registry.company.com/app:v2.1](https://registry.company.com/app:v2.1)": rpc error: code = Unknown desc = Error response from daemon: pull access denied.
2. Verify image name validity manually:
docker pull registry.company.com/app:v2.1
Resolution
- Correct image tag typo in Deployment spec.
- Create registry credential secret and attach to service account or pod spec:
kubectl create secret docker-registry regcred \
--docker-server=registry.company.com \
--docker-username=dbadmin \
--docker-password=secretpass -n production
Add to pod spec:
spec:
imagePullSecrets:
- name: regcred
Verification
Verify image is pulled successfully and pod enters Running state.
Scenario 3: CrashLoopBackOff
Symptoms
Pod starts, crashes, and continuously attempts restarting with exponential backoff delay. Restarts count keeps incrementing.
Possible Causes
- Application startup code exception (missing environment variable, configuration file missing).
- Application failed health check probe (Liveness/Readiness).
- Container entrypoint command invalid or target port already bound.
Investigation and Diagnosis
- Check logs from the crashed instance immediately prior to restart:
kubectl logs web-app-crashed-9812 -n production --previous
- Inspect exit code details:
kubectl describe pod web-app-crashed-9812 -n production | grep -A 4 "Last State"
Exit Code 1: General app failure/uncaught exception. Exit Code 137: Killed by OS due to Out-Of-Memory (OOM). Exit Code 127: Entrypoint command or binary not found.
Resolution
- Fix underlying code issue or supply missing ConfigMap/Secret environment variables.
- Update container command line parameters in manifest.
Verification
Check restart count stabilizes and pod health remains READY 1/1.
Scenario 4: Pod Terminated due to OOMKilled
Symptoms
Container crashes abruptly. kubectl describe pod shows Reason: OOMKilled and Exit Code: 137.
Possible Causes
- Container memory consumption exceeded its configured
spec.containers[].resources.limits.memory. - Application contains a memory leak.
Investigation and Diagnosis
- Confirm OOM termination event:
kubectl describe pod app-mem-6712 -n production | grep -E "OOMKilled|Exit Code"
- Monitor memory trend prior to crash using metrics:
kubectl top pod app-mem-6712 -n production
Resolution
- Increase memory resource limits in manifest:
resources:
limits:
memory: "512Mi" # Upgraded from 256Mi
- Profile application code to patch memory leaks.
Verification
Verify Pod running stable without crashing under load.
Scenario 5: Service Routing / Endpoints Not Reachable
Symptoms
Requests routed to internal Service IP or DNS name fail or connection times out.
Possible Causes
- Service selector labels do not match target Pod labels (causing empty Endpoints list).
- Application container not listening on target
containerPort. - NetworkPolicy blocking inter-pod communication.
Investigation and Diagnosis
- Verify if Service has active Pod IP endpoints attached:
kubectl get endpoints web-service -n production
If ENDPOINTS displays <none>, label selector mismatch exists.
2. Verify Pod labels against Service selector:
kubectl get svc web-service -n production -o yaml | grep -A 4 selector
kubectl get pods -n production --show-labels
- Test connectivity from inside pod network using a netshoot/busybox ephemeral pod:
kubectl run tmp-net-test --rm -it --image=nicolaka/netshoot -n production -- curl -v http://web-service:80
Resolution
- Align
spec.selectorin Service manifest with target Podmetadata.labels. - Adjust NetworkPolicy rules to permit ingress traffic on specified port.
Verification
kubectl get endpoints displays valid pod IP list and test curl returns HTTP 200.
Scenario 6: Forbidden / RBAC Authorization Errors
Symptoms
Client tool, service account, or application returns API error: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:..." cannot list resource....
Possible Causes
- Missing
Role/ClusterRolepermissions. - ServiceAccount missing appropriate
RoleBinding. - Command issued against wrong namespace context.
Investigation and Diagnosis
- Test RBAC permissions using
can-i:
kubectl auth can-i list pods --as=system:serviceaccount:production:my-sa -n production
- Verify active namespace setting:
kubectl config view --minify | grep namespace
Resolution
Create or update RoleBinding binding the required permissions to the ServiceAccount:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-read-binding
namespace: production
subjects:
- kind: ServiceAccount
name: my-sa
namespace: production
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Verification
Re-run kubectl auth can-i to confirm command returns yes.
Scenario 7: Node Status NotReady
Symptoms
Worker node status transitions to NotReady in kubectl get nodes. Pods on the node are marked for eviction after timeout.
NAME STATUS ROLES AGE VERSION
node-worker-01 NotReady <none> 45d v1.30.1
Possible Causes
kubeletdaemon service stopped or crashed on host.- Node experiencing severe disk pressure (
DiskPressure), memory pressure, or out of PIDs. - Container runtime service (
containerd) dead or unresponsive. - Network partition between worker node and control plane.
Investigation and Diagnosis
- Inspect Node conditions:
kubectl describe node node-worker-01 | grep -A 10 Conditions
- Log into target node via SSH and verify daemons:
systemctl status kubelet containerd
journalctl -u kubelet -n 50 --no-pager
- Check host disk space:
df -h /var/lib/containerd /var/lib/kubelet
Resolution
- Restart failed node services:
sudo systemctl restart containerd kubelet. - Prune unused container images if disk pressure exists:
sudo crictl rmi --prune.
Verification
Confirm node transitions back to Ready status.
Scenario 8: API Server Slow or kubectl Timeouts (etcd Bottleneck)
Symptoms
kubectl operations fail with client timeout errors. API server responses are severely delayed.
Possible Causes
etcddisk write latency degradation or high memory utilization.etcdcluster lost quorum (in multi-master setup).- Excessive watch requests or unindexed API queries flooding
kube-apiserver.
Investigation and Diagnosis
- Check overall control plane API health:
kubectl get --raw /healthz
- Check
etcdcluster health metrics (on control plane nodes runningetcdctl):
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
--key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
endpoint health
Resolution
- Store
etcddirectory/var/lib/etcdon fast dedicated NVMe SSD disks to minimize write latency. - Defragment etcd database and restore quorum if node lost.
Verification
API response times return to normal sub-second latencies.
Scenario 9: Helm Release Failed or Stuck
Symptoms
Helm deployments hang in pending-upgrade or fail mid-deployment, leaving half-applied resources.
Possible Causes
- Pre-install or post-upgrade hooks failed to execute.
- Resource validation failure during manifest application.
Investigation and Diagnosis
- Check release status:
helm status <RELEASE_NAME> -n <NS> - Inspect underlying failed pod objects created by Helm using standard
kubectlcommands:
kubectl get pods -n <NS> --selector=app.kubernetes.io/instance=<RELEASE_NAME>
Resolution
Roll back to last known healthy release version:
helm rollback <RELEASE_NAME> <REVISION_NUMBER> -n <NS>
11. Security and Best Practices
Least Privilege RBAC Management
Avoid using cluster-admin for daily administration. Create granular roles scoped strictly to relevant namespaces.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: production
name: developer-role
rules:
- apiGroups: [""]
resources: ["pods", "services", "configmaps"]
verbs: ["get", "list", "watch", "create", "update"]
Workload Isolation and Security Contexts
Enforce pod security standards to block running applications as root:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 10001
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
Network Policy Hardening
By default, all Pods in a Kubernetes cluster can talk to each other. Implement default-deny NetworkPolicies to isolate components:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
Secrets Security
- Do not store raw secrets in Git repositories. Use tools like SOPS or Sealed Secrets.
- Enable KMS (Key Management Service) encryption at rest in
kube-apiserverso secrets saved inetcdare strongly encrypted.
12. Performance and Resource Optimisation
Sizing Requests and Limits
Setting CPU and Memory requests and limits correctly is crucial for performance stability:
resources:
requests: # Used by scheduler to pick node
cpu: "250m"
memory: "256Mi"
limits: # Upper threshold; enforced by kernel cgroups
cpu: "500m"
memory: "512Mi"
- CPU Requests: Used by
kube-schedulerto place pods. CPU limits use CFS quotas; exceeding CPU limits causes throttling (slowness), but does not terminate the pod. - Memory Limits: Hard ceiling. Exceeding memory limits triggers kernel Out-Of-Memory (OOM) killer immediately, terminating the pod (
OOMKilled).
Node Allocatable Architecture
Worker nodes reserve capacity for system daemons and kernel tasks:
$$\text{Node Total Capacity} = \text{Allocatable} + \text{kube-reserved} + \text{system-reserved} + \text{eviction-threshold}$$
+-------------------------------------------------------------------+
| Node Total Capacity |
+-------------------+--------------------+------------------+-------+
| Allocatable | kube-reserved | system-reserved | Evict |
| (Pods run here) | (kubelet, CRI) | (sshd, systemd) | Thresh|
+-------------------+--------------------+------------------+-------+
13. Backup, Recovery and Maintenance
etcd Backup Snapshot Procedure
The entire cluster state lives inside etcd. Create regular snapshot backups before major upgrades:
# Save etcd snapshot
sudo ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
--key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
snapshot save /var/backups/etcd-snapshot-$(date +%Y%m%d).db
# Verify snapshot integrity
sudo ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd-snapshot-$(date +%Y%m%d).db -w table
Restoring etcd State
To restore cluster state from a snapshot:
sudo ETCDCTL_API=3 etcdctl snapshot restore /var/backups/etcd-snapshot-20261103.db \
--data-dir=/var/lib/etcd-restored
Update /etc/kubernetes/manifests/etcd.yaml host path to point to /var/lib/etcd-restored.
14. Automation and Integration
Package Management with Helm
Helm packages related Kubernetes manifests into reusable, versioned charts.
# Add third-party helm repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
# Install chart release into dedicated namespace
helm install ingress-controller ingress-nginx/ingress-nginx --namespace ingress-system --create-namespace
Continuous Deployment via GitOps (ArgoCD Concept)
Rather than manually running kubectl apply, SRE teams store declarative Kubernetes manifests in a Git repository. GitOps controllers (like ArgoCD or Flux) watch the Git repo and automatically reconcile live cluster state with Git commits.
+------------------+ Git Commit +-----------------------+
| Git Repository | ------------------------> | ArgoCD Controller |
| (YAML Manifests) | | (Runs inside Cluster) |
+------------------+ +-----------------------+
|
v Auto-Sync
+-----------------------+
| Kubernetes API |
+-----------------------+
15. Practical Labs and Exercises
Lab 1: Deploying a Scalable Web Application (Beginner)
Objective
Deploy a 3-replica NGINX web application and expose it internally via a ClusterIP service.
Instructions
- Save the following code as
lab1-web.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: lab1-nginx
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: lab1-nginx
template:
metadata:
labels:
app: lab1-nginx
spec:
containers:
- name: nginx
image: nginx:alpine
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: lab1-service
namespace: default
spec:
type: ClusterIP
selector:
app: lab1-nginx
ports:
- port: 80
targetPort: 80
- Apply the manifest:
kubectl apply -f lab1-web.yaml
Validation
Check pod statuses and test internal service resolution:
kubectl get pods -l app=lab1-nginx -o wide
kubectl run test-pod --rm -it --image=busybox -- wget -qO- http://lab1-service
Lab 2: Troubleshooting Broken Image & Missing Config (Intermediate)
Objective
Diagnose and fix a deployment stuck in ImagePullBackOff and CrashLoopBackOff.
Instructions
- Apply broken manifest
lab2-broken.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: broken-app
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: broken-app
template:
metadata:
labels:
app: broken-app
spec:
containers:
- name: app
image: nginx:nonexistent-tag-999
- Run
kubectl get podsand observe status (ImagePullBackOff). - Run
kubectl describe pod -l app=broken-appto read event failure reasons. - Fix the manifest by updating tag to
nginx:latestand re-apply.
Validation
Confirm the Pod reaches state READY 1/1.
Lab 3: Deep Connectivity Debugging with Ephemeral Debug Containers (Advanced)
Objective
Debug application environment and test network routing inside a distroless app container that has no shell using kubectl debug.
Instructions
- Attach an ephemeral container equipped with debugging tools to a running pod:
kubectl debug pod/lab1-nginx-7d84b986b-x291 -it --image=nicolaka/netshoot --target=nginx
- Inside the debug prompt, execute network probes:
netstat -tulpn
dig lab1-service.default.svc.cluster.local
- Exit shell (
exit).
16. Quick Reference and Cheat Sheet
Important Configuration Files
| File / Location | Purpose |
|---|---|
~/.kube/config |
Main local client configuration for cluster access. |
/etc/kubernetes/manifests/ |
Host static Pod manifests (Control plane daemons). |
/etc/kubernetes/admin.conf |
Cluster root administrative credentials. |
/var/lib/kubelet/ |
Internal runtime directory for Kubelet state data. |
Important Log Locations
| Log Target | Command / Path | Purpose |
|---|---|---|
kubelet service |
journalctl -u kubelet -f |
Node agent operational logs. |
containerd runtime |
journalctl -u containerd -f |
Container lifecycle & runtime errors. |
| Pod container logs | kubectl logs <POD> -c <CONTAINER> -n <NS> |
Application stdout/stderr output. |
| Crashed pod logs | kubectl logs <POD> --previous -n <NS> |
Last log stream prior to container exit. |
Common Administrative Commands
| Task | Command |
|---|---|
| List all Pods across namespaces | kubectl get pods -A -o wide |
| Switch cluster context | kubectl config use-context <CONTEXT_NAME> |
| Change default namespace | kubectl config set-context --current --namespace=<NS> |
| Stream logs matching label | stern -l app=<LABEL> -n <NS> |
| Decode Secret value | kubectl view-secret secret/<SECRET_NAME> -n <NS> |
| Drain node for maintenance | kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data |
| Execute temp debug container | kubectl run tmp-shell --rm -it --image=nicolaka/netshoot -- bash |
Troubleshooting Quick Reference
| Symptom | Initial Diagnostic Command | Primary Suspect |
|---|---|---|
Pending |
kubectl describe pod <POD> -n <NS> |
Insufficient CPU/Memory, Taints/Tolerations. |
ImagePullBackOff |
kubectl describe pod <POD> -n <NS> |
Misspelled image tag, missing secret. |
CrashLoopBackOff |
kubectl logs <POD> --previous -n <NS> |
App crash, missing env var/config. |
OOMKilled |
kubectl describe pod <POD> -n <NS> |
Memory limit set too low. |
| Service Unreachable | kubectl get endpoints <SVC> -n <NS> |
Label selector mismatch. |
Forbidden (403) |
kubectl auth can-i <VERB> <RESOURCE> |
Missing RBAC RoleBinding. |
**Node NotReady** |
journalctl -u kubelet on worker node |
kubelet service or container runtime down. |
Ports and Protocols
| Port | Protocol | Component / Purpose |
|---|---|---|
| 6443 | TCP | kube-apiserver HTTPS endpoint. |
| 2379 - 2380 | TCP | etcd client/peer communication. |
| 10250 | TCP | kubelet API (Control Plane to Worker). |
| 30000 - 32767 | TCP/UDP | NodePort Service default exposure range. |
17. Further Reading and References
- Kubernetes Official Documentation: https://kubernetes.io/docs
- Kubernetes Concepts Guide: https://kubernetes.io/docs/concepts
- kubectl CLI Command Reference: https://kubernetes.io/docs/reference/kubectl/
- Krew Plugin Index Repository: https://krew.sigs.k8s.io/plugins/
- SadServers Hands-On Debugging Playgrounds: https://sadservers.com