1. Introduction
In containerized environments managed by Kubernetes, one of the most common failure modes reported by SREs and system administrators is a Pod remaining stuck in the Pending state. While CrashLoopBackOff indicates runtime process failures within a container, Pending indicates that the Kubernetes control plane—specifically kube-scheduler—cannot place the Pod onto any cluster node.
This article explores a real-world scenario based on the “Bilbao” challenge from SadServers. We examine how to systematically diagnose a failure where an Nginx Deployment fails to start, preventing an external LoadBalancer Service from serving traffic.
By reading this guide, you will learn:
- How the Kubernetes scheduling cycle evaluates Pod constraints.
- How to inspect cluster resource capacity versus workload requests.
- How to use diagnostic commands like
kubectl describe,kubectl get nodes --show-labels, andkubectl get endpointsto isolate root causes. - Best practices for resource limits, node labeling, and operational recovery in production environments.
2. Problem Description and Environment
Problem Scenario
The scenario, titled “Bilbao: Basic Kubernetes Problems” (Difficulty: Easy), takes place on a Debian 13 (Trixie) virtual machine running a single-node Lightweight Kubernetes (k3s) cluster. The system context operates under a non-root user (admin) without sudo access.
Expected vs. Actual Behaviour
- Expected Behaviour: Accessing the LoadBalancer Service IP (
curl 10.43.216.196) returns the default Nginx Welcome HTML page. - Actual Behaviour: Running
curl 10.43.216.196fails immediately with a network connection error:curl: (7) Failed to connect to 10.43.216.196 port 80 after 0 ms: Could not connect to server. - Current Resources: A manifest file (
manifest.yml) in the home directory defines a Deployment (nginx-deployment) and a LoadBalancer Service (nginx-service). The backing Pod (nginx-deployment-7c4b9dc968-m8d29) has been stuck in thePendingstate for over 65 days.
# Source: manifest.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: docker.io/library/nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
resources:
limits:
memory: 2000Mi
cpu: 100m
requests:
cpu: 100m
memory: 2000Mi
nodeSelector:
disk: ssd
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
clusterIP: 10.43.216.196
type: LoadBalancer
(Manifest source:)
3. Initial Investigation and Hypotheses
Before making state changes, a systematic investigation must establish why the Pod is stuck in Pending.
In Kubernetes, a Pending status generally stems from one of four primary causes:
- Unsatisfiable Node Selector / Affinity: The Pod manifest specifies node requirements that no available node matches.
- Resource Exhaustion: Requested CPU or RAM exceeds total allocatable resources across all candidate nodes.
- Un-tolerated Taints: Cluster nodes carry taints (e.g., control-plane taints) without matching tolerations in the Pod template.
- Unbound PersistentVolumeClaims: The Pod requests storage that cannot be provisioned or bound.
Initial Diagnostic Matrix
| Hypothesis | Supporting Evidence | Evidence Against | Diagnostic Step |
|---|---|---|---|
| Node Selector Mismatch | Manifest specifies nodeSelector: disk: ssd. |
Node labels are unknown initially. | Run kubectl get nodes --show-labels. |
| Memory Resource Overcommit | Manifest requests 2000Mi RAM. |
Total node capacity is unknown. | Run kubectl describe node node1. |
| Container Image Pull Failure | Standard image specified. | Image pulling only starts after scheduling succeeds; status would show ErrImagePull. |
Run kubectl describe pod <pod-name>. |
| Control-Plane Taint Blocker | Single-node k3s cluster (node1). |
K3s typically removes control-plane taints on single-node installs. | Inspect Taints in kubectl describe node. |
4. Troubleshooting Process
Step 1: Cluster and Workload Audit
Objective: Query the Kubernetes API to evaluate existing workload health and Service routing status.
Command:
kubectl get all
Command Explanation:
kubectl get: Requests a tabular list of Kubernetes API resources.all: Target alias that expands to standard workload types (Pods, Services, Deployments, ReplicaSets).
Output and Analysis:
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-7c4b9dc968-m8d29 0/1 Pending 0 65d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 168d
service/nginx-service LoadBalancer 10.43.216.196 10.1.13.32 80:30422/TCP 65d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 0/1 1 0 65d
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-7c4b9dc968 1 1 0 65d
(Output source:)
Decision: The Deployment has successfully created a ReplicaSet, which instantiated Pod nginx-deployment-7c4b9dc968-m8d29. However, the Pod remains 0/1 READY in Pending state. Next, we inspect the events logged by kube-scheduler.
Step 2: Pod Event Inspection
Objective: Extract the exact scheduling failure events logged by the control plane.
Command:
kubectl describe pod nginx-deployment-7c4b9dc968-m8d29
Command Explanation:
describe: Fetches detailed runtime metadata, internal conditions, and recent audit events for the targeted resource.
Output and Analysis:
Name: nginx-deployment-7c4b9dc968-m8d29
Namespace: default
Node: <none>
Status: Pending
...
Node-Selectors: disk=ssd
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 65d default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
(Output source:)
Decision: The event log explicitly reveals 1 node(s) didn't match Pod's node affinity/selector. We must verify what labels exist on cluster node node1.
Step 3: Node Label Audit and Remediation
Objective: Inspect labels on node1 and satisfy the disk=ssd requirement.
Command:
kubectl get nodes --show-labels
Output and Analysis:
NAME STATUS ROLES AGE VERSION LABELS
node1 Ready control-plane,master 168d v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s
(Output source:)
node1 carries standard control-plane labels but lacks disk=ssd.
Action Executed:
kubectl label node node1 disk=ssd
Output: node/node1 labeled.
Step 4: Re-evaluating Pod Status & Discovering Secondary Blocker
Objective: Check if labeling node1 allowed the Pod to schedule.
Commands:
kubectl get pods -w
kubectl get endpoints nginx-service
Output and Analysis:
NAME READY STATUS RESTARTS AGE
nginx-deployment-7c4b9dc968-m8d29 0/1 Pending 0 65d
NAME ENDPOINTS AGE
nginx-service <none> 65d
(Output source:)
Even after labeling node1, the Pod remained in Pending. To force kube-scheduler to immediately evaluate a new scheduling cycle rather than staying in exponential backoff, we deleted the Pod:
kubectl delete pod -l app=nginx
Output: pod "nginx-deployment-7c4b9dc968-m8d29" deleted.
The parent ReplicaSet instantly created a new Pod (nginx-deployment-7c4b9dc968-qfjrz). Inspecting its events revealed a second scheduling blocker:
kubectl describe pod -l app=nginx
Output:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 27s default-scheduler 0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
(Output source:)
Step 5: Node Capacity Audit
Objective: Compare node allocatable memory against the Pod’s request of 2000Mi.
Command:
kubectl describe node node1
Output Excerpt:
Capacity:
cpu: 2
ephemeral-storage: 8025124Ki
memory: 962292Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 7806840622
memory: 962292Ki
pods: 110
Allocated resources:
Resource Requests Limits
-------- -------- ------
cpu 200m (10%) 0 (0%)
memory 140Mi (14%) 170Mi (18%)
(Output source:)
Decision: Node node1 has a total allocatable memory capacity of 962292Ki ($\approx 940\text{ MiB}$). The manifest requested 2000Mi ($\approx 1.95\text{ GiB}$). Because $2000\text{ MiB} > 940\text{ MiB}$, kube-scheduler correctly refused to schedule the Pod due to Insufficient memory.
5. Root Cause Analysis
The failure was caused by two distinct configuration blockers acting simultaneously:
- Primary Blocker (Missing Metadata): The deployment manifest specified
nodeSelector: disk: ssd. Nodenode1did not possess this label, failing the scheduler’s Filtering (Predicates) phase. - Secondary Blocker (Resource Overcommit): The deployment manifest requested
2000Miof RAM. The single node in the cluster (node1) only possesses962292Ki($\approx 940\text{ MiB}$) of allocatable memory.
+----------------------------------+
| Deployment Manifest Submitted |
+----------------------------------+
│
▼
+--------------------------+
| kube-scheduler Trigger |
+--------------------------+
│
▼
+----------------------------------+
| Phase 1: Filter Node Selectors |
+----------------------------------+
│
Label 'disk=ssd' missing on node1
│
▼
FAILED: Unmatched Node Selector
│
[ User Labels node1 disk=ssd ]
│
▼
+----------------------------------+
| Phase 2: Filter Resource Limits |
+----------------------------------+
│
Requested 2000Mi > Allocatable 940Mi
│
▼
FAILED: Insufficient Memory
│
▼
Pod Remains PENDING
Because Kubernetes treats resource requests as hard scheduling boundaries, kube-scheduler will never place a Pod on a node that cannot satisfy its requested CPU or memory allocation. Until the container spec was updated to fit within node capacity, the LoadBalancer Service remained without active endpoints, dropping incoming HTTP connections.
6. Solution and Remediation
Remediation Steps
- Satisfy Node Selector: Label
node1withdisk=ssd:
kubectl label node node1 disk=ssd
- Update Manifest Memory Limits: Edit
manifest.ymlto reduce memory requests and limits from2000Midown to a realistic value for Nginx (e.g.,128Mi).
# Excerpt of updated manifest.yml
resources:
limits:
memory: 128Mi
cpu: 100m
requests:
cpu: 100m
memory: 128Mi
(Manifest modification source:) 3. Apply Updated Configuration:
kubectl apply -f manifest.yml
Output: deployment.apps/nginx-deployment configured.
7. Verification and Validation
1. Verification of HTTP Connectivity
Test connectivity directly against the Service LoadBalancer IP:
curl 10.43.216.196
Output:
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...
</body>
</html>
(Output source:)
2. Automated Script Validation
Run the scenario evaluation script:
agent/check.sh
Output: OK.
8. Alternative Troubleshooting Approaches
Alternative 1: Modifying Node Selectors via kubectl patch
Instead of using a text editor, resource updates can be applied declaratively via CLI patches:
kubectl patch deployment nginx-deployment --type='json' -p='[
{"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value":"128Mi"},
{"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value":"128Mi"}
]'
Alternative 2: Scaling Replicas to Reset Pending State
When avoiding rollout restart during a Pending state (which leaves stagnant ReplicaSets), scale replicas down to 0 and back to 1:
kubectl scale deployment/nginx-deployment --replicas=0
kubectl scale deployment/nginx-deployment --replicas=1
Comparison Matrix of Remediation Approaches
| Approach | Command / Technique | Pros | Cons |
|---|---|---|---|
| Node Labeling + Manifest Update | kubectl label & kubectl apply |
Maintains git/manifest sync, preserves IaC principles. | Requires updating local YAML files. |
| CLI JSON Patching | kubectl patch |
Fast, programmatic execution without interactive editors. | Risks drift between cluster state and local manifest files. |
| Node Selector Removal | Delete nodeSelector block in YAML |
Eliminates node affinity restriction entirely. | May violate compliance if workloads require specific hardware (e.g. SSDs). |
9. Linux & Kubernetes Command Deep Dive
1. kubectl describe
- Purpose: Fetches detailed operational metadata and system events for Kubernetes objects.
- Syntax:
kubectl describe <resource-type> <resource-name> - Key Flags:
-l, --selector: Selects resources by label query (e.g.kubectl describe pod -l app=nginx).-n, --namespace: Specifies the target namespace.
2. kubectl label
- Purpose: Adds, modifies, or removes key-value metadata labels on cluster resources.
- Syntax:
kubectl label <resource-type> <resource-name> <key>=<value> - Useful Flags:
--overwrite: Overwrites existing label keys if already present.-l: Applies labels across multiple resources matching a selector query.
3. kubectl get
- Common Diagnostic Options:
| Command Variant | Description |
|---|---|
kubectl get pods -o wide |
Displays internal Pod IP addresses and assigned Worker Node names. |
kubectl get pods -w / --watch |
Watches Pod state transitions in real-time. |
kubectl get nodes --show-labels |
Lists all cluster nodes alongside their metadata labels. |
kubectl get endpoints <service> |
Displays active backend Pod IPs bound to a Service. |
10. Underlying Linux & Kubernetes Architecture
kube-scheduler Mechanics: Filtering vs. Scoring
kube-scheduler assigns Pods to nodes using a two-phase workflow:
[ Unscheduled Pod ] ──► ( 1. Filtering / Predicates ) ──► ( 2. Scoring / Priorities ) ──► [ Scheduled Node ]
- Filtering (Node Filtering / Predicates): Evaluates nodes against hard constraints.
NodeName/NodeSelector: Does the node possess matching labels?PodFitsResources: Does the node have sufficientAllocatableCPU and Memory capacity to satisfyrequests?NodeTolerations: Does the node carry taints not tolerated by the Pod?- If no nodes pass filtering, the Pod remains
PendingwithFailedSchedulingevents logged.
- Scoring (Priorities): Ranks the candidate nodes that passed the filtering phase to select the optimal location for placement.
Memory Allocation: Requests vs. Limits
- Resource Requests: Hard limits used strictly by
kube-schedulerduring node filtering. - Resource Limits: Hard limits enforced by Linux kernel control groups (
cgroups). If a running container exceeds its memory limit, the Linux kernel’s Out-Of-Memory (OOM) Killer terminates the container process with Exit Code137.
11. Real-World Sysadmin and SRE Applications
1. Production Monitoring & Alerting
In production environments, a Pod remaining in Pending represents a service availability risk. SREs monitor this using Prometheus metrics:
# Alert on Pods stuck in Pending state for more than 5 minutes
kube_pod_status_phase{phase="Pending"} > 0
2. Guardrails and Policy Enforcement
- LimitRanges: Enforce default minimum/maximum CPU and memory requests per namespace so developers cannot accidentally submit unfillable requests (like
2000Mion small nodes). - ResourceQuotas: Cap total aggregate resource consumption per namespace to prevent resource starvation.
- GitOps Alignment: Always ensure local changes made via
kubectl labelorkubectl patchare committed to version control repositories (e.g., via ArgoCD or Flux) to prevent configuration drift.
12. Further Learning and Experiments
Hands-on Experiments to Try in a Test Environment
- Simulate Out-Of-Memory (OOM) Kills:
- Deploy a workload with low limits (
memory: 32Mi) and use an image likepolinux/stressto allocate 64Mi. Observe the container status transition toOOMKilled(Exit Code 137).
- Taints and Tolerations:
- Taint a test node (
kubectl taint nodes node1 key1=val1:NoSchedule). Observe how workloads without matching tolerations remain inPendingstate even if node labels match.
Recommended Manual Pages & Documentation
man 8 kubelet- Kubernetes Official Documentation: Resource Management for Pods and Containers
- Kubernetes Official Documentation: Assigning Pods to Nodes
13. Key Takeaways
- Pending Status Meaning:
Pendingindicates a control-plane scheduling failure, whereasCrashLoopBackOffindicates runtime application failures. - Inspect Events First:
kubectl describe podis the definitive diagnostic starting point for scheduling failures. - Hard Constraints:
kube-schedulertreatsnodeSelectorandresources.requestsas absolute prerequisites. - Endpoint Dependency: LoadBalancers and ClusterIP Services will not route traffic until backend Pods reach
1/1 Readystatus. - Fresh Cycle Evaluation: Deleting a stuck
Pendingpod forces immediate scheduler re-evaluation, avoiding backoff delays.
14. Quick Troubleshooting Reference / Runbook
POD STUCK IN PENDING
│
▼
Run: kubectl describe pod <pod>
│
┌────────────────────────┴────────────────────────┐
▼ ▼
Event: NodeSelector Mismatch Event: Insufficient Memory
│ │
▼ ▼
1. Check labels: 1. Check node RAM:
kubectl get nodes --show-labels kubectl describe node <node>
2. Apply missing label: 2. Reduce request in manifest:
kubectl label node <node> key=val memory: 128Mi
│
▼
Apply changes: kubectl apply -f manifest.yml
Force reschedule: kubectl delete pod -l app=<name>
│
▼
Verify Endpoints & HTTP Access:
kubectl get endpoints <service>
curl <LoadBalancer-IP>