Domain 4Kubernetes EngineeringCORE

(Bilbao) Troubleshooting Kubernetes Pod Pending States: Node Selectors and Memory Limits

A comprehensive guide to diagnosing and fixing Kubernetes pods stuck in a Pending state due to nodeSelector mismatches and unfillable resource requests.

Domain: Domain 4: Kubernetes Cluster Engineering•Updated: 2026-11-11
#Kubernetes#k3s#Troubleshooting#SRE#PodPending#Scheduling

1. Introduction

In containerized environments managed by Kubernetes, one of the most common failure modes reported by SREs and system administrators is a Pod remaining stuck in the Pending state. While CrashLoopBackOff indicates runtime process failures within a container, Pending indicates that the Kubernetes control plane—specifically kube-scheduler—cannot place the Pod onto any cluster node.

This article explores a real-world scenario based on the “Bilbao” challenge from SadServers. We examine how to systematically diagnose a failure where an Nginx Deployment fails to start, preventing an external LoadBalancer Service from serving traffic.

By reading this guide, you will learn:

  • How the Kubernetes scheduling cycle evaluates Pod constraints.
  • How to inspect cluster resource capacity versus workload requests.
  • How to use diagnostic commands like kubectl describe, kubectl get nodes --show-labels, and kubectl get endpoints to isolate root causes.
  • Best practices for resource limits, node labeling, and operational recovery in production environments.

2. Problem Description and Environment

Problem Scenario

The scenario, titled “Bilbao: Basic Kubernetes Problems” (Difficulty: Easy), takes place on a Debian 13 (Trixie) virtual machine running a single-node Lightweight Kubernetes (k3s) cluster. The system context operates under a non-root user (admin) without sudo access.

Expected vs. Actual Behaviour

  • Expected Behaviour: Accessing the LoadBalancer Service IP (curl 10.43.216.196) returns the default Nginx Welcome HTML page.
  • Actual Behaviour: Running curl 10.43.216.196 fails immediately with a network connection error: curl: (7) Failed to connect to 10.43.216.196 port 80 after 0 ms: Could not connect to server.
  • Current Resources: A manifest file (manifest.yml) in the home directory defines a Deployment (nginx-deployment) and a LoadBalancer Service (nginx-service). The backing Pod (nginx-deployment-7c4b9dc968-m8d29) has been stuck in the Pending state for over 65 days.
# Source: manifest.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: docker.io/library/nginx:1.27.3
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        resources:
          limits:
            memory: 2000Mi
            cpu: 100m
          requests:
            cpu: 100m
            memory: 2000Mi
      nodeSelector:
        disk: ssd

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  clusterIP: 10.43.216.196
  type: LoadBalancer

(Manifest source:)


3. Initial Investigation and Hypotheses

Before making state changes, a systematic investigation must establish why the Pod is stuck in Pending.

In Kubernetes, a Pending status generally stems from one of four primary causes:

  1. Unsatisfiable Node Selector / Affinity: The Pod manifest specifies node requirements that no available node matches.
  2. Resource Exhaustion: Requested CPU or RAM exceeds total allocatable resources across all candidate nodes.
  3. Un-tolerated Taints: Cluster nodes carry taints (e.g., control-plane taints) without matching tolerations in the Pod template.
  4. Unbound PersistentVolumeClaims: The Pod requests storage that cannot be provisioned or bound.

Initial Diagnostic Matrix

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
Node Selector Mismatch Manifest specifies nodeSelector: disk: ssd. Node labels are unknown initially. Run kubectl get nodes --show-labels.
Memory Resource Overcommit Manifest requests 2000Mi RAM. Total node capacity is unknown. Run kubectl describe node node1.
Container Image Pull Failure Standard image specified. Image pulling only starts after scheduling succeeds; status would show ErrImagePull. Run kubectl describe pod <pod-name>.
Control-Plane Taint Blocker Single-node k3s cluster (node1). K3s typically removes control-plane taints on single-node installs. Inspect Taints in kubectl describe node.

4. Troubleshooting Process

Step 1: Cluster and Workload Audit

Objective: Query the Kubernetes API to evaluate existing workload health and Service routing status.

Command:

kubectl get all

Command Explanation:

  • kubectl get: Requests a tabular list of Kubernetes API resources.
  • all: Target alias that expands to standard workload types (Pods, Services, Deployments, ReplicaSets).

Output and Analysis:

NAME                                 READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-7c4b9dc968-m8d29   0/1     Pending   0          65d

NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes      ClusterIP      10.43.0.1       <none>        443/TCP        168d
service/nginx-service   LoadBalancer   10.43.216.196   10.1.13.32    80:30422/TCP   65d

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   0/1     1            0           65d

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-7c4b9dc968   1         1         0       65d

(Output source:)

Decision: The Deployment has successfully created a ReplicaSet, which instantiated Pod nginx-deployment-7c4b9dc968-m8d29. However, the Pod remains 0/1 READY in Pending state. Next, we inspect the events logged by kube-scheduler.


Step 2: Pod Event Inspection

Objective: Extract the exact scheduling failure events logged by the control plane.

Command:

kubectl describe pod nginx-deployment-7c4b9dc968-m8d29

Command Explanation:

  • describe: Fetches detailed runtime metadata, internal conditions, and recent audit events for the targeted resource.

Output and Analysis:

Name:               nginx-deployment-7c4b9dc968-m8d29
Namespace:          default
Node:               <none>
Status:             Pending
...
Node-Selectors:     disk=ssd
...
Events:
  Type     Reason            Age    From       Message
  ----     ------            ----   ----       -------
  Warning  FailedScheduling  65d    default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

(Output source:)

Decision: The event log explicitly reveals 1 node(s) didn't match Pod's node affinity/selector. We must verify what labels exist on cluster node node1.


Step 3: Node Label Audit and Remediation

Objective: Inspect labels on node1 and satisfy the disk=ssd requirement.

Command:

kubectl get nodes --show-labels

Output and Analysis:

NAME    STATUS   ROLES                  AGE    VERSION        LABELS
node1   Ready    control-plane,master   168d   v1.29.6+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s

(Output source:)

node1 carries standard control-plane labels but lacks disk=ssd.

Action Executed:

kubectl label node node1 disk=ssd

Output: node/node1 labeled.


Step 4: Re-evaluating Pod Status & Discovering Secondary Blocker

Objective: Check if labeling node1 allowed the Pod to schedule.

Commands:

kubectl get pods -w
kubectl get endpoints nginx-service

Output and Analysis:

NAME                                    READY   STATUS    RESTARTS   AGE
nginx-deployment-7c4b9dc968-m8d29       0/1     Pending   0          65d

NAME            ENDPOINTS   AGE
nginx-service   <none>      65d

(Output source:)

Even after labeling node1, the Pod remained in Pending. To force kube-scheduler to immediately evaluate a new scheduling cycle rather than staying in exponential backoff, we deleted the Pod:

kubectl delete pod -l app=nginx

Output: pod "nginx-deployment-7c4b9dc968-m8d29" deleted.

The parent ReplicaSet instantly created a new Pod (nginx-deployment-7c4b9dc968-qfjrz). Inspecting its events revealed a second scheduling blocker:

kubectl describe pod -l app=nginx

Output:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  27s   default-scheduler  0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

(Output source:)


Step 5: Node Capacity Audit

Objective: Compare node allocatable memory against the Pod’s request of 2000Mi.

Command:

kubectl describe node node1

Output Excerpt:

Capacity:
  cpu:                2
  ephemeral-storage:  8025124Ki
  memory:             962292Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  7806840622
  memory:             962292Ki
  pods:               110
Allocated resources:
  Resource           Requests     Limits
  --------           --------     ------
  cpu                200m (10%)   0 (0%)
  memory             140Mi (14%)  170Mi (18%)

(Output source:)

Decision: Node node1 has a total allocatable memory capacity of 962292Ki ($\approx 940\text{ MiB}$). The manifest requested 2000Mi ($\approx 1.95\text{ GiB}$). Because $2000\text{ MiB} > 940\text{ MiB}$, kube-scheduler correctly refused to schedule the Pod due to Insufficient memory.


5. Root Cause Analysis

The failure was caused by two distinct configuration blockers acting simultaneously:

  1. Primary Blocker (Missing Metadata): The deployment manifest specified nodeSelector: disk: ssd. Node node1 did not possess this label, failing the scheduler’s Filtering (Predicates) phase.
  2. Secondary Blocker (Resource Overcommit): The deployment manifest requested 2000Mi of RAM. The single node in the cluster (node1) only possesses 962292Ki ($\approx 940\text{ MiB}$) of allocatable memory.
               +----------------------------------+
               |  Deployment Manifest Submitted  |
               +----------------------------------+
                                │
                                ▼
                   +--------------------------+
                   |  kube-scheduler Trigger  |
                   +--------------------------+
                                │
                                ▼
               +----------------------------------+
               | Phase 1: Filter Node Selectors   |
               +----------------------------------+
                                │
                  Label 'disk=ssd' missing on node1
                                │
                                ▼
                  FAILED: Unmatched Node Selector
                                │
                 [ User Labels node1 disk=ssd ]
                                │
                                ▼
               +----------------------------------+
               | Phase 2: Filter Resource Limits  |
               +----------------------------------+
                                │
               Requested 2000Mi > Allocatable 940Mi
                                │
                                ▼
                  FAILED: Insufficient Memory
                                │
                                ▼
                      Pod Remains PENDING

Because Kubernetes treats resource requests as hard scheduling boundaries, kube-scheduler will never place a Pod on a node that cannot satisfy its requested CPU or memory allocation. Until the container spec was updated to fit within node capacity, the LoadBalancer Service remained without active endpoints, dropping incoming HTTP connections.


6. Solution and Remediation

Remediation Steps

  1. Satisfy Node Selector: Label node1 with disk=ssd:
kubectl label node node1 disk=ssd
  1. Update Manifest Memory Limits: Edit manifest.yml to reduce memory requests and limits from 2000Mi down to a realistic value for Nginx (e.g., 128Mi).
# Excerpt of updated manifest.yml
        resources:
          limits:
            memory: 128Mi
            cpu: 100m
          requests:
            cpu: 100m
            memory: 128Mi

(Manifest modification source:) 3. Apply Updated Configuration:

kubectl apply -f manifest.yml

Output: deployment.apps/nginx-deployment configured.


7. Verification and Validation

1. Verification of HTTP Connectivity

Test connectivity directly against the Service LoadBalancer IP:

curl 10.43.216.196

Output:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...
</body>
</html>

(Output source:)

2. Automated Script Validation

Run the scenario evaluation script:

agent/check.sh

Output: OK.


8. Alternative Troubleshooting Approaches

Alternative 1: Modifying Node Selectors via kubectl patch

Instead of using a text editor, resource updates can be applied declaratively via CLI patches:

kubectl patch deployment nginx-deployment --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value":"128Mi"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value":"128Mi"}
]'

Alternative 2: Scaling Replicas to Reset Pending State

When avoiding rollout restart during a Pending state (which leaves stagnant ReplicaSets), scale replicas down to 0 and back to 1:

kubectl scale deployment/nginx-deployment --replicas=0
kubectl scale deployment/nginx-deployment --replicas=1

Comparison Matrix of Remediation Approaches

Approach Command / Technique Pros Cons
Node Labeling + Manifest Update kubectl label & kubectl apply Maintains git/manifest sync, preserves IaC principles. Requires updating local YAML files.
CLI JSON Patching kubectl patch Fast, programmatic execution without interactive editors. Risks drift between cluster state and local manifest files.
Node Selector Removal Delete nodeSelector block in YAML Eliminates node affinity restriction entirely. May violate compliance if workloads require specific hardware (e.g. SSDs).

9. Linux & Kubernetes Command Deep Dive

1. kubectl describe

  • Purpose: Fetches detailed operational metadata and system events for Kubernetes objects.
  • Syntax: kubectl describe <resource-type> <resource-name>
  • Key Flags:
  • -l, --selector: Selects resources by label query (e.g. kubectl describe pod -l app=nginx).
  • -n, --namespace: Specifies the target namespace.

2. kubectl label

  • Purpose: Adds, modifies, or removes key-value metadata labels on cluster resources.
  • Syntax: kubectl label <resource-type> <resource-name> <key>=<value>
  • Useful Flags:
  • --overwrite: Overwrites existing label keys if already present.
  • -l: Applies labels across multiple resources matching a selector query.

3. kubectl get

  • Common Diagnostic Options:
Command Variant Description
kubectl get pods -o wide Displays internal Pod IP addresses and assigned Worker Node names.
kubectl get pods -w / --watch Watches Pod state transitions in real-time.
kubectl get nodes --show-labels Lists all cluster nodes alongside their metadata labels.
kubectl get endpoints <service> Displays active backend Pod IPs bound to a Service.

10. Underlying Linux & Kubernetes Architecture

kube-scheduler Mechanics: Filtering vs. Scoring

kube-scheduler assigns Pods to nodes using a two-phase workflow:

[ Unscheduled Pod ] ──► ( 1. Filtering / Predicates ) ──► ( 2. Scoring / Priorities ) ──► [ Scheduled Node ]
  1. Filtering (Node Filtering / Predicates): Evaluates nodes against hard constraints.
  • NodeName / NodeSelector: Does the node possess matching labels?
  • PodFitsResources: Does the node have sufficient Allocatable CPU and Memory capacity to satisfy requests?
  • NodeTolerations: Does the node carry taints not tolerated by the Pod?
  • If no nodes pass filtering, the Pod remains Pending with FailedScheduling events logged.
  1. Scoring (Priorities): Ranks the candidate nodes that passed the filtering phase to select the optimal location for placement.

Memory Allocation: Requests vs. Limits

  • Resource Requests: Hard limits used strictly by kube-scheduler during node filtering.
  • Resource Limits: Hard limits enforced by Linux kernel control groups (cgroups). If a running container exceeds its memory limit, the Linux kernel’s Out-Of-Memory (OOM) Killer terminates the container process with Exit Code 137.

11. Real-World Sysadmin and SRE Applications

1. Production Monitoring & Alerting

In production environments, a Pod remaining in Pending represents a service availability risk. SREs monitor this using Prometheus metrics:

# Alert on Pods stuck in Pending state for more than 5 minutes
kube_pod_status_phase{phase="Pending"} > 0

2. Guardrails and Policy Enforcement

  • LimitRanges: Enforce default minimum/maximum CPU and memory requests per namespace so developers cannot accidentally submit unfillable requests (like 2000Mi on small nodes).
  • ResourceQuotas: Cap total aggregate resource consumption per namespace to prevent resource starvation.
  • GitOps Alignment: Always ensure local changes made via kubectl label or kubectl patch are committed to version control repositories (e.g., via ArgoCD or Flux) to prevent configuration drift.

12. Further Learning and Experiments

Hands-on Experiments to Try in a Test Environment

  1. Simulate Out-Of-Memory (OOM) Kills:
  • Deploy a workload with low limits (memory: 32Mi) and use an image like polinux/stress to allocate 64Mi. Observe the container status transition to OOMKilled (Exit Code 137).
  1. Taints and Tolerations:
  • Taint a test node (kubectl taint nodes node1 key1=val1:NoSchedule). Observe how workloads without matching tolerations remain in Pending state even if node labels match.
  • man 8 kubelet
  • Kubernetes Official Documentation: Resource Management for Pods and Containers
  • Kubernetes Official Documentation: Assigning Pods to Nodes

13. Key Takeaways

  1. Pending Status Meaning: Pending indicates a control-plane scheduling failure, whereas CrashLoopBackOff indicates runtime application failures.
  2. Inspect Events First: kubectl describe pod is the definitive diagnostic starting point for scheduling failures.
  3. Hard Constraints: kube-scheduler treats nodeSelector and resources.requests as absolute prerequisites.
  4. Endpoint Dependency: LoadBalancers and ClusterIP Services will not route traffic until backend Pods reach 1/1 Ready status.
  5. Fresh Cycle Evaluation: Deleting a stuck Pending pod forces immediate scheduler re-evaluation, avoiding backoff delays.

14. Quick Troubleshooting Reference / Runbook

                         POD STUCK IN PENDING
                                  │
                                  ▼
                   Run: kubectl describe pod <pod>
                                  │
         ┌────────────────────────┴────────────────────────┐
         ▼                                                 ▼
Event: NodeSelector Mismatch     Event: Insufficient Memory
         │                                                 │
         ▼                                                 ▼
1. Check labels:                                   1. Check node RAM:
   kubectl get nodes --show-labels   kubectl describe node <node>
2. Apply missing label:                            2. Reduce request in manifest:
   kubectl label node <node> key=val memory: 128Mi
                                  │
                                  ▼
                   Apply changes: kubectl apply -f manifest.yml
                   Force reschedule: kubectl delete pod -l app=<name>
                                  │
                                  ▼
                   Verify Endpoints & HTTP Access:
                   kubectl get endpoints <service>
                   curl <LoadBalancer-IP>