Domain 4Kubernetes EngineeringCORE

(Gion) Resolving Kubernetes Secret SubPath Mismatches and Host Network Port Deadlocks

A comprehensive SRE deep dive into investigating and fixing a failing Kubernetes API service caused by secret subPath key mismatches and host network port scheduling deadlocks.

Domain: Domain 4: Kubernetes Cluster Engineering•Updated: 2026-11-10
#Kubernetes#Troubleshooting#SRE#PodLifecycle#Secrets#Networking

1. Introduction

In modern microservices architectures deployed on Kubernetes, application failures rarely stem from a single broken line of code. More often, outages are caused by subtle misconfigurations in workload specifications—such as volume mount definitions, environment projections, or network isolation settings.

This post-mortem details the investigation and remediation of an outage involving the Gion Ticket Office API, a lightweight HTTP service running inside a Kubernetes cluster. Following an infrastructure change during a previous engineering shift, the application entered a continuous crash loop and failed to bind to its designated port.

By reading this guide, you will learn:

  • How Kubernetes projects Secret keys into container filesystems using subPath mounts.
  • Why subPath volume misconfigurations cause immediate application startup failures.
  • How Linux network namespace sharing (hostNetwork: true) interacts with the Kubernetes scheduler and rollout strategies.
  • How to systematically diagnose CrashLoopBackOff and Pending pod states using native kubectl tools.
  • Practical SRE strategies to prevent container port collisions and configuration drift in single-node and production environments.

2. Problem Description and Environment

Scenario Overview

During a staging handover, the SRE team received an alert indicating that the Gion Ticket Office API was un-reachable. The API was expected to serve HTTP traffic on host port 80 and respond to health check queries.

Problem Statement & Requirements

  • Namespace: gion
  • Target Endpoint: GET [http://127.0.0.1/health](http://127.0.0.1/health)
  • Expected Response: {"status":"ok","office":"gion-tickets"}
  • Expected Pod Status: gion-api pod in namespace gion must be 1/1 Ready.
  • Constraint: Do not move the API off port 80.

Observed Behavior

Executing curl -sv [http://127.0.0.1/health](http://127.0.0.1/health) yielded an immediate connection error:

* Trying 127.0.0.1:80...
* connect to 127.0.0.1 port 80 from 127.0.0.1 port 59446 failed: Connection refused
* Failed to connect to 127.0.0.1 port 80 after 0 ms: Could not connect to server

Checking workload status revealed that existing pods were stuck in CrashLoopBackOff or Pending states.

Environment Architecture

  • Cluster: Single-node K3s cluster (v1.29.6+k3s1) on host node1.
  • Workload Manifests: Located in /home/admin/app/ (deployment.yaml, secret.yaml, configmap.yaml, service.yaml, namespace.yaml).
  • Handover Note: Located in /home/admin/HANDOVER.txt.
  • Base Image: busybox:1.36.1 (pre-loaded in container runtime storage).

3. Initial Investigation and Hypotheses

Before modifying cluster manifests or deleting resources, we established a systematic diagnostic plan. We formulated three primary hypotheses based on the handover context and initial symptoms.

Hypotheses Matrix

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
1. Missing / Misconfigured Environment File (/app/.env) Handover note stated recent changes modified secret mounting to project single key onto /app/.env. Pod is in CrashLoopBackOff. None yet. Inspect container logs (kubectl logs) and volume mount subPath specs in deployment.yaml.
2. Host Port Collision / Scheduling Deadlock Pod has hostNetwork: true enabled and second pod revision is stuck in Pending. Initial crash was CrashLoopBackOff (process exit), not Pending. Inspect scheduler events (kubectl describe pod) for port binding failures.
3. Invalid Script Logic in ConfigMap Entrypoint script executes directly from ConfigMap. ConfigMap applied in previous working shift without modified logic. Inspect live ConfigMap definition (kubectl get cm gion-api -o yaml).

4. Troubleshooting Process

The investigation followed a step-by-step, evidence-driven approach.

Step 1: Context Gathering & Handover Audit

Objective Review handover notes and manifest files on disk to understand recent operational changes.

Commands

cat /home/admin/HANDOVER.txt
cd /home/admin/app && ls -la

Command Explanation

  • cat: Prints file contents to standard output.
  • ls -la: Lists all files in the directory including hidden files (.env) in long format.

Output and Analysis

Gion ticket office API — staging handover
=========================================
GET http://127.0.0.1/health must return {"status":"ok","office":"gion-tickets"}

Namespace:  gion
Manifests:  /home/admin/app
Image:      busybox:1.36.1 (already imported; no Internet)

The process reads APP_TOKEN from /app/.env and will not start without that file. 
We used to mount the whole Secret into /app; last change projects only the dotenv file 
onto /app/.env so extra keys do not show up as sibling files.

Decision The handover note explicitly confirms that the application depends on /app/.env containing APP_TOKEN. The recent deployment change modified how Secret keys are projected onto /app/.env.


Step 2: Baseline Cluster & Pod Diagnostics

Objective Check the health of the node control plane and list the current status of pods in the gion namespace.

Commands

kubectl get nodes
kubectl get pods -n gion

Command Explanation

  • kubectl get nodes: Queries the API server for node readiness.
  • kubectl get pods -n gion: Displays pod lifecycle states within the gion namespace.

Output and Analysis

NAME    STATUS   ROLES                  AGE   VERSION
node1   Ready    control-plane,master   72d   v1.29.6+k3s1

NAME                        READY   STATUS             RESTARTS   AGE
gion-api-8589f4f45b-2lpfk   0/1     CrashLoopBackOff   9 (20s ago)72d
gion-api-864d65c6c-5gxft   0/1     Pending            0          72d

Decision node1 is healthy. Pod gion-api-8589f4f45b-2lpfk is continuously crashing (CrashLoopBackOff), while a second revision pod (864d65c6c-5gxft) is stuck in Pending. We must inspect the container logs of the crashing pod and its volume specification.


Step 3: Container Log & Manifest Inspection

Objective Retrieve container logs and compare the live Secret key definition against the Deployment’s volume mount spec.

Commands

kubectl logs gion-api-8589f4f45b-2lpfk -n gion
cat /home/admin/app/deployment.yaml
cat /home/admin/app/secret.yaml

Command Explanation

  • kubectl logs <pod>: Fetches stdout/stderr streams from the active container sandbox.
  • cat deployment.yaml / secret.yaml: Displays local manifest configurations.

Output and Analysis

Container Log Output:

missing /app/.env

Excerpt from secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: gion-dotenv
  namespace: gion
type: Opaque
stringData:
  .env: |
    APP_TOKEN=gion-ok-9c2e
    APP_NAME=gion-tickets
  extra.conf: |
    note=leftover-do-not-mount

Excerpt from deployment.yaml:

          volumeMounts:
            - name: entrypoint
              mountPath: /entrypoint.sh
              subPath: entrypoint.sh
            - name: dotenv
              mountPath: /app/.env
              subPath: env   # <--- MISMATCH DETECTED
      volumes:
        - name: dotenv
          secret:
            secretName: gion-dotenv

Key Discovery:

  1. The Secret gion-dotenv defines data keys named .env (with leading dot) and extra.conf.
  2. The Deployment volume mount specifies subPath: env (missing leading dot).
  3. Kubernetes searches the Secret volume for key env. Because env does not exist, Kubernetes fails to project the file into /app/.env.
  4. The entrypoint script (/entrypoint.sh) checks if [ ! -f /app/.env ]; then echo "missing /app/.env"; exit 1; fi and immediately exits.

Step 4: Applying the subPath Fix & Observing Scheduler Deadlock

Objective Correct the subPath mismatch in deployment.yaml and apply the manifest.

Commands

sed -i 's/subPath: env/subPath: .env/' /home/admin/app/deployment.yaml
kubectl apply -f /home/admin/app/deployment.yaml
kubectl get pods -n gion

Command Explanation

  • sed -i 's/.../.../': Performs an in-place string replacement.
  • kubectl apply -f: Declaratively updates the deployment in the Kubernetes API server.

Output and Analysis

deployment.apps/gion-api configured

NAME                        READY   STATUS             RESTARTS      AGE
gion-api-864d65c6c-5gxft   0/1     CrashLoopBackOff   1 (15s ago)   72d
gion-api-87bdd5557-kmmdj   0/1     Pending            0             17s

After applying the configuration, the new pod revision 87bdd5557-kmmdj remained stuck in Pending indefinitely.


Step 5: Scheduler Deadlock Investigation

Objective Determine why the new pod gion-api-87bdd5557-kmmdj cannot be scheduled.

Commands

kubectl describe pod gion-api-87bdd5557-kmmdj -n gion

Output and Analysis

Events:
  Type     Reason            Age   From              Message
  ----     ------            ----  ----              -------
  Warning  FailedScheduling  94s   default-scheduler 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Key Discovery:

  • deployment.yaml specifies hostNetwork: true, binding the container directly to host port 80 on node1.
  • Kubernetes default deployment strategy is RollingUpdate. During a rolling update, Kubernetes attempts to start the new pod (87bdd5557-kmmdj) before terminating the old pod (864d65c6c-5gxft).
  • Because node1 host port 80 is already occupied by the old crashing pod, the scheduler rejects the new pod due to port collision.

Step 6: Unblocking the Rollout & Verification

Objective Delete the stale crashing pod to free host port 80, allowing the scheduled pod to start.

Commands

kubectl delete pod gion-api-864d65c6c-5gxft -n gion
kubectl get pods -n gion
curl -s http://127.0.0.1/health
/home/admin/agent/check.sh

Output and Analysis

pod "gion-api-864d65c6c-5gxft" deleted

NAME                        READY   STATUS    RESTARTS   AGE
gion-api-87bdd5557-kmmdj   1/1     Running   0          2m3s

{"status":"ok","office":"gion-tickets"}

OK

Result: The new pod immediately bound to host port 80, passed validation, reached 1/1 Ready status, and responded with the required JSON payload.


5. Root Cause Analysis

The outage was caused by two distinct, interacting failure modes:

[Issue 1: Configuration Spec Error]
Secret Key: ".env"  <--->  Deployment subPath: "env" (Missing leading dot)
                                  │
                                  ▼
                     Container mounts empty file
                                  │
                                  ▼
                   /entrypoint.sh exits with code 1
                                  │
                                  ▼
                     Pod enters CrashLoopBackOff

                                  +

[Issue 2: Scheduling & Network Architecture Conflict]
Deployment: hostNetwork: true  +  Strategy: RollingUpdate (Default)
                                  │
                                  ▼
               New pod revision created before old pod deleted
                                  │
                                  ▼
                  Host Port 80 conflict on single node
                                  │
                                  ▼
                     New pod stuck in Pending

Detailed Breakdown

  1. Volume SubPath Key Mismatch: When mounting a Secret or ConfigMap volume using subPath, Kubernetes looks for a key inside the volume dictionary matching the subPath string. The Secret defined key .env, but deployment.yaml requested subPath: env. Because no key named env existed, /app/.env was not created. The application script failed its existence check (if [ ! -f /app/.env ]) and exited with code 1.
  2. Host Network Port Deadlock: Because hostNetwork: true was set, containers shared the node’s network namespace (127.0.0.1). Kubernetes default RollingUpdate deployment strategy attempts to create surge replicas before deleting old replicas. On a single-node cluster, two pods configured with hostNetwork: true cannot bind to port 80 concurrently, deadlocking the scheduler.

6. Solution and Remediation

Applied Changes

1. Corrected subPath Key Reference

File: /home/admin/app/deployment.yaml

--- app/deployment.yaml.orig
+++ app/deployment.yaml
@@ -23,3 +23,3 @@
             - name: dotenv
               mountPath: /app/.env
-              subPath: env
+              subPath: .env

2. Manual Pod Clearance

Executed kubectl delete pod gion-api-864d65c6c-5gxft -n gion to clear the host port conflict.


Production Remediation (Best Practice)

To prevent future scheduler deadlocks during rolling updates of host-network workloads, modify the deployment strategy in deployment.yaml from RollingUpdate to Recreate:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gion-api
  namespace: gion
spec:
  replicas: 1
  strategy:
    type: Recreate   # <--- Prevents port collision during updates
  selector:
    matchLabels:
      app: gion-api
  template:
    metadata:
      labels:
        app: gion-api
    spec:
      hostNetwork: true
      containers:
        - name: api
          image: busybox:1.36.1
          # ...

7. Verification and Validation

The fix was verified against three operational criteria:

  1. Pod Readiness:
kubectl get pods -n gion

Result: gion-api-87bdd5557-kmmdj reported 1/1 Ready and STATUS: Running. 2. HTTP API Response:

curl -s http://127.0.0.1/health

Result: Returned HTTP payload {"status":"ok","office":"gion-tickets"} on host port 80. 3. Automated Verification Script:

/home/admin/agent/check.sh

Result: Returned OK.


8. Alternative Troubleshooting Approaches

While direct log and manifest analysis resolved this incident quickly, alternative diagnostic methods provide additional insights in complex environments.

Diagnostic Approach Comparison

Approach Tools Primary Advantage Limitations
Manifest & Secret Inspection (Used) kubectl describe, kubectl get secret -o yaml Direct visibility into configuration data. Requires manual correlation between manifests.
Interactive Container Debugging kubectl debug / Ephemeral Containers Allows shell access to inspect /app directory inside a running/crashed pod. Requires Kubernetes v1.23+ and container runtime support.
Container Runtime Inspection crictl ps, crictl inspect <container_id> Bypasses API server to inspect low-level container mount points directly on host. Requires root SSH access to worker node.
Declarative Diffing kubectl diff -f app/ Shows exact differences between live cluster state and local files. Does not diagnose runtime scheduling deadlocks.

Example: Inspecting Volume Mounts via crictl

If kubectl logs is unavailable, you can inspect the exact mounts generated by containerd on the node:

# Obtain container ID from pod status
CONTAINER_ID=$(sudo crictl ps --name api -q | head -n 1)

# Inspect mount specs
sudo crictl inspect "$CONTAINER_ID" | jq '.info.runtimeSpec.mounts[] | select(.destination=="/app/.env")'

9. Linux & Kubernetes Command Deep Dive

Command 1: kubectl describe

Purpose

Retrieves detailed metadata, conditions, volume definitions, and historical lifecycle events for a Kubernetes resource.

Syntax & Key Flags

kubectl describe <resource_type> <resource_name> -n <namespace>
Flag Description
-n, --namespace Specifies the target namespace.
-A, --all-namespaces Lists resources across all namespaces.
-l, --selector Filters resources by label selector (e.g. app=gion-api).

Practical Examples

# View events for all failing pods in a namespace
kubectl describe pods -n gion | grep -E "Name:|State:|Events:" -A 10

Command 2: kubectl logs

Purpose

Fetches standard output (stdout) and standard error (stderr) logs from containers.

Syntax & Key Flags

kubectl logs <pod_name> -n <namespace> [flags]
Flag Description
-p, --previous Retrieves logs from a previously terminated instance of the container (vital for CrashLoopBackOff).
-c, --container Specifies container name when pod has multiple containers.
-f, --follow Streams live log output continuously.
--tail=N Displays only the last N lines of logs.

Common Pitfall

Running kubectl logs without --previous on a container actively undergoing a restart cycle may yield an empty output if the new container sandbox has just been created but has not yet executed.


Command 3: sed (Stream Editor)

Purpose

Non-interactive text parsing and transformation utility.

Syntax & Key Flags

sed [options] 's/pattern/replacement/flags' filename
Flag Description
-i, --in-place Edits files in-place rather than writing to standard output.
s/ Substitute command operator.
/g Global flag: replaces all matches in a line instead of just the first.

10. Underlying Linux & Kubernetes Concepts

1. Secret Volume Projections & subPath Mechanics

When mounting a Kubernetes Secret or ConfigMap into a container:

  • Without subPath: Kubernetes mounts a tmpfs volume containing symlinks pointing to the data keys. Updating the Secret in the API server automatically updates the file contents inside the running container.
  • With subPath: Kubernetes mounts only the specific individual file specified by subPath.
  • Trade-off 1: Allows mounting a single file into a pre-existing directory (e.g., /app/.env) without overwriting other files in /app.
  • Trade-off 2: Disables automatic updates. Files mounted via subPath do not receive dynamic secret/configmap updates when the underlying resource changes.
Secret Resource
┌─────────────────────────────────┐
│ Data Keys:                      │
│   .env: "APP_TOKEN=..."         │
│   extra.conf: "..."             │
└─────────────────────────────────┘
                │
                │ subPath: ".env"
                ▼
Container Directory (/app/)
├── .env (Mounted individually as single file)
├── server.js (Pre-existing file preserved)
└── package.json (Pre-existing file preserved)

2. Linux Network Namespaces & hostNetwork: true

By default, container runtimes isolate containers inside distinct Network Namespaces (netns). Each pod gets its own virtual network interface (eth0) and internal IP address.

Setting hostNetwork: true disables network namespace isolation for the pod:

  • The container attaches directly to the host node’s network interfaces (lo, eth0).
  • Listening on 0.0.0.0:80 inside the container binds port 80 directly on host system node1.
  • Security & Operational Risk: Bypasses Kubernetes NetworkPolicies and creates port conflict risks on multi-tenant nodes.

3. Kubernetes Scheduler Predicates (PodFitsHostPorts)

During the scheduling phase, the Kubernetes default-scheduler evaluates node readiness using a series of filter functions (predicates).

When a pod defines hostPort or hostNetwork: true, the scheduler executes the PodFitsHostPorts predicate:

$$\text{Fit} = \text{Requested Ports} \cap \text{Allocated Host Ports} == \emptyset$$

If host port 80 is already registered in the node’s allocated port registry by an active pod container, PodFitsHostPorts fails, and the pod is placed in Pending state with event FailedScheduling.


11. Real-World Sysadmin & SRE Applications

1. Observability & Alerting

In production environments, missing env files and port conflicts should be caught automatically by monitoring systems before impacting users:

  • Metric to Monitor: kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} and kube_pod_status_scheduled{condition="false"}.
  • Probes: Define explicit readiness and liveness probes in deployment specs:
readinessProbe:
  httpGet:
    path: /health
    port: 80
  initialDelaySeconds: 2
  periodSeconds: 5

2. CI/CD Manifest Validation (Shift-Left Security)

Prevent configuration typos like subPath: env prior to deployment by integrating linter checks into CI/CD pipelines:

  • Tools: kube-linter, conftest, or datree.
  • Example Rule: Enforce that all subPath entries in deployment manifests match defined keys in referenced ConfigMaps and Secrets.

3. Deployment Strategy Policies

Avoid combining hostNetwork: true with strategy.type: RollingUpdate on single-instance workloads. Always pair host-network workloads with strategy.type: Recreate or use standard Kubernetes NodePort / LoadBalancer Service abstractions.


12. Further Learning and Experiments

Safe Hands-on Lab Experiments

Experiment 1: Simulating subPath Behavior

Create a test pod in a local Minikube/K3s environment to observe how subPath isolates file mounts:

# 1. Create a secret with multiple keys
kubectl create secret generic test-secret --from-literal=key1=foo --from-literal=key2=bar

# 2. Run a pod mounting key1 via subPath into /tmp
kubectl run subpath-test --image=busybox:1.36.1 --restart=Never -- sh -c '
  cat /tmp/mykey; sleep 3600
' --overrides='
{
  "spec": {
    "containers": [{
      "name": "subpath-test",
      "image": "busybox:1.36.1",
      "command": ["sh", "-c", "cat /tmp/mykey && sleep 3600"],
      "volumeMounts": [{
        "name": "sec-vol",
        "mountPath": "/tmp/mykey",
        "subPath": "key1"
      }]
    }],
    "volumes": [{
      "name": "sec-vol",
      "secret": {"secretName": "test-secret"}
    }]
  }
}'

# 3. Check container log output
kubectl logs subpath-test
# Expected output: foo

13. Key Takeaways

  1. Exact Key Matching is Required: When mounting Secret keys via subPath, the string must match the target Secret key name character-for-character, including leading symbols (e.g., .env).
  2. hostNetwork Causes Port Collisions: Pods running with hostNetwork: true share the host IP and port space. Multiple instances cannot run on the same node on the same port.
  3. Deployment Strategies Matter: Pair hostNetwork: true or hostPort specs with strategy.type: Recreate to ensure old pod instances release ports before new instances start.
  4. Use --previous for Crash Logs: When a container crashes on boot, kubectl logs <pod> --previous fetches logs from the terminated execution sandbox.

14. Quick Troubleshooting Reference / Runbook

                  KUBERNETES POD TROUBLESHOOTING RUNBOOK
                  ======================================

[Step 1: Check Pod Status]
$ kubectl get pods -n <namespace> -o wide

├── If STATUS == CrashLoopBackOff:
│   ├── Check application exit logs:
│   │   $ kubectl logs <pod-name> -n <namespace> --previous
│   └── Check volume mounts & env variables:
│       $ kubectl describe pod <pod-name> -n <namespace>
│
└── If STATUS == Pending:
    ├── Inspect scheduler failure events:
    │   $ kubectl describe pod <pod-name> -n <namespace>
    └── Look for "FailedScheduling" messages:
        ├── "insufficient memory/cpu" -> Adjust resource requests.
        └── "didn't have free ports"  -> Host port collision:
            └── Check if hostNetwork: true is set and delete old pod,
                or change strategy to Recreate.

[Step 2: Validate Manifest Configurations]
$ kubectl get secret <secret-name> -n <namespace> -o yaml
$ kubectl get configmap <cm-name> -n <namespace> -o yaml

Ensure 'subPath' entries under spec.containers.volumeMounts match exact key names.

[Step 3: Apply & Verify]
$ kubectl apply -f <manifest.yaml>
$ kubectl rollout status deployment/<deployment-name> -n <namespace>
$ curl -s http://<endpoint>:<port>/health