1. Introduction
In modern microservices architectures deployed on Kubernetes, application failures rarely stem from a single broken line of code. More often, outages are caused by subtle misconfigurations in workload specifications—such as volume mount definitions, environment projections, or network isolation settings.
This post-mortem details the investigation and remediation of an outage involving the Gion Ticket Office API, a lightweight HTTP service running inside a Kubernetes cluster. Following an infrastructure change during a previous engineering shift, the application entered a continuous crash loop and failed to bind to its designated port.
By reading this guide, you will learn:
- How Kubernetes projects
Secretkeys into container filesystems usingsubPathmounts. - Why
subPathvolume misconfigurations cause immediate application startup failures. - How Linux network namespace sharing (
hostNetwork: true) interacts with the Kubernetes scheduler and rollout strategies. - How to systematically diagnose
CrashLoopBackOffandPendingpod states using nativekubectltools. - Practical SRE strategies to prevent container port collisions and configuration drift in single-node and production environments.
2. Problem Description and Environment
Scenario Overview
During a staging handover, the SRE team received an alert indicating that the Gion Ticket Office API was un-reachable. The API was expected to serve HTTP traffic on host port 80 and respond to health check queries.
Problem Statement & Requirements
- Namespace:
gion - Target Endpoint:
GET [http://127.0.0.1/health](http://127.0.0.1/health) - Expected Response:
{"status":"ok","office":"gion-tickets"} - Expected Pod Status:
gion-apipod in namespacegionmust be1/1 Ready. - Constraint: Do not move the API off port
80.
Observed Behavior
Executing curl -sv [http://127.0.0.1/health](http://127.0.0.1/health) yielded an immediate connection error:
* Trying 127.0.0.1:80...
* connect to 127.0.0.1 port 80 from 127.0.0.1 port 59446 failed: Connection refused
* Failed to connect to 127.0.0.1 port 80 after 0 ms: Could not connect to server
Checking workload status revealed that existing pods were stuck in CrashLoopBackOff or Pending states.
Environment Architecture
- Cluster: Single-node K3s cluster (
v1.29.6+k3s1) on hostnode1. - Workload Manifests: Located in
/home/admin/app/(deployment.yaml,secret.yaml,configmap.yaml,service.yaml,namespace.yaml). - Handover Note: Located in
/home/admin/HANDOVER.txt. - Base Image:
busybox:1.36.1(pre-loaded in container runtime storage).
3. Initial Investigation and Hypotheses
Before modifying cluster manifests or deleting resources, we established a systematic diagnostic plan. We formulated three primary hypotheses based on the handover context and initial symptoms.
Hypotheses Matrix
| Hypothesis | Supporting Evidence | Evidence Against | Diagnostic Step |
|---|---|---|---|
1. Missing / Misconfigured Environment File (/app/.env) |
Handover note stated recent changes modified secret mounting to project single key onto /app/.env. Pod is in CrashLoopBackOff. |
None yet. | Inspect container logs (kubectl logs) and volume mount subPath specs in deployment.yaml. |
| 2. Host Port Collision / Scheduling Deadlock | Pod has hostNetwork: true enabled and second pod revision is stuck in Pending. |
Initial crash was CrashLoopBackOff (process exit), not Pending. |
Inspect scheduler events (kubectl describe pod) for port binding failures. |
| 3. Invalid Script Logic in ConfigMap | Entrypoint script executes directly from ConfigMap. | ConfigMap applied in previous working shift without modified logic. | Inspect live ConfigMap definition (kubectl get cm gion-api -o yaml). |
4. Troubleshooting Process
The investigation followed a step-by-step, evidence-driven approach.
Step 1: Context Gathering & Handover Audit
Objective Review handover notes and manifest files on disk to understand recent operational changes.
Commands
cat /home/admin/HANDOVER.txt
cd /home/admin/app && ls -la
Command Explanation
cat: Prints file contents to standard output.ls -la: Lists all files in the directory including hidden files (.env) in long format.
Output and Analysis
Gion ticket office API — staging handover
=========================================
GET http://127.0.0.1/health must return {"status":"ok","office":"gion-tickets"}
Namespace: gion
Manifests: /home/admin/app
Image: busybox:1.36.1 (already imported; no Internet)
The process reads APP_TOKEN from /app/.env and will not start without that file.
We used to mount the whole Secret into /app; last change projects only the dotenv file
onto /app/.env so extra keys do not show up as sibling files.
Decision
The handover note explicitly confirms that the application depends on /app/.env containing APP_TOKEN. The recent deployment change modified how Secret keys are projected onto /app/.env.
Step 2: Baseline Cluster & Pod Diagnostics
Objective
Check the health of the node control plane and list the current status of pods in the gion namespace.
Commands
kubectl get nodes
kubectl get pods -n gion
Command Explanation
kubectl get nodes: Queries the API server for node readiness.kubectl get pods -n gion: Displays pod lifecycle states within thegionnamespace.
Output and Analysis
NAME STATUS ROLES AGE VERSION
node1 Ready control-plane,master 72d v1.29.6+k3s1
NAME READY STATUS RESTARTS AGE
gion-api-8589f4f45b-2lpfk 0/1 CrashLoopBackOff 9 (20s ago)72d
gion-api-864d65c6c-5gxft 0/1 Pending 0 72d
Decision
node1 is healthy. Pod gion-api-8589f4f45b-2lpfk is continuously crashing (CrashLoopBackOff), while a second revision pod (864d65c6c-5gxft) is stuck in Pending. We must inspect the container logs of the crashing pod and its volume specification.
Step 3: Container Log & Manifest Inspection
Objective Retrieve container logs and compare the live Secret key definition against the Deployment’s volume mount spec.
Commands
kubectl logs gion-api-8589f4f45b-2lpfk -n gion
cat /home/admin/app/deployment.yaml
cat /home/admin/app/secret.yaml
Command Explanation
kubectl logs <pod>: Fetches stdout/stderr streams from the active container sandbox.cat deployment.yaml / secret.yaml: Displays local manifest configurations.
Output and Analysis
Container Log Output:
missing /app/.env
Excerpt from secret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: gion-dotenv
namespace: gion
type: Opaque
stringData:
.env: |
APP_TOKEN=gion-ok-9c2e
APP_NAME=gion-tickets
extra.conf: |
note=leftover-do-not-mount
Excerpt from deployment.yaml:
volumeMounts:
- name: entrypoint
mountPath: /entrypoint.sh
subPath: entrypoint.sh
- name: dotenv
mountPath: /app/.env
subPath: env # <--- MISMATCH DETECTED
volumes:
- name: dotenv
secret:
secretName: gion-dotenv
Key Discovery:
- The Secret
gion-dotenvdefines data keys named.env(with leading dot) andextra.conf. - The Deployment volume mount specifies
subPath: env(missing leading dot). - Kubernetes searches the Secret volume for key
env. Becauseenvdoes not exist, Kubernetes fails to project the file into/app/.env. - The entrypoint script (
/entrypoint.sh) checksif [ ! -f /app/.env ]; then echo "missing /app/.env"; exit 1; fiand immediately exits.
Step 4: Applying the subPath Fix & Observing Scheduler Deadlock
Objective
Correct the subPath mismatch in deployment.yaml and apply the manifest.
Commands
sed -i 's/subPath: env/subPath: .env/' /home/admin/app/deployment.yaml
kubectl apply -f /home/admin/app/deployment.yaml
kubectl get pods -n gion
Command Explanation
sed -i 's/.../.../': Performs an in-place string replacement.kubectl apply -f: Declaratively updates the deployment in the Kubernetes API server.
Output and Analysis
deployment.apps/gion-api configured
NAME READY STATUS RESTARTS AGE
gion-api-864d65c6c-5gxft 0/1 CrashLoopBackOff 1 (15s ago) 72d
gion-api-87bdd5557-kmmdj 0/1 Pending 0 17s
After applying the configuration, the new pod revision 87bdd5557-kmmdj remained stuck in Pending indefinitely.
Step 5: Scheduler Deadlock Investigation
Objective
Determine why the new pod gion-api-87bdd5557-kmmdj cannot be scheduled.
Commands
kubectl describe pod gion-api-87bdd5557-kmmdj -n gion
Output and Analysis
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 94s default-scheduler 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
Key Discovery:
deployment.yamlspecifieshostNetwork: true, binding the container directly to host port80onnode1.- Kubernetes default deployment strategy is
RollingUpdate. During a rolling update, Kubernetes attempts to start the new pod (87bdd5557-kmmdj) before terminating the old pod (864d65c6c-5gxft). - Because
node1host port80is already occupied by the old crashing pod, the scheduler rejects the new pod due to port collision.
Step 6: Unblocking the Rollout & Verification
Objective
Delete the stale crashing pod to free host port 80, allowing the scheduled pod to start.
Commands
kubectl delete pod gion-api-864d65c6c-5gxft -n gion
kubectl get pods -n gion
curl -s http://127.0.0.1/health
/home/admin/agent/check.sh
Output and Analysis
pod "gion-api-864d65c6c-5gxft" deleted
NAME READY STATUS RESTARTS AGE
gion-api-87bdd5557-kmmdj 1/1 Running 0 2m3s
{"status":"ok","office":"gion-tickets"}
OK
Result:
The new pod immediately bound to host port 80, passed validation, reached 1/1 Ready status, and responded with the required JSON payload.
5. Root Cause Analysis
The outage was caused by two distinct, interacting failure modes:
[Issue 1: Configuration Spec Error]
Secret Key: ".env" <---> Deployment subPath: "env" (Missing leading dot)
│
▼
Container mounts empty file
│
▼
/entrypoint.sh exits with code 1
│
▼
Pod enters CrashLoopBackOff
+
[Issue 2: Scheduling & Network Architecture Conflict]
Deployment: hostNetwork: true + Strategy: RollingUpdate (Default)
│
▼
New pod revision created before old pod deleted
│
▼
Host Port 80 conflict on single node
│
▼
New pod stuck in Pending
Detailed Breakdown
- Volume SubPath Key Mismatch:
When mounting a
SecretorConfigMapvolume usingsubPath, Kubernetes looks for a key inside the volume dictionary matching thesubPathstring. The Secret defined key.env, butdeployment.yamlrequestedsubPath: env. Because no key namedenvexisted,/app/.envwas not created. The application script failed its existence check (if [ ! -f /app/.env ]) and exited with code1. - Host Network Port Deadlock:
Because
hostNetwork: truewas set, containers shared the node’s network namespace (127.0.0.1). Kubernetes defaultRollingUpdatedeployment strategy attempts to create surge replicas before deleting old replicas. On a single-node cluster, two pods configured withhostNetwork: truecannot bind to port80concurrently, deadlocking the scheduler.
6. Solution and Remediation
Applied Changes
1. Corrected subPath Key Reference
File: /home/admin/app/deployment.yaml
--- app/deployment.yaml.orig
+++ app/deployment.yaml
@@ -23,3 +23,3 @@
- name: dotenv
mountPath: /app/.env
- subPath: env
+ subPath: .env
2. Manual Pod Clearance
Executed kubectl delete pod gion-api-864d65c6c-5gxft -n gion to clear the host port conflict.
Production Remediation (Best Practice)
To prevent future scheduler deadlocks during rolling updates of host-network workloads, modify the deployment strategy in deployment.yaml from RollingUpdate to Recreate:
apiVersion: apps/v1
kind: Deployment
metadata:
name: gion-api
namespace: gion
spec:
replicas: 1
strategy:
type: Recreate # <--- Prevents port collision during updates
selector:
matchLabels:
app: gion-api
template:
metadata:
labels:
app: gion-api
spec:
hostNetwork: true
containers:
- name: api
image: busybox:1.36.1
# ...
7. Verification and Validation
The fix was verified against three operational criteria:
- Pod Readiness:
kubectl get pods -n gion
Result: gion-api-87bdd5557-kmmdj reported 1/1 Ready and STATUS: Running.
2. HTTP API Response:
curl -s http://127.0.0.1/health
Result: Returned HTTP payload {"status":"ok","office":"gion-tickets"} on host port 80.
3. Automated Verification Script:
/home/admin/agent/check.sh
Result: Returned OK.
8. Alternative Troubleshooting Approaches
While direct log and manifest analysis resolved this incident quickly, alternative diagnostic methods provide additional insights in complex environments.
Diagnostic Approach Comparison
| Approach | Tools | Primary Advantage | Limitations |
|---|---|---|---|
| Manifest & Secret Inspection (Used) | kubectl describe, kubectl get secret -o yaml |
Direct visibility into configuration data. | Requires manual correlation between manifests. |
| Interactive Container Debugging | kubectl debug / Ephemeral Containers |
Allows shell access to inspect /app directory inside a running/crashed pod. |
Requires Kubernetes v1.23+ and container runtime support. |
| Container Runtime Inspection | crictl ps, crictl inspect <container_id> |
Bypasses API server to inspect low-level container mount points directly on host. | Requires root SSH access to worker node. |
| Declarative Diffing | kubectl diff -f app/ |
Shows exact differences between live cluster state and local files. | Does not diagnose runtime scheduling deadlocks. |
Example: Inspecting Volume Mounts via crictl
If kubectl logs is unavailable, you can inspect the exact mounts generated by containerd on the node:
# Obtain container ID from pod status
CONTAINER_ID=$(sudo crictl ps --name api -q | head -n 1)
# Inspect mount specs
sudo crictl inspect "$CONTAINER_ID" | jq '.info.runtimeSpec.mounts[] | select(.destination=="/app/.env")'
9. Linux & Kubernetes Command Deep Dive
Command 1: kubectl describe
Purpose
Retrieves detailed metadata, conditions, volume definitions, and historical lifecycle events for a Kubernetes resource.
Syntax & Key Flags
kubectl describe <resource_type> <resource_name> -n <namespace>
| Flag | Description |
|---|---|
-n, --namespace |
Specifies the target namespace. |
-A, --all-namespaces |
Lists resources across all namespaces. |
-l, --selector |
Filters resources by label selector (e.g. app=gion-api). |
Practical Examples
# View events for all failing pods in a namespace
kubectl describe pods -n gion | grep -E "Name:|State:|Events:" -A 10
Command 2: kubectl logs
Purpose
Fetches standard output (stdout) and standard error (stderr) logs from containers.
Syntax & Key Flags
kubectl logs <pod_name> -n <namespace> [flags]
| Flag | Description |
|---|---|
-p, --previous |
Retrieves logs from a previously terminated instance of the container (vital for CrashLoopBackOff). |
-c, --container |
Specifies container name when pod has multiple containers. |
-f, --follow |
Streams live log output continuously. |
--tail=N |
Displays only the last N lines of logs. |
Common Pitfall
Running kubectl logs without --previous on a container actively undergoing a restart cycle may yield an empty output if the new container sandbox has just been created but has not yet executed.
Command 3: sed (Stream Editor)
Purpose
Non-interactive text parsing and transformation utility.
Syntax & Key Flags
sed [options] 's/pattern/replacement/flags' filename
| Flag | Description |
|---|---|
-i, --in-place |
Edits files in-place rather than writing to standard output. |
s/ |
Substitute command operator. |
/g |
Global flag: replaces all matches in a line instead of just the first. |
10. Underlying Linux & Kubernetes Concepts
1. Secret Volume Projections & subPath Mechanics
When mounting a Kubernetes Secret or ConfigMap into a container:
- Without
subPath: Kubernetes mounts atmpfsvolume containing symlinks pointing to the data keys. Updating the Secret in the API server automatically updates the file contents inside the running container. - With
subPath: Kubernetes mounts only the specific individual file specified bysubPath. - Trade-off 1: Allows mounting a single file into a pre-existing directory (e.g.,
/app/.env) without overwriting other files in/app. - Trade-off 2: Disables automatic updates. Files mounted via
subPathdo not receive dynamic secret/configmap updates when the underlying resource changes.
Secret Resource
┌─────────────────────────────────┐
│ Data Keys: │
│ .env: "APP_TOKEN=..." │
│ extra.conf: "..." │
└─────────────────────────────────┘
│
│ subPath: ".env"
▼
Container Directory (/app/)
├── .env (Mounted individually as single file)
├── server.js (Pre-existing file preserved)
└── package.json (Pre-existing file preserved)
2. Linux Network Namespaces & hostNetwork: true
By default, container runtimes isolate containers inside distinct Network Namespaces (netns). Each pod gets its own virtual network interface (eth0) and internal IP address.
Setting hostNetwork: true disables network namespace isolation for the pod:
- The container attaches directly to the host node’s network interfaces (
lo,eth0). - Listening on
0.0.0.0:80inside the container binds port80directly on host systemnode1. - Security & Operational Risk: Bypasses Kubernetes NetworkPolicies and creates port conflict risks on multi-tenant nodes.
3. Kubernetes Scheduler Predicates (PodFitsHostPorts)
During the scheduling phase, the Kubernetes default-scheduler evaluates node readiness using a series of filter functions (predicates).
When a pod defines hostPort or hostNetwork: true, the scheduler executes the PodFitsHostPorts predicate:
$$\text{Fit} = \text{Requested Ports} \cap \text{Allocated Host Ports} == \emptyset$$
If host port 80 is already registered in the node’s allocated port registry by an active pod container, PodFitsHostPorts fails, and the pod is placed in Pending state with event FailedScheduling.
11. Real-World Sysadmin & SRE Applications
1. Observability & Alerting
In production environments, missing env files and port conflicts should be caught automatically by monitoring systems before impacting users:
- Metric to Monitor:
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}andkube_pod_status_scheduled{condition="false"}. - Probes: Define explicit readiness and liveness probes in deployment specs:
readinessProbe:
httpGet:
path: /health
port: 80
initialDelaySeconds: 2
periodSeconds: 5
2. CI/CD Manifest Validation (Shift-Left Security)
Prevent configuration typos like subPath: env prior to deployment by integrating linter checks into CI/CD pipelines:
- Tools:
kube-linter,conftest, ordatree. - Example Rule: Enforce that all
subPathentries in deployment manifests match defined keys in referenced ConfigMaps and Secrets.
3. Deployment Strategy Policies
Avoid combining hostNetwork: true with strategy.type: RollingUpdate on single-instance workloads. Always pair host-network workloads with strategy.type: Recreate or use standard Kubernetes NodePort / LoadBalancer Service abstractions.
12. Further Learning and Experiments
Safe Hands-on Lab Experiments
Experiment 1: Simulating subPath Behavior
Create a test pod in a local Minikube/K3s environment to observe how subPath isolates file mounts:
# 1. Create a secret with multiple keys
kubectl create secret generic test-secret --from-literal=key1=foo --from-literal=key2=bar
# 2. Run a pod mounting key1 via subPath into /tmp
kubectl run subpath-test --image=busybox:1.36.1 --restart=Never -- sh -c '
cat /tmp/mykey; sleep 3600
' --overrides='
{
"spec": {
"containers": [{
"name": "subpath-test",
"image": "busybox:1.36.1",
"command": ["sh", "-c", "cat /tmp/mykey && sleep 3600"],
"volumeMounts": [{
"name": "sec-vol",
"mountPath": "/tmp/mykey",
"subPath": "key1"
}]
}],
"volumes": [{
"name": "sec-vol",
"secret": {"secretName": "test-secret"}
}]
}
}'
# 3. Check container log output
kubectl logs subpath-test
# Expected output: foo
13. Key Takeaways
- Exact Key Matching is Required: When mounting Secret keys via
subPath, the string must match the target Secret key name character-for-character, including leading symbols (e.g.,.env). hostNetworkCauses Port Collisions: Pods running withhostNetwork: trueshare the host IP and port space. Multiple instances cannot run on the same node on the same port.- Deployment Strategies Matter: Pair
hostNetwork: trueorhostPortspecs withstrategy.type: Recreateto ensure old pod instances release ports before new instances start. - Use
--previousfor Crash Logs: When a container crashes on boot,kubectl logs <pod> --previousfetches logs from the terminated execution sandbox.
14. Quick Troubleshooting Reference / Runbook
KUBERNETES POD TROUBLESHOOTING RUNBOOK
======================================
[Step 1: Check Pod Status]
$ kubectl get pods -n <namespace> -o wide
├── If STATUS == CrashLoopBackOff:
│ ├── Check application exit logs:
│ │ $ kubectl logs <pod-name> -n <namespace> --previous
│ └── Check volume mounts & env variables:
│ $ kubectl describe pod <pod-name> -n <namespace>
│
└── If STATUS == Pending:
├── Inspect scheduler failure events:
│ $ kubectl describe pod <pod-name> -n <namespace>
└── Look for "FailedScheduling" messages:
├── "insufficient memory/cpu" -> Adjust resource requests.
└── "didn't have free ports" -> Host port collision:
└── Check if hostNetwork: true is set and delete old pod,
or change strategy to Recreate.
[Step 2: Validate Manifest Configurations]
$ kubectl get secret <secret-name> -n <namespace> -o yaml
$ kubectl get configmap <cm-name> -n <namespace> -o yaml
Ensure 'subPath' entries under spec.containers.volumeMounts match exact key names.
[Step 3: Apply & Verify]
$ kubectl apply -f <manifest.yaml>
$ kubectl rollout status deployment/<deployment-name> -n <namespace>
$ curl -s http://<endpoint>:<port>/health