Every Kubernetes engineer has stared at it during a 2:00 AM incident: a pod status transitioning from Running to Error, and finally settling into the dreaded CrashLoopBackOff state.

Important Clarification: CrashLoopBackOff is not an error code itself. It is a state condition indicating that Kubernetes attempted to launch your container, the process terminated, and the kubelet is backing off restarts with an exponential delay (10s, 20s, 40s, up to 5 minutes) to avoid overloading the node.

The Root Cause Taxonomy: Decoding Exit Codes

Before blindly restarting deployments or changing image tags, check the container exit code. The exit code reveals the exact mechanism of death:

Exit Code	Name / Signal	Primary Root Cause
137	`SIGKILL (128 + 9)`	OOMKilled — Linux kernel killed the container for exceeding `limits.memory`, or host memory pressure.
143	`SIGTERM (128 + 15)`	Kubelet initiated termination due to failed liveness probe, preStop hook timeout, or rolling node draining.
1	Application Error	Uncaught exception, missing environment variable, syntax error, or unhandled database connection rejection.
127	Command Not Found	Entrypoint binary or startup command path is missing in the container image filesystem.
139	Segmentation Fault	C/Go binary memory corruption or incompatible native library dependencies (e.g. musl vs glibc on Alpine).

Step 1: Inspect Container Termination State & Exit Code

Standard kubectl get pods only shows that restarts are incrementing. Query the container status subresource directly to extract the exact termination details:

# Extract termination reason and exit code from the last crashed instance
kubectl get pod <pod-name> -n <namespace> \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

If you see the following JSON structure, you are facing an Out-Of-Memory termination:

{"exitCode": 137, "reason": "OOMKilled", "startedAt": "...", "finishedAt": "..."}

Step 2: Capture Logs From the Previous Crashed Instance

When a pod restarts, running kubectl logs <pod-name> only streams logs from the newly spawned instance, which may look healthy before it dies. To see why the previous container crashed, use the --previous (or -p) flag:

# Stream stdout/stderr from the container that just died
kubectl logs <pod-name> -n <namespace> --previous --tail=100

Multi-Container Pods: If your pod runs an Envoy proxy or logging sidecar, always specify the target container: kubectl logs <pod-name> -c app-container --previous.

Step 3: Check Pod Events & Liveness Probe Failures

A frequent cause of Exit Code 143 is an overly aggressive Liveness Probe. If your application takes 45 seconds to establish its database connection pool during startup, but your probe has initialDelaySeconds: 15, the kubelet will declare the container dead and issue a SIGTERM while the app is still initializing.

# Inspect the latest 20 events recorded for the pod
kubectl describe pod <pod-name> -n <namespace> | grep -A 10 Events:

Watch out for warnings like:

Warning  Unhealthy  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 500
Killing container with id containerd://app: Container failed liveness probe, will be restarted

The Solution: Decouple startup from ongoing health by implementing a startupProbe:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 2
# Liveness probe only activates AFTER startupProbe succeeds:
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Step 4: Investigate Node Kernel OOM Killer Events

When dealing with Exit Code 137, determine whether the container exceeded its own cgroup memory limit, or whether the entire worker node ran out of memory:

# 1. Identify which node is hosting the pod
NODE_NAME=$(kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeName}')

# 2. Inspect node dmesg logs for kernel OOM invocations
kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/logs/dmesg" | grep -i -E "oom_killer|killed process"

Runtime Heap Caveat: Runtimes like Node.js (V8) and Java (JVM) measure total host RAM by default. If your node has 64 GB RAM but your pod has a 1 GB limit, the runtime may allocate beyond 1 GB unless constrained via --max-old-space-size=768 or -XX:MaxRAMPercentage=75.

Step 5: Apply Runtime Memory & Probe Remediation

To permanently prevent CrashLoopBackOff:

Set memory requests equal to limits for latency-sensitive microservices to achieve a Guaranteed Quality of Service (QoS) tier.
Tune probe thresholds using startupProbe for slow boot routines.

Establish Prometheus alerts on container memory approaching limits:

container_memory_working_set_bytes / kube_pod_container_resource_limits{resource="memory"} > 0.85

Interactive Practice: Want to test your skills against real production scenarios? Try the scenario directly on our interactive platform:

👉 Practice Scenario: Silent OOMKilled & CrashLoopBackOff on Interview Hub →

Further Learning & Recommended Resources

Kubernetes Mastery Path: 24 hands-on interactive modules with live quizzes and local Minikube sandboxes.
DevOps & SRE Interview Hub: 950+ scenario-based incident runbooks with candidate storytelling elevator pitches.
Architecture Runbook: How to master Kubernetes in 30 days.

Naveed Ahmed

Senior DevOps, Cloud & SRE Engineer specializing in Kubernetes, AWS, and Cloud Architecture.

Portfolio Interview Hub K8s Mastery LinkedIn