Kubernetes CrashLoopBackOff Troubleshooting: The Complete Production Runbook

Every Kubernetes engineer has stared at it during a 2:00 AM incident: a pod status transitioning from Running to Error, and finally settling into the dreaded CrashLoopBackOff state.

Kubernetes CrashLoopBackOff is not an error in itself. It is the status Kubernetes reports when a container keeps crashing and the kubelet is waiting longer between restart attempts, so the real question is always why the container exits. This runbook follows the order that finds the cause fastest.

Kubernetes CrashLoopBackOff Root Causes: Decoding Exit Codes

Before blindly restarting deployments or changing image tags, check the container exit code. The exit code reveals the exact mechanism of death:

Step 1: Inspect Container Termination State & Exit Code

Standard kubectl get pods only shows that restarts are incrementing. Query the container status subresource directly to extract the exact termination details:

# Extract termination reason and exit code from the last crashed instance
kubectl get pod <pod-name> -n <namespace> \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

If you see the following JSON structure, you are facing an Out-Of-Memory termination:

{"exitCode": 137, "reason": "OOMKilled", "startedAt": "...", "finishedAt": "..."}

Step 2: Capture Logs From the Previous Crashed Instance

When a pod restarts, running kubectl logs <pod-name> only streams logs from the newly spawned instance, which may look healthy before it dies. To see why the previous container crashed, use the --previous (or -p) flag:

# Stream stdout/stderr from the container that just died
kubectl logs <pod-name> -n <namespace> --previous --tail=100

Step 3: Check Pod Events & Liveness Probe Failures

A frequent cause of Exit Code 143 is an overly aggressive Liveness Probe. If your application takes 45 seconds to establish its database connection pool during startup, but your probe has initialDelaySeconds: 15, the kubelet will declare the container dead and issue a SIGTERM while the app is still initializing.

# Inspect the latest 20 events recorded for the pod
kubectl describe pod <pod-name> -n <namespace> | grep -A 10 Events:

Watch out for warnings like:

Warning  Unhealthy  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 500
Killing container with id containerd://app: Container failed liveness probe, will be restarted

The Solution: Decouple startup from ongoing health by implementing a startupProbe:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 2
# Liveness probe only activates AFTER startupProbe succeeds:
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Step 4: Investigate Node Kernel OOM Killer Events

When dealing with Exit Code 137, determine whether the container exceeded its own cgroup memory limit, or whether the entire worker node ran out of memory:

# 1. Identify which node is hosting the pod
NODE_NAME=$(kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeName}')

# 2. Inspect node dmesg logs for kernel OOM invocations
kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/logs/dmesg" | grep -i -E "oom_killer|killed process"

Step 5: Apply Runtime Memory & Probe Remediation

To permanently prevent CrashLoopBackOff:

1. Set memory requests equal to limits for latency-sensitive microservices to achieve a Guaranteed Quality of Service (QoS) tier.
2. Tune probe thresholds using startupProbe for slow boot routines.
3. Establish Prometheus alerts on container memory approaching limits:

   container_memory_working_set_bytes / kube_pod_container_resource_limits{resource="memory"} > 0.85

Preventing Kubernetes CrashLoopBackOff Before It Pages You

Fixing a crash loop at 3 AM is expensive, and most of them are preventable in the pod spec and the release process.

Right-size requests and limits

Set memory limits from observed usage, not guesses. Look at peak working-set memory over a representative period, add headroom, and revisit after significant releases. A limit that is too tight is the most common reason a healthy application is OOMKilled under load.

Design probes that do not kill healthy pods

Use a startup probe for slow-starting applications so the liveness probe does not restart a container that is still booting. Keep liveness checks cheap and local. If a liveness probe depends on a database or another service, one outage can restart every replica at once.

Shut down gracefully

Handle SIGTERM, finish in-flight requests and set terminationGracePeriodSeconds to match how long the application really needs. A container that ignores SIGTERM is eventually killed with SIGKILL, which appears as exit code 137 even when memory is fine.

Roll out changes safely

Use rolling updates with readiness probes and a conservative maxUnavailable, and alert on restart counts, not only on pod phase. A bad release should stall at the first replica instead of taking the service down. These habits are the surest way to stop Kubernetes CrashLoopBackOff from becoming a recurring incident. For a wider triage order see the Kubernetes runbook for 12 production failures, and for hands-on drills try the kubectl practice game.

Frequently Asked Questions

Further Learning & Recommended Resources

• Kubernetes Mastery Path: 24 hands-on interactive modules with live quizzes and local Minikube sandboxes.
• DevOps & SRE Interview Hub: 950+ scenario-based incident runbooks with candidate storytelling elevator pitches.
• Architecture Runbook: How to master Kubernetes step by step.

Naveed Ahmed (Kumbhar)

Senior DevOps & Cloud Engineer with 10+ years specializing in AWS, Kubernetes, Platform Engineering, SRE incident response, and autonomous AI infrastructure agents.

Passionate about cloud architecture & systems engineering? Connect on LinkedIn →

Portfolio · Interview Hub (970 Scenarios) · K8s Mastery Path · Email Naveed ✉️