The Complete Kubernetes Incident Runbook Atlas: 12 Production Failures, Root Cause, and Fix

CrashLoopBackOff & Container Termination State

1. Diagnostic CLI Commands

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\tExitCode:"}{.state.waiting.message}{.lastState.terminated.exitCode}{"\tReason:"}{.lastState.terminated.reason}{"\n"}{end}'

kubectl logs <pod-name> -n <namespace> --previous --tail=100

# Run on the host node or via ephemeral debug container:
dmesg -T | grep -E -i "oom_reaper|killed process|out of memory"

2. Exit Code Decoder

• Exit Code 137: Container was terminated by SIGKILL (128 + 9). In 95% of cases, this is an OOMKilled trigger from the Linux kernel cgroup controller when memory limits are exceeded.
• Exit Code 143: Container received SIGTERM (128 + 15). Kubelet gracefully terminated the pod, often due to a failed liveness probe or preemption.
• Exit Code 1 / 2: Application-level runtime exception (missing configuration file, unhandled Promise rejection, database connection failure).

3. Permanent Remediation

If exit code is 137, increase memory limits in the deployment manifest or profile application heap allocation. If exit code is 143 due to probe failure, adjust initialDelaySeconds and failureThreshold to allow slow application bootstrap:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 45   # Increase from default 0-10s
  periodSeconds: 10
  failureThreshold: 5       # Allow up to 5 consecutive failures before killing

ImagePullBackOff & ErrImagePull

1. Diagnostic CLI Commands

kubectl describe pod <pod-name> -n <namespace> | grep -A 10 "Events:"

2. Root Cause Decision Tree

• 401 / 403 Forbidden: Missing or expired imagePullSecrets. For AWS ECR, the IAM node role lacks ecr:GetAuthorizationToken and ecr:BatchGetImage.
• 429 Too Many Requests: Docker Hub anonymous rate limiting (100 pulls per 6 hours per IP). Remediate by authenticating or proxying through an internal harbor/ECR pull-through cache.
• 404 Not Found: Image tag typo or multi-architecture mismatch (e.g. attempting to run an arm64 image on an amd64 node).

3. Permanent Remediation

# Create registry secret and patch default service account:
kubectl create secret docker-registry regcred \
  --docker-server=<your-registry-server> \
  --docker-username=<your-user> \
  --docker-password=<your-token> \
  -n <namespace>

# Add to pod deployment:
spec:
  imagePullSecrets:
    - name: regcred

Node NotReady & Kubelet Heartbeat Failures

1. Diagnostic CLI Commands

# Check node condition reasons:
kubectl describe node <node-name> | grep -A 8 "Conditions:"

# SSH into the node and inspect kubelet logs:
journalctl -u kubelet -n 150 --no-pager

# Check containerd / runtime daemon status:
systemctl status containerd

2. Remediation

If containerd has hung on zombie shim processes:

sudo systemctl restart containerd
sudo systemctl restart kubelet

If the node is under DiskPressure (filesystem >85% capacity):

# Clean dangling images and stopped containers:
crictl rmi --prune
df -h /var/lib/containerd /var/lib/kubelet

etcd Performance Degradation & Latency Spikes

1. Diagnostic CLI Commands

# Check etcd member health and cluster alarms:
etcdctl endpoint health --write-out=table
etcdctl alarm list

# Check disk fsync latency warnings in logs:
journalctl -u etcd | grep -E "took too long|lost leader|server is likely overloaded"

2. Remediation

Defragment the etcd database and disarm storage quota alarms:

# Defragment all endpoints:
etcdctl defrag --cluster

# Disarm alarms once defragmentation frees space:
etcdctl alarm disarm

Kube-Apiserver Unreachable / 503 Errors

1. Diagnostic CLI Commands

# Identify misconfigured validating or mutating webhooks:
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

# Check apiserver container logs:
kubectl logs -n kube-system -l component=kube-apiserver --tail=100 | grep -E "webhook|timeout|503"

2. Emergency Remediation

If a third-party webhook (e.g., Istio, Cert-Manager, or policy agent) is unreachable and configured with failurePolicy: Fail, it blocks all cluster admissions. Change policy to Ignore in an emergency:

kubectl patch validatingwebhookconfiguration <webhook-name> \
  --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"}]'

PVC & StorageClass Stuck in Pending

1. Diagnostic CLI Commands

kubectl describe pvc <pvc-name> -n <namespace>
kubectl get events -n <namespace> --field-selector reason=FailedBinding

2. Remediation

Ensure your StorageClass uses volumeBindingMode: WaitForFirstConsumer. This prevents the cloud provider from creating an EBS/Azure disk in us-east-1a when the pod is scheduled on a worker node in us-east-1b:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer   # Critical for multi-AZ clusters

Karpenter / Autoscaler Node Provisioning Failures

1. Diagnostic CLI Commands

# Check Karpenter controller logs:
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --tail=100 | grep -E "ERROR|failed|capacity"

# Inspect unschedulable pod requirements:
kubectl get pods -A --field-selector status.phase=Pending

2. Remediation

Broaden your EC2NodeClass / NodePool instance types. Avoid pinning to a single instance family (e.g. m5.large). Allow Karpenter to select across multiple generation families and availability zones:

requirements:
  - key: "karpenter.k8s.aws/instance-category"
    operator: In
    values: ["c", "m", "r"]
  - key: "karpenter.k8s.aws/instance-generation"
    operator: Gt
    values: ["4"]
  - key: "karpenter.sh/capacity-type"
    operator: In
    values: ["spot", "on-demand"]

ArgoCD Out-of-Sync & Infinite Sync Loops

1. Diagnostic CLI Commands

argocd app diff <app-name> --hard-refresh
argocd app get <app-name> --show-operation-details

2. Remediation

Add an ignoreDifferences block in your ArgoCD Application manifest to ignore dynamic fields populated by cloud controllers:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/template/spec/containers/0/imagePullPolicy

Network Policy Lockouts & CoreDNS Resolution Failures

1. Diagnostic CLI Commands

# Test DNS resolution from inside a debug pod:
kubectl run dns-test --rm -it --image=busybox:1.28 -- nslookup kubernetes.default

# Check CoreDNS pod status and logs:
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

2. Remediation

Ensure any default-deny egress NetworkPolicy includes an explicit rule permitting traffic to CoreDNS in the kube-system namespace:

egress:
  - to:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: kube-system
        podSelector:
          matchLabels:
            k8s-app: kube-dns
    ports:
      - protocol: UDP
        port: 53
      - protocol: TCP
        port: 53

ResourceQuota Exceeded & Scheduling Deadlocks

1. Diagnostic CLI Commands

kubectl get resourcequota -n <namespace>
kubectl describe resourcequota -n <namespace>

2. Remediation

Identify pods consuming quota without active workloads, right-size requests, or increase the namespace quota specification via Infrastructure as Code.

HPA Not Scaling & Metrics Server Failures

1. Diagnostic CLI Commands

kubectl describe hpa <hpa-name> -n <namespace>
kubectl top pods -n <namespace>

2. Remediation

HPA cannot calculate percentage utilization without a baseline request. Always declare container CPU requests:

resources:
  requests:
    cpu: 250m
    memory: 256Mi

Control Plane TLS Certificate Expiration

1. Diagnostic CLI Commands

kubeadm certs check-expiration

2. Remediation

Renew all control plane certificates with zero downtime:

kubeadm certs renew all
# Restart static control plane pods:
sudo kill -s SIGHUP $(pidof kube-apiserver kube-controller-manager kube-scheduler)