10 Kubernetes Production Problems I Check First: SRE Incident Runbook

Exit Code 137: Linux OOMKilled vs Node Memory Pressure

Exit code 137 means SIGKILL was dispatched. The critical question: Was it a container-level OOM (exceeded resources.limits.memory) or a node-level OOM (host kernel killed processes due to node exhaustion)?

# Check if container was OOMKilled
kubectl get pods -n prod -o jsonpath='{range .items[*]}{.metadata.name}{"	"}{.status.containerStatuses[*].lastState.terminated.reason}{"	"}{.status.containerStatuses[*].lastState.terminated.exitCode}{"
"}{end}' | grep OOMKilled

# Check node dmesg for kernel kill events
kubectl debug node/<node-name> -it --image=busybox -- chroot /host dmesg -T | grep -i -E "oom[-_]killer|killed process"

The Fix: Differentiate RSS memory from Page Cache. If JVM or Node.js heap is configured higher than container memory limits, tune heap flags (-XX:MaxRAMPercentage=75.0).

Exit Code 143: Graceful Termination Timeout Races

When a pod exits with 143, it received SIGTERM but failed to shutdown gracefully within terminationGracePeriodSeconds (default: 30s), forcing kubelet to send SIGKILL.

# Inspect pod shutdown duration events
kubectl get events -n prod --field-selector reason=Killing --sort-by='.metadata.creationTimestamp'

The Fix: Implement a preStop hook with a sleep (e.g. sleep 10) to allow Kubernetes endpoints and kube-proxy iptables to remove the pod IP before the application stops accepting traffic, and increase terminationGracePeriodSeconds: 60.

Liveness Probe Cascading Death Spirals

The most destructive configuration mistake in Kubernetes: configuring a liveness probe to check downstream dependencies (PostgreSQL database or Redis). If the database slows down, all pods fail their liveness probe simultaneously, kubelet restarts all containers in a stampede, destroying connection pools and taking down the entire system.

The Rule: Liveness probes must only verify if the container process is deadlocked internally (shallow /healthz check returning HTTP 200). Use readiness probes for dependency checks so traffic is temporarily detached without restarting the process.

CoreDNS 5-Second Latency & Conntrack Table Collisions

When external or inter-pod requests experience sudden 5.00-second latency spikes, it is almost always the Linux netfilter conntrack UDP race condition between IPv4 (A) and IPv6 (AAAA) DNS queries.

# Check conntrack insertion drops
sudo conntrack -S

# Verify CoreDNS response metrics
kubectl top pods -n kube-system -l k8s-app=kube-dns

The Fix: Deploy NodeLocal DNSCache daemonset on every node to serve DNS over local TCP loops, eliminating conntrack UDP races entirely.

AWS VPC CNI IP Address Exhaustion

Pods remain stuck in ContainerCreating or FailedCreatePodSandBox with error: "failed to assign an IP address to container".

# Check aws-node daemonset IP pool status
kubectl get pods -n kube-system -l k8s-app=aws-node
kubectl describe daemonset aws-node -n kube-system | grep -A 8 "Environment:"

The Fix: Enable Prefix Delegation on the AWS VPC CNI. Set ENABLE_PREFIX_DELEGATION=true and configure WARM_PREFIX_TARGET=1, expanding node IP capacity from ~30 IPs to over 250 IPs per instance.

Remaining Essential Checks

• 6. Node NotReady from Inode Exhaustion: Check df -ih. Millions of orphaned container logs or build cache artifacts fill the inode table even when disk space is 80% free.
• 7. etcd Fsync Latency Spikes (> 10ms): High disk IOPS on control plane nodes causes etcd leader election loss. Ensure etcd storage uses dedicated io2/gp3 volumes with guaranteed IOPS.
• 8. Ingress 502 Bad Gateway (Keep-Alive Mismatch): Occurs when the AWS ALB idle timeout (60s) is longer than the application server keep-alive timeout (5s in Node.js). Set upstream keep-alive to 65s.
• 9. ImagePullBackOff (Docker Hub / ECR Rate Limits): Verify IAM role permissions for ECR and deploy pull-through caches to prevent public registry rate throttling.
• 10. Karpenter Node Provisioning Failures: Check kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter for InsufficientInstanceCapacity or missing subnet tags.