Kubernetes Scenarios: 12 Real-World Production Problems and Battle-Tested Solutions

Packet sniffing on worker node interfaces revealed concurrent DNS lookups for IPv4 (A) and IPv6 (AAAA) records sent simultaneously over UDP. Linux Netfilter conntrack experienced a race condition:

# Check conntrack insertion drops
conntrack -S | grep insert_failed

# Trace CoreDNS UDP requests from inside container
tcpdump -i eth0 udp port 53 -nn

Because glibc defaults to ndots:5, an external query for api.stripe.com sequentially appended 5 cluster search domains before querying the root domain. When two queries hit the conntrack table on the same tuple simultaneously, the second packet was silently dropped by the kernel, triggering a 5.0-second UDP retransmission timeout.

1. Deploy NodeLocal DNSCache: Runs as a DaemonSet with a local loopback IP (169.254.20.10), answering queries directly on the node via TCP and bypassing Netfilter conntrack.
2. Inject pod dnsConfig: Set ndots: 2 and single-request-reopen in workload specs.
3. Trailing Dots: Configure microservice environment variables with fully qualified domain names (https://api.stripe.com.).

Developers deployed 10 new replicas of a data ingestion service without memory requests or limits (QoS class: BestEffort). When a data backlog arrived, these pods consumed all free host RAM:

# Check kernel dmesg for host OOM killer
dmesg -T | grep -i "Out of memory: Kill process"

# Inspect node allocatable vs total capacity
kubectl describe node ip-10-0-45-12.ec2.internal | grep -A 8 "Allocatable:"

Because nodes lacked system-reserved and kube-reserved memory cushions, the Linux kernel OOM killer activated globally and terminated the containerd runtime and kubelet processes to protect the kernel.

1. Enforce Hard Kubelet Reservations: Set --system-reserved=memory=1Gi and --kube-reserved=memory=1Gi in KubeletConfiguration.
2. Namespace LimitRanges: Enforce mandatory default memory requests and limits for all containers.
3. PriorityClasses: Assign system components (CoreDNS, CNI) a high PriorityClass to guarantee immunity from pod preemption.

The EKS cluster resided in a /23 VPC private subnet (506 usable IPs). By default, AWS VPC CNI assigns secondary private IP addresses directly from the subnet for each pod, and allocates warm pools via WARM_IP_TARGET=30 per node:

# Inspect available IPs in subnet via AWS CLI
aws ec2 describe-subnets --subnet-ids subnet-03b9e4a821e29e92a \
  --query "Subnets[*].[SubnetId,AvailableIpAddressCount]" --output table

With 15 nodes holding 30 warm IPs each, 450 IPs were reserved before workloads even ran, completely exhausting the subnet.

1. Enable Prefix Delegation: Configure AWS VPC CNI to allocate /28 CIDR blocks (16 IPs per ENI slot) instead of individual secondary IPs (ENABLE_PREFIX_DELEGATION=true).
2. Custom Networking (Secondary CIDR): Attach a non-routable 100.64.0.0/16 CGNAT CIDR block to the VPC dedicated solely to Kubernetes pod interfaces.

A security vendor's mutating admission webhook was configured with failurePolicy: Fail. When the node hosting the webhook crashed, the API server attempted to connect to its ClusterIP service and timed out after 10 seconds:

# View registered webhook configurations
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations

# Check webhook failure policies
kubectl get mutatingwebhookconfigurations security-agent-webhook -o yaml | grep -A 2 failurePolicy

Because the webhook pods themselves could not be rescheduled (since all pod creations were blocked by the missing webhook), the cluster entered a complete chicken-and-egg deadlock.

1. Emergency Unfreeze: Delete the blocking webhook configuration: kubectl delete mutatingwebhookconfiguration security-agent-webhook.
2. Exclude System Namespaces: Add namespaceSelector matching kubernetes.io/metadata.name: kube-system with NotIn so control plane pods are never intercepted.
3. High Availability Webhooks: Run webhooks with at least 3 replicas, spread across zones with a PodDisruptionBudget.

When a Deployment initiates a rolling replacement, the old pod receives a SIGTERM signal from kubelet while simultaneously being removed from the Service Endpoints object.

However, updating Ingress-NGINX routing tables and draining backend keep-alive connection pools takes 2 to 4 seconds. The application container terminated immediately upon receiving SIGTERM, causing Ingress-NGINX to forward active user requests to a dead TCP socket.

# Inject preStop sleep hook into container spec
lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]

The 15-second delay ensures Ingress-NGINX and kube-proxy completely purge the old pod IP from active routing tables before the application process stops accepting connections.

The DevOps team patched the PVC storage size, triggering the AWS EBS CSI driver to expand the cloud volume. However, AWS EBS imposes a hard constraint: once a volume is modified, it enters the optimizing state and cannot be modified again for 6 hours.

Additionally, the node hosting the pod was rebooted while the filesystem was actively resizing (resize2fs), corrupting the ext4 journal block.

1. Rescue Mount: Detached volume from the worker node and attached to a rescue EC2 instance to execute non-destructive e2fsck -f /dev/xvdf.
2. Volume Binding Mode: Enforce volumeBindingMode: WaitForFirstConsumer in StorageClasses.
3. VolumeSnapshots: Mandate a verified VolumeSnapshot execution prior to any manual PVC storage modification.

Kubernetes enforces CPU limits using Linux Completely Fair Scheduler (CFS) bandwidth quotas evaluated over 100ms periods (cfs_period_us=100000).

Inspect Prometheus metric:

# Query throttled periods in Prometheus
sum(rate(container_cpu_cfs_throttled_periods_total[5m])) by (pod)
  / sum(rate(container_cpu_cfs_periods_total[5m])) by (pod) * 100

A multi-threaded Go application with limit 1000m spawned 8 goroutines that collectively consumed 100ms of CPU time in the first 12ms of the period. The Linux kernel halted all execution threads for the remaining 88ms!

1. Remove Restrictive CPU Limits: In modern Kubernetes, remove CPU limits entirely on multi-threaded latency-sensitive microservices, keeping only accurate CPU requests for scheduling.
2. Tune CFS Period: If CPU limits are legally mandated, increase cpuCFSQuotaPeriod in KubeletConfiguration to smooth out bursty workloads.

etcd writes transaction write-ahead logs (WAL) to disk synchronously using fsync(). The etcd nodes were deployed on AWS EBS GP2 volumes that exhausted their IOPS burst balance credits:

# Test etcd cluster disk performance
ETCDCTL_API=3 etcdctl check perf

# Inspect etcd fsync histogram in Prometheus
histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))

WAL fsync latency jumped from 2ms to 140ms. etcd missed Raft heartbeats, triggering leader election thrashing across all 3 members.

1. Migrate to Dedicated Fast Storage: Run etcd on dedicated NVMe SSD instance stores or AWS EBS io2 volumes with guaranteed <10ms p99 fsync latency.
2. Automated Snapshots: Configure hourly etcdctl snapshot save routines to S3 with automated checksum verifications.

In self-managed Kubernetes clusters created via kubeadm, the control plane PKI client certificates expire exactly 1 year after generation:

# Check certificate expiration dates
kubeadm certs check-expiration

No monitoring alert was configured for the kube-apiserver.crt certificate expiration, resulting in an unannounced total control plane blackout at midnight.

1. Emergency Renewal:

   kubeadm certs renew all
   systemctl restart kubelet

2. Automated Expiration Alerting: Deploy cert-manager or Prometheus exporter monitoring x509_cert_not_after with alerts triggering at 30 days and 7 days before expiry.

The security engineer applied an ingress: [] default-deny NetworkPolicy to meet SOC2 requirements, but forgot that Kubernetes NetworkPolicies are whitelist-only:

# Inspect NetworkPolicies affecting pod
kubectl get networkpolicy -n production -o wide
calicoctl get endpoints -o wide

Crucially, the policy also omitted an explicit Egress rule for DNS (UDP/TCP port 53). Pods could not resolve CoreDNS service names, causing all HTTP client connections to fail at the DNS lookup stage.

1. Always Whitelist CoreDNS: Every egress network policy must explicitly permit traffic to port 53 in kube-system.
2. Staged Policy Enforcement: Deploy Calico in audit/permissive mode before switching to enforce mode, verifying allowed flows in telemetry dashboards.

The base image python:3.11-slim was pulled directly from public DockerHub without an authenticated account. Because all worker nodes in the VPC egressed through a single AWS NAT Gateway elastic IP, DockerHub treated the entire cluster as a single anonymous client and rate-limited all image pulls across the company.

1. Mirror to Private ECR: Never pull public images directly in production manifests. Mirror all verified base images into private Amazon ECR repositories.
2. ECR Pull-Through Cache: Configure ECR Pull-Through Cache rules to automatically cache upstream DockerHub images with zero code changes.

A microservice opened log files in an emptyDir volume, deleted them via rm, but kept the file descriptors open in uncollected zombie processes.

In Linux, unlinked files remain on disk allocating physical blocks until all processes holding the open file descriptor terminate:

# Find open file descriptors holding deleted files
lsof +L1 /var/lib/kubelet/pods/

Kubelet saw the root filesystem capacity exceed its imageGCHighThresholdPercent (85%) and began aggressively evicting pods.

1. Restart Leaking Process: Identify and kill the parent process holding the unlinked file descriptor to immediately release disk blocks.
2. Enforce Ephemeral Storage Limits: Add resources.limits.ephemeral-storage: 5Gi to all container specifications to prevent single-pod disk exhaustion.