Kubernetes Interview Questions 2026: 100+ Questions and Answers (From Primitives to Production Architecture)

When you execute kubectl apply, the client validates the YAML schema, merges local changes with metadata.generation, and issues an HTTP POST or PATCH to kube-apiserver. The API server performs: 1. Authentication & Authorization: Validates user X.509 cert or OIDC bearer token, followed by RBAC checks. 2. Admission Control: Passes the object through Mutating Admission Webhooks (e.g., sidecar injectors), Schema Object validation, and Validating Admission Webhooks (e.g., Kyverno, OPA Gatekeeper). 3. etcd Persistence: The object is serialized to JSON/protobuf and committed into etcd with a new resourceVersion. 4. Controller Reconciliation: kube-controller-manager (DeploymentController) detects the new object via watch API, calculates the ReplicaSet delta, and creates the ReplicaSet. The ReplicaSetController creates the individual Pod objects in Pending state. 5. Scheduling: kube-scheduler filters and scores worker nodes, binds the Pod to a node by setting nodeName, and updates etcd. 6. Kubelet Execution: The target node's kubelet notices the scheduled Pod via its watch channel, invokes the CNI plugin for pod network namespace creation, CSI plugin for volume attachment, and the CRI (containerd) to pull container images and spawn containers.

etcd uses the Raft consensus algorithm. It requires an active quorum of (N/2) + 1 nodes to accept writes. In a 3-node cluster, 2 nodes must agree; in a 5-node cluster, 3 must agree.

Quorum Loss Causes: Split-brain network partitions, high disk fsync latency (>10ms on slow disks), out-of-memory crashes on control plane nodes, or simultaneous node termination. When quorum is lost, kube-apiserver becomes read-only or completely unresponsive to write operations, freezing deployments and pod scheduling while existing running pods continue executing undisturbed.

👉 Deep Dive: Learn the disaster recovery runbook in Kubernetes Control Plane & Pod Scheduling Triage.

The Kubernetes scheduler assigns unscheduled Pods to nodes in two distinct phases:

• Filtering (formerly Predicates): Eliminates nodes that cannot run the Pod. Checks include NodeResourcesFit (CPU/memory requests), NodeName, NodePorts, PodTopologySpread, and NodeAffinity / Taints. If no node passes, the Pod stays in Pending with FailedScheduling event.
• Scoring (formerly Priorities): Ranks remaining candidate nodes on a scale of 0 to 100 based on plugins like NodeResourcesBalancedAllocation, ImageLocality (preferring nodes where container image layers are already cached), and NodeAffinity weights. The node with the highest cumulative score is selected.

PLEG is a sub-module inside the kubelet that periodically inspects the container runtime (e.g., containerd) to detect container state transitions (started, exited, died) and publish events to the kubelet's internal channels.

'PLEG is not healthy' triggers when the relist cycle takes longer than 3 minutes (default timeout). Root causes: 1. Container runtime deadlock or unresponsive runtime socket (/run/containerd/containerd.sock). 2. Heavy disk I/O wait on the root filesystem preventing rapid container status queries. 3. Excessive pod churn or zombie processes flooding the node's process table.
Diagnostic command: journalctl -u kubelet -e --no-pager | grep -i pleg.

• iptables mode: Uses sequential kernel packet matching rules. Rule traversal cost scales as O(N) with the number of Services and Endpoints. In clusters with >5,000 Services, updating rules creates massive CPU spikes and packet delay.
• IPVS (IP Virtual Server) mode: Uses Netfilter transport-layer switching with hash tables. Operates at O(1) complexity regardless of cluster size. Supports advanced load balancing algorithms (round-robin, least connections, locality).
• eBPF mode (e.g. Cilium / Calico eBPF): Bypasses the Netfilter stack entirely. Injects eBPF programs directly at the network interface (XDP / TC layers) and socket layers (sockops). Delivers near wire-speed routing, eliminating connection tracking overhead and conntrack table exhaustion.

kube-controller-manager runs multiple replicas for redundancy, but only one active leader executes the reconciliation loops at any given moment. Leaders are coordinated using a Lease resource in the kube-system namespace.

The active leader renews its lease every 2 seconds (default renewDeadline: 10s). If the leader crashes or loses network connectivity to the API server, standby instances detect the expired lease and acquire leadership within 15 seconds (leaseDuration), preventing dual-controller split-brain write conflicts.

• CRI (Container Runtime Interface): A gRPC interface defined by Kubernetes that allows kubelet to communicate with container runtimes without recompiling Kubernetes code.
• OCI (Open Container Initiative): Industry standards specifying container images (image-spec) and how container bundles are executed (runtime-spec).
• High-level runtime (containerd / CRI-O): Implements the CRI gRPC API, pulls container images over HTTPS, manages snapshot filesystems, and delegates container execution to low-level runtimes.
• Low-level runtime (runc / crun): Directly interacts with the Linux kernel using cgroups, namespaces, and seccomp to spawn and isolate the actual Linux process.

Imperative (create/replace): Instructs the API server to perform an exact command. kubectl create fails if the resource already exists. kubectl replace overwrites the full object, requiring you to supply the complete definition and stripping fields injected by admission controllers.

Declarative (apply): Focuses on the desired end state. Uses a three-way merge patch between the local file, the live object in etcd, and the last-applied-configuration annotation. Retains fields dynamically added by controllers (like replicas scaled by HPA).

Kubernetes enforces optimistic concurrency via the metadata.resourceVersion field stored in etcd. When a client reads an object, it receives its current resourceVersion. When submitting updates (PUT/PATCH), the client must supply this version. If another client updated the resource in the interim, etcd rejects the write with HTTP status 409 Conflict. Clients must re-fetch the latest state, re-apply modifications, and retry.

The API Aggregation layer allows extending the Kubernetes API by registering dedicated backend HTTP servers (APIServices) rather than storing custom resources in the primary etcd cluster.

Use CRDs (Custom Resource Definitions) for 95% of use cases: declarative workflows, custom operators, and application lifecycle definitions.
Use API Aggregation when: 1. You require a dedicated high-throughput datastore (e.g., metrics-server storing time-series metrics in in-memory RAM without thrashing etcd). 2. Custom business validation logic requiring non-standard protocol translations. 3. Strict multi-tenant data isolation where custom objects must not reside on the primary control plane disk.

Admission controllers intercept API requests after authentication and authorization, but before object persistence in etcd.

• Mutating Admission Webhooks: Executed first. Can modify the object request (e.g., injecting Istio sidecar proxies, setting default security contexts, adding corporate labels).
• Validating Admission Webhooks: Executed second. Read-only validation. Evaluates business rules (e.g., rejecting images with :latest tag, verifying non-root user). Rejection terminates the request immediately with an HTTP 400 error.

The behavior depends on the webhook configuration's failurePolicy:

• failurePolicy: Fail (Default in secure production clusters): If the webhook service is unreachable or times out (default 10s), the API request is rejected outright. If this webhook matches Pod creations, cluster-wide pod scheduling halts!
• failurePolicy: Ignore: The API server logs a warning and allows the request through without validation. High availability best practice: Webhooks must run at least 3 replicas spread across multiple availability zones with PodDisruptionBudgets.

The Garbage Collector controller deletes orphaned resources whose owner references are invalid.

• Foreground Cascading Deletion: The parent object transitions to 'deletion in progress' with a foregroundDeletion finalizer. The controller deletes all child objects (e.g., Pods) first; once all children are gone, the parent is deleted.
• Background Cascading Deletion: The parent object is deleted immediately from etcd, and the garbage collector subsequently deletes the child objects in the background.
• Orphan Deletion: The parent is deleted, but child objects have their ownerReferences removed and continue running independently.

Kubelet interacts with containerd via gRPC over the local UNIX domain socket /run/containerd/containerd.sock. Two distinct gRPC services are consumed: 1. RuntimeService: Responsible for sandbox creation (pod network namespace setup), starting/stopping containers, attaching/executing commands, and collecting metrics. 2. ImageService: Responsible for querying local images, pulling remote container layers from registries, and removing stale images during node disk garbage collection.

• Pending: The Pod manifest was accepted by the API server, but one or more containers are not yet created. Common causes: waiting for scheduler assignment, pulling container images, or waiting for PVC volume binding.
• Running: Bound to a node, all containers created, and at least one container is executing or in the process of starting.
• Succeeded: All containers in the Pod completed execution with exit code 0. Typical for batch Job workloads.
• Failed: All containers terminated, and at least one container terminated with a non-zero exit code or was terminated by the system (e.g., OOMKilled exit code 137).
• Unknown: The API server cannot communicate with the node's kubelet (typically due to network partition or node failure).

• Init Containers: Run sequentially to completion before any app container starts. If an init container exits with non-zero, the Pod restarts according to its restartPolicy.
• App Containers: Execute concurrently throughout the Pod lifecycle until stopped or crashed.
• Native Sidecar Containers (K8s 1.28+ feature gate SidecarContainers, GA in 1.29+): Init containers configured with restartPolicy: Always. They start before application containers, stay running throughout the Pod lifetime, and shut down after the main application containers finish, solving historical logging and proxy termination issues.

Deployments are for stateless applications where pods are interchangeable and fungible. Use StatefulSets when workloads require:

1. Stable, unique network identifiers: Predictable hostnames (pod-0, pod-1) resolvable via Headless Services (app-0.app-service.ns.svc.cluster.local).
2. Stable, persistent storage: Each replica receives a dedicated PVC generated by volumeClaimTemplates that persists across pod rescheduling.
3. Ordered, graceful deployment and scaling: Sequential startup (0, 1, 2) and reverse-order termination (2, 1, 0) essential for quorum databases (ZooKeeper, Kafka, Cassandra).

👉 Practice Scenario: Deployment vs StatefulSet vs DaemonSet Architectural Decision Matrix.

A DaemonSet guarantees that all (or some subset of) nodes run a copy of a Pod. Typical use cases: node log collectors (Fluentbit), monitoring agents (Node Exporter, Datadog), and CNI network plugins (Calico, AWS VPC CNI).

In modern Kubernetes (1.17+), DaemonSet pods are scheduled by the default kube-scheduler using node affinity rules rather than the DaemonSet controller directly. DaemonSet pods automatically apply default tolerations for node taints like node.kubernetes.io/not-ready and node.kubernetes.io/unschedulable to ensure continuous daemon execution.

A PDB limits the number of pods of a replicated application that can be down simultaneously from voluntary disruptions (e.g., kubectl drain, cluster autoscaler node consolidation, or EKS AMI rolling upgrades).

You specify either minAvailable: 2 or maxUnavailable: 1. When an engineer drains a node, the Eviction API checks active PDBs; if evicting the pod violates the budget, the eviction is blocked until replacements are healthy elsewhere, preventing unexpected production outages.

• RollingUpdate: Default strategy. Gradually replaces old pods with new ones without downtime. Governed by maxSurge (how many pods can be created above desired replica count) and maxUnavailable (how many pods can be unavailable during update).
• Recreate: Terminates all existing pods simultaneously before creating the new version. Incurs immediate downtime. Required only when multiple application versions cannot run concurrently (e.g., non-backward-compatible database schema migrations or storage locked to a single writer).

# Trigger a rolling update with new image
kubectl set image deployment/payment-svc payment=company/payment:v2.4.1 --record

# Inspect rollout status in real-time
kubectl rollout status deployment/payment-svc

# View revision history
kubectl rollout history deployment/payment-svc

# Rollback immediately to previous revision
kubectl rollout undo deployment/payment-svc

# Rollback to a specific historical revision
kubectl rollout undo deployment/payment-svc --to-revision=3

👉 Live SRE Runbook: Master emergency rollback triage in Fast & Safe Rollback of Bad Production Deployments.

A Job creates one or more pods and tracks them until a specified number of completions occur successfully. A CronJob runs Jobs on a repeating cron schedule.

concurrencyPolicy defines how overlapping CronJob executions are handled if a previous job is still running:

• Allow (default): Multiple job instances run concurrently. Can exhaust database connection pools or memory.
• Forbid: Skips the new job if the previous instance has not completed.
• Replace: Cancels the currently running job and spawns the new job.

Ephemeral containers are temporary debug containers injected into an already-running pod via kubectl debug. Unlike standard containers, they cannot have ports or resource reservations.

Use them to troubleshoot production pods built from distroless or minimal images that lack shells, curl, or networking tools:

kubectl debug -it pod/payment-api-7b9 --image=nicolaka/netshoot --target=payment-api

The debug container shares the target container's process, network, and IPC namespaces.

When a Pod is deleted: 1. Pod enters Terminating phase and endpoints are removed from Service Endpoints. 2. Kubelet executes the preStop hook (if defined). 3. Kubelet sends SIGTERM to container processes. 4. Kubelet waits up to terminationGracePeriodSeconds (default 30s). 5. If processes have not exited, kubelet sends SIGKILL to forcibly kill all processes.
A preStop sleep hook (e.g., sleep 15) is essential to give kube-proxy and Ingress controllers time to remove the pod IP from load balancer endpoints before the application stops accepting connections, eliminating 502/504 errors.

• Mounted as Volume: Kubelet automatically updates projected volume files within a few seconds via atomic symlink rotation without restarting the pod. However, applications must actively watch the filesystem or reload config upon file change.
• Injected as Environment Variables: Environment variables are resolved at pod startup. Updates to ConfigMaps/Secrets are NOT reflected in running containers; the Pod or Deployment must be restarted (kubectl rollout restart deployment/app).

• Opaque: Default secret type for arbitrary user-defined key-value strings or base64-encoded credentials.
• kubernetes.io/tls: Contains tls.crt and tls.key for TLS termination certificates used by Ingress controllers and Gateway API.
• kubernetes.io/dockerconfigjson: Stores serialized .docker/config.json authentication tokens used by imagePullSecrets to pull private container images.

Finalizers are metadata keys that inform controllers to execute pre-delete cleanup routines (e.g., removing cloud load balancers, detaching AWS EBS volumes) before deleting the object from etcd.

If a resource has a finalizer and the responsible controller is deleted or failing, the object remains stuck in Terminating indefinitely. Emergency fix:

kubectl patch pod/stuck-pod -p '{"metadata":{"finalizers":null}}' --type=merge

Manual scaling sets an explicit spec.replicas count via kubectl scale or GitOps commits. HorizontalPodAutoscaler (HPA) dynamically adjusts spec.replicas based on observed metrics (CPU, memory, custom Prometheus queries) using the formula: desiredReplicas = ceil[currentReplicas * (currentMetricValue / targetMetricValue)]. If an HPA is active, manual scaling changes will be immediately overwritten by HPA's reconciliation loop.

1. All pods can communicate with all other pods on any node without NAT.
2. All agents on a node (e.g. kubelet) can communicate with all pods on that same node.
3. The IP that a pod sees as its own IP is the same IP that other pods see it as (no internal NAT between pods).
4. Nodes can communicate with all pods in the cluster across host interfaces.

• ClusterIP: Default type. Allocates a virtual cluster-internal IP from the service-cluster-ip-range. Only accessible from inside the cluster.
• NodePort: Exposes the service on a static high-range port (30000–32767) across every worker node's external IP address.
• LoadBalancer: Provisions a cloud provider load balancer (e.g., AWS NLB/ALB, GCP Cloud LB) that routes external traffic to the NodePort.
• ExternalName: Maps the service to a CNAME DNS record (e.g. db.external-domain.com) without proxying or allocating cluster VIPs.

The infamous 5-second DNS delay is caused by Linux kernel conntrack race conditions when sending simultaneous UDP queries for IPv4 (A) and IPv6 (AAAA) records over the same socket. Combined with /etc/resolv.conf default ndots:5, every single query evaluates 5 local search domains before external resolution.

Production Solutions: 1. Deploy NodeLocal DNSCache on every node to answer DNS queries locally over TCP. 2. In client pod specs, adjust DNS options: dnsConfig: options: [{name: 'ndots', value: '2'}, {name: 'single-request-reopen'}]. 3. Append a trailing dot to external domain names (e.g. api.stripe.com.) to skip search path expansion.

In traditional Kubernetes, an Endpoints resource contains a single monolithic list of all pod IPs for a Service. When a service scaled to 5,000+ pods, updating a single pod required rewriting the entire massive Endpoints object, triggering API server serialization bottlenecks and kube-proxy CPU spikes.

EndpointSlices (default in 1.21+) partition endpoints into multiple lightweight resources (up to 100 endpoints per slice). When a single pod updates, only that specific slice is modified, reducing network bandwidth and API server processing from O(N) to O(1).

An Ingress Controller (e.g., Ingress-NGINX, Traefik, HAProxy) is a reverse proxy running in the cluster that watches Kubernetes Ingress resources via the API server.

It dynamically generates reverse proxy configuration files (like nginx.conf) and reloads routes without dropping TCP connections. It terminates TLS certificates, performs path-based (/api/v1) and host-based (app.domain.com) routing, and distributes traffic directly to backend Pod IPs using keep-alive connection pools.

👉 Detailed Architecture: Ingress Controller End-to-End OSI Layer 7 Traffic Flow.

The Gateway API is the next-generation evolution of Ingress providing role-oriented networking:

• Separation of Concerns: GatewayClass (managed by Cloud/Infrastructure team), Gateway (managed by Cluster Platform team configuring IP, ports, TLS), and HTTPRoute / GRPCRoute / TCPRoute (managed by Application Developers).
• Cross-Namespace Routing: Routes can attach to Gateways across different namespaces with ReferenceGrant security controls.
• Native Advanced Traffic Controls: Direct support for header-based routing, traffic splitting/canaries (80/20 weights), request mirroring, and URL rewriting without custom vendor annotations.

By default, pod networking is non-isolated; all pods can send and receive traffic to any destination. Applying a NetworkPolicy selects pods via podSelector and isolates them.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

With empty ingress/egress rules, all inbound and outbound traffic is immediately dropped by the CNI plugin (Calico/Cilium) unless explicitly allowed by subsequent white-listing policies.

• externalTrafficPolicy: Cluster (default): Traffic hitting any node's NodePort is forwarded to backend pods across the cluster. May incur an extra network hop across nodes, and masks the client's real source IP with the node's SNAT IP.
• externalTrafficPolicy: Local: Only routes traffic to pods running on the local node that received the packet. Preserves the client's original source IP and eliminates extra inter-node hops. If a node has no local pods for that service, incoming packets are dropped (health check NodePort routes traffic only to healthy nodes).

Follow this systematic network triage:

1. Check Endpoints: Run kubectl get endpoints <svc-name>. If empty, verify the Service's spec.selector matches the Pod's labels exactly.
2. Check TargetPort: Verify targetPort matches the container's listening port.
3. Check Readiness Probes: If the pod's readiness probe is failing, it remains in Running state but is omitted from Endpoints.
4. Check NetworkPolicy: Verify ingress rules allow traffic from the caller's namespace.
5. Test Pod Directly: kubectl exec into caller pod and curl the target Pod IP directly. If pod IP works but ClusterIP fails, check kube-proxy or CNI IPVS/iptables state.

👉 Interactive Guide: Pod Running but Service Inaccessible: End-to-End Network Runbook.

CNI is an ecosystem standard specifying how network plugins configure network interfaces and IP addresses for containers.

• Overlay CNI (Calico with VXLAN/IPIP): Encapsulates pod packets inside host UDP packets. Pod IPs are independent of the underlying VPC network. Consumes CPU for packet encapsulation and adds 50 bytes of MTU overhead.
• Native Routed CNI (AWS VPC CNI): Allocates secondary real ENI private IP addresses directly from the VPC subnet. No packet encapsulation overhead; pods achieve native AWS network throughput and can communicate directly with RDS, ElastiCache, and on-premises resources via DirectConnect. Limitation: May exhaust VPC subnet IP addresses.

When using overlay networks (like VXLAN or Geneve), encapsulation headers reduce the maximum payload size by 50 bytes. If the node physical interface has an MTU of 1500 but the CNI interface is also set to 1500, packets exceeding 1450 bytes will be dropped or require fragmentation.

Symptoms: Small HTTP ping/healthcheck packets succeed, but large TLS handshakes (Client Hello certificates) or large database payloads hang indefinitely and time out.

A Headless Service is created by explicitly setting spec.clusterIP: None.

Instead of returning a single load-balanced virtual IP (VIP), CoreDNS returns direct A records for all backing pod IPs. Clients can directly discover and connect to specific pod replicas. Essential for StatefulSet databases (Kafka brokers, MongoDB replica sets, Elasticsearch cluster discovery) that perform their own client-side load balancing and leader election.

Linux Netfilter uses the conntrack table to track the state of all active TCP/UDP connections. When high-volume microservices open millions of short-lived TCP connections without connection pooling, the table reaches nf_conntrack_max.

Once full, the kernel silently drops all new SYN packets with nf_conntrack: table full, dropping packet. Node-level network connectivity completely halts. Fix: Increase net.netfilter.nf_conntrack_max via sysctl, deploy NodeLocal DNS, and enforce HTTP keep-alive connection reuse.

Configured via service.spec.sessionAffinity: ClientIP with optional sessionAffinityConfig.clientIP.timeoutSeconds (default 10800s / 3 hours). Kube-proxy records the client's IP in Netfilter/IPVS tables and routes repeated requests from that IP to the same backing pod. Note: If traffic arrives through an external HTTP proxy or NAT gateway, all users behind that gateway share the same IP, causing extreme traffic imbalance on a single pod.

• StorageClass: Defines the storage provisioner (e.g., ebs.csi.aws.com), disk parameters (gp3, iops, throughput), and reclamation policies. Acts as an automated blueprint for provisioning.
• PersistentVolumeClaim (PVC): A user's request for storage. Specifies storage capacity (e.g. 50Gi) and access modes (ReadWriteOnce) without caring about underlying hardware.
• PersistentVolume (PV): The actual storage resource in the cluster provisioned either statically by an admin or dynamically by a CSI provisioner in response to a PVC. Once matched on capacity and access modes, the PV is bound to the PVC in a 1-to-1 relationship.

• Immediate (legacy default): The storage provisioner creates the cloud disk (e.g. AWS EBS volume) as soon as the PVC is created, picking an arbitrary Availability Zone before the Pod is scheduled. If the scheduler later schedules the Pod in a different AZ, the Pod will fail to start with Multi-Attach error or zone mismatch!
• WaitForFirstConsumer (production best practice): Delays disk provisioning until a Pod using the PVC is actually scheduled on a specific node. Guarantees the volume is provisioned in the exact Availability Zone where the Pod will run.

• ReadWriteOnce (RWO): Can be mounted as read-write by a single node at a time. Multiple pods on the same node can read/write to it (e.g., AWS EBS, GCP Persistent Disk).
• ReadOnlyMany (ROX): Can be mounted read-only by many nodes simultaneously.
• ReadWriteMany (RWX): Can be mounted read-write by multiple pods across multiple nodes simultaneously (e.g., AWS EFS, NFS, CephFS).
• ReadWriteOncePod (RWOP) (K8s 1.22+): Restricts access to a single Pod in the entire cluster, preventing dual-writer split-brain corruption during pod migrations.

1. CSI Controller Plugin: Runs as a Deployment. Interacts with the cloud API (e.g., AWS EC2 API) to create, delete, attach, and detach disk volumes from virtual machines.
2. CSI Node Plugin: Runs as a DaemonSet on every worker node. Responsible for formatting raw block devices with filesystems (ext4/xfs) and mounting the volume into the pod's container directory (NodeStageVolume / NodePublishVolume).
3. Kubernetes CSI Sidecars: Open-source helper containers developed by Kubernetes storage SIG (external-provisioner, external-attacher, external-resizer, node-driver-registrar) that bridge K8s API events to CSI gRPC calls.

To expand a volume, the StorageClass must specify allowVolumeExpansion: true.

The user edits the PVC: kubectl patch pvc data-pvc -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'.

1. The external-resizer sidecar calls the cloud provider to expand the underlying disk.
2. The kubelet on the node resizes the filesystem (resize2fs or xfs_growfs) once the volume is mounted.

Limitations: You can only expand volumes; Kubernetes does not support shrinking storage. EBS volumes have an AWS rate limit of one modification every 6 hours per volume.

Occurs when a Pod using an RWO volume (like EBS) is rescheduled to a new node, but the old node's kubelet did not cleanly unmount and detach the volume from the AWS EC2 instance before the new node attempts to attach it.

Resolution steps: 1. Verify if the pod on the old node is completely terminated: kubectl get pod <name> -o wide. 2. If the old node is NotReady, safely force terminate the dead pod or check VolumeAttachment objects: kubectl get volumeattachment. 3. If necessary, delete the orphaned VolumeAttachment object or detach the volume manually in AWS EC2 Console.

• Delete (default for dynamic StorageClasses): Deleting the PVC automatically triggers the deletion of the associated PV and the underlying cloud disk in AWS/GCP, destroying all data.
• Retain: Deleting the PVC does NOT delete the PV or cloud disk. The PV transitions to Released status. Data remains intact for disaster recovery, but the PV cannot be bound to a new PVC until an administrator manually scrubs the claimRef.

An emptyDir volume is created when a Pod is assigned to a node and exists as long as that Pod runs on that node. It starts empty and is wiped when the Pod is removed.

Caveat: By default, emptyDir writes to the node's root filesystem (/var/lib/kubelet/pods/). Heavy write workloads can consume root disk space, triggering kubelet DiskPressure evictions!
Optimization: Set emptyDir: { medium: Memory } to mount a RAM-backed tmpfs for high-performance caches, but remember it consumes pod memory limits.

VolumeSnapshot (CSI snapshotter) takes a point-in-time storage backup using the storage system's native snapshot mechanism.

To restore, create a new PVC with dataSource pointing to the VolumeSnapshot:

spec:
  dataSource:
    name: db-backup-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  resources:
    requests:
      storage: 100Gi

volumeMounts.subPath allows mounting a specific subdirectory inside a single shared volume rather than mounting the volume root.

Useful when multiple containers in a Pod share a single PersistentVolume, or when an application expects a specific folder name (e.g. /var/lib/mysql/data) but mounting the root volume would overwrite the container image's default directory files.

A Projected Volume maps multiple existing volume sources into the same directory:

• secret
• configMap
• downwardAPI (pod labels, annotations, pod IP)
• serviceAccountToken (time-bound, audience-scoped OIDC tokens)

Enables single-directory configuration projection for complex enterprise applications.

StatefulSets intentionally do NOT delete PVCs generated by volumeClaimTemplates when the StatefulSet or Helm release is deleted. The PVCs and cloud disks remain in Bound status. To recover, simply reinstall the Helm release with the exact same name; the new StatefulSet pods automatically reattach to the existing, un-deleted PVCs.

The 4C model represents defense-in-depth security layers:

1. Cloud: Securing the underlying infrastructure (VPC isolation, IAM roles, security groups, KMS keys).
2. Cluster: Securing Kubernetes components (API server access, RBAC, etcd encryption, audit logging, network policies).
3. Container: Securing container execution (distroless images, vulnerability scanning, non-root users, read-only root filesystems).
4. Code: Securing application logic (SAST/DAST testing, dependency vulnerability scans, TLS termination).

A compromise in an outer layer (e.g., Cloud IAM) invalidates security guarantees in inner layers.

• Role: Namespaced. Grants permissions (verbs on resources) strictly within a single namespace.
• ClusterRole: Non-namespaced. Grants permissions cluster-wide (e.g., viewing Nodes, PVs, or resources across all namespaces).
• RoleBinding: Binds a Role (or ClusterRole) to subjects (Users, Groups, ServiceAccounts) scoped strictly within a specific namespace.
• ClusterRoleBinding: Binds a ClusterRole to subjects across the entire cluster (e.g., cluster-admin).

Prior to Kubernetes 1.24, the API server generated static, everlasting Secret tokens for every ServiceAccount that lasted indefinitely until deleted.

In 1.24+ (Projected ServiceAccount Tokens), kubelet requests time-bound, audience-restricted OIDC tokens directly from the TokenRequest API. The tokens expire automatically (default 1 hour), are rotated automatically by kubelet, and are bound to the specific pod lifecycle—if the pod is deleted, the token is instantly invalid, eliminating credential leakage risk.

Pod Security Standards replace deprecated PodSecurityPolicies (PSP) with three standardized profiles:

1. Privileged: Unrestricted; allows root access, hostPID, hostNetwork. Intended for system daemons.
2. Baseline: Prevents known privilege escalations; blocks host namespaces and privileged capabilities.
3. Restricted: Hardened production profile; requires non-root execution, drops all capabilities except NET_BIND_SERVICE, and enforces read-only root filesystems.

Enforced directly via namespace labels:

kubectl label ns production pod-security.kubernetes.io/enforce=restricted   pod-security.kubernetes.io/warn=restricted

By default, Kubernetes Secrets stored in etcd are merely base64-encoded plain text. If an attacker gains raw etcd disk access, all secrets are exposed.

With KMS Envelope Encryption, the API server uses a local Data Encryption Key (DEK) to encrypt secrets with AES-GCM. The DEK is then encrypted using a Key Encryption Key (KEK) managed by an external Key Management Service (AWS KMS, HashiCorp Vault). Plaintext keys are never persisted to disk.

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL
  seccompProfile:
    type: RuntimeDefault

These guardrails guarantee that even if an attacker executes remote code execution inside the container, they cannot elevate privileges, modify system binaries, or access host kernel primitives.

IRSA eliminates the dangerous practice of assigning IAM permissions to worker node EC2 instances.

1. EKS establishes an OIDC identity provider in AWS IAM.
2. A Kubernetes ServiceAccount is annotated with the IAM Role ARN: eks.amazonaws.com/role-arn: arn:aws:iam::...
3. The pod injects a projected OIDC token and AWS environment variables (AWS_WEB_IDENTITY_TOKEN_FILE).
4. The AWS SDK calls sts:AssumeRoleWithWebIdentity, trading the Kubernetes token for temporary 15-minute AWS credentials scoped strictly to that specific Pod.

Kubernetes API server audit logs record every request across 4 stages: RequestReceived, ResponseStarted, ResponseComplete, and Panic.

Query the audit log (CloudWatch / Elasticsearch) for the HTTP method, URI, and user identity:

jq 'select(.verb == "delete" and .objectRef.resource == "deployments") | {user: .user.username, time: .requestReceivedTimestamp, name: .objectRef.name, namespace: .objectRef.namespace}' /var/log/kubernetes/audit.log

• User Account: Represents human operators or external CI/CD processes. Kubernetes does not have a User database; users are managed externally via X.509 client certs, OpenID Connect (Okta, Google), or webhook authenticators.
• ServiceAccount: Represents in-cluster software agents and pods managed natively via the Kubernetes API server within specific namespaces.

The Linux kernel divides root privileges into ~40 granular capabilities (e.g., CAP_CHOWN, CAP_NET_RAW, CAP_SYS_ADMIN). Standard Docker containers default to granting 14 capabilities. Dropping ALL capabilities strips potential attack vectors (like ARP spoofing or packet sniffing) that an attacker could use if container code is compromised.

• Seccomp (Secure Computing Mode): Restricts which Linux system calls (syscalls) a container process can execute. The RuntimeDefault profile blocks ~50 dangerous syscalls (like reboot, sys_chroot).
• AppArmor / SELinux: Mandatory Access Control (MAC) systems that enforce granular filesystem, network, and process execution permissions on the host operating system.

imagePullSecrets contains authentication credentials to pull images from private registries (AWS ECR, DockerHub, Quay). If omitted, misconfigured, or if credentials have expired, the pod scheduling halts in ImagePullBackOff or ErrImagePull with an HTTP 401 Unauthorized or 403 Forbidden error.

1. Never set privileged: true.
2. Never mount the host Docker socket (/var/run/docker.sock) or host root filesystem.
3. Enforce non-root user execution (runAsNonRoot: true).
4. Keep host kernel and container runtimes (containerd, runc) patched against CVEs.
5. Utilize sandboxed runtimes like gVisor (runsc) or Kata Containers for untrusted multi-tenant workloads.

Compliance standards mandate that production databases and payment processing services must not possess unrestricted internet egress. Applying explicit Egress NetworkPolicies blocks malicious data exfiltration, command-and-control beacons, and unauthorized API calls by whitelisting only essential internal subnets and DNS ports (UDP 53).

Kubernetes assigns QoS classes automatically based on CPU and memory configuration:

1. Guaranteed: Every container in the Pod has both CPU and Memory requests and limits set, and requests == limits for all resources. Highest priority; never evicted unless the node is terminating or out of memory.
2. Burstable: At least one container has requests or limits specified, but requests and limits do not match. Evicted during node memory pressure after BestEffort pods are terminated.
3. BestEffort: No container specifies any CPU or memory requests or limits. First candidates to be killed via OOM killer during node memory pressure.

👉 Resource Deep Dive: Kubernetes Resource Requests, Limits & Scheduling Guarantees.

Kubernetes enforces CPU limits using Linux Completely Fair Scheduler (CFS) bandwidth quotas over 100ms periods (cfs_period_us: 100000). If a container with a limit of 1 CPU executes 100ms of work in the first 20ms of the period, the Linux kernel completely throttles the process for the remaining 80ms!

This causes severe latency spikes and p99 timeouts in multi-threaded runtimes (Go, Java, Node.js). Best practice in modern production: Set CPU requests accurately for scheduling, but remove or dramatically increase CPU limits, relying instead on node autoscaling.

• Taints: Applied to nodes to repel pods unless the pod explicitly has a matching toleration. Syntax: key=value:Effect (NoSchedule, PreferNoSchedule, NoExecute). Used to dedicate nodes to GPU or batch workloads.
• Node Affinity: Configured on pods to attract them to specific nodes based on node labels (e.g. topology.kubernetes.io/zone: us-east-1a). Can be hard (requiredDuringSchedulingIgnoredDuringExecution) or soft (preferredDuringSchedulingIgnoredDuringExecution).

• podAntiAffinity: Prevents pods from running on the same node or availability zone as other pods with matching labels, ensuring high availability across failure domains.
• topologySpreadConstraints: Provides granular control over how pods are distributed across topology keys (zones, hosts, racks). Governed by maxSkew (the maximum allowable difference in pod count between any two zones), preventing cluster imbalances.

HPA v2 queries the custom metrics API (custom.metrics.k8s.io) or external metrics API (external.metrics.k8s.io) served by adapters like Prometheus Adapter or Datadog Metric Provider.

Enables scaling on business-critical telemetry such as SQS queue length, RabbitMQ message backlog, HTTP request rate (RPS), or API response latency rather than lagging CPU metrics.

• Cluster Autoscaler: Tightly coupled to cloud Node Groups / Auto Scaling Groups (ASGs). When a pod is pending, it triggers an ASG scale-out. Slow (takes 2–4 minutes to launch nodes) and bound to predefined EC2 instance types.
• Karpenter: Group-less, dynamic autoscaler developed by AWS. Directly evaluates unscheduled pod resource requests, taints, and architecture constraints, and calls cloud fleet APIs to provision the exact optimal mix of Spot and On-Demand instances in seconds. Consolidates underutilized nodes automatically to slash cloud spend.

VPA observes historical CPU and memory utilization and automatically updates pod resource requests and limits.

Can they run together? Only if HPA scales on custom/external business metrics (like SQS queue depth or HTTP RPS). If both HPA and VPA scale based on CPU/memory, they will create race conditions and thrash pod counts.

• LimitRange: Enforces constraints on individual containers or pods within a namespace (min/max CPU, default requests/limits if omitted by developer).
• ResourceQuota: Enforces hard ceilings on the aggregate resource consumption of an entire namespace (total CPU requests, total memory limits, max number of Pods, Services, or PVCs). Prevents a single team from starving the cluster.

The kubelet continuously evaluates node conditions against eviction thresholds (e.g. memory.available < 100Mi):

• MemoryPressure: Kubelet evicts pods based on their QoS class and memory usage relative to requests (BestEffort killed first, then Burstable exceeding requests).
• DiskPressure: Kubelet first cleans up unused container images and dead containers. If pressure persists, it evicts pods consuming significant local ephemeral storage.

A PriorityClass defines an integer priority (e.g., 1000000 for critical system pods). When cluster resources are exhausted and a high-priority pod is unscheduled, the scheduler initiates preemption: it evicts lower-priority pods from a node to free sufficient CPU and memory for the high-priority pod to schedule.

• Hard eviction threshold (e.g., memory.available < 100Mi): Kubelet immediately evicts pods without waiting for grace periods (sends SIGKILL).
• Soft eviction threshold (e.g., memory.available < 500Mi for 90s): Kubelet monitors the condition for the duration period; if unresolved, it evicts pods honoring their terminationGracePeriodSeconds.

Overcommit occurs when the sum of container resource limits across all scheduled pods exceeds the actual physical capacity of the worker node. While it maximizes resource utilization for bursty workloads, simultaneous traffic spikes across multiple pods will trigger the Linux kernel Out-Of-Memory (OOM) killer, crashing critical application containers.

In soft affinity (preferredDuringSchedulingIgnoredDuringExecution), each preference clause is assigned a weight from 1 to 100. The scheduler iterates through candidate nodes, evaluates matched label expressions, and adds the weight score to each node's total ranking. Nodes with the highest combined score are selected, but scheduling will still succeed on lower-scoring nodes if preferred ones are unavailable.

Prevents metric flapping ('thrashing') where pod count rapidly fluctuates up and down due to bursty traffic. HPA v2 provides behavior.scaleDown.stabilizationWindowSeconds (default 300s / 5 minutes), instructing the controller to evaluate the peak desired replica count observed over the past 5 minutes before executing pod termination.

• Startup Probe: Checks if the application has completed initialization. All other probes are disabled until the startup probe succeeds. Prevents slow-starting legacy apps (like Java JVM) from being prematurely killed by liveness probes.
• Liveness Probe: Checks if the container is healthy and running. If it fails, kubelet terminates the container and restarts it according to restartPolicy.
• Readiness Probe: Checks if the container is ready to accept user network traffic. If it fails, the pod IP is immediately stripped from Service Endpoints and Ingress routing without restarting the container.

The Cascading Outage Trap: If your application's /healthz liveness probe queries an external database, and that database experiences latency or an outage, every single pod in your cluster will fail its liveness probe simultaneously! Kubelet will kill and restart all pods in a catastrophic crash loop, amplifying database connection pressure when it recovers.

Rule of thumb: Liveness probes must only check local in-process deadlocks; only Readiness probes should inspect external dependencies.

metrics-server polls node kubelets over HTTPS (querying the internal cAdvisor summary API) every 60 seconds to retrieve instantaneous CPU and memory metrics. It exposes these metrics via the metrics.k8s.io API for HPA and kubectl top.

It purposefully keeps metrics strictly in volatile memory (RAM) and discards history to maintain lightweight, sub-second response times without thrashing etcd.

Prometheus Operator automates the deployment and management of Prometheus monitoring instances using CRDs:

• ServiceMonitor: Declaratively specifies how groups of Services should be scraped. Matches services using label selectors and defines scrape interval, endpoint path (e.g. /metrics), and port.
• The operator watches ServiceMonitors and dynamically writes scrape configurations into Prometheus configuration files without requiring manual Prometheus restarts.

Containers stream stdout and stderr to the container runtime. The runtime (containerd) writes these streams to JSON or CRI log files located on the node host at /var/log/pods/<pod-name>/<container-name>/*.log.

When you run kubectl logs, kubelet reads these local files and streams them back via the API server. In production, a DaemonSet agent (Fluentbit / Vector) tail-reads these files, enriches logs with pod metadata, and ships them to centralized storage (OpenSearch, Loki).

Writing logs to files inside container filesystems is an anti-pattern: 1. When containers crash or restart, unmounted local container files are lost. 2. It consumes the node root disk filesystem, triggering DiskPressure evictions. 3. kubectl logs cannot read them. Cloud-native best practice: Applications must strictly stream structured JSON logs to stdout/stderr.

OpenTelemetry Collector receives, processes, and exports telemetry (traces, metrics, logs) vendor-neutrally.

Common deployment topologies:

• DaemonSet (Agent): Collects host metrics and receives OTLP traces from pods over localhost:4317.
• Deployment (Gateway): Centralized collector pool handling batching, sampling, credential enrichment, and forwarding to storage backends (Jaeger, Datadog).

CoreDNS exports native Prometheus metrics on port 9153:

• coredns_dns_request_duration_seconds_bucket: Measures DNS query response latency.
• coredns_dns_responses_total{rcode="SERVFAIL"}: Measures server failure responses.
• coredns_cache_hits_total vs coredns_cache_misses_total: Evaluates DNS cache hit ratio.

When a request spans multiple microservices in Kubernetes, context propagation injects W3C TraceContext HTTP headers (traceparent, tracestate) into all outgoing HTTP/gRPC client calls. Allows APM tools to visualize the end-to-end distributed transaction as a single unified trace waterfall.

Audit logs support 4 levels: None, Metadata, Request, and RequestResponse.

Production best practice: Set None for noisy high-frequency read endpoints (endpoints, kube-proxy token leases), Metadata for routine read requests, and RequestResponse for all mutations on critical resources (Secrets, Roles, RoleBindings, Nodes).

Follow this deterministic runbook:

1. Inspect Exit Code & Reason: kubectl describe pod <pod> → Check Last State: Terminated, Exit Code, and Reason.
2. Read Crash Logs: kubectl logs <pod> --previous (reads logs from the crashed container before restart).
3. Check Exit Code Meanings:
   • Exit Code 0: Application finished normally but shouldn't have (missing foreground loop).
   • Exit Code 1/2: Application code exception, missing configuration, syntax error.
   • Exit Code 137: OOMKilled (Killed by Linux SIGKILL signal 9, 128+9).
   • Exit Code 139: Segmentation fault (SIGSEGV signal 11, 128+11).
   • Exit Code 143: Graceful termination via SIGTERM (signal 15, 128+15).
4. Verify Dependencies: Test database connectivity, Vault tokens, and external secrets.
5. Check Liveness Probe: Determine if an aggressive liveness probe is killing the container before startup.
6. Interactive Debugging: Override entrypoint with sleep 3600 to inspect container environment locally.

👉 Live Scenario Runbook: CrashLoopBackOff Diagnostic & Root Cause Workflow.

• Cgroup Limit OOMKilled: The container exceeded its Kubernetes memory limit. The Linux kernel kills the individual container process. The host node remains completely healthy and unimpacted. kubectl describe pod shows Reason: OOMKilled.
• Host Node OOM: The entire worker node ran out of physical memory (due to overcommit or BestEffort pods). The kernel OOM killer activates globally and terminates processes with the highest oom_score_adj to protect the kernel, often evicting pods without recording clean Kubernetes events.

Check kubectl describe pod <pod> under Events:

1. Insufficient Node Resources: 0/12 nodes are available: 12 Insufficient cpu/memory → Scale node group or adjust pod requests.
2. Unsatisfied NodeAffinity / NodeSelector: No node matches the pod's label constraints.
3. Taints and Tolerations: Nodes have taints that the pod cannot tolerate.
4. Unbound PVC: pod has unbound immediate PersistentVolumeClaims → StorageClass or CSI driver issue.
5. Node Port Conflicts: Multiple pods attempting to bind to the same host port.

👉 Complete Runbook: Kubernetes Pod Stuck in Pending: 6 Common Root Causes & Step-by-Step Triage.

1. Run kubectl describe pod <pod> | grep -A 5 -i "Failed to pull image".
2. Typo in image tag or registry name: Verify image tag exists in repository.
3. Private registry authentication failure: Missing or expired imagePullSecrets or incorrect IAM role permissions for ECR.
4. Docker Hub Rate Limiting: Exceeded 100 pulls/6 hours anonymously (HTTP 429 Too Many Requests). Fix: Authenticate with DockerHub or mirror to AWS ECR.

A node is marked NotReady when the node-controller has not received a heartbeat from the node's kubelet for node-monitor-grace-period (default 40s).

First 3 commands:

# 1. Inspect node events and conditions
kubectl describe node <node-name>

# 2. Check if kubelet systemd service is active on host
ssh <node-ip> "systemctl status kubelet"

# 3. Inspect recent kubelet error logs
ssh <node-ip> "journalctl -u kubelet -e --no-pager -n 50"

Common causes: Kubelet crash, OOM, disk space exhausted on /var/lib/docker, or network partition.

Follow this production sequence:

1. Pre-upgrade audit: Check deprecated API versions using pluto or kubent.
2. Upgrade Control Plane: Update EKS cluster version via AWS Console / Terraform. EKS upgrades the managed control plane nodes one by one.
3. Upgrade Core Add-ons: Upgrade vpc-cni, kube-proxy, and coredns to match the target Kubernetes minor version.
4. Upgrade Worker Nodes: Provision a new Node Group with the updated AMI. Cordon the old node group (kubectl cordon) and drain nodes sequentially (kubectl drain --ignore-daemonsets --delete-emptydir-data), respecting PodDisruptionBudgets.
5. Post-upgrade validation: Verify all services, endpoints, and ingress routing are functional before terminating old node groups.

👉 Step-by-Step Architecture: Zero-Downtime Amazon EKS Minor & Multi-Version Upgrade.

# 1. Take etcd snapshot
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key   snapshot save /tmp/etcd-backup-$(date +%F).db

# 2. Restore etcd snapshot to a new data directory
ETCDCTL_API=3 etcdctl --data-dir=/var/lib/etcd-restored   snapshot restore /tmp/etcd-backup-2026-10-12.db

# 3. Update static pod manifest (/etc/kubernetes/manifests/etcd.yaml) to point to /var/lib/etcd-restored

Split-brain occurs when a network partition divides a cluster into two isolated subsets, and both subsets believe they are the legitimate leader, accepting conflicting writes.

etcd completely prevents split-brain by requiring an absolute majority quorum (N/2)+1 for leader election and write commits. In a partition, only the subset containing more than 50% of members can elect a leader and accept writes; the minority subset refuses writes, preserving data integrity.

• kubectl drain <node>: Cordons the node (marks it unschedulable), contacts the Eviction API, honors PodDisruptionBudgets, invokes preStop hooks, and gives applications up to their termination grace period to terminate gracefully.
• Forced termination (e.g. terminating AWS EC2 instance directly): Bypasses Kubernetes coordination. Pods immediately drop connections without warning, risking in-flight database transaction corruption and transient 502 Bad Gateway errors for end users.

All internal service discovery and external domain resolution halts. Microservices attempting to connect to other services via DNS names (payment-svc.prod.svc.cluster.local) or external APIs (api.stripe.com) will timeout immediately.

Running pods with established keep-alive TCP connections continue functioning, but any new connection or lookup fails. Mitigation: Run CoreDNS as an autoscaled deployment with at least 3 replicas spread across distinct AZs, with PDB minAvailable: 2, and deploy NodeLocal DNSCache.

A deadlock occurs when a mutating or validating webhook is configured with failurePolicy: Fail, but the webhook service itself crashes or cannot schedule pods (e.g., after cluster reboot). Because the API server cannot reach the webhook, it rejects all incoming pod creations—including the pods needed to run the webhook!

Recovery:

# Temporarily disable the failing validating webhook configuration
kubectl get validatingwebhookconfigurations
kubectl delete validatingwebhookconfiguration <failing-webhook>

Execute this systematic triage sequence:

1. Check CPU Throttling: Inspect Prometheus metric container_cpu_cfs_throttled_periods_total.
2. Check CoreDNS Latency: Measure DNS lookup times inside pods using dig or dnsping.
3. Check conntrack tables: Verify whether nodes are reaching nf_conntrack_max limits.
4. Check CNI / Network interface drops: Run ethtool -S eth0 on worker nodes to inspect packet drops and ring buffer overruns.
5. Check Ingress keep-alive connection reuse: Ensure ingress controllers reuse backend connections rather than establishing new TCP/TLS handshakes on every request.

👉 Detailed Guide: Cluster Intermittent Pod Failures & High Latency Systematic Troubleshooting.

Never treat Kubernetes as a magic black box: Every high-level abstraction (Pod, Service, Ingress, Volume) compiles down to deterministic Linux kernel primitives (cgroups, network namespaces, iptables/eBPF, and filesystem mounts). When production fails at 2 AM, the engineer who understands these fundamental Linux primitives will diagnose and restore the system in minutes, while others stare helplessly at YAML manifests.