EKS Troubleshooting Cheat Sheet

First PublishedApr 24, 2026Last UpdatedMay 18, 2026ByAtif Alam

Symptom-driven debugging reference for EKS SRE work. Focus areas: networking and autoscaling.

---

Universal First Moves

Before diving into any specific symptom, run these:

# Pod-level
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous          # last crash output
kubectl logs <pod> -n <ns> -c <container> -f   # multi-container

# Cluster-level events (most recent)
kubectl get events -A --sort-by='.lastTimestamp' | tail -50

# Node health
kubectl get nodes -o wide
kubectl describe node <node> | grep -A5 Conditions

# Quick resource view
kubectl top nodes
kubectl top pods -A --sort-by=memory

Drop into a debug pod for in-cluster network testing:

kubectl run debug --rm -it --image=nicolaka/netshoot -- bash

For a stuck or dying container, use ephemeral debug containers:

kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container> -n <ns>

---

Symptom 1: Pod Stuck in Pending

kubectl describe pod <pod> -n <ns>             # check Events at the bottom
kubectl get events -n <ns> --sort-by='.lastTimestamp'

Event message	Cause	Fix
Insufficient cpu/memory	No node has capacity	Scale nodes or reduce requests
node(s) had untolerated taint	Taint mismatch	Add toleration or remove taint
node(s) didn't match node selector	Label/affinity mismatch	Check nodeSelector, nodeAffinity
pod has unbound immediate PVCs	PVC not bound	Check StorageClass and PV availability
0/N nodes are available	All filtered out	Read full message, usually a combination of constraints
No events / scheduler silent	Karpenter or CAS not provisioning	See Autoscaling section below

EKS-specific check for IP exhaustion that appears as Pending:

kubectl logs -n kube-system -l k8s-app=aws-node --tail=100 | grep -i "no available IP"

---

Symptom 2: CrashLoopBackOff

kubectl logs <pod> -n <ns> --previous
kubectl describe pod <pod> -n <ns> | grep -A5 "Last State"

Exit code	Meaning
0	Clean exit, but Kubernetes expected long-running process
1	Generic application error, inspect app logs
137	OOMKilled (SIGKILL)
139	Segfault (SIGSEGV)
143	SIGTERM (often shutdown exceeded grace period)

Liveness probe killing healthy pods:

kubectl describe pod <pod> -n <ns> | grep -A3 Liveness
# Look for "Liveness probe failed" in events

Common fix: increase initialDelaySeconds, raise failureThreshold, or add startupProbe for slow-booting apps.

---

Symptom 3: ImagePullBackOff (EKS)

kubectl describe pod <pod> -n <ns> | grep -A3 -i "failed to pull"

EKS-specific causes (in typical frequency order):

1. Node IAM role missing ECR permissions (AmazonEC2ContainerRegistryReadOnly)
2. Cross-account ECR pull missing source repo policy
3. Architecture mismatch (amd64 image on arm64 node, or reverse)
4. Docker Hub rate limit
5. Missing imagePullSecret in the same namespace

# Verify ECR auth from a node
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account>.dkr.ecr.<region>.amazonaws.com

---

Symptom 4: Service Not Reachable

Endpoints check (always step 1):

kubectl get endpoints <svc> -n <ns>
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>

If endpoints are empty:

• Selector does not match pod labels -> kubectl get pods -n <ns> --show-labels
• Pods are not Ready -> endpoints include only ready pods

In-cluster DNS test:

kubectl run debug --rm -it --image=nicolaka/netshoot -- bash
# Inside:
nslookup <svc>.<ns>.svc.cluster.local
curl -v http://<svc>.<ns>.svc.cluster.local:<port>

CoreDNS checks:

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
kubectl get cm coredns -n kube-system -o yaml

---

Symptom 5: Node NotReady

kubectl describe node <node> | grep -A10 Conditions

Condition quick meanings:

• MemoryPressure: eviction risk
• DiskPressure: disk cleanup / image GC pressure
• PIDPressure: process count pressure
• NetworkUnavailable: CNI not initialized

Get onto the node (EKS):

aws ssm start-session --target <instance-id>
sudo journalctl -u kubelet -f
sudo crictl ps -a
sudo crictl logs <container-id>

EKS join failures (node never becomes Ready):

• Check /var/log/cloud-init-output.log
• Verify aws-auth ConfigMap mapping (legacy) or EKS Access Entries (new model)
• Check VPC CNI pod startup and node/control-plane SG rules

---

Networking Deep Dive (EKS)

VPC CNI Mental Model

AWS VPC CNI assigns each pod a real VPC IP via node ENIs:

• Pods become first-class VPC IPs
• Pod density is limited by instance ENI/IP limits
• Subnet IP planning directly controls pod scale

Check current IP allocation:

kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, allocatable_pods: .status.allocatable.pods}'

# On aws-node pod:
kubectl exec -n kube-system <aws-node-pod> -- /app/grpc-health-probe -addr=:50051
kubectl logs -n kube-system <aws-node-pod> | grep -i "ip address"

IP Exhaustion Playbook

Symptom: pods stuck in ContainerCreating with CNI errors, or no assignable pod IPs.

kubectl describe pod <pod> -n <ns> | grep -i "failed to assign"
kubectl logs -n kube-system -l k8s-app=aws-node --tail=200 | grep -iE "ip|eni|exhaust"

Remediations (prefer in this order):

1. Prefix delegation on supported instances:

   kubectl set env ds aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

2. Custom networking (pod subnets via ENIConfig):

   kubectl set env ds aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

3. Add secondary VPC CIDR for pod subnets
4. Switch to an overlay-based approach if VPC-native pod IP is not required

Pod-to-Pod Traffic Path (Cross-Node)

Pod A (eth0) -> veth -> host namespace
             -> host route table / ENI routing
             -> node A ENI -> VPC routing -> node B ENI
             -> host route -> pod B veth -> Pod B (eth0)

No overlay and no pod-to-pod NAT in the default VPC CNI path, so VPC Flow Logs show pod IPs.

kube-proxy Modes

kubectl get cm kube-proxy-config -n kube-system -o yaml | grep mode

• iptables: default, rule-walk based
• ipvs: hash lookup model, better at larger service counts
• eBPF datapath (for example Cilium replacing kube-proxy): improved performance/visibility trade-offs

LoadBalancer / Ingress Issues (AWS Load Balancer Controller)

kubectl logs -n kube-system deployment/aws-load-balancer-controller --tail=200
kubectl describe svc <svc> -n <ns>
kubectl describe ingress <ing> -n <ns>

Symptom	Likely cause
ALB not provisioning	Missing subnet tags (kubernetes.io/role/elb or kubernetes.io/role/internal-elb)
Targets unhealthy	SG rules or readiness path mismatch
ALB 504	ALB idle timeout lower than backend response time
Wrong target type	instance mode hairpins via NodePort; use ip mode where supported

Subnet tag checklist:

aws ec2 describe-subnets --subnet-ids <id> --query 'Subnets[].Tags'

Required tags:

• kubernetes.io/cluster/<cluster-name> = shared|owned
• kubernetes.io/role/elb = 1 (public)
• kubernetes.io/role/internal-elb = 1 (internal)

NetworkPolicy on EKS

VPC CNI by itself does not always enforce Kubernetes NetworkPolicy semantics. Common options:

• Calico policy-only mode with VPC CNI
• VPC CNI native Network Policy feature (where enabled)
• Cilium for broader L3-L7 policy

kubectl run test --rm -it --image=nicolaka/netshoot -n <restricted-ns> -- curl <target>

---

Autoscaling Deep Dive

For concepts, install paths, and metric pipelines, see Autoscaling on EKS. Custom metrics: Prometheus Adapter for HPA, Container Insights for HPA.

Three layers that are orthogonal:

HPA / VPA / KEDA       -> pod count and size
CAS / Karpenter        -> node count and type
EC2 / Fargate          -> underlying compute supply

HPA (Horizontal Pod Autoscaler)

kubectl get hpa -A
kubectl describe hpa <name> -n <ns>

Debug “HPA not scaling”:

Symptom in describe	Likely cause
unable to get metrics	metrics-server unavailable
current: <unknown>	missing pod resource requests
desired equals current under load	target threshold/metric mismatch
scaling thrash	behavior windows not tuned

kubectl top pods -A
kubectl get apiservice v1beta1.metrics.k8s.io -o yaml

For queue/event-driven scaling, KEDA is commonly simpler than custom adapter plumbing. Metric install guides: Prometheus Adapter, Container Insights.

VPA (Vertical Pod Autoscaler)

Modes:

• Off: recommendations only
• Initial: set on pod creation
• Auto: can evict/recreate for resize

Avoid running HPA and VPA on the same signal (for example both on CPU).

Cluster Autoscaler vs Karpenter

	Cluster Autoscaler	Karpenter
Provisioning unit	ASG/node groups	direct EC2 launches
Instance selection	pre-defined by group	per-pod-fit selection
Scale-up speed	usually slower	usually faster
Best fit	predictable regulated environments	dynamic cost-optimized environments

Karpenter Debugging

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --tail=200
kubectl get nodepool
kubectl get ec2nodeclass
kubectl get nodeclaim

If pods are Pending and not provisioning:

1. Check NodePool requirements and matching
2. Check pod constraints (nodeSelector, taints, topology)
3. Check EC2 quotas
4. Check subnet IP capacity
5. Check NodeClass IAM/permissions and events

kubectl describe pod <pending-pod>

Consolidation blockers commonly include restrictive PDBs, non-evictable pods, and long termination windows.

Cluster Autoscaler Debugging

kubectl logs -n kube-system deployment/cluster-autoscaler --tail=200
kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

Common gotchas:

• ASG max size reached
• mixed instance simulation mismatch
• missing autoscaler discovery tags
• scale-down blocked by local-storage pods, restrictive PDBs, or safe-to-evict: "false"

Spot Interruption Handling

• Karpenter-managed environments: native interruption handling and drain path
• CAS-based environments: verify Node Termination Handler daemonset

kubectl get ds -n kube-system aws-node-termination-handler

---

Symptom 6: Intermittent 5xx (Layered Approach)

# 1. Are specific pods unhealthy?
kubectl get pods -n <ns> -o wide
kubectl top pods -n <ns>

# 2. CPU throttling
kubectl exec <pod> -n <ns> -- cat /sys/fs/cgroup/cpu.stat | grep throttled

# 3. Recent restarts
kubectl get pods -n <ns> -o json | jq '.items[] | {name: .metadata.name, restarts: [.status.containerStatuses[].restartCount]}'

# 4. ALB target health
aws elbv2 describe-target-health --target-group-arn <arn>

# 5. Correlate with deploys
kubectl rollout history deployment/<name> -n <ns>

---

Symptom 7: IRSA Not Working

# Service account annotation
kubectl get sa <name> -n <ns> -o yaml | grep eks.amazonaws.com/role-arn

# Pod service account usage
kubectl get pod <pod> -n <ns> -o yaml | grep serviceAccountName

# Projected token / env
kubectl describe pod <pod> -n <ns> | grep -A3 "aws-iam-token"
kubectl exec <pod> -n <ns> -- env | grep -E "AWS_ROLE_ARN|AWS_WEB_IDENTITY_TOKEN_FILE"

# Effective identity
kubectl exec <pod> -n <ns> -- aws sts get-caller-identity

Trust policy must match exact OIDC provider, service account subject, and audience:

{
  "Effect": "Allow",
  "Principal": {
    "Federated": "arn:aws:iam::ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/CLUSTER_ID"
  },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "oidc.eks.REGION.amazonaws.com/id/CLUSTER_ID:sub": "system:serviceaccount:NAMESPACE:SA_NAME",
      "oidc.eks.REGION.amazonaws.com/id/CLUSTER_ID:aud": "sts.amazonaws.com"
    }
  }
}

Alternative model to know: EKS Pod Identity.

---

Symptom 8: API Server / Control Plane Slow

Control plane is managed in EKS, but workload behavior can still overwhelm API usage.

If enabled, inspect EKS CloudWatch control-plane logs:

• audit
• api
• authenticator
• controllerManager
• scheduler

In-cluster API metrics samples:

kubectl get --raw /metrics | grep apiserver_request_duration_seconds | grep -v "#"
kubectl get --raw /metrics | grep apiserver_request_total | sort -t'"' -k4 | tail -20

Frequent stressors:

• high-churn operators watching broad scope
• oversized ConfigMaps or Secrets
• many CRDs with expensive reconciliation loops

---

3 AM Incident Playbook

1. Acknowledge and declare incident ownership
2. Establish blast radius (namespace/service/customer impact)
3. Check recent changes quickly:

   kubectl rollout history deployment -A | grep -v REVISION

4. Check AWS Service Health
5. Roll back suspicious changes rapidly:

   kubectl rollout undo deployment/<name> -n <ns>

6. Communicate status every 15 minutes
7. Capture evidence for post-incident analysis

---

Quick Reference: High-Frequency Commands

kubectl get events -A --sort-by='.lastTimestamp' | tail -50
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous
kubectl logs <pod> -n <ns> -f --tail=100
kubectl get pods -n <ns> -o wide
kubectl get endpoints <svc> -n <ns>
kubectl top pods -A --sort-by=memory
kubectl top nodes
kubectl describe node <node>
kubectl get hpa -A
kubectl describe hpa <name> -n <ns>
kubectl rollout status deployment/<name> -n <ns>
kubectl rollout history deployment/<name> -n <ns>
kubectl rollout undo deployment/<name> -n <ns>
kubectl exec -it <pod> -n <ns> -- sh
kubectl debug -it <pod> -n <ns> --image=nicolaka/netshoot --target=<container>
kubectl run debug --rm -it --image=nicolaka/netshoot -- bash
kubectl get nodepool
kubectl get nodeclaim
kubectl logs -n kube-system -l k8s-app=aws-node
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter