Diagnosing Failed or Unstable EKS Request Paths

Welcome back. In the previous lesson, you designed a highly available ECS service and treated health checks, load balancing, capacity headroom, and graceful draining as operational controls—not deployment details. EKS has the same reliability objectives, but an incident has more independently moving control loops: Kubernetes workload controllers, kubelet probes, Services and EndpointSlices, an Ingress controller, the AWS Load Balancer Controller, autoscalers, and node capacity.

This lesson gives you a disciplined way to diagnose an EKS request path when an exchange-facing API is failing outright, intermittently returning errors, or becoming unstable during scale-out and deployment. By the end, you should be able to state where the request is failing, gather evidence before changing anything, and explain the safe mitigation.

---

Begin with the request path, not with a favorite tool

Suppose the order-api service has elevated 5xx responses and p99 latency. Some requests succeed, but failures spike after a deployment or traffic increase. The request path may include:

1. A client reaches an ALB or NLB created or managed for the EKS application.
2. An Ingress rule matches the host and path, then names a Kubernetes Service and service port.
3. The Service selects eligible Pods using labels and EndpointSlices.
4. The traffic reaches a Pod, either directly or through node-level Service routing, depending on the load balancer target type and cluster networking.
5. The application accepts the request and reaches its dependencies.

The key diagnostic principle is:

Test the lowest viable layer first, then move outward until the failure appears.

Do not begin by editing an Ingress annotation, restarting all Pods, or scaling the node group. Those actions can erase evidence and turn a narrow failure into a wider outage.

A senior incident response starts by classifying the blast radius:

Observed symptom	Initial interpretation
All external requests fail, but in-cluster calls succeed	Focus on the load balancer, Ingress configuration/controller, target health, security groups, DNS, or public routing.
External and internal Service calls fail, but direct Pod calls work	Focus on the Service, EndpointSlices, network policy, or cluster dataplane.
Only some requests fail	Compare healthy and failing Pods, nodes, AZs, target health, application versions, and connection behavior.
Failures begin during a rollout	Suspect readiness, target registration, termination draining, replica availability, or deployment settings.
Pods are Pending during demand growth	Suspect HPA configuration, resource requests, node capacity, taints, affinity, subnet IP capacity, or pod-density limits.
Pods are Running but 0/1 Ready	Start with readiness probes, startup dependencies, events, logs, and resource throttling.

Before deeper investigation, establish two things:

• Customer impact: error rate, latency, affected API routes, affected tenants, and whether the problem is regional, AZ-specific, or version-specific.
• Recent change: deployment revision, HPA adjustment, node group rollout, security-policy update, DNS or Ingress change, dependency incident, certificate rotation, or AWS control-plane event.

For an order path, safe mitigation may mean pausing a faulty rollout, shifting traffic to healthy replicas, restoring a previously known-good immutable image, or temporarily increasing already-proven capacity. It does not mean deleting Pods randomly until the dashboard looks better.

---

First evidence: workload state, events, and probes

Start at the workload level. A Deployment can report a desired replica count while the actual serving population is much smaller. The distinction among desired, current, available, and ready replicas matters.

NAMESPACE=trading
APP=order-api

kubectl -n $NAMESPACE get deployment $APP
kubectl -n $NAMESPACE get replicasets -l app=$APP
kubectl -n $NAMESPACE get pods -l app=$APP -o wide
kubectl -n $NAMESPACE get events --sort-by=.lastTimestamp

Then inspect an affected Pod rather than only the Deployment summary:

kubectl -n $NAMESPACE describe pod <pod-name>
kubectl -n $NAMESPACE logs <pod-name> --all-containers=true
kubectl -n $NAMESPACE logs <pod-name> --previous --all-containers=true

kubectl describe pod and the event stream are usually the fastest way to identify the category of failure.

Interpret Pod states precisely

Pending with no assigned node

The scheduler has not found a valid placement. Common event messages include FailedScheduling and explanations such as:

• Insufficient requested CPU or memory.
• A taint has no matching toleration.
• Node affinity or pod anti-affinity excludes all available nodes.
• A node selector matches no nodes.
• The node has reached its maximum Pod count.
• A persistent volume constraint cannot be met.

The critical detail is that the scheduler uses resource requests, not observed average utilization. A container consuming little CPU can still be unschedulable if its declared request does not fit.

Pending after a node has been assigned

The Pod may have a node name but be stuck in ContainerCreating, Init:0/1, or a related state. Investigate:

• Missing ConfigMaps, Secrets, volumes, or service-account dependencies.
• Image pull failures and registry access.
• Init containers that have not completed.
• CNI networking or Pod IP allocation failures.
• Container runtime or node-level issues.

CrashLoopBackOff

The application is starting and terminating repeatedly. Use current and previous logs, inspect the exit code, and check whether an application configuration change, missing secret, failed dependency initialization, or an OOM kill is involved.

Running but not Ready

This is one of the most important production states. A running process is not necessarily safe to receive traffic. The readiness probe may be failing because:

• The application has not completed initialization.
• It cannot establish a required database, cache, or message-broker connection.
• A dependency is slow or unavailable.
• The probe path, port, host header, TLS expectation, or authentication requirement is wrong.
• The application is overloaded and probe requests are timing out.
• A recent change altered the health endpoint’s behavior.

Keep the three probe roles separate

A strong interview answer distinguishes the probes rather than calling all of them “health checks.”

Probe	Question it answers	Failure effect
Startup probe	Has a slow-starting application completed initialization?	Prevents liveness and readiness checks from acting too early.
Readiness probe	Can this Pod safely receive new requests now?	Removes the Pod from Service endpoints when it fails. It does not inherently restart the container.
Liveness probe	Is the process irrecoverably unhealthy, deadlocked, or stuck?	Kubernetes restarts the container after repeated failures.

For an order API, /livez should normally be cheap and local. It should not fail merely because an optional downstream dependency is slow; otherwise, a dependency incident can trigger a restart storm.

/readyz can test conditions genuinely required to accept work: completed initialization, loaded configuration, and usable connections to critical dependencies. But do not make it an expensive synthetic order flow or a broad dependency fan-out. A readiness check that overloads dependencies during an incident becomes part of the failure.

Watch this practical troubleshooting segment before proceeding. It is useful for reinforcing how status, events, logs, resource requests, probes, Services, and network components narrow the search space.

---

Prove the Service can reach the right Pods

Once you know whether Pods are actually healthy, verify that Kubernetes can route to them. A Service is a stable virtual address plus a selector and port mapping; it is not proof that backends exist.

kubectl -n $NAMESPACE get service $APP -o yaml
kubectl -n $NAMESPACE describe service $APP

kubectl -n $NAMESPACE get pods -l app=$APP --show-labels
kubectl -n $NAMESPACE get endpointslice \
  -l kubernetes.io/service-name=$APP -o yaml

Check four fields carefully:

1. Service selector: Does spec.selector match the labels on the actual Pods?
2. Service port: Is the client-facing Service port correct?
3. Target port: Does targetPort match the application port or the correctly named container port?
4. EndpointSlices: Do they contain the expected Pod IPs and ports, with endpoints marked ready?

An empty EndpointSlice is usually not a network problem. It commonly indicates a selector mismatch, Pods that are not Ready, or no running replicas. A selector typo can make an apparently healthy Service route to nobody.

A reliable isolation procedure tests from inside the cluster. Use an approved diagnostic image and namespace for production environments; do not normalize unrestricted debugging containers in sensitive namespaces.

kubectl -n $NAMESPACE run net-debug \
  --rm -it --restart=Never \
  --image=registry.k8s.io/e2e-test-images/agnhost:2.45 \
  -- sh

From that shell, test the Service DNS name and Service port:

nslookup order-api
wget -S -O- http://order-api:8080/readyz

If required, use the namespace-qualified name:

nslookup order-api.trading
wget -S -O- http://order-api.trading.svc.cluster.local:8080/readyz

Interpret the outcome carefully:

• DNS fails: investigate CoreDNS availability, namespace naming, the Pod resolver configuration, and DNS-related network policy.
• DNS works but the Service request fails: inspect Service ports, EndpointSlices, network policies, and the Service dataplane.
• Service works but one direct Pod request fails: compare that Pod’s logs, probe status, node, image digest, resource usage, and dependency connectivity with a healthy Pod.
• Direct Pod tests all work but the Service fails: the failure is between the Service abstraction and its endpoints, or in the underlying network dataplane.
• Service works from inside the cluster but not externally: move outward to Ingress and the AWS load balancer layer.

The Kubernetes documentation’s service-debugging approach is particularly useful because it forces this isolation instead of treating “Service unavailable” as a single generic fault.

---

Diagnose Ingress and AWS load-balancer behavior

An Ingress resource is a routing declaration, not the data plane itself. It needs an Ingress controller to interpret it. On EKS, a common design uses the AWS Load Balancer Controller to reconcile Ingress and Service configuration into an Application Load Balancer and target groups.

Inspect the Kubernetes objects first:

kubectl -n $NAMESPACE get ingress
kubectl -n $NAMESPACE describe ingress order-api
kubectl get ingressclass

kubectl -n kube-system get deployment aws-load-balancer-controller
kubectl -n kube-system get pods -l app.kubernetes.io/name=aws-load-balancer-controller
kubectl -n kube-system logs deployment/aws-load-balancer-controller --tail=200

For an Ingress incident, validate:

• The expected host and path rule match the incoming request.
• The ingressClassName is correct.
• The backend Service name and port match the actual Service.
• The controller is healthy and has IAM permission to reconcile AWS resources.
• The ALB listener and listener rules correspond to the intended host/path routing.
• The target group has healthy targets in the expected AZs.
• The ALB security group permits the intended client traffic.
• The target security group or Pod security group permits traffic from the ALB.
• The ALB subnets and route tables support the intended internet-facing or internal design.

Target type changes the request path

With the AWS Load Balancer Controller, the load balancer’s target type materially changes both performance and troubleshooting.

Instance target type registers worker-node IP addresses and NodePorts. Traffic reaches a node and then traverses node-level Service routing before reaching a selected Pod. This introduces more hops and can send traffic across nodes or Availability Zones.

IP target type registers Pod IPs directly. The ALB health check reaches the Pod directly, which makes target health a much closer representation of the application’s actual serving state. It also removes node-level Service routing from the north-south request path.

For an HTTP EKS API, IP targets are generally easier to reason about operationally. However, remember the nuance:

• For external traffic using IP targets, the ALB can deliver directly to a registered Pod IP.
• For internal traffic to a ClusterIP Service, Kubernetes still uses its Service dataplane and EndpointSlices.

That distinction explains a common incident pattern: internal Service tests can be healthy while external clients receive ALB 5xx errors, or the reverse.

Read the following AWS EKS guidance now. It directly connects target-type choice, readiness propagation, lifecycle behavior, and voluntary disruption controls.

Deployment-only failures: readiness gates and graceful termination

A particularly senior-level diagnosis is recognizing the mismatch between Kubernetes readiness and ALB target health during a rollout.

Without an appropriate readiness gate:

1. A new Pod passes its Kubernetes readiness probe.
2. Kubernetes marks it Ready and may allow the Deployment to terminate an older replica.
3. The AWS Load Balancer Controller is still registering the new Pod target and waiting for ELB health checks.
4. The external load balancer temporarily has fewer healthy targets than Kubernetes believes exist.

Under a tight replica count, this can produce short external request drops even while kubectl get pods looks acceptable.

Use Pod readiness gates with the AWS Load Balancer Controller where applicable. They delay the Pod’s overall Ready condition until the relevant target group reports it healthy, reducing the discrepancy between the Kubernetes and ALB views of availability.

Termination has the reverse problem. Kubernetes may start terminating a Pod before all data-plane layers stop sending it new traffic. A robust service should:

• Receive SIGTERM and stop accepting new work.
• Fail readiness quickly.
• Continue serving legitimate in-flight requests during the termination grace period.
• Close idle and new connections safely.
• Exit before terminationGracePeriodSeconds expires.
• Align ALB deregistration delay with the maximum safe request or connection duration.

A short PreStop delay can sometimes give endpoint and load-balancer deregistration time to propagate before the application begins shutdown. It is a mitigation for propagation timing, not a substitute for graceful application behavior.

---

Autoscaling and node signals: determine why capacity did not arrive

When a traffic spike causes failures, distinguish between workload scaling and cluster capacity scaling.

The Horizontal Pod Autoscaler, or HPA, adjusts the desired replica count of a workload. It does not create EC2 instances. The Cluster Autoscaler, or another node-provisioning mechanism, adds nodes only when Pods are unschedulable due to capacity or placement constraints.

Inspect both scaling layers:

kubectl -n $NAMESPACE get hpa
kubectl -n $NAMESPACE describe hpa order-api

kubectl -n $NAMESPACE get deployment order-api
kubectl -n $NAMESPACE get pods -l app=order-api -o wide
kubectl -n $NAMESPACE get events --sort-by=.lastTimestamp

HPA diagnostic questions

Ask:

• Is the HPA targeting the correct Deployment?
• Are the desired replicas increasing as demand rises?
• Are CPU or memory utilization metrics available?
• Are the workload’s CPU and memory requests meaningful?
• Is the chosen scaling signal related to the bottleneck?

CPU-based HPA requires resource requests to calculate utilization meaningfully. For an API path, request rate, concurrency, queue depth, or latency-aware custom metrics may be more representative than CPU alone. CPU can remain moderate while connection pools, thread pools, or downstream rate limits are saturated.

Also look for the dangerous pattern where HPA has increased desired replicas but the new Pods remain Pending. That means application-level scaling is working while cluster-level capacity is not.

Node-level evidence

kubectl get nodes -o wide
kubectl describe node <node-name>
kubectl top nodes
kubectl top pods -A --sort-by=memory

Look for these node conditions and signals:

Node signal	What it suggests
NotReady	Kubelet, network connectivity, node bootstrapping, or underlying instance failure.
MemoryPressure	Risk of eviction, OOM conditions, and unstable Pods.
DiskPressure	Image garbage collection, ephemeral storage, logs, or node filesystem exhaustion.
PIDPressure	Process exhaustion; containers may fail to start or behave unpredictably.
High requested CPU or memory despite low observed use	Requests have consumed schedulable capacity; the scheduler may correctly refuse more Pods.
Maximum Pods reached	EKS node ENI/IP or configured Pod-density limit is exhausted.
Repeated evictions	Capacity pressure, ephemeral-storage exhaustion, or a node that should be drained and investigated.

In EKS, insufficient Pod IP capacity can look deceptively like a Kubernetes application failure. With the Amazon VPC CNI, Pod networking consumes VPC address capacity. A node may have spare CPU but be unable to host more Pods because it has reached its supported Pod count or cannot allocate additional network interfaces/IPs. Subnet capacity and node instance type are therefore scaling dependencies.

For an unschedulable Pod, the event message is more valuable than guessing:

kubectl -n $NAMESPACE describe pod <pending-pod>

If the event says no node matches affinity, a larger node group will not fix the issue. If it says insufficient memory, adding suitable node capacity may help. If it says the node has reached its Pod limit, investigate CNI/IP capacity and instance configuration.

---

Disruption controls prevent self-inflicted capacity loss

A PodDisruptionBudget, or PDB, sets an availability constraint for voluntary evictions, such as a planned node drain. For example:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: order-api
  namespace: trading
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: order-api

This means a voluntary eviction should not reduce the matching healthy Pod population below two.

PDBs are important, but their limits matter:

• They do not protect against node crashes, an AZ outage, kernel failures, or abrupt network loss.
• They do not create capacity; the remaining replicas must have somewhere valid to run.
• A PDB requiring every replica to remain available can block node upgrades and drains indefinitely.
• Deployment rollout availability must also be controlled with maxUnavailable, maxSurge, replica count, readiness gates, and termination behavior. Do not assume a PDB alone makes a rollout safe.

For a three-replica API with a PDB of minAvailable: 2, one planned eviction is allowed. If the service is already degraded to two healthy Pods, the PDB blocks another voluntary eviction. That is desirable from the application’s perspective, but it may delay a node maintenance operation. The correct response is to restore capacity, not bypass protection casually.

A useful incident question is:

“Is this an involuntary failure that the PDB could never prevent, or are we attempting a voluntary operation while the application is already below its availability floor?”

That distinction changes the response. For an involuntary node failure, restore replicas and capacity. For a planned drain blocked by a PDB, determine why the workload cannot safely tolerate the disruption before forcing the eviction.

---

A production diagnostic runbook

For an unstable EKS API, use this sequence. It is intentionally evidence-led.

1. Stabilize and preserve evidence

• Confirm user impact using request rate, error rate, latency, synthetic checks, and traces.
• Identify the affected revision, node, AZ, route, and target group where possible.
• Pause a clearly harmful rollout if evidence links the failure to the new revision.
• Avoid broad restarts, deleting all Pods, or scaling blindly.

2. Validate the workload population

• Compare Deployment desired, updated, available, and ready replica counts.
• Inspect Pods for Pending, crash loops, readiness failures, OOM kills, and restart count.
• Read namespace events in time order.
• Compare healthy and unhealthy Pods by node, AZ, image digest, configuration version, and resource consumption.

3. Validate the application locally

• Inspect current and previous container logs.
• Use traces to find application latency, dependency failures, and request cancellations.
• Confirm the process listens on the expected container port.
• Test the readiness endpoint locally or through controlled port forwarding when appropriate.

4. Validate Service selection and internal routing

• Check Service selector, ports, and named target ports.
• Check EndpointSlices for ready Pod IPs.
• Test the Service DNS name from a diagnostic Pod.
• Test direct Pod IPs only as an isolation technique.
• Review network policies and any service-mesh sidecar policy or telemetry if present.

5. Validate Ingress and AWS infrastructure

• Inspect Ingress host/path/backend mappings and IngressClass.
• Confirm the relevant controller is healthy and reconcile logs show no AWS API or IAM errors.
• Check ALB listener rules and target-group health.
• Determine whether targets are Pods in IP mode or nodes in Instance mode.
• Verify security groups, TLS/certificates, subnet placement, and external DNS behavior.

6. Validate scaling, node health, and disruption state

• Inspect HPA desired replicas, current metrics, and scale events.
• Inspect Pending Pod scheduling events.
• Inspect node readiness, pressure conditions, allocatable resources, Pod density, and evictions.
• Confirm the node autoscaler or capacity mechanism can provision nodes compatible with workload constraints.
• Check PDB status before draining nodes or modifying disruption-sensitive capacity.

---

An interview-ready incident response

A concise senior-level answer could be:

“I would first classify the blast radius: external-only, internal Service-wide, Pod-specific, node-specific, or deployment-correlated. I would freeze a suspected rollout only if the evidence supports it, then preserve events, target health, logs, and traces before making broad changes.

I start at the workload: Deployment availability, Pod phase, readiness, restart count, events, prior container logs, and resource usage. A Pod being Running does not mean it is serving; readiness controls Service endpoint membership, while liveness should only restart an irrecoverable process.

Next I inspect the Service selector, ports, and EndpointSlices, then test the Service DNS name from a controlled diagnostic Pod. That isolates application, Pod, Service, and DNS or dataplane faults. If internal Service traffic works but external requests fail, I inspect Ingress rules, controller health, ALB listener rules, target-group health, and security groups. In EKS I also confirm whether the ALB uses IP targets or Instance targets because the request path and evidence differ.

For failures under load, I inspect HPA desired replicas and Pending Pod events separately from node capacity. HPA can request Pods, but Cluster Autoscaler only helps when those Pods are truly unschedulable and a compatible node group can scale. Finally, I use PDBs to protect planned disruption, while recognizing they do not protect against an abrupt node or AZ failure. The mitigation must restore serving capacity without hiding the root cause.”

---

Key takeaways

• Diagnose from the application and Pods outward through Service, Ingress, and AWS infrastructure; do not change everything at once.
• Pod phase, readiness, restart counts, events, logs, and previous-container logs are the first evidence set.
• A Service must have correct selectors, ports, and ready EndpointSlice members; DNS resolution alone does not prove traffic can reach a healthy Pod.
• In EKS, distinguish the Ingress declaration, its controller, the AWS load balancer, and the target group. IP target mode simplifies the external request path and target-health interpretation.
• Readiness gates and graceful termination prevent avoidable request drops during deployments.
• HPA scales Pods; cluster capacity scaling supplies nodes. Pending-Pod events reveal whether requests, taints, affinity, Pod limits, or IP capacity are blocking placement.
• PDBs protect against voluntary disruption, not node crashes or AZ failure; they must be paired with sufficient replicas and safe rollout settings.

Next, the course turns to the data layer: designing an RDS topology with Multi-AZ availability, read replicas, backups, failover behavior, connection management, and encryption matched to explicit recovery objectives.