Designing a Highly Available ECS Service and Evaluating EKS Requirements

Welcome. This module focuses on production container and data platforms: not merely starting containers, but designing services that remain available when tasks, hosts, or an Availability Zone fail.

For this lesson, treat the workload as a stateless, latency-sensitive exchange API component: it receives synchronous requests, performs authentication and validation, invokes internal dependencies, and must not expose credentials or collapse during a traffic surge. The aim is to give an interview-ready ECS design and, equally importantly, explain why each decision was made and when ECS should give way to EKS.

---

Start with the service boundary and assumptions

A senior-level ECS design begins by making the assumptions visible. Otherwise, choices such as Fargate versus EC2 or ALB versus NLB sound like preference rather than engineering.

For a representative design, assume:

• The service is stateless. Durable orders, balances, or events are persisted outside the task, for example in a database or messaging system.
• It runs in one AWS Region across three Availability Zones.
• The service is customer-facing and must continue serving after the loss of one task, one EC2 instance, or one AZ.
• Requests are HTTP/HTTPS API calls unless a specified client protocol requires raw TCP or TLS pass-through.
• The service has a known load-test capacity per task. This is essential; task counts should be derived from measured throughput and latency, not chosen arbitrarily.
• Secrets such as database credentials, API keys, and signing material are centrally managed and never embedded in images, task definitions, or pipeline variables.

The resulting service has distinct responsibilities:

Component	Responsibility
ECS service	Maintains the desired task count and replaces failed tasks.
ECS task definition	Declares the immutable application image, CPU/memory, ports, health checks, roles, and secret references.
Load balancer and target group	Sends traffic only to healthy, registered task IPs.
Capacity layer	Supplies Fargate capacity or EC2 capacity for the scheduler.
Application Auto Scaling	Changes desired task count in response to meaningful demand signals.
IAM and Secrets Manager	Give each workload only the permissions and secret values it needs.

The ECS service is a replica service, not a one-off task. A one-off task is appropriate for a batch job or migration. A replica service is the construct that maintains a population of long-running API tasks when individual tasks stop.

---

Select Fargate or ECS on EC2 deliberately

Both Fargate and EC2 use the ECS scheduler, task definitions, services, IAM task roles, load balancer integration, and CloudWatch integrations. The key difference is who owns the underlying compute fleet.

When Fargate is the better default

Choose Fargate for a standard stateless API or worker when operational simplicity, isolation, and rapid platform delivery are more valuable than instance-level tuning.

With Fargate:

• AWS manages host provisioning, patching, and container-instance lifecycle.
• Each task receives its own network interface when using awsvpc networking.
• You size compute per task rather than managing bin packing across instances.
• The platform team has fewer host-level failure modes to operate during an incident.

For most internal APIs, back-office services, and independently scaling microservices, Fargate is an excellent starting point. It avoids turning a container platform into an EC2 fleet-management problem.

However, “serverless” does not mean capacity planning disappears. Fargate still has regional quotas, task startup time, image-pull time, and a reactive scaling delay. It also requires private subnet IP capacity across all intended AZs.

When ECS on EC2 is justified

For the assumed high-throughput exchange gateway, I would choose ECS on EC2 using an Auto Scaling group capacity provider for the latency-critical service. This is not because ECS on EC2 is inherently more highly available; both options can be highly available. It is because EC2 offers greater control over performance and capacity characteristics.

The rationale would be:

• The workload has sustained, predictable throughput, making efficient packing of multiple tasks on larger instances economically attractive.
• Tail-latency targets warrant explicit choice of instance family, CPU architecture, network bandwidth, EBS throughput, and host-level tuning.
• Baseline capacity must already be running before market events or traffic shocks; EC2 fleet planning makes that capacity explicit.
• A controlled AMI and host-agent model may be required for approved operational tooling, although every additional host agent expands the patching and security surface.

The EC2 capacity provider should be backed by Auto Scaling groups distributed across three AZs. Its minimum capacity must be able to run the service after losing one AZ, not merely under normal conditions. Use On-Demand capacity for the critical baseline. Spot may be appropriate for interruption-tolerant batch workers or noncritical elastic capacity, but not for the guaranteed baseline of an order-entry path.

A concise interview answer is:

“I choose Fargate by default for stateless services where host management provides no product value. For the sustained, latency-sensitive exchange gateway, I would use ECS on EC2 with capacity providers across three AZs, because I need predictable instance characteristics, cost-efficient task density, and pre-provisioned headroom. The service remains an ECS replica service with task-level isolation and IAM; I simply take responsibility for the capacity fleet.”

Do not claim that EC2 is automatically “faster” or that Fargate cannot serve production traffic. The decision must follow measured latency, throughput, cost, and operational requirements.

---

Place capacity to survive failure, not just to look distributed

The architecture below illustrates the availability objective: a load balancer serves tasks placed in three AZs. The pictured load balancer is an NLB; whether it is the right choice depends on the application protocol, discussed shortly.

For an ECS service, configure the following baseline:

1. Deploy tasks to private application subnets in three AZs. Tasks do not need public IP addresses when a public load balancer is their ingress path.
2. Enable Availability Zone service rebalancing so ECS corrects an uneven task distribution after deployments or capacity changes.
3. Use a replica scheduling strategy with an explicit minimum task count.
4. For ECS on EC2, use capacity providers attached to Auto Scaling groups that span the same three AZs.
5. Spread tasks across AZs first. For a particularly high blast-radius concern, also spread tasks across multiple hosts rather than concentrating many replicas on one host.
6. Maintain enough spare task and host capacity to operate after an AZ loss.

The key calculation is that a three-AZ service cannot simply run its normal capacity divided evenly by three and call itself AZ-resilient.

If normal demand requires 12 task-equivalents, AWS’s guidance for three AZs is to run 1.5 times normal capacity:

$$\lceil 1.5\times 12\rceil =18$$

With six tasks in each AZ, losing an AZ leaves 12 tasks in the remaining two zones: the capacity required for ordinary demand. This calculation assumes all tasks have comparable capacity and there is no hidden single-AZ dependency, such as a database endpoint, NAT path, downstream service, or EC2 Auto Scaling group that cannot survive the same failure.

Capacity headroom is an availability control

Autoscaling is inherently delayed. Metrics must be collected and evaluated; ECS must schedule a task; capacity must exist; the image must be pulled; the application must start; and load-balancer health checks must pass. For the EC2 launch type, a new instance may also need to launch and register.

Therefore:

• Set a target utilization range with headroom. A target around $60\%$ to $80\%$ is often more defensible than driving tasks near saturation.
• Keep a meaningful minimum desired count. For the 12-task normal-load example, a minimum of 18 is justified if the service must retain normal capacity after losing an AZ.
• Pre-scale for known events, such as market open, promotions, planned migrations, or product launches.
• Validate AWS service quotas, subnet IP availability, and EC2 instance capacity before anticipated high-volume events.
• Define a safe degraded mode if one exists, such as disabling expensive noncritical response enrichment while preserving order acceptance and status retrieval.

---

Choose the load balancer from the traffic contract

The load balancer is not just a traffic-distribution device. It is the health gate that determines which task is eligible to receive customer traffic.

For a conventional HTTPS REST or gRPC-over-HTTP service, select an Application Load Balancer (ALB):

• It operates at Layer 7 and understands HTTP request properties.
• It supports host- and path-based routing, which is useful when several APIs share an entry point.
• It exposes the ALBRequestCountPerTarget metric, a valuable scaling signal for request-driven services.
• It integrates naturally with HTTP health paths and web application protections where relevant.

For a raw TCP protocol, extremely high connection rates, source-IP preservation requirements, static IP requirements, or a TCP/TLS service where Layer 7 routing is not desired, select a Network Load Balancer (NLB). The architecture image depicts this model. It is a valid choice for a protocol-specific trading connection, but it is not automatically superior for a JSON/HTTPS API merely because it sounds lower latency.

A defensible answer states the protocol decision explicitly:

“For the public HTTP order API, I use an internet-facing ALB across three AZs. For a separate persistent TCP market-data or trading protocol that needs Layer 4 behavior and static addressing, I use an NLB. I do not put both behind one generic load-balancer decision.”

Apply security groups as service-to-service contracts

With awsvpc networking, each task receives an ENI and can have a task-specific security group. That permits a clean least-privilege network model:

Resource	Inbound rule	Reason
Public ALB	HTTPS from approved client networks or the internet, depending on product requirements	Public entry point
ECS task security group	Application port only from the ALB security group	Prevents direct task access
Database security group	Database port only from the ECS task security group	Prevents arbitrary VPC access
Internal dependency	Only necessary ports from the calling workload’s security group	Limits lateral movement

Avoid the common “allow all traffic from the VPC” rule. It makes an internal compromise more useful to an attacker and makes application connectivity impossible to reason about during a security review.

---

Make “healthy” mean able to serve traffic

A container process running is not the same as a task being ready to serve a request. The service needs layered health signals.

1. Container health check

The container health check answers: is the local process alive enough for ECS to retain it?

A /livez endpoint or local health command should be cheap and should not fail due to an optional downstream dependency. If every temporary dependency problem fails liveness, ECS can create a restart storm and remove all tasks precisely during a downstream outage.

2. Load balancer target-group health check

The target group answers: should this task receive new client traffic?

Point it at a readiness endpoint, such as /readyz. Readiness should validate the conditions truly needed to handle a request safely: application initialization is complete, critical configuration has loaded, and required connection pools are usable. It should not execute an expensive end-to-end transaction on every probe.

ALB or NLB health checks prevent an unhealthy target from receiving new requests. The ECS service replaces tasks that remain unhealthy, maintaining the desired replica count.

3. Health-check grace period and graceful termination

The ECS health-check grace period prevents an application that is still starting from being terminated prematurely. Set it from observed startup data: image pull, runtime initialization, configuration load, and readiness time. A value chosen from a console default is not a design.

On shutdown, the application should:

• Stop accepting new work.
• Fail readiness so the target group removes it from new request routing.
• Complete or safely cancel in-flight work within the configured drain and stop periods.
• Flush critical telemetry where feasible.
• Exit cleanly.

The target group’s deregistration delay must align with maximum legitimate request duration. The default of 300 seconds may be excessive for a short HTTP API and too short for long-lived connections. Configure it from the protocol’s behavior, then test it during a deployment and forced task termination.

---

Scale on demand signals that predict user impact

For an HTTP API, use more than one perspective on load:

Signal	What it reveals	Typical use
ALB request count per target	Actual request pressure per healthy task	Primary scale-out signal for request-driven APIs
CPU utilization	Compute saturation	Protects CPU-heavy validation, encryption, or serialization
Memory utilization	Risk of memory pressure or OOM termination	Guards application stability
Request latency and 5xx rate	User-visible degradation	Alerting and deployment protection; sometimes a custom scaling trigger
Queue depth and age, for workers	Backlog and delayed processing	Primary scaling signal for asynchronous consumers

For a steady API, target tracking on ALBRequestCountPerTarget, with CPU and memory as guardrails, is usually a clear initial policy. The request target must come from load tests: identify the per-task request rate at which p95 or p99 latency begins to violate the service objective, then choose a target below that point.

For a known rapid demand shock, use scheduled scaling or a carefully designed step policy. Target tracking tends to be simpler, but it reacts after the metric rises. A market open or planned event is not a surprise; capacity should be raised in advance.

Scale-in should be slower and more conservative than scale-out. Removing tasks too quickly can disrupt connection reuse, induce repeated cold starts, and create oscillation around a threshold. Set a minimum count that preserves AZ distribution and failure headroom; never allow routine scale-in to reduce the service to one task per Region.

For ECS on EC2, task autoscaling is only half the mechanism. A correct design also has:

• ECS service scaling that increases or decreases desired task count.
• Capacity-provider or Auto Scaling group scaling that adds or removes EC2 capacity.
• Sufficient warm instance capacity or a tested instance-launch time that meets the scaling objective.
• Placement monitoring for PENDING tasks, which often reveals CPU, memory, ENI, subnet-IP, or host-capacity exhaustion.

---

Separate task execution permissions from workload permissions

An interviewer may use these two roles to test whether you have operated ECS securely.

Task execution role

The task execution role is used by ECS infrastructure during task startup and operation. It typically permits activities such as:

• Pulling the image from Amazon ECR.
• Writing container logs to CloudWatch Logs.
• Retrieving a secret value that ECS injects into the container configuration.

It is not the role the application should use to write an order event, call a queue, or read an S3 object.

Task role

The task role is the IAM identity presented to the application code inside the running task. Grant it only the APIs and resources the workload needs.

For example, an order API task role might permit:

• Publishing only to a specific order-event stream or queue.
• Reading one designated configuration object.
• Calling one required internal AWS service API.

It should not receive broad permissions such as s3:*, full Secrets Manager access, or administrator policies. On ECS EC2, using task roles also prevents every task on a shared host from inheriting the broad permissions of the EC2 instance profile.

Secrets

Use AWS Secrets Manager for credentials and sensitive values. The task definition should contain a reference to the secret, not its plaintext value. Use AWS Systems Manager Parameter Store where its feature set and rotation needs are sufficient; use Secrets Manager where managed rotation or rich secret handling is required.

Two operational details distinguish a complete answer:

1. A secret injected as an environment variable is generally read when the task starts. Rotating the underlying secret does not automatically update an already running process.
2. Rotation needs a coordinated application strategy: either the application retrieves and refreshes credentials at runtime, or the deployment process rolls tasks safely after rotation.

Do not log secret values, include them in CI/CD output, bake them into images, or pass them in plaintext Terraform variables or task-definition environment sections.

---

Know when the requirement has outgrown ECS

ECS is not an inferior version of Kubernetes. It is the appropriate orchestrator when AWS-managed scheduling, straightforward service deployment, task-level IAM, and a smaller operational surface meet the product need.

Move to EKS only when requirements are genuinely Kubernetes-specific or platform-wide. Strong justification includes:

• The organization operates Kubernetes-native workloads requiring custom resources, operators, or controllers.
• Teams require DaemonSets for node-level agents, custom scheduling behavior, or ecosystem tooling built around Kubernetes APIs.
• A service mesh, admission-control model, network-policy model, or GitOps platform is standardized on Kubernetes and must be consistently applied across many teams.
• Multi-cluster or multi-cloud portability is a real business requirement, not an abstract aspiration.
• The platform must expose Kubernetes primitives directly to internal engineering teams.

Weak justifications include “Kubernetes is popular,” “we may need portability someday,” or “we need autoscaling.” ECS supports service autoscaling, task-level networking, service discovery, sidecars, and production deployment controls without requiring Kubernetes cluster operations.

EKS adds operational responsibilities even with a managed control plane: Kubernetes version upgrades, add-on compatibility, node or Fargate profile management, CNI behavior, autoscaler behavior, policy controls, and more complex incident diagnosis. That cost is worth paying only when its capabilities are required.

---

A concise architecture-defense response

For an interview, lead with the design decision, then explain failure handling and trade-offs:

“I would implement this as an ECS replica service across private subnets in three AZs. For a standard stateless API I would choose Fargate, but for the sustained, latency-sensitive exchange gateway I would use ECS on EC2 with capacity providers and On-Demand baseline capacity across all three AZs.

For HTTP traffic, an ALB distributes requests only to healthy task IP targets; tasks allow inbound traffic solely from the ALB security group. I would use an NLB only if the client protocol requires Layer 4 behavior, static IPs, or source-IP preservation.

I size the minimum task and host capacity to survive one AZ loss. If normal traffic needs 12 tasks, I run 18 across three AZs, leaving 12 after one AZ fails. Autoscaling uses request count per target plus CPU and memory guardrails, but headroom and scheduled pre-scaling cover the period before autoscaling can react.

Health is layered: local liveness prevents dead processes from persisting, readiness controls load-balancer traffic, and ECS replaces failed replicas. Finally, the execution role pulls images, emits logs, and retrieves injected secrets, while the task role gives the application narrowly scoped runtime access. I would adopt EKS only if Kubernetes-specific platform requirements justify its additional operational complexity.”

That response shows an architecture owner’s mindset: requirements, decision, failure mode, safety control, and trade-off.

---

Key takeaways

• A highly available ECS service is a multi-AZ, replica-based system with capacity headroom, not simply a task count greater than one.
• Fargate reduces host-management burden; ECS on EC2 is justified when instance-level performance control, dense utilization, or pre-provisioned compute capacity materially matter.
• Use ALB for Layer 7 HTTP routing and NLB for genuine Layer 4 requirements.
• Health checks, graceful draining, and realistic startup grace periods determine whether replacements improve availability or create instability.
• Autoscaling is reactive. Minimum capacity, AZ-loss planning, and scheduled scaling are availability controls.
• Task roles, execution roles, security groups, and managed secrets form separate least-privilege boundaries.
• EKS is appropriate for Kubernetes-specific platform needs, not as a default substitute for ECS.

Next, the focus moves from designing an ECS service to diagnosing an unstable EKS request path: workloads, probes, Services, Ingress, autoscaling, disruption controls, and node signals.