Zero-Downtime ECS Delivery and Rollback Runbook with Immutable Promotion and Safe Database Migration

Hello. In the previous lesson, you treated a Terraform plan as a production change proposal: verify its account, state, privileges, drift, destructive changes, dependencies, and rollback implications before approving it. That is the infrastructure-control layer.

This lesson applies that discipline to an ECS application release. The aim is not simply to “deploy a new task definition,” but to operate a release process that preserves availability, produces evidence at each gate, and retains a fast, reliable recovery path. For an exchange or payments workload, the hardest part is usually not replacing containers; it is ensuring that old code, new code, and changing data structures can coexist safely.

By the end, you should be able to present a practical zero-downtime ECS delivery and rollback runbook, explain when to choose rolling versus blue/green deployment, and defend why database rollback is fundamentally different from application rollback.

---

1. What “zero downtime” actually requires

A deployment is not zero-downtime merely because ECS reports it as successful. A meaningful claim is that the service continues to meet its agreed availability and latency objectives while the change occurs.

That requires at least five conditions:

1. Enough healthy capacity remains available.
   Tasks must be distributed across Availability Zones, have sufficient headroom, and pass ECS/container and load-balancer health checks before receiving meaningful traffic.

2. New tasks are actually ready, not merely started.
   A process listening on a port is not necessarily ready to serve requests. It may still be loading configuration, retrieving secrets, warming a cache, establishing database connections, or synchronizing reference data.

3. Traffic drains safely from retiring tasks.
   The load balancer must deregister old tasks before they are stopped, allowing in-flight requests to complete within a consciously chosen drain timeout. Application shutdown handling matters here: the container should stop accepting new work and complete or safely abandon work before ECS forcibly terminates it.

4. Both application versions remain compatible during overlap.
   During a rolling deployment, and especially during blue/green bake time, clients can reach both versions. API contracts, message schemas, caches, and database access must tolerate that overlap.

5. Rollback is executable under stress.
   The old artifact, task definition, configuration, secret references, deployment settings, and database compatibility must still be available. “We can rebuild the previous version” is not an acceptable rollback plan.

A useful interview distinction is:

“Container rollback can be fast because I redeploy a known immutable task definition. Database rollback is asymmetric: once data has been written in a new shape, simply reverting application code or schema can lose or misinterpret data. Therefore, I design database changes for forward compatibility and delay destructive cleanup.”

---

2. Build once, promote the exact artifact

The foundation of a safe release is artifact identity. A production deployment should be traceable to a specific source commit, build, image digest, dependency set, test results, and security-scan result.

Do not treat a mutable-looking label such as latest, release, or even prod as the deployment identity. Those labels describe intent; they do not reliably identify bytes.

ECR tag immutability is a valuable control, but in a production runbook the strongest deployment reference is still the image digest:

123456789012.dkr.ecr.eu-west-1.amazonaws.com/orders-api@sha256:<digest>

A digest identifies the exact image content. The ECS task definition should reference that digest, or the pipeline should resolve the approved immutable tag to its digest and record that resolved value in deployment evidence.

Artifact-promotion model

Build the image once from a protected commit. Then promote the same digest through environments. Do not rebuild the source independently for test, staging, and production.

Release evidence	Why it matters
Git commit SHA and pull request	Identifies reviewed source and change context.
Image digest	Identifies the exact container bytes deployed.
SBOM and dependency scan result	Shows the dependency and vulnerability position of that specific artifact.
Image scan result	Detects known vulnerabilities in the packaged image.
Signed artifact or attestation, where used	Establishes provenance and reduces supply-chain ambiguity.
ECS task definition revision	Captures runtime configuration, image digest, ports, resource limits, task role, and secret references.
Migration version or checksum	Establishes exactly which schema change was executed.
Deployment record	Links approver, release window, dashboard links, decision points, and rollback revision.

This makes rollback concrete. Instead of saying “go back to yesterday’s release,” the runbook says:

Redeploy task definition revision 184, which references digest sha256:..., with feature flag new-risk-engine=false; preserve schema version 2025.03.08.2.

The delivery gates should produce evidence

For a senior-level process, a pipeline is a series of controls, not a linear chain of tools. Typical gates are:

Gate	Minimum evidence	Block release when
Source gate	Protected branch, reviewed pull request, commit SHA	Required review, checks, or change record are missing.
Build gate	Reproducible build, image digest, unit-test results	Build or tests fail.
Security gate	Secret scan, dependency scan, image scan, policy checks	Vulnerability threshold or policy threshold is breached.
Infrastructure gate	Reviewed Terraform plan, approved state/account/Region	The plan contains unexplained replacement, drift, privilege, network, or data risk.
Pre-production gate	Integration, contract, and smoke-test evidence	Critical user journeys or dependencies fail.
Production approval	Named accountable approver and release window	Risk, ownership, or rollback evidence is incomplete.
Release-health gate	Synthetic checks, SLI dashboards, alarm status	Error, latency, saturation, or business-integrity thresholds are breached.
Completion gate	Bake-period evidence and release decision	The candidate has not demonstrated stable behavior under production conditions.

The exact tools may be Jenkins, GitHub Actions, GitLab CI, Harness, or CodePipeline. The senior design principle is invariant: the same controlled pipeline identity promotes the same evidence-backed artifact across environments.

---

3. Choose rolling or blue/green based on rollback and compatibility

A rolling deployment and a blue/green deployment can both avoid an interruption in request handling. They have different operational properties.

Decision factor	Rolling ECS deployment	Blue/green ECS deployment
Running revisions	Old and new tasks coexist gradually in one service deployment.	Blue and green service revisions coexist as distinct target groups.
Capacity cost	Usually lower; only extra deployment capacity is needed.	Higher during deployment because both revisions must run.
Traffic control	ECS replaces tasks according to deployment configuration.	Explicit test traffic and controlled production traffic switching are available.
Rollback speed	Redeploy prior task definition; replacement takes time.	Shift traffic back to retained blue revision during bake time.
Best fit	Low-to-moderate risk, stateless service changes with stable compatibility.	High-risk release, payment/trading path, protocol change, major runtime upgrade, or change requiring controlled live validation.
Main risk	New and old tasks serve production simultaneously without a fully isolated test environment.	Additional operational complexity, capacity requirement, target-group and listener design.

When rolling deployment is sufficient

A rolling ECS deployment is reasonable when all of the following are true:

• The service is stateless or safely externalizes state.
• New and old versions can run concurrently.
• The database change is additive or absent.
• There is sufficient capacity across at least two Availability Zones.
• The load balancer health endpoint accurately represents readiness.
• The deployment configuration preserves the required healthy count.
• Redeploying the prior task definition is a sufficient application rollback.

For a critical service, a conservative setting often keeps the minimum healthy percentage at $100\%$ and permits temporary additional capacity through a maximum percentage above $100\%$. The precise values must account for available ECS capacity, Fargate quotas or EC2 cluster headroom, target-group limits, and the desired deployment duration.

A rolling deployment is not appropriate merely because it is cheaper. If rollback needs to be nearly immediate, or if you need to test a new revision against production-like requests before broad exposure, blue/green is the stronger operational choice.

Blue/green: separate revisions and controlled traffic

The two target groups are central. Blue and green tasks are not simply labels; they represent separate traffic destinations behind the load balancer. The old revision remains available through the verification and bake period.

For an HTTP service behind an ALB, test traffic can be directed to green through a separate test listener, path rule, or tightly controlled request-header rule. For example, an internal synthetic tool can send X-Test-Version: green, allowing it to exercise green without exposing normal users to the candidate revision.

The test route must be protected. It should not let an arbitrary external caller select an unreleased revision by adding a header. Restrict it through private access, authenticated internal tooling, network controls, or a combination of these.

The ECS blue/green lifecycle gives useful places to enforce tests and manual decisions:

• Before scale-up: validate deployment metadata, approved image digest, migration status, and change authorization.
• After scale-up: confirm green tasks are running, registered, healthy, and distributed correctly.
• After test traffic shift: run authenticated synthetic transactions against green.
• Before production traffic shift: pause for accountable approval when risk warrants it.
• After production traffic shift: validate production signals during the bake period.

The important principle is that a lifecycle hook should assess a meaningful release condition, not merely call the same shallow /health endpoint already used by the load balancer.

---

4. Health verification: prove user outcomes, not process existence

ECS deployment health is necessary but not sufficient. Use layered verification because each layer answers a different question.

Verification layer	Question answered	Example evidence
ECS task state	Did the desired tasks start and remain running?	Desired count equals running count; no repeated task stops.
Container health check	Is the application process alive?	The process responds locally and has not deadlocked.
ALB target health	Can the load balancer reach the task on the service port and path?	Healthy targets in every enabled Availability Zone.
Readiness behavior	Can this task safely accept production work?	Required configuration loaded; connection pools usable; essential dependencies reachable.
Synthetic transaction	Can a realistic authenticated request complete correctly?	Submit a test order or quote request in a safe test account and verify its expected result.
Service SLIs	Are real users receiving acceptable service?	Request success rate, tail latency, saturation, and dependency failures.
Business integrity	Is the release preserving the business outcome?	Order acceptance, event-processing lag, reconciliation discrepancy, or settlement failure signals remain normal.

A robust readiness check should be intentionally narrow. A liveness check that fails because a downstream dependency has a transient problem can restart every healthy task at once, making a dependency incident into an application outage. Conversely, a readiness check may reasonably prevent a newly started task from receiving traffic until essential initialization has completed.

Before starting a production deployment, define the rollback signals and decision owner. For example:

• Elevated server-error rate relative to the current baseline.
• Tail-latency breach for a critical API.
• Increased timeout or connection-pool exhaustion.
• Message-processing lag or consumer failure.
• Failed synthetic transaction.
• A critical business-integrity or reconciliation alarm.
• Security or authorization failures introduced by the candidate.

These conditions must be grounded in the service’s established SLIs and SLOs, not invented during an incident.

Amazon ECS can stop failed deployments when tasks fail to start through the deployment circuit breaker, and CloudWatch alarms can be configured to drive failure detection and automatic rollback. That is a safety net, not a substitute for deliberate SLI design.

---

5. Database migration is a release train, not a deployment step

The central rule is simple:

Never deploy application code that requires a destructive schema change while old application instances can still be running.

A zero-downtime database change uses expand, migrate, contract. The schema remains compatible with both revisions until the service has fully adopted the new representation and the team has verified the result.

Consider replacing customer_status with a richer risk_status structure.

Expand

Add the new column or table without breaking existing code:

• Add nullable columns, new tables, or additive indexes.
• Do not rename or drop a field in the same release.
• Check engine-specific locking behavior before executing DDL on a large RDS table.
• Make the migration idempotent, versioned, logged, and independently executable.
• Preserve a database backup and confirm the recovery approach before a significant migration.

At this point, old code continues to read and write the old model.

Migrate

Deploy a compatibility version of the application that can write both representations. Then backfill historical data in controlled batches.

A production backfill needs more than a one-time SQL statement:

• Limit batch size and rate to protect primary database latency.
• Make processing resumable and idempotent.
• Track progress, failure count, retry behavior, and replication lag.
• Reconcile counts and sampled records between old and new representations.
• Ensure concurrent writes are not missed while the backfill runs.

For financial or exchange workflows, also assess side effects. If a schema change affects order state, balances, or event handling, the migration and reconciliation model must be designed with the service owner and data owner. A technically complete backfill is not enough if business state can diverge.

Contract

After the data is migrated and verified, deploy a version that reads the new schema. Only later does the service stop writing the old representation. The old column, table, index, or code path is removed in a separate, deliberately approved cleanup release.

The time between “new code can read the new structure” and “old schema is deleted” is a safety buffer. It keeps application rollback possible.

Do not include destructive schema cleanup in the same production change as the feature rollout. The clean-up is usually low urgency and high irreversibility, which makes it a poor trade-off during a release window.

---

6. Production ECS delivery and rollback runbook

The following is a practical runbook template for a production ECS service. It assumes a high-value API behind an ALB, an immutable ECR image, Terraform-managed infrastructure, and an RDS-backed service.

A. Define release ownership and entry criteria

Before the pipeline starts, record:

• Release lead: owns go/no-go decisions and stakeholder communication.
• Service owner: validates application behavior and business outcomes.
• Platform/SRE owner: validates ECS, load balancer, capacity, observability, and rollback execution.
• Database owner: approves migration design and validates backfill/reconciliation where applicable.
• Incident commander: named in advance for high-risk releases.

The release ticket or change record must include:

• Commit SHA, image digest, task definition revision, and previous known-good revision.
• Chosen strategy: rolling or blue/green, with rationale.
• Migration version, compatibility statement, and cleanup plan.
• Approved production Terraform plan, if infrastructure changes are included.
• Dashboard, log, trace, and alarm links.
• Explicit rollback triggers, rollback owner, and stakeholder notification channel.
• Confirmed deployment window and capacity headroom.

Stop the release if the rollback revision is unavailable, the migration is not backward compatible, critical alarms are already active, or the release owner cannot identify the expected behavior and rollback trigger.

B. Build, test, secure, and promote the candidate

1. Build once from the reviewed commit using a controlled CI identity.
2. Generate and retain the image digest, SBOM, build metadata, and test results.
3. Push the image to ECR using immutable tags; resolve and record the digest.
4. Run unit, integration, contract, and relevant performance tests.
5. Run secret scanning, dependency scanning, image scanning, and policy checks.
6. Deploy the exact digest to a representative non-production environment.
7. Run smoke tests and realistic synthetic flows.
8. Obtain production approval only after the evidence is complete.

An approval should be a decision on observed risk, not a routine button click. For example, a critical vulnerability finding may require security approval or remediation; it should not be waived by the same person trying to accelerate the release.

C. Prepare production safely

1. Confirm production account, Region, ECS cluster, service, task definition family, target groups, and alarm names.
2. Confirm sufficient capacity across Availability Zones for the chosen deployment strategy. Blue/green requires capacity for both revisions during the bake period.
3. Confirm the currently deployed blue or rolling baseline is healthy: no active critical alarms, stable error rate, acceptable latency, and healthy target count.
4. Pause unrelated configuration changes to the same service, load balancer, task role, secrets, database, or Terraform state.
5. Run only the expand database migration, if applicable. Validate migration completion and ensure old application code still functions.
6. Create the candidate ECS task definition using the approved image digest and approved runtime configuration. Avoid unreviewed environment-variable or secret-reference changes in the same release.

D. Execute the deployment

For a rolling deployment:

1. Start the service deployment with the defined minimum healthy and maximum capacity settings.
2. Watch ECS service events, task stops, container logs, ALB target registration, and target health.
3. Confirm that healthy new tasks are added before old tasks are drained.
4. Run synthetic checks while both revisions are present.
5. Continue monitoring until the old revision is fully drained and the service reaches steady state.
6. Keep the previous task definition and release evidence available until the bake period completes.

For a blue/green deployment:

1. Scale up green using the approved candidate task definition.
2. Verify green task count, Availability Zone distribution, task health, target-group health, logs, traces, and secret access.
3. Route only controlled test traffic to green.
4. Run functional, authorization, dependency, and business-flow smoke tests against green.
5. Require the defined production approval or lifecycle-hook success before production traffic moves.
6. Shift production traffic according to the chosen strategy.
7. Keep blue running for the defined bake period; do not retire it just because the traffic switch succeeded.

E. Bake and verify

During bake time, monitor both leading and lagging indicators:

• ECS deployment state and task restart count.
• ALB healthy target count, request count, HTTP error rate, and target response time.
• Application error and latency SLIs, especially the relevant tail percentile.
• Database CPU, connections, locks, storage, replication health, and query latency.
• Queue depth, consumer lag, retry rate, and dead-letter activity where messaging is involved.
• Authentication, authorization, secret retrieval, and third-party dependency failures.
• Business measures appropriate to the service, such as accepted orders, rejected orders, reconciliation variance, or settlement workflow success.

Record a formal decision at the end of the bake period:

• Complete: retain release evidence, update deployment record, and schedule later contract cleanup if applicable.
• Hold: retain blue, stop further rollout, investigate without increasing exposure.
• Rollback: use the procedure below.

F. Rollback procedure

Application-only rollback

For a rolling deployment:

1. Stop further rollout activity.
2. Deploy the previous known-good ECS task definition revision, which must reference the prior immutable image digest.
3. Monitor replacement task health and ALB targets until the service returns to steady state.
4. Confirm recovery through synthetic checks, SLIs, logs, traces, and business indicators.
5. Preserve logs, traces, task-stop reasons, deployment events, and configuration evidence before cleanup.

For blue/green:

1. Stop the deployment if it has not completed.
2. Shift production traffic back to blue while blue is retained during bake time.
3. Confirm blue target health and service SLIs recover.
4. Keep green available only long enough to collect evidence, unless it creates security or cost risk.
5. Do not delete release evidence, failed-task logs, or diagnostic traces.

Database-aware rollback

If an additive migration and dual-write compatibility are in place, application rollback should be safe because the prior application still understands the old schema.

Do not immediately:

• Drop the new column or table.
• Reverse the migration blindly.
• Restore a database backup solely to undo an application deployment.
• Assume database rollback will remove already-issued external side effects.

Instead:

1. Roll back the application or disable the feature path first.
2. Preserve both schema representations during investigation.
3. Assess whether any data requires reconciliation or compensating action.
4. Use the established restore or point-in-time recovery process only when the incident is genuinely data-corruption or data-loss related, and when the business impact of recovery has been approved.
5. Delay contract cleanup until the root cause is understood and the system has stabilized.

---

7. A concise interview response

If asked, “How would you design zero-downtime ECS deployments and rollbacks?”, a strong response is:

“I build the container once from a reviewed commit, scan it, and promote the same immutable ECR image digest through environments. The ECS task definition references that approved artifact, so rollback means redeploying a known task-definition revision rather than rebuilding code.

For a compatible, low-risk stateless change, I use a rolling deployment with healthy-capacity settings, ALB readiness checks, connection draining, and CloudWatch alarms. For high-risk services or changes needing production-like validation, I use ECS blue/green with separate target groups, controlled test traffic, lifecycle hooks, an approval before production traffic changes, and a bake period where blue remains available.

I define rollback triggers in advance using service SLIs, synthetic transactions, dependency signals, and business-integrity metrics. The major constraint is database compatibility: I use expand, migrate, and contract. I add schema first, deploy code that supports both versions, backfill and reconcile, then delay destructive cleanup to a later release. That keeps application rollback safe even after deployment.”

---

Key takeaways

• Zero downtime is a service-level outcome: healthy capacity, readiness, safe draining, compatibility, and executable rollback all matter.
• Immutable ECR tags prevent tag overwrite; an image digest provides the most precise deployment identity.
• Promote the same tested artifact through environments rather than rebuilding per environment.
• Rolling ECS deployments are efficient when version compatibility and rollback time are acceptable.
• Blue/green deployments provide stronger isolation and faster traffic reversal, but require duplicate capacity and disciplined target-group, test-route, and bake-time design.
• Health checks must extend from task state to user-facing synthetic flows and business-integrity signals.
• Database changes must follow expand, migrate, and contract; destructive cleanup belongs in a later release.
• A credible runbook names owners, defines stop conditions, captures evidence, sets rollback triggers before release, and distinguishes application rollback from data recovery.

Next, the course moves from delivery safety into the access and operational controls that protect the platform: least-privilege IAM, short-lived credentials, permission boundaries, and managed secrets.