Designing and Defending Backup and Disaster Recovery Strategies

Good to see you again. In the last lesson, you defined customer-facing SLIs, SLOs, and error-budget policies so that reliability decisions are tied to measurable user impact. Disaster recovery uses a different lens: an SLO measures ongoing service behavior over a window, while DR asks what happens when a larger event makes a workload—or an entire Region—unusable.

For a senior AWS DevOps interview, avoid saying “we use Multi-AZ and backups” as though that is a complete DR design. You need to establish the business recovery targets, distinguish AZ-level high availability from regional recovery, choose a proportionate strategy, protect against data corruption as well as infrastructure loss, and demonstrate that the plan works through evidence from tests.

By the end of this lesson, you should be able to propose and defend a DR design for a critical exchange-style service, including its RTO, RPO, backup controls, failover and failback runbooks, and test plan.

---

1. Begin with the failure scope and recovery objectives

A resilient platform needs both high availability and disaster recovery, but they answer different questions.

• High availability (HA) limits the impact of common component, instance, network, software, load, or single-AZ failures.
• Disaster recovery (DR) restores the whole workload when a larger event makes the primary operating site unavailable, or when corruption or a security incident makes the active data unsafe.

A Multi-AZ ECS service behind an ALB, with tasks distributed across AZs and an RDS Multi-AZ database, is strong HA. If an ECS task, node, or AZ fails, healthy capacity continues serving requests; RDS can fail over to its standby. That does not constitute regional recovery:

• A full AWS Region disruption can affect the application, database, IAM and service dependencies in that Region.
• Multi-AZ replication does not protect against an accidental DELETE, faulty migration, ransomware encryption, or malicious administrative action. The destructive change can be replicated quickly.
• Multi-AZ does not give you a separately deployable environment, separately available data, or a tested means to reroute users across Regions.

The first architectural decision is therefore not “which AWS DR service?” It is: what event are we recovering from, for which business capability?

RTO and RPO are business commitments

Two targets drive the design:

• Recovery Time Objective (RTO): the maximum acceptable duration from service interruption to restored service.
• Recovery Point Objective (RPO): the maximum acceptable amount of data loss, expressed as time since the latest recoverable data point.

They are related but independent:

• Restoring a database snapshot from ten minutes ago might give an RPO of ten minutes, but rebuilding infrastructure and validating the application could take hours, producing a much longer RTO.
• A warm standby may be ready to accept traffic in minutes, providing a low RTO, but asynchronous data replication may still leave a non-zero RPO.

For an exchange platform, “RPO = zero” should not be accepted casually. It has a significant engineering and business meaning: no accepted transaction may be lost. Asynchronous cross-Region replication alone cannot guarantee that. A defensible design may instead state:

“For a regional outage, we target an RTO of 15 minutes and an RPO of less than one minute for operational order data. For financial correctness, accepted orders are idempotent and reconcilable against an authoritative event or ledger record. Orders whose final state is uncertain during the recovery window are reconciled before they are treated as final.”

This is far stronger than promising “zero data loss” without a mechanism to deliver it.

Use recovery tiers, not one target for the whole platform

An exchange is not a single homogeneous workload. The customer impact and compliance risk differ materially across services.

Capability	Example RTO	Example RPO	Recovery intent
Public order submission and risk checks	15 minutes	Under 1 minute	Restore controlled order entry quickly; prevent duplicate or uncertain processing
Order-status and account views	30 minutes	Under 5 minutes	Restore customer visibility and support investigation
Market-data cache or public price display	15–30 minutes	Seconds to minutes	Rebuild or refresh from the authoritative feed where possible
Ledger, audit trail, and reconciliation records	1–4 hours for full access	Business-defined; integrity is primary	Restore a verified, auditable state rather than merely a fast database endpoint
Analytics, reporting, batch ETL	24 hours	24 hours	Use backup and restore; do not pay warm-standby cost without business need
Infrastructure definitions and deployment artifacts	Hours	Near current version	Recreate through versioned Terraform, immutable images, and artifact replication

The important senior-level move is to state assumptions. If the business says an order is accepted only after a durable database commit and event publication, then the recovery plan must preserve or reconcile both facts. If the product tolerates a short “trading temporarily unavailable” state during a regional event, warm standby may be appropriate. If it cannot tolerate it, the architecture and its cost must change.

---

2. Choose a strategy that satisfies, rather than exceeds, the requirement

AWS DR approaches form a spectrum. Lower RTO and RPO normally require more continuously running capacity, replication, operational sophistication, and cost.

Strategy	Typical RPO	Typical RTO	Recovery-region state	Main trade-off
Backup and restore	Hours	Hours, up to a day	Backups only; infrastructure and data restored during incident	Lowest cost, longest restoration
Pilot light	Minutes	Tens of minutes	Core data and minimal foundational components live; application capacity created during recovery	Low operating cost, depends on deployment and scaling actions
Warm standby	Seconds to minutes	Minutes	Fully functional workload operates at reduced capacity	Faster recovery, ongoing regional cost
Multi-site active/active	Near real time	Potentially near zero	Multiple Regions actively serve production traffic	Highest complexity, especially for multi-Region writes

The diagram is useful in interviews because it prevents a common mistake: treating active/active as automatically “best.” It is only appropriate when the business impact justifies its cost and data-consistency complexity.

A proportionate design for the scenario

For a critical order-management platform with a 15-minute regional RTO and under-one-minute operational-data RPO, I would normally propose:

1. Multi-AZ HA in the primary Region for ordinary infrastructure and AZ failures.
2. Warm standby in a second Region for the critical order, risk, and customer-status services.
3. Cross-Region data replication for live operational recovery, with replication lag monitored against the RPO.
4. Independent point-in-time backups to protect against logical corruption, deletion, ransomware, and a bad deployment.
5. Manual authorization with automated execution for regional failover, rather than a fully automatic regional switch.

Warm standby is appropriate because the secondary Region can already process controlled requests at reduced scale. Recovery primarily involves promoting the data layer where required, increasing service capacity, validating dependencies, and shifting traffic.

Pilot light can be defensible if the RTO is closer to 30–60 minutes and the organization accepts dependency on control-plane actions to create compute capacity. Backup and restore is appropriate for reporting, analytics, and lower-tier workloads, but not for the customer-facing trading path under this target.

Why active/active is not the default

Multi-site active/active reduces application unavailability when one Region fails because the other Region is already serving traffic. But the hard problem becomes data ownership.

For example, routing writes to both Regions requires a defined conflict model:

• Single-writer model: all writes go to one Region; other Regions serve reads or forward writes. This simplifies consistency but does not make writes fully active/active.
• Write partitioning: a customer, instrument, account, or shard has one designated write Region. The partitioning rule prevents conflicting writes.
• Conflict-tolerant writes: only appropriate where the application can safely resolve concurrent updates. “Last writer wins” is usually dangerous for balances, orders, risk limits, and settlement state.

For an exchange workflow, a simple statement such as “we will use DynamoDB Global Tables for active/active” is incomplete. You must explain whether an order can be written in either Region, how duplicate submissions are prevented, how sequence and causality are maintained, and what happens if replication lags or a Region returns after an outage.

---

3. Protect the complete recovery set, not just the database

A recoverable application needs more than database backups. A useful interview framework is to inventory five categories:

Recovery category	Examples	Required protection
Authoritative data	RDS or Aurora data, ledger events, object records	Replication where needed, point-in-time recovery, cross-Region backups
Stateful supporting data	EBS, EFS, caches with important state, queues	Backup, replication, or explicit rebuild procedure
Application artifacts	Container images, versioned binaries, configuration bundles	Immutable artifact promotion and regional replication
Infrastructure and configuration	VPCs, subnets, ALBs, ECS services, IAM roles, WAF rules, DNS settings	Terraform in version control, separate regional state, tested deployment
Security and access dependencies	KMS keys, secrets, certificates, break-glass roles	Regional key and secret availability, controlled replication, tested recovery authorization

The architecture must answer a practical question: could an authorized operator build and operate the secondary Region without improvising during an outage?

A concrete AWS implementation

For a containerized ECS order platform, a credible warm-standby design could contain the following.

Primary Region

• ALB distributes traffic to ECS services across at least two AZs.
• RDS Multi-AZ or Aurora provides local HA.
• RDS automated backups and point-in-time recovery protect from accidental changes within the primary Region.
• Order processing emits durable, idempotent domain events with correlation and idempotency keys.
• S3 holds immutable or versioned artifacts, documents, reports, and exports as needed.

Recovery Region

• A separate VPC, subnet tiers, security groups, route configuration, ALB, ECS services, CloudWatch alarms, and IAM roles are deployed by Terraform.
• ECS services run a reduced but validated task count. Their configuration points to the recovery data path, not the unavailable primary endpoint.
• Service quotas are pre-reviewed. A DR plan that requires scaling from two tasks to 200 but lacks regional ECS, Fargate, EC2, NAT, or database quota capacity is not a plan.
• Images are available in the Region at the approved immutable digest. Do not assume that a container image can be pulled from an unavailable primary Region.
• Secrets and KMS access are available according to the security design. Encryption keys are regional resources, so key policy and decrypt permissions require explicit DR testing.

Data protection

• For a relational workload requiring low operational RPO, use an appropriate cross-Region replication pattern. For example, an Aurora Global Database can keep a secondary cluster ready for promotion, while traditional RDS read replicas may be appropriate where their replication and promotion characteristics meet the stated target.
• Monitor replication lag. “Typically low lag” is not proof that the RPO is met.
• Use AWS Backup for centralized backup policy, retention, recovery-point monitoring, and cross-Region or cross-account backup copies where supported.
• Enable S3 Versioning for protected object data; use cross-Region replication where regional availability is required.
• Keep point-in-time backups in addition to live replication. A replicated bad write is still a bad write.

Ransomware and account compromise require isolation

A backup in the same compromised production account may not be a usable backup. An attacker with broad privileges may disable backup plans, shorten retention, alter encryption access, or delete recovery points.

A stronger pattern uses:

• A dedicated backup or data-vault account.
• Cross-account backup copies into a vault controlled by separate administration.
• Separate KMS keys and narrowly scoped key policies.
• AWS Backup Vault Lock, where retention and regulatory requirements support immutable recovery points.
• Alerts generated outside the production account for backup-job failures, disabled plans, altered retention, and unusual deletion or access events.
• A separate recovery account or clean recovery environment for restoration when the production account itself may be untrustworthy.

A concise interview phrase is:

“Replication supports availability and a low operational RPO. Independent, immutable point-in-time backups support recovery from corruption and compromise. We need both.”

---

4. Make failover an operational process, not a diagram

An RTO is consumed by more than database promotion. The real recovery clock includes:

• Detection of the customer-impacting condition.
• Incident declaration and authorization.
• Fencing or isolating the impaired primary Region.
• Database promotion or selection of a clean restore point.
• Scaling application capacity.
• Routing traffic.
• Functional validation of critical customer journeys.
• Communication and controlled resumption of service.

If the target RTO is 15 minutes, assign a budget:

Activity	Target budget
Detect and confirm regional impact	2 minutes
Declare incident and authorize DR action	2 minutes
Fence primary writes and initiate recovery workflow	2 minutes
Promote regional data path and scale critical services	5 minutes
Shift traffic and execute synthetic validations	3 minutes
Resume controlled service and communicate status	1 minute

This is not a guarantee; it is a testable operational hypothesis. If an approved incident commander cannot be reached for 20 minutes, a 15-minute RTO cannot be met, regardless of technical design.

Failover runbook for a warm standby

The exact implementation varies, but a robust runbook should make these decisions explicit:

1. Confirm impact and declare the event. Use customer-journey synthetics, SLO symptoms, dependency health, and AWS Health information. A simple TCP or ALB health check alone is not sufficient.
2. Establish a single writer. Stop, fence, or otherwise prevent writes to the former primary where possible. This protects against split-brain behavior if partial connectivity remains.
3. Assess the data state. Confirm the latest replication checkpoint and whether the event is availability-related or a data-integrity incident. For corruption or ransomware, select a known-good point-in-time recovery point rather than promoting corrupted live data.
4. Activate the recovery application. Promote the designated database secondary where applicable; set the secondary ECS desired count or capacity policy to the recovery profile; verify secrets, configuration, dependencies, and deployment version.
5. Shift traffic deliberately. Use a prebuilt data-plane routing mechanism where possible, such as Route 53 Application Recovery Controller routing controls. A human should commonly authorize the switch, while the mechanics remain automated.
6. Validate before broad release. Run authenticated synthetic checks for order submission, a harmless business validation, order-status retrieval, read/write database health, queue processing, and critical third-party dependencies.
7. Operate in DR mode. Communicate customer impact, record the declared RPO gap, preserve evidence, and apply heightened change control.
8. Reconcile. Identify transactions around the recovery boundary using idempotency keys, event records, and ledger reconciliation. Do not silently assume every request timed out without effect.

This is especially relevant for financial transactions. A customer whose order submission timed out might retry. The platform must ensure that a retry is recognized as the same idempotent request, not processed as a second order.

Failback is a separate design problem

Failback is not “switch DNS back.” During the outage, the recovery Region becomes the source of truth and accumulates new writes. Before returning:

1. Rebuild or repair the former primary Region using the same Terraform and artifact pipeline.
2. Re-establish replication from the active recovery Region to the former primary.
3. Validate data consistency, replication lag, capacity, security controls, and customer journeys.
4. Plan a controlled write transition with the same single-writer protection used in failover.
5. Shift traffic in a defined maintenance or progressive-release window.
6. Keep the former recovery Region ready as the next recovery target.

A useful operational model is that the recovery Region may remain the new primary after a real incident. The original primary can be rebuilt as the standby. This reduces pressure to rush an unsafe reversal.

---

5. Test recovery and collect evidence

A backup policy is an intention. A completed restoration and validated application workflow are evidence.

Testing must prove both objectives:

$$\text{Observed RTO}=\text{time from detected interruption to validated service restoration}$$ $$\text{Observed RPO}=\text{time difference between the disruption and the latest verified recoverable data}$$

For a warm standby, establish a recurring test program:

Test	Example frequency	What it proves
Backup restore test in isolated environment	Monthly	Recovery points are readable, decryptable, complete, and usable
Recovery-region deployment validation	Per infrastructure or application release	Terraform, artifacts, configuration, secrets, and quotas remain current
Data replication-lag review	Continuous alerting; weekly review	Operational RPO is being maintained
Application-level DR drill	Quarterly	Recovery Region can process critical journeys at its intended capacity
Controlled regional failover game day	Semi-annually or according to business criticality	End-to-end RTO, traffic routing, roles, communications, reconciliation, and failback
Corruption or ransomware recovery scenario	At least annually	Teams can select a clean recovery point rather than promoting replicated corruption

A drill should record:

• Start and finish timestamps for every recovery stage.
• Actual RTO and RPO achieved.
• Replication lag at the decision point.
• Terraform and image versions deployed in the recovery Region.
• Capacity and service-quota constraints encountered.
• Results of customer-journey checks.
• Reconciliation results for test records.
• Runbook gaps, owners, and due dates.

Test safely. Many teams validate the recovery stack through an isolated endpoint, separate test tenant, synthetic records, or a restricted traffic segment before they conduct a broader production game day. The purpose is not simply to prove that an ALB returns 200; it is to prove that the application can perform its critical business operation with the right data and controls.

Interview-ready response

If asked, “How would you design DR for a critical AWS workload?”, a strong answer is:

“I would first separate Multi-AZ high availability from regional disaster recovery. The primary Region would be multi-AZ to survive ordinary instance and AZ failures, but that does not protect us from a regional outage, data corruption, or account compromise.

I would run a business impact analysis by service tier. For the order path, if the requirement is a 15-minute RTO and less than one minute RPO, I would use a warm standby in a second Region: the VPC, security controls, ALB, ECS services, observability, and deployment artifacts are already deployed, while the recovery service fleet runs at reduced capacity. The data layer uses monitored cross-Region replication appropriate to the database technology, but I would retain independent point-in-time backups because replication can propagate deletions and corruption.

Backups would be centrally governed, copied to a separate backup account, protected by appropriate immutability controls, and periodically restored into an isolated environment. Failover would be manually authorized but automated through a runbook: detect customer impact, fence primary writes, establish the recovery Region as the single writer, scale services, shift traffic, validate critical transactions, and reconcile the recovery window.

Finally, I would measure the actual RTO and RPO through regular drills, track configuration drift and regional quotas, and treat failback as a deliberate data synchronization event rather than a DNS change.”

---

Key takeaways

• Multi-AZ provides high availability within a Region; it is not sufficient protection from a full regional outage, data corruption, or account compromise.
• RTO is the maximum tolerable service-restoration time; RPO is the maximum tolerable time span of data loss. Both must come from business impact and risk analysis.
• Select backup and restore, pilot light, warm standby, or active/active according to the required RTO/RPO, cost, and operational complexity. Do not over-engineer a lower-tier service or under-design a critical transaction path.
• Live replication reduces operational data loss but does not replace point-in-time backups, versioning, and isolated immutable recovery copies.
• A credible DR design includes infrastructure, application artifacts, data, secrets, keys, quotas, traffic routing, access controls, reconciliation, and failback.
• The only convincing RTO and RPO are measured through restoration and failover tests.

Next, you will bring the course together in an end-to-end AWS production-readiness review, defending your network, compute, data, security, observability, scaling, and DR decisions under senior-level interview questioning.