Designing a Highly Available and Resilient RDS Topology

Welcome back. In the previous lesson, you traced an EKS request from the load balancer through Ingress, Services, endpoints, Pods, and dependencies, with particular attention to readiness and safe recovery during failure. That matters here because a database failover is experienced first as an application dependency failure: connections break, pools refill, retries occur, and some requests may need safe replay.

This lesson focuses on designing the RDS layer for an exchange-like service where the database is the system of record for orders, balances, or other financially consequential state. You will distinguish availability from read scale and disaster recovery, select a topology based on explicit recovery targets, and explain how Multi-AZ, read replicas, backups, connection handling, and encryption fit together.

---

Start with failure targets, not an RDS feature

“Highly available RDS” is not a complete requirement. A senior answer begins with the failure that must be survived and the business impact that is acceptable.

Assume a PostgreSQL-backed order and account service has these requirements:

Requirement	Example target	Architectural consequence
AZ or primary-instance failure	No acknowledged committed order lost	Synchronous Multi-AZ replication for the writer
Database service recovery after an AZ failure	Database failover completes within roughly 2 minutes; application recovers within its service SLO	Automatic failover, connection retry behavior, and tested client recovery
Read-after-write for order status and balances	No stale results	Read from the writer after a mutation; do not use asynchronous replicas
Analytics and reporting	Can tolerate a few seconds or minutes of staleness	Route explicitly to a read replica or reader endpoint
Regional outage	RPO of 5 minutes, RTO of 30 minutes, for example	A regional DR design beyond Multi-AZ: replica or backup restoration plus a complete application failover procedure
Credential exposure	No passwords in code or containers	Secrets Manager and, where suitable, IAM database authentication
Data confidentiality	Encrypted in transit and at rest	TLS plus KMS-backed encryption, restricted networking, and key-policy controls

Three distinctions should be crisp in an interview:

1. High availability addresses instance, storage, and Availability Zone failures within one Region.
2. Read scaling handles excess or isolated read workload. It does not automatically provide strongly consistent reads.
3. Disaster recovery addresses failures beyond the Multi-AZ fault domain: regional disruption, destructive data changes, or a compromised primary environment.

A Multi-AZ deployment is essential for a critical regional database, but it is neither a read-scaling feature nor a complete regional-recovery plan.

---

The primary topology: choose a writer path deliberately

For a production RDS PostgreSQL or MySQL service with strict write correctness, there are two common starting points.

Option A: Multi-AZ DB instance plus selected read replicas

A traditional Multi-AZ DB instance has:

• One active primary instance serving reads and writes.
• One passive standby in another Availability Zone.
• Synchronous replication to the standby.
• Automated failover when RDS determines that the primary or its AZ is unhealthy.
• One stable writer endpoint whose DNS target changes during failover.

The standby is not a read-scaling node. It exists to preserve data and provide a managed failover target.

This is often the clearest choice when the critical workload is write-oriented and reads requiring strong consistency can remain on the writer. Add one or more asynchronous read replicas only for explicitly stale-tolerant traffic such as statements, reporting, historical views, fraud analysis, or internal dashboards.

Option B: Multi-AZ DB cluster

For supported RDS engines and Regions, a Multi-AZ DB cluster has:

• One writer in one AZ.
• Two readable standby instances in two other AZs.
• A writer endpoint for read-write traffic.
• A reader endpoint for traffic that can tolerate replication lag.
• Faster typical failover behavior than a traditional Multi-AZ instance, though actual recovery still depends on workload and replication state.

This can be a strong fit when both availability and sustained read capacity are central requirements. However, readable standbys do not eliminate the consistency question. If a user submits an order and immediately checks its state, that follow-up read should normally stay on the writer until your application has a defined and safe consistency model.

The AWS Database Blog comparison is worth reading because it makes the deployment differences concrete.

The standby-and-read-replica diagram captures the important separation of concerns.

The design implication is simple:

Use the standby to meet the AZ-failure availability objective. Use readable replicas only when the application can state precisely which reads may be stale.

---

Read replicas: useful, asynchronous, and never invisible

An RDS read replica is a separate read-only DB instance that receives changes asynchronously from its source. It can scale read-heavy workloads, isolate expensive reporting, and serve as a possible recovery asset. But asynchronous replication means a replica can lag, and the lag can differ across replicas.

For an exchange-like platform, classify queries before routing them:

Query category	Appropriate destination	Why
Create or amend an order	Writer endpoint	The operation changes authoritative state.
Immediately retrieve an order just submitted	Writer endpoint	It requires read-after-write consistency.
Current account balance used for risk controls	Writer endpoint	A stale balance can cause incorrect business decisions.
Historical statements or end-of-day reports	Read replica or reader endpoint	Some lag is generally acceptable if the product agrees.
Heavy internal analytics	Dedicated read replica	Protects the writer from long-running scans and I/O contention.

Do not use a read replica merely because it exists. A read-routing decision is a product and correctness decision.

Also, read replicas do not autoscale automatically. You must set scaling thresholds, forecast demand, create or resize replicas, and validate that the primary can sustain the replication workload. A lagging replica is not a healthy read path merely because its process is running.

What happens if a replica is promoted?

Promotion makes a read replica an independent read-write DB instance. It is useful in a controlled recovery scenario, especially in a secondary Region. But it is not an automatic substitute for Multi-AZ failover.

A safe promotion runbook includes:

1. Declare the regional recovery event and prevent new writes to the old primary if it might still be reachable.
2. Record replica lag and explicitly accept the potential data-loss boundary against the stated RPO.
3. Promote the selected replica and wait until it becomes available for writes.
4. Shift the application’s writer configuration and traffic to the recovery environment.
5. Validate core invariants: schema version, order sequences, balances, jobs, and downstream processing.
6. Prevent the recovered old primary from accepting writes during reconciliation. Otherwise, split-brain data is possible.

An important senior-level phrase is: “A promoted asynchronous replica gives an RPO bounded by observed replication lag, not zero.”

---

Failover is an application event, even when RDS automates it

For a Multi-AZ DB instance, RDS detects the failure, promotes the standby, and updates the DB endpoint to the new primary. The database endpoint stays logically stable, but the IP behind it changes.

That means:

• In-flight transactions and queries can fail.
• Existing direct client connections cannot be assumed to survive.
• Clients must establish new connections and resolve the endpoint again.
• Retried writes must be idempotent or protected by unique request identifiers.
• Connection pools must evict broken connections promptly rather than handing them repeatedly to callers.

For an order API, a client retry after a timeout is not automatically safe. The server may have committed the order just before the connection failed. Use an idempotency key or a durable client request identifier so a retry returns the existing result rather than creating a second order.

Use RDS Proxy as a connection and failover boundary

ECS or EKS can scale from a few tasks to many tasks quickly. If every task opens a large connection pool directly to RDS, a deployment or HPA event can exhaust database connections long before CPU or memory is saturated.

Amazon RDS Proxy sits between clients and RDS:

• It maintains a managed pool of database connections.
• It reduces connection-creation pressure on the database.
• It can improve application recovery behavior during a database failover.
• It supports Secrets Manager integration and can require IAM authentication.
• It is deployed across multiple AZs.

Watch this short AWS video segment for the operational role of the proxy.

RDS Proxy does not make failures disappear:

• A transaction active during database failover can still fail and require safe retry.
• The proxy does not decide which SQL queries are safe to send to a stale replica.
• Connection multiplexing can be reduced when application sessions use stateful behavior, such as long transactions, temporary objects, or session-specific settings.
• You still need to size the database and proxy limits under peak concurrency.

A practical connection budget starts with the maximum number of application instances, not today’s average replica count:

$$\text{maximum client connections}=\text{maximum tasks}\times \text{connections per task}$$

Then reserve capacity for migrations, administration, monitoring, and recovery operations. If 100 ECS tasks can each create 20 direct connections, the potential 2,000 connections may be unreasonable for the selected DB class. Cap pools per task, use backpressure, and load-test scaling events.

---

Backups provide recovery from the failures replication preserves

Replication protects availability. It does not protect against a faulty deployment that deletes valid rows, a bad migration, malicious writes, or logical corruption replicated to every copy.

Your topology therefore needs automated backups and a tested restore procedure.

A defensible baseline includes:

• Automated backups with point-in-time recovery retained for the period set by operational and regulatory needs.
• Manual snapshots before irreversible changes, such as a major schema migration or engine upgrade.
• Cross-Region and cross-account backup strategy when the regional RPO and compliance model require independent recovery copies.
• Restore tests that measure actual elapsed recovery time, including application validation and performance warm-up.
• Documented recovery ownership, including who can authorize a point-in-time restore and who can change DNS or service configuration.

Restoring a backup creates a new instance. It is not an instantaneous switch to a replacement writer. The actual RTO includes provisioning, restoration, security-group attachment, secret and configuration validation, database checks, application cutover, and a functional verification of critical flows.

Use this recovery mapping:

Failure scenario	Primary mechanism	Expected data-loss posture
DB host or AZ failure	Multi-AZ automatic failover	Typically no acknowledged committed transaction loss for the infrastructure event
Read overload	Read replica or Multi-AZ DB cluster reader capacity	Replica data may be stale
Bad deployment or accidental deletion	Point-in-time restore	Data loss bounded by the selected restore point and last restorable time
Regional outage	Cross-Region replica promotion or restored backup	RPO depends on replication lag or backup-copy cadence; RTO includes full regional cutover

Do not state that Multi-AZ “solves DR.” A strong correction is:

“Multi-AZ provides regional high availability across AZs. It does not provide a recovery environment for a Region-wide outage or protect against application-level data corruption. I use backups and, where justified by the RTO and RPO, a separately operated regional recovery path.”

---

Encryption and network controls belong in the topology

A production database topology is not only instances and replicas. It has access boundaries.

Encryption at rest

Enable RDS storage encryption with a KMS key from initial creation. This protects underlying storage, automated backups, read replicas created from encrypted sources, and snapshots according to the service’s encryption model.

The operational design must also control who can:

• Use the KMS key for encryption or decryption.
• Create, share, copy, restore, or delete snapshots.
• Modify an RDS instance, replica, proxy, or associated parameter group.
• Disable backup-related retention or deletion protections.

Treat KMS key policy and snapshot sharing as data-access controls, not merely infrastructure settings.

Encryption in transit and network path

Use private DB subnets across at least two AZs. Avoid public database accessibility for the production writer.

A clean security-group pattern is:

Component	Inbound rule
Application security group	Receives only required traffic from the load-balancer or internal callers
RDS Proxy security group	Database port allowed only from the application security group
RDS DB security group	Database port allowed only from the RDS Proxy security group
Administrative access	No broad network rule; use tightly controlled operational access and audited tooling

Require TLS from application to proxy or database, and configure clients to validate the server certificate rather than merely encrypting the transport without identity verification.

For credentials, use Secrets Manager rather than environment variables containing static database passwords. Where engine and application-driver support make it appropriate, require IAM authentication at the proxy boundary. Independently of AWS IAM, maintain database-native least-privilege roles: an order writer, a read-only reporting user, migration identity, and monitoring identity should not share broad administrative privileges.

---

Operational controls that prove the design works

A topology is only credible if its failure signals and tests are defined.

Monitor at least:

• Writer health: CPU, free memory, free storage, read and write latency, I/O pressure, deadlocks, and database connection count.
• Read path health: replica lag, replica availability, read latency, and replica connection saturation.
• Failover health: RDS events, application database-connection errors, reconnection duration, retry rate, and user-facing error rate.
• Backup health: most recent restorable time, backup job failures, snapshot-copy status, and restore-test results.
• Proxy health: client connections, database connections, connection borrow latency, and pool saturation.

The most useful availability test is not simply “Can RDS fail over?” It is:

“Can the API continue to process an idempotent customer request correctly when the writer fails over under production-like connection volume?”

Run controlled tests in a non-production environment and, where governance permits, planned production game days:

1. Trigger or observe a controlled failover.
2. Measure the interval from the first connection error to successful critical requests.
3. Verify the application did not duplicate writes.
4. Verify connection pools recovered without a task restart storm.
5. Confirm dashboards, alerts, and incident runbooks led operators to the right evidence.
6. Restore a backup into an isolated environment and validate both data correctness and recovery duration.

---

A concise interview architecture defense

For the question, “How would you design RDS for a critical exchange API?”, a senior answer could be:

“I begin with failure targets. For the authoritative order and balance store, I need no acknowledged committed-write loss during an instance or Availability Zone failure, so I use an encrypted Multi-AZ RDS writer deployed privately across AZs. The application uses the stable writer endpoint through RDS Proxy, with bounded connection pools, TLS, Secrets Manager, and either IAM authentication or tightly scoped database credentials.

I do not route consistency-sensitive reads, such as post-trade order status or risk balances, to asynchronous replicas. I reserve read replicas or a Multi-AZ DB cluster reader endpoint for traffic that the product explicitly allows to be stale, such as reporting. I monitor replica lag and remove a lagging replica from noncritical routing.

Multi-AZ handles an AZ-level failure but not regional loss or logical corruption. Automated backups and point-in-time restore cover accidental changes. For a regional RTO and RPO that justify it, I maintain and test a cross-Region recovery path, with promotion, writer fencing, traffic cutover, and reconciliation steps. Finally, I test failover from the client perspective, because recovery depends on DNS behavior, driver retries, connection pools, and idempotent write handling as much as on the RDS failover itself.”

---

Key takeaways

• A Multi-AZ standby is for availability, not read scaling. It synchronously protects the writer within a Region and supports automatic failover.
• Read replicas are for read scale, workload isolation, and possible recovery use, but they replicate asynchronously and can return stale data.
• Multi-AZ DB clusters can provide readable standby capacity for supported engines, but reader traffic still needs an explicit consistency policy.
• RDS failover breaks or disrupts database work at the application boundary. Stable endpoints, reconnection logic, bounded pools, idempotency, and RDS Proxy determine the customer-visible recovery time.
• Backups and point-in-time recovery protect against logical errors that replication cannot solve; cross-Region recovery is a separate design with its own RPO and RTO.
• A secure topology uses private subnets, security-group chaining, TLS, KMS encryption, managed secrets, and least-privilege database and IAM identities.
• The design is credible only when failover, restore, replica-lag, and client-recovery behavior are monitored and tested.

Next, you will shift from runtime topology to change control: structuring reusable Terraform modules and isolated environments so that production infrastructure can be evolved safely.