Defending an End-to-End AWS Production-Readiness Review

Welcome. This lesson is a senior-level rehearsal: not a tour of AWS services, but a way to conduct a production-readiness review that turns architectural choices into explicit risks, controls, evidence, and business trade-offs.

For an exchange or high-value transaction platform, the central discipline is to avoid treating “high availability” as a feature list. A credible review distinguishes an Availability Zone failure from a regional disaster, a fast read API from an irreversible order-placement workflow, and a technically possible design from one the team can safely operate at 2 AM.

By the end, you should be able to give a structured 6–8 minute architecture review, then defend its key choices under follow-up questions about EKS, data, messaging, security, disaster recovery, and cost.

---

Treat the review as a decision meeting, not an architecture tour

A strong production-readiness review produces three things:

1. A decision record: what has been chosen, why it fits the requirements, and what alternatives were rejected.
2. A risk register: known failure modes, their impact, mitigation, owner, and accepted residual risk.
3. An evidence plan: what proves the system is ready, such as load-test results, restore-test evidence, access reviews, deployment records, and game-day outcomes.

Use the AWS Well-Architected Framework as a consistent lens across operational excellence, security, reliability, performance efficiency, and cost optimization. It is useful because it prevents a review from becoming dependent on whoever is in the room or on the most recent incident.

A practical opening statement

In an interview, begin by setting assumptions and separating critical paths. For example:

“I will assume this is an internet-facing exchange platform with a globally distributed user base, but with primary trading activity in one AWS Region. The design must tolerate an AZ loss without customer-visible interruption for critical APIs. Order submission and balance changes have stricter durability and audit requirements than market-data reads or the web UI. I will use a single write authority for financial state, and I will call out where regional recovery has a non-zero data-loss risk.”

This opening does four useful things:

• Establishes that requirements precede technology selection.
• Separates transactional correctness from general web availability.
• Avoids falsely promising “zero data loss” across Regions without defining the replication and consistency model.
• Invites the interviewer to correct an assumption rather than exposing an unspoken one later.

A senior reviewer should ask for measurable targets before approving an architecture:

Dimension	Example question	Why it changes the design
Availability	Which APIs require 99.9%, 99.95%, or higher availability?	Determines redundancy, capacity, and error-budget policy.
Latency	What are the p95 and p99 targets for order acceptance, account queries, and market data?	Determines caching, data locality, network choices, and synchronous dependencies.
Throughput	What are normal and peak orders per second, read requests per second, and event volume?	Drives partitioning, autoscaling, queue or stream selection, and load tests.
Data integrity	Can an accepted order ever be lost or duplicated?	Determines idempotency, transactional boundaries, and reconciliation controls.
Recovery	What RTO and RPO apply to each service and data class?	Determines whether backup and restore, pilot light, warm standby, or active/active is justified.
Compliance	Which audit, retention, encryption, residency, and segregation-of-duty controls apply?	Shapes account boundaries, logging, IAM, key policies, and immutable evidence.
Cost	What cost is acceptable per successful order, active customer, or unit of volume?	Prevents uncontrolled overprovisioning and makes resilience trade-offs explicit.

Do not accept “zero downtime” as a requirement without clarification. Ask whether it means no planned downtime, no single-AZ outage, no regional outage, or no interruption even during a bad deployment. Those are different engineering commitments.

---

Establish the baseline platform: isolate failure domains deliberately

For a production exchange platform, a defensible baseline is a multi-account AWS environment:

• Production workload account for application compute and data services.
• Network account where an enterprise model requires centralized connectivity, inspection, or Transit Gateway ownership.
• Security and log archive accounts for centrally retained CloudTrail, security findings, and tamper-resistant audit logs.
• Shared services or CI account for controlled build infrastructure, if organisational separation warrants it.

Within the production Region, use at least three Availability Zones when the Region and services support them. A typical VPC has:

• Public subnets: internet-facing ALBs, NAT Gateways, and only explicitly public edge components.
• Private application subnets: EKS worker nodes or ECS tasks, internal load balancers, and application services.
• Isolated data subnets: RDS, caches, MSK brokers, and other data-plane services that do not need default internet egress.

The public subnet is not “where public applications run.” It is where internet-routable infrastructure belongs. Application containers and database instances should not receive public IP addresses.

For EKS specifically, understand the boundary clearly: AWS operates the managed Kubernetes control plane, but you remain responsible for your VPC, worker nodes or compute configuration, workload configuration, identity design, container images, scaling choices, and operational controls.

Network choices you should be ready to defend

Ingress

For typical HTTPS APIs:

• Use Route 53 for DNS and health-aware routing decisions.
• Use CloudFront for static assets, cacheable public content, and edge protection where it materially improves user latency or absorbs traffic.
• Put AWS WAF in front of public endpoints for managed rules, rate limits, bot controls where appropriate, and application-specific protections.
• Use an ALB for HTTP/HTTPS routing, path- or host-based routing, and Kubernetes Ingress integration.
• Use an NLB only where Layer 4 behavior, static IPs, TLS pass-through, very high connection rates, or non-HTTP protocols are actual requirements.

Do not say “NLB is always faster.” A better answer is: “For HTTP APIs needing routing rules and WAF integration, I would choose ALB. I would introduce NLB for a specific Layer 4 or fixed-IP requirement, rather than as a default performance optimization.”

Egress

Private workloads commonly need ECR, S3, CloudWatch, STS, Secrets Manager, KMS, and other AWS services. Add VPC endpoints for the AWS services used by the workload. This can reduce NAT dependency, improve the network control boundary, and avoid unnecessary NAT data-processing charges.

A NAT Gateway still has a role for approved external destinations. For resilience, each private subnet should route to a NAT Gateway in its own AZ. Otherwise, an AZ loss can also remove egress for workloads in another AZ, while cross-AZ routing adds cost and an avoidable dependency.

Security groups and network ACLs

Use security groups as the principal, stateful control:

• The public ALB permits inbound HTTPS.
• Application workloads accept traffic only from the ALB security group or an explicitly approved internal caller.
• Databases accept only the required database port from the application security group.
• Administrative node access is through Systems Manager Session Manager, not public SSH.

Network ACLs are stateless and subnet-scoped. Use them sparingly for coarse guardrails or explicit organizational controls; do not make them the primary way to express application connectivity. A readiness reviewer will ask whether ephemeral return ports and both traffic directions have been accounted for.

---

Make the container decision based on operating requirements

An interview-quality architecture does not select EKS merely because Kubernetes appears in the job description.

Choose ECS on Fargate when the workload is primarily stateless HTTP or worker services, the organization does not need Kubernetes-specific extensibility, and reducing cluster operations is more valuable than Kubernetes portability or scheduling controls.

Choose EKS when there is a real platform need: several teams require Kubernetes APIs and ecosystem tooling, advanced scheduling, custom controllers, standardised policy enforcement, service-mesh capabilities, or a mature existing Kubernetes operating model.

For this rehearsal, assume EKS is justified because the platform hosts multiple independently deployed services and needs a standard deployment and policy model.

A production EKS review should verify:

• Managed node groups or an equivalent managed compute strategy span three AZs.
• Critical add-ons, such as CoreDNS and the AWS Load Balancer Controller, have multiple replicas and appropriate priority.
• Application deployments use multiple replicas across AZs with topology spread constraints.
• Pod Disruption Budgets protect availability during node rotation and upgrades, without blocking all maintenance.
• Liveness probes restart a genuinely unhealthy process; readiness probes remove a pod from traffic until it can safely serve requests.
• Horizontal Pod Autoscaling uses a meaningful signal, such as CPU plus request rate, queue depth, or custom application metrics.
• Cluster capacity can scale quickly enough for peak demand, and workload resource requests are realistic.
• System workloads and critical platform add-ons have reserved capacity rather than competing with business workloads during a surge.
• Pods use narrowly scoped AWS roles through EKS Pod Identity or IAM roles for service accounts, rather than inheriting node-role permissions.

A frequent failure mode is confusing a healthy Kubernetes Deployment with a healthy customer journey. A pod may be Ready while its dependency pool is exhausted, its downstream database is slow, or it is returning semantically invalid responses. Readiness must reflect the ability to serve the defined request safely, not merely that a process is listening on a port.

---

Protect correctness at the data and messaging boundary

The most important design distinction is between synchronous commands that change financial state and asynchronous events that distribute, notify, or reconcile that state.

Synchronous transaction path

For an order submission, the service should:

1. Authenticate and authorize the caller.
2. Validate account status, risk controls, and request schema.
3. enforce idempotency using a client-supplied idempotency key or a server-generated durable request identifier.
4. Commit the authoritative transaction to the system of record.
5. Return an acceptance response only after the system can durably account for the request.

An idempotency key protects against retries caused by client timeouts, load balancer retries, or a response lost after the server committed. It does not replace transaction design, reconciliation, or auditability.

For relational state, a production RDS design commonly includes:

• Multi-AZ deployment for automatic failover within a Region.
• Automated backups and point-in-time recovery.
• Encryption at rest with KMS and TLS in transit.
• Connection pooling or RDS Proxy when connection storms are plausible.
• Read replicas for read scaling, reporting isolation, or regional recovery needs.
• Monitoring for connection saturation, lock waits, slow queries, storage pressure, replication lag, and failover events.

Be exact in an interview: Multi-AZ provides regional high availability against an instance or AZ-level failure; it is not a cross-Region disaster-recovery strategy, and it is not a read-scaling mechanism.

Asynchronous events and messaging

After the authoritative transaction is committed, other systems may need an event: market-data distribution, notifications, fraud analytics, settlement, reporting, or audit processing.

Avoid a dual-write failure in which the database commits but the event publish fails. A common solution is a transactional outbox:

• Commit the business record and an outbox event in the same database transaction.
• Reliably publish outbox records through a relay process.
• Make consumers idempotent, since retries and at-least-once delivery can occur.
• Track consumer lag, failed events, duplicates, and reconciliation status.

Select the messaging technology based on delivery semantics:

Requirement	Suitable direction	Key control
Command queue with strict per-entity ordering and controlled throughput	SQS FIFO	Use an appropriate message-group key and idempotent consumers.
High-throughput event stream, replay, multiple consumers, and partitioned ordering	Amazon MSK or another Kafka-compatible managed platform	Choose partition keys deliberately; monitor consumer lag and replication health.
Lightweight event fan-out to independent subscribers	EventBridge or SNS, depending on routing and delivery requirements	Define retry, failure destination, and consumer ownership.

Kafka ordering applies within a partition, not globally. If order matters for a customer account or trading instrument, choose a stable partition key that preserves the ordering boundary you actually need. Global ordering is costly and often unnecessary.

A dead-letter queue is not a success state. A readiness review must establish who owns replay, how malformed messages are corrected, how duplicate processing is prevented, and how financial records are reconciled after a failure.

---

Make security and observability part of the design, not post-launch controls

Security controls should map to specific identities and trust boundaries.

For engineers:

• Federated SSO with MFA.
• Short-lived role credentials rather than IAM users and long-lived access keys.
• Separate read-only, deployer, break-glass, and security-audit roles.
• Permission boundaries or service control policies where the organization needs guardrails.
• Time-bound, logged emergency access with a post-use review.

For CI/CD:

• Use OIDC federation from the pipeline to an IAM role.
• Scope deployment roles to a specific environment and permitted resources.
• Separate the role that builds artifacts from the role that deploys production.
• Prevent pipelines from reading unrelated production secrets or assuming unrestricted administrator roles.

For workloads:

• Assign a role to each workload identity.
• Store credentials and sensitive configuration in Secrets Manager or Parameter Store with KMS protection.
• Retrieve secrets at runtime using tightly scoped permissions; do not bake secrets into container images, source control, task definitions, or Kubernetes manifests.
• Rotate secrets and prove that rotation works without customer impact.

Observability must answer operational questions

For each service, define:

• SLIs: availability, successful order acceptance, latency, queue age, processing lag, and correctness or reconciliation measures.
• SLOs: the target and measurement window for each SLI.
• Alerts: actionable signals that indicate customer impact or rapid error-budget consumption.
• Dashboards: a service view, dependency view, and business-journey view.
• Logs and traces: correlated by request ID, order ID, customer-safe identifiers, and deployment version.

For an order API, a useful availability SLI is not merely HTTP 200 rate. It may be the proportion of valid order requests that are durably accepted or correctly rejected within the agreed latency target. A 200 response that accepts an order but fails to persist it is worse than a visible 5xx.

A minimum observability stack should correlate:

• ALB request count, target response codes, latency, and rejected connections.
• EKS node and pod saturation, restarts, pending pods, and autoscaling events.
• Application error rate, latency percentiles, dependency calls, and business failures.
• RDS connections, failovers, locks, query latency, and replication lag.
• Queue depth, age of oldest message, consumer lag, retry rate, and DLQ volume.
• CloudTrail activity, EKS audit logs, GuardDuty findings, and unusual authorization failures.

Use both symptom-based alerts and burn-rate alerts. A symptom alert detects a current customer impact, such as a spike in failed order submissions. A burn-rate alert detects that the service is consuming its error budget so quickly that an outage is likely or already developing, even before a long-window SLO is technically breached.

---

Connect disaster recovery to specific RTO and RPO commitments

High availability and disaster recovery solve different problems:

• Multi-AZ handles local infrastructure and AZ failures within a Region.
• Backups provide recovery from deletion, corruption, and some security incidents, but recovery takes time.
• Cross-Region recovery addresses a regional outage, but the achievable RPO depends on replication design and the RTO depends on how much infrastructure is already running.

A sensible initial strategy for a critical, single-writer exchange service may be active/passive warm standby:

• The primary Region serves production traffic.
• A secondary Region contains pre-provisioned network, IAM, observability, container, and baseline capacity infrastructure through Terraform.
• Container images, infrastructure modules, secrets configuration, and operational runbooks are available in the recovery Region.
• Critical data is replicated or recoverable according to the stated RPO.
• Route 53 or another controlled traffic-management mechanism supports a tested failover procedure.
• Only one Region is permitted to accept authoritative writes at a time.

The final point matters. Multi-site active/active can reduce recovery time, but an active/active write design introduces distributed consistency, duplicate processing, conflict resolution, traffic steering, and operational complexity. Do not recommend it casually.

A strong answer sounds like this:

“For an RTO of 30 minutes and an RPO of five minutes, I would begin with warm standby, cross-Region replication for critical data, pre-provisioned infrastructure, and rehearsed failover. I would measure actual replication lag and restore time. If the business requires zero regional data loss for acknowledged financial transactions, asynchronous replication is insufficient; the requirement needs a more explicit consistency and acknowledgement design, with materially higher cost and complexity.”

The DR strategy is incomplete until tested. Required evidence includes:

• Restore test reports with measured restoration time.
• A controlled regional failover exercise.
• Documented DNS and traffic-management behavior.
• Validation that applications can start with recovery-region configuration and secrets.
• Reconciliation procedures for transactions around the failover boundary.
• A decision on who can declare disaster and authorize failover.

---

Review failure modes before they become incidents

A senior review anticipates failure chains rather than naming isolated components.

Failure mode	Expected behavior	Readiness evidence
One Availability Zone fails	Remaining pods, load balancer targets, nodes, and data services continue within capacity limits.	AZ-failure or capacity simulation; topology configuration; load-test headroom.
Bad application deployment	Health checks stop traffic to unhealthy version; rollout halts or reverts without breaking compatible clients.	Canary or blue/green deployment record; tested rollback; backward-compatible database migration.
RDS failover	Clients reconnect safely; idempotency prevents duplicate commands; latency impact is understood.	Failover test, connection-pool configuration, and observed recovery time.
Consumer failure or traffic surge	Queue retains work; autoscaling reacts; lag alerts fire before retention or business deadlines are breached.	Load test, lag dashboard, scaling evidence, and replay runbook.
Dependency degradation	Circuit breaking, timeouts, bounded retries, and graceful feature degradation prevent cascading failure.	Dependency fault injection and trace evidence.
Credential compromise	Access is detected, credentials can be revoked, activity is audited, and blast radius is contained.	CloudTrail and GuardDuty coverage, incident playbook, role design, and access review.
Regional outage	Recovery Region is activated within RTO; data loss is measured against RPO; write authority is fenced.	Game-day results and a signed recovery runbook.

The phrase to remember is: an untested recovery path is an unproven recovery path. Backups, autoscaling, failover routing, and rollback mechanisms are only architecture diagrams until they have been exercised.

---

Explain delivery, governance, and cost as operational controls

A senior DevOps engineer owns the management plane with the same rigor as the application plane.

Safe delivery

A production pipeline should use:

• Immutable artifacts, ideally promoted by image digest rather than rebuilt per environment.
• Terraform plans reviewed before apply, with remote encrypted state, locking, restricted access, and clear state ownership boundaries.
• Unit, integration, security, image-vulnerability, and policy checks.
• Environment-specific approvals for higher-risk production changes.
• Progressive delivery through rolling, canary, or blue/green deployment, selected by risk and platform capability.
• Automated health verification based on customer-impact metrics, not only pod status.
• Rollback procedures that are tested and understood.

For database changes, use the expand-contract pattern:

1. Add backward-compatible schema or data structures.
2. Deploy application code that works with both old and new forms.
3. Migrate and validate data.
4. Remove deprecated paths only after all consumers are compatible.

Rolling back application code does not necessarily roll back a destructive database migration. Calling out that distinction signals production maturity.

Cost without weakening resilience

Cost optimization is not “remove redundancy.” It is choosing the least expensive design that meets explicitly stated reliability and performance requirements.

Track cost by service and, where possible, by business unit such as cost per successful order, active customer, or million market-data events. Review:

• Rightsizing based on actual requests and limits, not only node utilisation.
• Scaling floors for critical services and scaling ceilings to control runaway failure costs.
• Savings Plans or reserved capacity for predictable baseline workloads.
• On-Demand capacity for critical, interruption-sensitive components.
• Spot capacity only for workloads engineered to tolerate interruption.
• NAT, cross-AZ, data-transfer, logging, and stream-retention costs.
• VPC endpoints when they reduce unnecessary NAT dependency and transfer costs.
• Storage lifecycle controls for logs and non-regulated data, while preserving mandated retention.

A credible trade-off statement is: “I will not use Spot capacity for the authoritative transaction path. I may use it for replayable analytics or asynchronous consumers with sufficient On-Demand baseline capacity and interruption testing.”

---

Deliver and defend the review in the interview

Use this five-part answer structure whenever challenged:

1. Decision: state the architecture choice.
2. Rationale: connect it to a measurable requirement.
3. Failure mode: state what can still go wrong.
4. Control and evidence: explain mitigation and how it is verified.
5. Trade-off: identify the cost, complexity, or limitation honestly.

For example:

“I would place EKS nodes in private subnets across three AZs, with public ALBs only in public subnets. This prevents direct internet access to workloads while retaining internet-facing API access. The remaining risk is loss of outbound connectivity or a NAT dependency during an AZ event, so I use VPC endpoints for AWS service traffic and a NAT Gateway in each AZ for approved external egress. I would validate this with route-table review, endpoint coverage, and an AZ-resilience test. The trade-off is endpoint and NAT cost, which is justified for the critical production path.”

Common interview challenges

“Why not simply use Multi-AZ and call it disaster recovery?”
Multi-AZ protects against an infrastructure or AZ failure in one Region. It does not recover the platform from a regional outage, account compromise, or some forms of logical corruption. Regional DR needs separate recovery-region capacity, data recovery or replication, traffic failover, and tests.

“Why do you need EKS instead of ECS?”
Do not defend EKS as a default. State the actual platform requirement. If Kubernetes extensibility and multi-team policy controls are not needed, ECS can reduce the operational burden. The appropriate choice is the simplest platform that meets workload and organisational needs.

“How do you avoid duplicate order processing?”
Use durable idempotency keys at the command boundary, transactional persistence, idempotent event consumers, and reconciliation. Messaging alone cannot guarantee business-level exactly-once outcomes.

“How would you prove this platform is ready?”
Show evidence: Terraform plan and ownership controls, threat-model and IAM review, performance test against peak plus headroom, restore and failover results, deployment rollback test, SLO dashboards, alert tests, and game-day reports.

Before the interview, practise delivering the architecture in one pass without opening the AWS console. Keep the explanation centred on customer journeys and failure handling, then use AWS service details to support each decision.

---

Key takeaways

A production-readiness review is a structured engineering decision process, not a checklist of AWS services.

• Start with differentiated SLOs, RTOs, RPOs, compliance needs, and workload assumptions.
• Design for AZ failure, but explicitly plan and test regional recovery separately.
• Keep workloads private, minimize ingress and egress paths, and use IAM roles with short-lived credentials.
• Treat transactional correctness, idempotency, event delivery, reconciliation, and auditability as first-class design concerns.
• Make operational evidence mandatory: load tests, restore tests, failover drills, access reviews, deployment verification, and game days.
• Defend choices through requirements, remaining failure modes, verification evidence, and honest trade-offs.

The next lesson moves from architecture defense to incident leadership: hypothesis-driven diagnosis, safe mitigation and rollback, stakeholder communication, evidence preservation, root-cause analysis, and preventive action.