Replication Patterns for Availability and Scalability

Hello! Welcome back to our course on system design.

In our last lesson, we tackled the challenge of scaling a database by partitioning data across multiple servers. We learned how sharding (horizontal partitioning) and vertical partitioning help us handle massive amounts of data and traffic. However, this introduced a new problem: if one of our partitions (or shards) fails, that piece of the system becomes unavailable, and we could lose data permanently.

Today's lesson provides the solution: data replication. We will create copies of our data to ensure that our system remains available and durable, even when hardware fails.

Our learning outcome is to apply replication patterns (leader-follower, leader-leader, leaderless) for availability and read scalability. We will explore these three fundamental architectures, understand their distinct trade-offs, and learn how to choose the right one for a given problem.

1. What is Database Replication?

At its core, replication is the process of keeping identical copies of your data on multiple database servers, which are called replicas. Why go to this trouble? There are two primary benefits we'll focus on:

1. High Availability & Durability: If one server fails, a replica can take its place, allowing the application to continue running with minimal disruption. This prevents data loss and downtime.
2. Read Scalability: For applications that have many more reads than writes (like a blog or an e-commerce catalog), we can direct read requests to the replicas. This distributes the load, improving performance and allowing the system to serve more users.

To start, let's watch a short video that provides a high-level overview of replication.

As the video explains, replication is a fundamental strategy for building robust systems that can handle failures and scale effectively.

2. An Overview of Replication Patterns

There are three main architectural patterns for managing how data is written to and copied among replicas. Let's get a quick visual sense of them before we dive into the details of each.

We will use the more modern terminology of Leader/Follower, but it's good to know the older Master/Slave terms as you will frequently encounter them. Now, let's explore each pattern in detail.

3. Pattern 1: Leader-Follower Replication

This is the most common replication pattern. The idea is simple: one replica is designated as the leader (or master), and all other replicas are followers (or slaves).

• Write operations must go to the leader.
• The leader records the change and then sends it to all its followers.
• Read operations can be served by the leader or any of the followers.

This model is excellent for read-heavy workloads because you can add more followers to scale out your read capacity.

Synchronous vs. Asynchronous Replication

A critical decision in this pattern is how the leader replicates data to its followers.

• Synchronous Replication: The leader waits for confirmation from one or more followers that they have received the write before confirming success to the client.

  • Pro: Guarantees that the data on the follower is perfectly up-to-date. If the leader fails, no data is lost. This provides strong consistency.
  • Con: Increases write latency, as the client must wait longer. A slow or failed follower can block all writes.

• Asynchronous Replication: The leader sends the update to followers and immediately confirms success to the client without waiting for a response.

  • Pro: Very low write latency.
  • Con: There is a replication lag. If the leader fails before the data has been replicated, the recent writes are lost. This leads to eventual consistency.

Failure Handling: Failover

If the leader fails, the system must perform a failover:

1. One of the followers is promoted to become the new leader.
2. The application clients are re-routed to send writes to the new leader.
3. The other followers start consuming updates from the new leader.

This process can be complex. The system needs to reliably detect the leader's failure and have a mechanism (often an election among followers) to choose the most up-to-date follower as the new leader.

4. Pattern 2: Leader-Leader Replication

What if having a single leader for writes becomes a bottleneck or a single point of failure? The leader-leader (or multi-master) pattern addresses this by allowing more than one node to act as a leader.

• Write operations can be sent to any leader node.
• Each leader is responsible for replicating its writes to all other leaders.

This pattern is often used in globally distributed systems where you want users to have a low-latency write experience by connecting to a nearby datacenter (which has its own leader).

The Challenge: Write Conflicts

The main benefit of leader-leader replication—multiple write locations—is also its greatest challenge. What happens if two users edit the same piece of data at the same time, but their requests go to two different leaders? This is a write conflict.

Since the leaders replicate changes to each other asynchronously, the conflict is only detected after the fact. The system needs a conflict resolution strategy to decide which write "wins." Common strategies include:

• Last Write Wins (LWW): The write with the latest timestamp is kept, and the other is discarded. This is simple but can lead to data loss if the "loser" write contained important information.
• Custom Logic: The application is responsible for merging the conflicting changes, possibly by asking the user to resolve it.

5. Pattern 3: Leaderless Replication

The leaderless pattern, popularized by Amazon's DynamoDB, takes a radical approach: there are no leaders. All replicas are equal.

• Write and read operations can be sent by the client to any replica.
• To ensure consistency without a leader, this pattern uses a quorum for read and write operations.

Quorum Reads and Writes

Here’s how a quorum works:

• N = The number of replicas for a piece of data.
• W = The write quorum. A write must be successfully acknowledged by at least W replicas to be considered successful.
• R = The read quorum. A read request is sent to N replicas, but the client waits for responses from at least R replicas and uses the most recent version of the data.

The key is the formula: W + R > N. This ensures that the set of nodes you read from (R) and the set of nodes you wrote to (W) always have at least one node in common. This guarantees that a read will always see the most recent successful write.

Example:
Imagine N=3, W=2, R=2.

• A client writes a value. The write must succeed on at least 2 out of 3 replicas.
• Later, a client reads that value. It must get a response from at least 2 out of 3 replicas.
• Because 2 + 2 > 3, the set of nodes read from is guaranteed to overlap with the set of nodes written to. The client can then identify and return the latest value.

This pattern provides extremely high availability for both reads and writes. An operation only fails if fewer than W or R nodes are available.

6. Applying the Patterns: A Comparison

Choosing a replication pattern is a critical design decision that involves balancing consistency, availability, latency, and operational complexity. There is no single "best" pattern; the right choice depends entirely on your system's requirements.

Let's solidify this with a few scenarios. Based on what you've learned, which pattern would you choose for each of the following systems?

1. A banking application processing financial transactions.
   • Hint: What is the tolerance for data loss or inconsistency?
2. A social media site's "like" button.
   • Hint: Is it more important that every "like" is instantly successful, or that the "like" count is always 100% accurate across the globe in real-time?
3. A collaborative document editor like Google Docs, used by a global team.
   • Hint: What are the requirements for write latency for users in different regions? What is the main technical challenge to solve?

Think about your answers before revealing the suggestions below.

Click to see suggested answers

1. Banking Application: Leader-Follower with Synchronous Replication. Consistency is paramount. You cannot afford to lose transactions or have conflicting balances. The higher write latency is an acceptable trade-off for correctness.
2. "Like" Button: Leaderless Replication. High availability is key. You want the user's "like" to register successfully even if some servers are down. Eventual consistency is perfectly acceptable; it doesn't matter if another user sees the updated count a few seconds later.
3. Collaborative Editor: Leader-Leader Replication. This allows users in different geographic regions to write to a local leader, providing low latency. The core challenge then becomes designing a robust conflict resolution strategy to merge simultaneous edits from different users.

Conclusion

Today we've seen how data replication is the key to building available and scalable data storage systems. You've learned the fundamental trade-offs between three major replication patterns.

Key Takeaways:

• Replication provides availability (fault tolerance) and read scalability by keeping copies of data on multiple servers.
• Leader-Follower is the most common pattern, ideal for read-heavy workloads. It offers strong consistency with synchronous replication or low write latency with asynchronous replication.
• Leader-Leader provides high write availability and low latency for globally distributed applications, but at the cost of complex conflict resolution.
• Leaderless offers the highest availability for both reads and writes by eliminating single points of failure. It relies on quorums to manage consistency, which is often eventual.
• The choice of pattern is a trade-off. You must analyze your system's specific needs for Consistency, Availability, and Performance.

Preview of the Next Lesson:

We've now covered how to partition our data (sharding) and how to make those partitions fault-tolerant (replication). Next, we will zoom back in to look at the data inside a single database. We will learn how to choose between normalized and denormalized data models based on access patterns. This will give us another powerful tool to optimize our system's performance, especially for read-heavy queries.