Eventual Consistency: Trade-offs & UX Mitigation

Hello! Let's dive into the final lesson of our module on Distributed Data Management.

In our last session, we implemented the CQRS pattern. We saw how separating command and query responsibilities allows for independent scaling and optimization. A key consequence of this separation, especially when linked by an asynchronous message bus like Kafka, is that the system becomes eventually consistent. There's a natural delay between when a command is executed on the write side and when that change becomes visible on the read side.

Today, we'll tackle this consequence head-on. Our learning outcome is to analyze the trade-offs of eventual consistency and describe strategies to mitigate user-experience issues. This is a critical topic in system design interviews, as it demonstrates your ability to think through the practical implications of architectural decisions.

1. Understanding Eventual Consistency and Its Trade-offs

At its core, eventual consistency is a guarantee that, if no new updates are made to a given data item, all reads of that item will eventually return the last updated value. The key word is "eventually"—for a period of time, different parts of the system may have different versions of the data.

This isn't a bug; it's a deliberate design choice. In distributed systems, we often trade strong consistency for other benefits.

The Trade-off:

We Gain (Benefits)	We Pay (Challenges)
High Availability & Performance: The system remains responsive. Write operations can return success immediately without waiting for all read models to be updated.	User Experience (UX) Issues: Users might see stale data after performing an action, leading to confusion. This is our main focus today.
Scalability: The write side (commands) and read side (queries) can be scaled independently based on their specific loads.	Increased Complexity: Developers must design for and reason about temporary data inconsistencies.
Resilience: Services are decoupled. The failure of a downstream read-model projector (e.g., a service for analytics) won't impact the primary service's ability to accept writes.	Read-Your-Own-Writes Problem: A user updates data but their subsequent read request is served old data from a replica that hasn't synced yet.

To get a formal definition and a practical example, let's start with a foundational reading.

This visual can also help you understand the different situations where eventual consistency manifests.

2. Strategies to Mitigate User-Experience Issues

Since eventual consistency can create a confusing or frustrating experience, we need strategies to manage it. The right strategy depends on the business context and user expectations. A "like" button not updating for a second is acceptable; a new bank transaction not appearing is not.

Let's explore several common strategies, ranging from simple to complex.

Let's summarize and expand upon these strategies:

Strategy 1: Inform the User

This is the simplest approach. After a user performs an action, the UI displays a message like, "Your request is being processed and will appear shortly."

• Pros: Easy to implement, sets clear expectations.
• Cons: The user has to trust the system. It can feel clunky if overused. What happens if the background processing fails? You need a separate mechanism (e.g., notifications, emails) to handle failures.

Strategy 2: Optimistic UI ("Faking it")

This is a very common and effective pattern. The UI updates immediately, assuming the operation will succeed. In the background, the request is sent to the server. If the server returns an error, the UI reverts the change and notifies the user.

• Pros: Creates a very responsive, snappy user experience. The user feels their actions have an immediate effect.
• Cons: More complex to implement on the client-side, as you need to handle state rollbacks on failure. It's not suitable for critical operations where success must be confirmed before proceeding (e.g., a payment).

Strategy 3: Synchronous Polling ("Waiting")

After the write request is sent, the client-side code can poll the query API endpoint in a loop until the updated data is returned. A loading spinner can be shown during this time.

• Pros: Guarantees that the user sees the updated state once the loading is complete.
• Cons: It's a blocking operation from the user's perspective, making the application feel slower. You need to implement a timeout to avoid polling indefinitely if something goes wrong.

3. A Deeper Dive: The Read-Your-Own-Writes Problem

The most jarring UX failure is when a user makes a change and immediately sees the old state. This is called the Read-Your-Own-Writes (RYOW) problem. It violates a user's natural expectation that their own actions are immediately visible to them.

Because this is such a critical issue, there are specific strategies to solve it, even within an eventually consistent system.

Based on that reading, here are two of the most common solutions you'd discuss in an interview:

1. Read from the Write Model: After a user performs a write, you can temporarily route their read requests to the primary/write database instead of the read replica. Since the write model is always strongly consistent, this guarantees they see their change. You might do this for a short window (e.g., 1 minute) or until you can verify the change has propagated.

   • Trade-off: You bypass the performance benefits of the read model for that specific user's queries.

2. Version Tracking: The write operation returns a version number or timestamp. The client then includes this version in subsequent read requests (GET /api/resource?minVersion=123). The server (or API gateway) can use this information to make an intelligent routing decision. If the query replica's data is older than minVersion, the server can either wait for it to catch up or route the request to the primary database.

   • Trade-off: This is more complex to implement but is also more robust and flexible than simply reading from the write model for a fixed time.

Test your understanding!

Imagine you are designing the "shopping cart" feature for an e-commerce site. A user clicks "Add to Cart" on a product page. This sends a command to the cart-service. The user is then immediately redirected to the main cart page.

Problem: Due to eventual consistency, the user lands on the cart page, but the item they just added isn't visible. After they refresh the page a second later, it appears.

Propose two different strategies to solve this UX problem. For each, describe how it would work and list one major pro and one major con.

Show answer

Here are two possible solutions:

Strategy 1: Optimistic UI

• How it works: When the user clicks "Add to Cart", the client-side code doesn't wait for the server's response. It immediately adds the item to the local UI state of the shopping cart. Then, it redirects the user to the cart page, which is populated from this local state. The command to the cart-service is sent in the background. If the server returns an error, the UI would then remove the item from the local state and show an error message.
• Pro: The application feels instantaneous. The user's action has an immediate, visible result.
• Con: It's more complex to implement client-side logic for managing the local state and handling potential rollbacks if the server call fails.

Strategy 2: Read from the Write Model (Temporarily)

• How it works: When the user clicks "Add to Cart," a command is sent to the cart-service. Upon a successful response, the client receives a unique session identifier or flag. When the client is redirected to the cart page, it includes this identifier in the request to fetch the cart contents. The API Gateway or backend service sees this flag and routes the request directly to the write database for the cart service, bypassing the potentially stale read replica. This flag could expire after a few seconds.
• Pro: It guarantees the user will see their most recent change without complex client-side state management.
• Con: It temporarily sacrifices the performance and scalability benefits of CQRS by hitting the write database for a read operation. This could become a bottleneck if many users are performing this action simultaneously.

Conclusion

We've covered the final, and perhaps most important, piece of the distributed data puzzle: managing the user experience in the face of eventual consistency. This isn't just a technical challenge but a product and design challenge that requires collaboration across teams.

Key Takeaways:

• Eventual consistency is a trade-off: We sacrifice immediate consistency for higher availability, performance, and scalability.
• UX is paramount: Unmanaged eventual consistency can lead to confusing and frustrating user experiences.
• A toolkit of strategies exists: We can inform the user, use optimistic UI patterns to create a responsive feel, or use synchronous polling when consistency is critical.
• Read-Your-Own-Writes is a special case: This common problem requires specific solutions like temporarily reading from the write model or implementing version tracking to meet user expectations.

In an interview, being able to articulate these trade-offs and propose concrete mitigation strategies for a given scenario is a hallmark of a senior engineer.

Next Up

This lesson concludes our module on data management. We've explored how to handle data within and across services. Now, we'll shift our focus inward to the services themselves.

Our next module, Advanced Concurrency & Performance, tackles how a single microservice can handle a massive number of requests efficiently. We'll start by analyzing the limitations of the traditional thread-per-request model and explore why modern, high-performance systems are moving towards non-blocking and reactive programming paradigms. This will connect directly to our discussion today on performance and responsiveness.