Handling Eventual Consistency and Data Staleness

Hello! Welcome to the next lesson in our series on distributed systems architecture.

In our last session, we implemented projections, the components that consume an event stream to build and maintain our read models. We noted that this process is asynchronous, which creates a time lag between a write operation and its reflection in the queryable views.

Today, we will directly confront this consequence. This lesson is designed to meet the learning outcome: Address eventual consistency between write and read models, including strategies for communicating data staleness to clients. We will define the problem, explore its theoretical underpinnings, and evaluate practical strategies for managing it in a real-world system.

1. The Core Problem: Reading Your Own Writes

The asynchronicity inherent in CQRS with event sourcing means that after a user performs an action (a write), a subsequent query (a read) might be served by a read model that hasn't yet processed the corresponding event. The user sees stale data, which can be confusing and lead to a poor user experience.

This video provides an excellent overview of why this happens and frames it as a user experience challenge.

The central issue described in the video is often called the "read-your-own-writes" problem. While eventual consistency across an entire system might be acceptable, an individual user generally expects to see the immediate result of their own actions.

2. A Formal Framework for Consistency

To discuss solutions effectively, it's useful to have a more precise vocabulary. The term "eventual consistency" is just one point on a spectrum of guarantees a system can provide. Given your background in stochastic systems and mathematics, you'll find the formal definitions of these models insightful.

The following blog post by Amazon's CTO, Werner Vogels, is a foundational text on this topic. It provides clear, client-centric definitions for various consistency models.

From that reading, we can extract two key properties that are highly desirable for creating a good user experience in an eventually consistent system:

• Read-Your-Writes Consistency: This is the guarantee we've been discussing. After a user updates an item, any subsequent read they perform will return the updated value (or a newer one).
• Monotonic Read Consistency: If a user performs a sequence of reads, they will never see an older version of the data after having seen a newer version. This prevents the UI from appearing to go "back in time."

Achieving these two properties for the acting user, while allowing the rest of the system to remain eventually consistent for other users, is often the primary goal.

The Azure Architecture Center documentation on the CQRS pattern explicitly acknowledges this challenge as a primary consideration.

3. Strategies for Achieving Read-Your-Writes Consistency

Now that we have a formal and practical understanding of the problem, let's explore concrete strategies to solve it. Most of these strategies focus on ensuring consistency for the user who initiated the write, without sacrificing the benefits of eventual consistency for the rest of the system.

The following video discusses several of these solutions. We will use it as a guide for our discussion.

Let's break down these strategies and analyze their trade-offs.

Strategy 1: Synchronous Update (Server-Side Wait)

• Mechanism: The command handler, after persisting the event, waits for the relevant projection(s) to confirm they have processed the event before returning a success response to the client.
• Pros: Provides strong, immediate consistency for the user. Simple to reason about.
• Cons:
  • High Write Latency: The user's request is blocked for the entire duration of the write and the projection update.
  • Tight Coupling: The write path becomes coupled to the read path's update speed. A slow projection can slow down the entire system. This negates a key performance benefit of CQRS.
• Applicability: Rarely used in high-performance systems. It might be acceptable for low-throughput, critical operations where the user absolutely must see the confirmed state immediately.

Strategy 2: Redirecting to the Primary (Reading from the Write Model)

• Mechanism: The command returns immediately. The client is given a hint (or the routing logic infers) that for a short period, any queries related to the recent write should be directed to the write model (the event store) or a synchronous replica, bypassing the eventually consistent read model.
• Pros: Low write latency. Guarantees read-your-writes consistency.
• Cons:
  • Increased Load on Write Model: The write model, optimized for writes, is now also handling read traffic, which can become a bottleneck.
  • Complex Routing: Requires logic (either on the client or in an API gateway) to route queries to the correct data store based on user context and time.
• Applicability: A strong pattern for many use cases. For example, after a user executes an FX trade, their "trade details" query for the next 30 seconds could be served by rehydrating the Trade aggregate from the event store, while all other users query the denormalized read models.

Strategy 3: Client-Side Polling with Versioning

• Mechanism: The command response includes a version identifier for the state change (e.g., the event's sequence number or a timestamp). The client then polls the read model, passing the version it expects. The read model API only returns a result once its own data is at or beyond the requested version.
• Pros: Decouples the server-side components. Write latency is low.
• Cons:
  • Chatty: Can lead to many HTTP requests from the client.
  • Perceived Latency: The user may still have to wait, but the waiting happens on the client side (e.g., showing a spinner).
  • API Complexity: The read API must be aware of versioning.
• Applicability: Can be effective, but often superseded by push notifications for a better UX. It's a viable option when WebSocket infrastructure is not available.

Strategy 4: Push Notifications (WebSockets)

• Mechanism: The command returns immediately. The client opens a persistent connection (e.g., a WebSocket). When the projection finishes updating the read model, it also publishes a notification event. A separate service pushes this notification to the specific client, which can then either refetch the data or be sent the updated data directly in the notification payload.
• Pros:
  • Excellent UX: Feels real-time and responsive.
  • Efficient: Avoids polling. The server pushes data only when it's ready.
• Cons:
  • Infrastructure Complexity: Requires managing persistent connections (WebSockets, SignalR, etc.), which adds statefulness and complexity to the backend.
• Applicability: The gold standard for modern, interactive UIs. The demo in the video of the e-shop order list updating automatically is a perfect example.

4. Communicating Staleness: The Fallback Strategy

Sometimes, the complexity of the above strategies isn't justified. Eventual consistency is often perfectly acceptable if the user's expectations are managed correctly.

• When is it acceptable? As the video "Eventual Consistency: Good, Bad, and a HUGE Mistake" points out, this is often the case for non-transactional contexts like reporting, analytics, or viewing data that other users have changed. For example, a trading desk's aggregate position view doesn't need to be real-time to the millisecond.

• How to communicate it?

  • Visual Indicators: Use spinners, loading bars, or disabled buttons with tooltips like "Processing your request...".
  • Explicit Text: Clearly label data with timestamps, such as "Data as of 14:32:15 UTC" or display a message like "Your new trade will appear in the list shortly."
  • Optimistic UI: Update the UI immediately with the expected state, but visually indicate that it's "unconfirmed." If the backend operation fails, revert the UI change and show an error. This provides a very responsive feel but requires careful state management.

Conclusion

Today we tackled the crucial challenge of eventual consistency, a direct result of the asynchronous architecture we've been building.

Key Takeaways:

• Eventual consistency in CQRS primarily becomes a user experience problem in "read-your-own-writes" scenarios.
• We can use a formal vocabulary (strong, eventual, read-your-writes, monotonic read) to precisely describe the guarantees we need.
• There are four primary strategies to ensure a user sees their own writes: server-side waiting, reading from the primary, client-side polling, and push notifications, each with distinct trade-offs.
• When immediate consistency isn't required, effectively communicating data staleness to the user is a valid and often simpler approach.

Preview of the Next Lesson:
We have now explored the full data lifecycle in our CQRS/ES system—from command handling, to event persistence, to projection, and finally to managing the consistency of the resulting read models. Our next step is to formalize and strengthen the heart of this system: the event store itself. In the next lesson, we will define the API contract for a generic event store service and then move on to implementing one of its most critical features: optimistic concurrency control.