Database-per-Service: Consistency Challenges

Hello! Welcome to the first lesson of our module on Distributed Transactions & Data Management.

In the previous module, we explored event-driven architecture and the trade-offs between tools like RabbitMQ and Kafka. We concluded by noting that these messaging systems are often the key to solving complex data problems in distributed systems. Today, we'll dive into one of the most fundamental—and challenging—of those problems.

This lesson addresses the learning outcome: Explain the database-per-service pattern and its data consistency challenges. Understanding this pattern is non-negotiable for designing microservices. It's a foundational concept that frequently comes up in system design interviews to gauge your grasp of architectural trade-offs.

1. The Database-per-Service Pattern: A Core Tenet of Microservices

In a monolithic architecture, it's common for multiple modules to share a single, large database. This simplifies data access and ensures strong consistency through ACID transactions. Microservices, however, are built on the principle of loose coupling and autonomy. To achieve this, we must also decentralize data ownership.

This leads us to the database-per-service pattern: each microservice is solely responsible for its own data and stores it in a private database. No other service is allowed to access this database directly. If one service needs data from another, it must go through the owner service's public API.

To understand the "why" behind this pattern, let's explore its benefits.

As the article highlights, the primary benefits are:

• Loose Coupling & Service Autonomy: This is the most critical benefit. If the User service's team wants to refactor their database schema (e.g., add a column, switch to a new table structure), they can do so without impacting the Order or Payment services. This autonomy is essential for independent development and deployment cycles. In a shared database model, a single schema change can have a ripple effect, requiring coordinated updates across multiple teams.
• Polyglot Persistence: Not all data is the same. The Product Catalog service might have data that is best represented as a JSON document, making a document database like MongoDB a great fit. The Order service, requiring strong transactional guarantees, would benefit from a relational database like PostgreSQL. This pattern allows each team to choose the best data storage technology for their specific needs.
• Independent Scaling: The Inventory service might experience a very high volume of writes, while the Product Catalog service is mostly read-heavy. With separate databases, you can scale the database for each service independently based on its specific load profile, optimizing both performance and cost.
• Fault Isolation: If the database for the Recommendation service goes down, it won't directly affect the Order service's ability to process new orders. The failure is contained within the boundaries of a single service, improving the overall resilience of the application.

2. The Trade-Off: Data Consistency Challenges

While the benefits are significant, decentralizing data introduces one of the biggest challenges in microservices architecture: maintaining data consistency across services.

In a monolith with a single database, you can rely on ACID (Atomicity, Consistency, Isolation, Durability) transactions to execute a business operation spanning multiple tables. For example, placing an order might involve inserting a row into the Orders table and updating a row in the Inventory table within a single, atomic transaction. If either step fails, the entire transaction is rolled back, leaving the database in a consistent state.

With the database-per-service pattern, this is no longer possible. The Orders data lives in the Order Service's database, and the Inventory data lives in the Inventory Service's database. There is no mechanism for a single, global transaction that spans these two separate databases.

This introduces several complex challenges.

Let's break down the key challenges mentioned in the article with a concrete example. Consider an e-commerce "place order" workflow that involves three services:

1. Order Service: Creates an order in a PENDING state.
2. Payment Service: Charges the customer's credit card.
3. Inventory Service: Decrements the stock for the ordered items.

Challenge 1: Distributed Transactions and Partial Failures

What happens if the Order Service and Payment Service succeed, but the Inventory Service fails (e.g., the item is unexpectedly out of stock)?

• The customer has been charged.
• An order record exists.
• But the product cannot be shipped.

The system is now in an inconsistent state. We have a "distributed transaction" that has partially failed. Since we can't use a traditional database transaction to automatically roll everything back, we need a new mechanism to handle this. This is the fundamental problem that patterns like Sagas (which we'll cover soon) are designed to solve.

Challenge 2: Eventual Consistency

The solution to the distributed transaction problem often involves asynchronous, event-driven communication (which you'll recall from our last module). For instance, the Order Service might publish an OrderCreated event. The Payment and Inventory services would then consume this event and perform their respective actions.

This leads to a state of eventual consistency. There will be a brief period where the order is created, but the payment has not yet been processed or the inventory has not been updated. The system is temporarily inconsistent but will become consistent eventually once all services have processed the event. For many use cases, this is acceptable. However, it's a major shift from the strong consistency guaranteed by monolithic systems and requires careful design to manage user expectations.

Challenge 3: Complex Reporting and Queries

In a monolith, generating a report that shows "all orders placed by users in California for products over $100" is a straightforward SQL query with a few JOIN clauses across the Users, Orders, and Products tables.

In a microservices architecture, this data is scattered across the User Service, Order Service, and Product Service databases. You can no longer run a simple JOIN. This creates significant complexity for analytics and reporting. Common solutions, which we will discuss later in the course, include:

• Building a dedicated reporting database or data warehouse that aggregates data from multiple services.
• Implementing the Command Query Responsibility Segregation (CQRS) pattern.

Conclusion

In this lesson, we established the database-per-service pattern as a foundational element of microservice design, essential for achieving loose coupling and autonomy.

Key Takeaways:

• Database-per-Service Pattern: Each microservice owns and manages its own private database. Communication for data access must happen through APIs.
• Key Benefits: It enables loose coupling, polyglot persistence (using the right DB for the job), independent scaling, and fault isolation.
• Core Challenge: The loss of global ACID transactions across services. This introduces significant data consistency challenges.
• Consequences: We must handle partial failures in distributed transactions and embrace eventual consistency as the default, which is a major architectural shift from monolithic systems.

Next Up

We've now clearly defined the problem of maintaining consistency across distributed services. The next logical question is: how do we solve it? In our next lesson, we will explore the two primary architectural patterns for coordinating workflows across microservices: Choreography vs. Orchestration. These patterns provide frameworks for implementing the distributed transactions we've discussed today.