Multi-Level Caching for Read-Heavy Services

Hello! Welcome to the final lesson in our module on "Advanced Concurrency & Performance."

In our last session, we explored advanced caching patterns like write-through and write-behind, along with crucial eviction policies. We established that a single cache layer, even a powerful distributed one like Redis, has inherent trade-offs, particularly around network latency for every request. For the high-traffic, low-latency systems you'd build at a FAANG or top fintech company, we can do better.

Today, we will address that by designing a system that uses multiple layers of caching. This is a very common topic in system design interviews for mid-to-senior roles, as it demonstrates your ability to make nuanced architectural decisions to optimize performance at scale.

Our learning outcome is to design a multi-level caching strategy for a high-traffic, read-heavy service. We will cover the rationale, a standard architectural blueprint, and how you can implement a key part of it in a Spring Boot application.

Why One Layer Isn't Always Enough

Let's quickly review the two main types of caching locations we've discussed:

1. In-Process Cache (e.g., Caffeine):

   • Pros: Extremely fast, as it lives in the application's own memory (RAM). No network calls are needed.
   • Cons: Data is siloed. Each instance of your microservice has its own separate cache. An update in one instance isn't reflected in others, leading to inconsistency. The cache size is also limited by the application's heap memory.

2. Distributed Cache (e.g., Redis):

   • Pros: Provides a large, shared cache accessible by all service instances, ensuring data consistency across the cluster.
   • Cons: Every cache access requires a network round-trip, which is orders of magnitude slower than an in-memory read.

A multi-level strategy aims to get the best of both worlds by combining these layers. The fundamental principle is to place caches at different points between the user and the data source, serving requests from the fastest, closest cache possible.

To see how this works in practice, let's watch a short clip illustrating a multi-layered approach for an e-commerce site.

Designing a Multi-Level Caching Architecture

A common and effective blueprint for a high-traffic service involves three distinct layers. Think of these like the L1, L2, and L3 caches in a CPU, each offering a different balance of speed and size.

For a structured overview of these layers, let's turn to a well-written article on the topic.

Let's summarize and expand on those layers:

Layer 1: In-Process Cache (L1)

This is the fastest cache, living inside each microservice instance. Its goal is to serve repeated requests for the same data from the same instance with near-zero latency.

• Purpose: Ultra-fast, low-latency access.
• Technology: In-memory libraries like Caffeine or Guava Cache.
• Characteristics:
  • Access is measured in nanoseconds.
  • Data is not shared between service instances.
  • Size is limited by the application's heap.
• Typical Use Cases: Caching frequently accessed, rarely changing data like application configuration, user permissions/roles, or metadata for a "hot" item that a single user is interacting with repeatedly.

Layer 2: Shared Distributed Cache (L2)

This is the workhorse layer, providing a unified cache for your entire application cluster. When L1 misses, the application checks L2.

• Purpose: A consistent, shared cache for all application instances.
• Technology: A dedicated caching service like Redis or Memcached.
• Characteristics:
  • Access is slower due to the network hop (typically <10ms).
  • Can store a much larger volume of data.
  • Acts as a single source of truth for cached data across the service.
• Typical Use Cases: Caching results of expensive database queries, pre-computed API responses, or user session data.

Layer 3: CDN / Edge Cache (L3)

This layer sits at the edge of the network, geographically close to your users. Its goal is to reduce network latency caused by physical distance.

• Purpose: Serve content to users from a location near them, reducing round-trip time.
• Technology: Content Delivery Networks like AWS CloudFront, Cloudflare, or Akamai.
• Characteristics:
  • Drastically reduces latency for global users.
  • Offloads a massive amount of traffic from your origin servers.
• Typical Use Cases:
  • Static Assets: Images, videos, CSS, and JavaScript files.
  • Public API Responses: Caching API responses that are not user-specific (e.g., a list of public products).

The following video provides an excellent summary of these cache locations and their trade-offs, which will help solidify your understanding.

Implementation: A Hybrid L1/L2 Cache in Spring Boot

Let's get practical. The most common multi-level pattern you will implement as a service developer is the L1/L2 hybrid cache. This combines an in-process cache (Caffeine) with a distributed cache (Redis) and is highly effective for read-heavy microservices.

The key challenge is consistency: if an item is updated or removed, how do we ensure it's evicted from the L1 cache of all service instances? The solution involves using the distributed cache (Redis) as a messaging bus to broadcast invalidation events.

The following article provides a complete, production-ready implementation guide for this pattern in Spring Boot.

Key Implementation Takeaways

Here's the workflow from the article, which is crucial to understand for an interview:

1. Read Operation (get):

   • The application first checks the L1 (Caffeine) cache. If it's a hit, data is returned instantly.
   • If it's an L1 miss, the application checks the L2 (Redis) cache.
   • If it's an L2 hit, the data is returned to the application and also written into the L1 cache for future requests.
   • If it's an L2 miss, the application queries the database, then writes the result to both L2 and L1.

2. Write/Update Operation (put):

   • The data is written to the L2 (Redis) cache.
   • It is also written to the L1 (Caffeine) cache of the current instance.

3. Eviction/Invalidation (evict):

   • A request is sent to L2 (Redis) to delete the key.
   • Redis, via Redisson's features, publishes a message about the key removal.
   • The CacheEntryRemovedListener in all service instances subscribes to these messages.
   • Upon receiving a removal message, each instance evicts that key from its own L1 (Caffeine) cache.

This final step is what solves the consistency problem of a simple in-process cache.

Conclusion

You have now designed a robust, multi-level caching strategy suitable for a high-traffic, read-heavy service. This approach moves beyond simple caching patterns and into the realm of advanced system architecture, which is exactly what interviewers for senior roles are looking for.

Key Takeaways:

• Multi-level caching combines different cache types (in-process, distributed, CDN) to optimize for both speed and scale.
• A common three-layer architecture consists of an L1 In-Process Cache (for speed), an L2 Distributed Cache (for consistency and shared state), and an L3 CDN (for geographic latency).
• The L1/L2 hybrid pattern is a powerful and practical implementation for microservices, but requires a mechanism (like a pub/sub listener) to synchronize invalidations and maintain consistency.
• In a design interview, you should be able to justify which layer is appropriate for which type of data and explain how you would handle challenges like data consistency.

Next Up

We have concluded our deep dive into advanced performance patterns. We will now shift our focus to another critical pillar of microservices architecture: security. In the next lesson, we will begin Module 6 by exploring the foundational framework for modern authentication and authorization: "Explain the OAuth 2.0 framework, including roles (Resource Owner, Client, etc.) and common grant types."