Competing Consumers for Pub/Sub Scaling

Introduction

Welcome to your course on scaling infrastructure on GKE! Your goal is to fix a production issue where a Pub/Sub subscriber service is struggling to keep up, leading to a growing message backlog. This course is designed to give you the specific, practical skills to solve this problem by implementing autoscaling.

In this first lesson, we will focus on the foundational concept that makes scaling possible for queue-based systems: the competing consumer pattern. You'll learn how running multiple instances—or replicas—of your subscriber service allows you to process messages in parallel, directly addressing the bottleneck you're experiencing. This pattern is the "why" behind the "how" of autoscaling that we'll implement in later lessons.

The Bottleneck: A Single Consumer

At the heart of your current problem is a classic bottleneck. You have a producer application sending messages to a Google Cloud Pub/Sub topic. Your subscriber service is connected via a subscription, which acts like a message queue. With only one instance of your service running, it must process every single message sequentially.

If the rate of incoming messages exceeds the rate at which your single instance can process them, a backlog is inevitable. This is likely why you're observing a growing number of undelivered messages.

The Solution: The Competing Consumer Pattern

To overcome this limitation, we use the competing consumer pattern. The idea is simple but powerful: instead of one consumer, we run multiple, identical consumers that all pull messages from the same subscription. These consumers "compete" to acquire and process messages.

This parallelizes the workload. If you have three consumers, you can potentially process three messages simultaneously, tripling your throughput.

To get a clear overview of this concept, please watch the following video.

As the video shows, adding more consumers allows the system to scale horizontally. Each consumer is independent, and they collectively work through the message queue much faster than a single consumer ever could.

Competing Consumers in Pub/Sub and Kubernetes

Now, let's map this pattern to your specific technology stack:

1. Google Cloud Pub/Sub: The competing consumer pattern is the default and intended way to build scalable subscribers. When you have multiple subscriber clients connected to the same subscription, Pub/Sub automatically distributes messages among them. The goal is to maximize overall throughput. As the documentation explains, Pub/Sub will even dynamically send more messages to consumers that acknowledge messages faster, effectively load-balancing the work.

2. Kubernetes: The "multiple consumers" in this pattern correspond to multiple Pods running your subscriber application. In Kubernetes, you manage this using a Deployment resource, where you can specify the number of replicas (i.e., identical Pods) you want to run.

By increasing the replicas count in your Deployment, you are directly implementing the competing consumer pattern. Kubernetes will ensure that the specified number of Pods, each an independent consumer, are running and ready to pull messages from your Pub/Sub subscription.

The following article provides a good bridge between the architectural pattern and its implementation in Kubernetes.

How Work is Distributed Fairly

A crucial detail for this pattern to work effectively is ensuring that messages are distributed fairly and efficiently. If one consumer grabs 100 messages but is slow, and a faster consumer sits idle, you haven't gained much.

To solve this, message brokers often implement two key mechanisms:

1. Message Acknowledgment (ack): A message is not removed from the queue when a consumer receives it. Instead, it is temporarily leased. The consumer must explicitly send an "acknowledgment" (ack) back to the broker after it has successfully processed the message. If the consumer crashes or fails to send an ack within a timeout, the broker makes the message available again for another consumer to process. This ensures reliability.

2. Prefetch Limit / Flow Control: Consumers can be configured to pull only a certain number of messages at a time (e.g., one). This is often called a "prefetch count" or "flow control limit." Setting a low limit (like prefetch_count=1) prevents a single slow consumer from hoarding messages, ensuring that available messages are dispatched to consumers that are actually free to do work. This promotes fair dispatch.

The following video demonstrates these concepts with a Python example. While it uses RabbitMQ, the principles of message acknowledgment and fair dispatch are directly applicable to building robust Pub/Sub consumers.

Tying it to Your Pub/Sub Problem

The Google Cloud documentation for Pub/Sub troubleshooting directly points to this pattern as a solution for backlogs and high latency.

By running multiple replicas of your subscriber Pod, you ensure there are "enough subscribers" with active connections to handle the message volume, reduce the backlog, and minimize processing latency.

Conclusion

In this lesson, we've established the core principle for scaling your event-driven service.

Key Takeaways:

• A single message consumer can become a bottleneck, leading to growing message backlogs.
• The competing consumer pattern solves this by running multiple, identical consumer instances that process messages in parallel from the same queue (or Pub/Sub subscription).
• In Kubernetes, this is achieved by setting the number of replicas in a Deployment.
• Mechanisms like message acknowledgment and prefetch limits ensure that work is distributed reliably and efficiently among consumers.

You now understand why running multiple replicas is the solution to your problem. The next logical question is: how many replicas should you run? And how does Kubernetes know where to place them?

Before we can scale, we must first define the resources each replica needs. In the next lesson, we will cover Kubernetes resource requests and limits, which are essential for ensuring predictable performance and enabling effective autoscaling.