Micrometer and Prometheus: Custom Metrics Configuration

Hello! Welcome back to our module on Observability & Monitoring.

In our last lesson, we dove into distributed tracing, the second pillar of observability. You learned how to use Micrometer Tracing and Zipkin to visualize the entire journey of a request across multiple services. This is invaluable for answering the question, "Why was this specific request slow?".

Today, we address the third and final pillar: Metrics. While tracing is about individual requests, metrics are about the big picture. They provide aggregated, numerical data about the health and performance of your system over time. This allows us to answer questions like, "What is the average API latency over the last hour?" or "Is our error rate increasing?".

Our learning outcome for this lesson is to configure custom application metrics using Micrometer and expose them in Prometheus format. You will learn how to instrument your Spring Boot application to produce the raw data that powerful monitoring and alerting systems consume. This is a fundamental skill for operating services at scale and a common topic in senior-level system design interviews.

1. The Key Players: Micrometer, Actuator, and Prometheus

To understand how metrics work in a modern Spring Boot application, we need to know the three main components and how they interact.

Let's break down their roles:

• Micrometer: Think of Micrometer as the "SLF4J for metrics." It's a vendor-neutral application metrics facade. You write your instrumentation code against the Micrometer API, which allows you to switch the underlying monitoring system (Prometheus, Datadog, New Relic, etc.) with minimal code changes. This decoupling is a critical design principle for building portable, cloud-native applications.
• Spring Boot Actuator: This is the component that provides production-ready features for your application. For our purposes, its most important job is to expose various operational endpoints, including the crucial /actuator/prometheus endpoint, which presents all registered Micrometer metrics in a format Prometheus can scrape.
• Prometheus: An open-source monitoring and alerting toolkit. It operates on a pull-based model, meaning it periodically "scrapes" (fetches) metrics from configured HTTP endpoints like the one our Actuator provides. We will focus on setting up Prometheus itself in the next lesson; today, our goal is to get our application ready for scraping.

2. Setting Up Your Application

Getting started with metrics in Spring Boot is straightforward thanks to its auto-configuration capabilities.

Step 1: Add Dependencies

First, you need to add the necessary dependencies to your pom.xml.

<!-- Provides production-ready endpoints, including for metrics -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<!-- The Micrometer registry that formats metrics for Prometheus -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

With these two dependencies, Spring Boot will automatically:

1. Configure a MeterRegistry for Prometheus.
2. Instrument common components like web controllers and data sources.
3. Set up the /actuator/prometheus endpoint.

Step 2: Expose the Prometheus Endpoint

By default, for security reasons, Spring Boot does not expose the Prometheus endpoint over HTTP. You need to enable it in your application.properties:

# Expose the prometheus endpoint alongside health and info
management.endpoints.web.exposure.include=health,info,prometheus

# You can also give your application a name, which helps in monitoring
spring.application.name=order-service

Once you've done this and started your application, you can visit http://<your-host>:<port>/actuator/prometheus to see a wealth of auto-configured metrics related to the JVM, system resources, HTTP requests, and more.

3. Implementing Custom Metrics: The Core Meter Types

While auto-configured metrics are useful, the real power comes from instrumenting your own business logic. Micrometer provides several types of "Meters" for this. The central piece of the API for creating these meters is the MeterRegistry, which you can inject into any Spring bean.

Let's explore the four most important meter types.

1. Counter: For Monotonically Increasing Values

A Counter is a single value that only ever goes up. It's perfect for counting events like "orders processed," "login attempts," or "exceptions thrown."

You can create and use a counter like this:

@Service
public class OrderService {

    private final Counter ordersCreatedCounter;

    public OrderService(MeterRegistry meterRegistry) {
        // Define the counter with a name
        this.ordersCreatedCounter = meterRegistry.counter("orders.created.total");
    }

    public void createOrder(Order order) {
        // Business logic to create an order...
        
        // Increment the counter
        ordersCreatedCounter.increment();
    }
}

When scraped by Prometheus, this metric will appear as:

# HELP orders_created_total  
# TYPE orders_created_total counter
orders_created_total 1.0

The .total suffix is a Prometheus convention for counters.

2. Gauge: For a Snapshot of a Value

A Gauge represents a value that can go up or down. It's a snapshot of a current state, like the number of items in a queue, the size of a cache, or the number of active user sessions.

Since a gauge is a snapshot, you don't set its value directly. Instead, you register an object or function that Micrometer will call whenever the gauge needs to be measured.

@Component
public class TaskQueueMonitor {

    private final List<String> taskQueue = new CopyOnWriteArrayList<>();

    public TaskQueueMonitor(MeterRegistry meterRegistry) {
        // Create a gauge that tracks the size of the queue
        Gauge.builder("task.queue.size", taskQueue, List::size)
             .description("The current number of tasks in the queue")
             .register(meterRegistry);
    }

    public void addTask(String task) {
        taskQueue.add(task);
    }
    
    // ... methods to remove tasks
}

The output will look like:

# HELP task_queue_size The current number of tasks in the queue
# TYPE task_queue_size gauge
task_queue_size 15.0

A crucial point for interviews: Micrometer only holds a weak reference to the object being gauged. If taskQueue in the example above were to be garbage-collected, the gauge would start reporting NaN or disappear. It is your responsibility to maintain a strong reference to the state object you are gauging.

3. Timer: For Measuring Latency and Frequency

A Timer is one of the most useful meters. It measures both the duration of events and their frequency. It's ideal for measuring the latency of API endpoints, database queries, or any time-bound operation. A single Timer provides multiple metrics: a count of events, the total time spent, and the maximum time recorded.

@Service
public class ReportGenerator {

    private final Timer reportGenerationTimer;

    public ReportGenerator(MeterRegistry meterRegistry) {
        this.reportGenerationTimer = meterRegistry.timer("reports.generation.latency");
    }

    public void generateReport() {
        reportGenerationTimer.record(() -> {
            // Simulate time-consuming report generation
            try {
                Thread.sleep(new Random().nextInt(500));
            } catch (InterruptedException e) {
                // handle exception
            }
        });
    }
}

This single Timer produces a rich set of metrics in Prometheus format:

# HELP reports_generation_latency_seconds  
# TYPE reports_generation_latency_seconds summary
reports_generation_latency_seconds_count 5.0
reports_generation_latency_seconds_sum 1.234
# HELP reports_generation_latency_seconds_max  
# TYPE reports_generation_latency_seconds_max gauge
reports_generation_latency_seconds_max 0.487

4. DistributionSummary: For Measuring Non-Time-Based Distributions

A DistributionSummary is similar to a Timer but is used to record the distribution of non-time-based values. A classic use case is tracking the size of request or response payloads.

@RestController
public class FileUploadController {
    
    private final DistributionSummary fileSizeSummary;

    public FileUploadController(MeterRegistry meterRegistry) {
        this.fileSizeSummary = DistributionSummary.builder("file.upload.size.bytes")
                                .description("Distribution of uploaded file sizes")
                                .baseUnit("bytes")
                                .register(meterRegistry);
    }

    @PostMapping("/upload")
    public void handleFileUpload(@RequestParam("file") MultipartFile file) {
        fileSizeSummary.record(file.getSize());
        // ... process file
    }
}

This gives you metrics like file_upload_size_bytes_count, _sum, and _max.

4. The Power of Dimensionality: Tags and Labels

A common mistake is to create separate metrics for different facets of an event, like orders_created_web_total and orders_created_mobile_total. This leads to a metric explosion and makes aggregation difficult.

The modern, correct approach is to use dimensional metrics. In Micrometer, you add Tags to a metric. These tags become Labels in Prometheus, which are key-value pairs that enrich the metric.

Instead of two metrics, you create one: orders.created.total, and add a tag for the source.

// In your service method
meterRegistry.counter("orders.created.total", "source", "web").increment();
// ...
meterRegistry.counter("orders.created.total", "source", "mobile").increment();

This produces two time series for the same metric:

orders_created_total{source="web"} 25.0
orders_created_total{source="mobile"} 12.0

This approach is incredibly powerful because you can now query your metrics in Prometheus by filtering or grouping by these labels. For example, you can easily get the total number of orders across all sources (sum(orders_created_total)).

Best Practices and Interview Hot Topics:

• Metric Naming: Use lowercase, dot-separated names in your code (e.g., http.server.requests). Micrometer will automatically translate this to the format expected by the monitoring system (e.g., http_server_requests for Prometheus).
• Tag Keys: Use meaningful, low-cardinality keys like status, method, region.
• High-Cardinality Warning: Avoid using tags with unbounded sets of values, such as user IDs, request IDs, or timestamps. Each unique combination of labels creates a new time series in Prometheus, and high cardinality can overwhelm your monitoring system. This is a key sign of seniority in an interview.
• Common Tags: You can define tags that will be applied to all metrics from your application in application.properties. This is perfect for environment-wide context like region or application name.

  management.metrics.tags.region=us-east-1
  management.metrics.tags.stack=prod
  

// On success
meterRegistry.counter("payment.gateway.calls.total", "status", "success").increment();

// On failure
meterRegistry.counter("payment.gateway.calls.total", "status", "failure").increment();

Conclusion

You have now learned how to instrument your application to produce meaningful, queryable metrics—the third pillar of observability. This data is the foundation for building dashboards, setting up alerts, and gaining quantitative insights into your system's behavior.

Key Takeaways:

• Micrometer is the standard metrics facade in Spring, decoupling your code from monitoring systems.
• Spring Boot Actuator and the micrometer-registry-prometheus dependency work together to expose a /actuator/prometheus scrape endpoint.
• The primary meter types are Counter (for counts), Gauge (for snapshots), Timer (for latency), and DistributionSummary (for value distributions).
• Using tags (which become Prometheus labels) is essential for creating powerful, dimensional metrics that can be easily filtered and aggregated.
• Avoid high-cardinality tags like user IDs to prevent overwhelming your monitoring system.

Next Up

Our application is now a well-instrumented source of metric data. But right now, that data just sits at an endpoint, waiting to be collected. In the next lesson, we will complete the loop. You will learn how to set up Prometheus to scrape these metrics and write basic PromQL queries to analyze them. This is where we turn raw data into actionable insights.