Prometheus Metrics for Go Applications

Welcome to the next lesson in our exploration of observability. In the previous session, we established the critical framework of SLIs and SLOs to define and measure the reliability of our services from a user's perspective. We answered the questions of what to measure and why. Now, we pivot from theory to practice and address the question of how.

This lesson will guide you through the process of instrumenting a Go application. Instrumentation is the process of adding code to your application to generate and expose telemetry data. Specifically, you will learn how to use the official Prometheus client library for Go to create metrics, track the behavior of your application, and make those metrics available for a Prometheus server to collect. This is the foundational skill needed to bring your SLIs to life.

How Prometheus Instrumentation Works

Before we write any code, let's establish a mental model of the process. Prometheus operates on a pull-based model. This means the Prometheus server periodically sends an HTTP GET request to a target application's /metrics endpoint. The application's responsibility is to respond with its current set of metrics in a specific text-based format that Prometheus understands.

We don't need to handle the metric formatting or the HTTP endpoint logic ourselves. A client library, in our case prometheus/client_golang, provides all the necessary components. The overall flow looks like this:

1. Your application code imports the Prometheus client library.
2. You define and register the metrics you want to track (e.g., total requests, active users). These metrics are held in memory.
3. As your application runs, it updates these metric objects (e.g., incrementing a counter, setting a gauge).
4. The client library exposes an HTTP handler (which you attach to the /metrics path) that automatically formats the current state of all registered metrics for Prometheus.

The diagram below illustrates this seven-step process, from setting up the registry in your application to Prometheus reading the final state from the /metrics endpoint.

Choosing the Right Tool: Prometheus Metric Types

The first step in instrumenting your code is choosing the correct metric type for the data you want to measure. Using the wrong type can make your data useless or misleading. Prometheus offers four primary metric types.

• Counter: A cumulative metric that only ever increases or resets to zero upon an application restart. It's perfect for counting events, such as the total number of HTTP requests, tasks completed, or errors encountered.
• Gauge: A metric that represents a single numerical value that can arbitrarily go up and down. Gauges are ideal for "point-in-time" measurements, like current memory usage, the number of active goroutines, or the number of items in a queue.
• Histogram: A metric that samples observations (usually request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values. Histograms allow you to calculate quantiles (e.g., 95th or 99th percentile) on the server side using PromQL queries. This is extremely powerful because you can aggregate data from multiple instances of your service before calculating percentiles, giving you a global view of performance.
• Summary: Similar to a histogram, a summary also samples observations. However, it calculates configurable quantiles on the client side (i.e., within your application) and exposes them directly. The main drawback is that you cannot aggregate quantiles from multiple service instances. For this reason, histograms are generally preferred for scalable, distributed systems.

This article gives a clear and concise explanation of the three most common types.

The following flowchart provides a simple decision-making guide for selecting the right metric type.

Hands-On: Instrumenting a Go Web Service

Now, let's apply these concepts to a practical example. We'll instrument a web service written in Go using the popular Gin framework. Your experience with Go and backend development will make this process straightforward. The pattern we'll use—a middleware to capture request-level metrics—is a standard practice in production environments.

The following guide walks you through building a simple API and instrumenting it from scratch.

Let's break down the key pieces of code from that article.

1. Defining and Registering Metrics

// Prometheus metrics
var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"path"}, // This is the label name
    )
    // ... other metrics
)

func init() {
    prometheus.MustRegister(httpRequestsTotal, ...)
}

• Metrics are typically declared as global variables.
• The ...Vec types (e.g., CounterVec) are crucial. They allow you to add labels to your metrics. Instead of a single counter, you get a vector of counters partitioned by the label values (in this case, the request path). This allows for much more granular analysis.
• The init() function is a convenient place to register all your metrics with the Prometheus client's global registry. MustRegister will panic if a metric fails to register (e.g., due to a duplicate name), which is useful for catching errors at startup.

2. Collecting Metrics via Middleware

func prometheusMiddleware(c *gin.Context) {
    path := c.Request.URL.Path
    timer := prometheus.NewTimer(httpRequestDuration.WithLabelValues(path))
    httpRequestsTotal.WithLabelValues(path).Inc()
    activeConnections.Inc()

    c.Next() // Process the request

    timer.ObserveDuration()
    activeConnections.Dec()
}

• This middleware function runs for every incoming request.
• WithLabelValues(path) selects the specific counter or histogram from the vector that corresponds to the current request's path.
• .Inc() increments the counter.
• prometheus.NewTimer(...) is a utility that simplifies measuring duration for a histogram. When timer.ObserveDuration() is called after the request is processed (c.Next()), it automatically records the elapsed time.
• The activeConnections gauge is incremented before processing the request and decremented after, accurately tracking the number of in-flight requests.

3. Exposing the /metrics Endpoint

func main() {
    router := gin.Default()
    router.Use(prometheusMiddleware)
    // ... your application endpoints
    router.GET("/metrics", gin.WrapH(promhttp.Handler()))
    router.Run(":8080")
}

• router.Use(prometheusMiddleware) applies our instrumentation logic to all routes.
• gin.WrapH(promhttp.Handler()) is the bridge between the standard net/http handler provided by the Prometheus library (promhttp.Handler()) and the Gin router. This single line creates the /metrics endpoint.

For a more detailed verbal explanation of the different metric types and their vector counterparts, this video is an excellent resource.

Viewing Your Metrics with Prometheus

While the learning outcome is focused on instrumenting the application, seeing the results is the most satisfying part. You can run a local Prometheus instance using Docker to scrape the metrics from your newly instrumented Go application.

1. Create a prometheus.yml configuration file:

   global:
     scrape_interval: 15s
   
   scrape_configs:
     - job_name: 'go-app'
       static_configs:
   
   
   
         # For Docker on Mac/Windows, use host.docker.internal
         # For Docker on Linux, you might need to use your machine's IP
         - targets: ['host.docker.internal:8080']
   

2. Run Prometheus in Docker:
   Execute this command from the same directory where you saved prometheus.yml.

   docker run -d \
     -p 9090:9090 \
     -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
     --name prometheus \
     prom/prometheus
   

After running your Go application and this Docker command, you can navigate to http://localhost:9090. In the "Targets" section of the Prometheus UI, you should see your go-app job with a green "UP" state. You can then use the expression browser to query the metrics you created, like http_requests_total.

Conclusion

In this lesson, you've bridged the gap between the theory of SLOs and the practice of measurement. You now have the practical skills to instrument a Go application, turning its internal state and behavior into quantifiable metrics ready for monitoring.

Key Takeaways:

• Instrumentation is the process of adding code to an application to export telemetry data.
• Prometheus uses a pull model, scraping metrics from a standard /metrics HTTP endpoint.
• Go applications are typically instrumented using the prometheus/client_golang library.
• Choosing the correct metric type (Counter, Gauge, Histogram) is essential for meaningful data. Histograms are generally preferred over Summaries for scalable services.
• Labels are the key to powerful, multidimensional metrics, allowing you to slice and dice your data (e.g., by path, status code, or method).
• Using middleware is an effective pattern for collecting request-level metrics in web services without cluttering your business logic.

We have now covered how to collect metrics from a single service. But modern, scalable systems are composed of many services working together. How do we understand the lifecycle of a request as it travels across multiple services? Our next lesson will address this by introducing you to distributed tracing with OpenTelemetry.