How and when to use a Prometheus gauge

A Prometheus gauge is a specific type of metric used for measurements. That means your service always returns to Prometheus the current value of whatever it is you’re measuring.

Prometheus is regularly scraping your service for metrics though, and when your gauge’s current value is returned Prometheus stores this against the current time. You’ll then be able to run queries against Prometheus to see what’s happening to your gauge over time.

A gauge in Prometheus is represented by a 64-bit floating point number. That means it can store very large or small decimal numbers, either positive or negative.

When to use a gauge?

Any time you want to measure something which can go up or down, you should use a gauge.

Here are some examples:

• memory or CPU usage
• queue size
• number of active sessions
• temperature

If you need to measure something that only ever goes up, such as request count, then use a counter instead (see this article for more info).

Client libraries for gauges

Client libraries to publish gauges from your service exist for Java, Go, Python, and Ruby. They allow you to register gauges and interact with them in these ways:

• increment the gauge by 1 or any floating point value
• decrement the gauge by 1 or any floating point value
• set the gauge to a specific floating point value

Java gauge client library

The Java client library for creating Prometheus gauges is io.prometheus:simpleclient and can be downloaded from Maven central.

Here’s an example of using it to measure the size of a queue.

package com.tom;

import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.Gauge;

public class GaugeExample {
    private final Gauge gauge;

    public GaugeExample(CollectorRegistry collectorRegistry) {
        gauge = Gauge.build()
                .name("queue_size")
                .labelNames("queue_name")
                .help("Size of queue.")
                .register(collectorRegistry);
    }

    public void addItemToQueue(QueueItem queueItem) {
        gauge.labels("my-awesome-queue").inc();

        // code to add to queue
    }

    public QueueItem readItemFromQueue() {
        gauge.labels("my-awesome-queue").dec();

        // code to read from queue
        return new QueueItem();
    }
}

• just like with all Prometheus metrics, the Gauge metric is created with:
  • a mandatory name
  • optional label names
  • optional help text
• the gauge is then registered with a CollectorRegistry, the central store in your application for Prometheus metrics
• when an item is added to the queue, the gauge can be incremented. The label value must also be passed at this point.
• likewise, when an item is read from the queue the gauge can be decremented, also passing the label value

How are gauges exposed to Prometheus?

By default Prometheus scrapes any configured targets once every minute. If your service publishes a gauge then Prometheus expects it to be in this format.

# HELP queue_size Size of queue.
# TYPE queue_size gauge
queue_size{queue_name="my-awesome-queue",} 3.0

• the HELP text contains the metric name plus any description that was configured
• the TYPE contains the metric name plus the type of the metric, in this case gauge
• the metric value is published using the standard Prometheus format of the name, any labels, followed by the value

Querying gauges in Prometheus

Prometheus stores observed gauge values over time, allowing you to run all sorts of queries against them using the PromQL query language.

Current value

To get the current value of a gauge metric, execute this simple query which will return all label combinations.

queue_size

Current value with labels

To get the current value of a gauge metric for a specific label, use this query.

queue_size{queue_name="my-awesome-queue"}

Average value over time

The avg_over_time function calculates the average value of the gauge over the provided time range.

avg_over_time(queue_size[10m])

Maximum value over time

The max_over_time function calculates the gauge’s maximum value over the time range.

max_over_time(queue_size[10m])

Functions not relevant to gauges

Most Prometheus functions do work with gauges, but there are some important exceptions such as increase and rate. This is because these functions make calculations based on the assumption that the value only ever goes up. You should never use these functions with a gauge.

Gauge-specific functions

Some Prometheus functions have been specifically designed with gauges in mind.

Calculate delta

The delta function calculates the difference between the value of a gauge at the end and start of the provided time range. This example shows us the difference between the queue size now and 10 minutes ago.

delta(queue_size[10m])

Predict future value

The predict_linear function tries to predict the value of your gauge at some point in the future.

It takes two arguments:

1. a range of data from which to base the prediction
2. the number of seconds into the future the prediction should be made

This example shows us the prediction of the queue size 1 minute from now, based on the last 10 minutes of data.

predict_linear(queue_size[10m], 60)

An example use case for predict_linear

Imagine a scenario where you have a queue with a hard size limit of 100. If something tries to add the 101st item to the queue, things go badly wrong! You can use Prometheus alerts to be notified if there’s a problem.

One approach would be to create an alert which triggers when the queue size goes above some pre-defined limit, say 80. Even if the queue size has been slowly increasing by 1 every week, if it gets to 80 in the middle of the night you get woken up with an alert. 😢

A smarter approach might be to set things up so that you only get an alert if the queue size is predicted to reach it’s maximum size in the next 48 hours. predict_linear allows you to create such an alert, waking you up in the night only if there is genuinely an imminent problem.

Be careful though, as your gauge must behave in predictable ways for this to be a valid use case.

For a full list of Prometheus functions see the official documentation.

Key takeaways

• the gauge is for values which go up and down.
• use a counter if you’re measuring something that only goes up
• the gauge represents the current value of whatever you’re measuring
• you can use most standard Prometheus functions to query a gauge, plus some interesting gauge-specific ones
• don’t try to use the increase or rate functions with a gauge

To learn more about the other types of Prometheus metrics, check out the article The 4 Types Of Prometheus Metrics.