Creating Histograms in Grafana from Prometheus Buckets

In the following example, we will be creating a histogram in Grafana. Our datasource is Prometheus’s cumulative histogram.

I have captured the metrics using micrometer’s distribution summary. I have 12 buckets for report size. I want to show size buckets on X axis and its count on Y-axis. e.g: 20 Kb - 100 Kb : 100 100 Kb -500 Kb : 500 500 Kb -1 Mb : 1000 1 Mb - 10 Mb : 15

rptsize_megabyte_bucket{le="1.0",} 5.0 rptsize_megabyte_bucket{le="5.0",} 8.0 rptsize_megabyte_bucket{le="10.0",} 10.0 rptsize_megabyte_bucket{le="50.0",} 15.0 rptsize_megabyte_bucket{le="100.0",} 16.0 rptsize_megabyte_bucket{le="300.0",} 16.0 rptsize_megabyte_bucket{le="500.0",} 16.0 rptsize_megabyte_bucket{le="1000.0",} 16.0 rptsize_megabyte_bucket{le="2000.0",} 16.0 rptsize_megabyte_bucket{le="3000.0",} 16.0 rptsize_megabyte_bucket{le="5000.0",} 16.0 rptsize_megabyte_bucket{le="+Inf",} 16.0 rptsize_megabyte_count 16.0 rptsize_megabyte_sum 239.8

Create a bar gauge graph, set Format in the query tab to Heatmap and query

sum(vmrs_rptsize_megabyte_bucket
) by (le)

In the panel option’s Display tab you’ll probably want to set Value to Last instead of Mean.

Grafana is aware of Prometheus’ cumulative buckets, so when you change format Time series to format Heatmap it’ll deduplicate the result and places the buckets in order.

There are three steps:

1. Grafana transforms the time series to a Grafana compatible format (same as for regular time series).
2. The results are sorted in incremental order.
3. Normalization where the series are iterated over in order and the previous series’ value is subtracted from the current (so that 10, 20, 30 => 10, 10, 10). https://github.com/grafana/grafana/blob/master/public/app/plugins/datasource/prometheus/result_transformer.ts#L22-L29

Check out Björn “Beorn” Rabenstein’s on Prometheus’s rate queries

Looking for an end-to-end incident alerting, on-call scheduling and response orchestration platform?

Sign up for a 14-day free trial of Zenduty. No CC required. Implement modern incident response and SRE best practices within your production operations and provide industry-leading SLAs to your customers.