SEAN K.H. LIAO

# metric histograms

## a bucket for all

### histograms

In observability/metrics, histograms are primarily used to record distributions in a space efficient manner that can be aggregated and summarized (usually with quantile measurements) at a later point.

But how can you pull out something like the 99.9 percentile from a few buckets? The answer is you can't. What you can do is find the bucket it would land in and attempt a (linear) interpolation between the bucket bounds and generate an estimate.

#### implementations

So it becomes apparent that having more buckets gives you higher resolutions. Now for a look at different implentations.

##### prometheus

Prometheus essentially stores each bucket as its own time series with the bucket bounds recorded as labels. This is quite innefficient and as a result, by default it only gives a few buckets, which while it covers the common range for things like http request latency, is subject to large error bars if the boundaries aren't close to the actual values (which is what is interesting... the outlier events).

``````DefBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}
``````
##### hdr histogram

HDR-Histograms based on this analysis divides the space between a given `(min, max)` pair into exponential buckets (`2^x`), then subdivides each bucket linearly. This apprently scales to ~1800 buckets in a ~1500 byte packet?

In other places this is called "log linear bounds" (logarithmic big buckets, linear sub buckets).

##### datatdog ddsketch

DDSketch based on this analysis also uses exponential buckets, but this time choosing a base that guarantees relative error. For a 1% error, `(1+0.1) / (1-0.1) = 1.02`, and so buckets are chosen with `1.02^x`. A fixed number of buckets are chosen, and the starting buckets (lowest indices) are merged if it ever grows out of space. Their blog post mentions somewhere on the order of a few hundred buckets (eg 802 for ns~day scale), with their implementation setting a default limit at 2048.

In a comment on an opentelemetry histogram proposal Google said they use a `(base = minimum_base^2^compression)^x` for buckets.
After much discussion opentelemetry proposed and finally defined their buckets with `base = 2 ^ (2 ^ -scale)`.