<p>I am recording and monitoring SLOs (server-side request duration) of Kubernetes Pods via Prometheus using a <a href="https://godoc.org/github.com/prometheus/client_golang/prometheus#HistogramVec" rel="nofollow noreferrer">HistogramVec</a> within a Golang HTTP server. Every request’s duration is timed and persisted as described in the <a href="https://prometheus.io/docs/practices/histograms/" rel="nofollow noreferrer">Prometheus practices</a> and partitioned by status code, method and HTTP path. </p>

<p>I am running autoscaling experiments therefore Pods are created &amp; terminated. After each experiment I fetch the metrics for all pods (including the ones already deleted) and plot a cumulative distribution, e.g.:
<a href="https://i.stack.imgur.com/KoQkS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KoQkS.png" alt="CDF"></a>
In order to make these plots more “accurate”, I opted for many, smaller histogram buckets and aggregate &amp; analyze the data locally and do not use the built-in <a href="https://prometheus.io/docs/practices/histograms/#quantiles" rel="nofollow noreferrer">Histogram Quantiles</a>. <strong>The ideal query would therefore return only the most recent value for all time series that have existed over a specified time range (green + red circles).</strong>
<a href="https://i.stack.imgur.com/Wfs6H.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Wfs6H.png" alt="Timeseries"></a>
Currently, I am using a <a href="https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries" rel="nofollow noreferrer">range query</a> within the script generating all the plots, e.g.:</p>

<pre><code>http://localhost:9090/api/v1/query_range?query=http_request_duration_milliseconds_bucket{path="/service/login"}&amp;start=1591803898&amp;end=1591804801&amp;step=5s
</code></pre>

<p>However, I am aware that this is highly inefficient and costly as it retrieves a huge amount of surplus data even though I am only interested in the very last value for each individual time series. On the other hand, if I use an instant query, I only get the values for a specified moment, thus I’d need to shoot multiple queries &amp; first find out when some time series (red circles) were marked stale - which doesn’t seem great either. </p>

<p>So, basically I'm looking for a way to work around the <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness" rel="nofollow noreferrer">Prometheus basics on staleness</a>, and stop stale time series from "disappearing":</p>

<blockquote>
  <p>If no sample is found (by default) 5 minutes before a sampling timestamp, no value is returned for that time series at this point in time. This effectively means that time series "disappear" from graphs at times where their latest collected sample is older than 5 minutes or after they are marked stale.</p>
</blockquote>

<p>I am almost certain that there is a way to do this (e.g. an option to simply include stale time series), but I haven’t been able to put it together so far.</p>

<p>The solution is to use <a href="https://docs.victoriametrics.com/MetricsQL.html#last_over_time" rel="nofollow noreferrer">last_over_time()</a> function. For example, the following query returns the last values seen during the last hour per each histogram bucket:</p>
<pre><code>last_over_time(http_request_duration_milliseconds_bucket{path=&quot;/service/login&quot;}[1h])
</code></pre>
<p>This query must be sent to <a href="https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries" rel="nofollow noreferrer">/api/v1/query</a> instead of <a href="https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries" rel="nofollow noreferrer">/api/v1/query_range</a>, since <code>/api/v1/query</code> calculates the query only once at the given <code>time</code> timestamp, while <code>/api/v1/query_range</code> calculates the query <code>1+(end-start)/step</code> times at every point on the timer range <code>[start ... end]</code> with interval <code>step</code>.</p>
<p>Note also that big number of histogram buckets multiplied by big number of unique <code>path</code> label values may result into too many time series, which is known as <a href="https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality" rel="nofollow noreferrer">high cardinality</a>. See <a href="https://www.robustperception.io/cardinality-is-key" rel="nofollow noreferrer">this article</a> for more details.</p>
<p>See also <a href="https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350" rel="nofollow noreferrer">VictoriaMetrics historgrams</a>, which solve common issues in Prometheus histograms.</p>
