<p>I have set up a Grafana dashboard, to monitor Kubernetes. It's working as expected but there's an issue of losing metrics/history at some point of time. See the attached screenshot. In the screenshot, no metrics are shown between 21:00 PM to 23:00 PM.</p>

<p><a href="https://imgur.com/lQFKV2W" rel="nofollow noreferrer">https://imgur.com/lQFKV2W</a></p>

<p>From the above image, is that behavior normal? Thanks.</p>

<p>I'm using Grafana as an addon on Istio.</p>

<p>Part of the Istio helm chart configuration:</p>

<pre><code>grafana:
  enabled: true
  persist: true  
  accessMode: ReadWriteMany
</code></pre>

<p>My expectation is that no history/metrics should be lost at any specific time.</p>

<p>Istio uses Prometheus to collect the metrics. What you want to do, is to change a Prometheus data retention.</p>

<p>For example, if you are installing it with Helm, just change this parameter in the chart</p>

<pre><code>--set prometheus.retention=2d
</code></pre>

<p>source: <a href="https://github.com/istio/istio/issues/12567" rel="nofollow noreferrer">https://github.com/istio/istio/issues/12567</a></p>
