<p>I installed the Prometheus helm chart to a kubernetes cluster for monitoring.
By default, </p>

<ul>
<li>persistent volume size for prometheus server is defined as 8Gi.</li>
<li>Prometheus server will store the metrics in this volume for 15 days (retention period)</li>
</ul>

<p>After some days of deploying the chart, the prometheus server pod enetered to a crashloopbackoff state.
The reason found from pod logs was:</p>

<pre><code>level=error ts=2019-10-09T11:03:10.802847347Z caller=main.go:625 err="opening storage failed: zero-pad torn page: write /data/wal/00000429: no space left on device"
</code></pre>

<p>That means there is no space available in the disk (persistent volume) to save the data.
So I cleared the existing data of the volume and fixed the issue temporarily.</p>

<p>What would be the proper solution for this? </p>

<p>The <a href="https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects" rel="nofollow noreferrer">Prometheus documentation</a> says:</p>

<p><strong>To plan the capacity of a Prometheus server, you can use the rough formula:</strong></p>

<pre><code>needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample
</code></pre>

<p>Can someone explain how to use this formula deliberately?</p>

<p>Why the 8Gi size is not enough with 15days retention period?</p>

<p><strong>EDIT :</strong></p>

<p>The default 8Gi space was 100% used after 6 days.</p>

<p>15 days is about 1.3 million seconds. Let’s overestimate 8 bytes per sample. So each metric takes about 10mb. So 8gb would let you store 800 metrics. You probably have more than that. Multiply the number of series you want to store by 10 and that’s the number of megabytes you need. Roughly, that will get you the right order of magnitude at least.</p>
