<p>I installed the Prometheus helm chart to a kubernetes cluster for monitoring.
By default, </p>

<ul>
<li>persistent volume size for prometheus server is defined as 8Gi.</li>
<li>Prometheus server will store the metrics in this volume for 15 days (retention period)</li>
</ul>

<p>After some days of deploying the chart, the prometheus server pod enetered to a crashloopbackoff state.
The reason found from pod logs was:</p>

<pre><code>level=error ts=2019-10-09T11:03:10.802847347Z caller=main.go:625 err="opening storage failed: zero-pad torn page: write /data/wal/00000429: no space left on device"
</code></pre>

<p>That means there is no space available in the disk (persistent volume) to save the data.
So I cleared the existing data of the volume and fixed the issue temporarily.</p>

<p>What would be the proper solution for this? </p>

<p>The <a href="https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects" rel="nofollow noreferrer">Prometheus documentation</a> says:</p>

<p><strong>To plan the capacity of a Prometheus server, you can use the rough formula:</strong></p>

<pre><code>needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample
</code></pre>

<p>Can someone explain how to use this formula deliberately?</p>

<p>Why the 8Gi size is not enough with 15days retention period?</p>

<p><strong>EDIT :</strong></p>

<p>The default 8Gi space was 100% used after 6 days.</p>

<p>As of Prometheus 2.7, theye've introduced a new flag to manage retention. From docs:</p>

<blockquote>
  <p><code>--storage.tsdb.retention.size</code>: [EXPERIMENTAL] This determines the maximum number of bytes that storage blocks can use (note that this does not include the WAL size, which can be substantial). The oldest data will be removed first. Defaults to 0 or disabled. This flag is experimental and can be changed in future releases. Units supported: KB, MB, GB, PB. Ex: "512MB"</p>
</blockquote>

<p>You can set this argument option in your Deployment configuration to limit the retention according to size, instead of time.</p>

<p>As it is experimental yet, according to <a href="https://www.robustperception.io/configuring-prometheus-storage-retention" rel="nofollow noreferrer">this</a> source, it would be safe to allow for space for the WAL and one maximum size block (which is the smaller of 10% of the retention time and a month).</p>
