<p>Prometheus deployed on kubernetes using prometheus operator is eating too much memory and it is at present at ~12G. I see <code>/prometheus/wal</code> directory is at ~12G. I have removed all <code>*.tmp</code> files but that couldn't help. 
Unable to figure out the solution for this problem. Any suggestions ??</p>

Yogesh Jilhawar
<p>Digging over google couple of days, I found that there are lot of unused metrics for which we can drop samples[1].<br>
Searched over prometheus directory for the use of metrices whose sampling is very high-
Query to find prometheus metrics-   </p>

<pre><code>topk(20, count by (__name__, job)({__name__=~".+"}))
</code></pre>

<p>If you found tcp or udp metrices here in the list. Try executing these metrices on prometheus, if value is zero then these are safe to drop since these metrices are already disabled at CAdvisor level as they produce large number of samples.<br>
bash commands to check for the use of these metrices anywhere in prometheus or grafana-</p>

<pre class="lang-sh prettyprint-override"><code>cd &lt;prometheus dir&gt;
grep -irn &lt;metric_name&gt;
</code></pre>

<p>if not being used anywhere then simply add drop action for that particular job.<br>
<strong>Note</strong>- you will get <code>jobname</code> in promQL query executed at first.   </p>

<p>I am using prometheus operator so I have to edit respective servicemonitor definition for that. If you are deploying prometheus in normal way, you may need to edit prometheus.yaml file.    </p>

<pre><code>metric_relabel_configs:
   - source_labels: [ __name__ ]
     regex: 'metric_name'
     action: drop
</code></pre>

<p><strong>Reference:-</strong>.<br>
<strong>[1]</strong> <a href="https://www.robustperception.io/dropping-metrics-at-scrape-time-with-prometheus" rel="nofollow noreferrer">https://www.robustperception.io/dropping-metrics-at-scrape-time-with-prometheus</a>      </p>

<p><strong>[2]</strong> Formula to calculate needed RAM-<br>
<a href="https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion" rel="nofollow noreferrer">https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion</a></p>
