<p>I am running <strong>prometheus</strong> in my <strong>kubernetes</strong> cluster.
I have the following system in kubernetes:</p>

<p>I have 4 nodes. I want to calculate free memory. I want to have the summation of those four nodes. Then I want to find the maximum over 1 day. So, for example,</p>

<p>at time=t1
node1: 500 MB
node2: 600 MB
node3: 200 MB
node4: 300 MB
Total = 1700 MB</p>

<p>at time=t2
node1: 400 MB
node2: 700 MB
node3: 100 MB
node4: 200 MB
Total = 1300 MB</p>

<p>at time=t3
node1: 600 MB
node2: 800 MB
node3: 1200 MB
node4: 1300 MB
Total = 3900 MB</p>

<p>at time=t4
node1: 100 MB
node2: 200 MB
node3: 300 MB
node4: 400 MB
Total = 1000 MB</p>

<p>So, The answer to my query should be 3900 MB. I am not able to do max_over_time for the sum.</p>

<p>I have done like this(which is not working at all): </p>

<pre><code>max_over_time(sum(node_memory_MemFree)[2m])
</code></pre>

<p><a href="https://prometheus.io/blog/2019/01/28/subquery-support/" rel="noreferrer">Since</a> version 2.7 (Jan 2019), Prometheus supports sub-queries:</p>

<pre><code>max_over_time( sum(node_memory_MemFree_bytes{instance=~"foobar.*"})[1d:1h] )
</code></pre>

<p>(metric for the past 2 days, with a resolution of 1 hour.)</p>

<p>Read the documentation for more informations on using recording rules:  <a href="https://prometheus.io/docs/prometheus/latest/querying/examples/#subquery" rel="noreferrer">https://prometheus.io/docs/prometheus/latest/querying/examples/#subquery</a></p>

<p>However, do note the blog recommendation:</p>

<blockquote>
  <p><strong>Epilogue</strong></p>
  
  <p>Though subqueries are very convenient to use in place of recording rules, using them unnecessarily has performance implications. Heavy subqueries should eventually be converted to recording rules for efficiency.</p>
  
  <p>It is also not recommended to have subqueries inside a recording rule. Rather create more recording rules if you do need to use subqueries in a recording rule.</p>
</blockquote>

<p>The use of recording rules is explained in brian brazil article:
 <a href="https://www.robustperception.io/composing-range-vector-functions-in-promql/" rel="noreferrer">https://www.robustperception.io/composing-range-vector-functions-in-promql/</a></p>
