<p>I'm trying to write a Prometheus query that can tell me how much, as a percentage, CPU (and another for memory and network) each namespace has used over a time frame, say a week.</p>
<p>The metrics I'm trying to use are <code>container_spec_cpu_shares</code> and <code>container_memory_working_set_bytes</code> but I can't figure out how sum them over time.  Whatever I try either returns 0 or errors.</p>
<p>Any help on how to write a query for this would be greatly appreciated.</p>

Steve
<p>To check the percentage of memory used by each namespace you will need a query similar to the one below:</p>
<pre><code>sum( container_memory_working_set_bytes{container=&quot;&quot;, namespace=~&quot;.+&quot;} )|
by (namespace) / ignoring (namespace) group_left 
sum( machine_memory_bytes{}) * 100 
</code></pre>
<p>Above query should produce a graph similar to this one:</p>
<p><a href="https://i.stack.imgur.com/eiqur.png" rel="noreferrer"><img src="https://i.stack.imgur.com/eiqur.png" alt="GRAFANA IMAGE" /></a></p>
<blockquote>
<p>Disclaimers!:</p>
<ul>
<li>The screenshot above is from Grafana for better visibility.</li>
<li><strong>This query does not acknowledge changes in available RAM (changes in nodes, autoscaling of nodes, etc.).</strong></li>
</ul>
</blockquote>
<p>To get the metric over a period of time in PromQL you will need to use additional function like:</p>
<ul>
<li><code>avg_over_time(EXP[time])</code>.</li>
</ul>
<p>To go back in time and calculate resources from specific point in time you will need to use:</p>
<ul>
<li><code>offset TIME</code></li>
</ul>
<p><strong>Using above pointers query should combine to:</strong></p>
<pre><code>avg_over_time( sum(container_memory_working_set_bytes{container=&quot;&quot;, namespace=~&quot;.+&quot;} offset 45m) by (namespace)[120m:])  / ignoring (namespace) group_left 
sum( machine_memory_bytes{}) 
</code></pre>
<p>Above query will calculate the average percentage of memory used by each namespace and divide it by all memory in the cluster in the span of 120 minutes to present time. It will also start 45 minutes earlier from present time.</p>
<p>Example:</p>
<ul>
<li>Time of running the query: 20:00</li>
<li><code>avg_over_time(EXPR[2h:])</code></li>
<li><code>offset 45 min</code></li>
</ul>
<p>Above example will start at 17:15 and it will run the query to the 19:15. You can modify it to include the whole week :).</p>
<p>If you want to calculate the CPU usage by namespace you can replace this metrics with the one below:</p>
<ul>
<li><code>container_cpu_usage_seconds_total{}</code> - please check <code>rate()</code> function when using this metric (counter)</li>
<li><code>machine_cpu_cores{}</code></li>
</ul>
<p>You could also look on this network metrics:</p>
<ul>
<li><code>container_network_receive_bytes_total</code> - please check <code>rate()</code> function when using this metric (counter)</li>
<li><code>container_network_transmit_bytes_total</code> - please check <code>rate()</code> function when using this metric (counter)</li>
</ul>
<p>I've included more explanation below with examples (memory), methodology of testing and dissection of used queries.</p>
<hr />
<p>Let's assume:</p>
<ul>
<li>Kubernetes cluster <code>1.18.6</code> (Kubespray) with 12GB of memory in total:
<ul>
<li>master node with <code>2GB</code> of memory</li>
<li>worker-one node with <code>8GB</code> of memory</li>
<li>worker-two node with <code>2GB</code> of memory</li>
</ul>
</li>
<li>Prometheus and Grafana installed with: <em><a href="https://github.com/coreos/kube-prometheus" rel="noreferrer">Github.com: Coreos: Kube-prometheus</a></em></li>
<li>Namespace <code>kruk</code> with single <code>ubuntu</code> pod set to generate artificial load with below command:
<ul>
<li><code>$ stress-ng --vm 1 --vm-bytes &lt;AMOUNT_OF_RAM_USED&gt; --vm-method all -t 60m -v</code></li>
</ul>
</li>
</ul>
<p>The artificial load was generated with <code>stress-ng</code> two times:</p>
<ul>
<li>60 minutes - <strong>1GB</strong> of memory used</li>
<li>60 minutes - <strong>2GB</strong> of memory used</li>
</ul>
<p>The percentage of memory used by namespace <code>kruk</code> in this timespan:</p>
<ul>
<li>1GB which accounts for about ~8.5% of all memory in the cluster (12GB)</li>
<li>2GB which accounts for about ~17.5% of all memory in the cluster (12GB)</li>
</ul>
<p>The load from Prometheus query for <code>kruk</code> namespace was looking like that:</p>
<p><a href="https://i.stack.imgur.com/x1FTl.png" rel="noreferrer"><img src="https://i.stack.imgur.com/x1FTl.png" alt="kruk namespace memory usage" /></a></p>
<p>Calculation using <code>avg_over_time(EXPR[time:]) / memory in the cluster</code> showed the usage in the midst of about 13% (<code>(17.5+8.5)/2</code>) when querying the time the artificial load was generated. This should indicate that the query was correct:</p>
<p><a href="https://i.stack.imgur.com/g3y6r.png" rel="noreferrer"><img src="https://i.stack.imgur.com/g3y6r.png" alt="Average usage" /></a></p>
<hr />
<p>As for the used query:</p>
<pre><code>avg_over_time( sum( container_memory_working_set_bytes{container=&quot;&quot;, namespace=&quot;kruk&quot;} offset 1380m )
by (namespace)[120m:]) / ignoring (namespace) group_left 
sum( machine_memory_bytes{}) * 100 
</code></pre>
<p>Above query is really similar to the one in the beginning but I've made some changes to show only the <code>kruk</code> namespace.</p>
<p>I divided the query explanation on 2 parts (dividend/divisor).</p>
<h3>Dividend</h3>
<pre><code>container_memory_working_set_bytes{container=&quot;&quot;, namespace=&quot;kruk&quot;}
</code></pre>
<p>This metric will output records of memory usage in namespace <code>kruk</code>. If you were to query for all namespaces look on additional explanation:</p>
<ul>
<li><code>namespace=~&quot;.+&quot;</code> &lt;- this regexp will match only when the value inside of namespace key is containing 1 or more characters. This is to avoid empty namespace result with aggregated metrics.</li>
<li><code>container=&quot;&quot;</code> &lt;- part is used to filter the metrics. If you were to query without it you would get multiple memory usage metrics for each container/pod like below. <code>container=&quot;&quot;</code> will match only when container value is empty (last row in below citation).</li>
</ul>
<pre class="lang-sh prettyprint-override"><code>container_memory_working_set_bytes{container=&quot;POD&quot;,endpoint=&quot;https-metrics&quot;,id=&quot;/kubepods/podab1ed1fb-dc8c-47db-acc8-4a01e3f9ea1b/e249c12010a27f82389ebfff3c7c133f2a5da19799d2f5bb794bcdb5dc5f8bca&quot;,image=&quot;k8s.gcr.io/pause:3.2&quot;,instance=&quot;192.168.0.124:10250&quot;,job=&quot;kubelet&quot;,metrics_path=&quot;/metrics/cadvisor&quot;,name=&quot;k8s_POD_ubuntu_kruk_ab1ed1fb-dc8c-47db-acc8-4a01e3f9ea1b_0&quot;,namespace=&quot;kruk&quot;,node=&quot;worker-one&quot;,pod=&quot;ubuntu&quot;,service=&quot;kubelet&quot;} 692224
container_memory_working_set_bytes{container=&quot;ubuntu&quot;,endpoint=&quot;https-metrics&quot;,id=&quot;/kubepods/podab1ed1fb-dc8c-47db-acc8-4a01e3f9ea1b/fae287e7043ff00da16b6e6a8688bfba0bfe30634c52e7563fcf18ac5850f6d9&quot;,image=&quot;ubuntu@sha256:5d1d5407f353843ecf8b16524bc5565aa332e9e6a1297c73a92d3e754b8a636d&quot;,instance=&quot;192.168.0.124:10250&quot;,job=&quot;kubelet&quot;,metrics_path=&quot;/metrics/cadvisor&quot;,name=&quot;k8s_ubuntu_ubuntu_kruk_ab1ed1fb-dc8c-47db-acc8-4a01e3f9ea1b_0&quot;,namespace=&quot;kruk&quot;,node=&quot;worker-one&quot;,pod=&quot;ubuntu&quot;,service=&quot;kubelet&quot;} 2186403840
container_memory_working_set_bytes{endpoint=&quot;https-metrics&quot;,id=&quot;/kubepods/podab1ed1fb-dc8c-47db-acc8-4a01e3f9ea1b&quot;,instance=&quot;192.168.0.124:10250&quot;,job=&quot;kubelet&quot;,metrics_path=&quot;/metrics/cadvisor&quot;,namespace=&quot;kruk&quot;,node=&quot;worker-one&quot;,pod=&quot;ubuntu&quot;,service=&quot;kubelet&quot;} 2187096064
</code></pre>
<blockquote>
<p>You can read more about pause container here:</p>
<ul>
<li><em><a href="https://www.ianlewis.org/en/almighty-pause-container" rel="noreferrer">Ianlewis.org: Almighty pause container</a></em></li>
</ul>
</blockquote>
<pre><code>sum( container_memory_working_set_bytes{container=&quot;&quot;, namespace=&quot;kruk&quot;} offset 1380m )
by (namespace)
</code></pre>
<p>This query will sum the results by their respective namespaces. <code>offset 1380m</code> is used to go back in time as the tests were made in the past.</p>
<pre><code>avg_over_time( sum( container_memory_working_set_bytes{container=&quot;&quot;, namespace=&quot;kruk&quot;} offset 1380m )
by (namespace)[120m:])
</code></pre>
<p>This query will calculate average from memory metric across namespaces in the specified time (120m to now) starting 1380m earlier than present time.</p>
<p>You can read more about <code>avg_over_time()</code> here:</p>
<ul>
<li><em><a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#aggregation_over_time" rel="noreferrer">Prometheus.io: Aggregation over time</a></em></li>
<li><em><a href="https://prometheus.io/blog/2019/01/28/subquery-support/" rel="noreferrer">Prometheus.io: Blog: Subquery support</a></em></li>
</ul>
<h3>Divisor</h3>
<pre><code>sum( machine_memory_bytes{})
</code></pre>
<p>This metric will sum the memory available in each node in the cluster.</p>
<pre><code>EXPR / ignoring (namespace) group_left 
sum( machine_memory_bytes{}) * 100 
</code></pre>
<p>Focusing on:</p>
<ul>
<li><code>/ ignoring (namespace) group_left</code> &lt;- this expression will allow you to divide each &quot;record&quot; in the dividend (each namespace with their memory average across time) by a divisor (all memory in the cluster). You can read more about it here: <em><a href="https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching" rel="noreferrer">Prometheus.io: Vector matching</a></em></li>
<li><code>* 100</code> is rather self explanatory and will multiply the result by a 100 to look more like percentages.</li>
</ul>
<hr />
<p>Additional resources:</p>
<ul>
<li><em><a href="https://prometheus.io/docs/prometheus/latest/querying/basics/" rel="noreferrer">Prometheus.io: Querying: Basics</a></em></li>
<li><em><a href="https://timber.io/blog/promql-for-humans/" rel="noreferrer">Timber.io: Blog: Promql for humans</a></em></li>
<li><em><a href="https://grafana.com/grafana/dashboards/315" rel="noreferrer">Grafana.com: Dashboards: 315</a></em></li>
</ul>
