<p>I run a <code>v1.9.2</code> custom setup of Kubernetes and scrape various metrics with Prometheus <code>v2.1.0</code>. Among others, I scrape the <code>kubelet</code> and <code>cAdvisor</code> metrics.</p>

<p>I want to answer the question: "<em>How much of the CPU resources defined by <code>requests</code> and <code>limits</code> in my deployment are actually used by a pod (and its containers) in terms of (milli)cores?</em>"</p>

<p>There are a lot of scraped metrics available, but nothing like that. Maybe it could be calculated by the CPU usage time in seconds, but I don't know how.</p>

<p>I was considering it's not possible - until a friend told me she runs Heapster in her cluster which has a graph in the built-in Grafana that tells <em>exactly</em> that: It shows the indivual CPU usage of a pod and its containers in (milli)cores.</p>

<p>Since Heapster also uses <code>kubelet</code> and <code>cAdvisor</code> metrics, I wonder: how can I calculate the same? The metric in InfluxDB is named <code>cpu/usage_rate</code> but even with Heapster's code, I couldn't figure out how they calculate it.</p>

<p>Any help is appreciated, thanks!</p>

<p>We're using the <code>container_cpu_usage_seconds_total</code> metric to calculate Pod CPU usage. This metrics contains the total amount of CPU seconds consumed <em>by container by core</em> (this is important, as a Pod may consist of multiple containers, each of which can be scheduled across multiple cores; however, the metric has a <code>pod_name</code> annotation that we can use for aggregation). Of special interest is the <strong>change rate</strong> of that metric (which can be calculated with <a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#rate()" rel="noreferrer">PromQL's <code>rate()</code> function</a>). If it increases by 1 within one second, the Pod consumes 1 CPU core (or 1000 milli-cores) in that second.</p>

<p>The following PromQL query does just that: Compute the CPU usage of all Pods (using the <a href="https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators" rel="noreferrer"><code>sum(...) by (pod_name)</code></a> operation) over a five minute average:</p>

<pre><code>sum(rate(container_cpu_usage_seconds_total[5m])) by (pod_name)
</code></pre>
