<p>I want to calculate the cpu usage of all pods in a kubernetes cluster. I found two metrics in prometheus may be useful:</p>

<pre><code>container_cpu_usage_seconds_total: Cumulative cpu time consumed per cpu in seconds.
process_cpu_seconds_total: Total user and system CPU time spent in seconds.

Cpu Usage of all pods = increment per second of sum(container_cpu_usage_seconds_total{id="/"})/increment per second of sum(process_cpu_seconds_total)
</code></pre>

<p>However, I found every second's increment of <code>container_cpu_usage{id="/"}</code> larger than the increment of <code>sum(process_cpu_seconds_total)</code>. So the usage may be larger than 1...</p>

<p>This I'm using to get CPU usage at cluster level:</p>

<pre><code>sum (rate (container_cpu_usage_seconds_total{id="/"}[1m])) / sum (machine_cpu_cores) * 100
</code></pre>

<p>I also track the CPU usage for each pod.</p>

<pre><code>sum (rate (container_cpu_usage_seconds_total{image!=""}[1m])) by (pod_name)
</code></pre>

<p>I have a complete kubernetes-prometheus solution on GitHub, maybe can help you with more metrics: <a href="https://github.com/camilb/prometheus-kubernetes" rel="noreferrer">https://github.com/camilb/prometheus-kubernetes</a></p>

<p><a href="https://i.stack.imgur.com/Qr7St.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Qr7St.png" alt="enter image description here"></a></p>

<p><a href="https://i.stack.imgur.com/uxbXv.png" rel="noreferrer"><img src="https://i.stack.imgur.com/uxbXv.png" alt="enter image description here"></a></p>
