<p>I have a Fluentd daemonset on the k8s cluster with 3 nodes. I would like to get a value which represents percentage which gives me an information on how much CPU (in %) on each node, fluentd pod occupies at this moment.</p>

<p>What would be the way to do it in Prometheus?</p>

<p>Thanks.</p>

Bakir Jusufbegovic
<p>You would want to use the <code>container_cpu_usage_seconds_total</code> query in Prometheus. <br></p>

<p>Like so:</p>

<pre><code>sum (rate (container_cpu_usage_seconds_total{}[5m])) by (container_name)
</code></pre>

<p>This will return the CPU usage of all the pods <code>by</code> container name in the system.</p>

<hr>

<p>You can apply some filters as well to fine grain the output. For example:</p>

<pre><code>sum (rate (container_cpu_usage_seconds_total{container_name=~"fluentd.*"}[5m])) by (container_name)
</code></pre>

<p>The above query will return the CPU usage with pods that matches containers name starting with <code>fluentd</code></p>

<hr>

<p>You can divide the usage of those pods by the total cpu cores of your cluster to find out the usage in percentage, like so:</p>

<pre><code>sum (rate (container_cpu_usage_seconds_total{container_name=~"fluentd.*"}[5m])) / sum (machine_cpu_cores{}) * 100
</code></pre>

<hr>

<p>Finally, in order to get percentage of total cpu cores usage on specific node for specific container_name, you would add additional filter: instance="INSTANCE_NAME": </p>

<pre><code>sum (rate (container_cpu_usage_seconds_total{container_name=~"fluentd.*", instance="INSTANCE_NAME"}[5m])) / sum (machine_cpu_cores{}) * 100
</code></pre>

<p><em>N.B: Depending on K8S version, the fields returned by the <code>container_cpu_usage_seconds_total</code> query may vary. On some system, the name of the containers is represented by the <code>container_name</code> field, wheres on some system it is <code>container</code>.</em></p>
