<p>I have some trouble in understanding why I get multiple results for the same pod in Prometheus/Grafana.</p>
<p>I'm trying to get cpu usage through <code>rate(container_cpu_usage_seconds_total{namespace=~&quot;.+-test&quot;, pod=~&quot;my-server-.+&quot;, image!~&quot;|.*pause.*&quot;, container!=&quot;POD&quot;}[5m])</code>.</p>
<p>The <code>container</code> label excludes the results with the <code>POD</code> string. I found that those refers to the <em>pause container</em> which holds namespace and other things before the container starts.</p>
<p>However I get pause containers in the <code>image</code> label. So I excluded them from that label.</p>
<p>Then I found some containers without the <code>image</code> label and I excluded them inserting an or (<code>|</code>) in the <code>image</code> label.</p>
<p>In some cases the cpu usage of the container without the <code>image</code> label is lower than the one of the &quot;correct&quot; container (the one with the correct <code>image</code> and <code>container</code> labels) and in other cases it is very similar, but never the same.</p>
<p>Example:</p>
<p><a href="https://i.stack.imgur.com/PKfMI.jpg" rel="nofollow noreferrer">Server 1 image</a></p>
<p><a href="https://i.stack.imgur.com/EVRzF.jpg" rel="nofollow noreferrer">Server 2 image</a></p>
<p>I would like to understand what are those containers and what they refer to.</p>
<p>PS. the metrics are from <code>cadvisor</code>.</p>

<p>Try this query:</p>
<pre><code>rate(container_cpu_usage_seconds_total{container!=&quot;POD&quot;, container=~&quot;.+&quot;}[5m])
</code></pre>
<p>In short, CPU usage is available at several resolutions (container, pod, <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#qos-classes" rel="nofollow noreferrer">QoS class</a>) and this query above effectively eliminates everything except containers that you defined explicitly. <code>!=&quot;POD&quot;</code> removes pause containers and <code>container=~&quot;.+&quot;</code> means &quot;not empty&quot;. No resolution besides &quot;per container&quot; has this label.</p>
