<p><strong>Situation:</strong> I just set up Prometheus for my Kubernetes cluster, and wanted to take a closer look at resource usage. Seemed like the way to do this was with <code>container_cpu_usage_seconds_total</code>, but I'm confused by the results.</p>
<p><strong>Problem:</strong> Prometheus seems to be returning three different elements for each (single-container) pod. I don't know what these different elements represent, and haven't been able to find a clear explanation on Google.</p>
<hr>
<p><strong>What I'm doing:</strong></p>
<p>I'm executing this query, where MY-POD-IDENTIFIER is the name of a Kubernetes Pod with one defined container:</p>
<pre><code>rate(container_cpu_usage_seconds_total{pod=&quot;MY-POD-IDENTIFIER&quot;}[5m])
</code></pre>
<p><strong>What I'm seeing:</strong></p>
<p>The <code>console</code> tab gives me a table of (element, value) with three entries, as seen in the image below. I've included a simplified list of the returned elements below.</p>
<p><a href="https://i.stack.imgur.com/4AnNx.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4AnNx.png" alt="enter image description here" /></a></p>
<pre><code>[
{
    beta_kubernetes_io_arch = &quot;amd64&quot;,
    beta_kubernetes_io_instance_type = &quot;m5a.xlarge&quot;,
    beta_kubernetes_io_os = &quot;linux&quot;,
    container = &quot;POD&quot;,
    cpu = &quot;total&quot;,
    id = &quot;/kubepods/podSOME-LONG-ID/A-LONG-ID&quot;,
    image = &quot;602401143452.dkr.ecr.ap-southeast-1.amazonaws.com/eks/pause-amd64:3.1&quot;,
    instance = &quot;MY-INTERNAL-NODE-INSTANCE&quot;,
    job = &quot;kubernetes-nodes-cadvisor&quot;,
    name = &quot;k8s_POD_MY-POD-IDENTIFIER-WITH-LONGID&quot;,
    namespace = &quot;staging&quot;,
    pod = &quot;MY-POD-IDENTIFIER&quot;
},
{
    beta_kubernetes_io_arch = &quot;amd64&quot;,
    beta_kubernetes_io_instance_type = &quot;m5a.xlarge&quot;,
    beta_kubernetes_io_os = &quot;linux&quot;,
    container = &quot;MY-CONTAINER-IMAGE-NAME&quot;,
    cpu = &quot;total&quot;,
    id = &quot;/kubepods/podSOME-LONG-ID/A-DIFFERENT-LONG-ID&quot;,
    image = &quot;111111111.dkr.ecr.ap-southeast-1.amazonaws.com/MY-KUBERNETES-CONTAINER-IMAGE&quot;,
    instance = &quot;MY-INTERNAL-NODE-INSTANCE&quot;,
    job = &quot;kubernetes-nodes-cadvisor&quot;,
    name = &quot;k8s_MY-POD-NAME_MY-POD-IDENTIFIER-WITH-LONGID&quot;,
    namespace = &quot;staging&quot;,
    pod = &quot;MY-POD-IDENTIFIER&quot;
},
{
    beta_kubernetes_io_arch = &quot;amd64&quot;,
    beta_kubernetes_io_instance_type = &quot;m5a.xlarge&quot;,
    beta_kubernetes_io_os = &quot;linux&quot;,
    cpu = &quot;total&quot;,
    id = &quot;/kubepods/podSOME-LONG-ID&quot;,
    instance = &quot;MY-INTERNAL-NODE-INSTANCE&quot;,
    job = &quot;kubernetes-nodes-cadvisor&quot;,
    namespace = &quot;staging&quot;,
    pod = &quot;MY-POD-IDENTIFIER&quot;
}]
</code></pre>
<p>The <code>graph</code> tab gives me this view:
<a href="https://i.stack.imgur.com/MYY0I.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MYY0I.png" alt="enter image description here" /></a></p>
<p>These lines match up to the previous list of elements:</p>
<ol>
<li>First element: red</li>
<li>Second element: light green</li>
<li>Third element: dark green</li>
</ol>
<hr>
<p><strong>My Guesses:</strong></p>
<p>Looking at the list of elements, one of the differentiating factors seems to be the <code>container</code> value. This is respectively: [<code>POD</code>, <code>MY-CONTAINER-IMAGE-NAME</code>, <code>None</code>]. The <code>POD</code> element has CPU usage of 0 throughout, while the other elements have similar-but-varying CPU usage of 0.6m.</p>
<p>My <em>guess</em> was that the <code>POD</code> element represents a kubernetes helper container within the Pod, the <code>MY-CONTAINER-IMAGE-NAME</code> element represents the container I defined, and the <code>None</code> element is a wrapper for both of those. This seems likely because of how closely the <code>None</code> element tracks the <code>MY-CONTAINER-IMAGE-NAME</code> element. But on the other hand, there is variation between them. So I'm not sure.</p>
<p><strong>My Question:</strong></p>
<p>Can someone please explain what these different Prometheus elements are? Or point me in the direction of a good resource?</p>

Hebe Hilhorst
<p>The &quot;elements&quot; that you see are the &quot;targets&quot; that Prometheus has discovered.  To fully understand how/what this is, the <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config" rel="nofollow noreferrer">Prometheus Kubernetes Service Discovery docs</a> are gonna be the most helpful.</p>
<p>Basically (without seeing your prom config to verify which one), Prometheus is finding 3x targets per pod per some of the rules you see on that page.  E.g. with <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod" rel="nofollow noreferrer"><code>pods</code></a> it will find 1 target per exposed port.</p>
<p>After you've figured out which of these you want and which you don't need, you can use the <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config" rel="nofollow noreferrer"><code>relabel_configs</code></a> to drop the targets that are extra and you don't need to scrape.</p>
