<p>I tested those queries. The first query was half the value of the second query:</p>
<p><code>sum(container_memory_working_set_bytes{image!=&quot;&quot;,name=~&quot;^k8s_.*&quot;,pod=~&quot;$pod&quot;}) by (pod)</code></p>
<p>and</p>
<p><code>sum (container_memory_working_set_bytes{pod=~&quot;$pod&quot;}) by (pod)</code></p>
<p>Why is writing <code>image! = &quot;&quot;, name = ~ &quot;^ k8s_. *&quot;</code> Halving the value?</p>

<p>That's because <code>cAdvisor</code> takes these values from <code>cgroups</code>. The structure of cgroups looks like a tree, where there are branches for each pod, and every pod has child cgroups for each container in it. This is how it looks (<code>systemd-cgls</code>):</p>
<pre><code>├─kubepods
│ ├─podb0c98680-4c6d-4788-95ef-0ea8b43121d4
│ │ ├─799e2d3f0afe0e43d8657a245fe1e97edfdcdd00a10f8a57277d310a7ecf4364
│ │ │ └─5479 /bin/node_exporter --path.rootfs=/host --web.listen-address=0.0.0.0:9100
│ │ └─09ce1040f746fb497d5398ad0b2fabed1e4b55cde7fb30202373e26537ac750a
│ │   └─5054 /pause
</code></pre>
<p>The resource value for each cgroup is <em><strong>a cumulative for all its children</strong></em>. That's how you got memory utilization doubled, you just summarized the total pod consumption with each container in it.</p>
<p>If you execute those queries in Prometheus, you would notice the duplicated values:</p>
<pre><code>{pod=&quot;cluster-autoscaler-58b9c77456-krl5m&quot;} 59076608
{container=&quot;POD&quot;,pod=&quot;cluster-autoscaler-58b9c77456-krl5m&quot;} 708608
{container=&quot;cluster-autoscaler&quot;,pod=&quot;cluster-autoscaler-58b9c77456-krl5m&quot;}  58368000
</code></pre>
<p>The first one is the parent cgroup. As you see, it has no <code>container</code> label. The two others in this example are <a href="https://stackoverflow.com/questions/48651269/what-are-the-pause-containers">the pause container</a> and the actual application. Combining their values you will get the value of the parent cgroup:</p>
<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; 708608 + 58368000 == 59076608
True
</code></pre>
<p>There are multiple ways to fix the problem. For example, you can exclude metrics without container name by using <code>container!=&quot;&quot;</code> label filter.</p>
<p>Another (more difficult) way to solve this is to drop the cumulative metrics in <code>metric_relabel_configs</code> (prometheus.yml). I.e. you can write a relabeling rule that will drop metrics without a container name. <strong>Be careful with this one</strong>, you may accidentally drop all non-cadvisor metrics.</p>
