<p>I'm trying to make sense of container_memory_rss or container_memory_working_set_bytes with respect to <strong>node_memory_used</strong> i.e (<em>node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes</em>)</p>
<p>Here what I meant</p>
<p><strong>PROMQL 1:</strong></p>
<pre><code>sum(container_memory_rss) by (instance) / 1024 / 1024 / 1024

{instance=&quot;172.19.51.8:10250&quot;}        7.537441253662109
</code></pre>
<p><strong>PROMQL 2:</strong></p>
<pre><code>sum(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) by (instance) / 1024 / 1024 / 1024

{instance=&quot;172.19.51.8:9100&quot;}         2.2688369750976562
</code></pre>
<p><strong>PROMQL 3:</strong></p>
<pre><code>sum(container_memory_working_set_bytes) by (instance) / 1024 / 1024 / 1024

{instance=&quot;172.19.51.8:10250&quot;}        9.285114288330078
</code></pre>
<p><strong>PROMQL 4:</strong></p>
<pre><code>sum(node_memory_MemAvailable_bytes) by (instance) / 1024 / 1024 / 1024

{instance=&quot;172.19.51.8:9100&quot;}         13.356605529785156
</code></pre>
<p>So if a Pod always runs on a Node. I fail to understand why is <strong>container_memory_rss</strong> or  <strong>container_memory_working_set_bytes</strong> is more than <strong>node_memory_used</strong></p>
<p>i.e values for PROMQL 1 and PROMQL 3 are way more than the value of PROMSQL 2 i.e the memory used by the node.</p>
<p>I don't know if I'm correct shouldn't the pod / container rss should always be &lt;= nodes memory used (even if no default resource limit is set)</p>

<h2>tl;dr</h2>
<p>Use container name filter (<code>container!=&quot;&quot;</code>) to exclude totals:</p>
<pre><code>sum(container_memory_rss{container!=&quot;&quot;}) by (instance) / 2^30
</code></pre>
<h2>Explanation</h2>
<p>If you ran the first query grouping results by container name, you would have noticed that most of the usage comes from a container without a name:</p>
<pre><code>sort_desc(sum(container_memory_rss{instance=&quot;ip-192-168-104-46&quot;}) by (name)) / 2^30

{}                          3.9971389770507812
{name=&quot;prometheus&quot;}         0.6084518432617188
{name=&quot;cluster-autoscaler&quot;} 0.04230499267578125
</code></pre>
<p>Actually there are several entries without name but they all have an <code>id</code>:</p>
<pre><code>sort_desc(sum(container_memory_rss{instance=&quot;ip-192-168-104-46&quot;}) by (id)) / 2^30

# these do not have a container name
{id=&quot;/&quot;}                                1.1889266967773438
{id=&quot;/kubepods&quot;}                        0.900482177734375
{id=&quot;/kubepods/burstable&quot;}              0.6727218627929688
{id=&quot;/system.slice/docker.service&quot;}     0.07495498657226562
{id=&quot;/system.slice/kubelet.service&quot;}    0.060611724853515625

# and this is an example id of a real container which has a name label
{id=&quot;/kubepods/burstable/pod562495f9-afa6-427e-8435-016c2b500c74/e73975d90b66772e2e17ab14c473a2d058c0b9ffecc505739ee1a94032728a78&quot;} 0.6027107238769531
</code></pre>
<p>These are accumulated values for each <code>cgroup</code>. <code>cAdvisor</code> takes the stats from <code>cgroups</code> and if you looks at them, you will find familiar entities:</p>
<pre><code># systemd-cgls -a
├─kubepods
│ ├─podc7dfcc4e-74fc-4469-ad56-c13fe5a9e7d8
│ │ ├─61a1a58e47968e7595f3458a6ded74f9088789a865bda2be431b8c8b07da1c6e
│ │ └─d47601e38a96076dd6e0205f57b0c365d4473cb6051eb0f0e995afb31143279b
│ ├─podfde9b8ca-ce80-4467-ba05-03f02a14d569
│ │ ├─9d3783df65085d54028e2303ccb2e143fecddfb85d7df4467996e82691892176
│ │ └─47702b7977bed65ddc86de92475be8f93b50b06ae8bd99bae9710f0b6f63d8f6
│ ├─burstable
│ │ ├─pod9ff634a5-fd2a-42e2-be27-7e1028e96b67
│ │ │ ├─5fa225aad10bdc1be372859697f53d5517ad28c565c6f1536501543a071cdefc
│ │ │ └─27402fed2e4bb650a6fc41ba073f9994a3fc24782ee366fb8b93a6fd939ba4d3
</code></pre>
<p>If you sum up all direct children of, say <code>kubepods</code>, you will get the same value <code>kubepods</code> has. Because of these totals <code>sum(container_memory_rss) by (instance)</code> shows several times the actual resource utilisation.</p>
<p>The solution is just to filter out any values without a container name. You can either do that when querying, as in the example at the top, or configure Prometheus with <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config" rel="nofollow noreferrer">relabel_config</a> to drop such metrics at the scrape time.</p>
