<p>I'm looking to understanding the relationship of</p>
<p><strong>container_memory_working_set_bytes</strong> vs <strong>process_resident_memory_bytes</strong> vs <strong>total_rss</strong> (container_memory_rss) + <strong>file_mapped</strong> so as to better equipped system for alerting on OOM possibility.</p>
<p><a href="https://i.stack.imgur.com/DTBSe.png" rel="noreferrer"><img src="https://i.stack.imgur.com/DTBSe.png" alt="enter image description here" /></a></p>
<p>It seems against my understanding (which is puzzling me right now) given <strong>if a container/pod is running a single process executing a compiled program written in Go.</strong></p>
<p>Why is the difference between <code>container_memory_working_set_bytes</code> is so big(nearly 10 times more) with respect to <code>process_resident_memory_bytes</code></p>
<p>Also the relationship between <code>container_memory_working_set_bytes</code> and <code>container_memory_rss + file_mapped</code> is weird here, something I did not expect, after reading <a href="https://stackoverflow.com/questions/65428558/what-is-the-difference-between-container-memory-working-set-bytes-and-contain">here</a></p>
<blockquote>
<p>The total amount of anonymous and swap cache memory (it includes transparent hugepages), and it equals to the value of total_rss from memory.status file. This should not be confused with the true resident set size or the amount of physical memory used by the cgroup. rss + file_mapped will give you the resident set size of cgroup. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory.</p>
</blockquote>
<p>So <code>cgroup</code> total resident set size is <code>rss + file_mapped</code> how does this value is less than <code>container_working_set_bytes</code> for a container that is running in the given cgroup</p>
<p>Which make me feels something with this stats that I'm not correct.</p>
<p>Following are the PROMQL used to build the above graph</p>
<ul>
<li>process_resident_memory_bytes{container=&quot;sftp-downloader&quot;}</li>
<li>container_memory_working_set_bytes{container=&quot;sftp-downloader&quot;}</li>
<li>go_memstats_heap_alloc_bytes{container=&quot;sftp-downloader&quot;}</li>
<li>container_memory_mapped_file{container=&quot;sftp-downloader&quot;} + container_memory_rss{container=&quot;sftp-downloader&quot;}</li>
</ul>

<p>So the relationship seems is like this</p>
<pre><code>container_working_set_in_bytes = container_memory_usage_bytes - total_inactive_file
</code></pre>
<p><code>container_memory_usage_bytes</code> as its name implies means the total memory used by the container (but since it also includes file cache i.e inactive_file which OS can release under memory pressure) substracting the inactive_file gives <code>container_working_set_in_bytes</code></p>
<p>Relationship between <code>container_memory_rss</code> and <code>container_working_sets</code> can be summed up using following expression</p>
<pre><code>container_memory_usage_bytes = container_memory_cache + container_memory_rss 
</code></pre>
<p>cache reflects data stored on a disk that is currently cached in memory. it contains active + inactive file (mentioned above)</p>
<p>This explains why the <code>container_working_set</code> was higher.</p>
<p>Ref <a href="https://www.datadoghq.com/blog/how-to-monitor-docker-resource-metrics/#memory" rel="noreferrer">#1</a></p>
<p>Ref <a href="https://www.docker.com/blog/gathering-lxc-docker-containers-metrics/#memory-metrics" rel="noreferrer">#2</a></p>
