<p>I'm trying to find out and understand how OOM-killer works on the container.</p>
<p>To figuring it out, I've read lots of articles and I found out that OOM-killer kills container  based on the <code>oom_score</code>. And <code>oom_score</code> is determined by <code>oom_score_adj</code> and memory usage of that process.</p>
<p>And there're two metrics <code>container_memory_working_set_bytes</code>, <code>container_memory_rss</code> from the cAdvisor for monitoring memory usage of the container.</p>
<p>It seems that RSS memory (<code>container_memory_rss</code>) has impact on <code>oom_score</code> so I can understand that with the <code>container_memory_rss</code> metric, if that metric reached to memory limit, the OOM-killer will kill the process.</p>
<ul>
<li><a href="https://github.com/torvalds/linux/blob/v3.10/fs/proc/base.c#L439" rel="noreferrer">https://github.com/torvalds/linux/blob/v3.10/fs/proc/base.c#L439</a></li>
<li><a href="https://github.com/torvalds/linux/blob/v3.10/mm/oom_kill.c#L141" rel="noreferrer">https://github.com/torvalds/linux/blob/v3.10/mm/oom_kill.c#L141</a></li>
<li><a href="https://github.com/torvalds/linux/blob/v3.10/include/linux/mm.h#L1136" rel="noreferrer">https://github.com/torvalds/linux/blob/v3.10/include/linux/mm.h#L1136</a></li>
</ul>
<p>But from the articles like below:</p>
<ul>
<li><a href="https://faun.pub/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d" rel="noreferrer">https://faun.pub/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d</a></li>
<li><a href="https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e66" rel="noreferrer">https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e66</a></li>
</ul>
<blockquote>
<p>The better metric is <code>container_memory_working_set_bytes</code> as this is what the OOM killer is watching for.</p>
</blockquote>
<p><strong>I cannot understand the fact that OOM-killer is watching for container's working set memory.</strong> I think I'm not understand the meaning of the working set memory on the container which is 'total usage - inactive file'.</p>
<ul>
<li><a href="https://github.com/google/cadvisor/issues/2582#issuecomment-644883028" rel="noreferrer">https://github.com/google/cadvisor/issues/2582#issuecomment-644883028</a></li>
</ul>
<p>Where can I find the reference? Or could you explain the relationship between working set memory and OOM-kill on the container?</p>

<p>As you already know, <code>container_memory_working_set_bytes</code> is:</p>
<blockquote>
<p>the amount of working set memory and it includes recently accessed
memory, dirty memory, and kernel memory. Therefore, Working set is
(lesser than or equal to) &lt;/= &quot;usage&quot;.</p>
</blockquote>
<p>The <code>container_memory_working_set_bytes</code> is being used for OoM decisions because it <strong>excludes cached data</strong> (<a href="https://www.thomas-krenn.com/en/wiki/Linux_Page_Cache_Basics" rel="noreferrer">Linux Page Cache</a>) that can be evicted in memory pressure scenarios.</p>
<p>So, if the <code>container_memory_working_set_bytes</code> is increased to the limit, it will lead to oomkill.</p>
<p>You can find the fact that when Linux kernel checking available memory, it calls <a href="https://github.com/torvalds/linux/blob/master/mm/util.c#L872" rel="noreferrer"><code>vm_enough_memory()</code></a> to find out how many pages are potentially available.</p>
<p>Then when the machine is low on memory, old page frames including cache will be reclaimed but kernel still may find that it was unable free enough pages to satisfy a request. Now it's time to call <a href="https://github.com/torvalds/linux/blob/master/mm/oom_kill.c#L1048" rel="noreferrer"><code>out_of_memory()</code></a> to kill the process. To determine the candidate process to be killed it uses <code>oom_score</code>.</p>
<p>So when Working Set bytes reached to limits, it means that kernel cannot find availables pages even after reclaiming old pages including cache so kernel will trigger OOM-killer to kill the process.</p>
<p>You can find more details on the Linux kernel documents:</p>
<ul>
<li><a href="https://www.kernel.org/doc/gorman/html/understand/understand016.html" rel="noreferrer">https://www.kernel.org/doc/gorman/html/understand/understand016.html</a></li>
<li><a href="https://www.kernel.org/doc/gorman/html/understand/understand013.html" rel="noreferrer">https://www.kernel.org/doc/gorman/html/understand/understand013.html</a></li>
</ul>
