<p>Situation :<br />
There are two containers in the prometheus pod ( config-reloader, prometheus )<br />
I set the resources.limits.memory as 50Mi, 32Gi respectively.</p>
<p>The metric container_memory_failcnt has been increased dramatically from 10 to 8000 within 5 minutes  (precisely rate(container_memory_failcnt{}[5m]) )</p>
<p>The mertic container_memory_failcnt tells how many times the container hits memory limit.</p>
<p>But according to the metric container_memory_working_set_bytes, the prometheus container used 18Gi of memory.<br />
The pod is not killed by OOM neither. But the metric container_memory_failcnt increased dramatically.</p>
<p>OOM is different from hitting memory limit?</p>
<p>And I want to know some candidates why the prometheus container used the memory(18Gi) so much within 5 minutes.  (It usually uses 10Gi or below )</p>

JAESANGPARK
<p>After searching on google 2 days, I got to know the answer.<br />
The container_memory_failcnt really checks whether the target container hits memory limits.<br />
This metric is along with the container_memory_usage_bytes.</p>
<p>And the metric container_memory_working_set_bytes is the actual memory size which the container currently is using.<br />
And the oom killer is watching this metrics.</p>
<p>So in my case, the container_memory_failcnt had been increasing but the container_memory_working_set_bytes was lower than the limits.memory of the container so that the pod was not oom killed.</p>
<p>special thanks to   Bob cotton </p>
<p><a href="https://faun.pub/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d" rel="nofollow noreferrer">https://faun.pub/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d</a></p>
