<p>I have a spark executor pod, which when goes to OOMKilled status, I want to alert it. I am exporting spark metrics using prometheus to grafana.</p>
<p>I have tried some queries to</p>
<pre><code>kube_pod_container_status_last_terminated_reason{reason=&quot;OOMKilled&quot;}
kube_pod_container_status_terminated_reason{reason=&quot;OOMKilled&quot;}
</code></pre>
<p>They don't seem to give proper results. I am cross checking the result using humio logs, which is logging the OOMKilled properly.</p>
<pre><code>container_memory_failures_total{pod=&quot;&lt;&lt;pod_name&gt;&gt;&quot;}
</code></pre>
<p>Even this is not able to capture the problems of OOMKilled which is in sync with the humio logs. Is there any other proper metric to catch OOMKilled ?</p>

user9920500
<p>As i know there is two metrics which allow you to monitor OOM.
The first one is used for tracking OOMKilled status of your main process/pid. If it breach the limit pod will be restarted with this status.</p>
<pre><code>kube_pod_container_status_last_terminated_reason{reason=&quot;OOMKilled&quot;}
</code></pre>
<p>And the second one for gathering total count of OOM events inside the container. So every time some child process or other process will breach the RAM limit they will be just killed and metric counter increased. But the container will be working as usual.</p>
<pre><code>container_oom_events_total
</code></pre>
