<p>In our Kuberenetes cluster, we are running into sporadic situations where a cluster node runs out of memory and Linux invokes OOM killer. Looking at the logs, it appears that the Pods scheduled onto the Node are requesting more memory than can be allocated by the Node.</p>

<p>The issue is that, when OOM killer is invoked, it prints out a list of processes and their memory usage. However, as all of our Docker containers are Java services, the "process name" just appears as "java", not allowing us to track down which particular Pod is causing the issues.</p>

<p>How can I get the history of which Pods were scheduled to run on a particular Node and when? </p>

<p>You can now use kube-state-metrics <code>kube_pod_container_status_terminated_reason</code> to detect OOM events</p>

<pre><code>kube_pod_container_status_terminated_reason{reason="OOMKilled"}

kube_pod_container_status_terminated_reason{container="addon-resizer",endpoint="http-metrics",instance="100.125.128.3:8080",job="kube-state-metrics",namespace="monitoring",pod="kube-state-metrics-569ffcff95-t929d",reason="OOMKilled",service="kube-state-metrics"}
</code></pre>
