<p>I am running apache in Kubernetes pod and I am troubled by random OOM kills of the apache process and I am unsure of the cause.</p>
<p>The pod has 600M memory limit and it seems it can consume it quite easily even though PHP has 128 MB limit per run. When observing <code>dmesg -w</code> I can regularly see:</p>
<pre><code>[4019392.626796] Memory cgroup out of memory: Kill process 4178127 (apache2) score 1137 or sacrifice child
[4019392.636520] Killed process 4178127 (apache2) total-vm:143960kB, anon-rss:22856kB, file-rss:10472kB, shmem-rss:28228kB
</code></pre>
<ol>
<li>How can I be sure, that this OOM kill is related to the machine or to the pod limits?</li>
<li>There are several apache pods and I am not sure how to recognize the pod to which this process belonged. Is there a way?</li>
<li>It would be great if these were accessible within the pods, so one could see killed threads which actually belonged to the pod (ideally in error output). Is it possible?</li>
</ol>

Vojtěch
<p>Covering your questions in the same order:</p>
<ol>
<li><p><code>kubectl get events</code> in the namespace where you run a pod. You will see Kubernetes OOM events among others. If there's nothing - it will be the kernel itself.</p>
</li>
<li><p>There's a <a href="https://github.com/heptiolabs/pid2pod" rel="nofollow noreferrer">project called <code>pid2pod</code></a>, but it greps only what's still running. The other option will be to route dmesg to stdout (on container OS level) and implement <a href="https://github.com/helm/charts/blob/d339305a27326a98681e654eb5f4f9bd6882f3aa/stable/fluent-bit/templates/config.yaml#L82" rel="nofollow noreferrer">centralized log collection with Kubernetes metadata enrichment</a>. Then every log event will have pod id in the metadata.</p>
</li>
<li><p>Not sure what are you trying to solve here. Usually, one pod (and I assume one container) is one process. Once it killed by OOM, there's nothing else that can take an action on this inside the pod. It's the controller's (<code>Deployment</code>) responsibility to react to this event and recreate the pod.</p>
<p>If you just want to watch for OOMs within the remaining pods you may implement smth like <a href="https://github.com/bitnami-labs/kubewatch" rel="nofollow noreferrer">kubewatch</a>. Unfortunately, I don't have a project that lets you subscribe within a pod directly, but this can be used as an example. Or in case of centralized logging, you can subscribe to a log stream inside a pod.</p>
</li>
</ol>
