<p>I am testing out Airflow on Kubernetes. Currently, I am running a simple DAG and purposefully trying to crash it to see what happens by running a memory-intensive task. My process queries a very large table and I'm trying to generate an Out Of Memory (OOM) error. I know it's happening because the task fails but the only logs I'm able to see give a vague error:</p>
<pre><code>subprocess.CalledProcessError: Command '['/tmp/venvuydlj38n/bin/python', '/tmp/venvuydlj38n/script.py', '/tmp/venvuydlj38n/script.in', '/tmp/venvuydlj38n/script.out', '/tmp/venvuydlj38n/string_args.txt']' died with &lt;Signals.SIGKILL: 9&gt;.
</code></pre>
<p>This is the only thing I see. Nothing in Kubernetes tells me why either. The pod doesn't restart and I don't see anything on <code>kubectl get events</code>. I don't understand why Kubernetes isn't evicting the Pod. From the Kubernetes documentation, Kubernetes should kill this pod and reschedule it but it doesn't look like it's happening - yet my process still gets the SigKill 9 error. At this point I know it's an OOME, however I want to find the logs that will tell confirm that. If I get Sigkill 9 error some other reason I want to know.</p>
<p>Some other info: I set my Celery worker to have a 2GB resource limit.</p>

<p>Take a look at this issue: <a href="https://github.com/kubernetes/kubernetes/issues/50632" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/issues/50632</a></p>
<p>and this article:
<a href="https://medium.com/back-market-engineering/a-story-of-kubernetes-celery-resource-and-a-rabbit-ec2ef9e37e9f" rel="nofollow noreferrer">https://medium.com/back-market-engineering/a-story-of-kubernetes-celery-resource-and-a-rabbit-ec2ef9e37e9f</a></p>
<blockquote>
<p>When a container consumes more memory than its limit, the OOM Killer
sends a SIGKILL (signal number 9) to one of its processes.</p>
<p>If the container runs only one process, as it is often advised, the
container will then exit with the same exit code. K8s will see this
and report it, marking the container as OOMKilled. If, however, there
are multiple processes running on the container, as is the case here
with our Celery server and its 4 worker children processes, the
SIGKILL may target any of those children. The Celery server will
notice it</p>
</blockquote>
