<p>Recently, the same container of several pods in a deployment restarted with <code>OOMKilled</code> event.
Here is the description of one of the containers:</p>
<pre><code>State:          Running
      Started:      Tue, 15 Feb 2022 23:33:06 +0000
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    1
      Started:      Fri, 11 Feb 2022 17:48:21 +0000
      Finished:     Tue, 15 Feb 2022 23:33:05 +0000
    Ready:          True
    Restart Count:  1
    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:      1
      memory:   512Mi
</code></pre>
<p>If the container would exceed the limit of the available memory then it would exit with code <code>137</code>. And I guess the container did not reached the limit. So my question what could happen if the exit code is 1 and the Reason is <code>OOMKilled</code>.</p>
<p><strong>Update:</strong>
The process is actually a python app which has threads, this is the code:</p>
<pre><code>ret = subprocess.run(args, stderr=subprocess.PIPE, universal_newlines=True, check=False)
if ret.returncode != 0:
    logging.warning(&quot;Executing cmd failed: %s, code: %d, stderr: %s&quot;, cmd, ret.returncode, ret.stderr)
    raise Exception(&quot;Failed&quot;)
</code></pre>
<p>and the relevant logs when called, it return with <code>-9</code>:</p>
<pre><code>2022-02-15T23:33:30.510Z WARNING &quot;MainThread - Executing cmd failed: iptables-restore -n -w 3 restore-filter, code: -9, stderr: &quot;
raise Exception(&quot;Failed&quot;)
Exception: Failed
</code></pre>
<p>from the description of subprocess.run(): <code>A negative value -N indicates that the child was terminated by signal N (POSIX only).</code></p>
<p>So because the exception is raised the python code exited with 1? Probably..</p>

<p>Two possible reasons:</p>
<h4>Reason #1</h4>
<p>Subprocess was killed by OOM killer (it received SIGKILL(9) from OOM killer), resulting in application crashing with <em>exit code 1</em>, and <em>OOMKilled</em> reason for termination.</p>
<h4>Reason #2</h4>
<p>If you have <code>initContainers</code> specified, init container could have been killed by OOM killer, resulting in <em>OOMKilled</em> reason, and application crashing with <em>exit code 1</em> due to the bad initialization.</p>
<hr />
<p>OOM kill is not very well documented in Kubernetes docs. For example</p>
<blockquote>
<p>Containers are marked as OOM killed only when the init pid gets killed by the kernel OOM killer. There are apps that can tolerate OOM kills of non init processes and so we chose to not track non-init process OOM kills.
<sup>[<a href="https://github.com/kubernetes/kubernetes/issues/50632#issuecomment-322849618" rel="nofollow noreferrer">source</a>]</sup></p>
</blockquote>
<p>I could not find any mentions of it anywhere, othen than this GitHub issue.</p>
<hr />
<p>First reason is more probable in my opinion.<br />
Possible solution is to increase memory limits (if you have any).</p>
