<p>I have Java pod that after few days is restarted.</p>
<p>Looking at <code>kubectl describe pod ...</code> there is just the following:</p>
<pre><code>Last State:     Terminated
  Reason:       Error
  Exit Code:    137
</code></pre>
<p>This message, in my experience, usually means that I have an OutOfMemoryError somewhere but looking at the log I don't see anything useful.</p>
<p>Is there a way to execute a script (or save few files) just before the inevitable restart?
Something that could help me to identify the problem.</p>
<p>For example: in case the restart was caused by an OutOfMemoryError, would be wonderful if I could save the memory dump or the garbage collection logs.</p>

freedev
<p>There is some solutions to do that:</p>
<ul>
<li>you can mount a volume to your application, and configure <code>log4j</code> to write the log to a file in the volume, so the log will be persistent</li>
<li>the best solution is using a log collector (fluentd, logstash) to save the log in Elastic Search or a S3 file, or using a managed service like AWS cloudWatch, datadog, ...</li>
</ul>
<p>To solve the problem of OOM, you can add a big request memory to your application (2-4G), then you can watch the memory usage by using <code>top</code> command or a tool to monitor your cluster (ex prometheus):</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: ...
spec:
  containers:
  - name: ...
    image: ...
    resources:
      requests:
        memory: &quot;2G&quot;
</code></pre>
