<p>I am running Docker containers containing JVM (java8u31). These containers are deployed as pods in a kubernetes cluster. Often I get OOM for the pods and Kubernetes kills the pods and restarts it. I am having issues in finding the root cause for these OOMs as I am new to Kubernetes.</p>

<ol>
<li><p>Here are the JVM parameters</p>

<pre><code>-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Xms700M -Xmx1000M  -XX:MaxRAM=1536M  -XX:MaxMetaspaceSize=250M 
</code></pre></li>
<li><p>These containers are deployed as stateful set and following is the resource allocation</p>

<pre><code>resources:
    requests:
        memory: "1.5G"
        cpu: 1
    limits:
        memory: "1.5G"
        cpu: 1
</code></pre>

<p>so the total memory allocated to the container matches the MaxRam </p></li>
<li><p>If I use <code>-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/etc/opt/jmx/java_pid%p.hprof</code> that doesn't help because the pod is getting killed and recreated and started as soon as there is a OOM so everything within the pod is lost</p>

<p>The only way to get a thread or HEAP dump is to SSH into the pod which also I am not able to take because the pod is recreated after an OOM so I don't get the memory footprint at the time of OOM. I SSH after an OOM which is not much help.</p></li>
<li><p>I also profiled the code using visualVM, jHat but couldn't find substantial memory footprint which could lead to a conclusion of too much memory consumption by the threads running within the JVM or a probable leak.</p></li>
</ol>

<p>Any help is appreciated to resolve the OOM thrown by Kubernetes.</p>

<p>When your application in a pod reaches the limits of memory you set by resources.limits.memory or namespace limit, Kubernetes restarts the pod.</p>
<p>The Kubernetes part of limiting resources is described in the following articles:</p>
<ul>
<li><a href="https://cloudplatform.googleblog.com/2018/05/Kubernetes-best-practices-Resource-requests-and-limits.html" rel="nofollow noreferrer">Kubernetes best practices: Resource requests and limits</a></li>
<li><a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/" rel="nofollow noreferrer">Resource Quotas</a></li>
<li><a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/admission_control_resource_quota.md" rel="nofollow noreferrer">Admission control plugin: ResourceQuota</a></li>
<li><a href="https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/" rel="nofollow noreferrer">Assign Memory Resources to Containers and Pods</a></li>
</ul>
<p>Memory consumed by Java application is not limited to the size of the Heap that you can set by specifying the options:</p>
<pre><code>-Xmssize Specifies the initial heap size.
-Xmxsize Specifies the maximum heap size.
</code></pre>
<p>Java application needs some additional memory for metaspace, class space, stack size, and JVM itself needs even more memory to do its tasks like garbage collection, JIT optimization, Off-heap allocations, JNI code.
It is hard to predict total memory usage of JVM with reasonable precision, so the best way is to measure it on the real deployment with usual load.</p>
<p>I would recommend you to set the Kubernetes pod limit to double <code>Xmx</code> size, check if you are not getting OOM anymore, and then gradually decrease it to the point when you start getting OOM. The final value should be in the middle between these points.<br />
You can get more precise value from memory usage statistics in a monitoring system like Prometheus.</p>
<p>On the other hand, you can try to limit java memory usage by specifying the number of  available options, like the following:</p>
<pre><code>-Xms&lt;heap size&gt;[g|m|k] -Xmx&lt;heap size&gt;[g|m|k]
-XX:MaxMetaspaceSize=&lt;metaspace size&gt;[g|m|k]
-Xmn&lt;young size&gt;[g|m|k]
-XX:SurvivorRatio=&lt;ratio&gt;
</code></pre>
<p>More details on that can be found in these articles:</p>
<ul>
<li><a href="https://medium.com/@matt_rasband/dockerizing-a-spring-boot-application-6ec9b9b41faf" rel="nofollow noreferrer">Properly limiting the JVM’s memory usage (Xmx isn’t enough)</a></li>
<li><a href="https://plumbr.io/blog/memory-leaks/why-does-my-java-process-consume-more-memory-than-xmx" rel="nofollow noreferrer">Why does my Java process consume more memory than Xmx</a></li>
</ul>
<p>The second way to limit JVM memory usage is to calculate heap size based on the amount of RAM(or MaxRAM). There is a good explanation of how it works in the <a href="https://web.archive.org/web/20191216024717/http://what-when-how.com:80/Tutorial/topic-684cn3k/Java-Performance-The-Definitive-Guide-218.html" rel="nofollow noreferrer">article</a>:</p>
<blockquote>
<p>The default sizes are based on the amount of memory on a machine, which can be set with the  <code>-XX:MaxRAM=N</code>  flag.
Normally, that value is calculated by the JVM by inspecting the amount of memory on the machine.
However, the JVM limits  <code>MaxRAM</code>  to <code>1 GB</code> for the client compiler, <code>4 GB</code> for 32-bit server compilers, and <code>128 GB</code> for 64-bit compilers.
The maximum heap size is one-quarter of  <code>MaxRAM</code> .
This is why the default heap size can vary: if the physical memory on a machine is less than  <code>MaxRAM</code> , the default heap size is one-quarter of that.
But even if hundreds of gigabytes of RAM are available, the most the JVM will use by default is <code>32 GB</code>: one-quarter of <code>128 GB</code>. The default maximum heap calculation is actually this:</p>
</blockquote>
<blockquote>
<p><code>Default Xmx = MaxRAM / MaxRAMFraction</code></p>
</blockquote>
<blockquote>
<p>Hence, the default maximum heap can also be set by adjusting the value of the  - <code>XX:MaxRAMFraction=N</code>  flag, which defaults to <code>4</code>.
Finally, just to keep things interesting, the <code>-XX:ErgoHeapSizeLimit=N</code>  flag can also be set to a maximum default value that the JVM should use.
That value is <code>0</code> by default (meaning to ignore it); otherwise, that limit is used if it is smaller than  <code>MaxRAM  /  MaxRAMFraction</code> .</p>
</blockquote>
<blockquote>
<p>The initial heap size choice is similar, though it has fewer complications. The initial heap size value is determined like this:</p>
</blockquote>
<blockquote>
<p><code>Default Xms =  MaxRAM / InitialRAMFraction</code></p>
</blockquote>
<blockquote>
<p>As can be concluded from the default minimum heap sizes, the default value of the  <code>InitialRAMFraction</code>  flag is <code>64</code>.
The one caveat here occurs if that value is less than <code>5 MB</code> —or, strictly speaking, less than the values specified by  <code>-XX:OldSize=N</code>  (which defaults to <code>4 MB</code>) plus  -<code>XX:NewSize=N</code>  (which defaults to <code>1 MB</code>).
In that case, the sum of the old and new sizes is used as the initial heap size.</p>
</blockquote>
<p>This article gives you a good point to start tuning your JVM for web-oriented application:</p>
<ul>
<li><a href="https://web.archive.org/web/20220326090501/https://blog.sokolenko.me/2014/11/javavm-options-production.html" rel="nofollow noreferrer">Java VM Options You Should Always Use in Production</a></li>
</ul>
