<p>My Java microservices are running in k8s cluster hosted on AWS EC2 instances.</p>

<p>I have around 30 microservice(a good mix of nodejs and Java 8) running in a K8s cluster. I am facing a challange where my java application pods gets restart unexpectedly which leads to increase in application 5xx count.</p>

<p>To debug this, I started a newrelic agent in pod along with application and found the following graph:</p>

<p><a href="https://i.stack.imgur.com/lSUfE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/lSUfE.png" alt="enter image description here"></a></p>

<p>Where I can see that, I have Xmx value as 6GB and my uses is max 5.2GB.</p>

<p>This clearly stats that JVM is not crossing the Xmx value.</p>

<p>But when I describe the pod and look for last state it says "Reason:Error" with "Exit code: 137"</p>

<p><a href="https://i.stack.imgur.com/PPHVw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PPHVw.png" alt="enter image description here"></a></p>

<p>Then on further investigation I find that my Pod average memory uses is close to its limit all the time.(Allocated 9Gib, uses ~9Gib). I am not able to understand why memory uses is so high in Pod even thogh I have only one process running((JVM) and that too is restricted with 6Gib Xmx.</p>

<p><a href="https://i.stack.imgur.com/Aw06D.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Aw06D.png" alt="enter image description here"></a> </p>

<p>When I login to my worker nodes and check the status of docker containers I can see the last container of that appriction with Exited state and says "Container exits with non-zero exit code 137"</p>

<p>I can see the wokernode kernel logs as:</p>

<p><a href="https://i.stack.imgur.com/LtDL0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LtDL0.png" alt="enter image description here"></a></p>

<p>which shows kernel is terminitaing my process running inside container.</p>

<p>I can see I have lot of free memory in my worker node.</p>

<p><a href="https://i.stack.imgur.com/R9uLY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/R9uLY.png" alt="enter image description here"></a></p>

<p>I am not sure why my pods get restart again and again is this k8s behaviour or something spoofy in my infrastructure. This force me to move my application from Container to VM again as this leades to increase in 5xx count.</p>

<p>EDIT: I am getting OOM after increasing memory to 12GB.</p>

<p><a href="https://i.stack.imgur.com/EWPrR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EWPrR.png" alt="enter image description here"></a></p>

<p>I am not getting sure why POD is getting killed because of OOM th
ough JVM xmx is 6 GB only.</p>

<p>Need help!</p>

<p>Some older Java versions( prior to Java 8 u131 release)  don’t recognize that they are running in a container.  So even if you specify maximum heap size for the JVM with -Xmx, the JVM will set the maximum heap size based on the host’s total memory instead of the memory available to the container and then when a process tries to allocate memory over its limit(defined in a pod/deployment spec) your container is getting OOMKilled.</p>

<p>These problems might not pop up when running your Java apps in K8 cluster locally, because the difference between pod memory limit and total local machine memory aren’t big. But when you run it in production on nodes with more memory available, then JVM may go over your container memory limit and will be OOMKilled.</p>

<p>Starting from Java 8(u131 release) it is possible to make JVM be “container-aware” so that it recognizes constraints set by container control groups (cgroups).</p>

<p>For <strong>Java 8</strong>(from U131 release) <strong>and Java9</strong> you can set this experimental flags to JVM:</p>

<pre><code>-XX:+UnlockExperimentalVMOptions 
-XX:+UseCGroupMemoryLimitForHeap
</code></pre>

<p>It will set the heap size based on your container cgroups memory limit, which is defined as "resources: limits" in your container definition part of the pod/deployment spec. 
There still probably can be cases of JVM’s off-heap memory increase in Java 8, so you might monitor that, but overall those experimental flags must be handling that as well.</p>

<p>From <strong>Java 10</strong>  these experimental flags are the new default and are enabled/disabled by using this flag:</p>

<pre><code> -XX:+UseContainerSupport 
 -XX:-UseContainerSupport
</code></pre>
