<p>On the kubernetes dashboard, there's a pod wherein the Memory Usage (bytes) is displayed as <code>904.38Mi</code>.</p>

<p>This pod holds the java app that was ran with <code>-Xms512m -Xmx1024m</code>, and on kubernetes deployment file -> <code>requests.memory = 512M</code>, <code>limits.memory = 1.5G</code>.</p>

<p>I've enabled gc logs and seeing these on the pod logs:</p>

<pre><code>[2020-04-29T15:41:32.051+0000] GC(1533) Phase 1: Mark live objects
[2020-04-29T15:41:32.133+0000] GC(1533) Phase 1: Mark live objects 81.782ms
[2020-04-29T15:41:32.133+0000] GC(1533) Phase 2: Compute new object addresses
[2020-04-29T15:41:32.145+0000] GC(1533) Phase 2: Compute new object addresses 11.235ms
[2020-04-29T15:41:32.145+0000] GC(1533) Phase 3: Adjust pointers
[2020-04-29T15:41:32.199+0000] GC(1533) Phase 3: Adjust pointers 54.559ms
[2020-04-29T15:41:32.199+0000] GC(1533) Phase 4: Move objects
[2020-04-29T15:41:32.222+0000] GC(1533) Phase 4: Move objects 22.406ms
[2020-04-29T15:41:32.222+0000] GC(1533) Pause Full (Allocation Failure) 510M-&gt;127M(680M) 171.359ms
[2020-04-29T15:41:32.222+0000] GC(1532) DefNew: 195639K-&gt;0K(195840K)
[2020-04-29T15:41:32.222+0000] GC(1532) Tenured: 422769K-&gt;130230K(500700K)
[2020-04-29T15:41:32.222+0000] GC(1532) Metaspace: 88938K-&gt;88938K(1130496K)
[2020-04-29T15:41:32.228+0000] GC(1532) Pause Young (Allocation Failure) 603M-&gt;127M(614M) 259.018ms
[2020-04-29T15:41:32.228+0000] GC(1532) User=0.22s Sys=0.05s Real=0.26s
</code></pre>

<p>How did kubernetes arrived at <code>904.38Mi</code> usage? If I understood correctly, the current usages are only:</p>

<pre><code>DefNew (young) -      0k
Tenured        - 130230K
Metaspace      -  88938K
Sum            - 216168K
</code></pre>

<p>Running <code>ps</code> shows there are no other processes running on the pod aside from this java app.<br>
Anyone can shed some light to this?</p>

<p>(edited)
When the pod first started and let run for few mins, the memory usage is displayed as around 500mb only, then let the requests come in it will burst to 900mb-1gb, then when all has been processed, the memory usage on k8s dashboard doesn't go down below 900mb, even though based on GC logs, heap is GC'ed ok. </p>

<p>There are a lot of things going on here. Let's take each at a time.</p>
<p>It seems you are using a single container per pod (though you could have <em>many</em> containers per pod). The <code>requests.memory</code> and <code>limits.memory</code> are specific to the <em>container</em>, Kubernetes computes <code>limits</code> and <code>requests</code> per <em>pod</em> as the sum of all the containers limitations.</p>
<p>So think about it - you are saying that a <code>pod</code> shows <code>904.38Mi</code>, but you show <code>requests.memory</code> and <code>limits.memory</code>, which is per <em>container</em>. That is why I assume you have one container per pod. This is a general intro that does not answer your question - but we will get there.</p>
<p>Then comes the fact that a <code>pod</code> is started by <code>docker</code>, which is started with <code>kubectl</code> and that reads <code>requires.memory</code> and <code>limits.memory</code>. To make this a bit simpler : what you set in <code>limits.memory</code>, will be passed as <code>docker -m</code>. So, in your case, the total memory used for the docker process is <code>1.5GC</code>. Remember that is the <em>entire</em> process limit, not just the heap. A java process is a lot more than the heap, which you specify with <code>-Xms512m -Xmx1024m</code>. So to answer your question:</p>
<blockquote>
<p>How did kubernetes arrived at 904.38Mi usage?</p>
</blockquote>
<p>This is what the <em>entire</em> process is currently taking, not just the heap. From the very short log files that you posted - your application is just fine.</p>
<p><strong>EDIT</strong></p>
<p>I actually did not had kubernetes dashboard in my environment to test this specifically, so had to install it to really understand what is going on. I had a hint against the majority of things, but to make sure, I did some tests.</p>
<p>First things first: what does that number in the dashboard mean? Took a while to find/understand, but that is <a href="https://github.com/kubernetes-sigs/metrics-server/issues/187" rel="noreferrer">the actual resident memory of the process</a>, which actually is a very good thing.</p>
<p>Any sane <code>OS</code> knows that when someone requests memory from it, it rarely needs/utilizes it all, as such, it gives the memory to it in a lazy fashion. This is easy to prove in <code>k8s</code>. Suppose I have a <code>jdk-13</code> JVM and start it with:</p>
<pre><code>kubectl run jdk-13 
    --image=jdk-13 
    --image-pull-policy=Never 
    --limits &quot;memory=100Mi&quot; 
    --requests &quot;memory=10Mi&quot; 
    --command -- /bin/sh -c &quot;while true; do sleep 5; done&quot;.
</code></pre>
<p>Notice <code>requests.memory=10Mi</code> and <code>limits.memory=100Mi</code>. Reading the answer from the beginning, you already know that the specific pod will be started with <code>docker -m 100m...</code> because <code>limits.memory=100Mi</code>. That is easy to prove, just <code>sh</code> into the <code>pod</code>:</p>
<pre><code> kubectl exec -it jdk-13-b8d656977-rpzrg -- /bin/sh
</code></pre>
<p>and find out what the <code>cgroup</code> says:</p>
<pre><code> # cat /sys/fs/cgroup/memory/memory.limit_in_bytes
 104857600 // 100MB
</code></pre>
<p>perfect! so the pod's memory limitation is <code>100 MB</code> max, but what is the <em>current</em> memory utilization, that is what is the resident memory taken?</p>
<pre><code>kubectl top pod
   NAME                          CPU(cores)   MEMORY(bytes)
   jdk-13-b8d656977-rpzrg           1m           4Mi
</code></pre>
<p>OK, so the current memory utilization is only <code>4MB</code>.
You can &quot;make sure&quot; that this is indeed accurate, if you do:</p>
<pre><code>kubectl exec -it jdk-13-b8d656977-rpzrg -- /bin/sh
</code></pre>
<p>and inside that pod issue:</p>
<pre><code>top -o %MEM
</code></pre>
<p>and notice that <code>RES</code> memory is on par with the one reported via dashboard or <code>kubectl top pod</code>.</p>
<p>And now let's do a test. Suppose I have this very simple code in that pod:</p>
<pre><code>// run this with: java &quot;-Xlog:gc*=debug&quot; -Xmx100m -Xms20m  HeapTest
import java.time.temporal.ChronoUnit;
import java.util.Arrays;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.LockSupport;

public class HeapTest {

    public static void main(String[] args) throws Exception {

        // allocate 1 MB every 3 seconds
        for (int i = 0; i &lt; 40; ++i) {
            byte[] b = new byte[1024 * 1024 * 1];
            b[i] = 1;
            System.out.println(Arrays.hashCode(b));
            LockSupport.parkNanos(TimeUnit.of(ChronoUnit.SECONDS).toNanos(3));
        }
    }
}

    
</code></pre>
<p>I allocate <code>1MB</code> every 3 seconds for around 2 minutes. When I look at this process in dashboard, I do see that at some point in time, the memory grows. After the program ends, dashboard reports the drop in memory back. Good! That means the memory is given back and RSS memory drops. This is how it looks like in dashboard:</p>
<p><a href="https://i.stack.imgur.com/d3w97.png" rel="noreferrer"><img src="https://i.stack.imgur.com/d3w97.png" alt="enter image description here" /></a></p>
<p>Now let's change this code just a bit. Let's add some GC in there and let's never finish this process (you know like typical spring-boot applications do):</p>
<pre><code>import java.time.temporal.ChronoUnit;
import java.util.Arrays;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.LockSupport;

public class HeapTest {

    public static void main(String[] args) throws Exception {

        // allocate 1 MB every 3 seconds
        for (int i = 0; i &lt; 40; ++i) {
            byte[] b = new byte[1024 * 1024 * 1];
            b[i] = 1;
            System.out.println(Arrays.hashCode(b));
            LockSupport.parkNanos(TimeUnit.of(ChronoUnit.SECONDS).toNanos(3));
        }
        for (int i = 0; i &lt; 10; i++) {
            Thread.sleep(500);
            System.gc();
        }
        
        while (true) {
            try {
                Thread.sleep(TimeUnit.of(ChronoUnit.SECONDS).toMillis(5));
                Thread.onSpinWait();
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }

    }
}
</code></pre>
<p>I run this with:</p>
<pre><code>java &quot;-Xlog:heap*=debug&quot; 
     &quot;-Xlog:gc*=debug&quot; 
     &quot;-Xlog:ergo*=debug&quot; 
     -Xmx100m 
     -Xms20m
     HeapTest
</code></pre>
<p>on inspecting logs (just like in your example), I do see that heap is collected just fine. But when I look at the dashboard, the memory does not drop (unlikes the previous example).</p>
<p><a href="https://i.stack.imgur.com/KzoSa.png" rel="noreferrer"><img src="https://i.stack.imgur.com/KzoSa.png" alt="enter image description here" /></a></p>
<p>Once <code>G1GC</code> takes memory, it is not very eager to give it back to the OS. It can do that in rare cases, here is <a href="https://stackoverflow.com/questions/59362760/does-g1gc-release-back-memory-to-the-os-even-if-xms-xmx">one example</a> or <a href="https://openjdk.java.net/jeps/346" rel="noreferrer">you can instruct it to do so</a>.</p>
<p>Both ways are rather painful, instead there are <code>GC</code> algorithms that are smarter (and in general <em>a lot</em> better). My personal love goes to <code>Shenandoah</code>, let's see what it does. If I slightly change the code (so that I could prove my point better):</p>
<pre><code>import java.time.temporal.ChronoUnit;
import java.util.Arrays;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.LockSupport;

public class HeapTest {

    public static void main(String[] args) throws Exception {

        // allocate 1/4 MB every 100 ms
        for (int i = 0; i &lt; 6000; ++i) {
            byte[] b = new byte[1024 * 256];
            b[i] = 1;
            System.out.println(Arrays.hashCode(b));
            LockSupport.parkNanos(TimeUnit.of(ChronoUnit.MILLIS).toNanos(100));
        }

        while (true) {
            try {
                Thread.sleep(TimeUnit.of(ChronoUnit.SECONDS).toMillis(5));
                Thread.onSpinWait();
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }

    }
}
</code></pre>
<p>And run it with:</p>
<pre><code> java &quot;-Xlog:gc*=debug&quot; 
      &quot;-Xlog:ergo*=debug&quot; 
      &quot;-Xlog:heap*=debug&quot; 
       -XX:+UnlockExperimentalVMOptions 
       -XX:+UseShenandoahGC 
       -XX:+ShenandoahUncommit 
       -XX:ShenandoahGCHeuristics=compact  
       -Xmx1g 
       -Xms1m  
       HeapTest
</code></pre>
<p>Here is what you are going to see:</p>
<p><a href="https://i.stack.imgur.com/rXLQO.png" rel="noreferrer"><img src="https://i.stack.imgur.com/rXLQO.png" alt="heap3" /></a></p>
<p>And you <a href="https://bugs.openjdk.java.net/browse/JDK-8204089" rel="noreferrer">should, for a fact care about this</a>:</p>
<blockquote>
<p><strong>This behavior is particularly disadvantageous in container environments where resources are paid by use. Even during phases where the VM only uses a fraction of its assigned memory resources due to inactivity, G1 will retain all of the Java heap. This results in customers paying for all resources all the time, and cloud providers not being able to fully utilize their hardware.</strong></p>
</blockquote>
<p>P.S. I would also add to that the fact that <em>other</em> pods are suffering too, because one pod decided to take as much memory as it could, at a particular spike, and never give it back.</p>
