<p>I have a Kubernetes Pod that has </p>

<ul>
<li>Requested Memory of 1500Mb</li>
<li>Memory Limit of 2048Mb</li>
</ul>

<p>I have 2 containers running inside this pod, one is the actual application (heavy Java app) and a lightweight log shipper.</p>

<p>The pod consistently reports a usage of 1.9-2Gb of memory usage. Because of this, the deployment is scaled (an autoscaling configuration is set which scales pods if memory consumption > 80%), naturally resulting in more pods and more costs</p>

<p><strong>Yellow Line represents application memory usage</strong></p>

<p><a href="https://i.stack.imgur.com/K3vpL.png" rel="noreferrer"><img src="https://i.stack.imgur.com/K3vpL.png" alt="enter image description here"></a></p>

<p>However, on deeper investigation, this is what I found.</p>

<p>On <code>exec</code>ing inside the application container, I ran the <code>top</code> command, and it reports a total of <code>16431508 KiB</code> or roughly 16Gb of memory available, which is the memory available on the Machine.</p>

<p>There are 3 processes running inside the application container, out of which the root process (application) takes <em>5.9%</em> of memory, which roughly comes out to 0.92Gb.</p>

<p>The log-shipper simply takes 6Mb of memory.</p>

<p>Now, what I don't understand is <em>WHY</em> my pod consistently reports such high usage metrics. Am I missing something ? We're incurring significant costs due to the unintended auto-scaling and would like to fix the same.</p>

<p>In Linux unused memory considered as wasted memory, that's why all "free" RAM i. e. memory not used by application or kernel itself is actively used for caching IO operations, file system metadata, etc. but would be provided to your application if required.</p>

<p>You can get detailed information about your container memory consumption in here:</p>

<blockquote>
  <p>/sys/fs/cgroup/memory/docker/{id}/memory.stat</p>
</blockquote>

<p>If you want to scale your cluster based on memory usage it is better to count only your application size, not container memory usage.  </p>
