<p>I've been looking all over StackOverflow for this, but can't find a satisfactory answer.</p>

<p>When running <code>kubectl top nodes &lt;node name&gt;</code> I get a memory utilisation of approx. 69% (Kubernetes showing roughly 21Gi of 32Gi being used). But if I go into the system itself and run the <code>free</code> command, as well as the <code>top</code> command, I see a total of 6GB of used memory (i.e. 20% - this is the information under the <code>used</code> column in the output of <code>free</code>) - way less than 69% of the total system memory of 32GB.</p>

<p>Even accounting for the differences in Gi and GB, there's still more than 40% difference unaccounted for. I know that Kubernetes uses the stats reported by <code>/sys/fs/cgroup/memory/memory.usage_in_bytes</code> to report on memory utilisation, but why would this be different than the utilisation reported by other processes on the system (especially sometimes higher)? Which one should I take as the source of truth?</p>

<p>Found answer to my question here: <a href="https://serverfault.com/questions/902009/the-memory-usage-reported-in-cgroup-differs-from-the-free-command">https://serverfault.com/questions/902009/the-memory-usage-reported-in-cgroup-differs-from-the-free-command</a>. In summary, it seems that Kubernetes uses the cgroup memory utilisation, which is reported in /sys/fs/cgroup/memory/memory.usage_in_bytes. The cgroup memory utilisation calculates not only the currently used memory in RAM, but also the "cached" memory (i.e. any memory no longer required by apps that are subsequently free to be reclaimed by the OS, but hasn't been reclaimed yet). The Linux system commands see "cached" memory as "free" but Kubernetes does not (not sure why).</p>
