<p>I've deployed <code>metrics-server</code> in my K8s cluster (ver. 1.15)<br>
I gather this is a standard way to perform simple mem utilization checks</p>

<p>I have a POD that contains multiple processes (wrapped with <code>dumb-init</code> for process reaping purposes)</p>

<p>I want to know the exact current memory usage of my POD.</p>

<p>The output <code>kube-capacity --util --pods</code>:</p>

<pre><code>NODE      NAMESPACE           POD                               CPU REQUESTS   CPU LIMITS   CPU UTIL      MEMORY REQUESTS   MEMORY LIMITS   MEMORY UTIL
sj-k8s1   kube-system         kube-apiserver-sj-k8s1            250m (6%)      0m (0%)      77m (1%)      0Mi (0%)          0Mi (0%)        207Mi (2%)

...
sj-k8s3   salt-provisioning   salt-master-7dcf7cfb6c-l8tth      0m (0%)        0m (0%)      220m (5%)     1536Mi (19%)      3072Mi (39%)    1580Mi (20%)
</code></pre>

<p>Shows that salt-master POD uses currently ~1.6Gi and kubeapi uses ~200Mi</p>

<p>However performing on <code>sj-k8s3</code>, command <code>ps aux | awk '$12 ~ /salt-master/ {sum += $6} END {print sum}'</code> (sum of RSS from PS output):</p>

<pre><code>2051208
</code></pre>

<p>Which is ~2Gi, the output of <code>/sys/fs/cgroup/memory/memory.stats</code>:</p>

<pre><code>cache 173740032
rss 1523937280
rss_huge 0
shmem 331776
mapped_file 53248
dirty 4096
writeback 0
pgpgin 34692690
pgpgout 34278218
pgfault 212566509
pgmajfault 6
inactive_anon 331776
active_anon 1523916800
inactive_file 155201536
active_file 18206720
unevictable 0
hierarchical_memory_limit 2147483648
total_cache 173740032
total_rss 1523937280
total_rss_huge 0
total_shmem 331776
total_mapped_file 53248
total_dirty 4096
total_writeback 0
total_pgpgin 34692690
total_pgpgout 34278218
total_pgfault 212566509
total_pgmajfault 6
total_inactive_anon 331776
total_active_anon 1523916800
total_inactive_file 155201536
total_active_file 18206720
total_unevictable 0
</code></pre>

<p>This POD actually contains two docker containers, so actual sum of RSS is:</p>

<pre><code>2296688
</code></pre>

<p>which is even bigger: 2.3Gi  </p>

<p>On apiserver Node, performing just <code>ps aux</code> reveals that process RSS is: <code>447948</code>
The output of <code>/sys/fs/cgroup/memory/memory.stats</code>:</p>

<pre><code>cache 78499840
rss 391188480
rss_huge 12582912
shmem 0
mapped_file 69423104
dirty 0
writeback 0
pgpgin 111883
pgpgout 1812
pgfault 100603
pgmajfault 624
inactive_anon 0
active_anon 215531520
inactive_file 253870080
active_file 270336
unevictable 0
hierarchical_memory_limit 8361357312
total_cache 78499840
total_rss 391188480
total_rss_huge 12582912
total_shmem 0
total_mapped_file 69423104
total_dirty 0
total_writeback 0
total_pgpgin 111883
total_pgpgout 1812
total_pgfault 100603
total_pgmajfault 624
total_inactive_anon 0
total_active_anon 215531520
total_inactive_file 253870080
total_active_file 270336
total_unevictable 0

</code></pre>

<p>Could someone explain why the reported POD memory utilization differs from simple <code>ps</code> by almost 40% (for apiserver process by 100) ?</p>

<p>EDIT: I've updated the memory reported values to include output of <code>/sys/fs/cgroup/memory/memory.stat</code> which seems to +- correspond to POD utilization reported by <code>kube-capacity</code><br>
As suggested in first comment: does it mean that the difference is the shared memory only (reported by PS, but not by POD metrics/cgroup)?<br>
The difference is pretty big</p>

lakier
<p>The <code>ps</code> does not reflect the actual amount of memory used by the application but only the memory reserved for it.  It can be very misleading if pages are shared by several processes or by using some dynamically linked libraries. </p>

<p><a href="http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html" rel="nofollow noreferrer">Understanding memory usage on Linux</a> is a very good article describing how memory usage in Linux works and what ps is actually reporting. </p>

<blockquote>
  <p><strong>Why <code>ps</code> is "wrong"</strong></p>
  
  <p>Depending on how you look at it, <code>ps</code> is not reporting the real memory usage of processes. What it is really doing is showing how much real memory each process would take up <strong>if it were the only process running</strong>. Of course, a typical Linux machine has several dozen processes running at any given time, which means that the VSZ and RSS numbers reported by <code>ps</code> are almost definitely <em>wrong</em>.</p>
</blockquote>

<p>That is why <code>ps</code> should not be used for some detailed data for memory consumption.  </p>

<p>Alternative to <code>ps</code> would be <code>smem</code>. 
It reports physical memory usage, taking shared memory pages into account. Then unshared memory is reported at the <code>USS</code> (Unique Set Size). So you can use <code>USS</code> when you want to ignore shared memory. </p>

<p>The unshared memory (<code>USS</code>) plus process's proportion of shared memory is reported at the <code>PSS</code> (Proportionial Set Size). Basically it add <code>USS</code> along with a proportion of its shared memory divided by the number of processes sharing that memory. </p>

<p>On the other hand <code>RSS</code>(Resident Set Size) is the amount of shared memory plus unshared memory used by each process. If any processes share memory, this will short report that over the amount of memory that is actually used. </p>

<p>Linux uses a resource management technique used in programming to efficiently implement a <code>duplicate</code> or <code>copy</code> operation. This is called <code>copy-on-write</code>.  So when you have parent and child process, they both will show the same RSS. With <code>copy-on-write</code> linux ensures that both processes are really using the same memory. </p>
