<p>"Kubernetes" (v1.10.2) says that my pod (which contains one container) is using about 5GB memory. Inside the container, RSS is saying more like 681MiB. Can anypony explain how to get from 681MiB to 5GB with the following data (or describe how to make up the difference with another command I've omitted, either from the container or from the docker host that is running this container in kubernetes)?</p>

<p>kubectl top pods says 5GB:</p>

<pre><code>% kubectl top pods -l app=myapp
NAME                             CPU(cores)   MEMORY(bytes)
myapp-56b947bf6d-2lcr7           39m          5039Mi
</code></pre>

<p>Cadvisor reports a similar number (might have been from a slightly different time, so please ignore small differences):</p>

<pre><code>container_memory_usage_bytes{pod_name=~".*myapp.*"}      5309456384

5309456384 / 1024.0 / 1024 ~= 5063 ~= 5039
</code></pre>

<p>Inside the container, this file appears to be where cadvisor is getting its data:</p>

<pre><code>% kubectl exec -it myapp-56b947bf6d-2lcr7 bash
meme@myapp-56b947bf6d-2lcr7:/app# cat /sys/fs/cgroup/memory/memory.usage_in_bytes
5309456384
</code></pre>

<p>The resident set size (RSS) inside the container does NOT match up (less than 1GB):</p>

<pre><code>meme@myapp-56b947bf6d-2lcr7:/app# kb=$(ps aux | grep -v grep | grep -v 'ps aux' | grep -v bash | grep -v awk | grep -v RSS | awk '{print $6}' | awk '{s+=$1} END {printf "%.0f", s}'); mb=$(expr $kb / 1024); printf "Kb: $kb\nMb: $mb\n"
Kb: 698076
Mb: 681
</code></pre>

<p>Full ps aux in case that is helpful:</p>

<pre><code>meme@myapp-56b947bf6d-2lcr7:/app# ps aux | grep -v grep | grep -v 'ps aux' | grep -v bash | grep -v awk
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
meme         1  0.0  0.0 151840 10984 ?        Ss   Jun04   0:29 /usr/sbin/apache2 -D FOREGROUND
www-data    10  0.0  0.0 147340  4652 ?        S    Jun04   0:00 /usr/sbin/apache2 -D FOREGROUND
www-data    11  0.0  0.0 148556  4392 ?        S    Jun04   0:16 /usr/sbin/apache2 -D FOREGROUND
www-data    12  0.2  0.0 2080632 11348 ?       Sl   Jun04  31:58 /usr/sbin/apache2 -D FOREGROUND
www-data    13  0.1  0.0 2080384 10980 ?       Sl   Jun04  18:12 /usr/sbin/apache2 -D FOREGROUND
www-data    68  0.3  0.0 349048 94272 ?        Sl   Jun04  47:09 hotapp
www-data   176  0.2  0.0 349624 92888 ?        Sl   Jun04  43:11 hotapp
www-data   179  0.2  0.0 349196 94456 ?        Sl   Jun04  42:20 hotapp
www-data   180  0.3  0.0 349828 95112 ?        Sl   Jun04  44:14 hotapp
www-data   185  0.3  0.0 346644 91948 ?        Sl   Jun04  43:49 hotapp
www-data   186  0.3  0.0 346208 91568 ?        Sl   Jun04  44:27 hotapp
www-data   189  0.2  0.0 350208 95476 ?        Sl   Jun04  41:47 hotapp
</code></pre>

<p>Memory section from docker's container stats API:</p>

<pre><code>curl --unix-socket /var/run/docker.sock 'http:/v1.24/containers/a45fc651e7b12f527b677e6a46e2902786bee6620484922016a135e317a42b4e/stats?stream=false' | jq . # yields:

"memory_stats": {
  "usage": 5327712256,
  "max_usage": 5368344576,
  "stats": {
    "active_anon": 609095680,
    "active_file": 74457088,
    "cache": 109944832,
    "dirty": 28672,
    "hierarchical_memory_limit": 5368709120,
    "inactive_anon": 1687552,
    "inactive_file": 29974528,
    "mapped_file": 1675264,
    "pgfault": 295316278,
    "pgmajfault": 77,
    "pgpgin": 85138921,
    "pgpgout": 84964308,
    "rss": 605270016,
    "rss_huge": 0,
    "shmem": 5513216,
    "total_active_anon": 609095680,
    "total_active_file": 74457088,
    "total_cache": 109944832,
    "total_dirty": 28672,
    "total_inactive_anon": 1687552,
    "total_inactive_file": 29974528,
    "total_mapped_file": 1675264,
    "total_pgfault": 295316278,
    "total_pgmajfault": 77,
    "total_pgpgin": 85138921,
    "total_pgpgout": 84964308,
    "total_rss": 605270016,
    "total_rss_huge": 0,
    "total_shmem": 5513216,
    "total_unevictable": 0,
    "total_writeback": 0,
    "unevictable": 0,
    "writeback": 0
  },
  "limit": 5368709120
},
</code></pre>

<p>A comment on <a href="https://github.com/google/cadvisor/issues/638" rel="noreferrer">https://github.com/google/cadvisor/issues/638</a> asserts:</p>

<blockquote>
  <p>Total (memory.usage_in_bytes) = rss + cache</p>
</blockquote>

<p><a href="https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt" rel="noreferrer">https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt</a> says:</p>

<blockquote>
  <p>usage_in_bytes: For efficiency, as other kernel components, memory cgroup uses some optimization
  to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
  method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz
  value for efficient access. (Of course, when necessary, it's synchronized.)
  If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
  value in memory.stat(see 5.2).</p>
</blockquote>

<p><a href="https://docs.docker.com/engine/reference/commandline/stats/#parent-command" rel="noreferrer">https://docs.docker.com/engine/reference/commandline/stats/#parent-command</a> says:</p>

<blockquote>
  <p>Note: On Linux, the Docker CLI reports memory usage by subtracting page cache usage from the total memory usage. The API does not perform such a calculation but rather provides the total memory usage and the amount from the page cache so that clients can use the data as needed.</p>
</blockquote>

<p>And indeed, most of the stuff in /sys/fs/cgroup/memory/memory.stat in the container shows up in the above docker stats api response (slight differences are from taking the samples at a different time, sorry):</p>

<pre><code>meme@myapp-56b947bf6d-2lcr7:/app# cat /sys/fs/cgroup/memory/memory.stat
cache 119492608
rss 607436800
rss_huge 0
shmem 5525504
mapped_file 1675264
dirty 69632
writeback 0
pgpgin 85573974
pgpgout 85396501
pgfault 296366011
pgmajfault 80
inactive_anon 1687552
active_anon 611213312
inactive_file 32800768
active_file 81166336
unevictable 0
hierarchical_memory_limit 5368709120
total_cache 119492608
total_rss 607436800
total_rss_huge 0
total_shmem 5525504
total_mapped_file 1675264
total_dirty 69632
total_writeback 0
total_pgpgin 85573974
total_pgpgout 85396501
total_pgfault 296366011
total_pgmajfault 80
total_inactive_anon 1687552
total_active_anon 611213312
total_inactive_file 32800768
total_active_file 81166336
total_unevictable 0
</code></pre>

<p>Memory info from <code>kubectl describe pod &lt;pod&gt;</code>:</p>

<pre><code>Limits:
  memory:  5Gi
Requests:
  memory:   4Gi
</code></pre>

<p>Here's what <code>pmap</code> says inside the container. In this one-liner, I get all process ids, run pmap -x on them, and pull the Kbytes column from the pmap results. The total result is 256 Megabytes (much less than ps's RSS, partially, I think, because many of the processes return no output from pmap -x):</p>

<pre><code>ps aux | awk '{print $2}' | grep -v PID | xargs sudo pmap -x | grep total | grep -v grep | awk '{print $3}' | awk '{s+=$1} END {printf "%.0f", s}'; echo
256820
</code></pre>

<p><a href="https://raw.githubusercontent.com/pixelb/ps_mem/master/ps_mem.py" rel="noreferrer">ps_mem.py</a> is mentioned at <a href="https://stackoverflow.com/a/133444/6090676">https://stackoverflow.com/a/133444/6090676</a>. It inspects <code>/proc/$pid/statm</code> and <code>/proc/$pid/smaps</code>. No illumination here (again, it seems to be ignoring some processes):</p>

<pre><code># python ps_mem.py
Private  +   Shared  =  RAM used    Program

  1.7 MiB +   1.0 MiB =   2.7 MiB   apache2
  2.0 MiB +   1.0 MiB =   3.0 MiB   bash (3)
---------------------------------
                          5.7 MiB
=================================
</code></pre>

<p>There is another question similar to this (but with less information) at <a href="https://stackoverflow.com/q/46677536/6090676">Incorrect reporting of container memory usage by cadvisor</a>. Thanks!</p>

<p>I don't know if you already find your answer or not but let me give you some information that may help.</p>
<ul>
<li><p>cAdvisor extract many memory-related metrics. We will focus on:</p>
<ol>
<li><p><code>container_memory_usage_bytes</code> = value in /sys/fs/cgroup/memory/memory.usage_in_bytes file. (Usage of the memory)</p>
</li>
<li><p><code>container_memory_working_set_bytes</code> = <code>container_memory_usage_bytes</code> - <code>total_inactive_file</code> (from /sys/fs/cgroup/memory/memory.stat), this is calculated in cAdvisor and is &lt;= <code>container_memory_usage_bytes</code></p>
</li>
<li><p><code>container_memory_rss</code> = <code>total_rss</code> value from /sys/fs/cgroup/memory/memory.stat</p>
</li>
</ol>
</li>
<li><p>Now you know how those metrics are gathered, you need to know that when you use the <code>kubectl top pods</code> command, you get the value of <code>container_memory_working_set_bytes</code> not <code>container_memory_usage_bytes</code> metric.</p>
<p>so from your values:</p>
<p>5039Mi &quot;working set fro kubectl command&quot; ~= 5064 &quot;from memory.usage file&quot; - 28 &quot;total_inactive_file from Memory section from docker's container stats API&quot;</p>
</li>
<li><p>It is also worth to mention that when the value of <code>container_memory_usage_bytes</code> reaches to the limits, your pod will NOT get oom-killed. BUT if <code>container_memory_working_set_bytes</code> or <code>container_memory_rss</code> reached to the limits, the pod will be killed.</p>
</li>
</ul>
