title: High RSS and OOM kill despite low value in runtime.MemStats.Sys
body: <p>I have a process which slowly consumes more RAM until it eventually hits its cgroup limit and is OOM killed, and I'm trying to figure out why.</p>
  <p>Oddly, go's runtime seems to think not much RAM is used, whereas the OS seems to think a lot is used.</p>
  <p>Specifically, looking at runtime.MemStats (via the extvar package) I see:</p>
  <pre class="lang-json prettyprint-override"><code>&quot;Alloc&quot;:51491072,
  &quot;TotalAlloc&quot;:143474637424,
  &quot;Sys&quot;:438053112,
  &quot;Lookups&quot;:0,
  &quot;Mallocs&quot;:10230571,
  &quot;Frees&quot;:10195515,
  &quot;HeapAlloc&quot;:51491072,
  &quot;HeapSys&quot;:388464640,
  &quot;HeapIdle&quot;:333824000,
  &quot;HeapInuse&quot;:54640640,
  &quot;HeapReleased&quot;:0,
  &quot;HeapObjects&quot;:35056,
  &quot;StackInuse&quot;:14188544,
  &quot;StackSys&quot;:14188544,
  &quot;MSpanInuse&quot;:223056,
  &quot;MSpanSys&quot;:376832,
  &quot;MCacheInuse&quot;:166656,
  &quot;MCacheSys&quot;:180224,
  &quot;BuckHashSys&quot;:2111104,
  &quot;GCSys&quot;:13234176,
  &quot;OtherSys&quot;:19497592
  </code></pre>
  <p>But from the OS perspective:</p>
  <pre><code>$ ps auxwf
  USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  root       178  0.0  0.0   3996  3372 pts/0    Ss   17:33   0:00 bash
  root       246  0.0  0.0   7636  2828 pts/0    R+   17:59   0:00  \_ ps auxwf
  root         1  166  2.8 11636248 5509288 ?    Ssl  17:24  57:15 app server -api-public
  </code></pre>
  <p>So, the OSS reports an RSS of 5380 MiB, but the <code>Sys</code> field in <code>MemStats</code> shows only 417 MiB. My understanding is these fields should be approximately the same.</p>
  <p>GC is running, as confirmed by setting <code>GODEBUG=gctrace=1,madvdontneed=1</code>. For example, I see output like:</p>
  <pre><code> gc 6882 @2271.137s 0%: 0.037+2.2+0.087 ms clock, 3.5+0.78/37/26+8.4 ms cpu, 71-&gt;72-&gt;63 MB, 78 MB goal, 96 P
  </code></pre>
  <p>The numbers vary a bit depending on the process, but they are all &lt;100 MB, whereas the OS is reporting &gt;1GB (and growing, until eventual OOM).</p>
  <p><code>madvdontneed=1</code> was a shot in the dark but seems to make no difference. I wouldn't think the <code>madvise</code> parameters would be relevant, since it doesn't seem there's any need to return memory to the kernel, as the Go runtime doesn't think it's using much memory anyway.</p>
  <p>What could explain this discrepancy? Am I not correctly understanding the semantics of these fields? Are there mechanisms that would result in the growth of RSS (and an eventual OOM kill) but not increase <code>MemStats.Sys</code>?</p>