<p>I'm running Ignite in a Kubernetes cluster with persistence enabled. Each machine has a Java Heap of 24GB with 20GB devoted to durable memory with a memory limit of 110GB. My relevant JVM options are <code>-XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC</code>. After running DataStreamers on every node for several hours, nodes on my cluster hit their k8s memory limit triggering an OOM kill. After running Java NMT, I was surprised to find a huge amount of space allocated to internal memory.</p>

<pre><code>Java Heap (reserved=25165824KB, committed=25165824KB)
(mmap: reserved=25165824KB, committed=25165824KB)  

Internal (reserved=42425986KB, committed=42425986KB)
(malloc=42425954KB #614365) 
(mmap: reserved=32KB, committed=32KB) 
</code></pre>

<p>Kubernetes metrics confirmed this:</p>

<p><a href="https://i.stack.imgur.com/RUYIe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RUYIe.png" alt="enter image description here"></a></p>

<p>"Ignite Cache" is kernel page cache. The last panel "Heap + Durable + Buffer" is the sum of the ignite metrics <code>HeapMemoryUsed</code> + <code>PhysicalMemorySize</code> + <code>CheckpointBufferSize</code>.</p>

<p>I knew this couldn't be a result of data build-up because the DataStreamers are flushed after each file they read (up to about 250MB max), and no node is reading more than 4 files at once. After ruling out other issues on my end, I tried setting <code>-XX:MaxDirectMemorySize=10G</code>, and invoking manual GC, but nothing seems to have an impact other than periodically shutting down all of my pods and restarting them.</p>

<p>I'm not sure where to go from here. Is there a workaround in Ignite that doesn't force me to use a third-party database?</p>

<p>EDIT: My DataStorageConfiguration</p>

<pre><code>    &lt;property name="dataStorageConfiguration"&gt;
        &lt;bean class="org.apache.ignite.configuration.DataStorageConfiguration"&gt;
            &lt;property name="metricsEnabled" value="true"/&gt;
            &lt;property name="checkpointFrequency" value="300000"/&gt;
            &lt;property name="storagePath" value="/var/lib/ignite/data/db"/&gt;
            &lt;property name="walFlushFrequency" value="10000"/&gt;
            &lt;property name="walMode" value="LOG_ONLY"/&gt;
            &lt;property name="walPath" value="/var/lib/ignite/data/wal"/&gt;
            &lt;property name="walArchivePath" value="/var/lib/ignite/data/wal/archive"/&gt;               
            &lt;property name="walSegmentSize" value="2147483647"/&gt;
            &lt;property name="maxWalArchiveSize" value="4294967294"/&gt;
            &lt;property name="walCompactionEnabled" value="false"/&gt;
            &lt;property name="writeThrottlingEnabled" value="False"/&gt;
            &lt;property name="pageSize" value="4096"/&gt;                
            &lt;property name="defaultDataRegionConfiguration"&gt;
                &lt;bean class="org.apache.ignite.configuration.DataRegionConfiguration"&gt;
                    &lt;property name="persistenceEnabled" value="true"/&gt;
                    &lt;property name="checkpointPageBufferSize" value="2147483648"/&gt;
                    &lt;property name="name" value="Default_Region"/&gt;
                    &lt;property name="maxSize" value="21474836480"/&gt;
                    &lt;property name="metricsEnabled" value="true"/&gt;
                &lt;/bean&gt;
            &lt;/property&gt;
        &lt;/bean&gt;
    &lt;/property&gt; 
</code></pre>

<p>UPDATE: When I disable persistence, internal memory is properly disposed of:</p>

<p><a href="https://i.stack.imgur.com/vDFxS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vDFxS.png" alt="enter image description here"></a></p>

<p>UPDATE: The issue is demonstrated <a href="https://github.com/kellanburket/ignite-leak" rel="nofollow noreferrer">here</a> with a reproducible example. It's runnable on a machine with at least 22GB of memory for docker and about 50GB of storage. Interestingly the leak is only really noticeable when passing in a Byte Array or String as the value.   </p>

<h1>TLDR</h1>

<p>Set <code>walSegmentSize=64mb</code> (or just remove the setting and use the default) AND set <code>-XX:MaxDirectMemorySize=&lt;walSegmentSize * 4&gt;</code>.</p>

<h1>Explanation</h1>

<p>One thing people often forget when calculating Ignite's memory needs is direct memory buffer size.</p>

<p>Direct memory buffers are JVM-managed buffers allocated from a separate space in the Java process - it is neither Java heap, Ignite data region or Ignite checkpoint buffer.</p>

<p>Direct memory buffers are the normal way of interacting with non-heap memory in Java. There is a lot of things that use that (from JVM's internal code to applications) but in Ignite servers the main user of the direct memory pool is write-ahead log.</p>

<p>By default, Ignite writes to WAL using a memory-mapped file - which works through a direct memory buffer. The size of that buffer is the size of the WAL segment. And here we get to the fun stuff.</p>

<p>Your WAL segments are huge! 2GB - it's A LOT. Default is 64mb, and I've rarely seen an environment that would use more than that. In some specific workloads and for some specific disks we would recommend to set 256mb.</p>

<p>So, you have a 2GB buffers that are being created in the direct memory pool. The maximum size of the direct memory by default is equal to <code>-Xmx</code> - in your case, 24GB. I can see a scenario when your direct memory pool would bloat to 24GB (from the non-yet-cleared old buffered), making the total size of your application at least <code>20 + 2 + 24 + 24 = 70GB</code>!.</p>

<p>This explains the 40GB of internal JVM memory (I think that's the data region + direct). This also explains why you don't see an issue when persistence is off - you don't have WAL in that case.</p>

<h1>What to do</h1>

<ol>
<li><p>Choose a sane <code>walSegmentSize</code>. I don't know the reason behind the 2GB choice but I would recommend to go either for the default of 64mb or for 256mb if you're sure you had issues with small WAL segments.</p></li>
<li><p>Set a limit to JVM's direct memory pool via <code>-XX:MaxDirectMemorySize=&lt;size&gt;</code>. I find it a safe choice to set it to the value of <code>walSegmentSize * 4</code>, i.e. somewhere in the range 256mb-1gb.</p></li>
</ol>

<p>Even if you see issues with memory consumption after making the above changes - keep them anyway, just because they are the best choice in for 99% of clusters.</p>
