<p>I'm running Ignite in a Kubernetes cluster with persistence enabled. Each machine has a Java Heap of 24GB with 20GB devoted to durable memory with a memory limit of 110GB. My relevant JVM options are <code>-XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC</code>. After running DataStreamers on every node for several hours, nodes on my cluster hit their k8s memory limit triggering an OOM kill. After running Java NMT, I was surprised to find a huge amount of space allocated to internal memory.</p>

<pre><code>Java Heap (reserved=25165824KB, committed=25165824KB)
(mmap: reserved=25165824KB, committed=25165824KB)  

Internal (reserved=42425986KB, committed=42425986KB)
(malloc=42425954KB #614365) 
(mmap: reserved=32KB, committed=32KB) 
</code></pre>

<p>Kubernetes metrics confirmed this:</p>

<p><a href="https://i.stack.imgur.com/RUYIe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RUYIe.png" alt="enter image description here"></a></p>

<p>"Ignite Cache" is kernel page cache. The last panel "Heap + Durable + Buffer" is the sum of the ignite metrics <code>HeapMemoryUsed</code> + <code>PhysicalMemorySize</code> + <code>CheckpointBufferSize</code>.</p>

<p>I knew this couldn't be a result of data build-up because the DataStreamers are flushed after each file they read (up to about 250MB max), and no node is reading more than 4 files at once. After ruling out other issues on my end, I tried setting <code>-XX:MaxDirectMemorySize=10G</code>, and invoking manual GC, but nothing seems to have an impact other than periodically shutting down all of my pods and restarting them.</p>

<p>I'm not sure where to go from here. Is there a workaround in Ignite that doesn't force me to use a third-party database?</p>

<p>EDIT: My DataStorageConfiguration</p>

<pre><code>    &lt;property name="dataStorageConfiguration"&gt;
        &lt;bean class="org.apache.ignite.configuration.DataStorageConfiguration"&gt;
            &lt;property name="metricsEnabled" value="true"/&gt;
            &lt;property name="checkpointFrequency" value="300000"/&gt;
            &lt;property name="storagePath" value="/var/lib/ignite/data/db"/&gt;
            &lt;property name="walFlushFrequency" value="10000"/&gt;
            &lt;property name="walMode" value="LOG_ONLY"/&gt;
            &lt;property name="walPath" value="/var/lib/ignite/data/wal"/&gt;
            &lt;property name="walArchivePath" value="/var/lib/ignite/data/wal/archive"/&gt;               
            &lt;property name="walSegmentSize" value="2147483647"/&gt;
            &lt;property name="maxWalArchiveSize" value="4294967294"/&gt;
            &lt;property name="walCompactionEnabled" value="false"/&gt;
            &lt;property name="writeThrottlingEnabled" value="False"/&gt;
            &lt;property name="pageSize" value="4096"/&gt;                
            &lt;property name="defaultDataRegionConfiguration"&gt;
                &lt;bean class="org.apache.ignite.configuration.DataRegionConfiguration"&gt;
                    &lt;property name="persistenceEnabled" value="true"/&gt;
                    &lt;property name="checkpointPageBufferSize" value="2147483648"/&gt;
                    &lt;property name="name" value="Default_Region"/&gt;
                    &lt;property name="maxSize" value="21474836480"/&gt;
                    &lt;property name="metricsEnabled" value="true"/&gt;
                &lt;/bean&gt;
            &lt;/property&gt;
        &lt;/bean&gt;
    &lt;/property&gt; 
</code></pre>

<p>UPDATE: When I disable persistence, internal memory is properly disposed of:</p>

<p><a href="https://i.stack.imgur.com/vDFxS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vDFxS.png" alt="enter image description here"></a></p>

<p>UPDATE: The issue is demonstrated <a href="https://github.com/kellanburket/ignite-leak" rel="nofollow noreferrer">here</a> with a reproducible example. It's runnable on a machine with at least 22GB of memory for docker and about 50GB of storage. Interestingly the leak is only really noticeable when passing in a Byte Array or String as the value.   </p>

<p>The memory leaks seems to be triggered by the <code>@QueryTextField</code> annotation on value object in my cache model, which supports Lucene queries in Ignite.</p>

<p>Originally: <code>case class Value(@(QueryTextField@field) theta: String)</code></p>

<p>Changing this line to: <code>case class Value(theta: String)</code> seems to solve the problem. I don't have an explanation as to why this works, but maybe somebody with a good understanding of the Ignite code base can explain why.</p>
