<p>I am using DSBulk to unload data into CSV from a DSE cluster installed under Kubernetes, My cluster consists of 9 Kubernetes Pods each with 120 GB Ram.</p>
<p>I have monitored the resources while unloading the data and observed that the more the data is fetched in CSV the more the ram is getting utilised and pods are restarting due to lack of memory.</p>
<p>If one Pod is down at a time the DSBulk unload won't fail, but if 2 Pods are down unload will fail with the exception :</p>
<blockquote>
<p><strong>Cassandra timeout during read query at consistency LOCAL_ONE (1 responses were required but
only 0 replica responded).</strong></p>
</blockquote>
<p>Is there a way to avoid this exceeding of memory happening or is there a way to increase the timeout duration.</p>
<p>The command I am using is :</p>
<pre><code>dsbulk unload -maxErrors -1 -h ‘[“ &lt; My Host &gt; ”]’ -port 9042 -u &lt; My user name &gt;
-p &lt; Password &gt; -k &lt; Key Space &gt; -t &lt; My Table &gt; -url &lt; My Table &gt; 
--dsbulk.executor.continuousPaging.enabled false --datastax-java-driver.basic.request.page-size 1000 
--dsbulk.engine.maxConcurrentQueries 128 --driver.advanced.retry-policy.max-retries 100000
</code></pre>

<p>After a lot of Trial and Error, we found out the problem was with Kubernetes Cassandra pods using the main server's memory size as <strong>Max Direct Memory Size</strong>, rather than using the pods max assigned Ram.</p>
<p>The pods were assigned 120 GB of Ram, but Cassandra on each pod was assigning 185 GB Ram to <strong>file_cache_size</strong>, which made the unloading process fails as Kubernetes was rebooting each Pod that utilises Ram more than 120 GB.</p>
<p>The reason is that <strong>Max Direct Memory Size</strong> is calculated as:</p>
<pre><code>Max direct memory = ((system memory - JVM heap size))/2
</code></pre>
<p>And each pod was using 325 GB as <strong>Max Direct Memory Size</strong> and each pods <strong>file_cache_size</strong> sets automatically to be half of <strong>Max Direct Memory Size</strong> value, So whenever a pod requests for memory more than 120 GB Kubernetes will restart it.</p>
<p>The solution to it was to set <strong>Max Direct Memory Size</strong> as an env variable in Kubernetes cluster's yaml file with a default value or to override it by setting the <strong>file_cache_size</strong> value on each pod's Cassandra yaml's file</p>
