<p>We are running a Spark Streaming application on a Kubernetes cluster using spark 2.4.5.
The application is receiving massive amounts of data through a Kafka topic (one message each 3ms). 4 executors and 4 kafka partitions are being used.</p>
<p>While running, the memory of the driver pod keeps increasing until it is getting killed by K8s with an 'OOMKilled' status. The memory of executors is not facing any issues.</p>
<p>When checking the driver pod resources using this command :</p>
<pre><code>kubectl top pod podName
</code></pre>
<p>We can see that the memory increases until it reaches 1.4GB, and the pod is getting killed.</p>
<p>However, when checking the storage memory of the driver on Spark UI, we can see that the storage memory is not fully used (50.3 KB / 434 MB). Is there any difference between the <strong>storage memory of the driver</strong>, and the <strong>memory of the pod containing the driver</strong> ?</p>
<p>Has anyone had experience with a similar issue before?</p>
<p>Any help would be appreciated.</p>
<p>Here are few more details about the app :</p>
<ul>
<li>Kubernetes version : 1.18</li>
<li>Spark version : 2.4.5</li>
<li>Batch interval of spark streaming context : 5 sec</li>
<li>Rate of input data : 1 kafka message each 3 ms</li>
<li>Scala language</li>
</ul>

Nab
<p>In brief, the Spark memory consists of three parts:</p>
<ul>
<li>Reversed memory (300MB)</li>
<li>User memory ((all - 300MB)*0.4), used for data processing logic.</li>
<li>Spark memory ((all-300MB)*0.6(<code>spark.memory.fraction</code>)), used for cache and shuffle in Spark.</li>
</ul>
<p>Besides this, there is also <code>max(executor memory * 0.1, 384MB)</code>(<code>0.1</code> is <code>spark.kubernetes.memoryOverheadFactor</code>)  extra memory used by non-JVM memory in K8s.</p>
<p>Adding executor memory limit by memory overhead in K8S should fix the OOM.</p>
<p>You can also decrease <code>spark.memory.fraction</code> to allocate more RAM to user memory.</p>
