<p>I kicked off some Spark job on Kubernetes with quite big volume of data, and job failed that there is no enough space in /var/data/spark-xxx directory.</p>

<p>As the Spark documentation says on <a href="https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md" rel="nofollow noreferrer">https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md</a></p>

<blockquote>
  <p>Spark uses temporary scratch space to spill data to disk during
  shuffles and other operations. When using Kubernetes as the resource
  manager the pods will be created with an emptyDir volume mounted for
  each directory listed in SPARK_LOCAL_DIRS. If no directories are
  explicitly specified then a default directory is created and
  configured appropriately</p>
</blockquote>

<p>Seems like  <em>/var/data/spark-xx</em> directory is the default one for emptyDir. Thus, I tried to map that emptyDir to Volume (with bigger space) which is already mapped to Driver and Executors Pods.</p>

<p>I mapped it in the properties file and I can see that is mounted in the shell:</p>

<pre><code>spark.kubernetes.driver.volumes.persistentVolumeClaim.checkvolume.mount.path=/checkpoint
spark.kubernetes.driver.volumes.persistentVolumeClaim.checkvolume.mount.readOnly=false
spark.kubernetes.driver.volumes.persistentVolumeClaim.checkvolume.options.claimName=sparkstorage
spark.kubernetes.executor.volumes.persistentVolumeClaim.checkvolume.mount.path=/checkpoint
spark.kubernetes.executor.volumes.persistentVolumeClaim.checkvolume.mount.readOnly=false
spark.kubernetes.executor.volumes.persistentVolumeClaim.checkvolume.options.claimName=sparkstorage
</code></pre>

<p>I am wondering if it's possible to mount emptyDir somehow on my persistent storage, so I can spill more data and avoid job failures?</p>

<p>I found that spark 3.0 has considered this problem and has completed the feature. </p>

<blockquote>
  <p>Spark supports using volumes to spill data during shuffles and other operations. To use a volume as local storage, the volume's name should starts with <code>spark-local-dir-</code>, for example:</p>
</blockquote>

<pre><code>--conf spark.kubernetes.driver.volumes.[VolumeType].spark-local-dir-[VolumeName].mount.path=&lt;mount path&gt;
--conf spark.kubernetes.driver.volumes.[VolumeType].spark-local-dir-[VolumeName].mount.readOnly=false
</code></pre>

<p>Reference:</p>

<ul>
<li><a href="https://issues.apache.org/jira/browse/SPARK-28042" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/SPARK-28042</a></li>
<li><a href="https://github.com/apache/spark/pull/24879" rel="nofollow noreferrer">https://github.com/apache/spark/pull/24879</a></li>
</ul>
