<p>While running a spark job with a Kubernetes cluster, we get the following error:</p>

<pre><code>2018-11-30 14:00:47 INFO  DAGScheduler:54 - Resubmitted ShuffleMapTask(1, 58), so marking it as still running.
2018-11-30 14:00:47 WARN  TaskSetManager:66 - Lost task 310.0 in stage 1.0 (TID 311, 10.233.71.29, executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: 
The executor with id 3 exited with exit code -1.
The API gave the following brief reason: Evicted
The API gave the following message: The node was low on resource: ephemeral-storage. Container executor was using 515228Ki, which exceeds its request of 0. 
The API gave the following container statuses:
</code></pre>

<p>How to configure the job so we can increase the ephemeral storage size of each container ?</p>

<p>We use spark 2.4.0 and Kubernetes 1.12.1</p>

<p>The spark submit option is as follow</p>

<pre><code>--conf spark.local.dir=/mnt/tmp \
--conf spark.executor.instances=4 \
--conf spark.executor.cores=8 \
--conf spark.executor.memory=100g \
--conf spark.driver.memory=4g \
--conf spark.driver.cores=1 \
--conf spark.kubernetes.memoryOverheadFactor=0.1 \
--conf spark.kubernetes.container.image=spark:2.4.0 \
--conf spark.kubernetes.namespace=visionlab \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.container.image.pullPolicy=Always \
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.myvolume.options.claimName=pvc \
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.myvolume.mount.path=/mnt/ \
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.myvolume.mount.readOnly=false \
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.myvolume.options.claimName=pvc \
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.myvolume.mount.path=/mnt/ \
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.myvolume.mount.readOnly=false
</code></pre>

<p>This is because of your spark job do not have enough storage to store temporary cache data</p>
<p>First you need to inspect your pod deployment, to identify the volume name that will be used for mounting such as <code>spark-local-dir-1</code>
Then dive into your spark executor pod, print out <code>$SPARK_LOCAL_DIR</code> to identify location of current temporary spark data location, modify the path with <code>&quot;spark.local.dir&quot;: &quot;/tmp/spark-temp/&quot;</code> (to change the <code>$SPARK_LOCAL_DIR</code>) in your config, and finally mount your directory with existing pvc claim (notice give large storage e.g. <code>100GB</code>). The below code is spark config when submit job on kubernetes.</p>
<pre><code>&quot;spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName&quot;: &quot;spark-temp-local&quot;,
&quot;spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path&quot;: &quot;/tmp/spark-temp&quot;,
&quot;spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly&quot;: &quot;false&quot;,

&quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName&quot;: &quot;spark-temp-local&quot;,
&quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path&quot;: &quot;/tmp/spark-temp&quot;,
&quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly&quot;: &quot;false&quot;,
&quot;spark.local.dir&quot;: &quot;/tmp/spark-temp/&quot;
</code></pre>
