<p>I am struggling to understand the Spark documentation in order to set up the local-dir correctly.</p>
<h2>Setup:</h2>
<p>I am running Spark 3.1.2 on Kubernetes via the Sparkoperator approach. The Number of executor Pods varying on job size and available resources on the cluster. A typical case is that i start the job with 20 requested executors but 3 Pods remain in pending state and spark complete the job with 17 executors.</p>
<h2>Base Problem:</h2>
<p>I am running in the error &quot;The node was low on resource: ephemeral-storage.&quot; due to much spilling of data into the default local-dir created via <code>empty-dir</code>on the kubernetes nodes.</p>
<p>This is a known issue and it should be solved by pointing the <code>local-dir</code> to a mounted presistent volume.</p>
<p>I tried to approaches but both are not working:</p>
<h2>Approach 1:</h2>
<p>Following the documentation <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#local-storage" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/running-on-kubernetes.html#local-storage</a> I added the following options into the spark-config</p>
<pre><code>&quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName&quot;: &quot;tmp-spark-spill&quot;
&quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass&quot;: &quot;csi-rbd-sc&quot;
&quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit&quot;: &quot;3000Gi&quot;
&quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path&quot;: =&quot;/spill-data&quot;
&quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly&quot;: &quot;false&quot;
</code></pre>
<p>the full yaml looks like</p>
<pre><code>apiVersion: &quot;sparkoperator.k8s.io/v1beta2&quot;
kind: SparkApplication
metadata:
  name: job1
  namespace: spark
spec:
  serviceAccount: spark
  type: Python
  pythonVersion: &quot;3&quot;
  mode: cluster
  image: &quot;xxx/spark-py:app-3.1.2&quot;
  imagePullPolicy: Always
  mainApplicationFile: local:///opt/spark/work-dir/nfs/06_dwh_core/jobs/job1/main.py
  sparkVersion: &quot;3.0.0&quot;
  restartPolicy:
    type: OnFailure
    onFailureRetries: 0
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 0
    onSubmissionFailureRetryInterval: 20
  sparkConf:
    &quot;spark.default.parallelism&quot;: &quot;400&quot;
    &quot;spark.sql.shuffle.partitions&quot;: &quot;400&quot;
    &quot;spark.serializer&quot;: &quot;org.apache.spark.serializer.KryoSerializer&quot;
    &quot;spark.sql.debug.maxToStringFields&quot;: &quot;1000&quot;
    &quot;spark.ui.port&quot;: &quot;4045&quot;
    &quot;spark.driver.maxResultSize&quot;: &quot;0&quot; 
    &quot;spark.kryoserializer.buffer.max&quot;: &quot;512&quot;
    &quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName&quot;: &quot;tmp-spark-spill&quot;
    &quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass&quot;: &quot;csi-rbd-sc&quot;
    &quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit&quot;: &quot;3000Gi&quot;
    &quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path&quot;: =&quot;/spill-data&quot;
    &quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly&quot;: &quot;false&quot;
  driver:
    cores: 1
    memory: &quot;20G&quot;
    labels:
      version: 3.1.2
    serviceAccount: spark
    volumeMounts:
      - name: nfs
        mountPath: /opt/spark/work-dir/nfs
  executor:
    cores: 20
    instances: 20
    memory: &quot;150G&quot;
    labels:
      version: 3.0.0
    volumeMounts:
      - name: nfs
        mountPath: /opt/spark/work-dir/nfs
  volumes:
    - name: nfs
      nfs:
        server: xxx
        path: /xxx
        readOnly: false
</code></pre>
<h2>Issue 1:</h2>
<p>this results in an error saying that the pvc already exists and it only creates effectively one executor.</p>
<pre><code>io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default.svc/api/v1/namespaces/spark-poc/persistentvolumeclaims. Message: persistentvolumeclaims &quot;tmp-spark-spill&quot; already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=persistentvolumeclaims, name=tmp-spark-spill, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=persistentvolumeclaims &quot;tmp-spark-spill&quot; already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
</code></pre>
<p>Do I have to define this local-dir claims for every executor? kind of</p>
<pre><code> &quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName&quot;: &quot;tmp-spark-spill&quot;
.
.
.
 &quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-2.options.claimName&quot;: &quot;tmp-spark-spill&quot;
.
.
.
 &quot;spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-3.options.claimName&quot;: &quot;tmp-spark-spill&quot;
.
.
.
</code></pre>
<p>But how can I make it dynamicaly if I have changing numbers of executors? Is it not automatically picking it up from the executor config?</p>
<h2>Approach 2:</h2>
<p>I created an pvc myself mounted it as volume and set the local-dir as spark config parameter</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-spark-spill
  namespace: spark-poc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 3000Gi
  storageClassName: csi-rbd-sc
  volumeMode: Filesystem
</code></pre>
<p>mounted into to executors like</p>
<pre><code>apiVersion: &quot;sparkoperator.k8s.io/v1beta2&quot;
kind: SparkApplication
metadata:
  name: job1
  namespace: spark
spec:
  serviceAccount: spark
  type: Python
  pythonVersion: &quot;3&quot;
  mode: cluster
  image: &quot;xxx/spark-py:app-3.1.2&quot;
  imagePullPolicy: Always
  mainApplicationFile: local:///opt/spark/work-dir/nfs/06_dwh_core/jobs/job1/main.py
  sparkVersion: &quot;3.0.0&quot;
  restartPolicy:
    type: OnFailure
    onFailureRetries: 0
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 0
    onSubmissionFailureRetryInterval: 20
  sparkConf:
    &quot;spark.default.parallelism&quot;: &quot;400&quot;
    &quot;spark.sql.shuffle.partitions&quot;: &quot;400&quot;
    &quot;spark.serializer&quot;: &quot;org.apache.spark.serializer.KryoSerializer&quot;
    &quot;spark.sql.debug.maxToStringFields&quot;: &quot;1000&quot;
    &quot;spark.ui.port&quot;: &quot;4045&quot;
    &quot;spark.driver.maxResultSize&quot;: &quot;0&quot; 
    &quot;spark.kryoserializer.buffer.max&quot;: &quot;512&quot;
    &quot;spark.local.dir&quot;: &quot;/spill&quot;
  driver:
    cores: 1
    memory: &quot;20G&quot;
    labels:
      version: 3.1.2
    serviceAccount: spark
    volumeMounts:
      - name: nfs
        mountPath: /opt/spark/work-dir/nfs
  executor:
    cores: 20
    instances: 20
    memory: &quot;150G&quot;
    labels:
      version: 3.0.0
    volumeMounts:
      - name: nfs
        mountPath: /opt/spark/work-dir/nfs
      - name: pvc-spark-spill
        mountPath: /spill
  volumes:
    - name: nfs
      nfs:
        server: xxx
        path: /xxx
        readOnly: false
    - name: pvc-spark-spill
      persistentVolumeClaim:
        claimName: pvc-spark-spill
</code></pre>
<h2>Issue 2</h2>
<p>This approach fails with the message that the <code>/spill</code> must be unique.</p>
<pre><code> Message: Pod &quot;job1-driver&quot; is invalid: spec.containers[0].volumeMounts[7].mountPath: Invalid value: &quot;/spill&quot;: must be unique.
</code></pre>
<h2>Summary and Questions</h2>
<p>It seems that every executor needs his own pvc or at least his own folder on the pvc to spill his data. But how do I configure it correctly? I am not getting it from the documentation</p>
<p>Thanks for your help
Alex</p>

Alex Ortner
<p>spark should be able to create PVC dynamically by setting up claimName= OnDemand.
Attaching multiple pods for the same pvc will be issue on Kubernetes end</p>
<p>attaching screenshot  for documentation
<a href="https://i.stack.imgur.com/jmIPT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jmIPT.png" alt="config screenshot" /></a></p>
<p>You can look into nfs share which will work outside kubenetes manage volumes.
Example
<a href="https://www.datamechanics.co/blog-post/apache-spark-3-1-release-spark-on-kubernetes-is-now-ga" rel="nofollow noreferrer">https://www.datamechanics.co/blog-post/apache-spark-3-1-release-spark-on-kubernetes-is-now-ga</a></p>
