<p>I'm trying to use a Kubernetes (version 1.21) mounted volume on my Spark (version 3.1.3) executors for a certain job and use this as the SPARK_LOCAL_DIR for my executor to spill to.</p>
<p>Mounting it on my executors works with the following lines in my spark-submit (inspired by the <a href="https://spark.apache.org/docs/3.1.3/running-on-kubernetes.html#using-kubernetes-volumes" rel="nofollow noreferrer">Spark docs</a>):</p>
<pre><code>--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.test.options.claimName=OnDemand
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.test.options.storageClass=someClass
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.test.options.sizeLimit=100Gi
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.test.mount.path=//local-data
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.test.mount.readOnly=false
</code></pre>
<p>I can indeed verify that inside of my executors, a <code>/local-data/</code> mount exists with 100Gi of space. Furthermore, I see that in my K8s namespace the necessary PVCs were made.</p>
<p>The problem that I'm encountering is writing to that disk, even just manually. If I set the <code>readOnly</code> parameter to <code>true</code> in those 5 lines of config, I get the following:</p>
<pre><code>cd /local-data/
touch testfile
touch: cannot touch 'testfile': Read-only file system
</code></pre>
<p>This makes complete sense, as I set <code>readOnly</code> to <code>true</code>.</p>
<p>Now, if I set <code>readOnly</code> to <code>false</code>, I get the following issue:</p>
<pre><code>cd /local-data/
touch testfile
touch: cannot touch 'testfile': Permission denied
</code></pre>
<p>So it seems like I have an issue of permissions, which is easily seen with an <code>ls -al</code>:</p>
<pre><code>185@job-0a37df82fdd123d2-exec-1:/local-data$ ls -al
total 24
drwxr-xr-x 3 root root  4096 Sep  2 10:48 .
drwxr-xr-x 1 root root  4096 Sep  2 10:49 ..
drwx------ 2 root root 16384 Sep  2 10:48 lost+found
</code></pre>
<p>As you can see, my user (default Spark user, we can't run as root), does not have permissions to write to <code>/local-data</code>. This seems to be blocking me from using this volume as local storage, as defined in the <a href="https://spark.apache.org/docs/3.1.3/running-on-kubernetes.html#local-storage" rel="nofollow noreferrer">Spark docs</a>.</p>
<p>Is there a way in which I can define the permissions on this mounted volume within my spark submit options? If not, how would you go about and try to be able to use this mounted volume in your executor as local dir? I can't, however, run spark as root user in my application.</p>
<p>Thanks a lot!</p>

Koedlt
<p>You can create a <code>pod-template.yaml</code> similar to this -</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
</code></pre>
<p>and use it in the <code>spark-submit</code> command. Change the UID, GID and especially <code>fsGroup</code> accordingly.</p>
<pre><code>spark-sumbit ... \
--conf spark.kubernetes.driver.podTemplateFile=pod-template.yaml \
--conf spark.kubernetes.executor.podTemplateFile=pod-template.yaml \
...
</code></pre>
