<p>I'm working on a POC for getting a Spark cluster set up to use Kubernetes for resource management using AKS (Azure Kubernetes Service). I'm using spark-submit to submit pyspark applications to k8s in cluster mode and I've been successful in getting applications to run fine.</p>
<p>I got Azure file share set up to store application scripts and Persistent Volume and a Persistent Volume Claim pointing to this file share to allow Spark to access the scripts from Kubernetes. This works fine for applications that don't write any output, like the pi.py example given in the spark source code, but writing any kind of outputs fails in this setup. I tried running a script to get wordcounts and the line</p>
<pre><code>wordCounts.saveAsTextFile(f&quot;./output/counts&quot;)
</code></pre>
<p>causes an exception where wordCounts is an rdd.</p>
<pre><code>Traceback (most recent call last):
File &quot;/opt/spark/work-dir/wordcount2.py&quot;, line 14, in &lt;module&gt;
wordCounts.saveAsTextFile(f&quot;./output/counts&quot;)
File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/rdd.py&quot;, line 1570, in saveAsTextFile
File &quot;/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py&quot;, line 1257, in __call__
File &quot;/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py&quot;, line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o65.saveAsTextFile.
: ExitCodeException exitCode=1: chmod: changing permissions of '/opt/spark/work-dir/output/counts': Operation not permitted
</code></pre>
<p>The directory &quot;counts&quot; has been created from the spark application just fine, so it seems like it has required permissions, but this subsequent <code>chmod</code> that spark tries to perform internally fails. I haven't been able to figure out the cause and what exact configuration I'm missing in my commands that's causing this. Any help would be greatly appreciated.</p>
<p>The <code>kubectl</code> version I'm using is</p>
<pre><code>Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;22&quot;, GitVersion:&quot;v1.22.1&quot;, GitCommit:&quot;632ed300f2c34f6d6d15ca4cef3d3c7073412212&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-08-19T15:45:37Z&quot;, GoVersion:&quot;go1.16.7&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;21&quot;, GitVersion:&quot;v1.21.2&quot;, GitCommit:&quot;881d4a5a3c0f4036c714cfb601b377c4c72de543&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-10-21T05:13:01Z&quot;, GoVersion:&quot;go1.16.5&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<p>The spark version is 2.4.5 and the command I'm using is</p>
<pre><code>&lt;SPARK_PATH&gt;/bin/spark-submit --master k8s://&lt;HOST&gt;:443  \
--deploy-mode cluster  \
--name spark-pi3 \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.container.image=docker.io/datamechanics/spark:2.4.5-hadoop-3.1.0-java-8-scala-2.11-python-3.7-dm14  \
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.azure-fileshare-pvc.options.claimName=azure-fileshare-pvc \
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.azure-fileshare-pvc.mount.path=/opt/spark/work-dir \
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.azure-fileshare-pvc.options.claimName=azure-fileshare-pvc \
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.azure-fileshare-pvc.mount.path=/opt/spark/work-dir \
--verbose /opt/spark/work-dir/wordcount2.py
</code></pre>
<p>The PV and PVC are pretty basic. The PV yml is:</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: azure-fileshare-pv
  labels:
    usage: azure-fileshare-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  azureFile:
    secretName: azure-storage-secret
    shareName: dssparktestfs
    readOnly: false
    secretNamespace: spark-operator
</code></pre>
<p>The PVC yml is:</p>
<pre><code>kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: azure-fileshare-pvc
  # Set this annotation to NOT let Kubernetes automatically create
  # a persistent volume for this volume claim.
  annotations:
    volume.beta.kubernetes.io/storage-class: &quot;&quot;
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  selector:
    # To make sure we match the claim with the exact volume, match the label
    matchLabels:
      usage: azure-fileshare-pv
</code></pre>
<p>Let me know if more info is needed.</p>

<blockquote>
<p>The owner and user are root.</p>
</blockquote>
<p>It looks like you've mounted your volume as root. Your problem:</p>
<pre><code>chmod: changing permissions of '/opt/spark/work-dir/output/counts': Operation not permitted
</code></pre>
<p>is due to the fact that you are trying to change the permissions of a file that you are not the owner of. So you need to change the owner of the file first.</p>
<p>The easiest solution is to <code>chown</code> on the resource you want to access. However, this is often not feasible as it can lead to privilege escalation as well as the image itself may block this possibility. In this case you can create <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/" rel="nofollow noreferrer">security context</a>.</p>
<blockquote>
<p>A security context defines privilege and access control settings for a Pod or Container. Security context settings include, but are not limited to:</p>
<ul>
<li><p>Discretionary Access Control: Permission to access an object, like a file, is based on  <a href="https://wiki.archlinux.org/index.php/users_and_groups" rel="nofollow noreferrer">user ID (UID) and group ID (GID)</a>.</p>
</li>
<li><p><a href="https://en.wikipedia.org/wiki/Security-Enhanced_Linux" rel="nofollow noreferrer">Security Enhanced Linux (SELinux)</a>: Objects are assigned security labels.</p>
</li>
<li><p>Running as privileged or unprivileged.</p>
</li>
<li><p><a href="https://linux-audit.com/linux-capabilities-hardening-linux-binaries-by-removing-setuid/" rel="nofollow noreferrer">Linux Capabilities</a>: Give a process some privileges, but not all the privileges of the root user.
&gt;</p>
</li>
<li><p><a href="https://kubernetes.io/docs/tutorials/clusters/apparmor/" rel="nofollow noreferrer">AppArmor</a>: Use program profiles to restrict the capabilities of individual programs.</p>
</li>
<li><p><a href="https://kubernetes.io/docs/tutorials/clusters/seccomp/" rel="nofollow noreferrer">Seccomp</a>: Filter a process's system calls.</p>
</li>
<li><p>AllowPrivilegeEscalation: Controls whether a process can gain more privileges than its parent process. This bool directly controls whether the  <a href="https://www.kernel.org/doc/Documentation/prctl/no_new_privs.txt" rel="nofollow noreferrer"><code>no_new_privs</code></a>  flag gets set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged OR 2) has  <code>CAP_SYS_ADMIN</code>.</p>
</li>
<li><p>readOnlyRootFilesystem: Mounts the container's root filesystem as read-only.</p>
</li>
</ul>
<p>The above bullets are not a complete set of security context settings -- please see  <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#securitycontext-v1-core" rel="nofollow noreferrer">SecurityContext</a>  for a comprehensive list.</p>
<p>For more information about security mechanisms in Linux, see  <a href="https://www.linux.com/learn/overview-linux-kernel-security-features" rel="nofollow noreferrer">Overview of Linux Kernel Security Features</a></p>
</blockquote>
<p>You can <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods" rel="nofollow noreferrer">Configure volume permission and ownership change policy for Pods</a>.</p>
<blockquote>
<p>By default, Kubernetes recursively changes ownership and permissions for the contents of each volume to match the <code>fsGroup</code> specified in a Pod's <code>securityContext</code> when that volume is mounted. For large volumes, checking and changing ownership and permissions can take a lot of time, slowing Pod startup. You can use the <code>fsGroupChangePolicy</code> field inside a <code>securityContext</code> to control the way that Kubernetes checks and manages ownership and permissions for a volume.</p>
</blockquote>
<p>Here is an example:</p>
<pre class="lang-yaml prettyprint-override"><code>securityContext:
  runAsUser: 1000
  runAsGroup: 3000
  fsGroup: 2000
  fsGroupChangePolicy: &quot;OnRootMismatch&quot;
</code></pre>
<p>See also <a href="https://stackoverflow.com/questions/51200115/chown-changing-ownership-of-data-db-operation-not-permitted/51203031#51203031">this similar question</a>.</p>
