<p>I'm using <code>spark-submit</code> to create a spark driver pod on my k8s cluster. When I run </p>

<pre class="lang-sh prettyprint-override"><code>bin/spark-submit 
--master k8s://https://my-cluster-url:443
--deploy-mode cluster 
--name spark-test 
--class com.my.main.Class 
--conf spark.executor.instances=3 
--conf spark.kubernetes.allocation.batch.size=3 
--conf spark.kubernetes.namespace=my-namespace 
--conf spark.kubernetes.container.image.pullSecrets=my-cr-secret 
--conf spark.kubernetes.container.image.pullPolicy=Always 
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.my-pvc.mount.path=/var/service/src/main/resources/ 
--conf spark.kubernetes.container.image=my-registry.io/spark-test:test-3.0.0 
local:///var/service/my-service-6.3.0-RELEASE.jar
</code></pre>

<p><code>spark-submit</code> successfully creates a pod in my k8s cluster. However, many of the config options I specified are not seen. For example, the pod does not have a volume mounted at <code>/var/service/src/main/resources/</code> despite the existence of a <code>persistentVolumeClaim</code> on the cluster called <code>my-pvc</code>. Further, the pod has not been given the specified image pull secret <code>my-cr-secret</code>, causing an <code>ImagePullBackOff</code> error. On the other hand, the pod <em>is</em> properly created in the <code>my-namespace</code> namespace and the pull policy <code>Always</code>.</p>

<p>I have attempted this using spark <a href="https://spark.apache.org/docs/3.0.0-preview/running-on-kubernetes.html" rel="nofollow noreferrer">3.0.0</a> and <a href="https://spark.apache.org/docs/2.4.5/running-on-kubernetes.html" rel="nofollow noreferrer">2.4.5</a></p>

<p>Why are some config options not reflected in the pod created on my cluster?</p>

Mike S
<p>Figured out the issue:</p>

<p>I currently have spark 2.3.1 installed locally and the variable <code>SPARK_HOME</code> points to <code>/usr/local/spark</code>. For this current project I downloaded a distribution of spark 2.4.5. I was in the 2.4.5 directory and running <code>bin/spark-submit</code>, which should have (as far as I can tell) pointed to the <code>spark-submit</code> bundled in 2.4.5. However, running <code>bin/spark-submit --version</code> revealed that the version being run was 2.3.1. The configurations that were being ignored in my question above were not available in 2.3.1.</p>

<p>Simply changing <code>SPARK_HOME</code> to the new directory fixed the issue</p>
