<p>We are migrating our Spark workloads from Cloudera to Kubernetes.</p>
<p>For demo purposes, we wish to run one of our spark jobs within a minikube cluster using spark-submit in cluster mode.</p>
<p>I would like to pass a typesafe config file to my executors using the spark.file conf (I tried --files as well). The configuration file has been copied to the spark docker image at build time at the /opt/spark/conf directory.</p>
<p>Yet when I submit my job, I have a <strong>java.io.FileNotFoundException: File file:/opt/spark/conf/application.conf does not exist</strong>.</p>
<p>My understanding is that spark.files copies the files from driver to executors' working directory.</p>
<p>Am I missing something ? Thank you for your help.</p>
<p>Here is my spark-submit command</p>
<pre><code>spark-submit \
        --master k8s://https://192.168.49.2:8443 \
        --driver-memory ${SPARK_DRIVER_MEMORY} --executor-memory ${SPARK_EXECUTOR_MEMORY} \
        --deploy-mode cluster \
        --class &quot;${MAIN_CLASS}&quot; \
        --conf spark.driver.defaultJavaOptions=&quot;-Dconfig.file=local://${POD_CONFIG_DIR}/application.conf $JAVA_ARGS&quot; \
        --conf spark.files=&quot;file:///${POD_CONFIG_DIR}/application.conf,file:///${POD_CONFIG_DIR}/tlereg.properties&quot; \
        --conf spark.executor.defaultJavaOptions=&quot;-Dconfig.file=local://./application.conf&quot; \
        --conf spark.executor.instances=5 \
        --conf spark.kubernetes.container.image=$SPARK_CONTAINER_IMAGE \
        --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
        --conf spark.kryoserializer.buffer.max=512M \
        --conf spark.driver.maxResultSize=8192M \
        --conf spark.kubernetes.authenticate.caCertFile=$HOME/.minikube/ca.crt \
        --conf spark.executor.extraClassPath=&quot;./&quot; \
        local:///path/to/uber/jar.jar \
        &quot;${PROG_ARGS[@]}&quot; &gt; $LOG_FILE 2&gt;&amp;1
</code></pre>

<p>I've figured it out. <code>spark-submit</code> sends a request to kubernetes master's api-server to create a driver pod. A configmap volume is mounted to the driver's pod at <code>mountPath: /opt/spark/conf</code>, which overrides my config files located at that path in the docker container.
Workaround : editing /opt/spark/conf to /opt/spark/config in Dockerfile so that my configuration files are copied from the latter.</p>
