<p>My goal is to setup configmap and then use the config file in the spark application. Here are the details:</p>
<p>I have a config file (test_config.cfg) that looks like this:</p>
<pre><code>[test_tracker]
url = http://localhost:8080/testsomething/
username = TEST
password = SECRET
</code></pre>
<p>I created the config map by running the following command:</p>
<pre><code>kubectl create configmap testcfg1 --from-file test_config.cfg
</code></pre>
<p>Now, I have a YAML file(testprog.yaml) with SparkApplication specs that look like this:</p>
<pre><code>apiVersion: &quot;sparkoperator.k8s.io/v1beta2&quot;
kind: SparkApplication
metadata:
  name: testprog
  namespace: default
spec:
  type: Python
  pythonVersion: &quot;3&quot;
  mode: cluster
  image: &quot;&lt;ip-url&gt;:5000/schemamatcher/schemamatcher-spark-py:latest&quot;
  imagePullPolicy: Always
  mainApplicationFile: local:///opt/spark/dependencies/testprog.py
  arguments: ['s3a://f1.parquet', 's3a://f2.parquet', '--tokenizer-type', 'param']
  sparkVersion: &quot;3.0.0&quot;
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    coreLimit: &quot;1200m&quot;
    memory: &quot;16g&quot;
    labels:
      version: 3.0.0
    serviceAccount: default
    configMaps:
      - name: testcfg1
        path: /mnt/config-maps
  executor:
    cores: 1
    instances: 2
    memory: &quot;20g&quot;
    labels:
      version: 3.0.0
  hadoopConf:
    &quot;fs.s3a.access.key&quot;: minio
    &quot;fs.s3a.secret.key&quot;: minio123
    &quot;fs.s3a.endpoint&quot;: http://&lt;ip-url&gt;:9000
</code></pre>
<p>Now, I am able to run the program using:</p>
<pre><code>kubectl apply -f testprog.yaml
</code></pre>
<p>the pod just runs fine and doesn't throw any error. But I am unable to see my config file at the path given and I don't understand why.
When the pod is executing I do:</p>
<pre><code>kubectl exec --stdin --tty test-driver -- /bin/bash
</code></pre>
<p>and I try to look for the config file in the path /mnt/config-maps I don't see anything. I tried a couple of things but no luck. Besides, some of the documentation says that mutation webhook should be setup and I think the previous guy did it but I am not sure how to check it (but I think it is there).</p>
<p>Any help would be great as I am new and I am still learning about k8s.</p>
<p>Update:
Have also tried to update the specs like this and run and still no luck.</p>
<pre><code>  volumes:
    - name: config
      configMap:
        name: testcfg1
  driver:
    cores: 1
    coreLimit: &quot;1200m&quot;
    memory: &quot;16g&quot;
    labels:
      version: 3.0.0
    serviceAccount: default
    volumeMounts:
      - name: config
        mountPath: /opt/spark
  executor:
    cores: 1
    instances: 2
    memory: &quot;20g&quot;
    labels:
      version: 3.0.0
    volumeMounts:
      - name: config
        mountPath: /opt/spark
</code></pre>

yguw
<p>Not sure if this issue was solved in Spark v3.0.0 (that you seem to be using), but there was a bug in Spark on Kubernetes that was preventing ConfigMaps from mounting properly. See this discussion: <a href="https://stackoverflow.com/a/58508313/8570169">https://stackoverflow.com/a/58508313/8570169</a></p>
