<p>I am having some issues when trying to launch Spark jobs via the Kubernetes scheduler.</p>
<p>I want all my driver/executor pods to be spawned onto nodes which has a certain taint. Because of this, I want to specify tolerations which will be directly injected into the pods configuration files. Currently, there is no default way directly from the <code>spark-submit</code> command</p>
<p>According to <a href="https://github.com/palantir/k8s-spark-scheduler#usage" rel="nofollow noreferrer">this</a> and <a href="https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md#pod-template" rel="nofollow noreferrer">this</a>, a user should be able to specify a pod template which can be set with the following parameters: <code>spark.kubernetes.driver.podTemplateFile</code> and <code>spark.kubernetes.executor.podTemplateFile</code>.</p>
<p>I tried specifying those parameters in the <code>spark-submit</code> command with the following file:</p>
<p><code>pod_template.template</code></p>
<pre><code>apiVersion: v1
kind: Pod
spec:
  tolerations:
  - effect: NoSchedule
    key: dedicated
    operator: Equal
    value: test
</code></pre>
<p>However, this toleration never gets added to the launched driver pod. Is currently a way to solve this?</p>
<p>For reference, here is the full spark-submit command:</p>
<pre><code>/opt/spark/bin/spark-submit --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.persistent.options.claimName=pvc-storage --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.persistent.mount.subPath=test-stage1/spark --conf spark.executor.memory=1G --conf spark.executor.instances=1 --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.persistent.mount.subPath=test-stage1/spark --conf spark.kubernetes.executor.limit.cores=1 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.namespace=test-stage1 --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.persistent.mount.path=/persistent --conf spark.kubernetes.driver.limit.memory=3G --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.persistent.mount.path=/persistent --conf spark.submit.deployMode=cluster --conf spark.kubernetes.container.image=&lt;SPARK IMAGE&gt; --conf spark.master=k8s://https://kubernetes.default.svc --conf spark.kubernetes.driver.limit.cores=1  --conf spark.executor.cores=1 --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.persistent.options.claimName=pvc-storage --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.kubernetes.executor.podTemplateFile=//opt/pod_template.template --conf spark.kubernetes.driver.podTemplateFile=//opt/pod_template.template local:///opt/spark/examples/src/main/python/pi.py 100
</code></pre>

<p>I have checked various documentations and found few things that might be misconfigured here:</p>

<ol>
<li>Your <code>pod_template.template</code> should have the <code>.yaml</code> at the end</li>
<li>You did not specify <code>spark.kubernetes.driver.pod.name</code> in your <code>spark-submit</code> command nor in the <code>pod_template.template.yaml</code> in a form of <code>metadata</code></li>
<li>You have used double <code>//</code> when specifing path for <code>spark.kubernetes.driver.podTemplateFile=</code> and <code>spark.kubernetes.executor.podTemplateFile=</code></li>
<li>You should put all your tolerations in <code>""</code>, for example: <code>effect: "NoSchedule"</code></li>
</ol>

<p>Please let me know if that helped.</p>
