<p>I am trying to submit the Spark application to Kubernetes cluster, the job file is at : /opt/spark/applications/ path, submitting spark using below command:</p>

<pre><code>${SPARK_PATH}/bin/spark-submit \
--master &lt;K8S_MASTER&gt; \
--deploy-mode cluster \
--name spark-py-driver \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=spark-py:2.4.0-rc3 \
--conf spark.kubernetes.driver.pod.name=spark-py-driver \
--conf spark.executor.memory=2g \
--conf spark.driver.memory=2g \
local:///opt/spark/applications/spark_submit_test_job.py
</code></pre>

<p>Am getting 'No such file or directory' error, though the job file is there at the path on the node.</p>

<pre><code>python: can't open file '/opt/spark/applications/spark_submit_test_job.py': [Errno 2] No such file or directory
</code></pre>

Lakshman Battini
<p>As the instruction on <a href="https://spark.apache.org/docs/latest/running-on-kubernetes#cluster-mode" rel="nofollow noreferrer">spark running on Kubernetes</a> states:</p>

<blockquote>
  <p>Finally, notice that in the above example we specify a jar with a specific URI with a scheme of <code>local://</code>. This URI is the location of the example jar that <strong>is already in the Docker image</strong>.</p>
</blockquote>

<p>You need to create a correct Dockerfile with the script already present in the image.</p>

<p>A better solution would be using a repository that can be loaded when you are running the image, or instead you can use <a href="https://spark.apache.org/docs/latest/running-on-kubernetes#dependency-management" rel="nofollow noreferrer">Remote Dependencies</a></p>
