<p>I have a PySpark job present locally on my laptop. If I want to submit it on my minikube cluster using spark-submit, any idea how to pass the python file ?</p>

<p>I'm using following command, but it isn't working</p>

<pre><code>./spark-submit \
        --master k8s://https://192.168.64.6:8443 \
        --deploy-mode cluster \
        --name amazon-data-review \
        --conf spark.kubernetes.namespace=jupyter \
        --conf spark.executor.instances=1 \
        --conf spark.kubernetes.driver.limit.cores=1 \
        --conf spark.executor.cores=1 \
        --conf spark.executor.memory=500m \
        --conf spark.kubernetes.container.image=prateek/spark-ubuntu-2.4.5 \
        --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
        --conf spark.kubernetes.container.image.pullPolicy=Always \
        --conf spark.kubernetes.container.image.pullSecrets=dockerlogin \
        --conf spark.eventLog.enabled=true \
        --conf spark.eventLog.dir=s3a://prateek/spark-hs/ \
        --conf spark.hadoop.fs.s3a.access.key=xxxxx \
        --conf spark.hadoop.fs.s3a.secret.key=xxxxx \
        --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
        --conf spark.hadoop.fs.s3a.fast.upload=true \
        /Users/prateek/apache-spark/amazon_data_review.py
</code></pre>

<p>Getting following error - </p>

<pre><code>python3: can't open file '/Users/prateek/apache-spark/amazon_data_review.py': [Errno 2] No such file or directory
</code></pre>

<p>Is it required to keep the file within the Docker image itself. Can't we run it locally by keeping it on laptop</p>

Prateek Dubey
<p>Spark on Kubernetes doesn't support submitting locally stored files with <code>spark-submit</code>.</p>

<p>What you could do to make it work in cluster mode is to build Spark Docker image based on <code>prateek/spark-ubuntu-2.4.5</code> with <code>amazon_data_review.py</code> put inside of it (eg using Docker <code>COPY /Users/prateek/apache-spark/amazon_data_review.py /amazon_data_review.py</code> statement).</p>

<p>Then just refer to it in the <code>spark-submit</code> command using <code>local://</code> file system, eg.:</p>

<pre class="lang-sh prettyprint-override"><code>spark-submit \
  --master ... \
  --conf ... \
  ...
  local:///amazon_data_review.py
</code></pre>

<p>The alternative is to store that file on <code>http(s)://</code> or <code>hdfs://</code>-like accessible location.</p>
