<p>Im trying to deploy spark (pyspark) in kubernetes using spark-submit, but I'm getting the following error :</p>
<blockquote>
<p>Exception in thread &quot;main&quot; org.apache.spark.SparkException: Please specify spark.kubernetes.file.upload.path property.    at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadFileUri(KubernetesUtils.scala:330)    at org.apache.spark.deploy.k8s.KubernetesUtils$.renameMainAppResource(KubernetesUtils.scala:300)    at</p>
</blockquote>
<p>Since i'm packing my dependencies trhough a virtual environment, I don't have the need of specify a remote cluster to retrieve them, so I'm no setting the parameter <code>spark.kubernetes.file.upload.path</code></p>
<p>I tried to include that parameter anyway, leaving an empty value, but it does'nt work.</p>
<p>My spark-submit command (which I trigger from a python script) is as follows:</p>
<pre><code>   cmd = f&quot;&quot;&quot; {SPARK_HOME}/bin/spark-submit
                                --master {SPARK_MASTER}
                                --deploy-mode cluster
                                --name spark-policy-engine
                                --executor-memory {EXECUTOR_MEMORY} \
                                --conf spark.executor.instances={N_EXECUTORS} 
                                --conf spark.kubernetes.container.image={SPARK_IMAGE}
                                --conf spark.kubernetes.file.upload.path=''
                                --packages org.mongodb.spark:mongo-spark-connector_2.12:3.0.1,org.apache.hadoop:hadoop-aws:3.3.1,com.amazonaws:aws-java-sdk-bundle:1.11.901,org.apache.hadoop:hadoop-common:3.3.1 
                                --archives pyspark_venv.tar.gz#environment {spark_files}
                                --format_id {format_id}
                                &quot;&quot;&quot;
</code></pre>
<p>As shown I'm including the parameter with within a --conf tag (as shown in <a href="https://spark.apache.org/docs/3.0.0-preview/running-on-kubernetes.html#:%7E:text=It%20can%20be%20found%20in,use%20with%20the%20Kubernetes%20backend.&amp;text=This%20will%20build%20using%20the%20projects%20provided%20default%20Dockerfiles%20." rel="nofollow noreferrer">https://spark.apache.org/docs/3.0.0-preview/running-on-kubernetes.html#:~:text=It%20can%20be%20found%20in,use%20with%20the%20Kubernetes%20backend.&amp;text=This%20will%20build%20using%20the%20projects%20provided%20default%20Dockerfiles%20.</a>), but wether is present or not, it just doesn't work</p>

<p>You need to specify a real path not an empty string, let's say in your image you have a tmp folder under /opt/spark, then the conf should be set like this:</p>
<pre><code>--conf spark.kubernetes.file.upload.path='local:///opt/spark/tmp'
</code></pre>
<p>If you don't want to use the</p>
<pre><code>   cmd = f&quot;&quot;&quot; {SPARK_HOME}/bin/spark-submit
                                --master {SPARK_MASTER}
                                --deploy-mode cluster
                                --name spark-policy-engine
                                --executor-memory {EXECUTOR_MEMORY} \
                                --conf spark.executor.instances={N_EXECUTORS} 
                                --conf spark.kubernetes.container.image={SPARK_IMAGE}
                                --packages org.mongodb.spark:mongo-spark-connector_2.12:3.0.1,org.apache.hadoop:hadoop-aws:3.3.1,com.amazonaws:aws-java-sdk-bundle:1.11.901,org.apache.hadoop:hadoop-common:3.3.1 
                                --archives pyspark_venv.tar.gz#environment {spark_files}
                                --format_id {format_id}
                                local:///opt/spark/work-dir/xxx.jar
                                &quot;&quot;&quot;
</code></pre>
