title: Spark-submit deploy mode cluster __spark_conf__.zip does not exist error
body: <p>As per title, I am submitting a spark submit job and got the error below. It works fine if I run deploy mode client, but not cluster mode. Anyone might have any experience dealing with this?</p>
  <p>Command:
  ./spark-submit --class &quot;LocalPi&quot; --master yarn --deploy-mode cluster /tmp/movierecs_2.11-1.0.jar</p>
  <p>(I've noticed that it tries to upload this spark conf zip into hdfs)</p>
  <p>Uploading resource file:/tmp/spark-a9ec721a-2733-450c-a4df-95a4fb41a816/__spark_conf__8933063333289205256.zip -&gt; hdfs://BDSSBX/user/hdfs/.sparkStaging/application_1637812068012_0017/<strong>spark_conf</strong>.zip</p>
  <p>(The actual error)</p>
  <p>java.io.FileNotFoundException: File does not exist: hdfs://BDSSBX/user/hdfs/.sparkStaging/application_1637812068012_0017/<strong>spark_conf</strong>.zip</p>