title: Spark in cluster mode throws error if a SparkContext is not started
body: <p>I have a Spark job that initializes the spark context only if it is really necessary:</p>
  
  <pre><code>val conf = new SparkConf()
  val jobs: List[Job] = ??? //get some jobs
  if(jobs.nonEmpty) {
    val sc = new SparkContext(conf)
    sc.parallelize(jobs).foreach(....)
  } else {
      //do nothing
  }
  </code></pre>
  
  <p>It worked fine on Yarn if deploy-mode is 'client'</p>
  
  <pre><code>spark-submit --master yarn --deploy-mode client
  </code></pre>
  
  <p>Then I switched deploy mode to 'cluster' and it started to crash in case of <code>jobs.isEmpty</code></p>
  
  <pre><code>spark-submit --master yarn --deploy-mode cluster
  </code></pre>
  
  <p>Below is the error text:</p>
  
  <blockquote>
    <p>INFO yarn.Client: Application report for
    application_1509613523426_0017 (state: ACCEPTED) 
    17/11/02 11:37:17</p>
    
    <p>INFO yarn.Client: Application report for
    application_1509613523426_0017 (state: FAILED) 17/11/02 11:37:17 </p>
    
    <p>INFO yarn.Client:      client token: N/A   diagnostics: Application
    application_1509613523426_0017 failed 2 times due to AM Container for
    appattempt_1509613523426_0017_000002 exited with  exitCode: -1000 For
    more detailed output, check application tracking</p>
    
    <p>page:<a href="http://xxxxxx.com:8088/cluster/app/application_1509613523426_0017Then" rel="nofollow noreferrer">http://xxxxxx.com:8088/cluster/app/application_1509613523426_0017Then</a>,
    click on links to logs of each attempt. Diagnostics: File does not
    exist:
    hdfs://xxxxxxx/.sparkStaging/application_1509613523426_0017/__spark_libs__997458388067724499.zip
    java.io.FileNotFoundException: File does not exist:
    hdfs://xxxxxxx/.sparkStaging/application_1509613523426_0017/__spark_libs__997458388067724499.zip
      at
    org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
      at
    org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
      at
    org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at
    org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
      at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
      at
    org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
      at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
      at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
      at java.security.AccessController.doPrivileged(Native Method)   at
    javax.security.auth.Subject.doAs(Subject.java:422)    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
      at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
      at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)     at
    java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)     at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)</p>
    
    <p>Failing this attempt. Failing the application.     ApplicationMaster
    host: N/A      ApplicationMaster RPC port: -1      queue: dev      start time:
    1509622629354      final status: FAILED    tracking URL:
    <a href="http://xxxxxx.com:8088/cluster/app/application_1509613523426_0017" rel="nofollow noreferrer">http://xxxxxx.com:8088/cluster/app/application_1509613523426_0017</a>      user: xxx Exception in thread "main"
    org.apache.spark.SparkException: Application
    application_1509613523426_0017 finished with failed status    at
    org.apache.spark.deploy.yarn.Client.run(Client.scala:1104)    at
    org.apache.spark.deploy.yarn.Client$.main(Client.scala:1150)  at
    org.apache.spark.deploy.yarn.Client.main(Client.scala)    at
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)   at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)     at
    org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
      at
    org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    17/11/02 11:37:17 INFO util.ShutdownHookManager: Shutdown hook called
    17/11/02 11:37:17 INFO util.ShutdownHookManager: Deleting directory
    /tmp/spark-a5b20def-0218-4b0c-b9f8-fdf8a1802e95</p>
  </blockquote>
  
  <p>Is it a bug in Yarn support or I'm missing something?</p>