title: java.io.FileNotFoundException: File file:/tmp/spark- /__spark_libs__ .zip does not exist
body: <p>My Spark environments are like below :</p>
  
  <pre><code>OS : CentOS 7
  Hadoop : 2.7.4
  Spark : 2.2 for hadoop 2.7
  Eclipse : Oxygen
  </code></pre>
  
  <p>My Hadoop and Spark installation is successful. Command jps throws the following message</p>
  
  <pre><code>6738 Jps
  5219 Worker
  5220 Worker
  5222 Worker
  5575 org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar
  2906 NameNode
  3660 DataNode
  3965 ResourceManager
  4381 NodeManager
  5038 Master
  4783 JobHistoryServer
  </code></pre>
  
  <p>And also spark-shell command works well.</p>
  
  <pre><code>$ spark-shell --master yarn
  17/09/17 08:48:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  17/09/17 08:48:25 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
  17/09/17 08:48:42 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
  Spark context Web UI available at http://192.168.200.51:4040
  Spark context available as 'sc' (master = yarn, app id = application_1505604907413_0003).
  Spark session available as 'spark'.
  Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
        /_/
  
  Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
  Type in expressions to have them evaluated.
  Type :help for more information.
  
  scala&gt; 
  </code></pre>
  
  <p>The problem is my Java Spark client on Eclipse IDE. The project is simple. The jars of Spark jars folder are included on build path. These are Java codes</p>
  
  <pre><code>package com.aaa.spark;
  
  import org.apache.spark.SparkConf;
  import org.apache.spark.api.java.JavaSparkContext;
  
  public class JavaClient {
  
      public static void main(String[] args) {
          // TODO Auto-generated method stub
          SparkConf conf = new SparkConf().setAppName("SparkTest").setMaster("yarn-client");
      JavaSparkContext context = new JavaSparkContext(conf);
  
      System.out.println(context.master() + " : " + context.version());
      context.stop();
      }
  }
  </code></pre>
  
  <p>But this simple code does not work. It throws the following exception.</p>
  
  <pre><code>17/09/17 08:42:50 INFO Client: Application report for application_1505604907413_0002 (state: ACCEPTED)
  17/09/17 08:42:50 INFO Client: 
       client token: N/A
       diagnostics: N/A
       ApplicationMaster host: N/A
       ApplicationMaster RPC port: -1
       queue: default
       start time: 1505605369876
       final status: UNDEFINED
       tracking URL: http://master:8088/proxy/application_1505604907413_0002/
       user: m_usr
  17/09/17 08:42:51 INFO Client: Application report for application_1505604907413_0002 (state: FAILED)
  17/09/17 08:42:51 INFO Client: 
       client token: N/A
       diagnostics: Application application_1505604907413_0002 failed 2 times due to AM Container for appattempt_1505604907413_0002_000002 exited with  exitCode: -1000
  For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1505604907413_0002Then, click on links to logs of each attempt.
  Diagnostics: File file:/tmp/spark-1b711310-8ca2-43ff-a15b-c5c7951d8b56/__spark_libs__5711484877820940876.zip does not exist
  java.io.FileNotFoundException: File file:/tmp/spark-1b711310-8ca2-43ff-a15b-c5c7951d8b56/__spark_libs__5711484877820940876.zip does not exist
      at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
      at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
      at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
      at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
      at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
      at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
      at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
      at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
      at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
      at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
  
  Failing this attempt. Failing the application.
       ApplicationMaster host: N/A
       ApplicationMaster RPC port: -1
       queue: default
       start time: 1505605369876
       final status: FAILED
       tracking URL: http://master:8088/cluster/app/application_1505604907413_0002
       user: m_usr
  17/09/17 08:42:51 INFO Client: Deleted staging directory file:/home/m_usr/.sparkStaging/application_1505604907413_0002
  17/09/17 08:42:51 ERROR SparkContext: Error initializing SparkContext.
  org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
      at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
      at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
      at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
      at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:509)
      at org.apache.spark.api.java.JavaSparkContext.&lt;init&gt;(JavaSparkContext.scala:58)
      at com.aaa.spark.JavaClient.main(JavaClient.java:11)
  17/09/17 08:42:51 INFO SparkUI: Stopped Spark web UI at http://192.168.200.51:4040
  </code></pre>
  
  <p>I am stuck here. I can not understand what configuration is wrong in my Eclipse Java code.</p>
  
  <h2>Update 1</h2>
  
  <p>When I execute the <code>spark-shell --master=yarn</code>, the error messages are not shown and the success message is shown,</p>
  
  <pre><code>17/09/20 20:42:26 INFO Client: Application report for application_1505905006868_0010 (state: ACCEPTED)
  17/09/20 20:42:26 INFO Client: 
       client token: N/A
       diagnostics: N/A
       ApplicationMaster host: N/A
       ApplicationMaster RPC port: -1
       queue: default
       start time: 1505907745945
       final status: UNDEFINED
       tracking URL: http://master:8088/proxy/application_1505905006868_0010/
       user: m_usr
  17/09/20 20:42:27 INFO Client: Application report for application_1505905006868_0010 (state: ACCEPTED)
  17/09/20 20:42:28 INFO Client: Application report for application_1505905006868_0010 (state: ACCEPTED)
  17/09/20 20:42:29 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
  17/09/20 20:42:29 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -&gt; master, PROXY_URI_BASES -&gt; http://master:8088/proxy/application_1505905006868_0010), /proxy/application_1505905006868_0010
  </code></pre>
  
  <p>However, when I execute the above Spark Java code on Eclipse IDE, error messages are thrown like below:</p>
  
  <pre><code>17/09/20 20:40:02 INFO Client: Application report for application_1505905006868_0009 (state: ACCEPTED)
  17/09/20 20:40:02 INFO Client: 
       client token: N/A
       diagnostics: N/A
       ApplicationMaster host: N/A
       ApplicationMaster RPC port: -1
       queue: default
       start time: 1505907601211
       final status: UNDEFINED
       tracking URL: http://master:8088/proxy/application_1505905006868_0009/
       user: m_usr
  17/09/20 20:40:03 INFO Client: Application report for application_1505905006868_0009 (state: FAILED)
  17/09/20 20:40:03 INFO Client: 
       client token: N/A
       diagnostics: Application application_1505905006868_0009 failed 2 times due to AM Container for appattempt_1505905006868_0009_000002 exited with  exitCode: -1000
  For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1505905006868_0009Then, click on links to logs of each attempt.
  Diagnostics: File file:/tmp/spark-abd84352-1219-4ffc-92dc-ae94140d936a/__spark_libs__3762603621456052516.zip does not exist
  java.io.FileNotFoundException: File file:/tmp/spark-abd84352-1219-4ffc-92dc-ae94140d936a/__spark_libs__3762603621456052516.zip does not exist
  </code></pre>
  
  <p>I think I miss some Spark configuration on Eclipse IDE.</p>
  
  <h2>Update 2</h2>
  
  <p>Spark Installation Process</p>
  
  <p>copy spark-bin-file to s_usr01 and s_usr02 with scp commands</p>
  
  <pre><code>scp spark-2.2.0-bin-hadoop2.7.tgz s_usr01@slave01:/home/s_usr0
  </code></pre>
  
  <p>tar spark gzip file on each s_usr01 and s_usr02</p>
  
  <pre><code>$ ssh s_usr01@slave01 tar zxvf spark-2.2.0-bin-hadoop2.7.tgz
  $ ssh s_usr01@slave01 rm spark-2.2.0-bin-hadoop2.7.tgz
  </code></pre>
  
  <p>set path on .bashrc file</p>
  
  <pre><code>############ Eclipse PATH ###########
  export ECLIPSE_HOME=./eclipse/jee-oxygen/eclipse
  export PATH=$PATH:$ECLIPSE_HOME
  
  ######### JDK8 PATH ############
  JAVA_HOME=/usr/java/jdk1.8.0_131
  CLASSPATH=.:$JAVA_HOME/lib/tools.jar
  PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
  export JAVA_HOME CLASSPATH
  export PATH
  
  ############ Hadoop PATH ###########
  export HADOOP_HOME=/home/m_usr/hadoop-2.7.4
  export PATH=$PATH:$HADOOP_HOME/bin
  export PATH=$PATH:$HADOOP_HOME/sbin
  export PATH=$PATH:/usr/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin
  export HADOOP_PID_DIR=/home/m_usr/hadoop-2.7.4/pids
  export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar:wq
  export HADOOP_INSTALL=$HADOOP_HOME
  export HADOOP_MAPRED_HOME=$HADOOP_HOME
  export HADOOP_COMMON_HOME=$HADOOP_HOME
  export HADOOP_HDFS_HOME=$HADOOP_HOME
  export HADOOP_YARN_HOME=$HADOOP_HOME
  export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
  export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
  export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
  export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
  export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
  
  export YARN_HOME=$HADOOP_HOME
  export PATH=$PATH:$YARN_HOME
  
  ############ Spark Path ############
  export SPARK_HOME=/home/m_usr/spark-2.2.0-bin-hadoop2.7
  export SPARK_SUBMIT=/home/m_usr/spark-2.2.0-bin-hadoop2.7/bin/spark-submit
  
  export PATH=$PATH:$SPARK_HOME/bin
  export PATH=$PATH:$SPARK_HOME/sbin
  </code></pre>
  
  <p>modify configuation files on $SPARK_HOME/conf</p>
  
  <pre><code>$ vi spark-env.sh
  export JAVA_HOME=/usr/java/jdk1.8.0_131
  export HADOOP_HOME=/home/m_usr/hadoop-2.7.4
  export SPARK_HOME=/home/m_usr/spark-2.2.0-bin-hadoop2.7
  export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
  export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
  
  $ vi spark-defaults.conf
  spark.master                     spark://master:7077
  spark.eventLog.enabled           true
  spark.eventLog.dir               file:///home/m_usr/spark-2.2.0-bin-hadoop2.7/sparkeventlogs
  spark.serializer                 org.apache.spark.serializer.KryoSerializer
  
  vi log4j.properties
  
  # Set everything to be logged to the console
  log4j.rootCategory=WARN, console
  log4j.appender.console=org.apache.log4j.ConsoleAppender
  
  
  $ vi slaves
  slave01
  slave02
  </code></pre>
  
  <p>remote copy the spark configuration files to s_usr01 and s_usr02 spark cluster</p>
  
  <pre><code>$ scp * s_usr01@slave01:/home/s_usr01/spark-2.2.0-bin-hadoop2.7/conf
  </code></pre>
  
  <p>And finally execute the spark with below sh command</p>
  
  <pre><code>./sbin/start-all.sh
  </code></pre>
  
  <p>As you see, spark-shell works well with Yarn. The problem is Spark Java Client on Eclipse IDE.</p>
  
  <h2>Updated 3</h2>
  
  <p>I found the something weird. The exception from Eclipse IDE is </p>
  
  <pre><code>java.io.FileNotFoundException: File file:/tmp/spark-cf36a407-60c1-4100-89aa-ad38fdce7a87/__spark_libs__3088825805082848673.zip does not exist
  </code></pre>
  
  <p>However this file is generated temporary like below,</p>
  
  <pre><code>ls -al /tmp
  drwxr-xr-x.  2 m_usr m_usr  32  9월 25 20:06 hsperfdata_m_usr
  drwxr-xr-x.  2 root  root    6  9월 25 20:04 hsperfdata_root
  drwx------.  3 m_usr m_usr 105  9월 25 20:06 spark-cf36a407-60c1-4100-89aa-ad38fdce7a87
  </code></pre>
  
  <p>and my Java client can not find this file for some wrong reason.</p>
  
  <pre><code>17/09/25 20:06:11 INFO Client: Application report for application_1506335085215_0010 (state: ACCEPTED)
  17/09/25 20:06:12 INFO Client: Application report for application_1506335085215_0010 (state: FAILED)
  17/09/25 20:06:12 INFO Client: 
       client token: N/A
       diagnostics: Application application_1506335085215_0010 failed 2 times due to AM Container for appattempt_1506335085215_0010_000002 exited with  exitCode: -1000
  </code></pre>
  
  <p>But the spark-shell command never throws this exception and works well.</p>
  
  <pre><code>spark-shell --master yarn
  </code></pre>
  
  <p>Any idea? </p>