title: Spark 1.6.2 & YARN: diagnostics: Application failed 2 times due to AM Container for exited with exitCode: -1
body: <p>I have a cluster of 2 machines and am trying to submit a spark job with YARN cluster manager.</p>
  
  <ul>
  <li>vanilla Spark 1.6.2 built aginst hadoop 2.6.2</li>
  <li>vanilla Hadoop 2.7.2</li>
  </ul>
  
  <p>I can successfully run map-reduce jobs and spark jobs with standalone cluster manager. But when I run it with YARN, I got an error.</p>
  
  <ul>
  <li>Any suggestions how to get it to work?</li>
  <li>How do I enable more verbose logging? The error message is absolutely unclear</li>
  <li>Why no log files are created under hadoop/logs/userlogs/applicationXXX?</li>
  <li>Rhetorical question: IMO: hadoop logging &amp; diagnostic isn't very well. Why is that? Hadoop seems to be an established product.</li>
  </ul>
  
  <p>Below is the output:</p>
  
  <pre><code>mike@mp-desktop ~/opt/hadoop $ spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster  ~/prg/scala/spark-examples_2.11-1.0.jar     10
  16/07/09 08:59:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  16/07/09 08:59:01 INFO client.RMProxy: Connecting to ResourceManager at mp-desktop/192.168.1.60:8050
  16/07/09 08:59:01 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers
  16/07/09 08:59:01 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
  16/07/09 08:59:01 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
  16/07/09 08:59:01 INFO yarn.Client: Setting up container launch context for our AM
  16/07/09 08:59:01 INFO yarn.Client: Setting up the launch environment for our AM container
  16/07/09 08:59:01 INFO yarn.Client: Preparing resources for our AM container
  16/07/09 08:59:02 INFO yarn.Client: Uploading resource file:/home/mike/opt/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar -&gt; hdfs://mp-desktop:9000/user/mike/.sparkStaging/application_1468043888852_0001/spark-assembly-1.6.2-hadoop2.6.0.jar
  16/07/09 08:59:06 INFO yarn.Client: Uploading resource file:/home/mike/prg/scala/spark-examples_2.11-1.0.jar -&gt; hdfs://mp-desktop:9000/user/mike/.sparkStaging/application_1468043888852_0001/spark-examples_2.11-1.0.jar
  16/07/09 08:59:06 INFO yarn.Client: Uploading resource file:/tmp/spark-2ee6dfd6-e9d3-4ca4-9e98-5ce9e75dc757/__spark_conf__7114661171911035574.zip -&gt; hdfs://mp-desktop:9000/user/mike/.sparkStaging/application_1468043888852_0001/__spark_conf__7114661171911035574.zip
  16/07/09 08:59:06 INFO spark.SecurityManager: Changing view acls to: mike
  16/07/09 08:59:06 INFO spark.SecurityManager: Changing modify acls to: mike
  16/07/09 08:59:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mike); users with modify permissions: Set(mike)
  16/07/09 08:59:07 INFO yarn.Client: Submitting application 1 to ResourceManager
  16/07/09 08:59:07 INFO impl.YarnClientImpl: Submitted application application_1468043888852_0001
  16/07/09 08:59:08 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:08 INFO yarn.Client: 
       client token: N/A
       diagnostics: N/A
       ApplicationMaster host: N/A
       ApplicationMaster RPC port: -1
       queue: default
       start time: 1468043947113
       final status: UNDEFINED
       tracking URL: http://mp-desktop:8088/proxy/application_1468043888852_0001/
       user: mike
  16/07/09 08:59:09 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:10 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:11 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:12 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:13 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:14 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:15 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:16 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:17 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:18 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:19 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:20 INFO yarn.Client: Application report for application_1468043888852_0001 (state: ACCEPTED)
  16/07/09 08:59:21 INFO yarn.Client: Application report for application_1468043888852_0001 (state: FAILED)
  16/07/09 08:59:21 INFO yarn.Client: 
       client token: N/A
       diagnostics: Application application_1468043888852_0001 failed 2 times due to AM Container for appattempt_1468043888852_0001_000002 exited with  exitCode: -1
  For more detailed output, check application tracking page:http://mp-desktop:8088/cluster/app/application_1468043888852_0001Then, click on links to logs of each attempt.
  Diagnostics: File /home/mike/hadoopstorage/nm-local-dir/usercache/mike/appcache/application_1468043888852_0001/container_1468043888852_0001_02_000001 does not exist
  Failing this attempt. Failing the application.
       ApplicationMaster host: N/A
       ApplicationMaster RPC port: -1
       queue: default
       start time: 1468043947113
       final status: FAILED
       tracking URL: http://mp-desktop:8088/cluster/app/application_1468043888852_0001
       user: mike
  16/07/09 08:59:21 INFO yarn.Client: Deleting staging directory .sparkStaging/application_1468043888852_0001
  Exception in thread "main" org.apache.spark.SparkException: Application application_1468043888852_0001 finished with failed status
      at org.apache.spark.deploy.yarn.Client.run(Client.scala:1034)
      at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1081)
      at org.apache.spark.deploy.yarn.Client.main(Client.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  16/07/09 08:59:21 INFO util.ShutdownHookManager: Shutdown hook called
  16/07/09 08:59:21 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2ee6dfd6-e9d3-4ca4-9e98-5ce9e75dc757
  </code></pre>
  
  <p>Thanks!</p>