title: Submitting spark job from eclipse to yarn-client with scala
body: <p>I am new to spark and scala and I am having a hard time to submit a Spark job as YARN client. Doing this via the spark shell (spark submit) is no problem same goes for: first creating a spark job in eclipse then compile it into jar and use spark submit via the kernel shell, like:</p>
  
  <pre><code> spark-submit --class ebicus.WordCount /u01/stage/mvn_test-0.0.1.jar
  </code></pre>
  
  <p>However using Eclipse to directly compile and submit it to YARN seems difficult.</p>
  
  <p>My project setup is the following: My cluster is running CDH cloudera 5.6. I have a maven project, using scala,<a href="http://i.stack.imgur.com/LliIY.png" rel="nofollow">My classpath / which is in sinc with my pom.xml</a></p>
  
  <p>My code is as follows:</p>
  
  <pre><code>package test
  
  import org.apache.spark.SparkConf
  import org.apache.spark.SparkContext
  import org.apache.hadoop.conf.Configuration;
  import org.apache.spark.deploy.yarn.Client;
  import org.apache.spark.TaskContext;
  import akka.actor
  import org.apache.spark.deploy.yarn.ClientArguments
  import org.apache.spark.deploy.ClientArguments
  
  object WordCount {
  
    def main(args: Array[String]): Unit = {
  //    val workaround = new File(".");
      System.getProperties().put("hadoop.home.dir",  "c:\\winutil\\");
      System.setProperty("SPARK_YARN_MODE", "true");
  
     val conf = new SparkConf()
        .setAppName("WordCount")
        .setMaster("yarn-client")
        .set("spark.hadoop.fs.defaultFS", "hdfs://namecluster.com:8020/user/username")
        .set("spark.hadoop.dfs.nameservices", "namecluster.com:8020")
        .set("spark.hadoop.yarn.resourcemanager.hostname", "namecluster.com")
        .set("spark.hadoop.yarn.resourcemanager.address", "namecluster:8032")
        .set("spark.hadoop.yarn.application.classpath",
                "/etc/hadoop/conf,"
            +"/usr/lib/hadoop/*,"
            +"/usr/lib/hadoop/lib/*,"
            +"/usr/lib/hadoop-hdfs/*,"
            +"/usr/lib/hadoop-hdfs/lib/*,"
            +"/usr/lib/hadoop-mapreduce/*,"
            +"/usr/lib/hadoop-mapreduce/lib/*,"
            +"/usr/lib/hadoop-yarn/*,"
            +"/usr/lib/hadoop-yarn/lib/*,"
            +"/usr/lib/spark/*,"
            +"/usr/lib/spark/lib/*,"
            +"/usr/lib/spark/lib/*"
        )
        .set("spark.driver.host","localhost");
  
      val sc = new SparkContext(conf);
  
      val file = sc.textFile("hdfs://namecluster.com:8020/user/root/testdir/test.csv")
      //Count number of words from a hive table (split is based on char 001)
      val counts = file.flatMap(line =&gt; line.split(1.toChar)).map(word =&gt; (word, 1)).reduceByKey(_ + _)
  
      //swap key and value with count value and sort from high to low 
      val test = counts.map(_.swap).sortBy(word =&gt;(word,1), false, 5)
  
      test.saveAsTextFile("hdfs://namecluster.com:8020/user/root/test1")
  
    }
  
  }
  </code></pre>
  
  <p>I'm receiving the next error message on the log files of the hadoop resource manager</p>
  
  <pre><code>YARN executor launch context:
    env:
      CLASSPATH -&gt; {{PWD}}&lt;CPS&gt;{{PWD}}/__spark__.jar&lt;CPS&gt;/etc/hadoop/conf&lt;CPS&gt;/usr/lib/hadoop/*&lt;CPS&gt;/usr/lib/hadoop/lib/*&lt;CPS&gt;/usr/lib/hadoop-hdfs/*&lt;CPS&gt;/usr/lib/hadoop-hdfs/lib/*&lt;CPS&gt;/usr/lib/hadoop-mapreduce/*&lt;CPS&gt;/usr/lib/hadoop-mapreduce/lib/*&lt;CPS&gt;/usr/lib/hadoop-yarn/*&lt;CPS&gt;/usr/lib/hadoop-yarn/lib/*&lt;CPS&gt;/usr/lib/spark/*&lt;CPS&gt;/usr/lib/spark/lib/*&lt;CPS&gt;/usr/lib/spark/lib/*&lt;CPS&gt;$HADOOP_MAPRED_HOME/*&lt;CPS&gt;$HADOOP_MAPRED_HOME/lib/*&lt;CPS&gt;$MR2_CLASSPATH
      SPARK_LOG_URL_STDERR -&gt; http://cloudera-002.fusion.ebicus.com:8042/node/containerlogs/container_1461679867178_0026_01_000005/hadriaans/stderr?start=-4096
      SPARK_YARN_STAGING_DIR -&gt; .sparkStaging/application_1461679867178_0026
      SPARK_YARN_CACHE_FILES_FILE_SIZES -&gt; 520473
      SPARK_USER -&gt; hadriaans
      SPARK_YARN_CACHE_FILES_VISIBILITIES -&gt; PRIVATE
      SPARK_YARN_MODE -&gt; true
      SPARK_YARN_CACHE_FILES_TIME_STAMPS -&gt; 1462288779267
      SPARK_LOG_URL_STDOUT -&gt; http://cloudera-002.fusion.ebicus.com:8042/node/containerlogs/container_1461679867178_0026_01_000005/hadriaans/stdout?start=-4096
      SPARK_YARN_CACHE_FILES -&gt; hdfs://cloudera-003.fusion.ebicus.com:8020/user/hadriaans/.sparkStaging/application_1461679867178_0026/spark-yarn_2.10-1.5.0.jar#__spark__.jar
  
    command:
      {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms1024m -Xmx1024m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=49961' -Dspark.yarn.app.container.log.dir=&lt;LOG_DIR&gt; org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url akka.tcp://sparkDriver@10.29.51.113:49961/user/CoarseGrainedScheduler --executor-id 4 --hostname cloudera-002.fusion.ebicus.com --cores 1 --app-id application_1461679867178_0026 --user-class-path file:$PWD/__app__.jar 1&gt; &lt;LOG_DIR&gt;/stdout 2&gt; &lt;LOG_DIR&gt;/stderr
  ===============================================================================
  
  16/05/03 17:19:58 INFO impl.ContainerManagementProtocolProxy: Opening proxy : cloudera-002.fusion.ebicus.com:8041
  16/05/03 17:20:01 INFO yarn.YarnAllocator: Completed container container_1461679867178_0026_01_000005 (state: COMPLETE, exit status: 1)
  16/05/03 17:20:01 INFO yarn.YarnAllocator: Container marked as failed: container_1461679867178_0026_01_000005. Exit status: 1. Diagnostics: Exception from container-launch.
  Container id: container_1461679867178_0026_01_000005
  Exit code: 1
  Stack trace: ExitCodeException exitCode=1: 
      at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
      at org.apache.hadoop.util.Shell.run(Shell.java:478)
      at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
      at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>Any tips or advises are welcome.</p>