title: Run Spark Job via Uber Jar with Oozie and Hue
body: <p>I'm currently learning how to use Apache Oozie to run Spark Jobs in CDH 5.8 but seems to find problems. </p>
  
  <p>I'm compiling my spark job using IntelliJ > Build Artifact (into Uber JAR / Fat JAR) , and later remove its manifest file. Then I run spark-submit to run the JAR. It works fine.</p>
  
  <p>But when I specified a Spark Action with Oozie. I get the following error:</p>
  
  <pre><code>Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found
  java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
      at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:234)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
  Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
      ... 9 more
  </code></pre>
  
  <p><strong>job.properties:</strong></p>
  
  <pre><code>oozie.use.system.libpath=false
  security_enabled=False
  dryrun=False
  jobTracker=master.meshiang:8032
  nameNode=hdfs://master.meshiang:8020
  </code></pre>
  
  <p><strong>My Workflow</strong> : </p>
  
  <pre><code>&lt;workflow-app name="CSV" xmlns="uri:oozie:workflow:0.4"&gt;
      &lt;start to="spark-2bab"/&gt;
      &lt;kill name="Kill"&gt;
          &lt;message&gt;Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]&lt;/message&gt;
      &lt;/kill&gt;
      &lt;action name="spark-2bab"&gt;
          &lt;spark xmlns="uri:oozie:spark-action:0.1"&gt;
              &lt;job-tracker&gt;${jobTracker}&lt;/job-tracker&gt;
              &lt;name-node&gt;${nameNode}&lt;/name-node&gt;
              &lt;master&gt;local[*]&lt;/master&gt;
              &lt;mode&gt;client&lt;/mode&gt;
              &lt;name&gt;MySpark&lt;/name&gt;
                &lt;class&gt;ETL.CSVTransform&lt;/class&gt;
              &lt;jar&gt;/user/meshiang/jar/Spark-GetData.jar&lt;/jar&gt;
                &lt;arg&gt;work_id&lt;/arg&gt;
                &lt;arg&gt;csv_table&lt;/arg&gt;
                &lt;arg&gt;id/FirstName/Lastname&lt;/arg&gt;
                &lt;arg&gt;/user/meshiang/csv/ST1471448595.csv&lt;/arg&gt;
                &lt;arg&gt;,&lt;/arg&gt;
          &lt;/spark&gt;
          &lt;ok to="End"/&gt;
          &lt;error to="Kill"/&gt;
      &lt;/action&gt;
      &lt;end name="End"/&gt;
  &lt;/workflow-app&gt;
  </code></pre>
  
  <p>What I already did:</p>
  
  <ol>
  <li>When I put the same jar into /lib folder of the workspace, and use it the same way as above. The job ran for 10 minutes, killed itself, and didn't show any Error Code or Message. </li>
  <li>I ran the Spark Example job in Hue. I got the following message</li>
  </ol>
  
  <p>Error:</p>
  
  <pre><code>JA018
  Error Message   Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: Stream '/jars/oozie-examples.jar' was not found. at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:219) at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:106) at org.apache.spark.network.server.TransportChannelHandler.channelRead0(Tr
  </code></pre>
  
  <p><strong>My Questions</strong> :</p>
  
  <ol>
  <li>Should I only compile  the classes that I need and use Oozie ShareLibs? Does Oozie support Uber JARS in general?</li>
  <li>If I'm using Pig/Sqoop, Do I need to do the same ?</li>
  </ol>