title: Spark 2 action fails in oozie workflow: ./assembly/target/scala-2.11/jars' does not exist; make sure Spark is built
body: <p>I am able to run my spark job with spark-submit, but am not able to run it from an oozie workflow.</p>
  
  <p>I get the following output error:</p>
  
  <pre><code>`java.lang.IllegalStateException: Library directory '/mnt/resource/hadoop/yarn/local/usercache/sshuser/appcache/application_1521646255340_0010/container_1521646255340_0010_01_000002/./assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
      at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:260)
      at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:359)
      at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
      at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:556)
      at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:845)
      at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:170)
      at org.apache.spark.deploy.yarn.Client.run(Client.scala:1174)
      at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1233)
      at org.apache.spark.deploy.yarn.Client.main(Client.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:312)
      at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:233)
      at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:58)
      at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:62)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)`
  </code></pre>
  
  <p>I have followed these instructions: <a href="https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_spark-component-guide/content/ch_oozie-spark-action.html#spark-config-oozie-spark2" rel="nofollow noreferrer">https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_spark-component-guide/content/ch_oozie-spark-action.html#spark-config-oozie-spark2</a></p>