title: Running Spark2 from Oozie (CDH)
body: <p>I am attempting to run a spark job (using spark2-submit) from Oozie, so this job can be run on a schedule.</p>
  
  <p>The job runs just fine when running we run the shell script from command-line under our service account (not Yarn). When we run it as a Oozie Workflow the following happens: </p>
  
  <pre><code>17/11/16 12:03:55 ERROR spark.SparkContext: Error initializing SparkContext.
  org.apache.hadoop.security.AccessControlException: Permission denied: 
  user=yarn, access=WRITE, inode="/user":hdfs:supergroup:drwxrwxr-x
  </code></pre>
  
  <p>Oozie is running the job as the user Yarn. IT has denied us any ability to change Yarn's permissions in HDFS, and there is not a single reference to the <code>user</code> directory in the Spark script. We have attempted to ssh into the server - though this doesn't work - we have to ssh out of our worker nodes, onto the master. </p>
  
  <p>The shell script:</p>
  
  <pre><code>spark2-submit --name "SparkRunner" --master yarn --deploy-mode client --class org.package-name.Runner  hdfs://manager-node-hdfs/Analytics/Spark_jars/SparkRunner.jar
  </code></pre>
  
  <p>Any help would be appreciated.</p>