title: Oozie Hadoop Streaming
body: <p>I am trying to write a simple map only hadoop streaming job reading data from hdfs and pushing it to vertica.</p>
  
  <p>I have written few shell scripts as below</p>
  
  <p>load.sh</p>
  
  <pre><code>  hadoop jar hadoop-streaming-2.7.3.2.5.3.0-37.jar -input $INPUT_DIR -mapper /user/oozie/adhoc_data_load/scripts/export.sh -output $OUTPUT_DIR
  </code></pre>
  
  <p>export.sh</p>
  
  <pre><code> ./vsql -c "copy $TABLE from stdin delimiter E'\t' direct null '\\N';" -U $DBUSER -w $DBPWD -h $DBHOST -p $DBPORT
  </code></pre>
  
  <p>Workflow:</p>
  
  <pre><code>&lt;action name="loadToVertica"&gt;
      &lt;shell xmlns="uri:oozie:shell-action:0.1"&gt;
              &lt;job-tracker&gt;${jobTracker}&lt;/job-tracker&gt;
              &lt;name-node&gt;${nameNode}&lt;/name-node&gt;
              &lt;exec&gt;loadVertica.sh&lt;/exec&gt;
              &lt;argument&gt;${STREAMING_JAR_PATH}&lt;/argument&gt;
              &lt;argument&gt;${nameNode}/user/oozie/optus_adhoc_data/${exportDataDate}&lt;/argument&gt;
              &lt;argument&gt;${TABLE_NAME_VERTICA}&lt;/argument&gt;
              &lt;argument&gt;${dbHost}&lt;/argument&gt;
              &lt;argument&gt;${dbName}&lt;/argument&gt;
              &lt;argument&gt;${dbPassword}&lt;/argument&gt;
              &lt;argument&gt;${dbPort}&lt;/argument&gt;
              &lt;argument&gt;${nameNode}/user/oozie/optus_adhoc_data/output/${exportDataDate}&lt;/argument&gt;
              &lt;argument&gt;vsql,export.sh&lt;/argument&gt;
              &lt;file&gt;${nameNode}/user/oozie/adhoc_data_load/scripts/loadVertica.sh#loadVertica.sh&lt;/file&gt;
              &lt;file&gt;${wfsBasePath}/libs/${STREAMING_JAR_PATH}#${STREAMING_JAR_PATH}&lt;/file&gt;
              &lt;file&gt;${wfsBasePath}/config/vsql#vsql&lt;/file&gt;
              &lt;file&gt;${wfsBasePath}/scripts/export.sh#export.sh&lt;/file&gt;
              &lt;capture-output/&gt;
          &lt;/shell&gt;
          &lt;ok to="end"/&gt;
         &lt;error to="end"/&gt;
      &lt;/action&gt;
  </code></pre>
  
  <p>Running it with Oozie user getting below exception:</p>
  
  <pre><code>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=yarn, access=WRITE, inode="/user/yarn/.staging":hdfs:hdfs:drwxr-xr-x
          at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
  </code></pre>
  
  <p>Solution:</p>
  
  <p>Add in workflow :</p>
  
  <p>HADOOP_USER_NAME=${wf:user()}</p>