title: Getting HADOOP_HOME not found when running spark application in yarn cluster mode
body: <p>I am trying to run an application in yarn cluster mode. Here are setting of the shell script:</p>
  
  <pre><code>spark-submit --class "com.Myclass"  \
  --num-executors 2 \
   --executor-cores 2 \
   --master yarn \
   --supervise \
   --deploy-mode cluster \
  ../target/ \
  </code></pre>
  
  <p>Further I am getting following error. here are ERROR DETAILS FROM YARN LOGS APPLICATIONID</p>
  
  <pre><code>INFO : org.apache.spark.deploy.yarn.ApplicationMaster - Registered signal handlers for [TERM, HUP, INT]
  DEBUG: org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory
  java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
      at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:307)
      at org.apache.hadoop.util.Shell.&lt;clinit&gt;(Shell.java:332)
      at org.apache.hadoop.util.StringUtils.&lt;clinit&gt;(StringUtils.java:79)
      at org.apache.hadoop.yarn.conf.YarnConfiguration.&lt;clinit&gt;(YarnConfiguration.java:590)
      at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.newConfiguration(YarnSparkHadoopUtil.scala:62)
      at org.apache.spark.deploy.SparkHadoopUtil.&lt;init&gt;(SparkHadoopUtil.scala:52)
      at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.&lt;init&gt;(YarnSparkHadoopUtil.scala:47)
  </code></pre>
  
  <p>I tried modifying spark-env.sh like following and I see Hadoop_Home logged but still getting above error. Modified and added following entries to spark-env.sh </p>
  
  <pre><code>export HADOOP_HOME="/usr/lib/hadoop"
  echo "&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp; HADOOP HOME " 
  echo "$HADOOP_HOME"
  export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"
  echo "&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp; HADOOP_CONF_DIR " 
  echo "$HADOOP_CONF_DIR"
  </code></pre>
  
  <p>I see hadoop home logged when I run the spark-submit but still it complains about the the hadoop-home.</p>