title: Missing hive-site when using spark-submit YARN cluster mode
body: <p>Using HDP 2.5.3 and I've been trying to debug some YARN container classpath issues. </p>
  
  <p>Since HDP includes both Spark 1.6 and 2.0.0, there have been some conflicting versions</p>
  
  <p>Users I support are successfully able to use Spark2 with Hive queries in YARN <code>client</code> mode, but not from <code>cluster</code> mode they get errors about tables not found, or something like that because the Metastore connection isn't established. </p>
  
  <p>I am guessing that setting either <code>--driver-class-path /etc/spark2/conf:/etc/hive/conf</code> or passing <code>--files /etc/spark2/conf/hive-site.xml</code> after <code>spark-submit</code> would work, but why isn't <code>hive-site.xml</code> loaded already from the <code>conf</code> folder?</p>
  
  <p>Accoringing to <a href="https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_command-line-installation/content/configuring_spark.html" rel="nofollow noreferrer">Hortonworks docs</a>, says <code>hive-site</code> should be placed in <code>$SPARK_HOME/conf</code>, and it is...</p>
  
  <p>I see <code>hdfs-site.xml</code> and <code>core-site.xml</code>, and other files that are part of <code>HADOOP_CONF_DIR</code>, for example, and this is the from the YARN UI container info.  </p>
  
  <pre><code>2232355    4 drwx------   2 yarn     hadoop       4096 Aug  2 21:59 ./__spark_conf__
  2232379    4 -r-x------   1 yarn     hadoop       2358 Aug  2 21:59 ./__spark_conf__/topology_script.py
  2232381    8 -r-x------   1 yarn     hadoop       4676 Aug  2 21:59 ./__spark_conf__/yarn-env.sh
  2232392    4 -r-x------   1 yarn     hadoop        569 Aug  2 21:59 ./__spark_conf__/topology_mappings.data
  2232398    4 -r-x------   1 yarn     hadoop        945 Aug  2 21:59 ./__spark_conf__/taskcontroller.cfg
  2232356    4 -r-x------   1 yarn     hadoop        620 Aug  2 21:59 ./__spark_conf__/log4j.properties
  2232382   12 -r-x------   1 yarn     hadoop       8960 Aug  2 21:59 ./__spark_conf__/hdfs-site.xml
  2232371    4 -r-x------   1 yarn     hadoop       2090 Aug  2 21:59 ./__spark_conf__/hadoop-metrics2.properties
  2232387    4 -r-x------   1 yarn     hadoop        662 Aug  2 21:59 ./__spark_conf__/mapred-env.sh
  2232390    4 -r-x------   1 yarn     hadoop       1308 Aug  2 21:59 ./__spark_conf__/hadoop-policy.xml
  2232399    4 -r-x------   1 yarn     hadoop       1480 Aug  2 21:59 ./__spark_conf__/__spark_conf__.properties
  2232389    4 -r-x------   1 yarn     hadoop       1602 Aug  2 21:59 ./__spark_conf__/health_check
  2232385    4 -r-x------   1 yarn     hadoop        913 Aug  2 21:59 ./__spark_conf__/rack_topology.data
  2232377    4 -r-x------   1 yarn     hadoop       1484 Aug  2 21:59 ./__spark_conf__/ranger-hdfs-audit.xml
  2232383    4 -r-x------   1 yarn     hadoop       1020 Aug  2 21:59 ./__spark_conf__/commons-logging.properties
  2232357    8 -r-x------   1 yarn     hadoop       5721 Aug  2 21:59 ./__spark_conf__/hadoop-env.sh
  2232391    4 -r-x------   1 yarn     hadoop        281 Aug  2 21:59 ./__spark_conf__/slaves
  2232373    8 -r-x------   1 yarn     hadoop       6407 Aug  2 21:59 ./__spark_conf__/core-site.xml
  2232393    4 -r-x------   1 yarn     hadoop        812 Aug  2 21:59 ./__spark_conf__/rack-topology.sh
  2232394    4 -r-x------   1 yarn     hadoop       1044 Aug  2 21:59 ./__spark_conf__/ranger-hdfs-security.xml
  2232395    8 -r-x------   1 yarn     hadoop       4956 Aug  2 21:59 ./__spark_conf__/metrics.properties
  2232386    8 -r-x------   1 yarn     hadoop       4221 Aug  2 21:59 ./__spark_conf__/task-log4j.properties
  2232380    4 -r-x------   1 yarn     hadoop         64 Aug  2 21:59 ./__spark_conf__/ranger-security.xml
  2232372   20 -r-x------   1 yarn     hadoop      19975 Aug  2 21:59 ./__spark_conf__/yarn-site.xml
  2232397    4 -r-x------   1 yarn     hadoop       1006 Aug  2 21:59 ./__spark_conf__/ranger-policymgr-ssl.xml
  2232374    4 -r-x------   1 yarn     hadoop         29 Aug  2 21:59 ./__spark_conf__/yarn.exclude
  2232384    4 -r-x------   1 yarn     hadoop       1606 Aug  2 21:59 ./__spark_conf__/container-executor.cfg
  2232396    4 -r-x------   1 yarn     hadoop       1000 Aug  2 21:59 ./__spark_conf__/ssl-server.xml
  2232375    4 -r-x------   1 yarn     hadoop          1 Aug  2 21:59 ./__spark_conf__/dfs.exclude
  2232359    8 -r-x------   1 yarn     hadoop       7660 Aug  2 21:59 ./__spark_conf__/mapred-site.xml
  2232378   16 -r-x------   1 yarn     hadoop      14474 Aug  2 21:59 ./__spark_conf__/capacity-scheduler.xml
  2232376    4 -r-x------   1 yarn     hadoop        884 Aug  2 21:59 ./__spark_conf__/ssl-client.xml
  </code></pre>
  
  <p>As you might see, <code>hive-site</code> is not there, even though I definitely have <code>conf/hive-site.xml</code> for spark-submit to take</p>
  
  <pre><code>[spark@asthad006 conf]$ pwd &amp;&amp; ls -l
  /usr/hdp/2.5.3.0-37/spark2/conf
  total 32
  -rw-r--r-- 1 spark spark   742 Mar  6 15:20 hive-site.xml
  -rw-r--r-- 1 spark spark   620 Mar  6 15:20 log4j.properties
  -rw-r--r-- 1 spark spark  4956 Mar  6 15:20 metrics.properties
  -rw-r--r-- 1 spark spark   824 Aug  2 22:24 spark-defaults.conf
  -rw-r--r-- 1 spark spark  1820 Aug  2 22:24 spark-env.sh
  -rwxr-xr-x 1 spark spark   244 Mar  6 15:20 spark-thrift-fairscheduler.xml
  -rw-r--r-- 1 hive  hadoop  918 Aug  2 22:24 spark-thrift-sparkconf.conf
  </code></pre>
  
  <p>So, I don't think I am supposed to place hive-site in <code>HADOOP_CONF_DIR</code> as <code>HIVE_CONF_DIR</code> is separated, but my question is that how do we get Spark2 to pick up the <code>hive-site.xml</code> without needing to manually pass it as a parameter at runtime?</p>
  
  <p><strong>EDIT</strong> Naturally, since I'm on HDP I am using Ambari. The previous cluster admin has installed Spark2 clients on all of the machines, so all of the YARN NodeManagers that could be potential Spark drivers should have the same config files </p>