title: How to configure SparkContext for a HA enabled Cluster
body: <p>When I am trying to run the spark application in YARN mode using the HDFS file system it works fine when I provide the below properties.</p>
  
  <pre><code>sparkConf.set("spark.hadoop.yarn.resourcemanager.hostname",resourcemanagerHostname);
  sparkConf.set("spark.hadoop.yarn.resourcemanager.address",resourcemanagerAddress);
  sparkConf.set("spark.yarn.stagingDir",stagingDirectory );
  </code></pre>
  
  <p>But the problems with this are:</p>
  
  <ol>
  <li>Since my HDFS is NamdeNode HA enabled it won't work when I provide spark.yarn.stagingDir the commons URL of hdfs</li>
  </ol>
  
  <p>E.g. <code>hdfs://hdcluster/user/tmp/</code> gives an error that says:</p>
  
  <blockquote>
    <p>has unknown host hdcluster</p>
  </blockquote>
  
  <p>But it works fine when I give the URL as <code>hdfs://&lt;ActiveNameNode&gt;/user/tmp/</code>, but we don't know in advance which will be active so how do I resolve this?</p>
  
  <p>And few things I have noticed are SparkContext takes the Hadoop configuration but SparkConfiguration class won't have any methods to accepts Hadoop configuration.</p>
  
  <ol start="2">
  <li>How do I provide the resource Manager address when Resource Manager are running in HA?</li>
  </ol>