title: Accessing HDFS HA from spark job (UnknownHostException error)
body: <p>I have Apache Mesos 0.22.1 cluster (3 masters &amp; 5 slaves), running Cloudera HDFS  (2.5.0-cdh5.3.1) in HA configuration and Spark 1.5.1 framework.</p>
  
  <p>When I try to spark-submit compiled HdfsTest.scala example app (from Spark 1.5.1 sources) - it fails with <code>java.lang.IllegalArgumentException: java.net.UnknownHostException: hdfs</code> error in executor logs. This error is only observed when I pass HDFS HA Path as an argument <code>hdfs://hdfs/&lt;file&gt;</code>, when I pass <code>hdfs://namenode1.hdfs.mesos:50071/tesfile</code> - everything works fine.</p>
  
  <p>What I've found after enabling TRACE logging is that Spark driver actually reads <code>hdfs://hdfs</code> URL correctly, but Spark executor - doesn't.</p>
  
  <p>My Scala app code:</p>
  
  <pre><code>import org.apache.spark._
  object HdfsTest {
    def main(args: Array[String]) {
      val sparkConf = new SparkConf().setAppName("HdfsTest")
      val sc = new SparkContext(sparkConf)
      val file = sc.textFile(args(0))
      val mapped = file.map(s =&gt; s.length).cache()
      for (iter &lt;- 1 to 10) {
        val start = System.currentTimeMillis()
        for (x &lt;- mapped) { x + 2 }
        val end = System.currentTimeMillis()
        println("Iteration " + iter + " took " + (end-start) + " ms")
      }
      sc.stop()
     }
    }
  </code></pre>
  
  <p>I compile this code and submit jar file to Spark in cluster mode:</p>
  
  <pre><code>/opt/spark/bin/spark-submit --deploy-mode cluster --class com.cisco.hdfs.HdfsTest http://1.2.3.4/HdfsTest-0.0.1.jar hdfs://hdfs/testfile
  </code></pre>
  
  <p>My spark-defaults.conf file:</p>
  
  <pre><code>spark.master                     spark://1.2.3.4:7077
  spark.eventLog.enabled           true
  spark.driver.memory              1g
  </code></pre>
  
  <p>My spark-env.sh file:</p>
  
  <pre><code>export HADOOP_HOME=/opt/spark
  export HADOOP_CONF_DIR=/opt/spark/conf
  </code></pre>
  
  <p>I have spark deployed on each slave in /opt/spark directory.</p>
  
  <p>I can accesses HDFS using "hdfs dfs -ls hdfs://hdfs/" command in console, without the need to specify active namenode address and port. </p>
  
  <pre><code>core-site.xml:
  ----------------------------------------------------------------------
  &lt;configuration&gt;
   &lt;property&gt;
    &lt;name&gt;fs.default.name&lt;/name&gt;
    &lt;value&gt;hdfs://hdfs&lt;/value&gt;
   &lt;/property&gt;
  &lt;/configuration&gt;
  
  hdfs-site.xml:
  ----------------------------------------------------------------------
  &lt;configuration&gt;
   &lt;property&gt;
    &lt;name&gt;dfs.ha.automatic-failover.enabled&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.nameservice.id&lt;/name&gt;
    &lt;value&gt;hdfs&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.nameservices&lt;/name&gt;
    &lt;value&gt;hdfs&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.ha.namenodes.hdfs&lt;/name&gt;
    &lt;value&gt;nn1,nn2&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.rpc-address.hdfs.nn1&lt;/name&gt;
    &lt;value&gt;namenode1.hdfs.mesos:50071&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.http-address.hdfs.nn1&lt;/name&gt;
    &lt;value&gt;namenode1.hdfs.mesos:50070&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.rpc-address.hdfs.nn2&lt;/name&gt;
    &lt;value&gt;namenode2.hdfs.mesos:50071&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.http-address.hdfs.nn2&lt;/name&gt;
    &lt;value&gt;namenode2.hdfs.mesos:50070&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.client.failover.proxy.provider.hdfs&lt;/name&gt;
    &lt;value&gt;org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider      &lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.shared.edits.dir&lt;/name&gt;
       &lt;value&gt;qjournal://journalnode1.hdfs.mesos:8485;journalnode2.hdfs.mesos:8485;journalnode3.hdfs.mesos:8485/hdfs&lt;/value&gt;
     &lt;/property&gt;
  
   &lt;property&gt;
     &lt;name&gt;ha.zookeeper.quorum&lt;/name&gt;
     &lt;value&gt;master.mesos:2181&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.journalnode.edits.dir&lt;/name&gt;
    &lt;value&gt;/var/lib/hdfs/data/jn&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
     &lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;
     &lt;value&gt;file:///var/lib/hdfs/data/name&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
     &lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;
     &lt;value&gt;file:///var/lib/hdfs/data/data&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.ha.fencing.methods&lt;/name&gt;
    &lt;value&gt;shell(/bin/true)&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.permissions&lt;/name&gt;
    &lt;value&gt;false&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.datanode.du.reserved&lt;/name&gt;
    &lt;value&gt;10485760&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.datanode.balance.bandwidthPerSec&lt;/name&gt;
    &lt;value&gt;41943040&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
     &lt;name&gt;dfs.namenode.safemode.threshold-pct&lt;/name&gt;
     &lt;value&gt;0.90&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.heartbeat.recheck-interval&lt;/name&gt;
    &lt;value&gt;60000&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.datanode.handler.count&lt;/name&gt;
    &lt;value&gt;10&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.handler.count&lt;/name&gt;
    &lt;value&gt;20&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.image.compress&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.image.compression.codec&lt;/name&gt;
    &lt;value&gt;org.apache.hadoop.io.compress.SnappyCodec&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.invalidate.work.pct.per.iteration&lt;/name&gt;
    &lt;value&gt;0.35f&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.replication.work.multiplier.per.iteration&lt;/name&gt;
    &lt;value&gt;4&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.namenode.datanode.registration.ip-hostname-check&lt;/name&gt;
    &lt;value&gt;false&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
     &lt;name&gt;dfs.client.read.shortcircuit&lt;/name&gt;
     &lt;value&gt;true&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.client.read.shortcircuit.streams.cache.size&lt;/name&gt;
    &lt;value&gt;1000&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.client.read.shortcircuit.streams.cache.size.expiry.ms&lt;/name&gt;
     &lt;value&gt;1000&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;dfs.domain.socket.path&lt;/name&gt;
    &lt;value&gt;/var/run/hadoop-hdfs/dn._PORT&lt;/value&gt;
   &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>