title: Hadoop / HBASE - Can't configure regionserver with HDFS High Availability (Failover)
body: <p>I'm trying to build an Hadoop Architecture with failover functionnalities.
  My issue is I can't correctly configure RegionServer with HDFS HA. I have the following errors in the RegionServer log</p>
  
  <pre><code>java.io.IOException: Port 9000 specified in URI hdfs://HAcluster:9000 but host 'HAcluster' is a logical (HA) namenode and does not use port information.
  at org.apache.hadoop.hdfs.NameNodeProxies.getFailoverProxyProviderClass(NameNodeProxies.java:396)
  at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:134)
  at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:510)
  at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:453)
  at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:2508)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:2492)
  at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:62)
  at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2543)
  </code></pre>
  
  <p>Below my components : </p>
  
  <ul>
  <li>Hadoop : 2.7.1 </li>
  <li>HBASE : 0.98.12</li>
  <li>Zookeeper : 3.4.6 </li>
  <li>Java : JDK Oracle 1.7_75</li>
  </ul>
  
  <p>About the architecture, I have 6 VM :</p>
  
  <ul>
  <li>2 Masters with :
  
  <ul>
  <li>HDFS (NameNode)</li>
  <li>YARN (Ressource manager)</li>
  <li>HBASE (Master)</li>
  </ul></li>
  </ul>
  
  <p>One master is active, the other is in standby (in case the first one crashes). The standby master is just a replication of active master</p>
  
  <ul>
  <li>One slave with :
  
  <ul>
  <li>HDFS (DataNode)</li>
  <li>YARN (Node manager)</li>
  <li>HBASE (Region Server)</li>
  </ul></li>
  <li>Three Zookeepers on dedicated server</li>
  </ul>
  
  <p>Every component is in HA (High Availability) mode.To make it, I had to create logical cluster for HDFS &amp; YARN</p>
  
  <p><strong>Below the different files they may be useful to have a better understanding :</strong></p>
  
  <p><em>hdfs-site.xml (where  the HAcluster is defined) - same for the 3 servers except for some properties out of the HA availability scope</em></p>
  
  <pre><code>&lt;configuration&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.replication&lt;/name&gt;
      &lt;value&gt;1&lt;/value&gt;
      &lt;description&gt;the value is the number of the copy of the file in the file system&lt;/description&gt;
  &lt;/property&gt;
  &lt;!-- High Availability Hadoop --&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.nameservices&lt;/name&gt;
      &lt;value&gt;HAcluster&lt;/value&gt; &lt;!-- HAcluster is consisted of SUNRAY009IV06 = MASTER 1 and SUNRAY009IV07 = MASTER 2 --&gt;
      &lt;final&gt;true&lt;/final&gt;
      &lt;description&gt;The name of your cluster which consists of Master 1 and Master 2&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.ha.namenodes.HAcluster&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV06,SUNRAY009IV07&lt;/value&gt; &lt;!--SUNRAY009IV06 = MASTER 1, SUNRAY009IV07 = MASTER 2 --&gt;
      &lt;final&gt;true&lt;/final&gt;
      &lt;description&gt;The namenodes in your cluster&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.namenode.rpc-address.HAcluster.SUNRAY009IV06&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV06:9000&lt;/value&gt; &lt;!--SUNRAY009IV06 = MASTER 1 --&gt;
      &lt;description&gt;the RPC adress of your Master 1&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.namenode.rpc-address.HAcluster.SUNRAY009IV07&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV07:9000&lt;/value&gt; &lt;!--SUNRAY009IV07 = MASTER 2 --&gt;
      &lt;description&gt;the RPC adress of your Master 2&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.namenode.http-address.HAcluster.SUNRAY009IV06&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV06:50070&lt;/value&gt; &lt;!--SUNRAY009IV06 = MASTER 1 --&gt;
      &lt;description&gt;the HTTP adress of your Master 1&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.namenode.http-address.HAcluster.SUNRAY009IV07&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV07:50070&lt;/value&gt; &lt;!--SUNRAY009IV07 = MASTER 2 --&gt;
      &lt;description&gt;the HTTP adress of your Master 2&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.namenode.shared.edits.dir&lt;/name&gt;
      &lt;value&gt;qjournal://SUNRAY009IV06:8485;SUNRAY009IV07:8485;SUNRAY009IV08:8485/HAcluster&lt;/value&gt;
      &lt;!--SUNRAY009IV06 = MASTER 1, SUNRAY009IV07 = MASTER 2, SUNRAY009IV08 = SLAVE 1 --&gt;
      &lt;description&gt;the location of the shared storage directory&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.client.failover.proxy.provider.HAcluster&lt;/name&gt;
      &lt;value&gt;org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider&lt;/value&gt;
      &lt;description&gt;the Java class that HDFS clients use to contact the Active NameNode&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt; 
      &lt;name&gt;dfs.permissions&lt;/name&gt;
      &lt;value&gt;false&lt;/value&gt;
      &lt;description&gt;disable hdfs permissions&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.ha.automatic-failover.enabled&lt;/name&gt;
      &lt;value&gt;true&lt;/value&gt;
      &lt;description&gt;The backup is defined as automatic&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;ha.zookeeper.quorum&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV09:2181,SUNRAY009IV11:2181,SUNRAY009IV13:2181&lt;/value&gt;
      &lt;description&gt;The list of your Zookeeper servers in your Hadoop architecture&lt;/description&gt;
      &lt;!--SUNRAY009IV09 = ZOOKEEPER 1, SUNRAY009IV11 = ZOOKEEPER 2, SUNRAY009IV13 = ZOOKEEPER 3 --&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.ha.fencing.methods&lt;/name&gt;
      &lt;value&gt;sshfence&lt;/value&gt;
      &lt;description&gt; method which will be used to fence the Active NameNode during a failover. 
      sshfence = SSH to the Active NameNode and kill the process&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.ha.fencing.ssh.private-key-files&lt;/name&gt;
      &lt;value&gt;/home/hadoopuser/.ssh/id_rsa&lt;/value&gt;
      &lt;description&gt;List of SSH private key files&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.ha.fencing.ssh.connect-timeout&lt;/name&gt;
      &lt;value&gt;3000&lt;/value&gt;
      &lt;description&gt;timeout&lt;/description&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p></p>
  
  <p><em>yarn-site.xml - same for the 3 servers except for some properties out of the HA availability scope</em></p>
  
  <pre><code>&lt;configuration&gt;
  
  &lt;!-- Site specific YARN configuration properties --&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.hostname&lt;/name&gt;
      &lt;value&gt;HAyarn&lt;/value&gt;
      &lt;!--HAyarn is consisted of SUNRAY009IV06 = MASTER 1 and SUNRAY009IV07 = MASTER 2 --&gt;
      &lt;description&gt;The name of the Resource Manager&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.log-aggregation-enable&lt;/name&gt;
      &lt;value&gt;true&lt;/value&gt;
      &lt;description&gt;to enable YARN logs&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.nodemanager.remote-app-log-dir&lt;/name&gt;
      &lt;value&gt;/tmp/logs&lt;/value&gt;
      &lt;description&gt;Where to store logs in HDFS&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;
      &lt;value&gt;mapreduce_shuffle&lt;/value&gt;
      &lt;description&gt;shuffle service that needs to be set for Map Reduce to run&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.nodemanager.aux-services.mapreduce_shuffle.class&lt;/name&gt;
      &lt;value&gt;org.apache.hadoop.mapred.ShuffleHandler&lt;/value&gt;
      &lt;description&gt;mapreduce_shuffle service to implement&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;mapreduce.jobhistory.address&lt;/name&gt;
      &lt;value&gt;HAyarn:8031&lt;/value&gt;
      &lt;!--HAyarn is consisted of SUNRAY009IV06 = MASTER 1 and SUNRAY009IV07 = MASTER 2 --&gt;
      &lt;description&gt;host is the hostname of the resource manager and  the port is the port on which the NodeManagers contact the Resource Manage&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;!-- High Availability YARN --&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.ha.enabled&lt;/name&gt;
      &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.cluster-id&lt;/name&gt;
      &lt;value&gt;HAyarn&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.ha.rm-ids&lt;/name&gt;
      &lt;value&gt;rm1,rm2&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.hostname.rm1&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV06&lt;/value&gt;
      &lt;!--SUNRAY009IV06 = MASTER 1, SUNRAY009IV07 = MASTER 2--&gt;
      &lt;description&gt;The hostname of MASTER 1&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.hostname.rm2&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV07&lt;/value&gt;
      &lt;!--SUNRAY009IV06 = MASTER 1, SUNRAY009IV07 = MASTER 2--&gt;
      &lt;description&gt;The hostnameof MASTER 2&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.webapp.address.rm1&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV06:8088&lt;/value&gt;
      &lt;!--SUNRAY009IV06 = MASTER 1, SUNRAY009IV07 = MASTER 2--&gt;
      &lt;description&gt;The Web application address of MASTER 1&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.webapp.address.rm2&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV07:8088&lt;/value&gt;
      &lt;!--SUNRAY009IV06 = MASTER 1, SUNRAY009IV07 = MASTER 2--&gt;
      &lt;description&gt;The Web application address of MASTER 2&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.zk-address&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV09:2181,SUNRAY009IV11:2181,SUNRAY009IV13:2181&lt;/value&gt;
      &lt;description&gt;The list of your Zookeeper servers in your Hadoop architecture&lt;/description&gt;
      &lt;!--SUNRAY009IV09 = ZOOKEEPER 1, SUNRAY009IV11 = ZOOKEEPER 2, SUNRAY009IV13 = ZOOKEEPER 3 --&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.client.failover-proxy-provider.HAyarn&lt;/name&gt;
      &lt;value&gt;org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider&lt;/value&gt;
      &lt;description&gt;the class used for the YARN failover&lt;/description&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p></p>
  
  <p><em>hbase-site.xml (same in the 3 servers)</em></p>
  
  <pre><code>&lt;property&gt;
      &lt;name&gt;hbase.rootdir&lt;/name&gt;
      &lt;value&gt;hdfs://HAcluster/hbase&lt;/value&gt; &lt;!--HAcluster is consisted of SUNRAY009IV06 = MASTER 1 and SUNRAY009IV07 = MASTER 2 --&gt;
      &lt;description&gt;The directory shared by RegionServers (slaves)&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
      &lt;value&gt;true&lt;/value&gt;
      &lt;description&gt;The mode the cluster will be in&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.zookeeper.property.clientPort&lt;/name&gt;
      &lt;value&gt;2181&lt;/value&gt;
      &lt;description&gt;Property from ZooKeeper's config zoo.cfg. The port at which the clients will connect.&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
      &lt;value&gt;SUNRAY009IV09,SUNRAY009IV11,SUNRAY009IV13&lt;/value&gt;
      &lt;descrption&gt;The list of your Zookeeper servers in your Hadoop architecture&lt;/descrption&gt;
      &lt;!--SUNRAY009IV09 = ZOOKEEPER 1, SUNRAY009IV11 = ZOOKEEPER 2, SUNRAY009IV13 = ZOOKEEPER 3 --&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.zookeeper.property.dataDir&lt;/name&gt;
      &lt;value&gt;/home/zookeeper&lt;/value&gt;
      &lt;description&gt;Property from ZooKeeper's config zoo.cfg. The directory where the snapshot is stored.&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;zookeeper.znode.parent&lt;/name&gt;
      &lt;value&gt;/hbase&lt;/value&gt;
      &lt;description&gt;The root znode that will contain all the znodes created/used byHBase&lt;/description&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p><em>hbase-env.sh - only the useful part</em></p>
  
  <pre><code>#Tell HBase whether it should manage it's own instance of Zookeeper or not.
  export HBASE_MANAGES_ZK=false
  </code></pre>
  
  <p>Before posting it, I did researchs on Google. Nothing useful according me, so I did some tries :
  - I try to change the HBASE version. I downloaded the last one (0.98.17-hadoop2). No effect
  - I try to start from scratch which means : format HDFS, delete Zookeeper metadatas, delete znodes etc...
  - I try to replace hdfs://HAcluster/hbase by hdfs://MASTER1:9000/hbase on every server where there is HBASE. No effect.</p>
  
  <p>So I'm a bit lost because even with no logical cluster, I've still got the error.</p>
  
  <p>PS: All the rest works as expected : the datanode / nodemanager is connected to the active namenode /resourcemanager (checked with the web application)
  The HBASE master runs correctly too and the backup master is also taken in account (check too with the webapp)
  That's also why I don't understand I have this error</p>
  
  <p>I hope I gave you all the elements expected to understand correctly my issue</p>