title: HBase Error IllegalStateException when starting Master: hsync
body: <p>I'm trying to install HBase on a hadoop cluster and can't figure out why the HMaster fails to initiate when called from start-hbase.sh. The log files indicate an issue with hsync. I have confirmed that zookeeper is running correctly in distributed mode,and I have not had any issues working with the hadoop cluster through spark. When attempting to start HBase the region servers start on all data nodes.</p>
  <p>I have hadoop version 3.0.0, zookeeper 3.4.11 and hbase 2.0.0 beta 1.</p>
  <p>I have cleared out the zookeeper hbase node and hdfs hbase directory and restarted with no luck. I have the myid zookeeper files stored in the same directory as the .pid files for zookeeper and hbase region servers /usr/local/zookeeper/data/</p>
  <p>Below I have an excerpt of the hbase master log file pertaining to the hsync exception.</p>
  <pre><code>2018-02-08 17:26:54,786 ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Failed to become active master
  java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
      at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036)
      at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532)
      at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232)
      at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145)
      at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837)
      at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
      at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547)
      at java.lang.Thread.run(Thread.java:748)
  2018-02-08 17:26:54,787 ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Master server abort: loaded coprocessors are: []
  2018-02-08 17:26:54,787 ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Unhandled exception. Starting shutdown.
  java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
      at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036)
      at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532)
      at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232)
      at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145)
      at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837)
      at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
      at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  <p>Below is the hbase-site.xml file</p>
  <pre><code>&lt;property&gt;
      &lt;name&gt;hbase.master&lt;/name&gt;
      &lt;value&gt;localhost:60000&lt;/value&gt;
      &lt;description&gt;The host and port that the HBase master runs at.A value of 'lo$
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.zookeeper.property.clientPort&lt;/name&gt;
      &lt;value&gt;2181&lt;/value&gt;
      &lt;description&gt;Property from ZooKeeper's config zoo.cfg.The port at which the$
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
      &lt;value&gt;MasterNode,Slave1,Slave2&lt;/value&gt;
      &lt;description&gt;Comma separated list of servers in the ZooKeeper Quorum.For ex$
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.rootdir&lt;/name&gt;
      &lt;value&gt;hdfs://MasterNode:9000/hbase&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.tmp.dir&lt;/name&gt;
      &lt;value&gt;/usr/local/zookeeper/data&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.zookeeper.property.dataDir&lt;/name&gt;
      &lt;value&gt;/usr/local/zookeeper/data&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
      &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;zookeeper.znode.parent&lt;/name&gt;
      &lt;value&gt;/hbase&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  <p>Below are the alterations made to the hbase-env.sh file settings</p>
  <pre><code>export JAVA_HOME=/usr/lib/jvm/java-8-oracle
  export HBASE_HOME=/usr/local/hadoop/HBase
  export HADOOP_HOME=/usr/local/hadoop
  export HBASE_MANAGES_ZK=false
  export HBASE_PID_DIR=/usr/local/zookeeper/data
  </code></pre>
  <p>Below is the core-site.xml</p>
  <pre><code>&lt;property&gt;
      &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;
      &lt;value&gt;/app/hadoop/tmp&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;fs.default.name&lt;/name&gt;
      &lt;value&gt;hdfs://MasterNode:9000&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  <p>Below is the zoo.cfg</p>
  <pre><code>tickTime=2000
  dataDir=/usr/local/zookeeper/data
  dataLogDir=/usr/local/zookeeper/log
  clientPort=2181
  initLimit=5
  syncLimit=2
  server.1=MasterNode:2888:3888
  server.2=Slave1:2888:3888
  server.3=Slave2:2888:3888
  </code></pre>
  <p>Below is the .bashrc settings related to the hadoop setup</p>
  <pre><code>export HADOOP_HOME=/usr/local/hadoop
  export HADOOP_MAPRED_HOME=$HADOOP_HOME
  export HADOOP_PREFIX=/usr/local/hadoop
  export HADOOP_COMMON_HOME=$HADOOP_HOME
  export HADOOP_HDFS_HOME=$HADOOP_HOME
  export YARN_HOME=$HADOOP_HOME
  export HADOOP_CONFDIR=$HADOOP_HOME/etc/hadoop
  export HADOOP_OPTS=&quot;-Djava.library.path=$HADOOP_PREFIX/lib&quot;
  export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
  export PATH=$PATH:$HADOOP_HOME/
  export PATH=$PATH:$HADOOP_HOME/bin
  export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
  export JAVA_HOME=/usr/lib/jvm/java-8-oracle
  export HBASE_HOME=/usr/local/hadoop/HBase
  export PATH=$PATH:$HBASE_HOME/bin
  </code></pre>