title: Not All Meta Regions Online Exception in hbase
body: <p>I have a 4 node hadoop,hbase cluster of which 1 serves as hadoopmaster &amp; hbasemaster.
   The rest three server as datanode &amp; regionserver &amp; zookeeper.
  Recently one of my machine running the (datanode,regionserver &amp; zookeeper) got crashed and the system is not rebooting. But still my hadoop-cluster is working fine but the <strong>hbase cluster</strong> is showing problem as no regions are online and i am not able to view hbase tables.. 
  <strong>NOTE</strong>: the node's name which went down is <strong>hadoopslave3</strong>
  for command:</p>
  
  <p>In <code>hbase shell</code> when i tried to create a table it gave an error:</p>
  
  <pre><code>hbase(main):001:0&gt; create 'USERS','COL-FAMILY1'
  
  ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms)
  </code></pre>
  
  <p>for command:</p>
  
  <pre><code>$hbase hbck 
  </code></pre>
  
  <p>it shows:</p>
  
  <pre><code>12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hbase/lib/native/Linux-amd64-64
  12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Client environment:java.compiler=&lt;NA&gt;
  12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
  12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
  12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-33-server
  12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Client environment:user.name=ocpe
  12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/ocpe
  12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Client environment:user.dir=/var/log/hbase
  12/08/27 11:10:20 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoopslave2:2181,hbasemaster:2181,hadoopslave3:2181 sessionTimeout=180000 watcher=hconnection
  12/08/27 11:10:20 INFO zookeeper.ClientCnxn: Opening socket connection to server hbasemaster/10.68.210.71:2181
  12/08/27 11:10:20 INFO zookeeper.ClientCnxn: Socket connection established to hbasemaster/10.68.210.71:2181, initiating session
  12/08/27 11:10:20 INFO zookeeper.ClientCnxn: Session establishment complete on server hbasemaster/10.68.210.71:2181, sessionid = 0x39668f686e0006, negotiated timeout = 180000
  Version: 0.90.4-cdh3u3
  12/08/27 11:11:21 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@69267649; hsa=null
  ERROR: Root Region or some of its attributes are null.
  ERROR: Encountered fatal error. Exiting...
  </code></pre>
  
  <p>and master log shows:</p>
  
  <pre><code>2012-08-27 11:00:41,329 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoopslave3/10.68.210.58:2181
  2012-08-27 11:00:42,832 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
  java.net.NoRouteToHostException: No route to host
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146)
  .
  .
  .
  .
  .
  2012-08-27 11:00:43,610 DEBUG org.apache.hadoop.hbase.master.HMaster: Started service threads
  2012-08-27 11:00:44,357 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=hadoopslave2,60020,1346045456923, regionCount=0, userLoad=false
  2012-08-27 11:00:45,110 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
  2012-08-27 11:00:45,703 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=hadoopslave1,60020,1346065230110, regionCount=0, userLoad=false
  2012-08-27 11:00:46,611 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=2
  2012-08-27 11:00:48,111 INFO org.apache.hadoop.hbase.master.ServerManager: Exiting wait on regionserver(s) to checkin; count=2, stopped=false, count of regions out on cluster=0
  2012-08-27 11:00:48,118 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://hadoopmaster:54310/hbase2/.logs/hadoopslave1,60020,1346065230110 belongs to an existing region server
  2012-08-27 11:00:48,118 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://hadoopmaster:54310/hbase2/.logs/hadoopslave2,60020,1346045456923 belongs to an existing region server
  2012-08-27 11:00:48,118 INFO org.apache.hadoop.hbase.master.MasterFileSystem: No logs to split
  2012-08-27 11:00:48,125 INFO org.apache.hadoop.hbase.master.AssignmentManager: Processing region -ROOT-,,0.70236052 in state RS_ZK_REGION_OPENED
  2012-08-27 11:00:48,128 INFO org.apache.hadoop.hbase.master.AssignmentManager: Failed to find hadoopslave2,60020,1345789680868 in list of online servers; skipping registration of open of -ROOT-,,0.70236052
  2012-08-27 11:00:48,128 INFO org.apache.hadoop.hbase.master.AssignmentManager: Waiting on 70236052/-ROOT-
  2012-08-27 11:00:53,213 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out:  -ROOT-,,0.70236052 state=OPEN, ts=1345790880487, server=hadoopslave2,60020,1345789680868
  2012-08-27 11:00:53,213 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPEN for too long, we don't know where region was opened so can't do anything
  2012-08-27 11:01:43,242 WARN org.apache.hadoop.hbase.master.TimeToLiveLogCleaner: Found a log newer than current time, probably a clock skew
  2012-08-27 11:02:43,234 WARN org.apache.hadoop.hbase.master.TimeToLiveLogCleaner: Found a log newer than current time, probably a clock skew
  </code></pre>
  
  <p>and one of the regionserver(hadoopslave2) log shows:</p>
  
  <pre><code>2012-08-27 16:30:30,388 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoopslave2:2181,hbasemaster:2181,hadoopslave3:2181 sessionTimeout=180000 watcher=regionserver:60020
  2012-08-27 16:30:30,418 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoopslave3/10.68.210.58:2181
  2012-08-27 16:30:30,533 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver60020
  2012-08-27 16:30:33,426 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
  java.net.NoRouteToHostException: No route to host
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146)
  .
  .
  .
  .
  .
  .
  .
  2012-08-27 16:30:37,649 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: starting
  2012-08-27 16:30:37,649 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: starting
  2012-08-27 16:30:37,649 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: starting
  2012-08-27 16:30:37,653 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: starting
  2012-08-27 16:30:37,653 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Serving as hadoopslave1,60020,1346065230110, RPC listening on /10.68.210.54:60020, sessionid=0x139668faf560000
  2012-08-27 16:30:37,655 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker hadoopslave1,60020,1346065230110 starting
  2012-08-27 16:30:37,665 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Allocating LruBlockCache with maximum size 199.2m
  2012-08-27 16:30:38,211 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: HDFS pipeline error detected. Found 1 replicas but expecting 3 replicas.  Requesting close of hlog.
  </code></pre>