title: How to properly configure HDFS high availability using Zookeeper?
body: <p>I'm a student of big data. I'm coming to you today for a question about the high availability of HDFS using Zookeeper. I am aware that there has already been bcp of topic dealing with this subject, I have read a lot of them already. It's already been 15 days that I've been browsing the forums without finding what I'm looking for (maybe I'm not looking in the right place too ;-) )</p>
  
  <p>I have followed the procedure three times here: <a href="https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html" rel="nofollow noreferrer">https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html</a>.</p>
  
  <p>I may have done everything right, but when I kill one of my namenodes, none of them take over.</p>
  
  <p>My architecture is as follows:
  - 5 VM
  - VM 1,3 and 5 are namenodes
  - VMs 1 to 5 are datanodes.</p>
  
  <p>I launched my journalnodes, I started my DFSZKFailoverController, I formatted my first namenode, I copied with -bootstrapStandby the configuration of my first namenode to the 2 others and I started my cluster.</p>
  
  <p>Despite all this and no obvious problems in the ZKFC and namenode logs, I can't get a namenode to take over a dying namenode.</p>
  
  <p>Does anyone have any idea how to help me?</p>
  
  <p>Many thanks for your help :)</p>
  
  <p><strong>zoo.cfg</strong></p>
  
  <pre><code># The number of milliseconds of each tick
  tickTime=2000
  # The number of ticks that the initial 
  # synchronization phase can take
  initLimit=5
  # The number of ticks that can pass between 
  # sending a request and getting an acknowledgement
  syncLimit=2
  # the directory where the snapshot is stored.
  # do not use /tmp for storage, /tmp here is just 
  # example sakes.
  dataDir=/home/zookeeper/zoo
  # the port at which the clients will connect
  clientPort=2181
  # the maximum number of client connections.
  # increase this if you need to handle more clients
  maxClientCnxns=60
  #
  # Be sure to read the maintenance section of the 
  # administrator guide before turning on autopurge.
  #
  # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
  #
  # The number of snapshots to retain in dataDir
  #autopurge.snapRetainCount=3
  # Purge task interval in hours
  # Set to "0" to disable auto purge feature
  #autopurge.purgeInterval=1
  
  ## Metrics Providers
  #
  # https://prometheus.io Metrics Exporter
  #metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
  #metricsProvider.httpPort=7000
  #metricsProvider.exportJvmInfo=true
  
  admin.serverPort=7979
  
  server.1=10.10.10.15:2888:3888
  server.2=10.10.10.16:2888:3888
  server.3=10.10.10.17:2888:3888
  server.4=10.10.10.18:2888:3888
  server.5=10.10.10.19:2888:3888
  </code></pre>
  
  <p><strong>core-site.xml</strong></p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
  &lt;!--
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at
  
      http://www.apache.org/licenses/LICENSE-2.0
  
    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License. See accompanying LICENSE file.
  --&gt;
  
  &lt;!-- Put site-specific property overrides in this file. --&gt;
  
  &lt;configuration&gt;
  
      &lt;!-- default configuration --&gt;
      &lt;property&gt;
          &lt;name&gt;fs.default.name&lt;/name&gt;
          &lt;value&gt;hdfs://my-cluster&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;io.file.buffer.size&lt;/name&gt;
          &lt;value&gt;131072&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;!-- zookeeper configuration --&gt;
      &lt;property&gt;
          &lt;name&gt;ha.zookeeper.quorum&lt;/name&gt;
          &lt;value&gt;10.10.10.15:2181,10.10.10.16:2181,10.10.10.17:2181,10.10.10.18:2181,10.10.10.19:2181&lt;/value&gt;
      &lt;/property&gt;
  
  &lt;/configuration&gt;
  </code></pre>
  
  <p><strong>hdfs-site.xml</strong></p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
  &lt;configuration&gt;
  
      &lt;!-- cluster configuration --&gt;
      &lt;property&gt;
          &lt;name&gt;dfs.nameservices&lt;/name&gt;
          &lt;value&gt;my-cluster&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;!-- namenode configuration --&gt;
      &lt;property&gt;
          &lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;
          &lt;value&gt;/home/hdfs/data/nameNode&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;!-- datanode configuration --&gt;
      &lt;property&gt;
          &lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;
          &lt;value&gt;/home/hdfs/data/dataNode&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;!-- secondary namenode configuration --&gt;
      &lt;property&gt;
          &lt;name&gt;dfs.namenode.checkpoint.dir&lt;/name&gt;
          &lt;value&gt;/home/hdfs/data/secondaryNameNode&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;!-- replication factor --&gt;
      &lt;property&gt;
          &lt;name&gt;dfs.replication&lt;/name&gt;
          &lt;value&gt;3&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;!-- webhdfs connector --&gt;
      &lt;property&gt; 
          &lt;name&gt;dfs.webhdfs.enabled&lt;/name&gt; 
          &lt;value&gt;true&lt;/value&gt; 
      &lt;/property&gt; 
  
      &lt;!-- high-availability configuration --&gt;
      &lt;property&gt;
          &lt;name&gt;dfs.ha.namenodes.my-cluster&lt;/name&gt;
          &lt;value&gt;nn1,nn2,nn3&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.namenode.rpc-address.my-cluster.nn1&lt;/name&gt;
          &lt;value&gt;10.10.10.15:9000&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.namenode.rpc-address.my-cluster.nn2&lt;/name&gt;
          &lt;value&gt;10.10.10.19:9000&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.namenode.rpc-address.my-cluster.nn3&lt;/name&gt;
          &lt;value&gt;10.10.10.17:9000&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.namenode.http-address.my-cluster.nn1&lt;/name&gt;
          &lt;value&gt;10.10.10.15:9870&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.namenode.http-address.my-cluster.nn2&lt;/name&gt;
          &lt;value&gt;10.10.10.19:9870&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.namenode.http-address.my-cluster.nn3&lt;/name&gt;
          &lt;value&gt;10.10.10.17:9870&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.namenode.shared.edits.dir&lt;/name&gt;
          &lt;value&gt;qjournal://10.10.10.15:8485;10.10.10.19:8485;10.10.10.17:8485/my-cluster&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.journalnode.edits.dir&lt;/name&gt;
          &lt;value&gt;/home/hdfs/data/journalNode&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;!-- failover configuration --&gt;
      &lt;property&gt;
          &lt;name&gt;dfs.ha.automatic-failover.enabled&lt;/name&gt;
          &lt;value&gt;true&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.client.failover.proxy.provider.my-cluster&lt;/name&gt;
          &lt;value&gt;org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.ha.fencing.methods&lt;/name&gt;
          &lt;value&gt;sshfence&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
          &lt;name&gt;dfs.ha.fencing.ssh.private-key-files&lt;/name&gt;
          &lt;value&gt;/home/hdfsuser/.ssh/id_rsa&lt;/value&gt;
      &lt;/property&gt;
  
  &lt;/configuration&gt;
  </code></pre>
  
  <p><strong>dfs.service</strong></p>
  
  <pre><code>[Unit]
  Description=Hadoop DFS namenode and datanode
  After=syslog.target network.target remote-fs.target nss-lookup.target network-online.target
  Requires=network-online.target
  
  [Service]
  User=hdfsuser
  Group=hdfsgroup
  Type=simple
  ExecStart=/apps/hadoop/sbin/start-dfs.sh
  ExecStop=/apps/hadoop/sbin/stop-dfs.sh
  RemainAfterExit=yes
  Restart=on-failure
  StartLimitInterval=350
  StartLimitBurst=10
  
  [Install]
  WantedBy=multi-user.target
  </code></pre>
  
  <p><strong>hadoop-hdfsuser-zkfc-node15-hdfs-spark-master.log</strong>
  <em>(before i crash a namenode)</em></p>
  
  <pre><code>2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
  /************************************************************
  STARTUP_MSG: Starting DFSZKFailoverController
  STARTUP_MSG:   host = node15-hdfs-spark-master/10.10.10.15
  STARTUP_MSG:   args = []
  STARTUP_MSG:   version = 3.2.1
  STARTUP_MSG:   classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hado$STARTUP_MSG:   build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
  STARTUP_MSG:   java = 1.8.0_242
  ************************************************************/
  2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
  2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
  2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoo$2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
  2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=&lt;NA&gt;
  2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
  2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
  2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
  2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
  2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
  2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
  2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 wat$2020-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
  2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
  2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
  2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
  2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.$2020-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
  2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
  2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
  2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
  2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
  2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
  2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
  2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
  2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
  2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
  2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
  2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state
  </code></pre>
  
  <p><strong>hadoop-hdfsuser-zkfc-node15-hdfs-spark-master.log</strong>
  <em>(after i crash a namenode)</em></p>
  
  <pre><code>2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
  /************************************************************
  STARTUP_MSG: Starting DFSZKFailoverController
  STARTUP_MSG:   host = node15-hdfs-spark-master/10.10.10.15
  STARTUP_MSG:   args = []
  STARTUP_MSG:   version = 3.2.1
  STARTUP_MSG:   classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/commons-net-$STARTUP_MSG:   build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
  STARTUP_MSG:   java = 1.8.0_242
  ************************************************************/
  2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
  2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
  2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
  2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/$2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
  2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=&lt;NA&gt;
  2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
  2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
  2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
  2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
  2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
  2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
  2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.$2020-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
  2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
  2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
  2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
  2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, i$2020-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
  2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
  2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
  2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
  2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
  2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
  2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
  2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
  2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
  2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
  2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
  2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state
  2020-04-09 13:38:59,910 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
  java.io.EOFException: End of File Exception between local host is: "node15-hdfs-spark-master/10.10.10.15"; destination host is: "hdfs-0":9000; : java.io.EOFException; For more details see:  http://wiki.apache.org/ha$
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
          at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:791)
          at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
          at org.apache.hadoop.ipc.Client.call(Client.java:1491)
          at org.apache.hadoop.ipc.Client.call(Client.java:1388)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
          at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
          at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
          at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
          at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
          at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
  Caused by: java.io.EOFException
          at java.io.DataInputStream.readInt(DataInputStream.java:392)
          at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1850)
          at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1183)
          at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079)
  2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING
  2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_NOT_RESPONDING
  2020-04-09 13:38:59,938 WARN org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Can't get local NN thread dump due to Connexion refusée (Connection refused)
  2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hdfs-0/10.10.10.15:9000 and marking that fencing is necessary
  2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election
  2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ZooKeeper: Session: 0x50000a3038f0000 closed
  2020-04-09 13:38:59,947 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x50000a3038f0000
  2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x50000a3038f0000
  2020-04-09 13:39:01,951 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=10$2020-04-09 13:39:01,953 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
  java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see:  http://wiki.apache.org/ha$
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
          at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
          at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
          at org.apache.hadoop.ipc.Client.call(Client.java:1491)
          at org.apache.hadoop.ipc.Client.call(Client.java:1388)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
          at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
          at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
          at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
          at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
          at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
  Caused by: java.net.ConnectException: Connexion refusée
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
          at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
          at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
          at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
          at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
          at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
          at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
          at org.apache.hadoop.ipc.Client.call(Client.java:1435)
          ... 8 more
  2020-04-09 13:39:03,956 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=10$2020-04-09 13:39:03,958 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
  java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see:  http://wiki.apache.org/ha$
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
          at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
          at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
          at org.apache.hadoop.ipc.Client.call(Client.java:1491)
          at org.apache.hadoop.ipc.Client.call(Client.java:1388)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
          at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
          at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
          at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
          at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
          at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
  Caused by: java.net.ConnectException: Connexion refusée
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
          at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
          at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
          at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
          at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
          at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
          at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
          at org.apache.hadoop.ipc.Client.call(Client.java:1435)
          ... 8 more
  ...
  ...
  ...
  </code></pre>