title: Apache NiFi: node state not changing from 'Connecting' to 'Connected'
body: <p>One of the nodes in my cluster lost connectivity due to a slow network. In the UI it showed up as disconnected. It allows me to reconnect it by clicking on the electrical plug icon. Now the node is in 'Connecting' state for the last 20 minutes. </p>
  
  <p>The log below shows the Connection state changed to RECONNECTED but UI doesn't see this and also when I try to make any change to the flow it errors out due to a node being not in the cluster</p>
  
  <pre><code>2019-08-01 23:22:09,912 INFO [Curator-Framework-0-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
  2019-08-01 23:22:09,912 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to SUSPENDED
  2019-08-01 23:22:09,912 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to SUSPENDED
  2019-08-01 23:22:10,876 INFO [Curator-Framework-0-EventThread] o.a.c.f.state.ConnectionStateManager State change: LOST
  2019-08-01 23:22:10,876 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to LOST
  2019-08-01 23:22:10,876 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to LOST
  2019-08-01 23:22:10,879 INFO [Curator-Framework-0-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
  2019-08-01 23:22:10,880 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to RECONNECTED
  2019-08-01 23:22:10,881 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to RECONNECTED
  2019-08-01 23:22:10,978 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2019-08-01 23:22:09,816 and sent to usprz3-sec-asept-dz03.prz.apple.com:11443 at 2019-08-01 23:22:10,978; send took 1161 millis
  2019-08-01 23:22:47,420 INFO [pool-13-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
  2019-08-01 23:22:47,497 INFO [pool-13-thread-1] o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 6409 Records and 92 Swap Files in 76 milliseconds (Stop-the-world time = 2 milliseconds), max Transaction ID 190725903
  2019-08-01 23:22:47,497 INFO [pool-13-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 6409 records in 76 milliseconds
  2019-08-01 23:22:47,519 INFO [Curator-Framework-0-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
  2019-08-01 23:22:47,520 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to SUSPENDED
  2019-08-01 23:22:47,520 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to SUSPENDED
  2019-08-01 23:22:47,686 INFO [Curator-Framework-0-EventThread] o.a.c.f.state.ConnectionStateManager State change: LOST
  2019-08-01 23:22:47,687 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to LOST
  2019-08-01 23:22:47,687 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to LOST
  2019-08-01 23:22:47,690 INFO [Curator-Framework-0-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
  2019-08-01 23:22:47,690 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to RECONNECTED
  2019-08-01 23:22:47,691 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to RECONNECTED
  2019-08-01 23:22:47,786 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2019-08-01 23:22:47,420 and sent to usprz3-sec-asept-dz03.prz.apple.com:11443 at 2019-08-01 23:22:47,786; send took 366 millis
  2019-08-01 23:23:15,605 INFO [Curator-Framework-0-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
  2019-08-01 23:23:15,606 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to SUSPENDED
  2019-08-01 23:23:15,606 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to SUSPENDED
  2019-08-01 23:23:39,950 WARN [Process Cluster Protocol Request-7] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from usprz3-sec-asept-dz03.prz.apple.com due to javax.net.ssl.SSLException: Received close_notify during handshake
  javax.net.ssl.SSLException: Received close_notify during handshake
      at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
      at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
      at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1898)
      at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1999)
      at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
      at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:931)
      at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
      at sun.security.ssl.AppInputStream.read(AppInputStream.java:71)
      at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:41)
      at java.io.FilterInputStream.read(FilterInputStream.java:83)
      at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:110)
      at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:149)
      at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  2019-08-01 23:24:03,610 INFO [Curator-Framework-0-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
  2019-08-01 23:24:03,612 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to RECONNECTED
  2019-08-01 23:24:03,612 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to RECONNECTED
  2019-08-01 23:24:28,174 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message due to: java.net.SocketException: Broken pipe (Write failed)
  2019-08-01 23:24:28,178 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@4df4f611 checkpointed with 2 Records and 0 Swap Files in 5 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 11
  2019-08-01 23:24:52,583 INFO [pool-13-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
  2019-08-01 23:24:52,584 INFO [Curator-Framework-0-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
  2019-08-01 23:24:52,589 ERROR [Curator-Framework-0-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
  org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:516)
      at org.apache.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:50)
      at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609)
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
  2019-08-01 23:24:52,589 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to SUSPENDED
  2019-08-01 23:24:52,589 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to SUSPENDED
  2019-08-01 23:24:52,589 ERROR [Curator-Framework-0-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
  org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:516)
      at org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:166)
      at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609)
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
  2019-08-01 23:24:52,589 ERROR [Curator-Framework-0-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
  org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:516)
      at org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:166)
      at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609)
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
  2019-08-01 23:25:39,703 WARN [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from usprz3-sec-asept-dz03.prz.apple.com due to javax.net.ssl.SSLException: Received close_notify during handshake
  javax.net.ssl.SSLException: Received close_notify during handshake
      at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
      at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
      at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1898)
      at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1999)
      at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
      at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:931)
      at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
      at sun.security.ssl.AppInputStream.read(AppInputStream.java:71)
      at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:41)
      at java.io.FilterInputStream.read(FilterInputStream.java:83)
      at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:110)
      at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:149)
      at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  2019-08-01 23:25:39,709 INFO [Heartbeat Monitor Thread-1-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
  2019-08-01 23:25:39,709 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@ed9fc74 Connection State changed to RECONNECTED
  2019-08-01 23:25:39,709 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1e13df Connection State changed to RECONNECTED
  
  2019-08-01 23:39:26,030 ERROR [pool-15-thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[pool-15-thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
  2019-08-01 23:39:26,030 ERROR [Validate Components Thread-5] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=903b7f39-558a-300d-a22d-f142b0f48862] Failed to perform validation due to java.lang.OutOfMemoryError: Java heap space: java.lang.OutOfMemoryError: Java heap space
  java.lang.OutOfMemoryError: Java heap space
  2019-08-01 23:52:40,424 ERROR [NAR Auto-Loader] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[NAR Auto-Loader,5,main]: java.lang.OutOfMemoryError: Java heap space
  2019-08-01 23:52:40,424 ERROR [Timer-Driven Process Thread-1] c.a.a.e.n.r.ASEPNiFiReportingTask ASEPNiFiReportingTask[id=1481f25c-0167-1000-ffa8-3e26ef2db28c] Error running task ASEPNiFiReportingTask[id=1481f25c-0167-1000-ffa8-3e26ef2db28c] due to java.lang.OutOfMemoryError: Java heap space
  2019-08-01 23:52:40,424 ERROR [pool-15-thread-2] org.apache.nifi.NiFi
  java.lang.OutOfMemoryError: Java heap space
  2019-08-01 23:32:06,238 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@4df4f611 checkpointed with 2 Records and 0 Swap Files in 262603 milliseconds (Stop-the-world time = 26748 milliseconds, Clear Edit Logs time = 166754 millis), max Transaction ID 11
  2019-08-01 23:52:40,424 ERROR [Timer-Driven Process Thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Timer-Driven Process Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
  2019-08-01 23:52:40,424 ERROR [Timer-Driven Process Thread-2] org.apache.nifi.NiFi
  java.lang.OutOfMemoryError: Java heap space
  2019-08-01 23:52:40,424 ERROR [NAR Auto-Loader] org.apache.nifi.NiFi
  java.lang.OutOfMemoryError: Java heap space
  2019-08-01 23:28:56,961 ERROR [Timer-Driven Process Thread-7] o.a.n.p.standard.HandleHttpResponse HandleHttpResponse[id=7cead718-52e1-3936-59bc-3dbbe6744ca1] Failed to respond to HTTP request for StandardFlowFileRecord[uuid=8ae5dbdc-680c-4182-8f14-55fa625ff3e8,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1564701496063-439, container=default, section=439], offset=1017170, length=72656],offset=0,name=8ae5dbdc-680c-4182-8f14-55fa625ff3e8,size=72656] because FlowFile had an 'http.context.identifier' attribute of 4481a8f9-4875-4bac-96d0-05d6867423b5 but could not find an HTTP Response Object for this identifier
  2019-08-01 23:53:25,528 ERROR [pool-13-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile Repository due to java.lang.OutOfMemoryError: Java heap space
  java.lang.OutOfMemoryError: Java heap space
  </code></pre>