title: Hashicorp Raft consensus deadlock state
body: <p>I am implementing a Raft service using Hashicorp Raft library for distributed consensus.
  <a href="https://github.com/hashicorp/raft" rel="nofollow noreferrer">https://github.com/hashicorp/raft</a></p>
  <p>I have a simple layout with Raft, 1 followers and a leader.
  I bootstrap the cluster with the leader and add 2 follower Raft nodes to the Raft cluster, things look fine. When I knock one of the followers offline, the leader gets:</p>
  <p><code>failed to contact quorum of nodes, stepping down</code>. The problem with this is now there are 0 nodes in leader state and no one can promote to leader because majority of votes required from quorum of nodes. Now because the previous leader is now a follower, even my service discovery tools can't remove the old ip address from the leader because it requires leader power to do so.</p>
  <p>My cluster enters this infinite loop (deadlock) of trying to connect to a node that's offline forever and no one can get promoted to leader. Any ideas?</p>
  <p>Edit: After realizing, I guess I need a system where there are an odd number of nodes to reach quorum. (ie 3 nodes, 1 gets knocked offline then I can tell the new leader to remove old IP address)</p>