title: Go Webapp Cluster Leader Election
body: <p>I'm writing a fairly complex Go webapp that I want to make highly available.  I'm planning on having multiple VMs running the app, with a load-balancer directing traffic between them for what I want.</p>
  
  <p>Where it gets complex is that the webapp has a sort of database book-keeping routine running, which I only want (at most) one instance of at any time.  So if I have three webapp VMs, only one of them should be doing the book-keeping.</p>
  
  <p>(Yes, I'm aware that I could split the book-keeping off into a separate VM instance entirely, but the code has been rather heavily integrated with the rest of the webapp.)</p>
  
  <p>I've spent several hours looking at things like <a href="https://github.com/coreos/etcd" rel="nofollow">etcd</a>, <a href="https://github.com/hashicorp/raft" rel="nofollow">raft</a>, <a href="https://github.com/monnand/bully" rel="nofollow">bully</a>, <a href="https://github.com/hashicorp/memberlist" rel="nofollow">memberlist</a>, <a href="http://clusterlabs.org/" rel="nofollow">Pacemaker</a>, and so on and so forth.  These all seem like quite a lot of information to absorb to accomplish what I'm after, or I can't see a clear way of using them.</p>
  
  <p>What I would like, in this specific use case, is a system whereby Go webapp nodes automatically detect each other and elect a "leader" to do the book-keeping.  Ideally this would scale anywhere from 2 to 10 nodes, and not require manually adding IP addresses to config files (but possible, if necessary).</p>
  
  <p>I was thinking in the case of a network partition or something, where one node cannot see the others, I wouldn't want it to elect itself as a leader, because it would be possible to have two nodes attempting to do book-keeping at the same time.  That also means that if I stripped down the cluster to being just a single VM, no book-keeping would occur, but that could be tolerated for a brief period during maintenance, or I could set some sort of flag somewhere.</p>
  
  <p>I'm wondering if someone could point me in the right direction, and hopefully how I can accomplish this with low complexity, while leveraging existing code libraries as much as possible.</p>