title: RabbitMQ inconsistent cluster
body: <p>Few questions about <strong>RabbitMQ v3.1.5</strong> clustering.
  I have a cluster with 2 nodes, <strong>rabbitmq.config</strong> is like this on both nodes:</p>
  
  <pre><code>[
    {rabbit, [
      {cluster_nodes, {['rabbit@rmq01', 'rabbit@rmq02'], ram}},
      {tcp_listeners, [5674]}
    ]}
  ].
  </code></pre>
  
  <p>I already seen issue like this, and now I'm watching it again:
  When sometimes all cluster is shutting down, in case second node (rmq02) starts before first (rmq01), it 'forgets' about rmq01:</p>
  
  <pre><code>[root@rmq2 rabbitmq]# rabbitmqctl cluster_status
  Cluster status of node 'rabbit@rmq2' ...
  [{nodes,[{disc,['rabbit@rmq2']}]},
   {running_nodes,['rabbit@rmq2']},
   {partitions,[]}]
  ...done.
  </code></pre>
  
  <p>After this first node (rmq01) can not start due to rmq2 disagrees about clustering:</p>
  
  <pre><code>{"init terminating in do_boot",{rabbit,failure_during_boot,{error,{inconsistent_cluster,"Node 'rabbit@rmq1' thinks it's clustered with node 'rabbit@rmq2', but 'rabbit@rmq2' disagrees"}}}}
  </code></pre>
  
  <p>I've tried to add rmq01 to rmq02, but seems I have to stop_app before this:</p>
  
  <pre><code>[root@rmq2 rabbitmq]# rabbitmqctl join_cluster rabbit@rmq1
  Clustering node 'rabbit@rmq2' with 'rabbit@rmq1' ...
  Error: mnesia_unexpectedly_running
  </code></pre>
  
  <p>Here I see that rmq02 forgot about rmq01:</p>
  
  <pre><code>[root@rmq2 ~]# cat /var/lib/rabbitmq/mnesia/rabbit\@rmq2/cluster_nodes.config 
  {['rabbit@rmq2'],['rabbit@rmq2']}.
  </code></pre>
  
  <p>Meanwhile on rmq01 (correct configuration):</p>
  
  <pre><code>[root@rmq1 ~]# cat /var/lib/rabbitmq/mnesia/rabbit\@rmq1/cluster_nodes.config 
  {['rabbit@rmq1','rabbit@rmq2'],['rabbit@rmq1']}.
  </code></pre>
  
  <p>Questions:</p>
  
  <ol>
  <li>Is it normal <strong>rmq02</strong> forgets about <strong>rmq01</strong>, or I have some missconfiguration? Why is this happening?</li>
  <li>In case it is ok, is it possible to fix up cluster health without <strong>rmq02</strong> downtime (I mean without stop_app)?</li>
  </ol>