<p>I'm trying to run Rabbitmq using Kubernetes on AWS. I'm using the <a href="https://hub.docker.com/_/rabbitmq/">official Rabbitmq docker container</a>. Each time the pod restarts the rabbitmq container gets a new hostname. I've setup a service (of type LoadBalancer) for the pod with a resolvable DNS name.</p>

<p>But when I use an EBS to make the rabbit config/messsage/queues persistent between restarts it breaks with:</p>

<pre><code>exception exit: {{failed_to_cluster_with,
                     ['rabbitmq@rabbitmq-deployment-2901855891-nord3'],
                     "Mnesia could not connect to any nodes."},
                 {rabbit,start,[normal,[]]}}
  in function  application_master:init/4 (application_master.erl, line 134)
</code></pre>

<p><code>rabbitmq-deployment-2901855891-nord3</code> is the previous hostname rabbitmq container. It is almost like Mnesia saved the old hostname :-/</p>

<p>The container's info looks like this:</p>

<pre><code>              Starting broker...
=INFO REPORT==== 25-Apr-2016::12:42:42 ===
node           : rabbitmq@rabbitmq-deployment-2770204827-cboj8
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : XXXXXXXXXXXXXXXX
log            : tty
sasl log       : tty
database dir   : /var/lib/rabbitmq/mnesia/rabbitmq
</code></pre>

<p>I'm only able to set the first part of the node name to <code>rabbitmq</code> using the <code>RABBITMQ_NODENAME</code> environment variable.</p>

<p>Setting <code>RABBITMQ_NODENAME</code> to a resolvable DNS name breaks with:</p>

<p><code>Can't set short node name!\nPlease check your configuration\n"</code></p>

<p>Setting <code>RABBITMQ_USE_LONGNAME</code> to <code>true</code> breaks with:</p>

<p><code>Can't set long node name!\nPlease check your configuration\n"</code></p>

<p>Update: </p>

<ul>
<li><p>Setting <code>RABBITMQ_NODENAME</code> to rabbitmq@<strong>localhost</strong> works but that negates any possibility to cluster instances. </p>

<pre><code>          Starting broker...
=INFO REPORT==== 26-Apr-2016::11:53:19 ===
node           : rabbitmq@localhost
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : 9WtXr5XgK4KXE/soTc6Lag==
log            : tty
sasl log       : tty
database dir   : /var/lib/rabbitmq/mnesia/rabbitmq@localhost
</code></pre></li>
<li><p>Setting <code>RABBITMQ_NODENAME</code> to the service name, in this case <code>rabbitmq-service</code> like so rabbitmq@<strong>rabbitmq-service</strong> also works since kubernetes service names are internally resolvable via DNS. </p>

<pre><code>          Starting broker...
=INFO REPORT==== 26-Apr-2016::11:53:19 ===
node           : rabbitmq@rabbitmq-service
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : 9WtXr5XgK4KXE/soTc6Lag==
log            : tty
sasl log       : tty
database dir   : /var/lib/rabbitmq/mnesia/rabbitmq@rabbitmq-service
</code></pre></li>
</ul>

<p>Is this the right way though? Will I still be able to cluster multiple instances if the node names are the same?</p>

<p>The idea is to use a different 'service' and 'deployment' for each of the node you want to create. </p>

<p>As you said, you have to create a custom NODENAME for each i.e:</p>

<pre><code>RABBITMQ_NODENAME=rabbit@rabbitmq-1
</code></pre>

<p>Also <code>rabbitmq-1,rabbitmq-2,rabbitmq-3</code> have to be resolved from each nodes.  For that you can use kubedns. The <code>/etc/resolv.conf</code> will look like:</p>

<pre><code>search rmq.svc.cluster.local 
</code></pre>

<p>and <code>/etc/hosts</code> must contains:</p>

<pre><code>127.0.0.1 rabbitmq-1  # or rabbitmq-2 on node 2...
</code></pre>

<p>The services are here to create a stable network identity for each nodes</p>

<pre><code>rabbitmq-1.svc.cluster.local
rabbitmq-2.svc.cluster.local
rabbitmq-3.svc.cluster.local
</code></pre>

<p>The different <code>deployments</code> resources will allow you to mount a different volume on each node.</p>

<p>I'm working on a deployment tool to simplify those actions:
I've done a demo on how I scale and deploy rabbitmq from 1 to 3 nodes on kubernetes:
<a href="https://asciinema.org/a/2ktj7kr2d2m3w25xrpz7mjkbu?speed=1.5" rel="nofollow">https://asciinema.org/a/2ktj7kr2d2m3w25xrpz7mjkbu?speed=1.5</a></p>

<p>More generally, the complexity your facing to deploy a clustered application is addressed in the 'petset proposal': <a href="https://github.com/kubernetes/kubernetes/pull/18016" rel="nofollow">https://github.com/kubernetes/kubernetes/pull/18016</a></p>
