<p>I have installed rabbitmq using helm chart on a kubernetes cluster. The rabbitmq pod keeps restarting. On inspecting the pod logs I get the below error</p>

<pre><code>2020-02-26 04:42:31.582 [warning] &lt;0.314.0&gt; Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_durable_queue]}
2020-02-26 04:42:31.582 [info] &lt;0.314.0&gt; Waiting for Mnesia tables for 30000 ms, 6 retries left
</code></pre>

<p>When I try to do kubectl describe pod I get this error</p>

<pre><code>Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-rabbitmq-0
    ReadOnly:   false
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rabbitmq-config
    Optional:  false
  healthchecks:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rabbitmq-healthchecks
    Optional:  false
  rabbitmq-token-w74kb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rabbitmq-token-w74kb
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/arch=amd64
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                      From                                               Message
  ----     ------     ----                     ----                                               -------
  Warning  Unhealthy  3m27s (x878 over 7h21m)  kubelet, gke-analytics-default-pool-918f5943-w0t0  Readiness probe failed: Timeout: 70 seconds ...
Checking health of node rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local ...
Status of node rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local ...
Error:
{:aborted, {:no_exists, [:rabbit_vhost, [{{:vhost, :"$1", :_, :_}, [], [:"$1"]}]]}}
Error:
{:aborted, {:no_exists, [:rabbit_vhost, [{{:vhost, :"$1", :_, :_}, [], [:"$1"]}]]}}
</code></pre>

<p>I have provisioned the above on Google Cloud on a kubernetes cluster. I am not sure during what specific situation it started failing. I had to restart the pod and since then it has been failing.  </p>

<p>What is the issue here ?</p>

jeril
<p><strong>TLDR</strong></p>
<p><code>helm upgrade rabbitmq --set clustering.forceBoot=true</code></p>
<p><strong>Problem</strong></p>
<p>The problem happens for the following reason:</p>
<ul>
<li>All RMQ pods are terminated at the same time due to some reason (maybe because you explicitly set the StatefulSet replicas to 0, or something else)</li>
<li>One of them is the last one to stop (maybe just a tiny bit after the others). It stores this condition (&quot;I'm standalone now&quot;) in its filesystem, which in k8s is the PersistentVolume(Claim). Let's say this pod is rabbitmq-1.</li>
<li>When you spin the StatefulSet back up, the pod rabbitmq-0 is always the first to start (see <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#rolling-updates" rel="noreferrer">here</a>).</li>
<li>During startup, pod rabbitmq-0 first checks whether it's supposed to run standalone. But as far as it can see on its own filesystem, it's part of a cluster. So it checks for its peers and doesn't find any. This <a href="https://www.rabbitmq.com/clustering.html#restarting-schema-sync" rel="noreferrer">results in a startup failure by default</a>.</li>
<li>rabbitmq-0 thus never becomes ready.</li>
<li>rabbitmq-1 is never starting because that's how StatefulSets are deployed - one after another. If it were to start, it would start successfully because it sees that it can run standalone as well.</li>
</ul>
<p>So in the end, it's a bit of a mismatch between how RabbitMQ and StatefulSets work. RMQ says: &quot;if everything goes down, just start everything and the same time, one will be able to start and as soon as this one is up, the others can rejoin the cluster.&quot; k8s StatefulSets say: &quot;starting everything all at once is not possible, we'll start with the 0&quot;.</p>
<p><strong>Solution</strong></p>
<p>To fix this, there is a <a href="https://www.rabbitmq.com/clustering.html#forced-boot" rel="noreferrer">force_boot</a> command for rabbitmqctl which basically tells an instance to start standalone if it doesn't find any peers. How you can use this from Kubernetes depends on the Helm chart and container you're using. In the <a href="https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq" rel="noreferrer">Bitnami Chart</a>, which uses the <a href="https://hub.docker.com/r/bitnami/rabbitmq/" rel="noreferrer">Bitnami Docker image</a>, there is a value <code>clustering.forceBoot = true</code>, which translates to an env variable <code>RABBITMQ_FORCE_BOOT = yes</code> in the container, which will then issue the above command for you.</p>
<p>But looking at the problem, you can also see why deleting PVCs will work (<a href="https://stackoverflow.com/a/60414496/9912731">other answer</a>). The pods will just all &quot;forget&quot; that they were part of a RMQ cluster the last time around, and happily start. I would prefer the above solution though, as no data is being lost.</p>
