<p>I had the same problem with <a href="https://stackoverflow.com/questions/60812042/rabbitmq-fails-to-start-after-restart-kubernetes-cluster">@Amir Soleimani</a> but the error result was a bit different, I tried all the solutions in that post but all of them didn't work.... I'm using Azure Kubernetes Service (AKS) and after upgrading from <strong>1.13.xx</strong> to <strong>1.18.xx</strong> can't start RabbitMQ anymore.</p>
<p><strong>UPDATED</strong> - Solution that worked for me (please consider this approach as it may affect your existing queues)</p>
<pre><code>Remove current rabbitmq StatefulSet including persistent disks
</code></pre>
<p>========</p>
<p>Here is my StatefulSet file:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-management
  labels:
    app: rabbitmq
spec:
  ports:
    - port: 80
      targetPort: 15672
      name: http
  selector:
    app: rabbitmq
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  labels:
    app: rabbitmq
spec:
  ports:
    - port: 5672
      name: amqp
    - port: 4369
      name: epmd
    - port: 25672
      name: rabbitmq-dist
  clusterIP: None
  selector:
    app: rabbitmq
---
apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-config
  namespace: default
type: Opaque
data:
  erlang.cookie: samplecookie==
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  labels:
    app: rabbitmq
spec:
  serviceName: rabbitmq
  selector:
    matchLabels:
      app: rabbitmq
  replicas: 3
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      containers:
        - name: rabbitmq
          image: 'rabbitmq:3.6.6-management-alpine'
          lifecycle:
            postStart:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - &gt;
                    if [ -z &quot;$(grep rabbitmq /etc/resolv.conf)&quot; ]; then
                      sed &quot;s/^search \([^ ]\+\)/search rabbitmq.\1 \1/&quot; /etc/resolv.conf &gt; /etc/resolv.conf.new;
                      cat /etc/resolv.conf.new &gt; /etc/resolv.conf;
                      rm /etc/resolv.conf.new;
                    fi;
                    until rabbitmqctl node_health_check; do sleep 1; done;
                    if [[ &quot;$HOSTNAME&quot; != &quot;rabbitmq-0&quot; &amp;&amp; -z &quot;$(rabbitmqctl cluster_status | grep rabbitmq-0)&quot; ]]; then
                      rabbitmqctl stop_app;
                      rabbitmqctl join_cluster rabbit@rabbitmq-0;
                      rabbitmqctl start_app;
                    fi;
                    rabbitmqctl set_policy ha-all &quot;.&quot; '{&quot;ha-mode&quot;:&quot;exactly&quot;,&quot;ha-params&quot;:3,&quot;ha-sync-mode&quot;:&quot;automatic&quot;}'
          env:
            - name: RABBITMQ_ERLANG_COOKIE
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-config
                  key: erlang.cookie
            - name: RABBITMQ_DEFAULT_USER
              value: username
            - name: RABBITMQ_DEFAULT_PASS
              value: password
          ports:
            - containerPort: 5672
              name: amqp
            - containerPort: 15672
              name: amqp-management
          volumeMounts:
            - mountPath: /var/lib/rabbitmq
              name: volume
  volumeClaimTemplates:
    - metadata:
        name: volume
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
</code></pre>
<p>Result of <code>kubectl describe pod rabbitmq-0</code></p>
<pre><code>DIAGNOSTICS
===========

attempted to contact: ['rabbit@rabbitmq-0']

rabbit@rabbitmq-0:
  * connected to epmd (port 4369) on rabbitmq-0
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on rabbitmq-0
  * suggestion: start the node

current node details:
- node name: 'rabbitmq-cli-91@rabbitmq-0'
- home dir: /var/lib/rabbitmq
- cookie hash: P1XNOe5pN3Ug2FCRFzH7Xg==

Error: unable to connect to node 'rabbit@rabbitmq-0': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@rabbitmq-0']

rabbit@rabbitmq-0:
  * connected to epmd (port 4369) on rabbitmq-0
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on rabbitmq-0
  * suggestion: start the node

current node details:
- node name: 'rabbitmq-cli-26@rabbitmq-0'
- home dir: /var/lib/rabbitmq
- cookie hash: P1XNOe5pN3Ug2FCRFzH7Xg==

Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}
Error: rabbit application is not running on node rabbit@rabbitmq-0.
 * Suggestion: start it with &quot;rabbitmqctl start_app&quot; and try again
, message: &quot;Timeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nTimeout: 70.0 seconds ...\nChecking health of node 'rabbit@rabbitmq-0' ...\nError: unable to connect to node 'rabbit@rabbitmq-0': nodedown\n\nDIAGNOSTICS\n===========\n\nattempted to contact: ['rabbit@rabbitmq-0']\n\nrabbit@rabbitmq-0:\n  * connected to epmd (port 4369) on rabbitmq-0\n  * epmd reports: node 'rabbit' not running at all\n                  no other nodes on rabbitmq-0\n  * suggestion: start the node\n\ncurrent node details:\n- node name: 'rabbitmq-cli-91@rabbitmq-0'\n- home dir: /var/lib/rabbitmq\n- cookie hash: P1XNOe5pN3Ug2FCRFzH7Xg==\n\nError: unable to connect to node 'rabbit@rabbitmq-0': nodedown\n\nDIAGNOSTICS\n===========\n\nattempted to contact: ['rabbit@rabbitmq-0']\n\nrabbit@rabbitmq-0:\n  * connected to epmd (port 4369) on rabbitmq-0\n  * epmd reports: node 'rabbit' not running at all\n                  no other nodes on rabbitmq-0\n  * suggestion: start the node\n\ncurrent node details:\n- node name: 'rabbitmq-cli-26@rabbitmq-0'\n- home dir: /var/lib/rabbitmq\n- cookie hash: P1XNOe5pN3Ug2FCRFzH7Xg==\n\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: {aborted,{no_exists,[rabbit_vhost,[{{vhost,'$1','_'},[],['$1']}]]}}\nError: rabbit application is not running on node rabbit@rabbitmq-0.\n * Suggestion: start it with \&quot;rabbitmqctl start_app\&quot; and try again\n&quot;
  Warning  FailedPostStartHook  23m  kubelet  Exec lifecycle hook ([/bin/sh -c if [ -z &quot;$(grep rabbitmq /etc/resolv.conf)&quot; ]; then
  sed &quot;s/^search \([^ ]\+\)/search rabbitmq.\1 \1/&quot; /etc/resolv.conf &gt; /etc/resolv.conf.new;
  cat /etc/resolv.conf.new &gt; /etc/resolv.conf;
  rm /etc/resolv.conf.new;
fi; until rabbitmqctl node_health_check; do sleep 1; done; if [[ &quot;$HOSTNAME&quot; != &quot;rabbitmq-0&quot; &amp;&amp; -z &quot;$(rabbitmqctl cluster_status | grep rabbitmq-0)&quot; ]]; then
  rabbitmqctl stop_app;
  rabbitmqctl join_cluster rabbit@rabbitmq-0;
  rabbitmqctl start_app;
fi; rabbitmqctl set_policy ha-all &quot;.&quot; '{&quot;ha-mode&quot;:&quot;exactly&quot;,&quot;ha-params&quot;:3,&quot;ha-sync-mode&quot;:&quot;automatic&quot;}'
]) for Container &quot;rabbitmq&quot; in Pod &quot;rabbitmq-0_default(3ac91d73-de7b-4cde-81f6-c31bacd10252)&quot; failed - error: command '/bin/sh -c if [ -z &quot;$(grep rabbitmq /etc/resolv.conf)&quot; ]; then
  sed &quot;s/^search \([^ ]\+\)/search rabbitmq.\1 \1/&quot; /etc/resolv.conf &gt; /etc/resolv.conf.new;
  cat /etc/resolv.conf.new &gt; /etc/resolv.conf;
  rm /etc/resolv.conf.new;
fi; until rabbitmqctl node_health_check; do sleep 1; done; if [[ &quot;$HOSTNAME&quot; != &quot;rabbitmq-0&quot; &amp;&amp; -z &quot;$(rabbitmqctl cluster_status | grep rabbitmq-0)&quot; ]]; then
  rabbitmqctl stop_app;
  rabbitmqctl join_cluster rabbit@rabbitmq-0;
  rabbitmqctl start_app;
fi; rabbitmqctl set_policy ha-all &quot;.&quot; '{&quot;ha-mode&quot;:&quot;exactly&quot;,&quot;ha-params&quot;:3,&quot;ha-sync-mode&quot;:&quot;automatic&quot;}'
' exited with 137: Error: unable to connect to node 'rabbit@rabbitmq-0': nodedown
</code></pre>
<p>Result of <code>kubectl logs rabbitmq-0</code></p>
<pre><code>=CRASH REPORT==== 18-Jul-2021::11:06:01 ===
  crasher:
    initial call: application_master:init/4
    pid: &lt;0.156.0&gt;
    registered_name: []
    exception exit: {{timeout_waiting_for_tables,
                         [rabbit_user,rabbit_user_permission,rabbit_vhost,
                          rabbit_durable_route,rabbit_durable_exchange,
                          rabbit_runtime_parameters,rabbit_durable_queue]},
                     {rabbit,start,[normal,[]]}}
      in function  application_master:init/4 (application_master.erl, line 134)
    ancestors: [&lt;0.155.0&gt;]
    messages: [{'EXIT',&lt;0.157.0&gt;,normal}]
    links: [&lt;0.155.0&gt;,&lt;0.31.0&gt;]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 98
  neighbours:

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: rabbit
    exited: {{timeout_waiting_for_tables,
                 [rabbit_user,rabbit_user_permission,rabbit_vhost,
                  rabbit_durable_route,rabbit_durable_exchange,
                  rabbit_runtime_parameters,rabbit_durable_queue]},
             {rabbit,start,[normal,[]]}}
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: amqp_client
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: rabbit_common
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: xmerl
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: os_mon
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: inets
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: asn1
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: syntax_tools
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: mnesia
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: crypto
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: ranch
    exited: stopped
    type: temporary

=INFO REPORT==== 18-Jul-2021::11:06:01 ===
    application: compiler
    exited: stopped
    type: temporary


BOOT FAILED
===========

Timeout contacting cluster nodes: ['rabbit@rabbitmq-1','rabbit@rabbitmq-2'].

BACKGROUND
==========

This cluster node was shut down while other nodes were still running.
To avoid losing data, you should start the other nodes first, then
start this one. To force this node to start, first invoke
&quot;rabbitmqctl force_boot&quot;. If you do so, any changes made on other
cluster nodes after this one was shut down may be lost.

DIAGNOSTICS
===========

attempted to contact: ['rabbit@rabbitmq-1','rabbit@rabbitmq-2']

rabbit@rabbitmq-1:
  * unable to connect to epmd (port 4369) on rabbitmq-1: nxdomain (non-existing domain)

rabbit@rabbitmq-2:
  * unable to connect to epmd (port 4369) on rabbitmq-2: nxdomain (non-existing domain)


current node details:
- node name: 'rabbit@rabbitmq-0'
- home dir: /var/lib/rabbitmq
- cookie hash: P1XNOe5pN3Ug2FCRFzH7Xg==



=INFO REPORT==== 18-Jul-2021::11:06:01 ===
Timeout contacting cluster nodes: ['rabbit@rabbitmq-1','rabbit@rabbitmq-2'].

BACKGROUND
==========

This cluster node was shut down while other nodes were still running.
To avoid losing data, you should start the other nodes first, then
start this one. To force this node to start, first invoke
&quot;rabbitmqctl force_boot&quot;. If you do so, any changes made on other
cluster nodes after this one was shut down may be lost.

DIAGNOSTICS
===========

attempted to contact: ['rabbit@rabbitmq-1','rabbit@rabbitmq-2']

rabbit@rabbitmq-1:
  * unable to connect to epmd (port 4369) on rabbitmq-1: nxdomain (non-existing domain)

rabbit@rabbitmq-2:
  * unable to connect to epmd (port 4369) on rabbitmq-2: nxdomain (non-existing domain)


current node details:
- node name: 'rabbit@rabbitmq-0'
- home dir: /var/lib/rabbitmq
- cookie hash: P1XNOe5pN3Ug2FCRFzH7Xg==


{&quot;init terminating in do_boot&quot;,timeout_waiting_for_tables}
init terminating in do_boot (timeout_waiting_for_tables)

Crash dump is being written to: erl_crash.dump...
</code></pre>
<p>What I tried but didn't work:</p>
<pre><code>rabbitmqctl stop_app
rabbitmqctl force_boot
Remove StatefulSet and re-install
Re-configure the yaml file
</code></pre>

Nguyen Thanh
<p>Please try force boot in post Start scipt:</p>
<p>...</p>
<p>fi;</p>
<pre><code>if [[ &quot;$HOSTNAME&quot; == &quot;rabbitmq-0&quot; ]]; then
                  rabbitmqctl stop_app;
                  rabbitmqctl force_boot;
                  
fi;
</code></pre>
<p>until rabbitmqctl node_health_check; do sleep 1; done;
...</p>
