<p>We are trying to run an instance of the RabbitMQ chart with Helm from the <a href="https://github.com/helm/charts/tree/master/stable/rabbitmq" rel="nofollow noreferrer">helm/charts/stable/rabbit</a> project. I had it running perfect but then I had to restart k8s for some maintenance. Now we are completely unable to launch the RabbitMQ chart in any way shape or form. I am not even trying to run the chart with any variables, i.e. just the default values. </p>

<p>Here is all I am doing:</p>

<pre><code>helm install stable/rabbitmq
</code></pre>

<p>I have confirmed I can simply run the default right on my local k8s which I'm running with Docker for Desktop. When we run the rabbit chart on our shared k8s the exact same way as on desktop and what we did before the restart, the following error is thrown:</p>

<pre><code>Failed to get nodes from k8s - 503
</code></pre>

<p>I have also posted an issue on the Helm charts repo as well. <a href="https://github.com/helm/charts/issues/10811" rel="nofollow noreferrer">Click here to see the issue on Github.</a> </p>

<p>We are suspecting the DNS but are unable to confirm anything yet. What is very frustrating is after the restart every single other chart we installed restarted perfectly except Rabbit which now will not start at all. </p>

<p>Anyone know what I could do to get Rabbits peer discovery to work? Anyone seen issue like this after restarting k8s? </p>

<p>So I actually got rabbit to run. Turns out my issue was the k8s peer discovery could not connect over the default port 443 and I had to use the external port 6443 because <code>kubernetes.default.svc.cluster.local</code> resolved to the public port and could not find the internal, so yeah our config is messed up too. </p>

<p>It took me a while to realize the variable below was not overriding when I overrode it with  <code>helm install . -f server-values.yaml</code>.  </p>

<pre><code>rabbitmq:
  configuration: |-
    ## Clustering
    cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.port = 6443
    cluster_formation.node_cleanup.interval = 10
    cluster_formation.node_cleanup.only_log_warning = true
    cluster_partition_handling = autoheal
    # queue master locator
    queue_master_locator=min-masters
    # enable guest user
    loopback_users.guest = false
</code></pre>

<p>I had to add <code>cluster_formation.k8s.port = 6443</code> to the main <code>values.yaml</code> file instead of my own. Once the port was changed specifically in the <code>values.yaml</code>, rabbit started right up. </p>
