<p>I am trying to get Ansible AWX installed on my Kubernetes cluster but the RabbitMQ container is throwing "Failed to get nodes from k8s" error.</p>

<p><strong>Below are the version of platforms I am using</strong> </p>

<pre><code>[node1 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", 
GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", 
BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", 
Platform:"linux/amd64"}
</code></pre>

<p>Kubernetes is deployed via the <a href="https://github.com/kubernetes-incubator/kubespray" rel="nofollow noreferrer">kubespray</a> playbook v2.5.0 and all the services and pods are up and running. (CoreDNS, Weave, IPtables)  </p>

<p>I am deploying <a href="https://github.com/ansible/awx" rel="nofollow noreferrer">AWX</a> via the 1.0.6 release using the 1.0.6 images for awx_web and awx_task.</p>

<p>I am using an external PostgreSQL database at v10.4 and have verified the tables are being created by awx in the db.</p>

<p><strong>Troubleshooting steps I have tried.</strong></p>

<ul>
<li>I tried to deploy AWX 1.0.5 with the etcd pod to the same cluster and it has worked as expected </li>
<li>I have deployed a stand alone <a href="https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/tree/master/examples/k8s_statefulsets" rel="nofollow noreferrer">RabbitMQ cluster</a> in the same k8s cluster trying to mimic the AWX rabbit deployment as much as possible and it works with the rabbit_peer_discovery_k8s backend.</li>
<li>I have tried tweeking some of the rabbitmq.conf for AWX 1.0.6 with no luck it just keeps thowing the same error.</li>
<li>I have verified the /etc/resolv.conf file has the  kubernetes.default.svc.cluster.local entry</li>
</ul>

<p><strong>Cluster Info</strong></p>

<pre><code>[node1 ~]# kubectl get all -n awx
NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/awx   1         1         1            0           38m

NAME                DESIRED   CURRENT   READY     AGE
rs/awx-654f7fc84c   1         1         0         38m

NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/awx   1         1         1            0           38m

NAME                DESIRED   CURRENT   READY     AGE
rs/awx-654f7fc84c   1         1         0         38m

NAME                      READY     STATUS             RESTARTS   AGE
po/awx-654f7fc84c-9ppqb   3/4       CrashLoopBackOff   11         38m

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
svc/awx-rmq-mgmt   ClusterIP   10.233.10.146   &lt;none&gt;        15672/TCP                        1d
svc/awx-web-svc    NodePort    10.233.3.75     &lt;none&gt;        80:31700/TCP                     1d
svc/rabbitmq       NodePort    10.233.37.33    &lt;none&gt;        15672:30434/TCP,5672:31962/TCP   1d
</code></pre>

<p>AWX RabbitMQ error log</p>

<pre><code>[node1 ~]# kubectl logs -n awx awx-654f7fc84c-9ppqb awx-rabbit
2018-07-09 14:47:37.464 [info] &lt;0.33.0&gt; Application lager started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.767 [info] &lt;0.33.0&gt; Application os_mon started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.767 [info] &lt;0.33.0&gt; Application crypto started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.768 [info] &lt;0.33.0&gt; Application cowlib started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.768 [info] &lt;0.33.0&gt; Application xmerl started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.851 [info] &lt;0.33.0&gt; Application mnesia started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.851 [info] &lt;0.33.0&gt; Application recon started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.852 [info] &lt;0.33.0&gt; Application jsx started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.852 [info] &lt;0.33.0&gt; Application asn1 started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.852 [info] &lt;0.33.0&gt; Application public_key started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.897 [info] &lt;0.33.0&gt; Application ssl started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.901 [info] &lt;0.33.0&gt; Application ranch started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.901 [info] &lt;0.33.0&gt; Application ranch_proxy_protocol started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.901 [info] &lt;0.33.0&gt; Application rabbit_common started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.907 [info] &lt;0.33.0&gt; Application amqp_client started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.909 [info] &lt;0.33.0&gt; Application cowboy started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.957 [info] &lt;0.33.0&gt; Application inets started on node 'rabbit@10.233.120.5'
2018-07-09 14:47:37.964 [info] &lt;0.193.0&gt;
 Starting RabbitMQ 3.7.4 on Erlang 20.1.7
 Copyright (C) 2007-2018 Pivotal Software, Inc.
 Licensed under the MPL.  See http://www.rabbitmq.com/

  ##  ##
  ##  ##      RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
  ######  ##
  ##########  Logs: &lt;stdout&gt;

              Starting broker...
2018-07-09 14:47:37.982 [info] &lt;0.193.0&gt;
 node           : rabbit@10.233.120.5
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
 cookie hash    : at619UOZzsenF44tSK3ulA==
 log(s)         : &lt;stdout&gt;
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@10.233.120.5
2018-07-09 14:47:39.649 [info] &lt;0.201.0&gt; Memory high watermark set to 11998 MiB (12581714329 bytes) of 29997 MiB (31454285824 bytes) total
2018-07-09 14:47:39.652 [info] &lt;0.203.0&gt; Enabling free disk space monitoring
2018-07-09 14:47:39.653 [info] &lt;0.203.0&gt; Disk free limit set to 50MB
2018-07-09 14:47:39.658 [info] &lt;0.205.0&gt; Limiting to approx 1048476 file handles (943626 sockets)
2018-07-09 14:47:39.658 [info] &lt;0.206.0&gt; FHC read buffering:  OFF
2018-07-09 14:47:39.658 [info] &lt;0.206.0&gt; FHC write buffering: ON
2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Node database directory at /var/lib/rabbitmq/mnesia/rabbit@10.233.120.5 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Configured peer discovery backend: rabbit_peer_discovery_k8s
2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Peer discovery backend does not support locking, falling back to randomized delay
2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay.
2018-07-09 14:47:39.665 [info] &lt;0.193.0&gt; Failed to get nodes from k8s - {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},
                 {inet,[inet],nxdomain}]}
2018-07-09 14:47:39.665 [error] &lt;0.192.0&gt; CRASH REPORT Process &lt;0.192.0&gt; with 0 neighbours exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n                 {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 164 in application_master:init/4 line 134
2018-07-09 14:47:39.666 [info] &lt;0.33.0&gt; Application rabbit exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n                 {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 164
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,\"{failed_connect,[{to_address,{\\"kubernetes.default.svc.cluster.local\\",443}},\n                 {inet,[inet],nxdomain}]}\"}},[{rabbit_mnesia,init_from_config,0,[{file,\"src/rabbit_mnesia.erl\"},{line,164}]},{rabbit_mnesia,init_with_lock,3,[{file,\"src/rabbit_mnesia.erl\"},{line,144}]},{rabbit_mnesia,init,0,[{file,\"src/rabbit_mnesia.erl\"},{line,111}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,793}]}]}}}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,"{failed_connect,[{to_address,{\"kubernetes.defau

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
</code></pre>

<p>Kubernetes API service </p>

<pre><code>[node1 ~]# kubectl describe service kubernetes
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                provider=kubernetes
Annotations:       &lt;none&gt;
Selector:          &lt;none&gt;
Type:              ClusterIP
IP:                10.233.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         10.237.34.19:6443,10.237.34.21:6443
Session Affinity:  ClientIP
Events:            &lt;none&gt;
</code></pre>

<p>nslookup from a busybox in the same kubernetes cluster</p>

<pre><code>[node2 ~]# kubectl exec -it busybox -- sh
/ # nslookup  kubernetes.default.svc.cluster.local
Server:    10.233.0.3
Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local

Name:      kubernetes.default.svc.cluster.local
Address 1: 10.233.0.1 kubernetes.default.svc.cluster.local
</code></pre>

<p>Please let me know if I am missing anything that could help troubleshooting.</p>

<p>I <em>believe</em> the solution is to omit <a href="https://github.com/ansible/awx/blob/1.0.6/installer/roles/kubernetes/templates/deployment.yml.j2#L40" rel="nofollow noreferrer">the explicit kubernetes host</a>. I can't think of any good reason one would need to <em>specify</em> the kubernetes api host from inside the cluster.</p>

<p>If for some terrible reason the RMQ plugin requires it, then try swapping in the <code>Service</code> IP (assuming your SSL cert for the master has its <code>Service</code> IP in the SANs list).</p>

<hr>

<p>As for <em>why</em> it is doing such a silly thing, the only good reason I can think of is that the RMQ <code>PodSpec</code> has somehow gotten a <code>dnsPolicy</code> of something other than <code>ClusterFirst</code>. If you truly wish to troubleshoot the RMQ Pod, then you can provide an explicit <code>command:</code> to run some debugging bash commands first, in order to interrogate the state of the container at launch, and then <code>exec /launch.sh</code> to resume booting up RMQ (<a href="https://github.com/ansible/awx-rabbitmq/blob/7f20bd7dc057beb4070d5a6032c3c70ac0b7f724/Dockerfile#L14" rel="nofollow noreferrer">as they do</a>)</p>
