title: Ansible AWX RabbitMQ container in Kubernetes Failed to get nodes from k8s with nxdomain
body: <p>I am trying to get Ansible AWX installed on my Kubernetes cluster but the RabbitMQ container is throwing "Failed to get nodes from k8s" error.</p>
  
  <p><strong>Below are the version of platforms I am using</strong> </p>
  
  <pre><code>[node1 ~]# kubectl version
  Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", 
  GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", 
  BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", 
  Platform:"linux/amd64"}
  </code></pre>
  
  <p>Kubernetes is deployed via the <a href="https://github.com/kubernetes-incubator/kubespray" rel="nofollow noreferrer">kubespray</a> playbook v2.5.0 and all the services and pods are up and running. (CoreDNS, Weave, IPtables)  </p>
  
  <p>I am deploying <a href="https://github.com/ansible/awx" rel="nofollow noreferrer">AWX</a> via the 1.0.6 release using the 1.0.6 images for awx_web and awx_task.</p>
  
  <p>I am using an external PostgreSQL database at v10.4 and have verified the tables are being created by awx in the db.</p>
  
  <p><strong>Troubleshooting steps I have tried.</strong></p>
  
  <ul>
  <li>I tried to deploy AWX 1.0.5 with the etcd pod to the same cluster and it has worked as expected </li>
  <li>I have deployed a stand alone <a href="https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/tree/master/examples/k8s_statefulsets" rel="nofollow noreferrer">RabbitMQ cluster</a> in the same k8s cluster trying to mimic the AWX rabbit deployment as much as possible and it works with the rabbit_peer_discovery_k8s backend.</li>
  <li>I have tried tweeking some of the rabbitmq.conf for AWX 1.0.6 with no luck it just keeps thowing the same error.</li>
  <li>I have verified the /etc/resolv.conf file has the  kubernetes.default.svc.cluster.local entry</li>
  </ul>
  
  <p><strong>Cluster Info</strong></p>
  
  <pre><code>[node1 ~]# kubectl get all -n awx
  NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
  deploy/awx   1         1         1            0           38m
  
  NAME                DESIRED   CURRENT   READY     AGE
  rs/awx-654f7fc84c   1         1         0         38m
  
  NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
  deploy/awx   1         1         1            0           38m
  
  NAME                DESIRED   CURRENT   READY     AGE
  rs/awx-654f7fc84c   1         1         0         38m
  
  NAME                      READY     STATUS             RESTARTS   AGE
  po/awx-654f7fc84c-9ppqb   3/4       CrashLoopBackOff   11         38m
  
  NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
  svc/awx-rmq-mgmt   ClusterIP   10.233.10.146   &lt;none&gt;        15672/TCP                        1d
  svc/awx-web-svc    NodePort    10.233.3.75     &lt;none&gt;        80:31700/TCP                     1d
  svc/rabbitmq       NodePort    10.233.37.33    &lt;none&gt;        15672:30434/TCP,5672:31962/TCP   1d
  </code></pre>
  
  <p>AWX RabbitMQ error log</p>
  
  <pre><code>[node1 ~]# kubectl logs -n awx awx-654f7fc84c-9ppqb awx-rabbit
  2018-07-09 14:47:37.464 [info] &lt;0.33.0&gt; Application lager started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.767 [info] &lt;0.33.0&gt; Application os_mon started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.767 [info] &lt;0.33.0&gt; Application crypto started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.768 [info] &lt;0.33.0&gt; Application cowlib started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.768 [info] &lt;0.33.0&gt; Application xmerl started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.851 [info] &lt;0.33.0&gt; Application mnesia started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.851 [info] &lt;0.33.0&gt; Application recon started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.852 [info] &lt;0.33.0&gt; Application jsx started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.852 [info] &lt;0.33.0&gt; Application asn1 started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.852 [info] &lt;0.33.0&gt; Application public_key started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.897 [info] &lt;0.33.0&gt; Application ssl started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.901 [info] &lt;0.33.0&gt; Application ranch started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.901 [info] &lt;0.33.0&gt; Application ranch_proxy_protocol started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.901 [info] &lt;0.33.0&gt; Application rabbit_common started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.907 [info] &lt;0.33.0&gt; Application amqp_client started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.909 [info] &lt;0.33.0&gt; Application cowboy started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.957 [info] &lt;0.33.0&gt; Application inets started on node 'rabbit@10.233.120.5'
  2018-07-09 14:47:37.964 [info] &lt;0.193.0&gt;
   Starting RabbitMQ 3.7.4 on Erlang 20.1.7
   Copyright (C) 2007-2018 Pivotal Software, Inc.
   Licensed under the MPL.  See http://www.rabbitmq.com/
  
    ##  ##
    ##  ##      RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc.
    ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
    ######  ##
    ##########  Logs: &lt;stdout&gt;
  
                Starting broker...
  2018-07-09 14:47:37.982 [info] &lt;0.193.0&gt;
   node           : rabbit@10.233.120.5
   home dir       : /var/lib/rabbitmq
   config file(s) : /etc/rabbitmq/rabbitmq.conf
   cookie hash    : at619UOZzsenF44tSK3ulA==
   log(s)         : &lt;stdout&gt;
   database dir   : /var/lib/rabbitmq/mnesia/rabbit@10.233.120.5
  2018-07-09 14:47:39.649 [info] &lt;0.201.0&gt; Memory high watermark set to 11998 MiB (12581714329 bytes) of 29997 MiB (31454285824 bytes) total
  2018-07-09 14:47:39.652 [info] &lt;0.203.0&gt; Enabling free disk space monitoring
  2018-07-09 14:47:39.653 [info] &lt;0.203.0&gt; Disk free limit set to 50MB
  2018-07-09 14:47:39.658 [info] &lt;0.205.0&gt; Limiting to approx 1048476 file handles (943626 sockets)
  2018-07-09 14:47:39.658 [info] &lt;0.206.0&gt; FHC read buffering:  OFF
  2018-07-09 14:47:39.658 [info] &lt;0.206.0&gt; FHC write buffering: ON
  2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Node database directory at /var/lib/rabbitmq/mnesia/rabbit@10.233.120.5 is empty. Assuming we need to join an existing cluster or initialise from scratch...
  2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Configured peer discovery backend: rabbit_peer_discovery_k8s
  2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
  2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Peer discovery backend does not support locking, falling back to randomized delay
  2018-07-09 14:47:39.660 [info] &lt;0.193.0&gt; Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay.
  2018-07-09 14:47:39.665 [info] &lt;0.193.0&gt; Failed to get nodes from k8s - {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},
                   {inet,[inet],nxdomain}]}
  2018-07-09 14:47:39.665 [error] &lt;0.192.0&gt; CRASH REPORT Process &lt;0.192.0&gt; with 0 neighbours exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n                 {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 164 in application_master:init/4 line 134
  2018-07-09 14:47:39.666 [info] &lt;0.33.0&gt; Application rabbit exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n                 {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 164
  {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,\"{failed_connect,[{to_address,{\\"kubernetes.default.svc.cluster.local\\",443}},\n                 {inet,[inet],nxdomain}]}\"}},[{rabbit_mnesia,init_from_config,0,[{file,\"src/rabbit_mnesia.erl\"},{line,164}]},{rabbit_mnesia,init_with_lock,3,[{file,\"src/rabbit_mnesia.erl\"},{line,144}]},{rabbit_mnesia,init,0,[{file,\"src/rabbit_mnesia.erl\"},{line,111}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,793}]}]}}}}}"}
  Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,"{failed_connect,[{to_address,{\"kubernetes.defau
  
  Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
  </code></pre>
  
  <p>Kubernetes API service </p>
  
  <pre><code>[node1 ~]# kubectl describe service kubernetes
  Name:              kubernetes
  Namespace:         default
  Labels:            component=apiserver
                  provider=kubernetes
  Annotations:       &lt;none&gt;
  Selector:          &lt;none&gt;
  Type:              ClusterIP
  IP:                10.233.0.1
  Port:              https  443/TCP
  TargetPort:        6443/TCP
  Endpoints:         10.237.34.19:6443,10.237.34.21:6443
  Session Affinity:  ClientIP
  Events:            &lt;none&gt;
  </code></pre>
  
  <p>nslookup from a busybox in the same kubernetes cluster</p>
  
  <pre><code>[node2 ~]# kubectl exec -it busybox -- sh
  / # nslookup  kubernetes.default.svc.cluster.local
  Server:    10.233.0.3
  Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local
  
  Name:      kubernetes.default.svc.cluster.local
  Address 1: 10.233.0.1 kubernetes.default.svc.cluster.local
  </code></pre>
  
  <p>Please let me know if I am missing anything that could help troubleshooting.</p>