{"action":"closed","issue":{"url":"https://api.github.com/repos/ansible/awx/issues/2201","repository_url":"https://api.github.com/repos/ansible/awx","labels_url":"https://api.github.com/repos/ansible/awx/issues/2201/labels{/name}","comments_url":"https://api.github.com/repos/ansible/awx/issues/2201/comments","events_url":"https://api.github.com/repos/ansible/awx/issues/2201/events","html_url":"https://github.com/ansible/awx/issues/2201","id":354384258,"node_id":"MDU6SXNzdWUzNTQzODQyNTg=","number":2201,"title":"Rabbitmq container in AWX pod failing to start - Failed to get nodes from k8s with NXDOMAIN","user":{"login":"krisayala","id":2204340,"node_id":"MDQ6VXNlcjIyMDQzNDA=","avatar_url":"https://avatars2.githubusercontent.com/u/2204340?v=4","gravatar_id":"","url":"https://api.github.com/users/krisayala","html_url":"https://github.com/krisayala","followers_url":"https://api.github.com/users/krisayala/followers","following_url":"https://api.github.com/users/krisayala/following{/other_user}","gists_url":"https://api.github.com/users/krisayala/gists{/gist_id}","starred_url":"https://api.github.com/users/krisayala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krisayala/subscriptions","organizations_url":"https://api.github.com/users/krisayala/orgs","repos_url":"https://api.github.com/users/krisayala/repos","events_url":"https://api.github.com/users/krisayala/events{/privacy}","received_events_url":"https://api.github.com/users/krisayala/received_events","type":"User","site_admin":false},"labels":[{"id":687979964,"node_id":"MDU6TGFiZWw2ODc5Nzk5NjQ=","url":"https://api.github.com/repos/ansible/awx/labels/component:installer","name":"component:installer","color":"d4c5f9","default":false},{"id":687172858,"node_id":"MDU6TGFiZWw2ODcxNzI4NTg=","url":"https://api.github.com/repos/ansible/awx/labels/state:needs_triage","name":"state:needs_triage","color":"A0522D","default":false},{"id":607283748,"node_id":"MDU6TGFiZWw2MDcyODM3NDg=","url":"https://api.github.com/repos/ansible/awx/labels/type:bug","name":"type:bug","color":"F79F81","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-08-27T16:19:20Z","updated_at":"2018-10-26T02:31:37Z","closed_at":"2018-10-26T02:31:37Z","author_association":"NONE","body":"##### ISSUE TYPE\r\n - Bug Report\r\n\r\n##### COMPONENT NAME\r\n<!-- Pick the area of AWX for this issue, you can have multiple, delete the rest: -->\r\n - Installer\r\n\r\n##### SUMMARY\r\n<!-- Briefly describe the problem. -->\r\nRabbitmq fails to start when deploy AWX on Kubernetes.\r\n\r\n`Failed to get nodes from k8s - {failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\r\n                 {inet,[inet],nxdomain}]}\r\n`\r\n\r\n\r\n##### ENVIRONMENT\r\n* AWX version: 1.0.7\r\n* AWX install method: Kubernetes 1.10.4\r\n* Ansible version:  2.4.2.0\r\n* Operating System: RHEL 7\r\n* Web Browser: Chrome Version 68.0.3440.106\r\n\r\n##### STEPS TO REPRODUCE\r\n\r\n<!-- Please describe exactly how to reproduce the problem. -->\r\nWhen following the instructions on the [install guide](https://github.com/ansible/awx/blob/devel/INSTALL.md#kubernetes) for Kubernetes.\r\n\r\nI am running `ansible-playbook -i inventory install.yml`  and rabbitmq ends up in a bad state unable to connect to the kubernetes dns service.\r\n\r\n`  Normal   Pulling                6m (x2 over 6m)    kubelet, vl3091oma1  pulling image \"ansible/awx_rabbitmq:3.7.4\"\r\n  Normal   Pulled                 6m (x2 over 6m)    kubelet, vl3091oma1  Successfully pulled image \"ansible/awx_rabbitmq:3.7.4\"\r\n  Normal   Created                6m (x2 over 6m)    kubelet, vl3091oma1  Created container\r\n  Normal   Started                6m (x2 over 6m)    kubelet, vl3091oma1  Started container\r\n  Warning  BackOff                30s (x27 over 6m)  kubelet, vl3091oma1  Back-off restarting failed container\r\n`\r\n\r\n```\r\n[vm ~]# kubectl get all -n awxmine\r\nNAME                      READY     STATUS             RESTARTS   AGE\r\npod/awx-df8946fb6-jp2fj   3/4       CrashLoopBackOff   161        13h\r\n\r\nNAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE\r\nservice/awx-rmq-mgmt   ClusterIP   10.233.46.253   <none>        15672/TCP                        13h\r\nservice/awx-web-svc    NodePort    10.233.59.55    <none>        80:30905/TCP                     13h\r\nservice/rabbitmq       NodePort    10.233.62.224   <none>        15672:30175/TCP,5672:30506/TCP   13h\r\n\r\nNAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE\r\ndeployment.apps/awx   1         1         1            0           13h\r\n\r\nNAME                            DESIRED   CURRENT   READY     AGE\r\nreplicaset.apps/awx-df8946fb6   1         1         0         13h\r\n```\r\n\r\n##### EXPECTED RESULTS\r\n\r\n<!-- What did you expect to happen when running the steps above? -->\r\n```\r\n[vm ~]# kubectl get all -n awxmine\r\nNAME                      READY     STATUS             RESTARTS   AGE\r\npod/awx-df8946fb6-jp2fj   4/4       Ready           0               13h\r\n```\r\n\r\n##### ACTUAL RESULTS\r\n\r\n<!-- What actually happened? -->\r\nRabbitmq is throwing the following error.\r\n\r\n```\r\n[vm ~]# kubectl logs -n awxmine awx-df8946fb6-nz9ck awx-rabbit\r\n2018-08-27 15:46:06.331 [info] <0.33.0> Application lager started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.591 [info] <0.33.0> Application os_mon started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.591 [info] <0.33.0> Application crypto started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.592 [info] <0.33.0> Application cowlib started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.592 [info] <0.33.0> Application xmerl started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.669 [info] <0.33.0> Application mnesia started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.669 [info] <0.33.0> Application recon started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.669 [info] <0.33.0> Application jsx started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.669 [info] <0.33.0> Application asn1 started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.669 [info] <0.33.0> Application public_key started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.712 [info] <0.33.0> Application ssl started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.715 [info] <0.33.0> Application ranch started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.715 [info] <0.33.0> Application ranch_proxy_protocol started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.715 [info] <0.33.0> Application rabbit_common started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.722 [info] <0.33.0> Application amqp_client started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.723 [info] <0.33.0> Application cowboy started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.770 [info] <0.33.0> Application inets started on node 'rabbit@10.233.96.2'\r\n2018-08-27 15:46:06.777 [info] <0.193.0>\r\n Starting RabbitMQ 3.7.4 on Erlang 20.1.7\r\n Copyright (C) 2007-2018 Pivotal Software, Inc.\r\n Licensed under the MPL.  See http://www.rabbitmq.com/\r\n\r\n  ##  ##\r\n  ##  ##      RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc.\r\n  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/\r\n  ######  ##\r\n  ##########  Logs: <stdout>\r\n\r\n              Starting broker...\r\n2018-08-27 15:46:06.794 [info] <0.193.0>\r\n node           : rabbit@10.233.96.2\r\n home dir       : /var/lib/rabbitmq\r\n config file(s) : /etc/rabbitmq/rabbitmq.conf\r\n cookie hash    : at619UOZzsenF44tSK3ulA==\r\n log(s)         : <stdout>\r\n database dir   : /var/lib/rabbitmq/mnesia/rabbit@10.233.96.2\r\n2018-08-27 15:46:08.255 [info] <0.201.0> Memory high watermark set to 25706 MiB (26955354931 bytes) of 64266 MiB (67388387328 bytes) total\r\n2018-08-27 15:46:08.258 [info] <0.203.0> Enabling free disk space monitoring\r\n2018-08-27 15:46:08.258 [info] <0.203.0> Disk free limit set to 50MB\r\n2018-08-27 15:46:08.260 [info] <0.205.0> Limiting to approx 1048476 file handles (943626 sockets)\r\n2018-08-27 15:46:08.260 [info] <0.206.0> FHC read buffering:  OFF\r\n2018-08-27 15:46:08.260 [info] <0.206.0> FHC write buffering: ON\r\n2018-08-27 15:46:08.262 [info] <0.193.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@10.233.96.2 is empty. Assuming we need to join an existing cluster or initialise from scratch...\r\n2018-08-27 15:46:08.262 [info] <0.193.0> Configured peer discovery backend: rabbit_peer_discovery_k8s\r\n2018-08-27 15:46:08.262 [info] <0.193.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s\r\n2018-08-27 15:46:08.262 [info] <0.193.0> Peer discovery backend does not support locking, falling back to randomized delay\r\n2018-08-27 15:46:08.262 [info] <0.193.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay.\r\n2018-08-27 15:46:08.266 [info] <0.193.0> Failed to get nodes from k8s - {failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\r\n                 {inet,[inet],nxdomain}]}\r\n2018-08-27 15:46:08.266 [error] <0.192.0> CRASH REPORT Process <0.192.0> with 0 neighbours exited with reason: no case clause matching {error,\"{failed_connect,[{to_address,{\\\"kubernetes.default.svc.cluster.local\\\",443}},\\n                 {inet,[inet]on_master:init/4 line 134\r\n2018-08-27 15:46:08.266 [info] <0.33.0> Application rabbit exited with reason: no case clause matching {error,\"{failed_connect,[{to_address,{\\\"kubernetes.default.svc.cluster.local\\\",443}},\\n                 {inet,[inet],nxdomain}]}\"} in rabbit_mnesia:\r\n{\"Kernel pid terminated\",application_controller,\"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,\\\"{failed_connect,[{to_address,{\\\\\"kubernetes.default.svc.cluster.local\\\\\",443}},\\n                \\\"src/rabbit_mnesia.erl\\\"},{line,164}]},{rabbit_mnesia,init_with_lock,3,[{file,\\\"src/rabbit_mnesia.erl\\\"},{line,144}]},{rabbit_mnesia,init,0,[{file,\\\"src/rabbit_mnesia.erl\\\"},{line,111}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\\\"src/rabbsrc/rabbit_boot_steps.erl\\\"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\\\"src/rabbit_boot_steps.erl\\\"},{line,26}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\\\"src/rabbit_boot_steps.erl\\\"},{line,26}]},{rabbit,start,2,[{file\r\nKernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,\"{failed_connect,[{to_address,{\\\"kubernetes.defau\r\n\r\nCrash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done\r\n```\r\n\r\n\r\n##### ADDITIONAL INFORMATION\r\n\r\n<!-- Include any links to sosreport, database dumps, screenshots or other\r\ninformation. -->\r\n\r\nI am using the following for the inventory file.\r\n\r\ndockerhub_base=ansible\r\ndockerhub_version=latest\r\n\r\nhttp_proxy=<corp_proxy>\r\nhttps_proxy=<corp_proxy>\r\nno_proxy=<corp_dns_names>,*cluster.local,127.0.0.1\r\n\r\nI am unable to even get into the rabbitmq container to even troubleshoot it.  But I have deployed https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/tree/master/examples/k8s_statefulsets just to verify that the rabbitmq-peer-discovery-k8s plugin is working.  The example works fine and is able to connect using the same setting the AWX rabbitmq pod is using.  Which leads me to believe that something is wonky with the AWX rabbitmq container.\r\n\r\nI am thinking maybe its something with how I have the proxys setup and how the rabbitmq pod is inheriting them or something?  Since the kube-dns service is working and its just rabbitmq that is complaining about the nxdomain.\r\n\r\n```\r\n[vm ~]# kubectl get all --all-namespaces\r\nNAMESPACE     NAME                                        READY     STATUS    RESTARTS   AGE\r\nawxmine       pod/awx-df8946fb6-nz9ck                     3/4       Running   22         55m\r\ndefault       pod/busybox                                 1/1       Running   21         21h\r\nkube-system   pod/kube-apiserver-vl3090oma1               1/1       Running   0          2d\r\nkube-system   pod/kube-controller-manager-vl3090oma1      1/1       Running   0          2d\r\nkube-system   pod/kube-dns-69f4c8fc58-lk97q               3/3       Running   0          2d\r\nkube-system   pod/kube-dns-69f4c8fc58-mzsk7               3/3       Running   0          2d\r\nkube-system   pod/kube-proxy-vl3090oma1                   1/1       Running   0          2d\r\nkube-system   pod/kube-proxy-vl3091oma1                   1/1       Running   0          2d\r\nkube-system   pod/kube-proxy-vl3092oma1                   1/1       Running   0          2d\r\nkube-system   pod/kube-scheduler-vl3090oma1               1/1       Running   0          2d\r\nkube-system   pod/kubedns-autoscaler-565b49bbc6-mxnqd     1/1       Running   0          2d\r\nkube-system   pod/kubernetes-dashboard-6d4dfd56cb-9psvq   1/1       Running   0          2d\r\nkube-system   pod/nginx-proxy-vl3091oma1                  1/1       Running   0          2d\r\nkube-system   pod/nginx-proxy-vl3092oma1                  1/1       Running   0          2d\r\nkube-system   pod/weave-net-9zjps                         2/2       Running   0          2d\r\nkube-system   pod/weave-net-dlxdk                         2/2       Running   0          2d\r\nkube-system   pod/weave-net-hr4r6                         2/2       Running   0          2d\r\n\r\nNAMESPACE     NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE\r\nawxmine       service/awx-rmq-mgmt           ClusterIP   10.233.46.253   <none>        15672/TCP                        14h\r\nawxmine       service/awx-web-svc            NodePort    10.233.59.55    <none>        80:30905/TCP                     14h\r\nawxmine       service/rabbitmq               NodePort    10.233.62.224   <none>        15672:30175/TCP,5672:30506/TCP   14h\r\ndefault       service/kubernetes             ClusterIP   10.233.0.1      <none>        443/TCP                          2d\r\nkube-system   service/kube-dns               ClusterIP   10.233.0.3      <none>        53/UDP,53/TCP                    2d\r\nkube-system   service/kubernetes-dashboard   ClusterIP   10.233.31.60    <none>        443/TCP                          2d\r\n\r\nNAMESPACE     NAME                       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE\r\nkube-system   daemonset.apps/weave-net   3         3         3         3            3           <none>          2d\r\n\r\nNAMESPACE     NAME                                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE\r\nawxmine       deployment.apps/awx                    1         1         1            0           56m\r\nkube-system   deployment.apps/kube-dns               2         2         2            2           2d\r\nkube-system   deployment.apps/kubedns-autoscaler     1         1         1            1           2d\r\nkube-system   deployment.apps/kubernetes-dashboard   1         1         1            1           2d\r\n\r\nNAMESPACE     NAME                                              DESIRED   CURRENT   READY     AGE\r\nawxmine       replicaset.apps/awx-df8946fb6                     1         1         0         56m\r\nkube-system   replicaset.apps/kube-dns-69f4c8fc58               2         2         2         2d\r\nkube-system   replicaset.apps/kubedns-autoscaler-565b49bbc6     1         1         1         2d\r\nkube-system   replicaset.apps/kubernetes-dashboard-6d4dfd56cb   1         1         1         2d\r\n\r\n```\r\n\r\n\r\n```\r\n[vm ~]# kubectl describe service kubernetes\r\nName:              kubernetes\r\nNamespace:         default\r\nLabels:            component=apiserver\r\n                   provider=kubernetes\r\nAnnotations:       <none>\r\nSelector:          <none>\r\nType:              ClusterIP\r\nIP:                10.233.0.1\r\nPort:              https  443/TCP\r\nTargetPort:        6443/TCP\r\nEndpoints:         10.237.34.116:6443\r\nSession Affinity:  ClientIP\r\nEvents:            <none>\r\n```\r\n\r\n```\r\n[vm ~]# kubectl exec -it -n awxmine awx-df8946fb6-nz9ck --container=awx-celery -- /bin/bash\r\nbash-4.2$ nslookup kubernetes.default\r\nServer:         10.233.0.3\r\nAddress:        10.233.0.3#53\r\n\r\nNon-authoritative answer:\r\nName:   kubernetes.default.svc.cluster.local\r\nAddress: 10.233.0.1\r\n```\r\n\r\nPlease let me know if you need anymore information to help troubleshoot this issue.\r\n\r\n"}}
{"actor":{"display_login":"matburt"}}