{"action":"created","issue":{"url":"https://api.github.com/repos/ansible/awx/issues/2616","repository_url":"https://api.github.com/repos/ansible/awx","labels_url":"https://api.github.com/repos/ansible/awx/issues/2616/labels{/name}","comments_url":"https://api.github.com/repos/ansible/awx/issues/2616/comments","events_url":"https://api.github.com/repos/ansible/awx/issues/2616/events","html_url":"https://github.com/ansible/awx/issues/2616","id":377922061,"node_id":"MDU6SXNzdWUzNzc5MjIwNjE=","number":2616,"title":"AWX containers don't communicate with an external postgresql server","user":{"login":"jean-christophe-manciot","id":13176858,"node_id":"MDQ6VXNlcjEzMTc2ODU4","avatar_url":"https://avatars3.githubusercontent.com/u/13176858?v=4","gravatar_id":"","url":"https://api.github.com/users/jean-christophe-manciot","html_url":"https://github.com/jean-christophe-manciot","followers_url":"https://api.github.com/users/jean-christophe-manciot/followers","following_url":"https://api.github.com/users/jean-christophe-manciot/following{/other_user}","gists_url":"https://api.github.com/users/jean-christophe-manciot/gists{/gist_id}","starred_url":"https://api.github.com/users/jean-christophe-manciot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jean-christophe-manciot/subscriptions","organizations_url":"https://api.github.com/users/jean-christophe-manciot/orgs","repos_url":"https://api.github.com/users/jean-christophe-manciot/repos","events_url":"https://api.github.com/users/jean-christophe-manciot/events{/privacy}","received_events_url":"https://api.github.com/users/jean-christophe-manciot/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-06T16:16:08Z","updated_at":"2018-11-06T16:16:44Z","closed_at":null,"author_association":"NONE","body":"##### ISSUE TYPE\r\n - Bug Report\r\n\r\n##### COMPONENT NAME\r\n - Installer\r\n\r\n##### SUMMARY\r\nNo AWX containers communicate with postgresql when it is setup as an external database server..\r\n\r\n##### ENVIRONMENT\r\n* AWX version: 2.1.0\r\n* AWX install method: docker on linux\r\n* Ansible version:  2.7.1\r\n* Operating System: Debian buster\r\n* Web Browser: chrome 70.0.3538.77 (Official Build) (64-bit)\r\n\r\n##### STEPS TO REPRODUCE\r\n\r\n- install all dependencies:\r\n   + ```apt install docker-ce``` from docker repo (deb [arch=amd64] https://download.docker.com/linux/debian buster stable edge #Docker)\r\n   + ```pip2 install -I docker-compose```\r\n   + ```pip2 install -I requests==2.18.1``` (2.18.1 as a workaround to issue https://github.com/docker/docker-py/issues/1502#issuecomment-374565304)\r\n   + ```apt install postgresql```\r\n   + create awx postgresql database with parameters matching the ones listed below\r\n- checkout awx sources at 2.1.0\r\n- build awx as non-root from scratch with the following setup in installer/inventory:\r\n   + pg_hostname=127.0.0.1\r\n   + pg_username=postgres\r\n   + pg_password=xxxxxxxxxxxxxxx\r\n   + pg_database=awx\r\n   + pg_port=5432\r\n   + docker_compose_dir=/path/to/awx/docker-compose\r\n   + use_docker_compose=true\r\n   + project_data_dir=/path/to/awx/projects\"\r\n- cd installer\r\n- ansible-playbook -i inventory install.yml\r\n\r\n##### EXPECTED RESULTS\r\n\r\nAccess to AWX login page\r\n\r\n##### ACTUAL RESULTS\r\n\r\n- \"A server error has occurred\" on login page\r\n- The awx containers are correctly created:\r\n```\r\nf3ad1b1c8e66        ansible/awx_task:2.1.0       \"/tini -- /bin/sh -c…\"   37 minutes ago      Up 37 minutes               8052/tcp                                              docker-compose_task_1_193d23c0d5bc\r\n7d599a070a18        ansible/awx_web:2.1.0        \"/tini -- /bin/sh -c…\"   37 minutes ago      Up 37 minutes               0.0.0.0:80->8052/tcp                                  docker-compose_web_1_cbd28ec65580\r\na026d7c26069        ansible/awx_rabbitmq:3.7.4   \"docker-entrypoint.s…\"   37 minutes ago      Up 37 minutes               4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   docker-compose_rabbitmq_1_ca7688aefab3\r\n5e8283624478        memcached:alpine             \"docker-entrypoint.s…\"   37 minutes ago      Up 37 minutes               11211/tcp                                             docker-compose_memcached_1_d990b0d5e4b5\r\n```\r\n- docker service seems OK:\r\n```\r\n# echo 'q' | systemctl --no-pager --full status docker\r\n● docker.service - Docker Application Container Engine\r\n   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)\r\n   Active: active (running) since Tue 2018-11-06 10:43:50 CET; 6h ago\r\n     Docs: https://docs.docker.com\r\n  Process: 13363 ExecReload=/bin/kill -s HUP $MAINPID (code=exited, status=0/SUCCESS)\r\n Main PID: 6838 (dockerd)\r\n    Tasks: 100\r\n   Memory: 209.2M\r\n   CGroup: /system.slice/docker.service\r\n           ├─ 6838 /usr/bin/dockerd -H fd://\r\n           ├─ 6850 docker-containerd --config /var/run/docker/containerd/containerd.toml\r\n           ├─14438 docker-containerd-shim -namespace moby -workdir /media/actionmystique/SAMSUNG5-Virtual/Docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/5e8283624478bd15e3b820aa16c31b98378d2d96d78ff9db4ac8af9e070ebd92 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc -debug\r\n           ├─14490 docker-containerd-shim -namespace moby -workdir /media/actionmystique/SAMSUNG5-Virtual/Docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/a026d7c260690cef37cf50b4524333b3f97003302632e427b06ca2a1341d8ad1 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc -debug\r\n           ├─14501 memcached\r\n           ├─14516 /bin/sh -c /launch.sh\r\n           ├─14676 /bin/sh /launch.sh\r\n           ├─14679 /bin/sh /opt/rabbitmq/sbin/rabbitmq-server\r\n           ├─14870 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.22.0.4 -container-port 8052\r\n           ├─14876 docker-containerd-shim -namespace moby -workdir /media/actionmystique/SAMSUNG5-Virtual/Docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/7d599a070a182b9757ec9780458cabf247e7962d2bb8452a4d6d93e99b946836 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc -debug\r\n           ├─14894 /tini -- /bin/sh -c /usr/bin/launch_awx.sh\r\n           ├─14965 bash /usr/bin/launch_awx.sh\r\n           ├─15000 docker-containerd-shim -namespace moby -workdir /media/actionmystique/SAMSUNG5-Virtual/Docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/f3ad1b1c8e660b1afe2d571cf03fa570dc2dfb159eb411632fad61a84884965e -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc -debug\r\n           ├─15021 /tini -- /bin/sh -c /usr/bin/launch_awx_task.sh\r\n           ├─15065 /usr/lib/erlang/erts-9.1.5/bin/epmd -daemon\r\n           ├─15143 bash /usr/bin/launch_awx_task.sh\r\n           ├─15369 /usr/lib/erlang/erts-9.1.5/bin/beam.smp -W w -A 128 -P 1048576 -t 5000000 -stbt db -zdbbl 1280000 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /opt/rabbitmq/ebin -noshell -noinput -s rabbit boot -sname rabbit@a026d7c26069 -boot start_sasl -conf /etc/rabbitmq/rabbitmq -conf_dir /var/lib/rabbitmq/config -conf_script_dir /opt/rabbitmq/sbin -conf_schema_dir /var/lib/rabbitmq/schema -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger tty -rabbit lager_log_root \"/var/log/rabbitmq\" -rabbit lager_default_file tty -rabbit lager_upgrade_file tty -rabbit enabled_plugins_file \"/etc/rabbitmq/enabled_plugins\" -rabbit plugins_dir \"/opt/rabbitmq/plugins\" -rabbit plugins_expand_dir \"/var/lib/rabbitmq/mnesia/rabbit@a026d7c26069-plugins-expand\" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir \"/var/lib/rabbitmq/mnesia/rabbit@a026d7c26069\" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672\r\n           ├─15553 erl_child_setup 1048576\r\n           ├─15613 inet_gethost 4\r\n           ├─15614 inet_gethost 4\r\n           ├─16387 /usr/bin/python2 /usr/bin/supervisord -c /supervisor.conf\r\n           ├─16394 python /usr/bin/config-watcher\r\n           ├─16395 /var/lib/awx/venv/awx/bin/python2 /var/lib/awx/venv/awx/bin/daphne -b 127.0.0.1 -p 8051 awx.asgi:channel_layer\r\n           ├─16396 nginx: master process nginx -g daemon off;\r\n           ├─16397 /var/lib/awx/venv/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --vacuum --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768\r\n           ├─16398 nginx: worker process\r\n           ├─16399 /var/lib/awx/venv/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --vacuum --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768\r\n           ├─16400 /var/lib/awx/venv/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --vacuum --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768\r\n           ├─16401 /var/lib/awx/venv/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --vacuum --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768\r\n           ├─16402 /var/lib/awx/venv/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --vacuum --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768\r\n           ├─16403 /var/lib/awx/venv/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --vacuum --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768\r\n           ├─16514 /usr/bin/python2 /usr/bin/supervisord -c /supervisor_task.conf\r\n           ├─16519 python /usr/bin/config-watcher\r\n           ├─16520 python /usr/bin/awx-manage runworker --only-channels websocket.*\r\n           ├─16521 python /usr/bin/awx-manage run_callback_receiver\r\n           ├─16605 python /usr/bin/awx-manage run_callback_receiver\r\n           ├─16606 python /usr/bin/awx-manage run_callback_receiver\r\n           ├─16607 python /usr/bin/awx-manage run_callback_receiver\r\n           ├─16608 python /usr/bin/awx-manage run_callback_receiver\r\n           └─29622 python /usr/bin/awx-manage run_dispatcher\r\n\r\nNov 06 16:54:48 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:48.499211280+01:00\" level=debug msg=\"Name To resolve: memcached.\"\r\nNov 06 16:54:48 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:48.499259459+01:00\" level=debug msg=\"[resolver] lookup for memcached.: IP [172.22.0.2]\"\r\nNov 06 16:54:51 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:51.272436873+01:00\" level=debug msg=\"Name To resolve: memcached.\"\r\nNov 06 16:54:51 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:51.272472602+01:00\" level=debug msg=\"[resolver] lookup for memcached.: IP [172.22.0.2]\"\r\nNov 06 16:54:54 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:54.014787217+01:00\" level=debug msg=\"Name To resolve: memcached.\"\r\nNov 06 16:54:54 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:54.014843252+01:00\" level=debug msg=\"[resolver] lookup for memcached.: IP [172.22.0.2]\"\r\nNov 06 16:54:56 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:56.767222258+01:00\" level=debug msg=\"Name To resolve: memcached.\"\r\nNov 06 16:54:56 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:56.767257220+01:00\" level=debug msg=\"[resolver] lookup for memcached.: IP [172.22.0.2]\"\r\nNov 06 16:54:59 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:59.478015673+01:00\" level=debug msg=\"Name To resolve: memcached.\"\r\nNov 06 16:54:59 samsung5-debian dockerd[6838]: time=\"2018-11-06T16:54:59.478069732+01:00\" level=debug msg=\"[resolver] lookup for memcached.: IP [172.22.0.2]\"\r\n```\r\n- **no container seems to try to access postgresql TCP port**: cf. pcap trace below.\r\n\r\n##### ADDITIONAL INFORMATION\r\n\r\n**Login page**:\r\n![awx server error](https://user-images.githubusercontent.com/13176858/48075781-3bfc8c80-e1e4-11e8-8ad1-5e0322403ac1.png)\r\n**pcap trace**: https://drive.google.com/file/d/162zFvknyYQKL8ZAQ-4aJYnSOkkW21pjk/view?usp=sharing\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/ansible/awx/issues/comments/436311903","html_url":"https://github.com/ansible/awx/issues/2616#issuecomment-436311903","issue_url":"https://api.github.com/repos/ansible/awx/issues/2616","id":436311903,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjMxMTkwMw==","user":{"login":"jean-christophe-manciot","id":13176858,"node_id":"MDQ6VXNlcjEzMTc2ODU4","avatar_url":"https://avatars3.githubusercontent.com/u/13176858?v=4","gravatar_id":"","url":"https://api.github.com/users/jean-christophe-manciot","html_url":"https://github.com/jean-christophe-manciot","followers_url":"https://api.github.com/users/jean-christophe-manciot/followers","following_url":"https://api.github.com/users/jean-christophe-manciot/following{/other_user}","gists_url":"https://api.github.com/users/jean-christophe-manciot/gists{/gist_id}","starred_url":"https://api.github.com/users/jean-christophe-manciot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jean-christophe-manciot/subscriptions","organizations_url":"https://api.github.com/users/jean-christophe-manciot/orgs","repos_url":"https://api.github.com/users/jean-christophe-manciot/repos","events_url":"https://api.github.com/users/jean-christophe-manciot/events{/privacy}","received_events_url":"https://api.github.com/users/jean-christophe-manciot/received_events","type":"User","site_admin":false},"created_at":"2018-11-06T16:16:44Z","updated_at":"2018-11-06T16:16:44Z","author_association":"NONE","body":"Everything is fine when AWX is setup with its own postgresql container."}}
{"actor":{"display_login":"jean-christophe-manciot"}}