{"action":"opened","issue":{"url":"https://api.github.com/repos/StackStorm/st2-docker/issues/133","repository_url":"https://api.github.com/repos/StackStorm/st2-docker","labels_url":"https://api.github.com/repos/StackStorm/st2-docker/issues/133/labels{/name}","comments_url":"https://api.github.com/repos/StackStorm/st2-docker/issues/133/comments","events_url":"https://api.github.com/repos/StackStorm/st2-docker/issues/133/events","html_url":"https://github.com/StackStorm/st2-docker/issues/133","id":325922907,"number":133,"title":"Mistral fails to start with - password authentication failed for user","user":{"login":"nmaludy","id":3693851,"avatar_url":"https://avatars1.githubusercontent.com/u/3693851?v=4","gravatar_id":"","url":"https://api.github.com/users/nmaludy","html_url":"https://github.com/nmaludy","followers_url":"https://api.github.com/users/nmaludy/followers","following_url":"https://api.github.com/users/nmaludy/following{/other_user}","gists_url":"https://api.github.com/users/nmaludy/gists{/gist_id}","starred_url":"https://api.github.com/users/nmaludy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmaludy/subscriptions","organizations_url":"https://api.github.com/users/nmaludy/orgs","repos_url":"https://api.github.com/users/nmaludy/repos","events_url":"https://api.github.com/users/nmaludy/events{/privacy}","received_events_url":"https://api.github.com/users/nmaludy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-24T00:36:28Z","updated_at":"2018-05-24T00:36:28Z","closed_at":null,"author_association":"NONE","body":"**Describe the problem**\r\nThe `mistral-api` and `mistral-server` services fail to start in the `stackstorm` container. The\r\n\r\n**Versions**\r\n - Host OS: [e.g. Ubuntu 16.04]\r\n - docker: [e.g. 18.03.1-ce]\r\n - docker-compose: [e.g. 1.21.1]\r\n - stackstorm/stackstorm image: [e.g. v2.7.1]\r\n\r\n\r\n**To Reproduce**\r\nFresh clone of `st2-docker`, follow the normal steps in the TLDR section in the readme.\r\n\r\n```\r\ngit clone git@github.com:stackstorm/st2-docker\r\ncd st2-docker\r\nmake env\r\ndocker-compose up -d\r\ndocker-compose exec stackstorm bash\r\n```\r\n\r\n** Debugging info **\r\n\r\n#### StackStorm container\r\n\r\nNoticed that `mistral-api` and `mistral-server` weren't running:\r\n\r\n```\r\nroot@faf971c95e40:/# st2ctl status\r\n##### st2 components status #####\r\nst2actionrunner PID: 95\r\nst2actionrunner PID: 103\r\nst2actionrunner PID: 108\r\nst2actionrunner PID: 114\r\nst2actionrunner PID: 129\r\nst2actionrunner PID: 146\r\nst2actionrunner PID: 156\r\nst2actionrunner PID: 164\r\nst2api PID: 57\r\nst2api PID: 289\r\nst2stream PID: 59\r\nst2stream PID: 298\r\nst2auth PID: 48\r\nst2auth PID: 290\r\nst2garbagecollector PID: 46\r\nst2notifier PID: 52\r\nst2resultstracker PID: 50\r\nst2rulesengine PID: 55\r\nst2sensorcontainer PID: 43\r\nst2chatops is not running.\r\nmistral-server is not running.\r\nmistral.api is not running.\r\n```\r\n\r\nTrying to restart the service didn't help.\r\n\r\nLooking at the logs `/var/log/mistral/mistral-server.log` the following error is present multiple times:\r\n\r\n```\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service [-] Error starting thread.: DBError: Failed to setup database: (psycopg2.OperationalError) FATAL:  password authentication failed for user \"mistral-user\"\r\n (Background on this error at: http://sqlalche.me/e/e3q8)\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service Traceback (most recent call last):\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service   File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_service/service.py\", line 729, in run_service\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service     service.start()\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service   File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/engine/engine_server.py\", line 49, in start\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service     db_api.setup_db()\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service   File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/db/v2/api.py\", line 29, in setup_db\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service     IMPL.setup_db()\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service   File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/api.py\", line 73, in setup_db\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service     raise exc.DBError(\"Failed to setup database: %s\" % e)\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service DBError: Failed to setup database: (psycopg2.OperationalError) FATAL:  password authentication failed for user \"mistral-user\"\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service  (Background on this error at: http://sqlalche.me/e/e3q8)\r\n2018-05-24 00:11:50.878 929 ERROR oslo_service.service \r\n```\r\n\r\nChecking `conf/postgres.env`:\r\n\r\n```\r\n$ cat conf/postgres.env \r\nPOSTGRES_USER=mistral-user\r\nPOSTGRES_PASSWORD=Af8zZoL24vQtAnQIIn2XjaHC\r\nPOSTGRES_HOST=postgres\r\nPOSTGRES_PORT=5432\r\nPOSTGRES_DB=mistral\r\n```\r\n\r\nComparing against `/etc/mistral/mistral.conf` looks the same\r\n\r\n```\r\nroot@faf971c95e40:/# cat /etc/mistral/mistral.conf  | grep postg\r\nconnection = postgresql+psycopg2://mistral-user:Af8zZoL24vQtAnQIIn2XjaHC@postgres:5432/mistral\r\n```\r\n\r\n#### Postgres container\r\n\r\nLogging into the database:\r\n\r\n```\r\nsu - postgres\r\npsql\r\n```\r\n\r\nMaking sure the user was created:\r\n\r\n```\r\npostgres=# select * from pg_authid ;\r\n      rolname      | rolsuper | rolinherit | rolcreaterole | rolcreatedb | rolca\r\nnlogin | rolreplication | rolbypassrls | rolconnlimit |             rolpassword \r\n            | rolvaliduntil \r\n-------------------+----------+------------+---------------+-------------+------\r\n-------+----------------+--------------+--------------+-------------------------\r\n------------+---------------\r\n postgres          | t        | t          | t             | t           | t    \r\n       | t              | t            |           -1 |                         \r\n            | \r\n pg_signal_backend | f        | t          | f             | f           | f    \r\n       | f              | f            |           -1 |                         \r\n            | \r\n mistral-user      | t        | t          | f             | f           | t    \r\n       | f              | f            |           -1 | md50d27f1a3b9d45e2fa8942\r\n61a0ab1a598 | \r\n(3 rows)\r\n```\r\n\r\nWhen i changed the password to the new one, the md5sum changed for the password hash:\r\n\r\n```\r\npostgres=# ALTER USER \"mistral-user\" WITH PASSWORD 'Af8zZoL24vQtAnQIIn2XjaHC';\r\nALTER ROLE\r\npostgres=# select * from pg_authid ;\r\n      rolname      | rolsuper | rolinherit | rolcreaterole | rolcreatedb | rolca\r\nnlogin | rolreplication | rolbypassrls | rolconnlimit |             rolpassword \r\n            | rolvaliduntil \r\n-------------------+----------+------------+---------------+-------------+------\r\n-------+----------------+--------------+--------------+-------------------------\r\n------------+---------------\r\n postgres          | t        | t          | t             | t           | t    \r\n       | t              | t            |           -1 |                         \r\n            | \r\n pg_signal_backend | f        | t          | f             | f           | f    \r\n       | f              | f            |           -1 |                         \r\n            | \r\n mistral-user      | t        | t          | f             | f           | t    \r\n       | f              | f            |           -1 | md5c68d82ac9f0e4f8b67274\r\n69a599748ef | \r\n(3 rows)\r\n```\r\n\r\nI changed it a second time to see if the md5sum changed, and it did NOT. This leads me to believe that the password sent in initially was incorrect.\r\n\r\n```\r\npostgres=# ALTER USER \"mistral-user\" WITH PASSWORD 'Af8zZoL24vQtAnQIIn2XjaHC';\r\nALTER ROLE\r\npostgres=# select * from pg_authid ;\r\n      rolname      | rolsuper | rolinherit | rolcreaterole | rolcreatedb | rolca\r\nnlogin | rolreplication | rolbypassrls | rolconnlimit |             rolpassword \r\n            | rolvaliduntil \r\n-------------------+----------+------------+---------------+-------------+------\r\n-------+----------------+--------------+--------------+-------------------------\r\n------------+---------------\r\n postgres          | t        | t          | t             | t           | t    \r\n       | t              | t            |           -1 |                         \r\n            | \r\n pg_signal_backend | f        | t          | f             | f           | f    \r\n       | f              | f            |           -1 |                         \r\n            | \r\n mistral-user      | t        | t          | f             | f           | t    \r\n       | f              | f            |           -1 | md5c68d82ac9f0e4f8b67274\r\n69a599748ef | \r\n(3 rows)\r\n```\r\n\r\n#### Afher changing the password\r\n\r\nRestarted the mistral service and it connected immediately and no longer reported as not running in `st2ctl status`"}}
{"actor":{"display_login":"nmaludy"}}