{"action":"created","issue":{"url":"https://api.github.com/repos/curationexperts/ansible-hydra/issues/147","repository_url":"https://api.github.com/repos/curationexperts/ansible-hydra","labels_url":"https://api.github.com/repos/curationexperts/ansible-hydra/issues/147/labels{/name}","comments_url":"https://api.github.com/repos/curationexperts/ansible-hydra/issues/147/comments","events_url":"https://api.github.com/repos/curationexperts/ansible-hydra/issues/147/events","html_url":"https://github.com/curationexperts/ansible-hydra/issues/147","id":165636591,"number":147,"title":"solr 5 restart fails on centos 7 (centos branch)","user":{"login":"acozine","id":879121,"avatar_url":"https://avatars.githubusercontent.com/u/879121?v=3","gravatar_id":"","url":"https://api.github.com/users/acozine","html_url":"https://github.com/acozine","followers_url":"https://api.github.com/users/acozine/followers","following_url":"https://api.github.com/users/acozine/following{/other_user}","gists_url":"https://api.github.com/users/acozine/gists{/gist_id}","starred_url":"https://api.github.com/users/acozine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/acozine/subscriptions","organizations_url":"https://api.github.com/users/acozine/orgs","repos_url":"https://api.github.com/users/acozine/repos","events_url":"https://api.github.com/users/acozine/events{/privacy}","received_events_url":"https://api.github.com/users/acozine/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-07-14T19:14:07Z","updated_at":"2016-07-15T19:08:12Z","closed_at":null,"body":"The deploy role's restart of Solr 5 fails on CentOS 7 with the error:\r\n```\r\nTASK [deploy : restart solr 5.x] ***********************************************\r\nfatal: [23.23.8.207]: FAILED! => {\"changed\": false, \"failed\": true, \"msg\": \"Job for solr.service failed because the control process exited with error code. See \\\"systemctl status solr.service\\\" and \\\"journalctl -xe\\\" for details.\\n\"}\r\n```\r\nIf I ssh into the machine, I see:\r\n```\r\n[centos@ip-10-0-0-182 ~]$ sudo service solr status\r\nFound 1 Solr nodes: \r\nSolr process 4924 running on port 8983\r\n{\r\n  \"solr_home\":\"/var/solr/data\",\r\n  \"version\":\"5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:22:52\",\r\n  \"startTime\":\"2016-07-14T15:19:19.171Z\",\r\n  \"uptime\":\"0 days, 0 hours, 45 minutes, 1 seconds\",\r\n  \"memory\":\"48.5 MB (%9.9) of 490.7 MB\"}\r\n\r\n[centos@ip-10-0-0-182 ~]$ sudo systemctl status solr\r\n● solr.service - LSB: Controls Apache Solr as a Service\r\n   Loaded: loaded (/etc/rc.d/init.d/solr)\r\n   Active: failed (Result: exit-code) since Thu 2016-07-14 11:59:51 EDT; 4min 37s ago\r\n     Docs: man:systemd-sysv-generator(8)\r\n  Process: 27722 ExecStart=/etc/rc.d/init.d/solr start (code=exited, status=1/FAILURE)\r\n\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: Starting LSB: Controls Apache Solr as a Service...\r\nJul 14 11:59:51 ip-10-0-0-182 su[27724]: (to solr) root on none\r\nJul 14 11:59:51 ip-10-0-0-182 solr[27722]: Port 8983 is already being used by another process (pid: 4924)\r\nJul 14 11:59:51 ip-10-0-0-182 solr[27722]: Please choose a different port using the -p option.\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: solr.service: control process exited, code=exited status=1\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: Failed to start LSB: Controls Apache Solr as a Service.\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: Unit solr.service entered failed state.\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: solr.service failed.\r\n```\r\nif I then restart Solr from the command line with `service` the restart succeeds:\r\n```\r\n[centos@ip-10-0-0-182 ~]$ sudo service solr restart\r\nSending stop command to Solr running on port 8983 ... waiting 5 seconds to allow Jetty process 4924 to stop gracefully.\r\nWaiting up to 30 seconds to see Solr running on port 8983 [/]  \r\nStarted Solr server on port 8983 (pid=28848). Happy searching!\r\n```\r\nand ps shows the new PID:\r\n```\r\n[centos@ip-10-0-0-182 ~]$ ps aux | grep 8983\r\nsolr     28848 28.6  1.5 4132516 234680 ?      Sl   12:05   0:06 java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/solr/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/var/solr/data -Dsolr.install.dir=/opt/solr -Dlog4j.configuration=file:/var/solr/log4j.properties -Xss256k -jar start.jar -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/solr/logs --module=http\r\n```\r\nbut systemctl still shows the old PID and the same error message (with an update to how long ago it happened):\r\n```\r\n[centos@ip-10-0-0-182 ~]$ sudo systemctl status solr\r\n● solr.service - LSB: Controls Apache Solr as a Service\r\n   Loaded: loaded (/etc/rc.d/init.d/solr)\r\n   Active: failed (Result: exit-code) since Thu 2016-07-14 11:59:51 EDT; 5min ago\r\n     Docs: man:systemd-sysv-generator(8)\r\n  Process: 27722 ExecStart=/etc/rc.d/init.d/solr start (code=exited, status=1/FAILURE)\r\n\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: Starting LSB: Controls Apache Solr as a Service...\r\nJul 14 11:59:51 ip-10-0-0-182 su[27724]: (to solr) root on none\r\nJul 14 11:59:51 ip-10-0-0-182 solr[27722]: Port 8983 is already being used by another process (pid: 4924)\r\nJul 14 11:59:51 ip-10-0-0-182 solr[27722]: Please choose a different port using the -p option.\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: solr.service: control process exited, code=exited status=1\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: Failed to start LSB: Controls Apache Solr as a Service.\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: Unit solr.service entered failed state.\r\nJul 14 11:59:51 ip-10-0-0-182 systemd[1]: solr.service failed.\r\n``` \r\nI can manually fix the box at the command line by stopping Solr with `sudo service solr stop` and then restarting seems to work with either command (`systemctl` or `service`).\r\nHacky Ansible fix would be to add a centos-only shell task to stop/start instead of restart but I'd rather understand the problem better and fix it right if possible."},"comment":{"url":"https://api.github.com/repos/curationexperts/ansible-hydra/issues/comments/233043860","html_url":"https://github.com/curationexperts/ansible-hydra/issues/147#issuecomment-233043860","issue_url":"https://api.github.com/repos/curationexperts/ansible-hydra/issues/147","id":233043860,"user":{"login":"acozine","id":879121,"avatar_url":"https://avatars.githubusercontent.com/u/879121?v=3","gravatar_id":"","url":"https://api.github.com/users/acozine","html_url":"https://github.com/acozine","followers_url":"https://api.github.com/users/acozine/followers","following_url":"https://api.github.com/users/acozine/following{/other_user}","gists_url":"https://api.github.com/users/acozine/gists{/gist_id}","starred_url":"https://api.github.com/users/acozine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/acozine/subscriptions","organizations_url":"https://api.github.com/users/acozine/orgs","repos_url":"https://api.github.com/users/acozine/repos","events_url":"https://api.github.com/users/acozine/events{/privacy}","received_events_url":"https://api.github.com/users/acozine/received_events","type":"User","site_admin":false},"created_at":"2016-07-15T19:08:12Z","updated_at":"2016-07-15T19:08:12Z","body":"The `solr.service` definition is in `/run/systemd/generator.late/solr.service`. The ExecStart and ExecStop commands point to `/etc/rc.d/init.d/solr` which contains\r\n```\r\nif [ -n \"$RUNAS\" ]; then\r\n  su -c \"SOLR_INCLUDE=\\\"$SOLR_ENV\\\" \\\"$SOLR_INSTALL_DIR/bin/solr\\\" $SOLR_CMD\" - \"$RUNAS\"\r\nelse\r\n  SOLR_INCLUDE=\"$SOLR_ENV\" \"$SOLR_INSTALL_DIR/bin/solr\" \"$SOLR_CMD\"\r\nfi\r\n```\r\n"}}
{"actor":{"display_login":"acozine"}}