{"action":"created","issue":{"url":"https://api.github.com/repos/perfsonar/pscheduler/issues/238","repository_url":"https://api.github.com/repos/perfsonar/pscheduler","labels_url":"https://api.github.com/repos/perfsonar/pscheduler/issues/238/labels{/name}","comments_url":"https://api.github.com/repos/perfsonar/pscheduler/issues/238/comments","events_url":"https://api.github.com/repos/perfsonar/pscheduler/issues/238/events","html_url":"https://github.com/perfsonar/pscheduler/issues/238","id":190970172,"number":238,"title":"stalled processes","user":{"login":"igarny","id":11942774,"avatar_url":"https://avatars.githubusercontent.com/u/11942774?v=3","gravatar_id":"","url":"https://api.github.com/users/igarny","html_url":"https://github.com/igarny","followers_url":"https://api.github.com/users/igarny/followers","following_url":"https://api.github.com/users/igarny/following{/other_user}","gists_url":"https://api.github.com/users/igarny/gists{/gist_id}","starred_url":"https://api.github.com/users/igarny/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/igarny/subscriptions","organizations_url":"https://api.github.com/users/igarny/orgs","repos_url":"https://api.github.com/users/igarny/repos","events_url":"https://api.github.com/users/igarny/events{/privacy}","received_events_url":"https://api.github.com/users/igarny/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-22T11:07:52Z","updated_at":"2016-11-22T11:12:15Z","closed_at":null,"body":"Hi guys,\r\n\r\n\r\nI started with\r\n\r\n>> service [pscheduler-archiver, pscheduler-scheduler] stop\r\n[root@psps-test2-mgmt ~]# pscheduler internal reset\r\n\r\nWARNING\r\n\r\nYou are about to obliterate the contents of the pScheduler database on\r\nthis host.  Please confirm this by typing today's date in YYYY-MM-DD\r\nformat at the prompt below.\r\n\r\nConfirm (YYYY-MM-DD): 2016-11-22\r\n\r\nResetting database...ERROR:  database \"pscheduler\" is being accessed by other users\r\nDETAIL:  There is 1 other session using the database.\r\n\r\n\r\n**Checking processes:**\r\n[root@psps-test2-mgmt ~]# ps ax | grep psched\r\n 2070 ?        Ss     0:19 postgres: pscheduler pscheduler 127.0.0.1(54190) SELECT\r\n17752 pts/0    S+     0:00 /bin/sh /usr/libexec/pscheduler/internals/reset\r\n17810 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(57788) UPDATE waiting\r\n17874 pts/0    S+     0:00 /bin/sh /usr/bin/postgresql-load --role pscheduler /usr/share/pscheduler/database-build.sql\r\n17884 pts/0    S+     0:00 /bin/sh /usr/bin/postgresql-load --role pscheduler /usr/share/pscheduler/database-build.sql\r\n17886 pts/0    S+     0:00 cat /usr/share/pscheduler/database-build.sql\r\n17915 ?        Ss     0:00 postgres: postgres pscheduler [local] DROP TRIGGER waiting\r\n18150 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(57802) idle\r\n18151 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(57804) UPDATE waiting\r\n18711 pts/1    S+     0:00 grep psched\r\n25271 ?        S      0:00 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/test/throughput/participants\r\n\r\nI assume pschedID: 57788 blocks the operation.\r\n\r\n**I have tried all:**\r\nservice [pscheduler-archiver, pscheduler-runner, pscheduler-scheduler, pscheduler-ticker] stop\r\n\r\n**There were no powstream processes. I have even owamp restarted.**\r\n\r\n[root@psps-test2-mgmt ~]# grep 57788 /var/log/messages | grep \"Nov 22\"\r\nNov 22 02:09:04 psps-test2-mgmt owampd[7279]: FILE=sapi.c, LINE=303, Connection to ([psps-test2-mgmt.rrze.uni-erlangen.de]:861) from ([ma-dev2.bldc.grnoc.iu.edu]:57788)\r\nNov 22 02:09:04 psps-test2-mgmt owampd[7279]: FILE=sapi.c, LINE=303, Connection to ([psps-test2-mgmt.rrze.uni-erlangen.de]:861) from ([ma-dev2.bldc.grnoc.iu.edu]:57788)\r\nNov 22 02:09:04 psps-test2-mgmt owampd[7279]: FILE=sapi.c, LINE=466, ControlSession([psps-test2-mgmt.rrze.uni-erlangen.de]:861) accepted from userid(nil):([ma-dev2.bldc.grnoc.iu.edu]:57788)\r\nNov 22 02:09:04 psps-test2-mgmt owampd[7279]: FILE=sapi.c, LINE=466, ControlSession([psps-test2-mgmt.rrze.uni-erlangen.de]:861) accepted from userid(nil):([ma-dev2.bldc.grnoc.iu.edu]:57788)\r\n\r\n\r\n**IMHO the process had stalled and there is nothing to clean it**\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/perfsonar/pscheduler/issues/comments/262213548","html_url":"https://github.com/perfsonar/pscheduler/issues/238#issuecomment-262213548","issue_url":"https://api.github.com/repos/perfsonar/pscheduler/issues/238","id":262213548,"user":{"login":"igarny","id":11942774,"avatar_url":"https://avatars.githubusercontent.com/u/11942774?v=3","gravatar_id":"","url":"https://api.github.com/users/igarny","html_url":"https://github.com/igarny","followers_url":"https://api.github.com/users/igarny/followers","following_url":"https://api.github.com/users/igarny/following{/other_user}","gists_url":"https://api.github.com/users/igarny/gists{/gist_id}","starred_url":"https://api.github.com/users/igarny/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/igarny/subscriptions","organizations_url":"https://api.github.com/users/igarny/orgs","repos_url":"https://api.github.com/users/igarny/repos","events_url":"https://api.github.com/users/igarny/events{/privacy}","received_events_url":"https://api.github.com/users/igarny/received_events","type":"User","site_admin":false},"created_at":"2016-11-22T11:12:15Z","updated_at":"2016-11-22T11:12:15Z","body":"Hmm in fact it was another process:\r\n\r\n[root@psps-test2-mgmt ~]# kill 17810\r\n[root@psps-test2-mgmt ~]#\r\n[root@psps-test2-mgmt ~]#\r\n[root@psps-test2-mgmt ~]# ps ax | grep psch\r\n 2070 ?        Ss     0:19 postgres: pscheduler pscheduler 127.0.0.1(54190) SELECT\r\n17752 pts/0    S+     0:00 /bin/sh /usr/libexec/pscheduler/internals/reset\r\n17874 pts/0    S+     0:00 /bin/sh /usr/bin/postgresql-load --role pscheduler /usr/share/pscheduler/database-build.sql\r\n17884 pts/0    S+     0:00 /bin/sh /usr/bin/postgresql-load --role pscheduler /usr/share/pscheduler/database-build.sql\r\n17886 pts/0    S+     0:00 cat /usr/share/pscheduler/database-build.sql\r\n17915 ?        Ss     0:00 postgres: postgres pscheduler [local] DROP TRIGGER waiting\r\n18150 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(57802) idle\r\n18151 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(57804) UPDATE waiting\r\n18875 ?        S      0:00 /usr/bin/python /usr/libexec/pscheduler/daemons/scheduler --daemon --pid-file /var/run/pscheduler-scheduler.pid --dsn @/etc/pscheduler/database/database-dsn\r\n18876 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(58182) SELECT waiting\r\n19151 pts/1    S+     0:00 grep psch\r\n25271 ?        S      0:00 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/test/throughput/participants\r\n\r\n\r\n**[root@psps-test2-mgmt ~]# kill 2070**\r\n[root@psps-test2-mgmt ~]# ps ax | grep psch\r\n 2070 ?        Ss     0:19 postgres: pscheduler pscheduler 127.0.0.1(54190) SELECT\r\n17752 pts/0    S+     0:00 /bin/sh /usr/libexec/pscheduler/internals/reset\r\n17874 pts/0    S+     0:00 /bin/sh /usr/bin/postgresql-load --role pscheduler /usr/share/pscheduler/database-build.sql\r\n17884 pts/0    S+     0:00 /bin/sh /usr/bin/postgresql-load --role pscheduler /usr/share/pscheduler/database-build.sql\r\n17886 pts/0    S+     0:00 cat /usr/share/pscheduler/database-build.sql\r\n17915 ?        Ss     0:00 postgres: postgres pscheduler [local] DROP TRIGGER waiting\r\n18150 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(57802) idle\r\n18151 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(57804) UPDATE waiting\r\n18875 ?        S      0:00 /usr/bin/python /usr/libexec/pscheduler/daemons/scheduler --daemon --pid-file /var/run/pscheduler-scheduler.pid --dsn @/etc/pscheduler/database/database-dsn\r\n18876 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(58182) SELECT waiting\r\n19184 pts/1    S+     0:00 grep psch\r\n25271 ?        S      0:00 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/test/throughput/participants\r\n**[root@psps-test2-mgmt ~]# kill -9 2070**\r\n[root@psps-test2-mgmt ~]# ps ax | grep psch\r\n18875 ?        S      0:00 /usr/bin/python /usr/libexec/pscheduler/daemons/scheduler --daemon --pid-file /var/run/pscheduler-scheduler.pid --dsn @/etc/pscheduler/database/database-dsn\r\n19390 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(58474) idle\r\n19401 pts/1    S+     0:00 grep psch\r\n25271 ?        S      0:00 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/test/throughput/participants\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"igarny"}}