{"action":"created","issue":{"url":"https://api.github.com/repos/radical-cybertools/saga-python/issues/533","repository_url":"https://api.github.com/repos/radical-cybertools/saga-python","labels_url":"https://api.github.com/repos/radical-cybertools/saga-python/issues/533/labels{/name}","comments_url":"https://api.github.com/repos/radical-cybertools/saga-python/issues/533/comments","events_url":"https://api.github.com/repos/radical-cybertools/saga-python/issues/533/events","html_url":"https://github.com/radical-cybertools/saga-python/issues/533","id":148429243,"number":533,"title":"flushing failed","user":{"login":"dmanubens","id":1502044,"avatar_url":"https://avatars.githubusercontent.com/u/1502044?v=3","gravatar_id":"","url":"https://api.github.com/users/dmanubens","html_url":"https://github.com/dmanubens","followers_url":"https://api.github.com/users/dmanubens/followers","following_url":"https://api.github.com/users/dmanubens/following{/other_user}","gists_url":"https://api.github.com/users/dmanubens/gists{/gist_id}","starred_url":"https://api.github.com/users/dmanubens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmanubens/subscriptions","organizations_url":"https://api.github.com/users/dmanubens/orgs","repos_url":"https://api.github.com/users/dmanubens/repos","events_url":"https://api.github.com/users/dmanubens/events{/privacy}","received_events_url":"https://api.github.com/users/dmanubens/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/radical-cybertools/saga-python/labels/pty","name":"pty","color":"207de5"},{"url":"https://api.github.com/repos/radical-cybertools/saga-python/labels/shell%20job%20adaptor","name":"shell job adaptor","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"andre-merzky","id":1319674,"avatar_url":"https://avatars.githubusercontent.com/u/1319674?v=3","gravatar_id":"","url":"https://api.github.com/users/andre-merzky","html_url":"https://github.com/andre-merzky","followers_url":"https://api.github.com/users/andre-merzky/followers","following_url":"https://api.github.com/users/andre-merzky/following{/other_user}","gists_url":"https://api.github.com/users/andre-merzky/gists{/gist_id}","starred_url":"https://api.github.com/users/andre-merzky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andre-merzky/subscriptions","organizations_url":"https://api.github.com/users/andre-merzky/orgs","repos_url":"https://api.github.com/users/andre-merzky/repos","events_url":"https://api.github.com/users/andre-merzky/events{/privacy}","received_events_url":"https://api.github.com/users/andre-merzky/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2016-04-14T17:22:58Z","updated_at":"2016-04-27T17:32:21Z","closed_at":null,"body":"Hello,\r\n\r\nI am using this example modified slightly for our HPC resource (http://saga-python.readthedocs.org/en/latest/tutorial/part3.html) and when I launch multiple background scripts I get a flushing failed error.\r\n\r\n```\r\ndmanuben@bscearth309:~> module load autosubmit\r\ndmanuben@bscearth309:~> export SAGA_ADAPTOR_PATH=saga.adaptors.shell.shell_job:saga.adaptors.shell.shell_file:autosubmit.platforms.mn_adaptor\r\ndmanuben@bscearth309:~/tmp/testing-saga> SAGA_VERBOSE=DEBUG python saga_example_remote.py & SAGA_VERBOSE=DEBUG python saga_example_remote.py &\r\n\r\n2016-04-14 19:05:25,341: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : read : [    4] [   10] (PROMPT-0->)\r\n2016-04-14 19:05:25,341: radical.saga.cpi    : MainProcess                     : Thread-1       : INFO    : Job monitoring thread updating Job <autosubmit.platforms.mn_adaptor.MNJob object at 0x7fa5ab6e62d0> (state: Running)\r\n2016-04-14 19:05:28,344: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : flush: [    4] [     ] (flush pty read cache)\r\n2016-04-14 19:05:28,344: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : flush: [    4] [     ] (flush pty read cache)\r\n2016-04-14 19:05:28,446: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : run_sync: bjobs -noheader 1741003\r\n2016-04-14 19:05:28,446: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : run_sync: bjobs -noheader 1741004\r\n2016-04-14 19:05:28,447: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : write: [    4] [   24] (bjobs -noheader 1741003\\n)\r\n2016-04-14 19:05:28,447: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : write: [    4] [   24] (bjobs -noheader 1741004\\n)\r\n2016-04-14 19:05:28,566: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : read : [    4] [   82] (1741003 bsc3270 DONE  sequential login2      s01         *MYOUTPUT  Apr 14 19:04\\n)\r\n2016-04-14 19:05:28,568: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : read : [    4] [   82] (1741004 bsc3270 RUN   sequential login2      s01         *MYOUTPUT  Apr 14 19:04\\n)\r\n2016-04-14 19:05:28,570: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : read : [    4] [   10] (PROMPT-0->)\r\n2016-04-14 19:05:28,570: radical.saga.cpi    : MainProcess                     : Thread-1       : INFO    : Job monitoring thread updating Job <autosubmit.platforms.mn_adaptor.MNJob object at 0x7f51412692d0> (state: Done)\r\n2016-04-14 19:05:28,571: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : read : [    4] [   10] (PROMPT-0->)\r\n2016-04-14 19:05:28,571: radical.saga.cpi    : MainProcess                     : Thread-1       : INFO    : Job monitoring thread updating Job <autosubmit.platforms.mn_adaptor.MNJob object at 0x7fa5ab6e62d0> (state: Running)\r\nJob State : Done\r\nExitcode  : None\r\n2016-04-14 19:05:31,574: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : flush: [    4] [     ] (flush pty read cache)\r\n2016-04-14 19:05:31,575: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : read : [    4] [   41] (Shared connection to mn2.bsc.es closed.\\n)\r\n2016-04-14 19:05:31,575: radical.saga.pty    : MainProcess                     : Thread-1       : WARNING : flush: [    4] [   40] (discard data: 'Shared connection to mn2.bsc.es closed.\r\n')\r\n2016-04-14 19:05:31,615: radical.saga.pty    : MainProcess                     : Thread-1       : ERROR   : flushing failed\r\nTraceback (most recent call last):\r\n  File \"/shared/earth/software/Python/2.7.9-foss-2015a/lib/python2.7/site-packages/saga_python-0.40-py2.7.egg/saga/utils/pty_process.py\", line 174, in flush\r\n    tmp = self.read (timeout=0.1)\r\n  File \"/shared/earth/software/Python/2.7.9-foss-2015a/lib/python2.7/site-packages/saga_python-0.40-py2.7.egg/saga/utils/pty_process.py\", line 679, in read\r\n    % (e, self.tail))\r\nNoSuccess: read from process failed '[Errno 5] Input/output error' : (*MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->1741004 bsc3270 RUN   sequential login2      s01         *MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->1741004 bsc3270 RUN   sequential login2      s01         *MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->Shared connection to mn2.bsc.es closed.\r\n) (/shared/earth/software/Python/2.7.9-foss-2015a/lib/python2.7/site-packages/saga_python-0.40-py2.7.egg/saga/utils/pty_process.py +679 (read)  :  % (e, self.tail)))\r\n2016-04-14 19:05:31,615: radical.saga.cpi    : MainProcess                     : Thread-1       : WARNING : Exception caught in job monitoring thread: read from process failed '[Errno 5] Input/output error' : (*MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->1741004 bsc3270 RUN   sequential login2      s01         *MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->1741004 bsc3270 RUN   sequential login2      s01         *MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->Shared connection to mn2.bsc.es closed.\r\n) (/shared/earth/software/Python/2.7.9-foss-2015a/lib/python2.7/site-packages/saga_python-0.40-py2.7.egg/saga/utils/pty_process.py +679 (read)  :  % (e, self.tail)))\r\n2016-04-14 19:05:31,615: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : flush: [    4] [     ] (flush pty read cache)\r\n2016-04-14 19:05:31,616: radical.saga.pty    : MainProcess                     : Thread-1       : ERROR   : flushing failed\r\nTraceback (most recent call last):\r\n  File \"/shared/earth/software/Python/2.7.9-foss-2015a/lib/python2.7/site-packages/saga_python-0.40-py2.7.egg/saga/utils/pty_process.py\", line 174, in flush\r\n    tmp = self.read (timeout=0.1)\r\n  File \"/shared/earth/software/Python/2.7.9-foss-2015a/lib/python2.7/site-packages/saga_python-0.40-py2.7.egg/saga/utils/pty_process.py\", line 679, in read\r\n    % (e, self.tail))\r\nNoSuccess: read from process failed '[Errno 5] Input/output error' : (*MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->1741004 bsc3270 RUN   sequential login2      s01         *MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->1741004 bsc3270 RUN   sequential login2      s01         *MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->Shared connection to mn2.bsc.es closed.\r\n) (/shared/earth/software/Python/2.7.9-foss-2015a/lib/python2.7/site-packages/saga_python-0.40-py2.7.egg/saga/utils/pty_process.py +679 (read)  :  % (e, self.tail)))\r\n2016-04-14 19:05:31,616: radical.saga.cpi    : MainProcess                     : Thread-1       : WARNING : Exception caught in job monitoring thread: read from process failed '[Errno 5] Input/output error' : (*MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->1741004 bsc3270 RUN   sequential login2      s01         *MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->1741004 bsc3270 RUN   sequential login2      s01         *MYOUTPUT  Apr 14 19:04\r\nPROMPT-0->Shared connection to mn2.bsc.es closed.\r\n) (/shared/earth/software/Python/2.7.9-foss-2015a/lib/python2.7/site-packages/saga_python-0.40-py2.7.egg/saga/utils/pty_process.py +679 (read)  :  % (e, self.tail)))\r\n2016-04-14 19:05:31,616: radical.saga.pty    : MainProcess                     : Thread-1       : DEBUG   : flush: [    4] [     ] (flush pty read cache)\r\n```"},"comment":{"url":"https://api.github.com/repos/radical-cybertools/saga-python/issues/comments/215162286","html_url":"https://github.com/radical-cybertools/saga-python/issues/533#issuecomment-215162286","issue_url":"https://api.github.com/repos/radical-cybertools/saga-python/issues/533","id":215162286,"user":{"login":"andre-merzky","id":1319674,"avatar_url":"https://avatars.githubusercontent.com/u/1319674?v=3","gravatar_id":"","url":"https://api.github.com/users/andre-merzky","html_url":"https://github.com/andre-merzky","followers_url":"https://api.github.com/users/andre-merzky/followers","following_url":"https://api.github.com/users/andre-merzky/following{/other_user}","gists_url":"https://api.github.com/users/andre-merzky/gists{/gist_id}","starred_url":"https://api.github.com/users/andre-merzky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andre-merzky/subscriptions","organizations_url":"https://api.github.com/users/andre-merzky/orgs","repos_url":"https://api.github.com/users/andre-merzky/repos","events_url":"https://api.github.com/users/andre-merzky/events{/privacy}","received_events_url":"https://api.github.com/users/andre-merzky/received_events","type":"User","site_admin":false},"created_at":"2016-04-27T17:32:21Z","updated_at":"2016-04-27T17:32:21Z","body":"Hmm, the fifo creation seems also to be a race condition.  I'll open another ticket for this one, that likely needs a creation lock.  Thanks!\r\n\r\nAbout the original problem: yes, let me try to reproduce this.  Either way though, that will be difficult to fix.  The pty layer, and connection management, has been tripping us very frequently, and more so in concurrent setups.  We do have some ideas on how to refactor that (rather central) piece of code, but alas, could not yet allocate the manpower to approach this thoroughly.  So, lets see if we can find yet another partial fix :P \r\n\r\nAre all ssh connections running under the same user ID, and toward the same ID on the target machines?"}}