{"action":"created","issue":{"url":"https://api.github.com/repos/OceansAus/access-om2/issues/77","repository_url":"https://api.github.com/repos/OceansAus/access-om2","labels_url":"https://api.github.com/repos/OceansAus/access-om2/issues/77/labels{/name}","comments_url":"https://api.github.com/repos/OceansAus/access-om2/issues/77/comments","events_url":"https://api.github.com/repos/OceansAus/access-om2/issues/77/events","html_url":"https://github.com/OceansAus/access-om2/issues/77","id":292598495,"number":77,"title":"access-om2-01 hangs unpredictably - would also affect access-om2-025","user":{"login":"aekiss","id":31054815,"avatar_url":"https://avatars3.githubusercontent.com/u/31054815?v=4","gravatar_id":"","url":"https://api.github.com/users/aekiss","html_url":"https://github.com/aekiss","followers_url":"https://api.github.com/users/aekiss/followers","following_url":"https://api.github.com/users/aekiss/following{/other_user}","gists_url":"https://api.github.com/users/aekiss/gists{/gist_id}","starred_url":"https://api.github.com/users/aekiss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aekiss/subscriptions","organizations_url":"https://api.github.com/users/aekiss/orgs","repos_url":"https://api.github.com/users/aekiss/repos","events_url":"https://api.github.com/users/aekiss/events{/privacy}","received_events_url":"https://api.github.com/users/aekiss/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-01-29T23:25:36Z","updated_at":"2018-02-06T04:33:41Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"access-om2-01 hangs unpredictably for about a third of the submissions, burning 28kSU for nothing. I expect this problem to also affect access-om2-025 since this also relies on mxm.\r\n\r\nThere's been extensive email discussion with @benmenadue, @nicjhan, @marshallward  under subject \"HELP-8799 MPI library problem? MOM not timestepping\" (https://track.nci.org.au/servicedesk/customer/portal/5/HELP-8799) and Slack https://arccss.slack.com/archives/C08KM5VRA/p1516305480000282 but I want to organise key findings here.\r\n\r\nSymptoms: no timestepping (eg no output to `work/ice/OUTPUT`, which should get daily files). Eventually times out.\r\n\r\nProblem arose on 2018-01-17 when Ben set the MXM logfile output to /dev/null for performance reasons. Working hypothesis: MXM fsyncs its log target every 50 milliseconds by default, and this fails because MXM_LOG_FILE is pointing to a non-fsync-able file (/dev/null), causing it to abort processing the asynchronous callbacks.\r\n\r\nWorkaround: use `-x MXM_LOG_FILE=$PBS_JOBFS/mxm.log` in mpirun in config.yaml, and submit with `/projects/v45/apps/payu/aek` which was modified to do shell substitution in the mpirun commands. This works most of the time but we still get unpredictable hangs.\r\n\r\nThis seems to be an oasis problem. See Bens' email of 2018-01-25 re. `/short/v45/aek156/access-om2/control/01deg_jra55_ryf/work-3333791`:\r\n```\r\nThere's 250 ranks stuck on line 123 of mod_oasis_method.F90\r\n\r\ncall oasis_mpi_barrier(mpi_comm_global)\r\nyet at least 5160 ranks have made it past that point and are stuck at line 231 of the same file\r\n\r\ncall MPI_COMM_SPLIT(MPI_COMM_WORLD,icolor,ikey,mpi_comm_local,mpi_err)\r\nThat should not have been able to happen, unless mpi_comm_global /= MPI_COMM_WORLD.\r\n```"},"comment":{"url":"https://api.github.com/repos/OceansAus/access-om2/issues/comments/363308087","html_url":"https://github.com/OceansAus/access-om2/issues/77#issuecomment-363308087","issue_url":"https://api.github.com/repos/OceansAus/access-om2/issues/77","id":363308087,"user":{"login":"aidanheerdegen","id":6063709,"avatar_url":"https://avatars2.githubusercontent.com/u/6063709?v=4","gravatar_id":"","url":"https://api.github.com/users/aidanheerdegen","html_url":"https://github.com/aidanheerdegen","followers_url":"https://api.github.com/users/aidanheerdegen/followers","following_url":"https://api.github.com/users/aidanheerdegen/following{/other_user}","gists_url":"https://api.github.com/users/aidanheerdegen/gists{/gist_id}","starred_url":"https://api.github.com/users/aidanheerdegen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aidanheerdegen/subscriptions","organizations_url":"https://api.github.com/users/aidanheerdegen/orgs","repos_url":"https://api.github.com/users/aidanheerdegen/repos","events_url":"https://api.github.com/users/aidanheerdegen/events{/privacy}","received_events_url":"https://api.github.com/users/aidanheerdegen/received_events","type":"User","site_admin":false},"created_at":"2018-02-06T04:33:40Z","updated_at":"2018-02-06T04:33:40Z","author_association":"CONTRIBUTOR","body":"Love the name of the repo @russfiedler \r\n"}}
{"actor":{"display_login":"aidanheerdegen"}}