{"action":"created","issue":{"url":"https://api.github.com/repos/choderalab/yank/issues/1109","repository_url":"https://api.github.com/repos/choderalab/yank","labels_url":"https://api.github.com/repos/choderalab/yank/issues/1109/labels{/name}","comments_url":"https://api.github.com/repos/choderalab/yank/issues/1109/comments","events_url":"https://api.github.com/repos/choderalab/yank/issues/1109/events","html_url":"https://github.com/choderalab/yank/issues/1109","id":369767410,"node_id":"MDU6SXNzdWUzNjk3Njc0MTA=","number":1109,"title":"Some experiments in multi-experiment MPI SAMS runs hang","user":{"login":"jchodera","id":3656088,"node_id":"MDQ6VXNlcjM2NTYwODg=","avatar_url":"https://avatars1.githubusercontent.com/u/3656088?v=4","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"labels":[{"id":390179672,"node_id":"MDU6TGFiZWwzOTAxNzk2NzI=","url":"https://api.github.com/repos/choderalab/yank/labels/Priority%20high","name":"Priority high","color":"d93f0b","default":false},{"id":62138043,"node_id":"MDU6TGFiZWw2MjEzODA0Mw==","url":"https://api.github.com/repos/choderalab/yank/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-10-13T04:54:14Z","updated_at":"2018-10-13T18:12:24Z","closed_at":null,"author_association":"MEMBER","body":"I'm still seeing an issue where some experiments in multi-experiment MPI SAMS runs will hang and not make progress, or restart and not make progress. I've attached an example logfile where you can see both behaviors.\r\n\r\n[cmethingesinglehinge5.log.zip](https://github.com/choderalab/yank/files/2475079/cmethingesinglehinge5.log.zip)\r\n\r\nNo `CRITICAL` logs appear in this experiment."},"comment":{"url":"https://api.github.com/repos/choderalab/yank/issues/comments/429563335","html_url":"https://github.com/choderalab/yank/issues/1109#issuecomment-429563335","issue_url":"https://api.github.com/repos/choderalab/yank/issues/1109","id":429563335,"node_id":"MDEyOklzc3VlQ29tbWVudDQyOTU2MzMzNQ==","user":{"login":"jchodera","id":3656088,"node_id":"MDQ6VXNlcjM2NTYwODg=","avatar_url":"https://avatars1.githubusercontent.com/u/3656088?v=4","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"created_at":"2018-10-13T18:12:24Z","updated_at":"2018-10-13T18:12:24Z","author_association":"MEMBER","body":"That set of experiments was generated with \r\n```\r\n/data/chodera/chodera/2018-08-05/yank-benchmark/c-Met_hinge_binders/run-lsf-neutral-rmsd.sh\r\n```\r\nusing the YAML file\r\n```\r\n/data/chodera/chodera/2018-08-05/yank-benchmark/c-Met_hinge_binders/neutral-sams-rmsd.yaml\r\n```\r\nThese simulations were run in 6-hour blocks chained together by LSF.\r\n\r\nSeparately, I tested whether chained 1-hour simulations were likely to show the same behavior more quickly, suspecting that perhaps the second job would start before the first job's NetCDF file was fully synced by the filesystem. If I add\r\n```\r\nsync\r\nsleep 10\r\nsync\r\nsleep 10\r\nsync\r\n```\r\nto the beginning of the bash script (`run-lsf-neutral-rmsd-safe.sh` and `neutral-sams-rmsd-safe.yaml`) I seem to get the same sort of behavior where replicas stall out, though I get a bit more information because `verbose` is on. See\r\n```\r\n/data/chodera/chodera/2018-08-05/yank-benchmark/c-Met_hinge_binders/neutral-rmsd-safe/experiments/cmethingesinglehinge10\r\n```"}}
{"actor":{"display_login":"jchodera"}}