{"action":"created","issue":{"url":"https://api.github.com/repos/geopm/geopm/issues/336","repository_url":"https://api.github.com/repos/geopm/geopm","labels_url":"https://api.github.com/repos/geopm/geopm/issues/336/labels{/name}","comments_url":"https://api.github.com/repos/geopm/geopm/issues/336/comments","events_url":"https://api.github.com/repos/geopm/geopm/issues/336/events","html_url":"https://github.com/geopm/geopm/issues/336","id":350990591,"node_id":"MDU6SXNzdWUzNTA5OTA1OTE=","number":336,"title":"Debug ControlMessage::wait() errors on Theta","user":{"login":"bgeltz","id":6174349,"node_id":"MDQ6VXNlcjYxNzQzNDk=","avatar_url":"https://avatars3.githubusercontent.com/u/6174349?v=4","gravatar_id":"","url":"https://api.github.com/users/bgeltz","html_url":"https://github.com/bgeltz","followers_url":"https://api.github.com/users/bgeltz/followers","following_url":"https://api.github.com/users/bgeltz/following{/other_user}","gists_url":"https://api.github.com/users/bgeltz/gists{/gist_id}","starred_url":"https://api.github.com/users/bgeltz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgeltz/subscriptions","organizations_url":"https://api.github.com/users/bgeltz/orgs","repos_url":"https://api.github.com/users/bgeltz/repos","events_url":"https://api.github.com/users/bgeltz/events{/privacy}","received_events_url":"https://api.github.com/users/bgeltz/received_events","type":"User","site_admin":false},"labels":[{"id":274078186,"node_id":"MDU6TGFiZWwyNzQwNzgxODY=","url":"https://api.github.com/repos/geopm/geopm/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/geopm/geopm/milestones/20","html_url":"https://github.com/geopm/geopm/milestone/20","labels_url":"https://api.github.com/repos/geopm/geopm/milestones/20/labels","id":3558741,"node_id":"MDk6TWlsZXN0b25lMzU1ODc0MQ==","number":20,"title":"WW32 Sprint","description":"","creator":{"login":"cmcantalupo","id":378319,"node_id":"MDQ6VXNlcjM3ODMxOQ==","avatar_url":"https://avatars1.githubusercontent.com/u/378319?v=4","gravatar_id":"","url":"https://api.github.com/users/cmcantalupo","html_url":"https://github.com/cmcantalupo","followers_url":"https://api.github.com/users/cmcantalupo/followers","following_url":"https://api.github.com/users/cmcantalupo/following{/other_user}","gists_url":"https://api.github.com/users/cmcantalupo/gists{/gist_id}","starred_url":"https://api.github.com/users/cmcantalupo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmcantalupo/subscriptions","organizations_url":"https://api.github.com/users/cmcantalupo/orgs","repos_url":"https://api.github.com/users/cmcantalupo/repos","events_url":"https://api.github.com/users/cmcantalupo/events{/privacy}","received_events_url":"https://api.github.com/users/cmcantalupo/received_events","type":"User","site_admin":false},"open_issues":23,"closed_issues":3,"state":"open","created_at":"2018-08-09T17:07:52Z","updated_at":"2018-08-16T18:30:47Z","due_on":"2018-08-22T07:00:00Z","closed_at":null},"comments":2,"created_at":"2018-08-15T22:31:16Z","updated_at":"2018-08-20T17:53:42Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Very rarely, I'm seeing these messages in my job logs on Theta at job start:\r\n\r\n```\r\nError: <geopm> Runtime error: ControlMessage::wait(): Timed out waiting for status 1: at geopm/src/ControlMessage.cpp:87             \r\nError: <geopm> Runtime error: ControlMessage::wait(): Timed out waiting for status 4: at geopm/src/ControlMessage.cpp:87\r\n...\r\n<the error message for status 4 is repeated many times, once for each rank on each node I believe>\r\n```\r\n\r\nOnce this happens, the job hangs indefinitely.  It will eventually be killed by the resource manager when it times out.  The app appears to run to completion, but the Controller is hung.\r\n\r\nThe last job I have that reproduced this problem is 262202.  Here are the relevant docs:\r\n\r\n```\r\n/projects/intel/geopm-home/shared/output/262202.output\r\n/projects/intel/geopm-home/shared/output/262202.error\r\n/projects/intel/geopm-home/shared/output/262202.cobaltlog\r\n/projects/intel/geopm-home/shared/output/262202-nekbone-256node-1rank-62thread-2018-08-15_1011```"},"comment":{"url":"https://api.github.com/repos/geopm/geopm/issues/comments/414405919","html_url":"https://github.com/geopm/geopm/issues/336#issuecomment-414405919","issue_url":"https://api.github.com/repos/geopm/geopm/issues/336","id":414405919,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNDQwNTkxOQ==","user":{"login":"bgeltz","id":6174349,"node_id":"MDQ6VXNlcjYxNzQzNDk=","avatar_url":"https://avatars3.githubusercontent.com/u/6174349?v=4","gravatar_id":"","url":"https://api.github.com/users/bgeltz","html_url":"https://github.com/bgeltz","followers_url":"https://api.github.com/users/bgeltz/followers","following_url":"https://api.github.com/users/bgeltz/following{/other_user}","gists_url":"https://api.github.com/users/bgeltz/gists{/gist_id}","starred_url":"https://api.github.com/users/bgeltz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgeltz/subscriptions","organizations_url":"https://api.github.com/users/bgeltz/orgs","repos_url":"https://api.github.com/users/bgeltz/repos","events_url":"https://api.github.com/users/bgeltz/events{/privacy}","received_events_url":"https://api.github.com/users/bgeltz/received_events","type":"User","site_admin":false},"created_at":"2018-08-20T17:53:42Z","updated_at":"2018-08-20T17:53:42Z","author_association":"CONTRIBUTOR","body":"New failures over the weekend:\r\n262863\r\n262877\r\n262884\r\n262945\r\n262946"}}
{"actor":{"display_login":"bgeltz"}}