{"action":"created","issue":{"url":"https://api.github.com/repos/UIUC-PPL/charm-test/issues/1616","repository_url":"https://api.github.com/repos/UIUC-PPL/charm-test","labels_url":"https://api.github.com/repos/UIUC-PPL/charm-test/issues/1616/labels{/name}","comments_url":"https://api.github.com/repos/UIUC-PPL/charm-test/issues/1616/comments","events_url":"https://api.github.com/repos/UIUC-PPL/charm-test/issues/1616/events","html_url":"https://github.com/UIUC-PPL/charm-test/issues/1616","id":432355789,"node_id":"MDU6SXNzdWU0MzIzNTU3ODk=","number":1616,"title":"Race condition during ckRestart causes hang after merging patch for bug #1576","user":{"login":"juanjgalvez","id":35979610,"node_id":"MDQ6VXNlcjM1OTc5NjEw","avatar_url":"https://avatars1.githubusercontent.com/u/35979610?v=4","gravatar_id":"","url":"https://api.github.com/users/juanjgalvez","html_url":"https://github.com/juanjgalvez","followers_url":"https://api.github.com/users/juanjgalvez/followers","following_url":"https://api.github.com/users/juanjgalvez/following{/other_user}","gists_url":"https://api.github.com/users/juanjgalvez/gists{/gist_id}","starred_url":"https://api.github.com/users/juanjgalvez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juanjgalvez/subscriptions","organizations_url":"https://api.github.com/users/juanjgalvez/orgs","repos_url":"https://api.github.com/users/juanjgalvez/repos","events_url":"https://api.github.com/users/juanjgalvez/events{/privacy}","received_events_url":"https://api.github.com/users/juanjgalvez/received_events","type":"User","site_admin":false},"labels":[{"id":1314561688,"node_id":"MDU6TGFiZWwxMzE0NTYxNjg4","url":"https://api.github.com/repos/UIUC-PPL/charm-test/labels/bug","name":"bug","color":"d73a4a","default":true}],"state":"closed","locked":false,"assignee":{"login":"juanjgalvez","id":35979610,"node_id":"MDQ6VXNlcjM1OTc5NjEw","avatar_url":"https://avatars1.githubusercontent.com/u/35979610?v=4","gravatar_id":"","url":"https://api.github.com/users/juanjgalvez","html_url":"https://github.com/juanjgalvez","followers_url":"https://api.github.com/users/juanjgalvez/followers","following_url":"https://api.github.com/users/juanjgalvez/following{/other_user}","gists_url":"https://api.github.com/users/juanjgalvez/gists{/gist_id}","starred_url":"https://api.github.com/users/juanjgalvez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juanjgalvez/subscriptions","organizations_url":"https://api.github.com/users/juanjgalvez/orgs","repos_url":"https://api.github.com/users/juanjgalvez/repos","events_url":"https://api.github.com/users/juanjgalvez/events{/privacy}","received_events_url":"https://api.github.com/users/juanjgalvez/received_events","type":"User","site_admin":false},"assignees":[{"login":"juanjgalvez","id":35979610,"node_id":"MDQ6VXNlcjM1OTc5NjEw","avatar_url":"https://avatars1.githubusercontent.com/u/35979610?v=4","gravatar_id":"","url":"https://api.github.com/users/juanjgalvez","html_url":"https://github.com/juanjgalvez","followers_url":"https://api.github.com/users/juanjgalvez/followers","following_url":"https://api.github.com/users/juanjgalvez/following{/other_user}","gists_url":"https://api.github.com/users/juanjgalvez/gists{/gist_id}","starred_url":"https://api.github.com/users/juanjgalvez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juanjgalvez/subscriptions","organizations_url":"https://api.github.com/users/juanjgalvez/orgs","repos_url":"https://api.github.com/users/juanjgalvez/repos","events_url":"https://api.github.com/users/juanjgalvez/events{/privacy}","received_events_url":"https://api.github.com/users/juanjgalvez/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/UIUC-PPL/charm-test/milestones/7","html_url":"https://github.com/UIUC-PPL/charm-test/milestone/7","labels_url":"https://api.github.com/repos/UIUC-PPL/charm-test/milestones/7/labels","id":4221746,"node_id":"MDk6TWlsZXN0b25lNDIyMTc0Ng==","number":7,"title":"6.8.0","description":null,"creator":{"login":"pplimport","id":40410281,"node_id":"MDQ6VXNlcjQwNDEwMjgx","avatar_url":"https://avatars2.githubusercontent.com/u/40410281?v=4","gravatar_id":"","url":"https://api.github.com/users/pplimport","html_url":"https://github.com/pplimport","followers_url":"https://api.github.com/users/pplimport/followers","following_url":"https://api.github.com/users/pplimport/following{/other_user}","gists_url":"https://api.github.com/users/pplimport/gists{/gist_id}","starred_url":"https://api.github.com/users/pplimport/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pplimport/subscriptions","organizations_url":"https://api.github.com/users/pplimport/orgs","repos_url":"https://api.github.com/users/pplimport/repos","events_url":"https://api.github.com/users/pplimport/events{/privacy}","received_events_url":"https://api.github.com/users/pplimport/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":264,"state":"open","created_at":"2019-04-11T15:54:58Z","updated_at":"2019-04-12T02:12:33Z","due_on":null,"closed_at":null},"comments":6,"created_at":"2017-06-21T19:22:31Z","updated_at":"2019-04-12T02:12:55Z","closed_at":"2017-06-29T02:41:32Z","author_association":"NONE","body":"*Original issue: https://charm.cs.illinois.edu/redmine/issues/1616*\n\n---\nBug is exposed by doing a CmiNodeAllBarrier after InitCPUTopology, which may be causing broadcast messages sent by PE0 in InitCPUTopology to overlap with restart messages."},"comment":{"url":"https://api.github.com/repos/UIUC-PPL/charm-test/issues/comments/482409322","html_url":"https://github.com/UIUC-PPL/charm-test/issues/1616#issuecomment-482409322","issue_url":"https://api.github.com/repos/UIUC-PPL/charm-test/issues/1616","id":482409322,"node_id":"MDEyOklzc3VlQ29tbWVudDQ4MjQwOTMyMg==","user":{"login":"juanjgalvez","id":35979610,"node_id":"MDQ6VXNlcjM1OTc5NjEw","avatar_url":"https://avatars1.githubusercontent.com/u/35979610?v=4","gravatar_id":"","url":"https://api.github.com/users/juanjgalvez","html_url":"https://github.com/juanjgalvez","followers_url":"https://api.github.com/users/juanjgalvez/followers","following_url":"https://api.github.com/users/juanjgalvez/following{/other_user}","gists_url":"https://api.github.com/users/juanjgalvez/gists{/gist_id}","starred_url":"https://api.github.com/users/juanjgalvez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juanjgalvez/subscriptions","organizations_url":"https://api.github.com/users/juanjgalvez/orgs","repos_url":"https://api.github.com/users/juanjgalvez/repos","events_url":"https://api.github.com/users/juanjgalvez/events{/privacy}","received_events_url":"https://api.github.com/users/juanjgalvez/received_events","type":"User","site_admin":false},"created_at":"2019-04-12T02:12:55Z","updated_at":"2019-04-12T02:12:55Z","author_association":"NONE","body":"*Original date: 2017-06-28 14:40:15*\n\n---\nHave been investigating and have much better understanding of what's going on.\r\n\r\nIn the original code (before implementing second reduction in cputopology), cputopology\r\ndoes CmiReduce followed by broadcast from PE0 to all PEs. Adding CmiNodeAllBarrier right\r\nafter cputopology sometimes causes worker threads to reach CkRestartMain (and schedule bdcastR0)\r\nbefore the commthread has sent the cputopology broadcast.\r\nIn this situation, initDone by worker threads in node 1 might be called before those\r\nthreads exit cputopology, causing a hang. initDone is called because they receive\r\nbdcastRO from PE0 (which is processed before the broadcast from cputopology).\r\n\r\nAfter this patch (~~https://charm.cs.illinois.edu/gerrit/#/c/2723/~~ https://github.com/UIUC-PPL/charm/commit/e9191d90ba87d917ad9ed202d0be06ebc98ab553), PE0 now waits for\r\nother PEs to acknowledge reception of the cputopology broadcast before exiting cputopology.\r\nThis is implemented using a second CmiReduce. So, above issue should not occur anymore.\r\n\r\nHowever, the new issue is that the second CmiReduce msg from node 1 is not sent to node 0.\r\nThis is because rank0 in node 1, when entering initDone via CkRestartMain, encounters a\r\nCmiNodeBarrier. And CmiReduce doesn't progress if worker thread is stopped.\r\n\r\n\r\n*Options I currently see to address this issue:*\r\n\r\n- Instead of CmiReduce to notify PE0 that nodes have received cputopology broadcast,\r\ndo point-to-point CmiSend to PE0 (one send for each physical node).\r\nPotential scalability concerns.\r\nThis is easy to implement and initial testing works and addresses issue.\r\n\r\n- See if there is a way for CmiReduce to work even if worker thread is blocked, or\r\nmaybe implement CmiReduce with empty reducer that works this way. The second\r\nCmiReduce in cputopology uses empty reducer.\r\n\r\n- Don't call initDone from CkRestartMain and instead just wait for R0 broadcast from\r\nPE0. I don't yet have a very thorough understanding of CkRestart to know if the\r\nR0 broadcast will always be sent. And init.C mentions a potential race condition\r\nthat seems related to this case.\r\n"}}
{"actor":{"display_login":"juanjgalvez"}}