{"action":"created","issue":{"url":"https://api.github.com/repos/chapel-lang/chapel/issues/11621","repository_url":"https://api.github.com/repos/chapel-lang/chapel","labels_url":"https://api.github.com/repos/chapel-lang/chapel/issues/11621/labels{/name}","comments_url":"https://api.github.com/repos/chapel-lang/chapel/issues/11621/comments","events_url":"https://api.github.com/repos/chapel-lang/chapel/issues/11621/events","html_url":"https://github.com/chapel-lang/chapel/issues/11621","id":380064277,"node_id":"MDU6SXNzdWUzODAwNjQyNzc=","number":11621,"title":"Heterogeneous GASNET_DOMAIN_COUNT value causes hangs","user":{"login":"ronawho","id":1588337,"node_id":"MDQ6VXNlcjE1ODgzMzc=","avatar_url":"https://avatars1.githubusercontent.com/u/1588337?v=4","gravatar_id":"","url":"https://api.github.com/users/ronawho","html_url":"https://github.com/ronawho","followers_url":"https://api.github.com/users/ronawho/followers","following_url":"https://api.github.com/users/ronawho/following{/other_user}","gists_url":"https://api.github.com/users/ronawho/gists{/gist_id}","starred_url":"https://api.github.com/users/ronawho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronawho/subscriptions","organizations_url":"https://api.github.com/users/ronawho/orgs","repos_url":"https://api.github.com/users/ronawho/repos","events_url":"https://api.github.com/users/ronawho/events{/privacy}","received_events_url":"https://api.github.com/users/ronawho/received_events","type":"User","site_admin":false},"labels":[{"id":496591322,"node_id":"MDU6TGFiZWw0OTY1OTEzMjI=","url":"https://api.github.com/repos/chapel-lang/chapel/labels/area:%20Third-Party","name":"area: Third-Party","color":"d4c5f9","default":false},{"id":723282311,"node_id":"MDU6TGFiZWw3MjMyODIzMTE=","url":"https://api.github.com/repos/chapel-lang/chapel/labels/type:%20Portability","name":"type: Portability","color":"e99695","default":false}],"state":"open","locked":false,"assignee":{"login":"ronawho","id":1588337,"node_id":"MDQ6VXNlcjE1ODgzMzc=","avatar_url":"https://avatars1.githubusercontent.com/u/1588337?v=4","gravatar_id":"","url":"https://api.github.com/users/ronawho","html_url":"https://github.com/ronawho","followers_url":"https://api.github.com/users/ronawho/followers","following_url":"https://api.github.com/users/ronawho/following{/other_user}","gists_url":"https://api.github.com/users/ronawho/gists{/gist_id}","starred_url":"https://api.github.com/users/ronawho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronawho/subscriptions","organizations_url":"https://api.github.com/users/ronawho/orgs","repos_url":"https://api.github.com/users/ronawho/repos","events_url":"https://api.github.com/users/ronawho/events{/privacy}","received_events_url":"https://api.github.com/users/ronawho/received_events","type":"User","site_admin":false},"assignees":[{"login":"ronawho","id":1588337,"node_id":"MDQ6VXNlcjE1ODgzMzc=","avatar_url":"https://avatars1.githubusercontent.com/u/1588337?v=4","gravatar_id":"","url":"https://api.github.com/users/ronawho","html_url":"https://github.com/ronawho","followers_url":"https://api.github.com/users/ronawho/followers","following_url":"https://api.github.com/users/ronawho/following{/other_user}","gists_url":"https://api.github.com/users/ronawho/gists{/gist_id}","starred_url":"https://api.github.com/users/ronawho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronawho/subscriptions","organizations_url":"https://api.github.com/users/ronawho/orgs","repos_url":"https://api.github.com/users/ronawho/repos","events_url":"https://api.github.com/users/ronawho/events{/privacy}","received_events_url":"https://api.github.com/users/ronawho/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2018-11-13T05:19:59Z","updated_at":"2018-11-16T14:33:22Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Under gasnet-aries we set `GASNET_DOMAIN_COUNT` to `<numCores+1>`. We've been seeing intermittent hangs on a heterogenous system, which we've narrowed down to a single job getting different node types. As a specific example a job on a 28-core BW and 68-core KNL will hang. However, not all combinations seem to hang as a job on 10-core IV and 20-core IV seems to work.\r\n\r\nHere's an awful patch that I've been using to reproduce. It sets `GASNET_DOMAIN_COUNT` to `SLURM_NODEID+1`\r\n\r\n```diff\r\ndiff --git a/runtime/src/comm/gasnet/comm-gasnet.c b/runtime/src/comm/gasnet/comm-gasnet.c\r\nindex 2b1e0c6f25..76971f6014 100644\r\n--- a/runtime/src/comm/gasnet/comm-gasnet.c\r\n+++ b/runtime/src/comm/gasnet/comm-gasnet.c\r\n@@ -766,12 +766,13 @@ static void set_max_segsize() {\r\n   }\r\n }\r\n\r\n+#include \"chpl-env.h\"\r\n static void set_num_comm_domains() {\r\n #if defined(GASNET_CONDUIT_GEMINI) || defined(GASNET_CONDUIT_ARIES)\r\n   char num_cpus_val[22]; // big enough for an unsigned 64-bit quantity\r\n   int num_cpus;\r\n\r\n-  num_cpus = chpl_topo_getNumCPUsPhysical(true) + 1;\r\n+  num_cpus = chpl_env_str_to_int(\"SLURM_NODEID\", getenv(\"SLURM_NODEID\"), 0) + 1;\r\n\r\n   snprintf(num_cpus_val, sizeof(num_cpus_val), \"%d\", num_cpus);\r\n   if (setenv(\"GASNET_DOMAIN_COUNT\", num_cpus_val, 0) != 0) {\r\n```\r\n\r\nThis fails for me with 2 or more locales."},"comment":{"url":"https://api.github.com/repos/chapel-lang/chapel/issues/comments/439411217","html_url":"https://github.com/chapel-lang/chapel/issues/11621#issuecomment-439411217","issue_url":"https://api.github.com/repos/chapel-lang/chapel/issues/11621","id":439411217,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTQxMTIxNw==","user":{"login":"PHHargrove","id":6687992,"node_id":"MDQ6VXNlcjY2ODc5OTI=","avatar_url":"https://avatars1.githubusercontent.com/u/6687992?v=4","gravatar_id":"","url":"https://api.github.com/users/PHHargrove","html_url":"https://github.com/PHHargrove","followers_url":"https://api.github.com/users/PHHargrove/followers","following_url":"https://api.github.com/users/PHHargrove/following{/other_user}","gists_url":"https://api.github.com/users/PHHargrove/gists{/gist_id}","starred_url":"https://api.github.com/users/PHHargrove/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PHHargrove/subscriptions","organizations_url":"https://api.github.com/users/PHHargrove/orgs","repos_url":"https://api.github.com/users/PHHargrove/repos","events_url":"https://api.github.com/users/PHHargrove/events{/privacy}","received_events_url":"https://api.github.com/users/PHHargrove/received_events","type":"User","site_admin":false},"created_at":"2018-11-16T14:33:22Z","updated_at":"2018-11-16T14:33:22Z","author_association":"CONTRIBUTOR","body":"Regarding the cost of the unused domains due to the MAX:\r\n\r\nAs mentioned before, the allocation of more GNI-level Cdm's will potentially push the implementation into FMA sharing mode.\r\n\r\nWe also create additional GNI-level Cq's and Ep's.\r\nOther than the memory they consume, I am not aware of any associated cost (nothing analogous to the FMA-sharing overhead).\r\n\r\nIn addition, I find 416KB of pinned memory allocated for each domain.  \r\nThis will consume physical page frames for each \"extra\" domain, but will not get used.  \r\nHowever, with this memory coming from 2MB huge pages, it may or may not be visible.\r\n\r\nI am hoping I will have a solution which will avoid allocation of the GNI-level resources."}}
{"actor":{"display_login":"PHHargrove"}}