{"action":"created","issue":{"url":"https://api.github.com/repos/chapel-lang/chapel/issues/11533","repository_url":"https://api.github.com/repos/chapel-lang/chapel","labels_url":"https://api.github.com/repos/chapel-lang/chapel/issues/11533/labels{/name}","comments_url":"https://api.github.com/repos/chapel-lang/chapel/issues/11533/comments","events_url":"https://api.github.com/repos/chapel-lang/chapel/issues/11533/events","html_url":"https://github.com/chapel-lang/chapel/issues/11533","id":376130078,"node_id":"MDU6SXNzdWUzNzYxMzAwNzg=","number":11533,"title":"Portable way to launch a process per node for GASNet launchers","user":{"login":"ronawho","id":1588337,"node_id":"MDQ6VXNlcjE1ODgzMzc=","avatar_url":"https://avatars1.githubusercontent.com/u/1588337?v=4","gravatar_id":"","url":"https://api.github.com/users/ronawho","html_url":"https://github.com/ronawho","followers_url":"https://api.github.com/users/ronawho/followers","following_url":"https://api.github.com/users/ronawho/following{/other_user}","gists_url":"https://api.github.com/users/ronawho/gists{/gist_id}","starred_url":"https://api.github.com/users/ronawho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronawho/subscriptions","organizations_url":"https://api.github.com/users/ronawho/orgs","repos_url":"https://api.github.com/users/ronawho/repos","events_url":"https://api.github.com/users/ronawho/events{/privacy}","received_events_url":"https://api.github.com/users/ronawho/received_events","type":"User","site_admin":false},"labels":[{"id":496591153,"node_id":"MDU6TGFiZWw0OTY1OTExNTM=","url":"https://api.github.com/repos/chapel-lang/chapel/labels/area:%20Runtime","name":"area: Runtime","color":"d4c5f9","default":false},{"id":496591322,"node_id":"MDU6TGFiZWw0OTY1OTEzMjI=","url":"https://api.github.com/repos/chapel-lang/chapel/labels/area:%20Third-Party","name":"area: Third-Party","color":"d4c5f9","default":false},{"id":723282311,"node_id":"MDU6TGFiZWw3MjMyODIzMTE=","url":"https://api.github.com/repos/chapel-lang/chapel/labels/type:%20Portability","name":"type: Portability","color":"e99695","default":false}],"state":"open","locked":false,"assignee":{"login":"ronawho","id":1588337,"node_id":"MDQ6VXNlcjE1ODgzMzc=","avatar_url":"https://avatars1.githubusercontent.com/u/1588337?v=4","gravatar_id":"","url":"https://api.github.com/users/ronawho","html_url":"https://github.com/ronawho","followers_url":"https://api.github.com/users/ronawho/followers","following_url":"https://api.github.com/users/ronawho/following{/other_user}","gists_url":"https://api.github.com/users/ronawho/gists{/gist_id}","starred_url":"https://api.github.com/users/ronawho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronawho/subscriptions","organizations_url":"https://api.github.com/users/ronawho/orgs","repos_url":"https://api.github.com/users/ronawho/repos","events_url":"https://api.github.com/users/ronawho/events{/privacy}","received_events_url":"https://api.github.com/users/ronawho/received_events","type":"User","site_admin":false},"assignees":[{"login":"ronawho","id":1588337,"node_id":"MDQ6VXNlcjE1ODgzMzc=","avatar_url":"https://avatars1.githubusercontent.com/u/1588337?v=4","gravatar_id":"","url":"https://api.github.com/users/ronawho","html_url":"https://github.com/ronawho","followers_url":"https://api.github.com/users/ronawho/followers","following_url":"https://api.github.com/users/ronawho/following{/other_user}","gists_url":"https://api.github.com/users/ronawho/gists{/gist_id}","starred_url":"https://api.github.com/users/ronawho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronawho/subscriptions","organizations_url":"https://api.github.com/users/ronawho/orgs","repos_url":"https://api.github.com/users/ronawho/repos","events_url":"https://api.github.com/users/ronawho/events{/privacy}","received_events_url":"https://api.github.com/users/ronawho/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-10-31T19:40:14Z","updated_at":"2018-10-31T22:37:17Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"For gasnet launchers like gasnetrun_ibv, gasnetrun_mpi, gasnetrun_ofi, etc we currently just specify how many processes to launch with `-n <numLocales>`. By default gasnet will use mpirun to launch and how these processes get mapped to target locales is up to the mpirun implementation. MPICH seems to default to scattering processes so we end up with a process per node, but Open MPI seems to pack processes so we end up oversubscribing a single node, which is really bad for us.\r\n\r\nOn a system with Open MPI here's what we see: \r\n\r\n```sh \r\n$GASNET_BIN/gasnetrun_ibv -n 2 ./hello6-taskpar-dist_real -nl2\r\n```\r\n>Hello, world! (from locale 0 of 2 named chapcs11)\r\n>Hello, world! (from locale 1 of 2 named chapcs11)\r\n\r\nAdding `-N 2` to specify the number of nodes to run on seems to fix this:\r\n\r\n```sh\r\n$GASNET_BIN/gasnetrun_ibv -n 2 -N 2 ./hello6-taskpar-dist_real -nl2\r\n```\r\n>Hello, world! (from locale 0 of 2 named chapcs11)\r\n>Hello, world! (from locale 1 of 2 named chapcs10)\r\n\r\nHowever,  I'm not sure how portable of a solution this is because this option is listed as \"not always supported\", and I'm not sure what happens in cases where it's not supported."},"comment":{"url":"https://api.github.com/repos/chapel-lang/chapel/issues/comments/434871412","html_url":"https://github.com/chapel-lang/chapel/issues/11533#issuecomment-434871412","issue_url":"https://api.github.com/repos/chapel-lang/chapel/issues/11533","id":434871412,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDg3MTQxMg==","user":{"login":"PHHargrove","id":6687992,"node_id":"MDQ6VXNlcjY2ODc5OTI=","avatar_url":"https://avatars1.githubusercontent.com/u/6687992?v=4","gravatar_id":"","url":"https://api.github.com/users/PHHargrove","html_url":"https://github.com/PHHargrove","followers_url":"https://api.github.com/users/PHHargrove/followers","following_url":"https://api.github.com/users/PHHargrove/following{/other_user}","gists_url":"https://api.github.com/users/PHHargrove/gists{/gist_id}","starred_url":"https://api.github.com/users/PHHargrove/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PHHargrove/subscriptions","organizations_url":"https://api.github.com/users/PHHargrove/orgs","repos_url":"https://api.github.com/users/PHHargrove/repos","events_url":"https://api.github.com/users/PHHargrove/events{/privacy}","received_events_url":"https://api.github.com/users/PHHargrove/received_events","type":"User","site_admin":false},"created_at":"2018-10-31T22:37:17Z","updated_at":"2018-10-31T22:37:17Z","author_association":"CONTRIBUTOR","body":"If I understand correctly, the concern is over GASNet's \"not always supported\" disclaimer for the `-N` option.  The truth is that we have correct support for that option on the vast majority of `mpirun` implementations (including use of `aprun` and `srun`).  However, we cannot always know in advance what changes the MPICH or Open MPI folks will make when we aren't looking.\r\n\r\nIn response to \"I'm not sure what happens in cases where it's not supported\":\r\nWhen an mpirun not recognized/supported, the user will get a message.\r\n```\r\nWARNING: Don't know how to control process->node layout with your mpirun\r\nWARNING: PROCESS LAYOUT MIGHT NOT MATCH YOUR REQUEST\r\n```\r\n\r\nThe other good news is that the support for this is in perl, not C.\r\nSo adding support for an unrecognized mpirun could potentially be done \"in the field\" w/o the need to rebuild or reinstall GASNet.\r\n\r\nIf one does encounter a system where `-N` is not working, we would try to add the necessary recognition and support.  However, we'd also suggest that ssh-based spawning (which always supports `-N`) is a decent alternative when the system is configured to allow ssh among the allocated nodes."}}
{"actor":{"display_login":"PHHargrove"}}