{"action":"created","issue":{"url":"https://api.github.com/repos/uber/horovod/issues/616","repository_url":"https://api.github.com/repos/uber/horovod","labels_url":"https://api.github.com/repos/uber/horovod/issues/616/labels{/name}","comments_url":"https://api.github.com/repos/uber/horovod/issues/616/comments","events_url":"https://api.github.com/repos/uber/horovod/issues/616/events","html_url":"https://github.com/uber/horovod/pull/616","id":378970056,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI5NTQ5Mzg0","number":616,"title":"Parallelized, hierarchical allgather with GPU support","user":{"login":"karakusc","id":11875690,"node_id":"MDQ6VXNlcjExODc1Njkw","avatar_url":"https://avatars0.githubusercontent.com/u/11875690?v=4","gravatar_id":"","url":"https://api.github.com/users/karakusc","html_url":"https://github.com/karakusc","followers_url":"https://api.github.com/users/karakusc/followers","following_url":"https://api.github.com/users/karakusc/following{/other_user}","gists_url":"https://api.github.com/users/karakusc/gists{/gist_id}","starred_url":"https://api.github.com/users/karakusc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karakusc/subscriptions","organizations_url":"https://api.github.com/users/karakusc/orgs","repos_url":"https://api.github.com/users/karakusc/repos","events_url":"https://api.github.com/users/karakusc/events{/privacy}","received_events_url":"https://api.github.com/users/karakusc/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-11-09T00:07:39Z","updated_at":"2018-11-27T03:02:20Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/uber/horovod/pulls/616","html_url":"https://github.com/uber/horovod/pull/616","diff_url":"https://github.com/uber/horovod/pull/616.diff","patch_url":"https://github.com/uber/horovod/pull/616.patch"},"body":"This PR modifies the Allgather implementation, by decomposing the single MPI Allgather call it into the following steps: \r\n\r\n1. Create a shared memory buffer at CPU (shared across all local ranks)\r\n2. If the data is at the GPU, copy it into the shared buffer, with an offset based on where the tensor needs to be located in the Allgather output.\r\n3. For each local rank `i`, in parallel, perform cross-node MPI Allgather with other local rank `i`'s, into the shared memory buffer, with appropriate offsets. The shared memory buffer has the Allgather output at the end of this step.\r\n4. Copy the shared buffer contents into the rank-specific output buffer, at CPU or GPU depending on where the original tensor was located.\r\n\r\nImplementation does not change the final output of Allgather. For large tensors (>10s MB), it results in a significant speed-up (which grows with tensor and cluster size, up to several times). For smaller tensor sizes, the speed-up is limited. This also implicitly adds GPU Allgather support without requiring CUDA-aware MPI.\r\n\r\nNote: This might require a change in the HOROVOD_GPU_ALLGATHER environment variable semantics during installation, as currently HOROVOD_GPU_ALLGATHER=MPI seems to refer to CUDA-aware MPI. This implementation enables GPU Allgather without CUDA-aware MPI, but based on the current version, installation still requires HOROVOD_GPU_ALLGATHER=MPI and HOROVOD_ALLOW_MIXED_GPU_IMPL=1 so that the Allgather GPU op kernel gets registered.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/uber/horovod/issues/comments/441880909","html_url":"https://github.com/uber/horovod/pull/616#issuecomment-441880909","issue_url":"https://api.github.com/repos/uber/horovod/issues/616","id":441880909,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTg4MDkwOQ==","user":{"login":"karakusc","id":11875690,"node_id":"MDQ6VXNlcjExODc1Njkw","avatar_url":"https://avatars0.githubusercontent.com/u/11875690?v=4","gravatar_id":"","url":"https://api.github.com/users/karakusc","html_url":"https://github.com/karakusc","followers_url":"https://api.github.com/users/karakusc/followers","following_url":"https://api.github.com/users/karakusc/following{/other_user}","gists_url":"https://api.github.com/users/karakusc/gists{/gist_id}","starred_url":"https://api.github.com/users/karakusc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karakusc/subscriptions","organizations_url":"https://api.github.com/users/karakusc/orgs","repos_url":"https://api.github.com/users/karakusc/repos","events_url":"https://api.github.com/users/karakusc/events{/privacy}","received_events_url":"https://api.github.com/users/karakusc/received_events","type":"User","site_admin":false},"created_at":"2018-11-27T03:02:20Z","updated_at":"2018-11-27T03:02:20Z","author_association":"CONTRIBUTOR","body":"@alsrgv Sorry didn't get a chance to look at them the past few days, should be able to address the other comments this week."}}
{"actor":{"display_login":"karakusc"}}