{"action":"created","issue":{"url":"https://api.github.com/repos/uber/horovod/issues/411","repository_url":"https://api.github.com/repos/uber/horovod","labels_url":"https://api.github.com/repos/uber/horovod/issues/411/labels{/name}","comments_url":"https://api.github.com/repos/uber/horovod/issues/411/comments","events_url":"https://api.github.com/repos/uber/horovod/issues/411/events","html_url":"https://github.com/uber/horovod/pull/411","id":345985221,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA0OTYzNTA1","number":411,"title":"Parallelize hierarchical allreduce algorithm","user":{"login":"karakusc","id":11875690,"node_id":"MDQ6VXNlcjExODc1Njkw","avatar_url":"https://avatars0.githubusercontent.com/u/11875690?v=4","gravatar_id":"","url":"https://api.github.com/users/karakusc","html_url":"https://github.com/karakusc","followers_url":"https://api.github.com/users/karakusc/followers","following_url":"https://api.github.com/users/karakusc/following{/other_user}","gists_url":"https://api.github.com/users/karakusc/gists{/gist_id}","starred_url":"https://api.github.com/users/karakusc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karakusc/subscriptions","organizations_url":"https://api.github.com/users/karakusc/orgs","repos_url":"https://api.github.com/users/karakusc/repos","events_url":"https://api.github.com/users/karakusc/events{/privacy}","received_events_url":"https://api.github.com/users/karakusc/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"alsrgv","id":16640218,"node_id":"MDQ6VXNlcjE2NjQwMjE4","avatar_url":"https://avatars2.githubusercontent.com/u/16640218?v=4","gravatar_id":"","url":"https://api.github.com/users/alsrgv","html_url":"https://github.com/alsrgv","followers_url":"https://api.github.com/users/alsrgv/followers","following_url":"https://api.github.com/users/alsrgv/following{/other_user}","gists_url":"https://api.github.com/users/alsrgv/gists{/gist_id}","starred_url":"https://api.github.com/users/alsrgv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alsrgv/subscriptions","organizations_url":"https://api.github.com/users/alsrgv/orgs","repos_url":"https://api.github.com/users/alsrgv/repos","events_url":"https://api.github.com/users/alsrgv/events{/privacy}","received_events_url":"https://api.github.com/users/alsrgv/received_events","type":"User","site_admin":false},"assignees":[{"login":"alsrgv","id":16640218,"node_id":"MDQ6VXNlcjE2NjQwMjE4","avatar_url":"https://avatars2.githubusercontent.com/u/16640218?v=4","gravatar_id":"","url":"https://api.github.com/users/alsrgv","html_url":"https://github.com/alsrgv","followers_url":"https://api.github.com/users/alsrgv/followers","following_url":"https://api.github.com/users/alsrgv/following{/other_user}","gists_url":"https://api.github.com/users/alsrgv/gists{/gist_id}","starred_url":"https://api.github.com/users/alsrgv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alsrgv/subscriptions","organizations_url":"https://api.github.com/users/alsrgv/orgs","repos_url":"https://api.github.com/users/alsrgv/repos","events_url":"https://api.github.com/users/alsrgv/events{/privacy}","received_events_url":"https://api.github.com/users/alsrgv/received_events","type":"User","site_admin":false}],"milestone":null,"comments":10,"created_at":"2018-07-31T00:11:46Z","updated_at":"2018-09-05T20:30:27Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/uber/horovod/pulls/411","html_url":"https://github.com/uber/horovod/pull/411","diff_url":"https://github.com/uber/horovod/pull/411.diff","patch_url":"https://github.com/uber/horovod/pull/411.patch"},"body":"Replacing the existing hierarchical allreduce algorithm consisting of NCCL Reduce - MPI Allreduce - NCCL Broadcast steps with a new one consisting of NCCL ReduceScatter - Parallelized MPI Allreduce - NCCL Allgather to better make use of network bandwidth and improve performance."},"comment":{"url":"https://api.github.com/repos/uber/horovod/issues/comments/418870874","html_url":"https://github.com/uber/horovod/pull/411#issuecomment-418870874","issue_url":"https://api.github.com/repos/uber/horovod/issues/411","id":418870874,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODg3MDg3NA==","user":{"login":"alsrgv","id":16640218,"node_id":"MDQ6VXNlcjE2NjQwMjE4","avatar_url":"https://avatars2.githubusercontent.com/u/16640218?v=4","gravatar_id":"","url":"https://api.github.com/users/alsrgv","html_url":"https://github.com/alsrgv","followers_url":"https://api.github.com/users/alsrgv/followers","following_url":"https://api.github.com/users/alsrgv/following{/other_user}","gists_url":"https://api.github.com/users/alsrgv/gists{/gist_id}","starred_url":"https://api.github.com/users/alsrgv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alsrgv/subscriptions","organizations_url":"https://api.github.com/users/alsrgv/orgs","repos_url":"https://api.github.com/users/alsrgv/repos","events_url":"https://api.github.com/users/alsrgv/events{/privacy}","received_events_url":"https://api.github.com/users/alsrgv/received_events","type":"User","site_admin":false},"created_at":"2018-09-05T20:30:27Z","updated_at":"2018-09-05T20:30:27Z","author_association":"COLLABORATOR","body":"@karakusc, here are the timelines for the base and fixed state:\r\n[baseline-tl.json.gz](https://github.com/uber/horovod/files/2354445/baseline-tl.json.gz) and \r\n[fix64-v2-tl.json.gz](https://github.com/uber/horovod/files/2354450/fix64-v2-tl.json.gz).  It appears that `ncclAllGather` gets a big speedup with padding.\r\n\r\nCould you convert `64` to the constant and rebase your branch on master?  I'll run a couple of convergence tests and we should be good to merge :-)\r\n\r\n"}}
{"actor":{"display_login":"alsrgv"}}