{"action":"created","issue":{"url":"https://api.github.com/repos/uber/horovod/issues/228","repository_url":"https://api.github.com/repos/uber/horovod","labels_url":"https://api.github.com/repos/uber/horovod/issues/228/labels{/name}","comments_url":"https://api.github.com/repos/uber/horovod/issues/228/comments","events_url":"https://api.github.com/repos/uber/horovod/issues/228/events","html_url":"https://github.com/uber/horovod/issues/228","id":309980913,"number":228,"title":"Any advice with large embedding layer with large batch size?","user":{"login":"TomorrowIsAnOtherDay","id":25046619,"avatar_url":"https://avatars0.githubusercontent.com/u/25046619?v=4","gravatar_id":"","url":"https://api.github.com/users/TomorrowIsAnOtherDay","html_url":"https://github.com/TomorrowIsAnOtherDay","followers_url":"https://api.github.com/users/TomorrowIsAnOtherDay/followers","following_url":"https://api.github.com/users/TomorrowIsAnOtherDay/following{/other_user}","gists_url":"https://api.github.com/users/TomorrowIsAnOtherDay/gists{/gist_id}","starred_url":"https://api.github.com/users/TomorrowIsAnOtherDay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TomorrowIsAnOtherDay/subscriptions","organizations_url":"https://api.github.com/users/TomorrowIsAnOtherDay/orgs","repos_url":"https://api.github.com/users/TomorrowIsAnOtherDay/repos","events_url":"https://api.github.com/users/TomorrowIsAnOtherDay/events{/privacy}","received_events_url":"https://api.github.com/users/TomorrowIsAnOtherDay/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-30T05:41:15Z","updated_at":"2018-03-31T20:02:01Z","closed_at":null,"author_association":"NONE","body":"Thanks for your nice job, Horovod can be used to train lots of  image models which have a lot of dense parameter.\r\nBut I found its performance with large embedding layer with large batch size is very slow,\r\nI supposed its bottleneck is \r\nhttps://github.com/uber/horovod/blob/85882b0b04afe207ae9d5b91cc1eef726dea0083/horovod/tensorflow/__init__.py#L68\r\nBecause it have to collect  lots of sparse gradients and broadcast it to all workers.\r\nAny advice on this task?"},"comment":{"url":"https://api.github.com/repos/uber/horovod/issues/comments/377719390","html_url":"https://github.com/uber/horovod/issues/228#issuecomment-377719390","issue_url":"https://api.github.com/repos/uber/horovod/issues/228","id":377719390,"user":{"login":"alsrgv","id":16640218,"avatar_url":"https://avatars2.githubusercontent.com/u/16640218?v=4","gravatar_id":"","url":"https://api.github.com/users/alsrgv","html_url":"https://github.com/alsrgv","followers_url":"https://api.github.com/users/alsrgv/followers","following_url":"https://api.github.com/users/alsrgv/following{/other_user}","gists_url":"https://api.github.com/users/alsrgv/gists{/gist_id}","starred_url":"https://api.github.com/users/alsrgv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alsrgv/subscriptions","organizations_url":"https://api.github.com/users/alsrgv/orgs","repos_url":"https://api.github.com/users/alsrgv/repos","events_url":"https://api.github.com/users/alsrgv/events{/privacy}","received_events_url":"https://api.github.com/users/alsrgv/received_events","type":"User","site_admin":false},"created_at":"2018-03-31T20:02:01Z","updated_at":"2018-03-31T20:02:01Z","author_association":"COLLABORATOR","body":"@TomorrowIsAnOtherDay, it would be great if we make Horovod work efficiently with sparse models as well.  Is there a good public benchmark which would approximate your workload so we could profile it internally?"}}
{"actor":{"display_login":"alsrgv"}}