{"action":"created","issue":{"url":"https://api.github.com/repos/uber/horovod/issues/255","repository_url":"https://api.github.com/repos/uber/horovod","labels_url":"https://api.github.com/repos/uber/horovod/issues/255/labels{/name}","comments_url":"https://api.github.com/repos/uber/horovod/issues/255/comments","events_url":"https://api.github.com/repos/uber/horovod/issues/255/events","html_url":"https://github.com/uber/horovod/issues/255","id":318593995,"number":255,"title":"How to analyze timeline log with horovod","user":{"login":"scotthuang1989","id":5325686,"avatar_url":"https://avatars3.githubusercontent.com/u/5325686?v=4","gravatar_id":"","url":"https://api.github.com/users/scotthuang1989","html_url":"https://github.com/scotthuang1989","followers_url":"https://api.github.com/users/scotthuang1989/followers","following_url":"https://api.github.com/users/scotthuang1989/following{/other_user}","gists_url":"https://api.github.com/users/scotthuang1989/gists{/gist_id}","starred_url":"https://api.github.com/users/scotthuang1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scotthuang1989/subscriptions","organizations_url":"https://api.github.com/users/scotthuang1989/orgs","repos_url":"https://api.github.com/users/scotthuang1989/repos","events_url":"https://api.github.com/users/scotthuang1989/events{/privacy}","received_events_url":"https://api.github.com/users/scotthuang1989/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-28T02:55:01Z","updated_at":"2018-04-28T08:01:38Z","closed_at":null,"author_association":"NONE","body":"I post this  issue on [stackoverlfow](https://stackoverflow.com/questions/50072398/how-to-analyze-timeline-log-with-horovod), but I found that no horovod tag. so I put it on tensorflow tag and hope I can find more useful advice at here.\r\n\r\nAfter some debug, I found a very extreme situation, the **NEGOTIATE_ALLREDUCE**  takes a lot time. \r\nNCCL is installed and I have set the **LD_LIBRARY_PATH** correctly. \r\n\r\nAnyone know what's going on?\r\n\r\n![image](https://user-images.githubusercontent.com/5325686/39393010-908a304c-4af2-11e8-8b6f-6c56b34ea8af.png)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/uber/horovod/issues/comments/385150970","html_url":"https://github.com/uber/horovod/issues/255#issuecomment-385150970","issue_url":"https://api.github.com/repos/uber/horovod/issues/255","id":385150970,"user":{"login":"alsrgv","id":16640218,"avatar_url":"https://avatars2.githubusercontent.com/u/16640218?v=4","gravatar_id":"","url":"https://api.github.com/users/alsrgv","html_url":"https://github.com/alsrgv","followers_url":"https://api.github.com/users/alsrgv/followers","following_url":"https://api.github.com/users/alsrgv/following{/other_user}","gists_url":"https://api.github.com/users/alsrgv/gists{/gist_id}","starred_url":"https://api.github.com/users/alsrgv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alsrgv/subscriptions","organizations_url":"https://api.github.com/users/alsrgv/orgs","repos_url":"https://api.github.com/users/alsrgv/repos","events_url":"https://api.github.com/users/alsrgv/events{/privacy}","received_events_url":"https://api.github.com/users/alsrgv/received_events","type":"User","site_admin":false},"created_at":"2018-04-28T08:01:38Z","updated_at":"2018-04-28T08:01:38Z","author_association":"COLLABORATOR","body":"@scotthuang1989, `NEGOTIATE_ALLREDUCE` captures the timeline of when each worker sent readiness event for a particular tensor.  If you see this phase take 30 seconds, it means there's a very large imbalance between your workers.  What's your time per iteration on a single GPU?"}}
{"actor":{"display_login":"alsrgv"}}