{"action":"created","issue":{"url":"https://api.github.com/repos/uber/horovod/issues/297","repository_url":"https://api.github.com/repos/uber/horovod","labels_url":"https://api.github.com/repos/uber/horovod/issues/297/labels{/name}","comments_url":"https://api.github.com/repos/uber/horovod/issues/297/comments","events_url":"https://api.github.com/repos/uber/horovod/issues/297/events","html_url":"https://github.com/uber/horovod/issues/297","id":330238277,"node_id":"MDU6SXNzdWUzMzAyMzgyNzc=","number":297,"title":"horovod hang on with TensorBoard","user":{"login":"burness","id":3112825,"node_id":"MDQ6VXNlcjMxMTI4MjU=","avatar_url":"https://avatars2.githubusercontent.com/u/3112825?v=4","gravatar_id":"","url":"https://api.github.com/users/burness","html_url":"https://github.com/burness","followers_url":"https://api.github.com/users/burness/followers","following_url":"https://api.github.com/users/burness/following{/other_user}","gists_url":"https://api.github.com/users/burness/gists{/gist_id}","starred_url":"https://api.github.com/users/burness/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burness/subscriptions","organizations_url":"https://api.github.com/users/burness/orgs","repos_url":"https://api.github.com/users/burness/repos","events_url":"https://api.github.com/users/burness/events{/privacy}","received_events_url":"https://api.github.com/users/burness/received_events","type":"User","site_admin":false},"labels":[{"id":665452437,"node_id":"MDU6TGFiZWw2NjU0NTI0Mzc=","url":"https://api.github.com/repos/uber/horovod/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-06-07T11:51:45Z","updated_at":"2018-06-13T09:07:31Z","closed_at":null,"author_association":"NONE","body":"I run my job successfuly with no TensorBoard:\r\n\r\n![image](https://user-images.githubusercontent.com/3112825/41097844-f58e715c-6a8b-11e8-9984-a0c97c811d85.png)\r\nBut if I uncomment the `callbacks.append(keras.callbacks.TensorBoard(config.log_dir, histogram_freq=1, batch_size = config.batch_size, write_grads=True))`\r\n\r\nIt seems to be hang on and here is the log:\r\n![image](https://user-images.githubusercontent.com/3112825/41097880-1f471684-6a8c-11e8-8469-9e7e83408ed5.png)\r\nwhy there is no rank 0 in ready ranks ?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/uber/horovod/issues/comments/396869018","html_url":"https://github.com/uber/horovod/issues/297#issuecomment-396869018","issue_url":"https://api.github.com/repos/uber/horovod/issues/297","id":396869018,"node_id":"MDEyOklzc3VlQ29tbWVudDM5Njg2OTAxOA==","user":{"login":"burness","id":3112825,"node_id":"MDQ6VXNlcjMxMTI4MjU=","avatar_url":"https://avatars2.githubusercontent.com/u/3112825?v=4","gravatar_id":"","url":"https://api.github.com/users/burness","html_url":"https://github.com/burness","followers_url":"https://api.github.com/users/burness/followers","following_url":"https://api.github.com/users/burness/following{/other_user}","gists_url":"https://api.github.com/users/burness/gists{/gist_id}","starred_url":"https://api.github.com/users/burness/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burness/subscriptions","organizations_url":"https://api.github.com/users/burness/orgs","repos_url":"https://api.github.com/users/burness/repos","events_url":"https://api.github.com/users/burness/events{/privacy}","received_events_url":"https://api.github.com/users/burness/received_events","type":"User","site_admin":false},"created_at":"2018-06-13T09:07:31Z","updated_at":"2018-06-13T09:07:31Z","author_association":"NONE","body":"@alsrgv I use the 0.12.1 horovod, what mean is the `validation_data` ? "}}
{"actor":{"display_login":"burness"}}