{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5276","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5276/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5276/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5276/events","html_url":"https://github.com/tensorflow/tensorflow/issues/5276","id":186073917,"number":5276,"title":"TensorBoard histograms NaN value handling","user":{"login":"timmeinhardt","id":7427306,"avatar_url":"https://avatars.githubusercontent.com/u/7427306?v=3","gravatar_id":"","url":"https://api.github.com/users/timmeinhardt","html_url":"https://github.com/timmeinhardt","followers_url":"https://api.github.com/users/timmeinhardt/followers","following_url":"https://api.github.com/users/timmeinhardt/following{/other_user}","gists_url":"https://api.github.com/users/timmeinhardt/gists{/gist_id}","starred_url":"https://api.github.com/users/timmeinhardt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timmeinhardt/subscriptions","organizations_url":"https://api.github.com/users/timmeinhardt/orgs","repos_url":"https://api.github.com/users/timmeinhardt/repos","events_url":"https://api.github.com/users/timmeinhardt/events{/privacy}","received_events_url":"https://api.github.com/users/timmeinhardt/received_events","type":"User","site_admin":false},"labels":[{"id":283394348,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-29T11:35:20Z","updated_at":"2016-10-29T16:52:07Z","closed_at":null,"body":"Recently I had to fight a lot with exploding gradients and thereby weights turning NaN after only several iterations event hough I was using _tf.contrib.layers.variance_scaling_initializer_ correctly. To monitor the problem I started saving weight histograms which seemed totally messed up. I thought this was due to me using the initializer wrongly and only after some debugging I realised my initialization was totally fine. The NaN values appearing in iteration _n_ seem to mess with the visualization of values in previous iterations which seems like a very misleading behaviour to me. Histograms and distributions should be visualized correctly until this is not really possible because of ill values.\r\n\r\nI can easily provide tensorflow summary data to reproduce this behaviour."},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/257102438","html_url":"https://github.com/tensorflow/tensorflow/issues/5276#issuecomment-257102438","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5276","id":257102438,"user":{"login":"drpngx","id":20959853,"avatar_url":"https://avatars.githubusercontent.com/u/20959853?v=3","gravatar_id":"","url":"https://api.github.com/users/drpngx","html_url":"https://github.com/drpngx","followers_url":"https://api.github.com/users/drpngx/followers","following_url":"https://api.github.com/users/drpngx/following{/other_user}","gists_url":"https://api.github.com/users/drpngx/gists{/gist_id}","starred_url":"https://api.github.com/users/drpngx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drpngx/subscriptions","organizations_url":"https://api.github.com/users/drpngx/orgs","repos_url":"https://api.github.com/users/drpngx/repos","events_url":"https://api.github.com/users/drpngx/events{/privacy}","received_events_url":"https://api.github.com/users/drpngx/received_events","type":"User","site_admin":false},"created_at":"2016-10-29T16:52:07Z","updated_at":"2016-10-29T16:52:07Z","body":"Yes, @timmeinhardt please attach summary data in case we want to repro."}}
{"actor":{"display_login":"drpngx"}}