{"action":"opened","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1812","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1812/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1812/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1812/events","html_url":"https://github.com/tensorflow/tensorflow/issues/1812","id":146679636,"number":1812,"title":"model parameters contains a lot of 'nan' at certain stage of training using AdamOptimizer","user":{"login":"jiaboli007","id":15619907,"avatar_url":"https://avatars.githubusercontent.com/u/15619907?v=3","gravatar_id":"","url":"https://api.github.com/users/jiaboli007","html_url":"https://github.com/jiaboli007","followers_url":"https://api.github.com/users/jiaboli007/followers","following_url":"https://api.github.com/users/jiaboli007/following{/other_user}","gists_url":"https://api.github.com/users/jiaboli007/gists{/gist_id}","starred_url":"https://api.github.com/users/jiaboli007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jiaboli007/subscriptions","organizations_url":"https://api.github.com/users/jiaboli007/orgs","repos_url":"https://api.github.com/users/jiaboli007/repos","events_url":"https://api.github.com/users/jiaboli007/events{/privacy}","received_events_url":"https://api.github.com/users/jiaboli007/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-07T16:39:25Z","updated_at":"2016-04-07T16:39:25Z","closed_at":null,"body":"I am training a deep neural network using a simple and well defined case. The training goes very well and reach a high accuracy (~95%), and then, suddenly, just one step iteration, the accuracy is drop to almost zero. I then printed out all the model parameters, and there are lot of 'nan', something like this \r\n....0.076455489, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, 0.08646702....\r\n\r\nHere is the information about my system.\r\n\r\n1) Installation. I installed tensor flow from source code on AWS ubuntu g2 instance (GPU instance). During the installation, I use customized configuration so that the tensor flow can run on g2 instance with computing capacity 3.0. \r\n\r\n2) DNN structure: I have a 30 hidden fully connected layers. Each layer has 32 nodes. To make the DNN more stable, I have added short-cuts for every three layers (similar to a recent publication from Microsoft Research \"Deep Residual Learning for Image Recognition\", by Kaiming He et al.) \r\n\r\n3) Optimizer: AdamOptimizer. \r\n  I first found the problem with AdamOptimzer. However, I just realized that the same problem also happens to GradientDescentOptimizer.\r\n\r\n4) Procedure to produce the bug. \r\n\r\nI have a saved model, which has good accuracy, and I restart the training by reading the saved model, and the problem shows up for the first iteration. \r\n\r\nI wonder if this is a known issue or not. I would be happy to provide my code and input data if this is necessary. I need to create a simple case with simplified code as I am working on a production code that I cannot post it here. \r\n\r\n"}}