{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2037","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2037/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2037/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2037/events","html_url":"https://github.com/tensorflow/tensorflow/issues/2037","id":149814648,"number":2037,"title":"NaN problem on tutorials_example_trainer with tensorflow 0.8","user":{"login":"ernest-tg","id":4499498,"avatar_url":"https://avatars.githubusercontent.com/u/4499498?v=3","gravatar_id":"","url":"https://api.github.com/users/ernest-tg","html_url":"https://github.com/ernest-tg","followers_url":"https://api.github.com/users/ernest-tg/followers","following_url":"https://api.github.com/users/ernest-tg/following{/other_user}","gists_url":"https://api.github.com/users/ernest-tg/gists{/gist_id}","starred_url":"https://api.github.com/users/ernest-tg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ernest-tg/subscriptions","organizations_url":"https://api.github.com/users/ernest-tg/orgs","repos_url":"https://api.github.com/users/ernest-tg/repos","events_url":"https://api.github.com/users/ernest-tg/events{/privacy}","received_events_url":"https://api.github.com/users/ernest-tg/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/awaiting%20response","name":"awaiting response","color":"5319e7"}],"state":"closed","locked":false,"assignee":{"login":"prb12","id":11547801,"avatar_url":"https://avatars.githubusercontent.com/u/11547801?v=3","gravatar_id":"","url":"https://api.github.com/users/prb12","html_url":"https://github.com/prb12","followers_url":"https://api.github.com/users/prb12/followers","following_url":"https://api.github.com/users/prb12/following{/other_user}","gists_url":"https://api.github.com/users/prb12/gists{/gist_id}","starred_url":"https://api.github.com/users/prb12/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prb12/subscriptions","organizations_url":"https://api.github.com/users/prb12/orgs","repos_url":"https://api.github.com/users/prb12/repos","events_url":"https://api.github.com/users/prb12/events{/privacy}","received_events_url":"https://api.github.com/users/prb12/received_events","type":"User","site_admin":false},"assignees":[{"login":"prb12","id":11547801,"avatar_url":"https://avatars.githubusercontent.com/u/11547801?v=3","gravatar_id":"","url":"https://api.github.com/users/prb12","html_url":"https://github.com/prb12","followers_url":"https://api.github.com/users/prb12/followers","following_url":"https://api.github.com/users/prb12/following{/other_user}","gists_url":"https://api.github.com/users/prb12/gists{/gist_id}","starred_url":"https://api.github.com/users/prb12/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prb12/subscriptions","organizations_url":"https://api.github.com/users/prb12/orgs","repos_url":"https://api.github.com/users/prb12/repos","events_url":"https://api.github.com/users/prb12/events{/privacy}","received_events_url":"https://api.github.com/users/prb12/received_events","type":"User","site_admin":false}],"milestone":null,"comments":15,"created_at":"2016-04-20T16:09:25Z","updated_at":"2016-08-21T21:07:40Z","closed_at":"2016-07-19T17:57:19Z","body":"I worked for several months on versions of tensorflow installed from sources. Today, I tried to upgrade it to 0.8. The installation seemed to go fine.  \"bazel-bin/tensorflow/cc/tutorials_example_trainer\" gave the expected result. However, the output of \"bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu\" is unexpected: the first lines seem to be fine but then NaN values appear, the proportion of NaN values increases until there is nothing but NaN values (see the first half of the output here http://pastebin.com/EhuD2Z5S , the remaining lines are only \"nan\" lines which I omitted because the total output exceeded pastebin limit).\r\n\r\nI expected this tutorials_example_trainer to either work seamlessly or not at all (if I provided wrong paths for CUDA and CuDNN), but this in-between puzzles me.\r\n\r\nI tried to debug it with my own programs, but I can not import tensorflow :\r\n```\r\nimport tensorflow\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"tensorflow/__init__.py\", line 23, in <module>\r\n    from tensorflow.python import *\r\n  File \"tensorflow/python/__init__.py\", line 45, in <module>\r\n    from tensorflow.python import pywrap_tensorflow\r\nImportError: cannot import name pywrap_tensorflow\r\n```\r\nI am not sure whether the two problems are related.\r\n\r\nIn the case I made a mistake during configuration, I typed:\r\n```\r\nmatthieu@gpu2:~/external/tensorflow$ ./configure \r\nPlease specify the location of python. [Default is /usr/bin/python]: \r\nDo you wish to build TensorFlow with GPU support? [y/N] y\r\nGPU support will be enabled for TensorFlow\r\nPlease specify which gcc nvcc should use as the host compiler. [Default is /usr/bin/gcc]: \r\nPlease specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 7.0\r\nPlease specify the location where CUDA 7.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: \r\nPlease specify the Cudnn version you want to use. [Leave empty to use system default]: 4\r\nPlease specify the location where cuDNN 4 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: \r\nPlease specify a list of comma-separated Cuda compute capabilities you want to build with.\r\nYou can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.\r\nPlease note that each additional compute capability significantly increases your build time and binary size.\r\n[Default is: \"3.5,5.2\"]: \r\nSetting up Cuda include\r\nSetting up Cuda lib64\r\nSetting up Cuda bin\r\nSetting up Cuda nvvm\r\nSetting up CUPTI include\r\nSetting up CUPTI lib64\r\nConfiguration finished\"\r\n```\r\n\r\nI hope this issue belongs here, else I will try stackoverflow.\r\n\r\nThanks in advance,\r\n\r\n\r\n\r\n\r\n### Environment info\r\nI work on ubuntu 14.04\r\n\r\nWith CUDA 7.0 and cuDNN 4\r\n```\r\nls /usr/local/cuda-7.0/lib/libcud*        \r\n/usr/local/cuda-7.0/lib/libcudadevrt.a  /usr/local/cuda-7.0/lib/libcudart.so.7.0     /usr/local/cuda-7.0/lib/libcudart_static.a\r\n/usr/local/cuda-7.0/lib/libcudart.so    /usr/local/cuda-7.0/lib/libcudart.so.7.0.28\r\n```\r\n\r\nCommit hash: e1c017624fd6bd8562c1da27e0014b32f389b524"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/241282181","html_url":"https://github.com/tensorflow/tensorflow/issues/2037#issuecomment-241282181","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2037","id":241282181,"user":{"login":"tuba","id":187892,"avatar_url":"https://avatars.githubusercontent.com/u/187892?v=3","gravatar_id":"","url":"https://api.github.com/users/tuba","html_url":"https://github.com/tuba","followers_url":"https://api.github.com/users/tuba/followers","following_url":"https://api.github.com/users/tuba/following{/other_user}","gists_url":"https://api.github.com/users/tuba/gists{/gist_id}","starred_url":"https://api.github.com/users/tuba/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tuba/subscriptions","organizations_url":"https://api.github.com/users/tuba/orgs","repos_url":"https://api.github.com/users/tuba/repos","events_url":"https://api.github.com/users/tuba/events{/privacy}","received_events_url":"https://api.github.com/users/tuba/received_events","type":"User","site_admin":false},"created_at":"2016-08-21T21:07:40Z","updated_at":"2016-08-21T21:07:40Z","body":"Ubuntu 16.04, nVidia driver 367.35, GTX 1080"}}
{"actor":{"display_login":"tuba"}}