{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4823","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4823/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4823/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4823/events","html_url":"https://github.com/tensorflow/tensorflow/issues/4823","id":181704145,"number":4823,"title":"Tensorflow not using GPU (according to TensorBoard)","user":{"login":"seeb0h","id":4989990,"avatar_url":"https://avatars.githubusercontent.com/u/4989990?v=3","gravatar_id":"","url":"https://api.github.com/users/seeb0h","html_url":"https://github.com/seeb0h","followers_url":"https://api.github.com/users/seeb0h/followers","following_url":"https://api.github.com/users/seeb0h/following{/other_user}","gists_url":"https://api.github.com/users/seeb0h/gists{/gist_id}","starred_url":"https://api.github.com/users/seeb0h/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seeb0h/subscriptions","organizations_url":"https://api.github.com/users/seeb0h/orgs","repos_url":"https://api.github.com/users/seeb0h/repos","events_url":"https://api.github.com/users/seeb0h/events{/privacy}","received_events_url":"https://api.github.com/users/seeb0h/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/awaiting%20googler","name":"awaiting googler","color":"b60205"},{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/awaiting%20response","name":"awaiting response","color":"5319e7"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-10-07T15:49:32Z","updated_at":"2016-10-10T14:43:31Z","closed_at":"2016-10-10T14:43:31Z","body":"GTX 1070, ubuntu 16.04.\r\n\r\nI am retraining [inception](https://github.com/tensorflow/models/tree/master/inception) model on my own data. Everything is fine until the final command :\r\n\r\n    bazel-bin/inception/flowers_train \\\r\n      --config=cuda \\\r\n      --train_dir=\"${TRAIN_DIR}\" \\\r\n      --data_dir=\"${OUTPUT_DIRECTORY}\" \\\r\n      --pretrained_model_checkpoint_path=\"${MODEL_PATH}\" \\\r\n      --fine_tune=True \\\r\n      --initial_learning_rate=0.001 \\\r\n      --input_queue_memory_factor=1\r\n\r\nAccording to the logs, **Tensorflow seems to be using the GPU** :\r\n\r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:\r\n    name: GeForce GTX 1070\r\n    major: 6 minor: 1 memoryClockRate (GHz) 1.7715\r\n    pciBusID 0000:03:00.0\r\n    Total memory: 7.92GiB\r\n    Free memory: 7.77GiB\r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0\r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y\r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:03:00.0)\r\n\r\nBut when I am checking the learning in TensorBoard, **the net is using mainly the CPU** (blue /device:CPU:0, green /device:GPU:0):\r\n\r\nTensorBoard graph:\r\n\r\n![graph](https://cloud.githubusercontent.com/assets/4989990/19196523/33a2e6c2-8cb6-11e6-85b1-1f35950657e3.png)\r\n\r\n\r\nI have tried this two TensorFlow setups :\r\n\r\n 1. Install from the source (3b75eb34ea2c4982fb80843be089f02d430faade) with nvidia-367 drivers, CUDA8 8.0, cuDNN\r\n    v5,    source from the master (16/10/06 - r11?). compiled for GPU\r\n    use:\r\n\r\n        bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer\r\n        bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu    \r\n        bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package\r\n\r\n 1. docker GPU image of Tensorflow on a PC with a GTX\r\n    1070 8Go\r\n\r\n        nvidia-docker run -it -p 8888:8888 -p 6006:6006 gcr.io/tensorflow/tensorflow:latest-gpu /bin/bash\r\n\r\nAccording to the doc Cuda >=7.0 can be used when installed from sources. \r\n\r\n> The GPU version works best with Cuda Toolkit 7.5 and cuDNN v5. Other versions are supported (Cuda toolkit >= 7.0 and cuDNN >= v3) only when installing from sources\r\n\r\n\r\nAnd the dockerfile begins with `FROM nvidia/cuda:7.5-cudnn5-devel`. So it is not using cuda and cudnn of the host system\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/252641912","html_url":"https://github.com/tensorflow/tensorflow/issues/4823#issuecomment-252641912","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4823","id":252641912,"user":{"login":"dsmilkov","id":2294279,"avatar_url":"https://avatars.githubusercontent.com/u/2294279?v=3","gravatar_id":"","url":"https://api.github.com/users/dsmilkov","html_url":"https://github.com/dsmilkov","followers_url":"https://api.github.com/users/dsmilkov/followers","following_url":"https://api.github.com/users/dsmilkov/following{/other_user}","gists_url":"https://api.github.com/users/dsmilkov/gists{/gist_id}","starred_url":"https://api.github.com/users/dsmilkov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsmilkov/subscriptions","organizations_url":"https://api.github.com/users/dsmilkov/orgs","repos_url":"https://api.github.com/users/dsmilkov/repos","events_url":"https://api.github.com/users/dsmilkov/events{/privacy}","received_events_url":"https://api.github.com/users/dsmilkov/received_events","type":"User","site_admin":false},"created_at":"2016-10-10T14:43:31Z","updated_at":"2016-10-10T14:43:31Z","body":"@seeb0h tower_0 is the namespace that holds the GPU ops. If you expand it, you will probably see that tower_0 contains conv_0,1,2... nodes as well which correspond to the actual convolution operations, whereas the external conv_0,1,2... nodes correspond to variables holding the weights of that convolution and variables are always allocated to the CPU.\r\n\r\nI will close this bug for now, but feel free to reopen if you have any other concerns.\r\n\r\nThanks!"}}
{"actor":{"display_login":"dsmilkov"}}