{"action":"closed","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3377","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3377/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3377/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3377/events","html_url":"https://github.com/tensorflow/tensorflow/issues/3377","id":166212411,"number":3377,"title":"Moving data from CPU to GPU is slow ","user":{"login":"lizallendorf","id":20078720,"avatar_url":"https://avatars.githubusercontent.com/u/20078720?v=3","gravatar_id":"","url":"https://api.github.com/users/lizallendorf","html_url":"https://github.com/lizallendorf","followers_url":"https://api.github.com/users/lizallendorf/followers","following_url":"https://api.github.com/users/lizallendorf/following{/other_user}","gists_url":"https://api.github.com/users/lizallendorf/gists{/gist_id}","starred_url":"https://api.github.com/users/lizallendorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lizallendorf/subscriptions","organizations_url":"https://api.github.com/users/lizallendorf/orgs","repos_url":"https://api.github.com/users/lizallendorf/repos","events_url":"https://api.github.com/users/lizallendorf/events{/privacy}","received_events_url":"https://api.github.com/users/lizallendorf/received_events","type":"User","site_admin":false},"labels":[{"id":386191887,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/awaiting%20response","name":"awaiting response","color":"5319e7","default":false}],"state":"closed","locked":false,"assignee":{"login":"zheng-xq","id":15736910,"avatar_url":"https://avatars.githubusercontent.com/u/15736910?v=3","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false},"assignees":[{"login":"zheng-xq","id":15736910,"avatar_url":"https://avatars.githubusercontent.com/u/15736910?v=3","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false}],"milestone":null,"comments":12,"created_at":"2016-07-18T22:56:16Z","updated_at":"2016-11-14T19:04:24Z","closed_at":"2016-11-14T19:04:24Z","body":"I have noticed that when moving MNIST data from the CPU to the GPU, there is a significant time lag when using TensorFlow in comparison to Theano. Specifically, we have noticed this problem in the context of feed_dict, which moves information from the CPU to the GPU when running minibatches. I am using python 2.7. Our current solution to this problem is to move all of the data directly to the GPU at the beginning of the program, which is of course not sustainable unless one has a significant amount of space on their GPU. \n\nWhen we time 25000 minibatches of size 100 each, TensorFlow is approximately four times as slow as Theano. I have attached both files, and the time difference is evident in the final output. \n### Environment info\n\nOperating System: Ubuntu 14.04\n\nCPU Information:\n\nArchitecture:          x86_64\nCPU op-mode(s):        32-bit, 64-bit\nByte Order:            Little Endian\nCPU(s):                12\nOn-line CPU(s) list:   0-11\nThread(s) per core:    2\nCore(s) per socket:    6\nSocket(s):             1\nNUMA node(s):          1\nVendor ID:             GenuineIntel\nCPU family:            6\nModel:                 63\nStepping:              2\nCPU MHz:               1246.328\nBogoMIPS:              6995.89\nVirtualization:        VT-x\nL1d cache:             32K\nL1i cache:             32K\nL2 cache:              256K\nL3 cache:              15360K\nNUMA node0 CPU(s):     0-11\n\nGPU: NVidia GeForce GTX TITAN X Graphics Card (12GB) \n\nInstalled version of CUDA and cuDNN: 7.5.17\n### Logs or other output that would be helpful\n\n(If logs are large, please upload as attachment).\n\nRelevant files (I apologize for the poor naming conventions):\n\n1.[mnist_softmax.txt](https://github.com/tensorflow/tensorflow/files/370296/mnist_softmax.txt)\nThis file contains the relevant program that uses TensorFlow. \n1. [mnist.pkl.gz](https://github.com/tensorflow/tensorflow/files/370306/mnist.pkl.gz)\n   This file contains the dataset for the TensorFlow file. \n2.  [mnist_softmax_theano.txt(https://github.com/tensorflow/tensorflow/files/370314/mnist_softmax_theano.txt)\n   This file contains the relevant program that uses Theano.\n3. [tf_data.pkl.gz](https://github.com/tensorflow/tensorflow/files/370310/tf_data.pkl.gz)\n   This file contains the dataset necessary for the _Theano_ file.\n"}}
{"actor":{"display_login":"girving"}}