{"action":"opened","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3563","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3563/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3563/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3563/events","html_url":"https://github.com/tensorflow/tensorflow/issues/3563","id":168291455,"number":3563,"title":"[BUG?] Memory overflow (?) on Nvidia Quadro M6000 using feed dictionary with conv2d","user":{"login":"eickenberg","id":1306635,"avatar_url":"https://avatars.githubusercontent.com/u/1306635?v=3","gravatar_id":"","url":"https://api.github.com/users/eickenberg","html_url":"https://github.com/eickenberg","followers_url":"https://api.github.com/users/eickenberg/followers","following_url":"https://api.github.com/users/eickenberg/following{/other_user}","gists_url":"https://api.github.com/users/eickenberg/gists{/gist_id}","starred_url":"https://api.github.com/users/eickenberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eickenberg/subscriptions","organizations_url":"https://api.github.com/users/eickenberg/orgs","repos_url":"https://api.github.com/users/eickenberg/repos","events_url":"https://api.github.com/users/eickenberg/events{/privacy}","received_events_url":"https://api.github.com/users/eickenberg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-07-29T09:59:33Z","updated_at":"2016-07-29T09:59:33Z","closed_at":null,"body":"Hi everybody,\r\n\r\njust got several machines with Quadro M6000 (12G RAM) for our lab, and they all seem to suffer the same problem when feeding too much data to a convolution via a feed dictionary. The problem does not appear when using CPU on the same machine, and neither when using an equivalent machine with a different GPU (GeForce GTX Titan X).\r\n\r\nIt looks like when using a feed dictionary to provide data for the evaluation of a graph, part of the weights in the graph may be overwritten depending on the size of the data fed through the dictionary.\r\n\r\n#### Two ways to reproduce\r\n##### A minimal example \r\n\r\n```\r\nimport tensorflow as tf\r\nimport numpy as np\r\nx = tf.placeholder(tf.float32, shape=(None, 1024))\r\nx_r = tf.reshape(x, (-1, 32, 32, 1))\r\nW = tf.Variable(tf.ones((5, 5, 1, 32)))\r\nW_sum = tf.reduce_sum(W)\r\nconv = tf.nn.conv2d(x_r, W, strides=(1, 1, 1, 1), padding='SAME')\r\nsess = tf.Session()\r\nsess.run(tf.initialize_all_variables())\r\ndata = np.ones((5001, 1024), dtype='float32')\r\nprint(sess.run(W_sum))\r\nc = sess.run(conv, feed_dict={x: data})\r\nprint(sess.run(W_sum))\r\nsess.close()\r\n```\r\nIt outputs\r\n```\r\n800.0\r\n0.0\r\n```\r\nwhich indicates that `W` has changed through the feed-dictionary evaluation of the convolution.\r\n\r\nNote that\r\n* By reducing `data.shape[0]` to 5000 or less, the problem disappears\r\n* Going up to 5070, the second value stays at 0.0. \r\n* at exactly 5080 the problem seems to disappear (we get `800.0 800.0`)\r\n* from 5100 the second number can take arbitrary values (not zero)\r\n* The problem does not arise when replacing the `tf.nn.conv2d` operation by an equivalently sized  `matmul` operation (same dimensions of weight vector, not same dimensionality of output)\r\n\r\n##### The expert MNIST tutorial\r\nThe problem also arises in one of the very first tutorials https://www.tensorflow.org/versions/r0.9/tutorials/mnist/pros/index.html\r\n\r\nEverything works perfect until the last line of the tutorial, where a large array is fed by feed-dictionary for test accuracy. This modifies all the weight vectors of the convnet!\r\n\r\nAgain, the error disappears when feeding only 5000 test examples instead of 10000 to the architecture.\r\n\r\n#### Environment\r\n* OS: Ubuntu 1504\r\n* CUDA version 7.5\r\n* cuDNN version 4\r\n* library files\r\n```\r\n(tensorflow) meickenb@g-1504:~/code/tensorflow$ ls -l /usr/local/cuda/lib64/libcud*\r\n-rw-r--r-- 1 root root 322936 Aug 15  2015 /usr/local/cuda/lib64/libcudadevrt.a\r\nlrwxrwxrwx 1 root root     16 Aug  15  2015 /usr/local/cuda/lib64/libcudart.so -> libcudart.so.7.5\r\nlrwxrwxrwx 1 root root     19 Aug  15  2015 /usr/local/cuda/lib64/libcudart.so.7.5 -> libcudart.so.7.5.18\r\n-rwxr-xr-x 1 root root 383336 Aug  15  2015 /usr/local/cuda/lib64/libcudart.so.7.5.18\r\n-rw-r--r-- 1 root root 720192 Aug  15  2015 /usr/local/cuda/lib64/libcudart_static.a\r\n\r\n(tensorflow) meickenb@g-1504:/usr/lib/x86_64-linux-gnu$ ls -l libcudnn*\r\nlrwxrwxrwx 1 root root       13 Jul 21 18:22 libcudnn.so -> libcudnn.so.4\r\nlrwxrwxrwx 1 root root       17 Jul 21 18:22 libcudnn.so.4 -> libcudnn.so.4.0.7\r\n-rwxr-xr-x 1 root root 61453024 Jul 21 18:22 libcudnn.so.4.0.7\r\n-rw-r--r-- 1 root root 62025862 Jul 21 18:22 libcudnn_static.a\r\n```\r\n* Anaconda 3\r\n* pip install from wheel `https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.9.0-cp35-cp35m-linux_x86_64.whl`\r\n* tf version 0.9.0\r\n```\r\n(tensorflow) meickenb@g-1504:~$ python -c \"import tensorflow; print(tensorflow.__version__)\"\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally\r\n0.9.0\r\n\r\n```\r\n\r\n\r\nPlease let me know if you need any other info, or what else I could try to narrow down the reason for this overflow.\r\n\r\nCC @tmanglesl @CarmineCella @andreuxmath @beedotkiran"}}
{"actor":{"display_login":"eickenberg"}}