{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/24196","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/24196/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/24196/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/24196/events","html_url":"https://github.com/tensorflow/tensorflow/issues/24196","id":388218128,"node_id":"MDU6SXNzdWUzODgyMTgxMjg=","number":24196,"title":"Possibly buffer overflow in tf.nn.conv2d on GPU","user":{"login":"noamwies","id":3121971,"node_id":"MDQ6VXNlcjMxMjE5NzE=","avatar_url":"https://avatars3.githubusercontent.com/u/3121971?v=4","gravatar_id":"","url":"https://api.github.com/users/noamwies","html_url":"https://github.com/noamwies","followers_url":"https://api.github.com/users/noamwies/followers","following_url":"https://api.github.com/users/noamwies/following{/other_user}","gists_url":"https://api.github.com/users/noamwies/gists{/gist_id}","starred_url":"https://api.github.com/users/noamwies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noamwies/subscriptions","organizations_url":"https://api.github.com/users/noamwies/orgs","repos_url":"https://api.github.com/users/noamwies/repos","events_url":"https://api.github.com/users/noamwies/events{/privacy}","received_events_url":"https://api.github.com/users/noamwies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-06T13:39:27Z","updated_at":"2018-12-10T08:00:32Z","closed_at":null,"author_association":"NONE","body":"**System information**\r\n- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes\r\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux \r\n- TensorFlow installed from (source or binary): binary\r\n- TensorFlow version (use command below): 1.12\r\n- Python version: 3.5.3\r\n- CUDA/cuDNN version: 9.0 / 7.2.1\r\n- GPU model and memory: Tesla M60 (also tested on GeForce GTX TITAN Black)\r\n\r\n**Describe the current behavior**\r\nI've created a tensor set to zero, except a single entry at [:,-1,-1,0] that was set to a large number 1e10. I've then convolved the tensor with a random kernel. When the batch size is small (e.g. 5) then the output at [:,0,0,:] is zero (because the input array is zero there), but when the batch size is large (e.g. 100) then these entries contain numbers significantly larger than zero. This only happens on the GPU.\r\n**Describe the expected behavior**\r\nThe expected behavior should be zero regardless if running on CPU or GPU, because windows at these locations contain only zero entries. Because the only non-zero entry is outside the window used to compute the value, then this is most likely a buffer overflow (or some other memory access issue).\r\n**Code to reproduce the issue**\r\nIf the following does not immediately produces the bug, then try increasing the batch size from 100 to something greater.\r\n```python\r\nimport tensorflow as tf\r\nimport numpy as np\r\n\r\n\r\ndef test(sess):\r\n\tw = np.random.uniform(size=(5,5,64,64)).astype(np.float32)\r\n\tx_t = tf.placeholder(dtype=tf.float32, shape=(None, 32, 32, 64))\r\n\tw_t = tf.constant(w)\r\n\to_t = tf.nn.conv2d(x_t, w_t, [1,1,1,1], 'VALID', data_format='NHWC')\r\n\tx_good = np.zeros((5,32,32, 64), dtype=np.float32)\r\n\tx_good[:,-1,-1,0] = 1e10\r\n\tx_bad = np.zeros((100,32,32, 64), dtype=np.float32)\r\n\tx_bad[:,-1,-1,0] = 1e10\r\n\to_good = sess.run(o_t, feed_dict={x_t : x_good})\r\n\to_bad = sess.run(o_t, feed_dict={x_t : x_bad})\r\n\tprint('Number nonzero (good): ', np.count_nonzero(o_good[:,0,0,:]))\r\n\tprint('Number nonzero (bad): ', np.count_nonzero(o_bad[:,0,0,:]))\r\n\tif np.count_nonzero(o_bad[:,0,0,:]) > 0:\r\n\t\tprint(o_bad[0,0,0,:])\r\n\r\nwith tf.Session() as sess:\r\n\twith tf.device('cpu:0'):\r\n\t\tprint('Testing on cpu -- should succeed')\r\n\t\ttest(sess)\r\n\twith tf.device('gpu:0'):\r\n\t\tprint('Testing on gpu -- typically fails')\r\n\t\ttest(sess)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/445724295","html_url":"https://github.com/tensorflow/tensorflow/issues/24196#issuecomment-445724295","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/24196","id":445724295,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTcyNDI5NQ==","user":{"login":"the-butterfly","id":29935671,"node_id":"MDQ6VXNlcjI5OTM1Njcx","avatar_url":"https://avatars3.githubusercontent.com/u/29935671?v=4","gravatar_id":"","url":"https://api.github.com/users/the-butterfly","html_url":"https://github.com/the-butterfly","followers_url":"https://api.github.com/users/the-butterfly/followers","following_url":"https://api.github.com/users/the-butterfly/following{/other_user}","gists_url":"https://api.github.com/users/the-butterfly/gists{/gist_id}","starred_url":"https://api.github.com/users/the-butterfly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/the-butterfly/subscriptions","organizations_url":"https://api.github.com/users/the-butterfly/orgs","repos_url":"https://api.github.com/users/the-butterfly/repos","events_url":"https://api.github.com/users/the-butterfly/events{/privacy}","received_events_url":"https://api.github.com/users/the-butterfly/received_events","type":"User","site_admin":false},"created_at":"2018-12-10T08:00:32Z","updated_at":"2018-12-10T08:00:32Z","author_association":"NONE","body":"I have encountered the same problem, with CUDA/cuDNN 9.0/7.0.4, and I update cuDNN to 7.1.4 version, then open a new terminal, then it works fine. So I think you can try cuDNN v7.1.4. Good Luck!"}}
{"actor":{"display_login":"the-butterfly"}}