{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4968","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4968/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4968/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4968/events","html_url":"https://github.com/tensorflow/tensorflow/issues/4968","id":183073062,"number":4968,"title":"TensorFlow produces arbitrary results without error when using large amounts of GPU memory","user":{"login":"jonasrauber","id":5837385,"avatar_url":"https://avatars.githubusercontent.com/u/5837385?v=3","gravatar_id":"","url":"https://api.github.com/users/jonasrauber","html_url":"https://github.com/jonasrauber","followers_url":"https://api.github.com/users/jonasrauber/followers","following_url":"https://api.github.com/users/jonasrauber/following{/other_user}","gists_url":"https://api.github.com/users/jonasrauber/gists{/gist_id}","starred_url":"https://api.github.com/users/jonasrauber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonasrauber/subscriptions","organizations_url":"https://api.github.com/users/jonasrauber/orgs","repos_url":"https://api.github.com/users/jonasrauber/repos","events_url":"https://api.github.com/users/jonasrauber/events{/privacy}","received_events_url":"https://api.github.com/users/jonasrauber/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/awaiting%20response","name":"awaiting response","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-10-14T15:01:10Z","updated_at":"2016-10-24T20:54:52Z","closed_at":null,"body":"### Environment info\r\nOperating System: Docker container based on `nvidia/cuda:7.5-cudnn5-devel` running on CentOS\r\n\r\nIf installed from binary pip package, provide:\r\n\r\n1. A link to the pip package you installed: `pip3 install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.11.0rc0-cp34-cp34m-linux_x86_64.whl`\r\n2. The output from `python -c \"import tensorflow; print(tensorflow.__version__)\"`.\r\n\r\n```\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally\r\n0.11.0rc0\r\n```\r\n\r\n### If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)\r\n\r\n```python\r\nimport numpy as np\r\nimport tensorflow as tf\r\n\r\nmb = 100\r\nsz = int(mb * 1e6 / 4)\r\ntotal_gb = 11\r\nn = int(total_gb / mb * 1000)\r\nx = np.array(np.random.randn(sz), dtype=np.float32)\r\n\r\nwith tf.Graph().as_default():\r\n    place = tf.placeholder(tf.float32, shape=[sz])\r\n    with tf.device('/gpu:0'):\r\n        x_gpu = [tf.Variable(place, dtype=tf.float32) for i in range(n)]\r\n    init = tf.initialize_all_variables()\r\n    with tf.Session() as sess:\r\n        sess.run(init, feed_dict={place: x})\r\n        # I know it's not optimal, but to make it more comparable to the Theano code below\r\n        x_out = [sess.run(v) for v in x_gpu]\r\n\r\ndelta = [np.sum(np.abs(xi - x)) for xi in x_out]\r\nprint(delta)\r\n```\r\n\r\nExecuted using `python3 memcheck_tensorflow.py`\r\n\r\n#### Expected output:\r\n```\r\nsome tensorflow output ...\r\n\r\n[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\r\n```\r\n\r\n#### Actual output:\r\n\r\n```\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally\r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties: \r\nname: Tesla K40m\r\nmajor: 3 minor: 5 memoryClockRate (GHz) 0.745\r\npciBusID 0000:81:00.0\r\nTotal memory: 11.25GiB\r\nFree memory: 11.15GiB\r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 \r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y \r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40m, pci bus id: 0000:81:00.0)\r\n[28209880.0, 28208760.0, 28208760.0, 28209880.0, 28206538.0, 28208760.0, 28212806.0, 28209880.0, 28209880.0, 28208760.0, 28211926.0, 28209880.0, 28208760.0, 28209880.0, 28209880.0, 28206538.0, 28206538.0, 28209880.0, 28208760.0, 28212806.0, 28208760.0, 28209880.0, 28209880.0, 28209880.0, 28207726.0, 28208760.0, 28206538.0, 28209880.0, 28206538.0, 28210806.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209880.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28209472.0, 28206602.0, 28209472.0, 28209472.0, 28208760.0, 28208760.0, 28209880.0, 28208760.0, 28207726.0, 28209880.0, 28208760.0, 28208760.0, 28209880.0, 28209880.0, 28206602.0, 28209880.0, 28209880.0, 28208760.0, 28209880.0, 28209880.0, 28209880.0, 28208760.0, 28206538.0, 28209880.0, 28209880.0, 28208760.0, 28208760.0, 28206602.0, 28207726.0, 28209880.0, 28207726.0, 28209880.0, 28209880.0, 28209880.0, 28211926.0, 28209880.0, 28208760.0, 28209880.0, 28208760.0, 28208760.0, 28209880.0, 28209880.0, 28209880.0, 28209880.0, 28209880.0, 28209880.0, 28206538.0, 28209880.0, 28209880.0, 28209880.0, 28209880.0, 28210806.0, 28209880.0, 28208760.0, 28208760.0, 28209880.0, 28206538.0, 28208760.0, 28209880.0, 28206602.0, 28208760.0, 28208760.0]\r\n```\r\n\r\n### What other attempted solutions have you tried?\r\n\r\nI've verified that this problem exists on at least one other GPU (I've seen it on Nvidia GPUs K40m and K80).\r\n\r\nI've run similar code in Theano without any problems (same GPU, same Docker container):\r\n\r\n```python\r\nimport numpy as np\r\nimport theano\r\n\r\nmb = 100\r\nsz = int(mb * 1e6 / 4)\r\ntotal_gb = 11\r\nn = int(total_gb / mb * 1000)\r\nx = np.array(np.random.randn(sz), dtype=np.float32)\r\n\r\nx_gpu = [theano.shared(x) for i in range(n)]\r\nx_out = [v.get_value() for v in x_gpu]\r\n\r\ndelta = [np.sum(np.abs(xi - x)) for xi in x_out]\r\nprint(delta)\r\n```\r\n\r\nExecuted using `THEANO_FLAGS=device=gpu,floatX=float32 python3 memcheck_theano.py`\r\n\r\n#### Output\r\n\r\n```\r\nUsing gpu device 0: Tesla K40m (CNMeM is disabled, cuDNN 5103)\r\n[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/255863293","html_url":"https://github.com/tensorflow/tensorflow/issues/4968#issuecomment-255863293","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4968","id":255863293,"user":{"login":"yifeif","id":1192265,"avatar_url":"https://avatars.githubusercontent.com/u/1192265?v=3","gravatar_id":"","url":"https://api.github.com/users/yifeif","html_url":"https://github.com/yifeif","followers_url":"https://api.github.com/users/yifeif/followers","following_url":"https://api.github.com/users/yifeif/following{/other_user}","gists_url":"https://api.github.com/users/yifeif/gists{/gist_id}","starred_url":"https://api.github.com/users/yifeif/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yifeif/subscriptions","organizations_url":"https://api.github.com/users/yifeif/orgs","repos_url":"https://api.github.com/users/yifeif/repos","events_url":"https://api.github.com/users/yifeif/events{/privacy}","received_events_url":"https://api.github.com/users/yifeif/received_events","type":"User","site_admin":false},"created_at":"2016-10-24T20:54:52Z","updated_at":"2016-10-24T20:54:52Z","body":"Not sure what is the root cause of this issue, but wondering if upgrading to CUDA 8.0 and using the latest release 0.11.0rc1(w/ CUDA support) would help?"}}
{"actor":{"display_login":"yifeif"}}