{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/953","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/953/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/953/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/953/events","html_url":"https://github.com/tensorflow/tensorflow/issues/953","id":130514566,"number":953,"title":"Failed to enqueue async memcpy from device to host, with latest code (relapse?)","user":{"login":"caisq","id":16824702,"avatar_url":"https://avatars.githubusercontent.com/u/16824702?v=3","gravatar_id":"","url":"https://api.github.com/users/caisq","html_url":"https://github.com/caisq","followers_url":"https://api.github.com/users/caisq/followers","following_url":"https://api.github.com/users/caisq/following{/other_user}","gists_url":"https://api.github.com/users/caisq/gists{/gist_id}","starred_url":"https://api.github.com/users/caisq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caisq/subscriptions","organizations_url":"https://api.github.com/users/caisq/orgs","repos_url":"https://api.github.com/users/caisq/repos","events_url":"https://api.github.com/users/caisq/events{/privacy}","received_events_url":"https://api.github.com/users/caisq/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"zffchen78","id":7943790,"avatar_url":"https://avatars.githubusercontent.com/u/7943790?v=3","gravatar_id":"","url":"https://api.github.com/users/zffchen78","html_url":"https://github.com/zffchen78","followers_url":"https://api.github.com/users/zffchen78/followers","following_url":"https://api.github.com/users/zffchen78/following{/other_user}","gists_url":"https://api.github.com/users/zffchen78/gists{/gist_id}","starred_url":"https://api.github.com/users/zffchen78/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zffchen78/subscriptions","organizations_url":"https://api.github.com/users/zffchen78/orgs","repos_url":"https://api.github.com/users/zffchen78/repos","events_url":"https://api.github.com/users/zffchen78/events{/privacy}","received_events_url":"https://api.github.com/users/zffchen78/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2016-02-01T23:07:16Z","updated_at":"2016-02-01T23:54:37Z","closed_at":null,"body":"On a Linux machine with a GPU, with the GPU configuration, I ran the following command: \r\n`bazel test -c opt --config=cuda //tensorflow/python:framework_function_test`\r\n\r\nThen I got the following error, which seems to be similar to the closed issues. I'm using the latest code of the master branch\r\nhttps://github.com/tensorflow/tensorflow/issues/719\r\nhttps://github.com/tensorflow/tensorflow/issues/713\r\n\r\nI tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so.7.0 locally\r\nI tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so.6.5 locally\r\nI tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so.7.0 locally\r\nI tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:99] Couldn't open CUDA library libcurand.so.7.0. LD_LIBRARY_PATH: \r\nI tensorflow/stream_executor/cuda/cuda_rng.cc:333] Unable to load cuRAND DSO.\r\nI tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:909] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: \r\nname: GeForce GTX 980\r\nmajor: 5 minor: 2 memoryClockRate (GHz) 1.2785\r\npciBusID 0000:01:00.0\r\nTotal memory: 4.00GiB\r\nFree memory: 3.91GiB\r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 \r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y \r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:680] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0)\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 1.0KiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 2.0KiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 4.0KiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 8.0KiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 16.0KiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 32.0KiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 64.0KiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 128.0KiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 256.0Ki\r\nB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 512.0KiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 1.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 2.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 4.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 8.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 16.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 32.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 64.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 128.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 256.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 512.00MiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 1.00GiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 2.00GiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 4.00GiB\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:73] Allocating 3.62GiB bytes.\r\nI tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:83] GPU 0 memory begins at 0x704a80000 extends to 0x7ec261000\r\n...I tensorflow/core/common_runtime/gpu/gpu_device.cc:680] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0)\r\n..I tensorflow/core/common_runtime/gpu/gpu_device.cc:680] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0)\r\n**E tensorflow/stream_executor/cuda/cuda_driver.cc:1197] failed to enqueue async memcpy from device to host: CUDA_ERROR_INVALID_VALUE; host dst: 0x7f16e40008c0; GPU src: 0x265d480; size: 4=0x4**\r\nF tensorflow/core/common_runtime/gpu/gpu_util.cc:232] GPU->CPU Memcpy failed\r\nexternal/bazel_tools/tools/test/test-setup.sh: line 51:   714 Aborted                 (core dumped) \"$@\""},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/178264106","html_url":"https://github.com/tensorflow/tensorflow/issues/953#issuecomment-178264106","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/953","id":178264106,"user":{"login":"prb12","id":11547801,"avatar_url":"https://avatars.githubusercontent.com/u/11547801?v=3","gravatar_id":"","url":"https://api.github.com/users/prb12","html_url":"https://github.com/prb12","followers_url":"https://api.github.com/users/prb12/followers","following_url":"https://api.github.com/users/prb12/following{/other_user}","gists_url":"https://api.github.com/users/prb12/gists{/gist_id}","starred_url":"https://api.github.com/users/prb12/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prb12/subscriptions","organizations_url":"https://api.github.com/users/prb12/orgs","repos_url":"https://api.github.com/users/prb12/repos","events_url":"https://api.github.com/users/prb12/events{/privacy}","received_events_url":"https://api.github.com/users/prb12/received_events","type":"User","site_admin":false},"created_at":"2016-02-01T23:54:37Z","updated_at":"2016-02-01T23:54:37Z","body":"This does look like a problem, and there have been changes to this code recently.  "}}