title: GPU ->CPU Memcpy failed in tensorflow word2vec in gpu occured
body: <p><strong>I I am studying word2vec of tensorflow.
  We bought two 1080i for parallel processing of gpu.
  Mounting was successful and p2p was successful.
  However, I tried to assign it to gpu using the command with tf.device ('/ gpu: 0')
  The following error occurs :</strong></p>
  
  <p>I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties: </p>
  
  <p>name: GeForce GTX 1080 Ti</p>
  
  <p>major: 6 minor: 1 memoryClockRate (GHz) 1.645</p>
  
  <p>pciBusID 0000:66:00.0</p>
  
  <p>Total memory: 10.91GiB</p>
  
  <p>Free memory: 10.21GiB</p>
  
  <p>tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1 </p>
  
  <p>tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y Y </p>
  
  <p>tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1:   Y Y </p>
  
  <p>I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0)</p>
  
  <p>I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:66:00.0)</p>
  
  <p>I word2vec_kernels.cc:246] Data file: data/spouse_freebase/input2.nt contains 34966827 bytes, 2620786 words, 11769 unique words, 11769 unique frequent words.</p>
  
  <p>E tensorflow/stream_executor/cuda/cuda_driver.cc:1276] failed to enqueue async memcpy from device to host: CUDA_ERROR_INVALID_VALUE; host dst: 0x104d5000000; GPU src: 0x7f12c800cbc0; size: 8=0x8</p>
  
  <p>I tensorflow/stream_executor/stream.cc:1338] stream 0x39c2160 did not wait for stream: 0x39bf9a0</p>
  
  <p>I tensorflow/stream_executor/stream.cc:3775] stream 0x39c2160 did not memcpy device-to-host; source: 0x3bd0d00</p>
  
  <p><strong>F tensorflow/core/common_runtime/gpu/gpu_util.cc:296] GPU->CPU Memcpy failed</strong></p>
  
  <p>I think this error is the out of memory of the gpu.
  I wait for your help.
  thank you.</p>