title: Why TensorFlow spent so many time on HtoD memcpy with Titan X?
body: <p>I'm experiencing running AlexNet model from <a href="https://github.com/tensorflow/models/tree/master/research/slim/nets" rel="nofollow noreferrer">here</a> in TensorFlow for evaluating time spent on GPU by library, with the following parameters and hardware:</p>
  
  <ul>
  <li>1024 images on train dataset</li>
  <li>10 epochs with mini-batch sizes of 128</li>
  <li>using GPU GTX Titan X</li>
  </ul>
  
  <p>I stated that the real execution time on GPU is just a fraction of the all execution time of training (the graph belows compares TensorFlow and AlexNet vs Caffe and its AlexNet implementation)</p>
  
  <p><a href="https://i.stack.imgur.com/gGEVX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gGEVX.png" alt="enter image description here"></a>
  (information captured with nvidia-smi. 'Porcentagem' means percentage and 'Tempo (s)' means time (seconds))</p>
  
  <p>The GPU utilization rate oscilates frenetically between 0 and 100% in training. Why that? Caffe doesn't oscilates to much beyond 40%</p>
  
  <p>Also, Tensorflow spent many time doing memory copy from Host to Device, while Caffe doesn't. But why?
  <a href="https://i.stack.imgur.com/llo04.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/llo04.png" alt="enter image description here"></a>
  (tensorflow)</p>
  
  <p><a href="https://i.stack.imgur.com/js7AW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/js7AW.png" alt="enter image description here"></a>
  (caffe)</p>