title: Tensorflow Dataset performances?
body: <p>I am implementing a model inspired by the <a href="https://github.com/tensorflow/nmt" rel="nofollow noreferrer">NMT</a> model. I am using a training set stored as TFRecords files, using a TFRecordDataset to fetch it and feed the model. Following Google's <a href="https://www.tensorflow.org/performance/performance_guide#input_pipeline_optimization" rel="nofollow noreferrer">recommendations</a> about input pipeline performances improvement, I have:</p>
  
  <ul>
  <li>preprocessed as much as possible beforehand on CPU</li>
  <li>stacked several training examples up to about 100 MB TFrecords files (having less files containing more examples)</li>
  <li>used <code>num_parallel_calls</code> and <code>prefetch</code> on the Dataset <code>map</code> operations.</li>
  </ul>
  
  <p>However, GPU remains at maximum 40%, and it is barely as slow as when run on CPU. I am thus wondering about the <code>prefetch</code> operation. </p>
  
  <ul>
  <li><p>If I understand correctly, it will create a special thread that buffers <em>N</em> examples. But what does it mean ? What happens to the other examples not buffered ?</p></li>
  <li><p>is there an optimal relation between the prefetch buffer size, the number of examples in the complete Dataset and the batch size ? In the NMT code, prefetch buffer size is set at <code>1000*batch_size</code>, but why ? If e.g. I am using 10000 examples, a batch size of 100, what should be the prefetch buffer size ?</p></li>
  </ul>
  
  <p>Any other advice regarding Dataset speedup would be appreciated.</p>