title: Numpy to TFrecords: Is there a more simple way to handle batch inputs from tfrecords?
body: <p>My question is about how to get batch inputs from multiple (or sharded) tfrecords. I've read the example <a href="https://github.com/tensorflow/models/blob/master/inception/inception/image_processing.py#L410" rel="noreferrer">https://github.com/tensorflow/models/blob/master/inception/inception/image_processing.py#L410</a>. The basic pipeline is, take the training set as as example, (1) first generate a series of tfrecords (e.g., <code>train-000-of-005</code>, <code>train-001-of-005</code>, ...), (2) from these filenames, generate a list and fed them into the <code>tf.train.string_input_producer</code> to get a queue, (3) simultaneously generate a <code>tf.RandomShuffleQueue</code> to do other stuff, (4) using <code>tf.train.batch_join</code> to generate batch inputs. </p>
  
  <p>I think this is complex, and I'm not sure the logic of this procedure. In my case, I have a list of <code>.npy</code> files, and I want to generate sharded tfrecords(multiple seperated tfrecords, not just one single large file). Each of these <code>.npy</code> files contains different number of positive and negative samples (2 classes). A basic method is to generate one single large tfrecord file. But the file is too large (<code>~20Gb</code>). So I resort to sharded tfrecords. Are there any simpler way to do this? Thanks.</p>