title: Input pipeline w/ keras.utils.Sequence object or tf.data.Dataset?
body: <p>I am currently using a <code>tf.keras.utils.Sequence</code> object to generate image batches for a CNN. I am using Tensorflow 2.2 and the <code>Model.fit</code> method for the model.  When I fit the model, the following warning is thrown in each epoch when I set <code>use_multiprocessing=True</code> in <code>tf.keras.model.fit(...)</code>:</p>
  
  <pre><code>WARNING:tensorflow:multiprocessing can interact badly with TensorFlow,
   causing nondeterministic deadlocks. For high performance data pipelines tf.data is recommended
  </code></pre>
  
  <p>The model is optimizing just fine, as expected from the docs and the fact that I am using a <code>Sequence</code>-based generator. But if <code>use_multiprocessing</code> is going to be a deprecated functionality in lieu of <code>tf.data</code> objects, I would like to be using the most up-to-date input pipeline. I currently use the following <code>tf.keras.utils.Sequence</code>-based generator inspired by this article on good practices for partitioning large datasets:
  <a href="https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly" rel="noreferrer">https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly</a></p>
  
  <pre><code>class DataGenerator(keras.utils.Sequence):
  'Generates data for Keras'
  def __init__(self, list_IDs, labels, data_dir, batch_size=32, dim=(128,128), n_channels=1,
               n_classes=2, shuffle=True, **augmentation_kwargs):
      'Initialization'
      self.dim = dim
      self.batch_size = batch_size
      self.labels = labels
      self.list_IDs = list_IDs
      self.data_dir = data_dir
      self.n_channels = n_channels
      self.n_classes = n_classes
      self.shuffle = shuffle
      self.on_epoch_end()
      self.augmentor = keras.preprocessing.image.ImageDataGenerator(**augmentation_kwargs)
  
  def __len__(self):
      'Denotes the number of batches per epoch'
      return int(np.floor(len(self.list_IDs) / self.batch_size))
  
  def __getitem__(self, index):
      'Generate one batch of data'
      # Generate indexes of the batch
      indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]
  
      # Find list of IDs
      list_IDs_temp = [self.list_IDs[k] for k in indexes]
  
      # Generate data
      X, y = self.__data_generation(list_IDs_temp)
  
      return X, y
  
  def on_epoch_end(self):
      'Updates indexes after each epoch'
      self.indexes = np.arange(len(self.list_IDs))
      if self.shuffle == True:
          np.random.shuffle(self.indexes)
  
  def __data_generation(self, list_IDs_temp):
      'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
      # Initialization
      X = np.empty((self.batch_size, *self.dim))
      y = np.empty((self.batch_size), dtype=int)
  
      # Generate data
      for i, ID in enumerate(list_IDs_temp):
  
          # Store sample
          X[i,] = np.load(self.data_dir +'/{}_stars.npy'.format(ID))
  
          # Store class
          y[i] = self.labels[ID]
  
      # Reshape and apply augmentation to sample
      X,y = self.augmentor.flow(X.reshape(self.batch_size,*self.dim,1),y=y,
                                shuffle=False,batch_size=self.batch_size)[0]
  
      return X, y
  </code></pre>
  
  <p>All data from all classes is in the <code>data_dir</code> directory and are stored as individual <code>.npy</code> files. The IDs come from a list of strings. The class labels are taken from a dictionary whose keys are the IDs -- as in the article. </p>
  
  <p>I really like the intuition of the <code>Sequence</code> generator set-up. I can also easily generator random batches to check that it is behaving as I would expect. But how can I reproduce this set-up with <code>tf.data</code>? How do I reproduce the multiprocessing batch generation of a <code>Sequence</code> generator with the <code>interleave</code> and <code>prefetch</code> methods of <code>tf.data.Dataset</code>? And/or can I simply ingest this <code>Sequence</code>-based generator with the <code>tf.data.Dataset.from_generator()</code> method?</p>
  
  <p>Many thanks in advance.</p>