title: How to change python generator into Keras Sequence object?
body: <p>I am working on a regression problem. My CNN trains on data which has shape <strong>32x513x30</strong> - <strong>32 instances of 513x30</strong> per batch, and then <strong>4810</strong> batches.</p>
  
  <p>I have these batches saved in a directory, each named "batch#number.npy".</p>
  
  <p>Whilst using a Python generator, I keep getting a warning from TensorFlow: </p>
  
  <blockquote>
    <p>WARNING:tensorflow:Using a generator with <code>use_multiprocessing=True</code>
    and multiple workers may duplicate your data. Please consider using
    the <code>keras.utils.Sequence</code> class.</p>
  </blockquote>
  
  <p>I figured out how to load them using a Python generator. However, whilst using multiprocessing, it is advised to use Keras' Sequence class: <a href="https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence" rel="nofollow noreferrer">https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence</a></p>
  
  <p>Unfortunately, that's where things get a bit too complicated for me.
  Here's the code I've been using: </p>
  
  <pre><code>def batch_gen(num):
  
      os.chdir('mydirectory/train')
  
      for n in num:
          placeholder = np.load('batch#' + str(n) + '.npy')
          X = placeholder[:,:513,:]
          Y1= placeholder[:,513:,:]
  
          Y = X * Y1
  
          X = X / normalization # normalize X
          X = scale_mag*X.astype(np.float32)
  
          Y = Y / normalization 
          Y = scale_mag*Y.astype(np.float32)
  
  
          X = np.reshape(X,(32,513,30,1))
          Y = np.reshape(Y,(32,513,30,1))
          yield (X, Y)
  
  my_gen = batch_gen(C)   # C is an array with indexes 1 to 4810 (looped by number of training epochs)
  </code></pre>
  
  <p>Is the way I am using the generator causing my data to duplicate during training? If so, how could I translate it into a Sequence class? </p>
  
  <p>Thank you.</p>