title: How to write an efficient custom Keras data generator
body: <p>I would like to train a convolutional recurrent neural network for video frame prediction. The individual frames are quite big so it is challenging to fit the entire training data in memory at once. As such, I followed some tutorials online to create a custom data generator. When testing it, it seems to work but it is slower by a factor of at least 100 than using the pre-loaded data directly. Since I can only fit about a batch size of 8 on the GPU I understand that the data needs to be generated really fast, however, this does not seem to be the case.</p>
  
  <p>I train my model on a single P100 and have 32 GB of memory available to be used by up to 16 cores. </p>
  
  <pre><code>class DataGenerator(tf.keras.utils.Sequence):
  
  def __init__(self, images, input_images=5, predict_images=5, batch_size=16, image_size=(200, 200),
               channels=1):
  
      self.images = images
      self.input_images = input_images
      self.predict_images = predict_images
      self.batch_size = batch_size
      self.image_size = image_size
      self.channels = channels
      self.nr_images = int(len(self.images)-input_images-predict_images)
  
  def __len__(self):
  
      return int(np.floor(self.nr_images) / self.batch_size)
  
  def __getitem__(self, item):
  
      # Randomly select the beginning image of each batch
      batch_indices = random.sample(range(0, self.nr_images), self.batch_size)
  
      # Allocate the output images
      x = np.empty((self.batch_size, self.input_images,
                    *self.image_size, self.channels), dtype='uint8')
      y = np.empty((self.batch_size, self.predict_images,
                    *self.image_size, self.channels), dtype='uint8')
  
      # Get the list of input an prediction images
      for i in range(self.batch_size):
          list_images_input = range(batch_indices[i], batch_indices[i]+self.input_images)
          list_images_predict = range(batch_indices[i]+self.input_images,
                                           batch_indices[i]+self.input_images+self.predict_images)
  
          for j, ID in enumerate(list_images_input):
              x[i, ] = np.load(np.reshape(self.images[ID], (*self.imagesize, self.channels))
  
          # Read in the prediction images
          for j, ID in enumerate(list_images_predict):
              y[i, ] = np.load(np.reshape(self.images[ID], (*self.imagesize, self.channels))
  
      return x, y
  
  
  # Training the model using fit_generator
  
  params = {'batch_size': 8,
        'input_images': 5,
        'predict_images': 5,
        'image_size': (100, 100),
        'channels': 1
        }
  
  data_path = "input_frames/"
  input_images = sorted(glob.glob(data_path + "*.png"))
  training_generator = DataGenerator(input_images, **params)
  
  model.fit_generator(generator=training_generator, epochs=10, workers=6)
  </code></pre>
  
  <p>I would have expected that Keras will prepare the next data batch while the current batch is being processed on the GPU but it does not seem to catch up. In other words, preparing the data before sending it to the GPU seems to be the bottleneck. </p>
  
  <p>Any idea on how to improve the performance of a data generator like this? Is there something missing that guarantees that the data is being prepared in a timely manner?</p>
  
  <p>Thanks a lot!</p>