title: Keras data generator memory error, but there's enough memory to load whole data set
body: <p>I have a Keras model, and data that I load into a Pandas dataframe. For the purposes of testing and debugging, my data set is rather modest. In fact, I can process and load the entire data set into memory and train the model without issue.</p>
  
  <p>Later I'm going to transition to training the model on a much larger data set, so I wrote the following generator:</p>
  
  <pre><code>augment_image = keras.preprocessing.image.ImageDataGenerator(
  
      rotation_range = 20,
      zoom_range = 0.1,
      width_shift_range = 0.1,
      height_shift_range = 0.1,
      shear_range = 0.1,
      horizontal_flip = True,
      vertical_flip = True,
      fill_mode = "nearest"
  
  )
  </code></pre>
  
  <pre><code>def data_generator(df, batch_size, augment = None):
  
      df = df.sample(frac = 1).reset_index(drop = True) # Shuffle
      num_rows = len(df.index)
      batch_num = 0
  
      while True:
  
          index_low  = min(batch_num * batch_size, num_rows - 1)
          index_high = min(index_low + batch_size, num_rows)
          batch_num += 1
          if batch_num * batch_size &gt; num_rows - 1:
              batch_num = 0
  
          subframe = df.iloc[index_low:index_high]
          images = load_images(subframe, path, image_type)
          if augment is not None:
              images = augment.flow(np.array(images))
          targets = subframe["target"]
  
          yield (np.array(images), np.array(targets))
  </code></pre>
  
  <p>The function "load_images" simply takes a list of filenames (i.e. subframe["image_type"]) and loads the actual images associated with those filenames. I know it (and everything not unique to the generator) works because, as I mentioned before, I can train the model when I simply load the entire data set into memory. (That is, process and load the entire data set -- actual images and all -- into a single variable, and pass that to model.fit.)</p>
  
  <p>But when I try to use the above generator to pass data to model.fit, like so ...</p>
  
  <pre><code>history = model.fit(
  
      data_generator(train_set, batch_size=32, augment=augment_image),
      verbose = 2,
      epochs = 1,
      steps_per_epoch = len(train_set.index) // 32,
      validation_data = data_generator(test_set, batch_size=32),
      validation_steps = len(test_set.index) // 32,
      callbacks = [checkpoint, early_stopping, tensorboard]
  
  )
  </code></pre>
  
  <p>... it hangs for about 15 minutes before finally exiting with what seems like an absurd MemoryError:</p>
  
  <pre><code>Traceback (most recent call last):
    File "datagen_test.py", line 193, in &lt;module&gt;
      callbacks = [checkpoint, early_stopping, tensorboard]
    File "C:\Users\Username\Anaconda3\envs\tensorflow2\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 819, in fit
      use_multiprocessing=use_multiprocessing)
    File "C:\Users\Username\Anaconda3\envs\tensorflow2\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 235, in fit
      use_multiprocessing=use_multiprocessing)
    File "C:\Users\Username\Anaconda3\envs\tensorflow2\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 593, in _process_training_inputs
      use_multiprocessing=use_multiprocessing)
    File "C:\Users\Username\Anaconda3\envs\tensorflow2\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 706, in _process_inputs
      use_multiprocessing=use_multiprocessing)
    File "C:\Users\Username\Anaconda3\envs\tensorflow2\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py", line 747, in __init__
      peek, x = self._peek_and_restore(x)
    File "C:\Users\Username\Anaconda3\envs\tensorflow2\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py", line 850, in _peek_and_restore
      peek = next(x)
    File "datagen_test.py", line 84, in data_generator
      yield (np.array(images), np.array(targets))
    File "C:\Users\Username\Anaconda3\envs\tensorflow2\lib\site-packages\keras_preprocessing\image\iterator.py", line 104, in __next__
      return self.next(*args, **kwargs)
    File "C:\Users\Username\Anaconda3\envs\tensorflow2\lib\site-packages\keras_preprocessing\image\iterator.py", line 116, in next
      return self._get_batches_of_transformed_samples(index_array)
    File "C:\Users\Username\Anaconda3\envs\tensorflow2\lib\site-packages\keras_preprocessing\image\numpy_array_iterator.py", line 148, in _get_batches_of_transformed_samples
      dtype=self.dtype)
  MemoryError: Unable to allocate 18.4 MiB for an array with shape (32, 224, 224, 3) and data type float32
  </code></pre>
  
  <p>That seems like a rather small amount of memory to be unable to allocate. (And again, I have enough memory to simply load the entire data set and train the model that way.) What am I doing wrong?</p>