title: Is Keras fit_generator the best thing to use when handling data that does not fit in RAM?
body: <p>I am working to build a classifier that can classify <a href="https://en.wikipedia.org/wiki/Knot_(mathematics)" rel="nofollow noreferrer">Knots</a>. Currently I have a dataset that contains 100,000 images of "unknots", 100,000 "plus trefoil", and 100,000 "minus trefoil".</p>
  
  <p>I have been trying to get a classifier working on this large data set for the last 4ish days. The list of problems I had so far were:</p>
  
  <p>1) Data set does not fit in CPU main memory: fixed this by making a number of <code>EArrays</code> with PyTables and Hdf5 and appending them on disk. So now I have one 1.2Gb file which is the dataset.</p>
  
  <p>2) Even a very simple neural network in Keras was hitting 100% GPU (nvidia k80) memory usage after the <strong>model compiled</strong>... hadn't even fit the model yet. I read this was due to Keras backend automatically allocating near 100% of available resources upon compile: I fixed this as well.</p>
  
  <p>3) Once errors 1 and 2 were fixed, I was still getting strange accuracy from Keras <code>fit_generator()</code>.  </p>
  
  <p>Questions: </p>
  
  <p>1) Is the method I have described of merging small numpy arrays into one large <code>EArray</code> using PyTables a good way to make a very large (300,000 images 128x128, total size = 1.2Gb) dataset?</p>
  
  <p>2) Should fit generator be used over <code>train_on_batch</code> in Keras? Will they return significantly different final loss/accuracy scores?</p>
  
  <p>3) What is wrong with my generator method if I want to train the neural network on batches of 50 images from the hdf5 file, and after each training epoch, delete the images the network just trained on out of main memory?</p>
  
  <pre><code>import tables
  hdf5_path = "300K_Knot_data.hdf5"
  extendable_hdf5_file = tables.open_file(hdf5_path, mode='r')
  
  def imageLoader(files, batch_size):
  
      L = len(files.root.train_data)
  
      #this line is just to make the generator infinite, keras needs that    
      while True:
  
          batch_start = 0
          batch_end = batch_size
  
          while batch_start &lt; L:
              limit = min(batch_end, L)
              X = files.root.train_data[batch_start:limit]
              X = np.reshape(X, (X.shape[0],128,128,3))
              X = X/255
              Y = files.root.train_label[batch_start:limit]
  
              yield (X,Y) #a tuple with two numpy arrays with batch_size samples     
  
              batch_start += batch_size   
              batch_end += batch_size
  
  img_rows, img_cols = 128,128
  ###################################
  # TensorFlow wizardry
  config = tf.ConfigProto()
  
  # Don't pre-allocate memory; allocate as-needed
  config.gpu_options.allow_growth = True
  
  # Only allow a total of half the GPU memory to be allocated
  config.gpu_options.per_process_gpu_memory_fraction = 0.5
  
  # Create a session with the above options specified.
  K.tensorflow_backend.set_session(tf.Session(config=config))
  ###################################
  
  model = Sequential()
  model.add(Conv2D(64, (3,3), input_shape=(img_rows,img_cols,3)))
  model.add(BatchNormalization())
  model.add(Activation('relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  
  model.add(Conv2D(96, (3,3), padding='same'))
  model.add(Activation('relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  
  model.add(Conv2D(128, (3,3)))
  model.add(Activation('relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  
  
  model.add(Flatten())
  model.add(Dense(128))
  model.add(Activation('relu'))
  model.add(Dropout(0.5))
  model.add(Dense(3))
  model.add(Activation('softmax', name='preds'))
  
  
  #lr was originally 0.01
  
  model.compile(loss='categorical_crossentropy',
  optimizer=keras.optimizers.Adagrad(lr=0.01, epsilon=None, decay=0.0),metrics= 
  ['accuracy'])
  
  # fits the model on batches with real-time data augmentation:
  
  model.fit_generator(imageLoader(extendable_hdf5_file, 50),
                  steps_per_epoch=240000 / 50, epochs=50)
  </code></pre>
  
  <p>Here is the output of fitting the model:</p>
  
  <p><a href="https://i.stack.imgur.com/ZtviP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZtviP.png" alt="Output of fitting the model."></a></p>
  
  <p>Sorry if this is not the correct place for the post.
  My research adviser is out of town, and I have spent a considerable amount of time trying to fix this issue and I just need some input, because I cannot find an adequate solution on the internet, or cannot seem to wrap my head around what I do find well enough to implement it in a nice manner.  I am not a new programmer, but am relatively inexperienced  in python.</p>