title: GridSearch in Keras + TensorFlow resulting in Resource exhausted
body: <p>I know that this error is recurrent and I understand what can cause it.
  For example, running this model with 163 images of 150x150 gives me the error (however it's not clear to me why setting batch_size Keras still seems to try to allocate all images at a time in the GPU):</p>
  
  <pre><code>model = Sequential()
      model.add(Conv2D(64, kernel_size=(6, 6), activation='relu', input_shape=input_shape, padding='same', name='b1_conv'))
      model.add(MaxPooling2D(pool_size=(2, 2), name='b1_poll'))
      model.add(Conv2D(128, kernel_size=(6, 6), activation='relu', padding='same', name='b2_conv'))
      model.add(MaxPooling2D(pool_size=(2, 2), name='b2_pool'))
      model.add(Conv2D(256, kernel_size=(6, 6), activation='relu', padding='same', name='b3_conv'))
      model.add(MaxPooling2D(pool_size=(2, 2), name='b3_pool'))
      model.add(Flatten())
      model.add(Dense(500, activation='relu', name='fc1'))
      model.add(Dropout(0.5))
      model.add(Dense(500, activation='relu', name='fc2'))
      model.add(Dropout(0.5))
      model.add(Dense(n_targets, activation='softmax', name='prediction'))
      model.compile(optimizer=optim, loss='categorical_crossentropy',  metrics=['accuracy'])
  </code></pre>
  
  <p>Given that, I reduced the images size to 30x30 (which resulted in an accuracy drop, as expected). However, running grid search in this model Resource exhausted. </p>
  
  <pre><code>model = KerasClassifier(build_fn=create_model, verbose=0)
  
  # grid initial weight, batch size and optimizer
  sgd  = optimizers.SGD(lr=0.0005)
  rms  = optimizers.RMSprop(lr=0.0005)
  adag = optimizers.Adagrad(lr=0.0005)
  adad = optimizers.Adadelta(lr=0.0005)
  adam = optimizers.Adam(lr=0.0005)
  adamm = optimizers.Adamax(lr=0.0005)
  nadam = optimizers.Nadam(lr=0.0005)
  
  optimizers = [sgd, rms, adag, adad, adam, adamm, nadam]
  init = ['glorot_uniform', 'normal', 'uniform', 'he_normal']
  batches = [32, 64, 128]
  param_grid = dict(optim=optimizers, batch_size=batches, init=init)
  grid = GridSearchCV(estimator=model, param_grid=param_grid)
  grid_result = grid.fit(X_train, y_train)
  
  # summarize results
  print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
  </code></pre>
  
  <p>I wonder if it's possible to "clean" things before each combination used by the grid search (not sure if I made myself clear, this is all new to me).</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>Using the <code>fit_generator</code> also gives me the same error:</p>
  
  <pre><code>def generator(features, labels, batch_size):
      # Create empty arrays to contain batch of features and labels#
      batch_features = np.zeros((batch_size, size, size, 1))
      batch_labels = np.zeros((batch_size, n_targets))
      while True:
          for i in range(batch_size):
              # choose random index in features
              index = np.random.choice(len(features),1)
              batch_features[i] = features[index]
              batch_labels[i] = labels[index]
          yield batch_features, batch_labels
  
  sgd  = optimizers.SGD(lr=0.0005)
  rms  = optimizers.RMSprop(lr=0.0005)
  adag = optimizers.Adagrad(lr=0.0005)
  adad = optimizers.Adadelta(lr=0.0005)
  adam = optimizers.Adam(lr=0.0005)
  adamm = optimizers.Adamax(lr=0.0005)
  nadam = optimizers.Nadam(lr=0.0005)
  
  optim = [rms, adag, adad, adam, adamm, nadam]
  init = ['normal', 'uniform', 'he_normal']
  
  combinations = [(a, b) for a in optim for b in init]
  for combination in combinations:
      init  = combination[1]
      optim = combination[0]
      model = create_model(init=init, optim=optim)
      model.fit_generator(generator(X_train, y_train, batch_size=32),
                          steps_per_epoch=X_train.shape[0] // 32,
                          epochs=100, verbose=0, validation_data=(X_test, y_test))
      scores = model.model.evaluate(X_test, y_test, verbose=0)
      print("%s: %.2f%% Model %s %s" % (model.model.metrics_names[1], scores[1]*100, optim, init))
  </code></pre>