title: Very good validation accuracy but bad predictions
body: <p>I'm building a keras model to classify cats and dogs. I used transfer learning with bottleneck features and fine tuning with vgg model. Now I get very good validation accuracy like 97% but when I get to predict I get very bad results regarding the classification report and confusion matrix. What could be the problem?</p>
  
  <p>Here is the code of fine tuning and the results I get</p>
  
  <pre><code>base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(150,150,3))
  print('Model loaded.')
  
  # build a classifier model to put on top of the convolutional model
  top_model = Sequential()
  top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
  top_model.add(Dense(256, activation='relu'))
  top_model.add(Dropout(0.5))
  top_model.add(Dense(2, activation='sigmoid'))
  
  # note that it is necessary to start with a fully-trained
  # classifier, including the top classifier,
  # in order to successfully do fine-tuning
  top_model.load_weights(top_model_weights_path)
  
  # add the model on top of the convolutional base
  # model.add(top_model)
  model = Model(inputs=base_model.input, outputs=top_model(base_model.output))
  
  # set the first 25 layers (up to the last conv block)
  # to non-trainable (weights will not be updated)
  for layer in model.layers[:15]:
      layer.trainable = False
  
  # compile the model with a SGD/momentum optimizer
  # and a very slow learning rate.
  model.compile(loss='binary_crossentropy',
                optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
                metrics=['accuracy'])
  
  # prepare data augmentation configuration
  train_datagen = ImageDataGenerator(
      rescale=1. / 255,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True)
  
  test_datagen = ImageDataGenerator(rescale=1. / 255)
  
  train_generator = train_datagen.flow_from_directory(
      train_data_dir,
      target_size=(img_height, img_width),
      batch_size=batch_size,
      class_mode='categorical')
  
  validation_generator = test_datagen.flow_from_directory(
      validation_data_dir,
      target_size=(img_height, img_width),
      batch_size=batch_size,
      class_mode='categorical')
  
  model.summary()
  
  # fine-tune the model
  model.fit_generator(
      train_generator,
      steps_per_epoch=nb_train_samples // batch_size,
      epochs=epochs,
      validation_data=validation_generator,
      validation_steps=nb_validation_samples // batch_size,
      verbose=2)
  scores=model.evaluate_generator(generator=validation_generator,
  steps=nb_validation_samples // batch_size)
  print("Accuracy = ", scores[1])
  
  Y_pred = model.predict_generator(validation_generator, nb_validation_samples // batch_size)
  
  y_pred = np.argmax(Y_pred, axis=1)
  
  print('Confusion Matrix')
  
  print(confusion_matrix(validation_generator.classes, y_pred))
  
  print('Classification Report')
  
  target_names = ['Cats', 'Dogs']
  
  print(classification_report(validation_generator.classes, y_pred, target_names=target_names))
  model.save("model_tuned.h5")
  </code></pre>
  
  <p>Accuracy =  0.974375</p>
  
  <p>Confusion Matrix
  [[186 214]
   [199 201]]</p>
  
  <p>Classification Report</p>
  
  <pre><code>          precision    recall  f1-score   support
  
      Cats       0.48      0.47      0.47       400
      Dogs       0.48      0.50      0.49       400
  </code></pre>
  
  <p>micro avg       0.48      0.48      0.48       800
     macro avg       0.48      0.48      0.48       800
  weighted avg       0.48      0.48      0.48       800</p>