title: Tensorflow Keras - High accuracy during training, low accuracy during prediction
body: <p>I have a very basic multiclass CNN model for classifying vehicles into 4 classes <code>[pickup, sedan, suv, van]</code> that I have written using Tensorflow 2.0 tf.keras:</p>
  <pre><code>he_initialiser = tf.keras.initializers.VarianceScaling()
  model = tf.keras.Sequential()
  model.add(tf.keras.layers.Conv2D(32, kernel_size=(3,3), input_shape=(3,128,128), activation='relu', padding='same', data_format='channels_first', kernel_initializer=he_initialiser))
  model.add(tf.keras.layers.Conv2D(32, kernel_size=(3,3), activation='relu', padding='same', data_format='channels_first', kernel_initializer=he_initialiser))
  model.add(tf.keras.layers.MaxPooling2D((2, 2), data_format=cfg_data_fmt))
  model.add(tf.keras.layers.Conv2D(64, kernel_size=(3,3), activation='relu', padding='same', data_format='channels_first', kernel_initializer=he_initialiser))
  model.add(tf.keras.layers.Conv2D(64, kernel_size=(3,3), activation='relu', padding='same', data_format='channels_first', kernel_initializer=he_initialiser))
  model.add(tf.keras.layers.MaxPooling2D((2, 2), data_format=cfg_data_fmt))
  model.add(tf.keras.layers.Conv2D(128, kernel_size=(3,3), activation='relu', padding='same', data_format='channels_first', kernel_initializer=he_initialiser))
  model.add(tf.keras.layers.Conv2D(128, kernel_size=(3,3), activation='relu', padding='same', data_format='channels_first', kernel_initializer=he_initialiser))
  model.add(tf.keras.layers.MaxPooling2D((2, 2), data_format='channels_first'))
  model.add(tf.keras.layers.Flatten(data_format='channels_first'))
  model.add(tf.keras.layers.Dense(128, activation='relu', kernel_initializer=he_initialiser))
  model.add(tf.keras.layers.Dense(128, activation='relu', kernel_initializer=he_initialiser))
  model.add(tf.keras.layers.Dense(4, activation='softmax', kernel_initializer=he_initialiser))
  </code></pre>
  <p>I use the following configuration for training:</p>
  <ul>
  <li>Image size: 3x128x128 (planar data)</li>
  <li>Number of epochs: 45</li>
  <li>Batch size: 32</li>
  <li>Loss function: <code>tf.keras.losses.CategoricalCrossentropy(from_logits=True)</code></li>
  <li>Optimizer: <code>optimizer=tf.optimizers.Adam</code></li>
  <li>training data size: 67.5% of all data</li>
  <li>validation data size: 12.5% of all data</li>
  <li>test data size: 20% of all data</li>
  </ul>
  <p>I have an unbalanced dataset, which has the following distribution:</p>
  <pre><code>pickups: 1202
  sedans: 1954
  suvs: 2510
  vans: 196
  </code></pre>
  <p>For this reason I have used class weights to mitigate this imbalance:</p>
  <pre><code>pickup_weight: 4.87
  sedan_weight: 3.0
  suv_weight: 2.33
  van_weight: 30.0
  </code></pre>
  <p>This seems like a small dataset but I am using this for fine tuning since I first train the model on a larger dataset of 16k images of these classes, though with images of vehicles taken from different angles as compared to my fine tune dataset.</p>
  <p>Now the questions that I'm having stem from the following observations:</p>
  <p>At the end of the final epoch, the results returned by  <code>model.fit</code> gave:</p>
  <ul>
  <li>training accuracy of <strong>0.9229</strong></li>
  <li>training loss of <strong>3.5055</strong></li>
  <li>validation accuracy of <strong>0.7906</strong></li>
  <li>validation loss of <strong>0.9382</strong></li>
  <li>training precision for class <strong>pickup</strong> of <strong>0.9186</strong></li>
  <li>training precision for class <strong>sedan</strong> of <strong>0.9384</strong></li>
  <li>training precision for class <strong>suv</strong> of <strong>0.9196</strong></li>
  <li>training precision for class <strong>van</strong> of <strong>0.8378</strong></li>
  <li>validation precision for class <strong>pickup</strong> of <strong>0.7805</strong></li>
  <li>validation precision for class <strong>sedan</strong> of <strong>0.8026</strong></li>
  <li>validation precision for class <strong>suv</strong> of <strong>0.0.8029</strong></li>
  <li>validation precision for class <strong>van</strong> of <strong>0.4615</strong></li>
  </ul>
  <p>The results returned by <code>model.evaluate</code> on my hold-out test set after training gave similar accuracy and loss values as the corresponding validation values in the last epoch and the precision values for each class were also nearly identical to the corresponding validation precisions.</p>
  <p>The lower, but still high enough, validation accuracy leads me to believe there is no overfitting problem as the model can generalize.</p>
  <p><em><strong>My first question</strong></em> is how can the validation loss be so much lower than the training loss?</p>
  <p>Furthermore, when I created a confusion matrix using:</p>
  <pre><code>test_images = np.array([x[0].numpy() for x in list(labeled_ds_test)])
  test_labels = np.array([x[1].numpy() for x in list(labeled_ds_test)])
  test_predictions = model.predict(test_images, batch_size=32)
  print(tf.math.confusion_matrix(tf.argmax(test_labels, 1), tf.argmax(test_predictions, 1)))
  </code></pre>
  <p>The results I got back were:</p>
  <pre><code>tf.Tensor(
  [[ 42  85 109   3]
   [ 72 137 177   4]
   [ 91 171 228  11]
   [  9  12  16   1]], shape=(4, 4), dtype=int32)
  </code></pre>
  <p>This shows an accuracy of only 35%!!</p>
  <p><em><strong>My second question</strong></em> is therefore this: how can the accuracy given by <code>model.predict</code> be so small when during training and evaluation the values seemed to indicate that my model was quite precise with its predictions?</p>
  <p>Am I using the predict method wrong or is my theoretical understanding of what's expected to happen completely off?</p>
  <p>I am at a bit of a loss here and would greatly appreciate any feedback. Thanks for reading this.</p>