title: Is it normal that transfer learning (VGG16) performs worse on CIFAR-10?
body: <p><strong>Note:</strong> I am not sure this is the right website to ask these kind of questions. Please tell me where I should ask them before downvoting this "because this isn't the right place to ask". Thanks!</p>
  
  <p>I am currently experimenting with deep learning using <a href="https://keras.io" rel="nofollow noreferrer">Keras</a>. I tried already a model similar to the one to be found on <a href="https://keras.io/examples/cifar10_cnn/" rel="nofollow noreferrer">the Keras example</a>. This yields expecting results:</p>
  
  <ul>
  <li>80% after 10-15 epochs without data augmentation before overfitting around the 15th epoch and</li>
  <li>80% after 50 epochs with data augmentation without any signs of overfitting.</li>
  </ul>
  
  <p>After this I wanted to try transfer learning. I did this by using the <a href="https://keras.io/applications/#vgg16" rel="nofollow noreferrer">VGG16</a> network without retraining its weights (see code below). This gave very poor results: 63% accuracy after 10 epochs with a very shallow curve (see picture below) which seems to be indicating that it will achieve acceptable results only (if ever) after a very long training time (I would expect 200-300 epochs before it reaches 80%).</p>
  
  <p><a href="https://i.stack.imgur.com/MWmsT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MWmsT.png" alt="Transfer learning results"></a></p>
  
  <p>Is this normal behavior for this kind of application? Here are a few things I could imagine to be the cause of these bad results:</p>
  
  <ul>
  <li>the CIFAR-10 dataset has images of <code>32x32</code> pixels, which might be too few for the VGG16 net</li>
  <li>The filters of VGG16 are not good for CIFAR-10, which would be solved by setting the weights to <code>trainable</code> or by starting with random weights (only copying the model and not the weights)</li>
  </ul>
  
  <p>Thanks in advance!</p>
  
  <hr>
  
  <p><strong>My code:</strong></p>
  
  <p><em>Note that the inputs are 2 datasets (50000 training images and 10000 testing images) which are labeled images with shape <code>32x32x3</code>. Each pixel value is a float in the range <code>[0.0, 1.0]</code>.</em></p>
  
  <pre class="lang-py prettyprint-override"><code>import keras
  
  # load and preprocess data...
  
  # get VGG16 base model and define new input shape
  vgg16 = keras.applications.vgg16.VGG16(input_shape=(32, 32, 3),
                                         weights='imagenet',
                                         include_top=False)
  
  # add new dense layers at the top
  x = keras.layers.Flatten()(vgg16.output)
  x = keras.layers.Dense(1024, activation='relu')(x)
  x = keras.layers.Dropout(0.5)(x)
  x = keras.layers.Dense(128, activation='relu')(x)
  predictions = keras.layers.Dense(10, activation='softmax')(x)
  
  # define and compile model
  model = keras.Model(inputs=vgg16.inputs, outputs=predictions)
  for layer in vgg16.layers:
      layer.trainable = False
  model.compile(optimizer='adam',
                loss='categorical_crossentropy',
                metrics=['accuracy'])
  
  # training and validation
  model.fit(x_train, y_train,
            batch_size=256,
            epochs=10,
            validation_data=(x_test, y_test))
  
  model.evaluate(x_test, y_test)
  </code></pre>