<p>Try this:</p>

<ol>
<li>Train the first model, which sets <code>trainable</code> to <code>False</code>. You don't have to train it to saturation, so I would start with your 5 epochs.</li>
<li>Go back and set <code>trainable</code> to <code>True</code> for all the <code>vgg19</code> parameters. Then, <a href="https://keras.io/getting-started/faq/#how-can-i-freeze-keras-layers" rel="nofollow noreferrer">per the documentation</a>, you can rebuild and recompile the model to have these changes take effect.</li>
<li>Continue training on the rebuilt model, which now has all parameters available for tuning.</li>
</ol>

<p>It is very common in transfer learning to completely freeze the transferred layers in order to preserve them. In the early stages of training your additional layers don't know what to do. That means a noisy gradient by the time it gets to the transferred layers, which will quickly "detune" them away from their previously well-tuned weights.</p>

<p>Putting it all together into some code, it would look something like this.</p>

<pre><code># Original code. Transfer VGG and freeze the weights.
vgg19 = keras.applications.vgg19.VGG19(
  weights='imagenet',
  include_top=False,
  input_shape=(img_height, img_width, img_channels))

for layer in vgg19.layers:
  layer.trainable = False

model = Sequential(layers=vgg19.layers)
model.add(Flatten())
model.add(Dense(1024, activation='relu'))
model.add(Dense(512, activation='relu'))
model.add(Dense(10, activation='softmax'))

opt = Adam(learning_rate=0.001, beta_1=0.9)
model.compile(
    loss='categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy'])

model.fit()

# New second stage: unfreeze and continue training.
for layer in vgg19.layers:
  layer.trainable = True

full_model = Sequential(layers=model.layers)
full_model.compile(
    loss='categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy'])

full_model.fit()
</code></pre>

<p>You may want to tune the learning rate for the fine-tuning stage. It's not essential to start, just something to keep in mind.</p>

<hr>

<p>A third option is to use discriminative learning rates, as introduced by Jeremy Howard and Sebastian Ruder in the <a href="https://arxiv.org/abs/1801.06146" rel="nofollow noreferrer">ULMFiT paper</a>. The idea is that, in Transfer Learning, you usually want the later layers to learn faster than the earlier, transferred layers. So you actually set the learning rates to be different for different sets of layers. The fastai library has <a href="https://docs.fast.ai/basic_train.html#Discriminative-layer-training" rel="nofollow noreferrer">a PyTorch implementation</a> that works by dividing the model into "layer groups" and allowing different parameters for each.</p>
Transfer learning only works with trainable set to false
<p>I have two models initialized like this</p>

<pre><code>vgg19 = keras.applications.vgg19.VGG19(
  weights='imagenet',
  include_top=False,
  input_shape=(img_height, img_width, img_channels))

for layer in vgg19.layers:
  layer.trainable = False

model = Sequential(layers=vgg19.layers)
model.add(Flatten())
model.add(Dense(1024, activation='relu'))
model.add(Dense(512, activation='relu'))
model.add(Dense(10, activation='softmax'))

opt = Adam(learning_rate=0.001, beta_1=0.9)
model.compile(
    loss='categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy'])
</code></pre>

<p>and</p>

<pre><code>vgg19_2 = keras.applications.vgg19.VGG19(
    weights='imagenet',
    include_top=False,
    input_shape=(img_height, img_width, img_channels))

model2 = Sequential(layers=vgg19_2.layers)
model2.add(Dense(1024, activation='relu'))
model2.add(Dense(512, activation='relu'))
model2.add(Dense(10, activation='softmax'))

opt = Adam(learning_rate=0.001, beta_1=0.9)
model2.compile(
    loss='categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy'])
</code></pre>

<p>In other words the only difference is the second model doesn't set vgg19 layers' trainable parameter to false. Unfortunately the model with trainable set to true does not learn the data.</p>

<p>When I use model.fit I get</p>

<pre><code>Trainable set to false:
Epoch 1/51
2500/2500 [==============================] - 49s 20ms/step - loss: 1.4319 - accuracy: 0.5466 - val_loss: 1.3951 - val_accuracy: 0.5693
Epoch 2/51
2500/2500 [==============================] - 47s 19ms/step - loss: 1.1508 - accuracy: 0.6009 - val_loss: 0.7832 - val_accuracy: 0.6023
Epoch 3/51
2500/2500 [==============================] - 48s 19ms/step - loss: 1.0816 - accuracy: 0.6256 - val_loss: 0.6782 - val_accuracy: 0.6153
Epoch 4/51
2500/2500 [==============================] - 47s 19ms/step - loss: 1.0396 - accuracy: 0.6450 - val_loss: 1.3045 - val_accuracy: 0.6103
</code></pre>

<p>The model trains to about 65% accuracy within a few epochs. However using model2 which should be able to make even better predictions (since there are more trainable parameters) I get:</p>

<pre><code>Epoch 1/5
2500/2500 [==============================] - 226s 90ms/step - loss: 2.3028 - accuracy: 0.0980 - val_loss: 2.3038 - val_accuracy: 0.1008
Epoch 2/5
2500/2500 [==============================] - 311s 124ms/step - loss: 2.3029 - accuracy: 0.0980 - val_loss: 2.2988 - val_accuracy: 0.1017
Epoch 3/5
2500/2500 [==============================] - 306s 123ms/step - loss: 2.3029 - accuracy: 0.0980 - val_loss: 2.3052 - val_accuracy: 0.0997
Epoch 4/5
2500/2500 [==============================] - 321s 129ms/step - loss: 2.3029 - accuracy: 0.0972 - val_loss: 2.3028 - val_accuracy: 0.0997
Epoch 5/5
2500/2500 [==============================] - 300s 120ms/step - loss: 2.3028 - accuracy: 0.0988 - val_loss: 2.3027 - val_accuracy: 0.1007
</code></pre>

<p>When I then try to compute weights gradients on my data I get only zeros. I understand that it may take a long time to train such a big neural net like vgg to optimum but considering the calculated gradients for the last 3 layers should be very similar in both cases why is the accuracy so low? Training for more time gives no improvement.</p>