title: Fine-tune lstm in Keras
body: <p>I would like to compare the performance of a baseline LSTM and a fine-tuned model. I found many examples online, but all of them use pretrained networks, such as VGG16, ResNet50, and I was wondering if anybody can help with a language model.
  My network is the following:</p>
  
  <pre><code>model = Sequential()
  model.add(Embedding(vocab_size, embedding_size, input_length=5, weights=[pretrained_weights]))
  model.add(Bidirectional(LSTM(units=embedding_size, return_sequences=True)))
  model.add(Bidirectional(LSTM(units=embedding_size, return_sequences=True)))
  model.add(Bidirectional(LSTM(units=embedding_size, return_sequences=True)))
  model.add(Bidirectional(LSTM(units=embedding_size, return_sequences=True)))
  model.add(Bidirectional(LSTM(units=embedding_size, return_sequences=True)))
  model.add(Bidirectional(LSTM(units=embedding_size, return_sequences=True)))
  model.add(Bidirectional(LSTM(units=embedding_size, return_sequences=True)))
  model.add(Bidirectional(LSTM(units=embedding_size, return_sequences=True)))
  model.add(Bidirectional(LSTM(units=embedding_size, return_sequences=True)))
  model.add(Bidirectional(LSTM(units=embedding_size)))
  model.add(Dense(2000, activation='softmax'))
  
  model.compile(loss='categorical_crossentropy',
            optimizer = RMSprop(lr=0.001),
            metrics=['accuracy'])
  
  # fit network
  model.fit(X_train, y_train, epochs=100, verbose=2, batch_size=16)
  </code></pre>
  
  <p>I tried to fine-tune it by freezing the first layers, by adding:</p>
  
  <pre><code>for layer in model.layers[:3]:
      layer.trainable = False
  </code></pre>
  
  <p>before compiling, but the accuracy drops, with the baseline I have around 10% accuracy (I want to test on a corpus that is totally different from the training and evaluate the performance), while if I freeze the layers I get something like 0.002.
  Can somebody help me understand what I am doing wrong or suggest me a different fine-tuning method?
  Thank you in advance!</p>