title: adding CRF layer to LSTM flattens out learning curve
body: <p>I have implemented a bi-LSTM named entity tagger in keras with tensorflow backend (tf version 1.13.1). The task of the network, given a sequence of word tokens, is to tag every element of the sequence with an entity type label. I subsequently added a Conditional Random Field (CRF) layer on top on the network; in state-of-the-art named entity taggers, CRF is typically used to improve the result of bi-LSTM by imposing adjacency constraints on neighboring elements in the sequence. </p>
  
  <p>This is the network code. The loop performs custom mini-batch training due to the nature of the data.</p>
  
  <pre><code>from keras.models import Sequential
  from keras.layers import LSTM, Dense, TimeDistributed, Bidirectional, Dropout
  from keras import optimizers
  from keras_contrib.layers import CRF
  
  model = Sequential()
  model.add(Bidirectional(LSTM(params['hidden_size'], return_sequences=True), input_shape=(params['max_seq_len'], params['emb_size'])))
  model.add(Dropout(params['dropout']))
  model.add(TimeDistributed(Dense(params['n_classes'], activation=params['activation'])))
  model.add(CRF(params['n_classes'], sparse_target=True))
  
  model.compile(loss=crf.loss_function, optimizer='sgd'), metrics=['mse']) 
  
  for epoch in range(params['n_epochs']):
  
      for X,y in data:
  
          X = X.reshape(1, params['max_seq_len'], params['emb_size'])
          y = y.reshape(1, params['max_seq_len'], 1)
  
          history = model.fit(X, y, epochs=1, batch_size=params['batch_size'], verbose=1)
  </code></pre>
  
  <p>I expected the addition of CRF layer to improve the network's performance; however, it not only did not improve the results, but has also completely flattened out the learning curve, as measured by the mean squared error (shown in the bottom panel plot of mean squared error below, averaged over each epoch). This is puzzling, especially because the loss keeps on decreasing (as shown in the top panel plot of loss below, likewise averaged over each epoch). In addition, the model's performance on the test set was brought down to zero.</p>
  
  <p><a href="https://i.stack.imgur.com/4DBKF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4DBKF.png" alt="enter image description here"></a></p>
  
  <p>Is there an error in the implementation on the CRF? Or it be because the CRF loss function is not measured via mean squared error?</p>
  
  <p>A simulated data set that can be used to reproduce the problem is provided in <a href="https://stackoverflow.com/questions/59282996/zero-predictions-despite-masking-support-for-zero-padded-mini-batch-lstm-trainin">this</a> SO question (same implementation as here but without CRF layer).</p>