<p>There are many possibilities why something like this occurs:</p>

<ol>
<li><p><strong>Your parameters trajectory changed its basin of attraction</strong> - this means that your system left a stable trajectory and switched to another one. This was probably due to randomization like e.g. batch sampling or dropout.</p></li>
<li><p><strong>LSTM instability</strong>- <strong>LSTM</strong>s are believed to be extremely unstable in terms of training. It was also reported that very often it's really time consuming for them to stabilize.</p></li>
</ol>

<p>Due to the latest research (e.g. from <a href="https://arxiv.org/pdf/1706.10239.pdf" rel="noreferrer">here</a>) I would recommend you decreasing the batch size and leaving it for more epochs. I would also try to check if e.g. topology of a network is not to complexed (or plain) in terms of amount of patterns it need to learn. I would also try switch to either <code>GRU</code> or <code>SimpleRNN</code>.</p>
Strange loss curve while training LSTM with Keras
<p>I'm trying to train an LSTM for some a binary classification problem. When I plot <code>loss</code> curve after the training, there are strange picks in it. Here are some examples:</p>
<p><a href="https://i.stack.imgur.com/oxBpf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/oxBpf.png" alt="enter image description here" /></a></p>
<p><a href="https://i.stack.imgur.com/9cvie.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9cvie.png" alt="enter image description here" /></a></p>
<p>Here is the basic code</p>
<pre><code>model = Sequential()
model.add(recurrent.LSTM(128, input_shape = (columnCount,1), return_sequences=True))
model.add(Dropout(0.5))
model.add(recurrent.LSTM(128, return_sequences=False))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(optimizer='adam', 
             loss='binary_crossentropy', 
             metrics=['accuracy'])

new_train = X_train[..., newaxis]

history = model.fit(new_train, y_train, nb_epoch=500, batch_size=100, 
                    callbacks = [EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=2, verbose=0, mode='auto'), 
                                 ModelCheckpoint(filepath=&quot;model.h5&quot;, verbose=0, save_best_only=True)],
                    validation_split=0.1)

# list all data in history
print(history.history.keys())
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
</code></pre>
<p>I don't understand why do that picks occur? Any ideas?</p>