<p>The Batch normalization in LSTM is not that easy to implement. Some papers present some amazing results <a href="https://arxiv.org/pdf/1603.09025.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1603.09025.pdf</a> called Recurrent Batch normalization. The authors apply following the equations</p>

<p><a href="https://i.stack.imgur.com/2FhjQ.png" rel="nofollow noreferrer">BATCH-NORMALIZED LSTM</a></p>

<p>Unfortunately, this model is not implemented in keras yet only in tensorflow <a href="https://github.com/OlavHN/bnlstm" rel="nofollow noreferrer">https://github.com/OlavHN/bnlstm</a></p>

<p>However, I was able to get good results using (default) batch normalization after the activation function with the without centering and shifting. This approach is different from the paper above applying BN after c_t and h_t, maybe it is worth a try.</p>

<pre><code>model = Sequential()
model.add(LSTM(neurons1,
               activation=tf.nn.relu,
               return_sequences=True,
               input_shape=(timesteps, data_dim)))
model.add(BatchNormalization(momentum=m, scale=False, center=False))
model.add(LSTM(neurons2,
               activation=tf.nn.relu))
model.add(BatchNormalization(momentum=m, scale=False, center=False))
model.add(Dense(1))
</code></pre>
How to effectively use batch normalization in LSTM?
<p>I am trying to use batch normalization in LSTM using keras in R. In my dataset the target/output variable is the <code>Sales</code> column, and every row in the dataset records the <code>Sales</code> for each day in a year (2008-2017). The dataset looks like below:</p>

<p><a href="https://i.stack.imgur.com/mFJgq.png" rel="noreferrer"><img src="https://i.stack.imgur.com/mFJgq.png" alt="Sales data"></a></p>

<p>My objective is to build a LSTM model based on such dataset, which should be able to provide prediction at the end of training. I am training this model on the data from 2008-2016, and using half of the 2017 data as validation, and the rest as test set.</p>

<p>Previously, I tried creating a model using dropout and early stopping. This looks like below:</p>

<pre><code>mdl1 &lt;- keras_model_sequential()
mdl1 %&gt;%
  layer_lstm(units = 512, input_shape = c(1, 3), return_sequences = T ) %&gt;%  
  layer_dropout(rate = 0.3) %&gt;%
  layer_lstm(units = 512, return_sequences = FALSE) %&gt;%
  layer_dropout(rate = 0.2) %&gt;%
  layer_dense(units = 1, activation = "linear")

mdl1 %&gt;% compile(loss = 'mse', optimizer = 'rmsprop')
</code></pre>

<p>The model looks as follows</p>

<pre><code>___________________________________________________________
Layer (type)               Output Shape         Param #    
===========================================================
lstm_25 (LSTM)             (None, 1, 512)       1056768    
___________________________________________________________
dropout_25 (Dropout)       (None, 1, 512)       0          
___________________________________________________________
lstm_26 (LSTM)             (None, 512)          2099200    
___________________________________________________________
dropout_26 (Dropout)       (None, 512)          0          
___________________________________________________________
dense_13 (Dense)           (None, 1)            513        
===========================================================
Total params: 3,156,481
Trainable params: 3,156,481
Non-trainable params: 0
___________________________________________________________
</code></pre>

<p>To train the model, early stopping is used with a validation set.</p>

<pre><code>mdl1.history &lt;- mdl1 %&gt;% 
  fit(dt.tr, dt.tr.out, epochs=500, shuffle=F,
      validation_data = list(dt.val, dt.val.out),
      callbacks = list(
        callback_early_stopping(min_delta = 0.000001,  patience = 10, verbose = 1)
      ))
</code></pre>

<p>On top of this, I want to use batch normalization to speed up the training. As per my understanding, to use batch normalization, I need to divide the data into batches, and apply <code>layer_batch_normalization</code> for the input of each hidden layer. The model layers looks like as follows:</p>

<pre><code>batch_size &lt;- 32
mdl2 &lt;- keras_model_sequential()
mdl2 %&gt;%
  layer_batch_normalization(input_shape = c(1, 3), batch_size = batch_size) %&gt;%

  layer_lstm(units = 512, return_sequences = T) %&gt;%
  layer_dropout(rate = 0.3) %&gt;%
  layer_batch_normalization(batch_size = batch_size) %&gt;%

  layer_lstm(units = 512, return_sequences = F) %&gt;%
  layer_dropout(rate = 0.2) %&gt;%
  layer_batch_normalization(batch_size = batch_size) %&gt;%

  layer_dense(units = 1, activation = "linear")

mdl2 %&gt;% compile(loss = 'mse', optimizer = 'rmsprop')
</code></pre>

<p>This model looks as follows:</p>

<pre><code>______________________________________________________________________________
Layer (type)                                    Output Shape       Param #    
==============================================================================
batch_normalization_34 (BatchNormalization)     (32, 1, 3)         12         
______________________________________________________________________________
lstm_27 (LSTM)                                  (32, 1, 512)       1056768    
______________________________________________________________________________
dropout_27 (Dropout)                            (32, 1, 512)       0          
______________________________________________________________________________
batch_normalization_35 (BatchNormalization)     (32, 1, 512)       2048       
______________________________________________________________________________
lstm_28 (LSTM)                                  (32, 1, 512)       2099200    
______________________________________________________________________________
dropout_28 (Dropout)                            (32, 1, 512)       0          
______________________________________________________________________________
batch_normalization_36 (BatchNormalization)     (32, 1, 512)       2048       
______________________________________________________________________________
dense_14 (Dense)                                (32, 1, 1)         513        
==============================================================================
Total params: 3,160,589
Trainable params: 3,158,535
Non-trainable params: 2,054
______________________________________________________________________________
</code></pre>

<p>Training the model looks like before. Only difference lies in the training and validation dataset, which are made of sizes that are multiple of <code>batch_size</code> (32 here), by resampling data from the 2nd last batch to the last batch.</p>

<p>However, the performance of <code>mdl1</code> is much better than that of <code>mdl2</code>, as can be seen below.</p>

<p><a href="https://i.stack.imgur.com/cyzak.png" rel="noreferrer"><img src="https://i.stack.imgur.com/cyzak.png" alt="models"></a></p>

<p>I am not sure exactly what I am doing wrong, as I am starting with keras (and practical neural net in general). Additionally, the performance of first model is not so good as well; any suggestion on how to improve that would also be great.</p>