<p>No, you cannot use Batch Normalization on a recurrent neural network, as the statistics are computed per batch, this does not consider the recurrent part of the network. Weights are shared in an RNN, and the activation response for each "recurrent loop" might have completely different statistical properties.</p>

<p>Other techniques similar to Batch Normalization that take these limitations into account have been developed, for example <a href="https://arxiv.org/abs/1607.06450" rel="noreferrer">Layer Normalization</a>. There are also reparametrizations of the LSTM layer that allow Batch Normalization to be used, for example as described in <a href="https://arxiv.org/abs/1603.09025" rel="noreferrer">Recurrent Batch Normalization by Coijmaans et al. 2016.</a></p>
Is it normal to use batch normalization in RNN & LSTM?
<p>I know in regular neural nets people use batch norm before activation and it will reduce the reliance on good weight initialization. I wonder if it would do the same to RNN/lstm RNN when i use it. Does anyone have any experience with it?</p>