<p>Idea #1: Gradient clipping is often applied in RNNs. Here is an example of implementation: <a href="https://stackoverflow.com/questions/36498127/how-to-effectively-apply-gradient-clipping-in-tensor-flow">How to effectively apply gradient clipping in tensor flow?</a></p>

<p>Idea #2: Using <a href="https://arxiv.org/pdf/1603.09025.pdf" rel="nofollow noreferrer">Recurrent Batch Normalization (arXiv)</a> (<a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">Batch Normalization</a>)</p>

<p>Here is a Tensorflow implementation of a batch normalized LSTM cell: <a href="https://github.com/OlavHN/bnlstm/blob/master/lstm.py" rel="nofollow noreferrer">https://github.com/OlavHN/bnlstm/blob/master/lstm.py</a></p>

<p>This implementation is explained in the article here : <a href="http://olavnymoen.com/2016/07/07/rnn-batch-normalization" rel="nofollow noreferrer">Batch normalized LSTM for Tensorflow</a></p>
State Normalization of RNNs
<p>Perhaps a question better posed to Computer Science or Cross Validated?</p>

<hr>

<p>I'm beginning some work with LSTM on sequences of arbitrary length and one problem I'm experiencing and that I haven't seen addressed, is that my network seems to have developed a couple parameters that grow linearly (perhaps as a measure of time?).</p>

<p>The obvious issue with this is that the training data is bounded at a sequence of length <code>x</code> and so the network grows this parameter reasonably up until tilmestep <code>x</code>. But after that, the network will eventually NAN because values are getting too extreme.</p>

<p>Has anyone read anything about the normalization of stabilization of states over time?</p>

<p>Any suggestions would be much appreciated.</p>