<p>I like to do the following, yet the only thing I know is that some parameters prefers not to be regularized with L2, such as batch norm parameters and biases. LSTMs contains one Bias tensor (despite conceptually it has many biases, they seem to be concatenated or something, for performance), and for the batch normalization I add "noreg" in the variables' name to ignore it too. </p>

<pre><code>loss = your regular output loss
l2 = lambda_l2_reg * sum(
    tf.nn.l2_loss(tf_var)
        for tf_var in tf.trainable_variables()
        if not ("noreg" in tf_var.name or "Bias" in tf_var.name)
)
loss += l2
</code></pre>

<p>Where <code>lambda_l2_reg</code> is the small multiplier, e.g.: <code>float(0.005)</code></p>

<p>Doing this selection (which is the full <code>if</code> in the loop discarding some variables in the regularization) once made me jump <strong>from 0.879 F1 score to 0.890 in one shot</strong> of testing the code without readjusting the value of the config's <code>lambda</code>, well this was including both the changes for the batch normalisation and the Biases and I had other biases in the neural network.</p>

<p>According to <a href="http://jmlr.csail.mit.edu/proceedings/papers/v28/pascanu13.pdf" rel="noreferrer">this paper</a>, regularizing the recurrent weights may help with exploding gradients. </p>

<p>Also, according to <a href="https://arxiv.org/pdf/1409.2329.pdf" rel="noreferrer">this other paper</a>, dropout would be better used between stacked cells and not inside cells if you use some. </p>

<p>About the exploding gradient problem, if you use gradient clipping with the loss that has the L2 regularization already added to it, that regularization will be taken into account too during the clipping process.</p>

<hr>

<p>P.S. Here is the neural network I was working on: <a href="https://github.com/guillaume-chevalier/HAR-stacked-residual-bidir-LSTMs" rel="noreferrer">https://github.com/guillaume-chevalier/HAR-stacked-residual-bidir-LSTMs</a> </p>
Regularization for LSTM in tensorflow
<p>Tensorflow offers a nice LSTM wrapper.</p>

<pre><code>rnn_cell.BasicLSTM(num_units, forget_bias=1.0, input_size=None,
           state_is_tuple=False, activation=tanh)
</code></pre>

<p>I would like to use regularization, say L2 regularization. However, I don't have direct access to the different weight matrices used in the LSTM cell, so I cannot explicitly do something like</p>

<pre><code>loss = something + beta * tf.reduce_sum(tf.nn.l2_loss(weights))
</code></pre>

<p>Is there a way to access the matrices or use regularization somehow with LSTM?</p>