<p>Regularizers that'll work best will depend on your specific architecture, data, and problem; as usual, there isn't a single cut to rule all, but there <em>are</em> do's and (especially) don't's, as well as <em>systematic means</em> of determining what'll work best - via careful introspection and evaluation.</p>

<hr>

<p><strong>How does RNN regularization work?</strong></p>

<p>Perhaps the best approach to understanding it is <em>information</em>-based. First, see "How does 'learning' work?" and "RNN: Depth vs. Width". To understand RNN regularization, one must understand how RNN handles information and learns, which the referred sections describe (though not exhaustively). Now to answer the question:</p>

<p>RNN regularization's goal is any regularization's goal: maximizing information utility and traversal of the test loss function. The specific <em>methods</em>, however, tend to differ substantially for RNNs per their recurrent nature - and some work better than others; see below.</p>

<hr>

<p><strong>RNN regularization methods</strong>: </p>

<p><strong>WEIGHT DECAY</strong> </p>

<ol>
<li><p><strong>General</strong>: shrinks the norm ('average') of the weight matrix</p>

<ul>
<li><em>Linearization</em>, depending on activation; e.g. <code>sigmoid</code>, <code>tanh</code>, but less so <code>relu</code></li>
<li><em>Gradient boost</em>, depending on activation; e.g. <code>sigmoid</code>, <code>tanh</code> grads flatten out for large activations - linearizing enables neurons to keep learning</li>
</ul></li>
<li><p><strong>Recurrent weights</strong>: default <code>activation='sigmoid'</code> </p>

<ul>
<li><strong>Pros</strong>: linearizing can help BPTT (remedy vanishing gradient), hence also <em>learning long-term dependencies</em>, as <em>recurrent information utility</em> is increased</li>
<li><strong>Cons</strong>: linearizing can harm representational power - however, this can be offset by stacking RNNs</li>
</ul></li>
<li><p><strong>Kernel weights</strong>: for many-to-one (<code>return_sequences=False</code>), they work similar to weight decay on a typical layer (e.g. <code>Dense</code>). For many-to-many (<code>=True</code>), however, kernel weights operate on every timestep, so pros &amp; cons similar to above will apply.</p></li>
</ol>

<p><strong>Dropout</strong>:</p>

<ul>
<li><strong>Activations</strong> (kernel): can benefit, but only if limited; values are usually kept less than <code>0.2</code> in practice. Problem: tends to introduce too much noise, and erase important context information, especially in problems w/ limited timesteps.</li>
<li><strong>Recurrent activations</strong> (<code>recurrent_dropout</code>): the <a href="https://stackoverflow.com/questions/44924690/keras-the-difference-between-lstm-dropout-and-lstm-recurrent-dropout">recommended dropout</a></li>
</ul>

<p><strong>Batch Normalization</strong>:</p>

<ul>
<li><strong>Activations</strong> (kernel): worth trying. Can benefit substantially, or not.</li>
<li><strong>Recurrent activations</strong>: should work better; see <a href="https://arxiv.org/abs/1603.09025" rel="noreferrer">Recurrent Batch Normalization</a>. No Keras implementations yet as far as I know, but I may implement it in the future.</li>
</ul>

<p><strong>Weight Constraints</strong>: set hard upper-bound on weights l2-norm; possible alternative to weight decay. </p>

<p><strong>Activity Constraints</strong>: don't bother; for most purposes, if you have to manually constrain your outputs, the layer itself is probably learning poorly, and the solution is elsewhere.</p>

<hr>

<p><strong>What should I do?</strong> Lots of info - so here's some concrete advice:</p>

<ol>
<li><p><strong>Weight decay</strong>: try <code>1e-3</code>, <code>1e-4</code>, see which works better. Do <em>not</em> expect the same value of decay to work for <code>kernel</code> and <code>recurrent_kernel</code>, especially depending on architecture. Check weight shapes - if one is much smaller than the other, apply smaller decay to former</p></li>
<li><p><strong>Dropout</strong>: try <code>0.1</code>. If you see improvement, try <code>0.2</code> - else, scrap it</p></li>
<li><p><strong>Recurrent Dropout</strong>: start with <code>0.2</code>. Improvement --> <code>0.4</code>. Improvement --> <code>0.5</code>, else <code>0.3</code>.</p></li>
<li><strong>Batch Normalization</strong>: try. Improvement --> keep it - else, scrap it.</li>
<li><strong>Recurrent Batchnorm</strong>: same as 4.</li>
<li><strong>Weight constraints</strong>: advisable w/ higher learning rates to prevent exploding gradients - else use higher weight decay</li>
<li><strong>Activity constraints</strong>: probably not (see above)</li>
<li><strong>Residual RNNs</strong>: introduce significant changes, along a regularizing effect. See application in <a href="https://arxiv.org/abs/1803.04831" rel="noreferrer">IndRNNs</a></li>
<li><strong>Biases</strong>: weight decay and constraints become important upon attaining good backpropagation properties; without them on bias weights but <em>with</em> them on kernel (K) &amp; recurrent kernel (RK) weights, bias weights may grow much faster than the latter two, and dominate the transformation - also leading to exploding gradients. I recommend weight decay / constraint less than or equal to that used on K &amp; RK. Also, with <code>BatchNormalization</code>, you <s>can</s> <em>cannot</em> set <code>use_bias=False</code> as an "equivalent"; BN applies to <em>outputs</em>, not <em>hidden-to-hidden transforms</em>.</li>
<li><strong>Zoneout</strong>: don't know, never tried, might work - see <a href="https://arxiv.org/abs/1606.01305" rel="noreferrer">paper</a>.</li>
<li><strong>Layer Normalization</strong>: some report it working better than BN for RNNs - but my application found it otherwise; <a href="https://arxiv.org/abs/1607.06450" rel="noreferrer">paper</a></li>
<li><strong>Data shuffling</strong>: is a strong regularizer. Also shuffle <em>batch samples</em> (samples in batch). See relevant info on <a href="https://stackoverflow.com/questions/58276337/proper-way-to-feed-time-series-data-to-stateful-lstm/58277760#58277760">stateful RNNs</a></li>
<li><strong>Optimizer</strong>: can be an inherent regularizer. Don't have a full explanation, but in my application, Nadam (&amp; NadamW) has stomped every other optimizer - worth trying.</li>
</ol>

<p><strong>Introspection</strong>: bottom section on 'learning' isn't worth much without this; don't just look at validation performance and call it a day - <em>inspect</em> the effect that adjusting a regularizer has on <em>weights</em> and <em>activations</em>. Evaluate using info toward bottom &amp; relevant theory.</p>

<p><strong>BONUS</strong>: weight decay can be powerful - even more powerful when done right; turns out, <em>adaptive optimizers</em> like Adam can harm its effectiveness, as described in <a href="https://arxiv.org/abs/1711.05101" rel="noreferrer">this paper</a>. <em>Solution</em>: use AdamW. My Keras/TensorFlow implementation <a href="https://github.com/OverLordGoldDragon/keras-adamw" rel="noreferrer">here</a>.</p>

<hr>

<p><strong>This is too much!</strong> Agreed - welcome to Deep Learning. Two tips here:</p>

<ol>
<li><a href="https://philipperemy.github.io/visualization/" rel="noreferrer"><em>Bayesian Optimization</em></a>; will save you time especially on prohibitively expensive training.</li>
<li><code>Conv1D(strides &gt; 1)</code>, for many timesteps (<code>&gt;1000</code>); slashes dimensionality, shouldn't harm performance (may in fact improve it).</li>
</ol>

<hr>

<p><strong>Introspection Code</strong>: </p>

<p><strong>Gradients</strong>: see <a href="https://stackoverflow.com/questions/59017288/how-to-visualize-rnn-lstm-gradients-in-keras-tensorflow/59017289#59017289">this answer</a></p>

<p><strong>Weights</strong>: see <a href="https://stackoverflow.com/questions/59275959/how-to-visualize-rnn-lstm-weights-in-keras-tensorflow/59275960#59275960">this answer</a></p>

<p><strong>Weight norm tracking</strong>: see <a href="https://stackoverflow.com/questions/61481921/how-to-set-and-track-weight-decays/61481922#61481922">this Q &amp; A</a></p>

<p><strong>Activations</strong>: see <a href="https://stackoverflow.com/questions/58356868/how-visualize-attention-lstm-using-keras-self-attention-package/58357581#58357581">this answer</a></p>

<p><strong>Weights</strong>: <a href="https://github.com/OverLordGoldDragon/see-rnn/blob/master/see_rnn/visuals_rnn.py#L10" rel="noreferrer"><code>see_rnn.rnn_histogram</code></a> or <a href="https://github.com/OverLordGoldDragon/see-rnn/blob/master/see_rnn/visuals_rnn.py#L264" rel="noreferrer"><code>see_rnn.rnn_heatmap</code></a> (examples in README)</p>

<hr>

<p><strong>How does 'learning' work?</strong></p>

<p>The 'ultimate truth' of machine learning that is seldom discussed or emphasized is, <strong>we don't have access to the function we're trying to optimize</strong> - the <em>test loss function</em>. <em>All</em> of our work is with what are <em>approximations</em> of the true loss surface - both the train set and the validation set. This has some critical implications:</p>

<ol>
<li>Train set global optimum can lie <em>very far</em> from test set global optimum</li>
<li>Local optima are unimportant, and irrelevant:

<ul>
<li>Train set local optimum is almost always a better test set optimum</li>
<li>Actual local optima are almost impossible for high-dimensional problems; for the case of the "saddle", you'd need the gradients w.r.t. <em>all of the millions of parameters</em> to equal zero at once</li>
<li><a href="https://www.wikiwand.com/en/Attractor" rel="noreferrer">Local attractors</a> are lot more relevant; the analogy then shifts from "falling into a pit" to "gravitating into a strong field"; once in that field, your loss surface topology is bound to that set up by the field, which defines its own local optima; high LR can help exit a field, much like "escape velocity"</li>
</ul></li>
</ol>

<p>Further, loss functions are way too complex to analyze directly; a better approach is to <em>localize</em> analysis to individual layers, their weight matrices, and roles relative to the entire NN. Two key considerations are:</p>

<ol start="3">
<li><p><strong>Feature extraction capability</strong>. <em>Ex</em>: the driving mechanism of deep classifiers is, given input data, to <em>increase class separability</em> with each layer's transformation. Higher quality features will filter out irrelevant information, and deliver what's essential for the output layer (e.g. softmax) to learn a separating hyperplane.</p></li>
<li><p><strong>Information utility</strong>. <em>Dead neurons</em>, and <em>extreme activations</em> are major culprits of poor information utility; no single neuron should dominate information transfer, and too many neurons shouldn't lie purposeless. Stable activations and weight distributions enable gradient propagation and continued learning.</p></li>
</ol>

<hr>

<p><strong>How does regularization work?</strong> read above first</p>

<p>In a nutshell, via maximizing NN's information utility, and improving estimates of the test loss function. Each regularization method is unique, and no two exactly alike - see "RNN regularizers".</p>

<hr>

<p><strong>RNN: Depth vs. Width</strong>: not as simple as "one is more nonlinear, other works in higher dimensions".</p>

<ul>
<li><strong>RNN width</strong> is defined by (1) # of input channels; (2) # of cell's filters (output channels). As with CNN, each RNN filter is an <em>independent feature extractor</em>: <em>more</em> is suited for higher-complexity information, including but not limited to: dimensionality, modality, noise, frequency.</li>
<li><strong>RNN depth</strong> is defined by (1) # of stacked layers; (2) # of timesteps. Specifics will vary by architecture, but from information standpoint, unlike CNNs, RNNs are <em>dense</em>: every timestep influences the ultimate output of a layer, hence the ultimate output of the next layer - so it again isn't as simple as "more nonlinearity"; stacked RNNs exploit both spatial and temporal information.</li>
</ul>

<hr>

<p><strong>Update</strong>:</p>

<p>Here is an example of a near-ideal RNN gradient propagation for 170+ timesteps:</p>

<p><img src="https://i.stack.imgur.com/71seM.png" width="550"></p>

<p>This is rare, and was achieved via careful regularization, normalization, and hyperparameter tuning. Usually we see a large gradient for the last few timesteps, which drops off sharply toward left - as <a href="https://stackoverflow.com/questions/59017288/how-to-visualize-rnn-lstm-gradients-in-keras-tensorflow/59017289#59017289">here</a>. Also, since the model is stateful and fits 7 equivalent windows, gradient effectively spans <strong>1200 timesteps</strong>.</p>

<p><strong>Update 2</strong>: see 9 w/ new info &amp; correction</p>

<p><strong>Update 3</strong>: add weight norms &amp; weights introspection code</p>
RNN Regularization: Which Component to Regularize?
<p>I am building an RNN for classification (there is a softmax layer after the RNN). There are so many options for what to regularize and I am not sure if to just try all of them, would the effect be the same? which components do I regularize for what situation?</p>

<p>The components being:</p>

<ul>
<li>Kernel weights (layer input)</li>
<li>Recurrent weights</li>
<li>Bias</li>
<li>Activation function (layer output)</li>
</ul>