<p>While nothing prevents you from clipping them as well, there is no reason to do so. A nice paper with reasons is <a href="https://arxiv.org/pdf/1211.5063.pdf" rel="nofollow noreferrer">here</a>, I'll try to give you an overview.</p>

<p>The problem we're trying to solve by gradient clipping is that of <strong>exploding gradients:</strong> Let's assume that your RNN layer is computed like this:</p>

<pre><code> h_t = sigmoid(U * x + W * h_tm1 + b)
</code></pre>

<p>So forgetting about the nonlinearity for a while, you could say that a current state <code>h_t</code> depends on some earlier state <code>h_{t-T}</code> as <code>h_t = W^T * h_tmT + input</code>. So if the matrix <code>W</code> inflates the hidden state, the influence of that old hidden state is growing exponentially with time. And the same happens as you backpropagate the gradient, resulting in gradients that will most likely get you to to some useless point in the parameter space.</p>

<p>On the other hand, the output layer is applied just once during both forward and backward pass, so while it may complicate the learning, it will only be by a 'constant' factor, independent of the unrolling in time.</p>

<p>To get a bit more technical: The crucial quantity which determines whether you get exploding gradient is the largest eigenvalue of <code>W</code>. If it is larger than one (or smaller than -1, then it's real fun :-)), then you get exploding gradients. Conversely, if it's smaller than one, you'll suffer from <strong>vanishing gradients</strong>, making it difficult to learn long-term dependencies. You can find a nice discussion of these phenomena <a href="https://arxiv.org/pdf/1607.03474.pdf" rel="nofollow noreferrer">here</a>, with pointers to classical literature.</p>

<p>If we take the sigmoid back into the picture, it becomes more difficult to get exploding gradients, as the gradients get dampened by at least a factor of 4 when being backpropagated through it. But still, have an eigenvalue larger than 4 and you'll have adventures :-) It's rather important to initialize carefully, the second paper gives some hints. With <code>tanh</code>, there is little dampening around zero and ReLU just propagates the gradient through, so these are rather prone to gradient explodions and thus sensitive to initialization and gradient clipping.</p>

<p>Overall, <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">LSTMs</a> have better learning properties than vanilla RNNs, esp. with regard to the vanishing gradients. Though from my experience, gradient clipping is usually necessary with them as well.</p>

<p>EDIT: When to clip?
Right before the update of the weights, i.e. you do the backprop unaltered. The thing is that gradient clipping is kind of a dirty hack. You still want your gradient as precise as possible, so you better don't distort it in the middle of the backprop. Just that if you see the gradient become very large, you say <em>Nah, this smells. I better make a tiny step.</em> and clipping is an easy way to do it (it may be that only some elements of the gradient are exploded while the others are still well behaved and informative). With most of the toolkits, <strong>you don't have the choice anyway</strong>, because the backpropagation happens atomically.</p>
Should The Gradients For The Output Layer of an RNN Clipped?
<p>I am currently training an LSTM RNN for time-series forecasting. I understand that it is common practice to clip the gradients of the RNN when it crosses a certain threshold. However, I am not completely clear on whether or not this includes the output layer. </p>

<p>If we call the hidden layer of an RNN h, then the output is sigmoid(connected_weights*h + bias). I know that the gradients for the weights for determining the hidden layer are clipped, but does the same go for the output layer? </p>

<p>In other words, are the gradients for the connected_weights also clipped in gradient clipping?</p>