<p><strong>TL;DR</strong>: use <code>tf.clip_by_global_norm</code> for gradient clipping.</p>

<h3>clip_by_value</h3>

<p><code>tf.clip_by_value</code> clips each value inside one tensor, regardless of the other values in the tensor. For instance,</p>

<pre><code>tf.clip_by_value([-1, 2, 10], 0, 3)  -&gt; [0, 2, 3]  # Only the values below 0 or above 3 are changed
</code></pre>

<p>Consequently, it can change the direction of the tensor, so it should be used if the values in the tensor are decorrelated one from another (which is not the case for gradient clipping), or to avoid zero / infinite values in a tensor that could lead to Nan / infinite values elsewhere (by clipping with a minimum of epsilon=1e-8 and a very big max value for instance).</p>

<h3>clip_by_norm</h3>

<p><code>tf.clip_by_norm</code> rescales one tensor if necessary, so that its L2 norm does not exceed a certain threshold. It's useful typically to avoid exploding gradient on one tensor, because you keep the gradient direction. For instance:</p>

<pre><code>tf.clip_by_norm([-2, 3, 6], 5)  -&gt; [-2, 3, 6]*5/7  # The original L2 norm is 7, which is &gt;5, so the final one is 5
tf.clip_by_norm([-2, 3, 6], 9)  -&gt; [-2, 3, 6]  # The original L2 norm is 7, which is &lt;9, so it is left unchanged
</code></pre>

<p>However, <code>clip_by_norm</code> works on only one gradient, so if you use it on all your gradient tensors, you'll unbalance them (some will be rescaled, others not, and not all with the same scale).</p>

<p>Note that the two first ones work on only one tensor, while the last one is used on a list of tensors.</p>

<h3>clip_by_global_norm</h3>

<p><code>tf.clip_by_global_norm</code> rescales a list of tensors so that the total norm of the vector of all their norms does not exceed a threshold. The goal is the same as  <code>clip_by_norm</code> (avoid exploding gradient, keep the gradient directions), but it works on all the gradients at once rather than on each one separately (that is, all of them are rescaled by the same factor if necessary, or none of them are rescaled). This is better, because the balance between the different gradients is maintained.</p>

<p>For instance: </p>

<pre><code>tf.clip_by_global_norm([tf.constant([-2, 3, 6]),tf.constant([-4, 6, 12])] , 14.5)
</code></pre>

<p>will rescale both tensors by a factor <code>14.5/sqrt(49 + 196)</code>, because the first tensor has a L2 norm of 7, the second one 14, and <code>sqrt(7^2+ 14^2)&gt;14.5</code></p>

<p>This (<code>tf.clip_by_global_norm</code>) is the one that you should use for gradient clipping. See <a href="https://arxiv.org/pdf/1211.5063.pdf" rel="noreferrer">this</a> for instance for more information.</p>

<h3>Choosing the value</h3>

<p>Choosing the max value is the hardest part. You should use the biggest value such that you don't have exploding gradient (whose effects can be <code>Nan</code>s or <code>infinite</code> values appearing in your tensors, constant loss /accuracy after a few training steps). The value should be bigger for <code>tf.clip_by_global_norm</code> than for the others, since the global L2 norm will be mechanically bigger than the other ones due to the number of tensors implied.</p>
Difference between tf.clip_by_value and tf.clip_by_global_norm for RNN's and how to decide max value to clip on?
<p>Want to understand the difference in roles of <code>tf.clip_by_value</code> and <code>tf.clip_by_global_norm</code> during the implementation of Gradient Clipping in TensorFlow. Which one is preferred and how to decide the max value to clip on?</p>