<p>Truncated back-propagation aims at speeding up learning sequences (e.g. with LSTM), by computing approximate gradients on "short" sequences, instead of on full sequences. I guess this is what the documentation means by making "the learning process tractable".</p>

<p>This method seems to originate from the work of Mikolov on "Statistical Language Models based on Neural Networks" (his thesis). And as explained by Alex Graves in a <a href="http://arxiv.org/abs/1308.0850" rel="nofollow">well-cited paper</a> (page 9):</p>

<blockquote>
  <p>This form of truncated back propagation has been considered before for RNN language modelling [23], and found to speed up training (by reducing the sequence length and hence increasing the frequency of stochastic weight updates) without affecting the network’s ability to learn long-range dependencies.</p>
</blockquote>

<p>[23] is Mikolov's thesis.</p>

<hr>

<p>In short, truncated back-propagation is a "trick" to speed up learning over sequences, without losing (too much) significant information. Note that truncating too much can backdraft (i.e. lose significant information).</p>
What does "truncated gradients" mean in LSTM?
<p>I'm learning a tensorflow tutorial about LSTM: <a href="https://www.tensorflow.org/versions/master/tutorials/recurrent/index.html#truncated-backpropagation" rel="nofollow">Truncated Backpropagation</a>. </p>

<p>This section says the code uses "truncated backpropagation", so what exactly does this mean?</p>