<p>Both RNN and LSTM can be sequence learners. RNN suffers from vanishing gradient point problem. This problem causes the RNN to have trouble in remembering values of past inputs after more than 10 timesteps approx. (RNN can remember previously seen inputs for a few time steps only)</p>

<p>LSTM is designed to solve the vanishing gradient point problem in RNN. LSTM has the capability of bridging long time lags between inputs. In other words, it is able to remember inputs from up to 1000 time steps in the past (some papers even made claims it can go more than this). This capability makes LSTM an advantage for learning long sequences with long time lags. Refer to Alex Graves Ph.D. thesis <a href="http://www.cs.toronto.edu/~graves/phd.pdf" rel="nofollow noreferrer">Supervised Sequence Labelling
with Recurrent Neural Networks</a> for some details. If you are new to LSTM, I recommend <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">Colah's blog</a> for super simple and easy explanation.</p>

<p>However, recent advances in RNN also claim that with careful initialization, RNN can also learn long sequences comparable to the performance of LSTM. <a href="http://arxiv.org/abs/1504.00941" rel="nofollow noreferrer">A Simple Way to Initialize Recurrent Networks of Rectified Linear Units</a>.</p>
What is a Recurrent Neural Network, what is a Long Short Term Memory (LSTM) network, and is it always better?
<p>First, let me apologize for cramming three questions in that title. I'm not sure what better way is there.</p>

<p>I'll get right to it. I think I understand feedforward neural networks pretty well. </p>

<p>But LSTM really escapes me, and I feel maybe this is because I don't have a very good grasp of Recurrent neural networks in general. I have went through Hinton's and Andrew Ng's course on Coursera. A lot of it still doesn't make sense to me.</p>

<p>From what I understood, recurrent neural networks are different from feedforward neural networks in that past values influence the next prediction. Recurrent neural network are generally used for sequences.</p>

<p>The example I saw of recurrent neural network was binary addition.</p>

<pre><code>    010
+   011
</code></pre>

<p>A recurrent neural network would take the right most 0 and 1 first, output a 1. Then take the 1,1 next, output a zero, and carry the 1. Take the next 0,0 and output a 1 because it carried the 1 from last calculation. Where does it store this 1? In feed forward networks the result is basically:</p>

<pre><code>    y = a(w*x + b)
where w = weights of connections to previous layer
and x = activation values of previous layer or inputs
</code></pre>

<p>How is a recurrent neural network calculated? I am probably wrong but from what I understood, recurrent neural networks are pretty much feedforward neural network with T hidden layers, T being number of timesteps. And each hidden layer takes the X input at timestep T and it's outputs are then added to the next respective hidden layer's inputs.</p>

<pre><code>    a(l) = a(w*x + b + pa)
where l = current timestep
and x = value at current timestep
and w = weights of connections to input layer
and pa = past activation values of hidden layer 
    such that neuron i in layer l uses the output value of neuron i in layer l-1

    y = o(w*a(l-1) + b)
where w = weights of connections to last hidden layer
</code></pre>

<p>But even if I understood this correctly, I don't see the advantage of doing this over simply using past values as inputs to a normal feedforward network (sliding window or whatever it's called).</p>

<p>For example, what is the advantage of using a recurrent neural network for binary addition instead of than training a feedforward network with two output neurons. One for the binary result and the other for the carry? And then take the carry output and plug it back into the feedforward network.</p>

<p>However, I'm not sure how is this different than simply having past values as inputs in a feedforward model.</p>

<p>It seems to me that the more timesteps there are, recurrent neural networks are only a disadvantage over feedforward networks because of vanishing gradient. Which brings me to my second question, from what I understood, LSTM is a solution to the problem of vanishing gradient. But I have no actual grasp of how they work. Furthermore, are they simply better than recurrent neural networks, or are there sacrifices to using a LSTM?</p>