title: What is a Recurrent Neural Network, what is a Long Short Term Memory (LSTM) network, and is it always better?
body: <p>First, let me apologize for cramming three questions in that title. I'm not sure what better way is there.</p>
  
  <p>I'll get right to it. I think I understand feedforward neural networks pretty well. </p>
  
  <p>But LSTM really escapes me, and I feel maybe this is because I don't have a very good grasp of Recurrent neural networks in general. I have went through Hinton's and Andrew Ng's course on Coursera. A lot of it still doesn't make sense to me.</p>
  
  <p>From what I understood, recurrent neural networks are different from feedforward neural networks in that past values influence the next prediction. Recurrent neural network are generally used for sequences.</p>
  
  <p>The example I saw of recurrent neural network was binary addition.</p>
  
  <pre><code>    010
  +   011
  </code></pre>
  
  <p>A recurrent neural network would take the right most 0 and 1 first, output a 1. Then take the 1,1 next, output a zero, and carry the 1. Take the next 0,0 and output a 1 because it carried the 1 from last calculation. Where does it store this 1? In feed forward networks the result is basically:</p>
  
  <pre><code>    y = a(w*x + b)
  where w = weights of connections to previous layer
  and x = activation values of previous layer or inputs
  </code></pre>
  
  <p>How is a recurrent neural network calculated? I am probably wrong but from what I understood, recurrent neural networks are pretty much feedforward neural network with T hidden layers, T being number of timesteps. And each hidden layer takes the X input at timestep T and it's outputs are then added to the next respective hidden layer's inputs.</p>
  
  <pre><code>    a(l) = a(w*x + b + pa)
  where l = current timestep
  and x = value at current timestep
  and w = weights of connections to input layer
  and pa = past activation values of hidden layer 
      such that neuron i in layer l uses the output value of neuron i in layer l-1
  
      y = o(w*a(l-1) + b)
  where w = weights of connections to last hidden layer
  </code></pre>
  
  <p>But even if I understood this correctly, I don't see the advantage of doing this over simply using past values as inputs to a normal feedforward network (sliding window or whatever it's called).</p>
  
  <p>For example, what is the advantage of using a recurrent neural network for binary addition instead of than training a feedforward network with two output neurons. One for the binary result and the other for the carry? And then take the carry output and plug it back into the feedforward network.</p>
  
  <p>However, I'm not sure how is this different than simply having past values as inputs in a feedforward model.</p>
  
  <p>It seems to me that the more timesteps there are, recurrent neural networks are only a disadvantage over feedforward networks because of vanishing gradient. Which brings me to my second question, from what I understood, LSTM is a solution to the problem of vanishing gradient. But I have no actual grasp of how they work. Furthermore, are they simply better than recurrent neural networks, or are there sacrifices to using a LSTM?</p>