<p>Recurrent Layers are like Feed Forward Neural Networks with a Feedback Loop in it. They just pass on the useful information from the past to present.</p>

<p>A decent explanation is at: <a href="https://kevinzakka.github.io/2017/07/20/rnn/" rel="nofollow noreferrer">https://kevinzakka.github.io/2017/07/20/rnn/</a></p>

<p>And coming to Adding More layers to RNN you can find the details for that Deep RNNs in the <a href="https://arxiv.org/pdf/1312.6026.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1312.6026.pdf</a></p>

<p>The paper says that Deep RNNs are better than conventional RNNs</p>
Recurrent neural networks as feature extraction
<p>For convolutional networks, one can view the convolutional part (convolutional, max-pooling etc) as feature extraction which then gets feed into feedforward networks which does the classifying (more or less).</p>

<p>Is the same true for recurrent networks (RNN, LSTM etc), i.e. the recurrent layers creates a represenation of the data/features which then gets feed into a feed-forward layers?</p>

<p>I was think in terms of sentiment analysis, i.e. "sequence to one" model. Do you think that having one recurrent layer + one feed-forward layer would outperform only one recurrent layer network?</p>