<p>TLDR: Each LSTM cell at time t and level l has inputs x(t) and hidden state h(l,t)
In the first layer, the input is the actual sequence input x(t), and previous hidden state h(l, t-1), and in the next layer the input is the hidden state of the corresponding cell in the previous layer h(l-1,t).</p>

<p>From <a href="https://arxiv.org/pdf/1710.02254.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1710.02254.pdf</a>:</p>

<p>To increase the capacity of GRU networks (Hermans and
Schrauwen 2013), recurrent layers can be stacked on top of
each other. 
Since GRU does not have two output states, the same output hidden state h'2
is passed to the next vertical layer. In other words, the h1 of the next layer will be equal to h'2. 
This forces GRU to learn transformations that are useful along depth as well as time.</p>
Understanding multi-layer LSTM
<p>I'm trying to understand and implement multi-layer LSTM. The problem is i don't know how they connect. I'm having two thoughs in mind:</p>

<ol>
<li><p>At each timestep, the hidden state H of the first LSTM will become the input of the second LSTM.</p></li>
<li><p>At each timestep, the hidden state H of the first LSTM will become the initial value for the hidden state of the sencond LSTM, and the input of the first LSTM will become the input for the second LSTM.</p></li>
</ol>

<p>Please help!</p>