<p>If you want to have have multiple layers that pass the information backward or forward in time, there are two ways how to design this. Assume the forward layer consists of two layers F1, F2 and the backword layer consists of two layers B1, B2.</p>

<p>If you use <code>tf.nn.bidirectional_dynamic_rnn</code> the model will look like this (time flows from left to right):</p>

<p><a href="https://i.stack.imgur.com/wrpA9.png" rel="noreferrer"><img src="https://i.stack.imgur.com/wrpA9.png" alt="enter image description here"></a></p>

<p>If you use <code>tf.contrib.rnn.stack_bidirectional_dynamic_rnn</code> the model will look like this:</p>

<p><a href="https://i.stack.imgur.com/qODfk.png" rel="noreferrer"><img src="https://i.stack.imgur.com/qODfk.png" alt="enter image description here"></a></p>

<p>Here the black dot between first and second layer represents a concatentation. I.e., the outputs of the forward and backward cells are concatenated together and fed to the backward and forward layers of the next upper layer. This means both F2 and B2 receive exactly the same input and there is an explicit connection between backward and forward layers. In <a href="https://arxiv.org/pdf/1303.5778.pdf" rel="noreferrer">"Speech Recognition with Deep Recurrent Neural Networks"</a> Graves et al. summarize this as follows:</p>

<blockquote>
  <p>... every hidden layer receives input from both the
  forward and backward layers at the level below.</p>
</blockquote>

<p>This connection only happens implicitly in the unstacked BiRNN (first image), namely when mapping back to the output. The stacked BiRNN usually performed better for my purposes, but I guess that depends on your problem setting. But for sure it is worthwile to try it out!</p>

<p><strong>EDIT</strong></p>

<p>In response to your comment: I base my answer on the documentation of the function <code>tf.contrib.rnn.stack_bidirectional_dynamic_rnn</code> which says:</p>

<blockquote>
  <p>Stacks several bidirectional rnn layers. The combined forward and
  backward  layer outputs are used as input of the next layer.
  tf.bidirectional_rnn does not allow to share forward and backward
  information between layers.</p>
</blockquote>

<p>Also, I looked at the implementation available under <a href="https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/contrib/rnn/python/ops/rnn.py" rel="noreferrer">this link</a>. </p>
Difference between bidirectional_dynamic_rnn and stack_bidirectional_dynamic_rnn in Tensorflow
<p>I am building a dynamic RNN network with stacking multiple LSTMs. I see there are 2 options</p>

<pre><code># cells_fw and cells_bw are list of cells eg LSTM cells
stacked_cell_fw = tf.contrib.rnn.MultiRNNCell(cells_fw)
stacked_cell_bw = tf.contrib.rnn.MultiRNNCell(cells_bw)

output = tf.nn.bidirectional_dynamic_rnn(
          stacked_cell_fw, stacked_cell_bw, INPUT,
          sequence_length=LENGTHS, dtype=tf.float32)
</code></pre>

<p>vs </p>

<pre><code>output = tf.contrib.rnn.stack_bidirectional_dynamic_rnn(cells_fw, cells_bw, INPUT,
sequence_length=LENGTHS, dtype=tf.float32)
</code></pre>

<p>What is the difference between the 2 approaches and is one better than the other?</p>