<p>One problem with a Tensorflow placeholder is that you can only feed it with a Python list or Numpy array (I think). So you can't save the state between runs in tuples of LSTMStateTuple. </p>

<p>I solved this by saving the state in a tensor like this</p>

<p><code>initial_state = np.zeros((num_layers, 2, batch_size, state_size))</code></p>

<p>You have two components in an LSTM layer, the <strong>cell state</strong> and <strong>hidden state</strong>, thats what the "2" comes from. (this article is great: <a href="https://arxiv.org/pdf/1506.00019.pdf" rel="noreferrer">https://arxiv.org/pdf/1506.00019.pdf</a>)</p>

<p>When building the graph you unpack and create the tuple state like this:</p>

<pre><code>state_placeholder = tf.placeholder(tf.float32, [num_layers, 2, batch_size, state_size])
l = tf.unpack(state_placeholder, axis=0)
rnn_tuple_state = tuple(
         [tf.nn.rnn_cell.LSTMStateTuple(l[idx][0],l[idx][1])
          for idx in range(num_layers)]
)
</code></pre>

<p>Then you get the new state the usual way</p>

<pre><code>cell = tf.nn.rnn_cell.LSTMCell(state_size, state_is_tuple=True)
cell = tf.nn.rnn_cell.MultiRNNCell([cell] * num_layers, state_is_tuple=True)

outputs, state = tf.nn.dynamic_rnn(cell, series_batch_input, initial_state=rnn_tuple_state)
</code></pre>

<p>It shouldn't be like this... perhaps they are working on a solution.</p>
How do I set TensorFlow RNN state when state_is_tuple=True?
<p>I have written an <a href="https://github.com/wpm/tfrnnlm" rel="noreferrer">RNN language model using TensorFlow</a>. The model is implemented as an <code>RNN</code> class. The graph structure is built in the constructor, while <code>RNN.train</code> and <code>RNN.test</code> methods run it.</p>

<p>I want to be able to reset the RNN state when I move to a new document in the training set, or when I want to run a validation set during training. I do this by managing the state inside the training loop, passing it into the graph via a feed dictionary.</p>

<p>In the constructor I define the the RNN like so</p>

<pre><code>    cell = tf.nn.rnn_cell.LSTMCell(hidden_units)
    rnn_layers = tf.nn.rnn_cell.MultiRNNCell([cell] * layers)
    self.reset_state = rnn_layers.zero_state(batch_size, dtype=tf.float32)
    self.state = tf.placeholder(tf.float32, self.reset_state.get_shape(), "state")
    self.outputs, self.next_state = tf.nn.dynamic_rnn(rnn_layers, self.embedded_input, time_major=True,
                                                  initial_state=self.state)
</code></pre>

<p>The training loop looks like this</p>

<pre><code> for document in document:
     state = session.run(self.reset_state)
     for x, y in document:
          _, state = session.run([self.train_step, self.next_state], 
                                 feed_dict={self.x:x, self.y:y, self.state:state})
</code></pre>

<p><code>x</code> and <code>y</code> are batches of training data in a document. The idea is that I pass the latest state along after each batch, except when I start a new document, when I zero out the state by running <code>self.reset_state</code>.</p>

<p>This all works.  Now I want to change my RNN to use the recommended <code>state_is_tuple=True</code>. However, I don't know how to pass the more complicated LSTM state object via a feed dictionary. Also I don't know what arguments to pass to the <code>self.state = tf.placeholder(...)</code> line in my constructor.</p>

<p>What is the correct strategy here? There still isn't much example code or documentation for <code>dynamic_rnn</code> available.</p>

<hr>

<p>TensorFlow issues <a href="https://github.com/tensorflow/tensorflow/issues/2695" rel="noreferrer">2695</a> and <a href="https://github.com/tensorflow/tensorflow/issues/2838" rel="noreferrer">2838</a> appear relevant.</p>

<p>A <a href="http://www.wildml.com/2016/08/rnns-in-tensorflow-a-practical-guide-and-undocumented-features/" rel="noreferrer">blog post</a> on WILDML addresses these issues but doesn't directly spell out the answer.</p>

<p>See also <a href="https://stackoverflow.com/questions/38241410/tensorflow-remember-lstm-state-for-next-batch-stateful-lstm">TensorFlow: Remember LSTM state for next batch (stateful LSTM)</a>.</p>