title: feeding data into an LSTM - Tensorflow RNN PTB tutorial
body: <p>
  I am trying to feed data into an LSTM.  I am reviewing the code from <a href="https://www.tensorflow.org/tutorials/recurrent" rel="nofollow noreferrer">Tensorflow's RNN tutorial here</a>.</p>
  
  <p>The segment of code of interest is from the <code>reader.py</code> file of the tutorial, in particular the <code>ptb_producer</code> function, which ouputs the <code>X</code> and <code>Y</code> that is used by the LSTM.</p>
  
  <p><code>raw_data</code> is a list of indexes of words from the <code>ptb.train.txt</code> file. The length of <code>raw_data</code> is 929,589. <code>batch_size</code> is 20, <code>num_steps</code> is 35. Both <code>batch_size</code> and <code>num_steps</code> are based on the <code>LARGEconfig</code> which feeds the data to an LSTM. </p>
  
  <p>I have walked through the code (and added comments for what I've printed) and I understand it up till <code>tf.strided_slice</code>.  From the reshape, we have a matrix of indexes of shape <code>(20, 46497)</code>.</p>
  
  <p>Strided slice in the first iteration of <code>i</code>, tries to take data from <code>[0, i * num_steps + 1]</code> which is <code>[0,1*35+1]</code> till <code>[batch_size, (i + 1) * num_steps + 1]</code> which is <code>[20, (1+1)*35+1]</code>.</p>
  
  <p>Two questions:</p>
  
  <ol>
  <li><p>where in the matrix is <code>[0,1*35+1]</code> and <code>[20, (1+1)*35+1]</code>?  What spots in the <code>(20, 46497)</code> the begin and end in strided_slice is trying to access?</p></li>
  <li><p>It seems like EVERY iteration of <code>i</code>, will take in data from 0?  the very start of the data matrix <code>(20, 46497)</code>?</p></li>
  </ol>
  
  <p>I guess what I am not understanding is how you would feed data into an LSTM, given the batch size and the <code>num_steps</code> (sequence length). </p>
  
  <p>I have read <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">colahs blog on LSTM</a> and <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow noreferrer">Karpathy's blog on RNN</a> which helps greatly in the understanding of LSTMs, but don't seem to address the exact mechanics of getting data into an LSTM. (maybe I missed something?)</p>
  
  <pre class="lang-py prettyprint-override"><code>def ptb_producer(raw_data, batch_size, num_steps, name=None):
    """Iterate on the raw PTB data.
    This chunks up raw_data into batches of examples and returns Tensors that
    are drawn from these batches.
    Args:
      raw_data: one of the raw data outputs from ptb_raw_data.
      batch_size: int, the batch size.
      num_steps: int, the number of unrolls.
      name: the name of this operation (optional).
    Returns:
      A pair of Tensors, each shaped [batch_size, num_steps]. The second 
      element of the tuple is the same data time-shifted to the right by one.
    Raises:
      tf.errors.InvalidArgumentError: if batch_size or num_steps are too high.
    """
    with tf.name_scope(name, "PTBProducer", [raw_data, batch_size, num_steps]):
      raw_data = tf.convert_to_tensor(raw_data, name="raw_data", dtype=tf.int32)
      data_len = tf.size(raw_data)   # prints 929,589
      batch_len = data_len // batch_size   # prints 46,497
      data = tf.reshape(raw_data[0 : batch_size * batch_len],
                        [batch_size, batch_len])      
      #this truncates raw data to a multiple of batch_size=20, 
      #then reshapes to [20, 46497].  prints (20,?)
  
      epoch_size = (batch_len - 1) // num_steps   #prints 1327 (number of epoches)
      assertion = tf.assert_positive(
          epoch_size,
          message="epoch_size == 0, decrease batch_size or num_steps")
      with tf.control_dependencies([assertion]):
        epoch_size = tf.identity(epoch_size, name="epoch_size")
  
      i = tf.train.range_input_producer(epoch_size, shuffle=False).dequeue()
      #for each of the 1327 epoches
      x = tf.strided_slice(data, [0, i * num_steps], [batch_size, (i + 1) * num_steps])   # prints (?, ?) 
      x.set_shape([batch_size, num_steps])  #prints (20,35) 
      y = tf.strided_slice(data, [0, i * num_steps + 1], [batch_size, (i + 1) * num_steps + 1])
      y.set_shape([batch_size, num_steps])
      return x, y
  </code></pre>