title: Tensorflow backprop through rnn ValueError
body: <p>I am using a recurrent neural network in tensorflow with BasicLSTMCells. Basically, I have an input sequence of word ids, I convert each id to word embeddings, pass the word embeddings one at a time through the rnn, and then make a prediction for a single word after reading the whole sequence. My embedding matrix is of dimension V x H where V is the size of my vocabulary and H is the number of hidden units in my rnn. In order to make a prediction for the next word, I multiply my hidden vector by a weight matrix of size H x V and then compute a softmax. With the setup I described, everything seems to work as expected. I'm able to train on some examples and make reasonable predictions.
  However, I've noticed that if I try to use the transpose of the embedding matrix, which will be a matrix of size H x V, instead of a separate matrix for the softmax layer, tensorflow raises a value error claiming that the dimensions of something it's not specifying don't have the same rank. I've verified that the dimensions of my embedding matrix (well, its transpose) are the same as those of the separate softmax matrix I'm creating. Changing just the one line of code from using my embedding matrix vs a separate softmax weight matrix causes the error.
  I created a relatively small program to demonstrate what I'm trying to do and to show what causes the error. I was not able to make the error occur on a smaller network when I tried with just a single hidden layer network. </p>
  
  <pre><code>import sys
  import time
  import tensorflow as tf
  from tensorflow.models.rnn import rnn
  from tensorflow.models.rnn import rnn_cell
  from tensorflow.models.rnn.rnn_cell import BasicLSTMCell
  import numpy as np
  
  INPUT_LENGTH = 17
  BATCH_SIZE = 20
  VOCAB_SIZE = 11
  NUM_EPOCHS = 1000
  HIDDEN_UNITS = 100
  
  class Model(object):
      def __init__(self, is_training):
          initializer = tf.random_uniform_initializer(-1.0, 1.0)
  
          self._target = tf.placeholder(tf.float32, [BATCH_SIZE, VOCAB_SIZE])
          self._input_data=tf.placeholder(tf.int32,[BATCH_SIZE, INPUT_LENGTH])
  
          self.embedding = tf.get_variable("embedding",
                                      [VOCAB_SIZE, HIDDEN_UNITS],
                                      initializer=initializer)
  
          self.inputs = tf.split(1, INPUT_LENGTH,
                            tf.nn.embedding_lookup(self.embedding, self._input_data))
  
          self.inputs2 = [tf.squeeze(input_, [1]) for input_ in self.inputs]
  
          cell = rnn_cell.BasicLSTMCell(num_units=HIDDEN_UNITS)
  
          initial_state = cell.zero_state(BATCH_SIZE, tf.float32)
          outputs, states = rnn.rnn(cell, self.inputs2,
                                    initial_state=initial_state)
  
          self._outputs = outputs[-1]
  
          self.soft_w = tf.get_variable("softmax_w",
                                   [HIDDEN_UNITS, VOCAB_SIZE],
                                   initializer=initializer)
  
  
          prod = tf.matmul(self._outputs, self.soft_w)
  #uncommenting out the following line causes the error
  #        prod = tf.matmul(self._outputs, self.embedding, False, True)
          soft_b = tf.get_variable("softmax_b", [VOCAB_SIZE],
                                   initializer=initializer)
          self._logits = tf.nn.bias_add(prod,soft_b)
          self._loss = tf.nn.softmax_cross_entropy_with_logits(self._logits,
                                                               self._target)
  
          if not is_training:
              return
  
          learning_rate = .010001
          optimizer = tf.train.GradientDescentOptimizer(learning_rate)
          self._train_op = optimizer.minimize(self._loss)
  
      def train(self, sess, inputs, targets):
          t = np.zeros((BATCH_SIZE, VOCAB_SIZE))
          for i, target in enumerate(targets):
              t[i,target] = 1.0
  
          inputs = np.array(inputs)
          inputs = inputs.reshape(BATCH_SIZE,INPUT_LENGTH)
  
          fd = {self._target:t,
                self._input_data:inputs}
          o = sess.run([self._train_op, self._loss, self._outputs, self.embedding, self.soft_w], feed_dict = fd)
          print o[2].shape
          print o[3].shape
          print o[4].shape
          sys.exit()
          return np.mean(o[1])
  
  #this just generates dummy data
  def read_data_rows(count):
      ret = []
      for i in range(count):
          inputs = [4] * INPUT_LENGTH
          output = 1
          ret.append((inputs, output))
      return ret
  
  
  def main():
      start = time.time()
      tf.set_random_seed(1)
  
      print "creating model",time.time()-start
      m = Model(is_training=True)
  
      with tf.Session() as sess:
          print "initializing variables", time.time()-start
          tf.initialize_all_variables().run()
  
          for epoch in range(NUM_EPOCHS):
              train_rows = read_data_rows(100)
              for row_num in range(0, len(train_rows), BATCH_SIZE):
                  qs = []
                  ans = []
                  batch = train_rows[row_num:row_num+BATCH_SIZE]
                  for b in batch:
                      qs.append(b[0])
                      ans.append(b[1])
                  m.train(sess, qs, ans)
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>The error I see is ValueError: Shapes TensorShape([Dimension(100)]) and TensorShape([Dimension(17), Dimension(100)]) must have the same rank
  when uncommenting the line I mentioned above. What is the cause of the error I'm seeing? Why is the embedding matrix not treated the same way as the matrix self.soft_w?</p>