title: language modeling in tensorflow - how to tie embedding and softmax weights
body: <p>As suggested by recent language modeling papers, I want to use weight tying in my RNN language model. That is, I want to share the weights between the embedding and softmax layer. However, I am not sure how this can be done in TensorFlow.</p>
  
  <p>My network receives inputs of shape <code>(batch_size, sequence_length)</code>. The embedding matrix has shape <code>(vocab_size, embedding_size)</code> and is created as follows (I am using pre-trained word2vec embeddings):</p>
  
  <pre><code>        with tf.variable_scope('embedding'):
              self.embedding_matrix = tf.Variable(tf.constant(0.0, shape=[self.vocab_size, self.embd_size]), trainable=False, name='embedding')
              self.embed_placeholder = tf.placeholder(tf.float32, [self.vocab_size, self.embd_size])
              self.embed_init = self.embedding_matrix.assign(self.embed_placeholder)
  </code></pre>
  
  <p>The logits are computed as follows:</p>
  
  <pre><code>            output, self.final_state = tf.nn.dynamic_rnn(
                  cell,
                  inputs=self.inputs,
                  initial_state=self.init_state)
  
              self.output_flat = tf.reshape(output, [-1, cell.output_size])
              softmax_w = tf.get_variable("softmax_w", [self.n_hidden, self.vocab_size], dtype=tf.float32)
  
              softmax_b = tf.get_variable("softmax_b", [self.vocab_size], dtype=tf.float32)
              logits = tf.nn.xw_plus_b(self.output_flat, softmax_w, softmax_b)
              # Reshape logits to be a 3-D tensor
              self.logits = tf.reshape(logits, [self.batch_size, self.seq_length, self.vocab_size])
  </code></pre>
  
  <p>My questions are:</p>
  
  <ol>
  <li>The matrix that has to be changed to using the embeddings weights is <code>softmax_w</code>, correct?</li>
  <li><code>softmax_w</code> has shape <code>(n_hidden, vocab_size)</code>. How does that fit the size of the embedding matrix? Or do I have to ensure that n_hidden = embedding_size?</li>
  <li>How can I reuse the embedding weights in TensorFlow? I know that I have to use <code>reuse=True</code> in the variable_scope.</li>
  </ol>