title: Training a RNN to output word2vec embedding instead of logits
body: <p>Traditionally it seems that RNNs use logits to predict next time step in the sequence. In my case I need the RNN to output a word2vec (50 depth) vector prediction. This means that the cost function has be based off 2 vectors: <code>Y</code> the actual vector of the next word in the series and <code>Y_hat</code>, the network prediction.</p>
  
  <p>I've tried using a <code>cosine distance</code> cost function but the network does not seem to learn (I've let it run other 10 hours on a AWS P3 and the cost is always around 0.7)</p>
  
  <p>Is such a model possible at all ? If so what cost function should be used ?</p>
  
  <p>Cosine distance in TF:</p>
  
  <pre><code>cosine_distance = tf.losses.cosine_distance(tf.nn.l2_normalize(outputs, 2), tf.nn.l2_normalize(targets, 2), axis=2)
  </code></pre>
  
  <p><strong>Update:</strong>
  I am trying to predict a word2vec so during sampling I could pick next word based on the closest neighbors of the predicted vector. </p>