<p>You are probably talking about <a href="https://arxiv.org/pdf/1508.02096.pdf" rel="nofollow noreferrer" title="this very cool paper">this paper</a>. They don't seem to have released their code..</p>

<p>I don't think you would have to "undo the word embedding", because you can do the training simultaneously on two objectives :</p>

<ul>
<li>For each word, one loss for the characters produced, with respect to the true characters of the word. CE loss looks good for this.</li>
<li>One loss for the "word embeddings" produced in the decoder (by the inner LSTM), with respect to the word embeddings produced in the encoder (by the outer LSTM applied on characters). I'm thinking MSE loss for example.</li>
</ul>

<p>(Does it make sense?)</p>

<p>Out of curiosity, did you manage to implement this idea ? I'm going to try it in Pytorch</p>
Tensorflow sequence-to-sequence LSTM within LSTM (nested)
<p>I would like to build an LSTM with a special word embedding, but I have some questions about how this would work. </p>

<p>As you might know, some LSTMs operate on characters, so it is characters in, characters out. I would like to do the same, with an abstraction on words to learn a robust embedding on them with a nested LSTM to be resistant to slight character-level errors. </p>

<p>So, a tiny LSTM would unroll on every letter of a word, then this would create an embedding of the word. Each embedded word in a sentence would then be fed as an input to a higher level LSTM, which would operate on a word level at every time step, rather than on characters. </p>

<p>Questions:
- I cannot find anymore the research paper that talked about that. If you know of what I talk about, I would like to put a name on what I want to do. 
- Does some TensorFlow open-source code already exist for that?
- Else, do you have an idea on how to implement that? The output of the neural network might be harder to deal with, as we would need to undo the word embedding for the training on characters with an output nested LSTM. The whole thing should be trained once as a single unit (workflow: LSTM chars in, LSTM on words, LSTM chars out). </p>

<p>I guess that <code>rnn_cell.MultiRNNCell</code> would stack LSTMs on top of each other rather than nesting them. </p>

<p>Else would you recommend training the embeddings (in and out) as an autoencoder outside the main LSTM ?</p>