<p>I found an answer to my own question in Section 3.2 of <a href="https://arxiv.org/abs/1710.07654" rel="nofollow noreferrer">the paper</a> (Deep Voice 3).
So, they trained both of phoneme-based model and character-based model, using phoneme inputs mainly except that character-based model is used if words cannot be converted to their phoneme representations.</p>
Fixing error output from seq2seq model
<p>I want to ask you how we can effectively re-train a trained seq2seq model to remove/mitigate a specific observed error output. I'm going to give an example about Speech Synthesis, but any idea from different domains, such as Machine Translation and Speech Recognition, using seq2seq model will be appreciated.</p>

<p>I learned the basics of seq2seq with attention model, especially for Speech Synthesis such as <a href="https://github.com/Rayhane-mamah/Tacotron-2" rel="nofollow noreferrer">Tacotron-2</a>.
Using a distributed well-trained model showed me how naturally our computer could speak with the seq2seq (end-to-end) model (you can listen to some audio samples <a href="https://r9y9.github.io/blog/2018/05/20/tacotron2/" rel="nofollow noreferrer">here</a>). But still, the model fails to read some words properly, e.g., it fails to read "obey [əˈbā]" in multiple ways like [əˈbī] and [əˈbē].</p>

<p>The reason is obvious because the word "obey" appears too little, only three times out of 225,715 words, in our dataset (<a href="https://keithito.com/LJ-Speech-Dataset/" rel="nofollow noreferrer">LJ Speech</a>), and the model had no luck.</p>

<p>So, how can we re-train the model to overcome the error? Adding extra audio clips containing the "obey" pronunciation sounds impractical, but reusing the three audio clips has the danger of overfitting. And also, I suppose we use a well-trained model and "simply training more" is not an effective solution.</p>

<p>Now, this is one of the drawbacks of seq2seq model, which is not talked much. The model successfully simplified the pipelines of the traditional models, e.g., for Speech Synthesis, it replaced an acoustic model and a text analysis frontend etc by a single neural network. But we lost the controllability of our model at all. It's impossible to make the system read in a specific way.</p>

<p>Again, if you use a seq2seq model in any field and get an undesirable output, how do you fix that? Is there a data-scientific workaround to this problem, or maybe a cutting-edge Neural Network mechanism to gain more controllability in seq2seq model?</p>

<p>Thanks.</p>