<p>As soon as you find the answer, you'll be able to write a nice paper about it since that's kind of an open research question right now. :)</p>

<p>To my best knowledge, your evaluation has to combine syntactic and semantic plausibility of the output, context-coherence, personality consistency and discourse dynamic progression. There's no consensus on how to optimally measure these, but there's plenty of current papers on the topic.</p>

<p>Related introductory read by Liu et al: <a href="https://arxiv.org/abs/1603.08023" rel="nofollow noreferrer">https://arxiv.org/abs/1603.08023</a></p>
Natural language generation evaluation
<p>I was making a natural language generator using LSTM networks but now I am stuck in the part , how to evaluate my output. Suppose i have a input training data-set that consists of a dialogue act representation and the correct output for that particular dialogue act. Now suppose i generate a output sentence y from my LSTM network, so how to evaluate that sentence in comparison to the one in the data-set. I mean is there any way to compare output so that I can use gradient descent to train my weights. </p>