<p>There is also the very recent <strong>BERTScore</strong> metric (arXiv'19, ICLR'20, already almost 90 citations) that does not suffer from the well-known issues of ROUGE and BLEU.</p>
<p>Abstract from the paper:</p>
<blockquote>
<p>We propose BERTScore, an automatic evaluation metric for text
generation. Analogously to common metrics, BERTScore computes a
similarity score for each token in the candidate sentence with each
token in the reference sentence. However, instead of exact matches, we
compute token similarity using contextual embeddings. We evaluate
using the outputs of 363 machine translation and image captioning
systems. BERTScore correlates better with human judgments and provides
stronger model selection performance than existing metrics. Finally,
we use an adversarial paraphrase detection task to show that BERTScore
is more robust to challenging examples when compared to existing
metrics.</p>
</blockquote>
<ul>
<li><p>Paper: <a href="https://arxiv.org/pdf/1904.09675.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1904.09675.pdf</a></p>
</li>
<li><p>Code: <a href="https://github.com/Tiiiger/bert_score" rel="nofollow noreferrer">https://github.com/Tiiiger/bert_score</a></p>
</li>
<li><p>Full reference:</p>
<p>Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. &quot;Bertscore: Evaluating text generation with bert.&quot; arXiv preprint arXiv:1904.09675 (2019).</p>
</li>
</ul>
How do I evaluate a text summarization tool?
<p>I have written a system that summarizes a long document containing thousands of words. Are there any norms on how such a system should be evaluated in the context of a user survey?</p>

<p>In short, is there a metric for evaluating the time that my tool has saved a human? Currently, I was thinking of using the (Time taken to read the original document/Time taken to read the summary) as a way of determining the time saved, but are there better metrics?</p>

<p>Currently, I am asking the user subjective questions about the accuracy of the summary.</p>