<p>I recommend BartScore. Check the <a href="https://github.com/neulab/BARTScore" rel="nofollow noreferrer">Github</a> page and the <a href="https://arxiv.org/abs/2106.11520#:%7E:text=BARTScore%3A%20Evaluating%20Generated%20Text%20as%20Text%20Generation,-Weizhe%20Yuan%2C%20Graham&amp;text=The%20general%20idea%20is%20that,the%20generated%20text%20is%20better." rel="nofollow noreferrer">article</a>. The authors issued also a meta-evaluation on the <a href="http://explainaboard.nlpedia.ai/leaderboard/task-meval/" rel="nofollow noreferrer">ExplainaBoard platform</a>, &quot;which allows to interactively understand the strengths, weaknesses, and complementarity of each metric&quot;. You can find the list of most of the state-of-the-art metrics there.</p>
How do I evaluate a text summarization tool?
<p>I have written a system that summarizes a long document containing thousands of words. Are there any norms on how such a system should be evaluated in the context of a user survey?</p>

<p>In short, is there a metric for evaluating the time that my tool has saved a human? Currently, I was thinking of using the (Time taken to read the original document/Time taken to read the summary) as a way of determining the time saved, but are there better metrics?</p>

<p>Currently, I am asking the user subjective questions about the accuracy of the summary.</p>