The primary concerns include,
* The necessity of evaluating the degree of personalization is not clear to me.
- According to this paper, I only found three previous research that did personalized summarizers. And all of them utilize the current common metrics to measure performance. It seems these metrics are enough for this task.
- Let's assume the new evaluation metric is necessary. When we have the pairs of user profiles (such as user-expected summaries) and generated summaries for each user, why can we not use the average and variance of current metrics (such as Rouge) to show the degree of personalization? The average presents the performance of the summarizer to generate high-quality summaries, and variance can represent the performance of generating summaries close to each user. It is much easier to evaluate based on current metrics than new ones.
* The new proposed metric is only tested on a single dataset.
* There is no human judgment for this new metric. I notice the authors said, in Limitations, they are trying for the human evaluation. I think it is better to accept the next version with human judgment results.
* The metric is high time-cost due to the eight Jenson-Shannon Divergence calculations.
Besides, the details of $rot()$ were missed in Line 184.
* The new proposed metric is only tested on a single dataset.