* The plan-based method requires manually designing a plan based on the ground truth in advance, which is unrealistic in real-world scenarios. The learned plan methods are not comparable to the methods with pre-defined plans based on Table 2. It indicates that the proposed method may be difficult to generalize to a new dataset without the ground truth summary.
* The novelty of the proposed method is limited. The most effective part is the manually designed plan. Based on that, they should also discuss some plan-based /outline-based prompt studies, such as [1-3].
* Some experimental details are missing. For example,
* The detailed information of the proposed new data sets. For example, the size, average document length, average summary length, average citation number, training/validation/testing split, and so on.
* How to choose the X (sentence number) and Y (words) in plan-based methods?
* What generation configuration is used in LLaMA-2, ChatGPT-3.5, and GPT-4? For example, the greedy decoding or the sampling with temperature.
* How was the human evaluation (In Section 5) conducted? The number of annotators, the inner agreement among annotators, the average compensation, the working hours, and the procedure of annotation should be described in detail.
* How are Avg Rating, Avg Win Rate, and Coverage in Table 8 calculated?
[1] Re3: Generating Longer Stories With Recursive Reprompting and Revision
[2] Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
[3] Self-planning Code Generation with Large Language Models
* The plan-based method requires manually designing a plan based on the ground truth in advance, which is unrealistic in real-world scenarios. The learned plan methods are not comparable to the methods with pre-defined plans based on Table 2. It indicates that the proposed method may be difficult to generalize to a new dataset without the ground truth summary.