of different chain-of-thought prompting methods: including zero-shot-CoT, few-shot-CoT, manual-CoT, and Auto-CoT. The paper conducts case study experiments to look into the limitations of existing methods and proposes improvement directions. Finally, the paper proposes an improved method for Auto-CoT that could achieve a competitive advantage over Manual-CoT.
2. The experiment part is very detailed and comprehensive.
3. The paper is well-organized.
The writing is good and most of the content is very clear to me. Weaknesses/Feedback
1. The writing could be improved.
It would be helpful to draw a table to compare different CoT prompting methods across different dimensions.
How and why shall we make an assumption that “questions of all the wrong demonstrations fall into the same frequent-error cluster”?
Is the selection criteria in section 4.2 reasonable? Namely, why do we not choose questions with more than 60 tokens and rationales with more than 5 reasoning steps?
2. Some experimental details are missing.
The experimental details for Table 1 are not very clear and lack some details. For example, how the manual-CoT examples are built. What is the number of demonstration examples?
The experimental details for the Codex baseline are missing. I am curious about the instructions you used to prompt the Codex model.
3. It would be better to discuss more recent related work.
I understand some work has been released very recently. Since this work is closely related to the paper, it would be nice to include it in the revised paper. Recent work includes but is not limited to: [1] Calibrate Before Use: Improving Few-Shot Performance of Language Models, 2021 [2] Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning, 2022 [3] Complexity-Based Prompting for Multi-Step Reasoning, 2022 [4] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering, 2022 [5] What Makes Pre-trained Language Models Better Zero/Few-shot Learners?, 2022
4. Is it possible to apply Auto-CoT to sample the examples with manually designed CoT?
3. The paper is well-organized. The writing is good and most of the content is very clear to me. Weaknesses/Feedback 1. The writing could be improved. It would be helpful to draw a table to compare different CoT prompting methods across different dimensions. How and why shall we make an assumption that “questions of all the wrong demonstrations fall into the same frequent-error cluster”? Is the selection criteria in section 4.2 reasonable? Namely, why do we not choose questions with more than 60 tokens and rationales with more than 5 reasoning steps?