* The paper could have provided a clearer use case for why the NMT model is still necessary. When powerful and large general language models are used for post-editing, why should one use a specialized neural machine translation model to get the translations in the first place? Including GPT-4 translations plus GPT-4 post-editing scores would have been insightful.
* The paper lacks an ablation study explaining why they chose the prompt in this specific way, e.g., few-shot examples for CoT might improve performance.
* The reliance on an external model via API, where it's unclear how the underlying model changes, makes it hard to reproduce the results. There is also a risk of data pollution since translations might already be in the training data of GPT-4. The authors only state that the WMT-22 test data is after the cutoff date of the GPT-4 training data, but they do not say anything about the WMT-20 and WMT-21 datasets that they also use.
* The nature of post-edited translation experiments is only partially done: En-Zh and Zh-En for GPT-4 but not for En-De and De-En for GPT-3.5.
* The title is misleading since the authors also evaluate GPT-3.5.
* The paper lacks an ablation study explaining why they chose the prompt in this specific way, e.g., few-shot examples for CoT might improve performance.