1. Not clear if the contribution of the paper are sufficient for a long *ACL paper. By tightening the writing and removing unnecessary details, I suspect the paper will make a nice short paper, but in its current form, the paper lacks sufficient novelty. 2. The writing is difficult to follow in many places and can be simplified.
1. Line 360-367 are occupying too much space than needed. 2. It was not clear to me that Vikidia is the new dataset that was introduced by the paper until I read the last section :) 3. Too many metrics used for evaluation. While I commend the paper’s thoroughness by using different metrics for evaluation, I believe in this case the multiple metrics create more confusion than clarity in understanding the results. I recommend using the strictest metric (such as RA) because it will clearly highlight the differences in performance. Also consider marking the best results in each column/row using boldface text. 4. I suspect that other evaluation metrics NDCG, SRRR, KTCC are unable to resolve the differences between NPRM and the baselines in some cases. For e.g., Based on the extremely large values (>0.99) for all approaches in Table 4, I doubt the difference between NPRM’s 0.995 and Glove+SVMRank 0.992 for Avg. SRR on NewsEla-EN is statistically significant. 5. I did not understand the utility of presenting results in Table 2 and Table 3. Why not simplify the presentation by selecting the best regression based and classification based approaches for each evaluation dataset and compare them against NPRM in Table 4 itself? 6. From my understanding, RA is the strictest evaluation metric, and NPRM performs worse on RA when compared to the baselines (Table 4) where simpler approaches fare better. 7. I appreciate the paper foreseeing the limitations of the proposed NPRM approach. However, I find the discussion of the first limitation somewhat incomplete and ending abruptly. The last sentence has the tone of “despite the weaknesses, NPRM is useful'' but it does not flesh out why it’s useful. 8. I found ln616-632 excessively detailed for a conclusion paragraph. Maybe simply state that better metrics are needed for ARA evaluation? Such detailed discussion is better suited for Sec 4.4 9. Why was a classification based model not used for the zero shot experiments in Table 5 and Table 6? These results in my opinion are the strongest aspect of the paper, and should be as thorough as the rest of the results. 10. Line 559: “lower performance on Vikidia-Fr compared to Newsela-Es …” – Why? These are different languages after all, so isn’t the performance difference in-comparable?
5. I did not understand the utility of presenting results in Table 2 and Table 3. Why not simplify the presentation by selecting the best regression based and classification based approaches for each evaluation dataset and compare them against NPRM in Table 4 itself?