- A key part – the rules to derive error types – are not described.
- The classifier evaluation lacks a thorough error analysis and based upon that it lacks directions of future work on how to improve the classifier.
- The evaluation was only performed for English and it is unclear how difficult it would be to use the approach on another language.
Classifier and Classifier Evaluation ==================================== It is unclear on what basis the error categories were devised. Are they based on previous work?
Although the approach in general is independent of the alignment algorithm, the rules are probably not, but the authors don't provide details on that. The error categories are a major part of the paper and the reader should at least get a glimpse of how a rule to assign an error type looks like.
Unfortunately, the paper does not apply the proposed evaluation on languages other than English. It also does not elaborate on what changes would be necessary to run the classifier on other languages. I assume that the rules used for determining edit boundaries as well as for determining the error tags depend on the language/the pre-processing pipeline to a certain extent and therefore need to be adapted. Also, the error categories might need to be changed. The authors do not provide any detail on the rules for assigning error categories (how many are there overall/per error type? how complex are they?) to estimate the effort necessary to use the approach on another language.
The error spans computed in the pre-processing step seem to be inherently continuous (which is also the case with the M2 scorer), which is problematic since there are errors which can only be tagged accurately when the error span is discontinuous. In German, for example, verbs with separable prefixes are separated from each other in the main clause: [1st constituent] [verb] [other constituents] [verb prefix]. Would the classifier be able to tag discontinuous edit spans?
The authors write that all human judges rated at least 95\% of the automatically assigned error tags as appropriate "despite the degree of noise introduced by automatic edit extraction" (295). I would be more cautious with this judgment since the raters might also have been more forgiving when the boundaries were noisy. In addition, they were not asked to select a tag without knowing the system output but could in case of noisy boundaries be more biased towards the system output. Additionally, there was no rating option between "Bad (Not Appropriate)" and "Appropriate", which might also have led raters to select "Appropriate" over "Bad". To make the evaluation more sound, the authors should also evaluate how the human judges rate the classifier output if the boundaries were manually created, i.e. without the noise introduced by faulty boundaries.
The classifier evaluation lacks a thorough error analysis. It is only mentioned that "Bad" is usually traced back to a wrong POS tag. Questions I'd like to see addressed: When did raters select "Bad", when "Appropriate"? Does the rating by experts point at possibilities to improve the classifier?
Gold Reference vs. Auto Reference ================================= It is unclear on what data the significance test was performed exactly. Did you test on the F0.5 scores? If so, I don't think this is a good idea since it is a derived measure with weak discriminative power (the performance in terms of recall an precision can be totally different but have the same F0.5 score). Also, at the beginning of Section 4.1 the authors refer to the mismatch between automatic and reference in terms of alignment and classification but as far as I can tell, the comparison between gold and reference is only in terms of boundaries and not in terms of classification.
Error Type Evaluation ===================== I do not think it is surprising that 5 teams (~line 473) failed to correct any unnecessary token error. For at least two of the systems there is a straightforward explanation why they cannot handle superfluous words. The most obvious is UFC: Their rule-base approach works on POS tags (Ng et al., 2014) and it is just not possible to determine superfluous words based on POS alone. Rozovskaya & Roth (2016) provide an explanation why AMU performs poorly on superfluous words.
The authors do not analyze or comment the results in Table 6 with respect to whether the systems were designed to handle the error type. For some error types, there is a straight-forward mapping between error type in the gold standard and in the auto reference, for example for word order error. It remains unclear whether the systems failed completely on specific error types or were just not designed to correct them (CUUI for example is reported with precision+recall=0.0, although it does not target word order errors). In the CUUI case (and there are probably similar cases), this also points at an error in the classification which is neither analyzed nor discussed.
Please report also raw values for TP, FP, TN, FN in the appendix for Table 6. This makes it easier to compare the systems using other measures. Also, it seems that for some error types and systems the results in Table 6 are based only on a few instances. This would also be made clear when reporting the raw values.
Your write "All but 2 teams (IITB and IPN) achieved the best score in at least 1 category, which suggests that different approaches to GEC complement different error types." ( 606) It would be nice to mention here that this is in line with previous research.
Multi-token error analysis is helpful for future work but the result needs more interpretation: Some systems are probably inherently unable to correct such errors but none of the systems were trained on a parallel corpus of learner data and fluent (in the sense of Sakaguchi et al, 2016) corrections.
Other ===== - The authors should have mentioned that for some of the GEC approaches, it was not impossible before to provide error annotations, e.g. systems with submodules for one error type each.
Admittedly, the system would need to be adapted to include the submodule responsible for a change in the system output. Still, the proposed approach enables to compare GEC systems for which producing an error tagged output is not straightforward to other systems in a unified way.
- References: Some titles lack capitalizations. URL for Sakaguchi et al. (2016) needs to be wrapped. Page information is missing for Efron and Tibshirani (1993).
Author response =============== I agree that your approach is not "fatally flawed" and I think this review actually points out quite some positive aspects. The approach is good, but the paper is not ready.
The basis for the paper are the rules for classifying errors and the lack of description is a major factor. This is not just a matter about additional examples. If the rules are not seen as a one-off implementation, they need to be described to be replicable or to adapt them.
Generalization to other languages should not be an afterthought. It would be serious limitation if the approach only worked on one language by design. Even if you don't perform an adaption for other languages, your approach should be transparent enough for others to estimate how much work such an adaptation would be and how well it could reasonably work. Just stating that most research is targeted at ESL only reinforces the problem.
You write that the error types certain systems tackle would be "usually obvious from the tables". I don't think it is as simple as that -- see the CUUI example mentioned above as well as the unnecessary token errors. There are five systems that don't correct them (Table 5) and it should therefore be obvious that they did not try to tackle them. However, in the paper you write that "There is also no obvious explanation as to why these teams had difficulty with this error type".
- A key part – the rules to derive error types – are not described.