This paper introduces the NLP-TEA 2015 shared task for Chinese grammatical error diagnosis. We describe the task, data preparation, performance metrics, and evaluation results. The hope is that such an evaluation campaign may produce more advanced Chinese grammatical error diagnosis techniques. All data sets with gold standards and evaluation tools are publicly available for research purposes.
Introduction Human language technologies for English grammatical error correction have attracted more attention in recent years (Ng et al., 2013; . In contrast to the plethora of research related to develop NLP tools for learners of English as a foreign language, relatively few studies have focused on detecting and correcting grammatical errors for use by learners of Chinese as a foreign language (CFL). A classifier has been designed to detect word-ordering errors in Chinese sentences (Yu and Chen, 2012) . A ranking SVMbased model has been further explored to suggest corrections for word-ordering errors (Cheng et al., 2014) . Relative positioning and parse template language models have been proposed to detect Chinese grammatical errors written by US learners (Wu et al., 2010) . A penalized probabilistic first-order inductive learning algorithm has been presented for Chinese grammatical error diagnosis (Chang et al. 2012) . A set of linguistic rules with syntactic information was manually crafted to detect CFL grammatical errors (Lee et al., 2013) . A sentence judgment system has been further developed to integrate both rule-based linguistic analysis and n-gram statistical learning for grammatical error detection . The ICCE-2014 workshop on Natural Language Processing Techniques for Educational Applications (NLP-TEA) organized a shared task on CFL grammatical error diagnosis . Due to the greater challenge in identifying grammatical errors in CFL leaners' written sentences, the NLP-TEA 2015 shared task features a Chinese Grammatical Error Diagnosis (CGED) task, providing an evaluation platform for the development and implementation of NLP tools for computer-assisted Chinese learning. The developed system should identify whether a given sentence contains grammatical errors, identify the error types, and indicate the range of occurred errors. This paper gives an overview of this shared task. The rest of this article is organized as follows. Section 2 provides the details of the designed task. Section 3 introduces the developed data sets. Section 4 proposes evaluation metrics. Section 5 presents the results of participant approaches for performance comparison. Section 6 summarizes the findings and offers futures research directions. Task Description The goal of this shared task is to develop NLP tools for identifying the grammatical errors in sentences written by the CFL learners. Four PADS error types are included in the target modification taxonomy, that is, mis-ordering (Permutation), redundancy (Addition), omission (Deletion), and mis-selection (Substitution). For the sake of simplicity, the input sentence is selected to contain one defined error types. The developed tool is expected to identify the error types and its position at which it occurs in the sentence. The input instance is given a unique sentence number sid. If the inputs contain no grammatical errors, the tool should return "sid, correct". If an input sentence contains a grammatical error, the output format should be a quadruple of "sid, start_off, end_off, error_type", where "start_off" and "end_off" respectively denote the characters at which the grammatical error starts and ends, where each character or punctuation mark occupies 1 space for counting positions. "Error_type" represents one defined error type in terms of "Redundant," "Missing," "Selection," and "Disorder". Examples are shown as follows. • Example 1 Input: (sid=B2-0080) 他是我的以前的室友 Output: B2-0080, 4, 4, Redundant • Example 2 Input: (sid=A2-0017) 那電影是機器人的故事 Output: A2-0017, 2, 2, Missing • Example 3 Input: (sid=A2-0017) 那部電影是機器人的故事 Output: A2-0017, correct • Example 4 Input: (sid=B1-1193) 吳先生是修理腳踏車的拿手 Output: B1-1193, 11, 12, Selection • Example 5 Input: (sid=B2-2292) 所 以 我 不 會 讓 失 望 她 Output: B2-2292, 7, 9, Disorder The character "的" is a redundant character in Ex. 1. There is a missing character between "那" and "電影" in Ex. 2, and a missed character "部" is shown in the correct sentence in Ex. 3. In Ex. 4, "拿手" is a wrong word. One of correct words may be "好手". "失望她" is a word ordering error in Ex. 5. The correct order should be "她失 望". Data Preparation The learner corpus used in our task was collected from the essay section of the computer-based Test of Chinese as a Foreign Language (TOCFL), administered in Taiwan. Native Chinese speakers were trained to manually annotate grammatical errors and provide corrections corresponding to each error. The essays were then split into three sets as follows. (1) Training Set: This set included 2,205 selected sentences with annotated grammatical errors and their corresponding corrections. Each sentence is represented in SGML format as shown in Fig. 1 . Error types were categorized as redundant (430 instances), missing (620), selection (849), and disorder (306). All sentences in this set were collected to use for training the grammatical diagnostic tools. <DOC> <SENTENCE id="B1-1120"> 我的中文進步了非常快 </SENTENCE> <MISTAKE start_off="7" end_off="7"> <TYPE> Selection </TYPE> <CORRECTION> 我的中文進步得非常快 </CORRECTION> </MISTAKE> </DOC> Figure 1. An sentence denoted in SGML format (2) Dryrun Set: A total of 55 sentences were distributed to participants to allow them familiarize themselves with the final testing process. Each participant was allowed to submit several runs generated using different models with different parameter settings of their developed tools. In addition, to ensure the submitted results could be correctly evaluated, participants were allowed to fine-tune their developed models in the dryrun phase. The purpose of dryrun is to validate the submitted output format only, and no dryrun outcomes were considered in the official evaluation (3) Test Set: This set consists of 1,000 testing sentences. Half of these sentences contained no grammatical errors, while the other half included a single defined grammatical error: redundant (132 instances), missing (126), selection (110), and disorder (132). The evaluation was conducted as an open test. In addition to the data sets provided, registered research teams were allowed to employ any linguistic and computational resources to identify the grammatical errors. Table 1 shows the confusion matrix used for performance evaluation. In the matrix, TP (True Positive) is the number of sentences with grammatical errors that are correctly identified by the developed tool; FP (False Positive) is the number of sentences in which non-existent grammatical errors are identified; TN (True Negative) is the number of sentences without grammatical errors that are correctly identified as such; FN (False Negative) is the number of sentences with grammatical errors for which no errors are identified. Performance Metrics The criteria for judging correctness are determined at three levels as follows. (1) Detection level: binary classification of a given sentence, that is, correct or incorrect should be completely identical with the gold standard. All error types will be regarded as incorrect. (2) Identification level: this level could be considered as a multi-class categorization problem. All error types should be clearly identified. A correct case should be completely identical with the gold standard of the given error type. (3) Position level: in addition to identifying the error types, this level also judges the occurred range of grammatical error. That is to say, the system results should be perfectly identical with the quadruples of gold standard. The following metrics are measured at all levels with the help of the confusion matrix. For example, given 8 testing inputs with gold standards shown as "B1-1138, 7, 10, Disorder", "A2-0087, 12, 13, Missing", "A2-0904, correct", "B1-0990, correct", "A2-0789, 2, 3, Selection", "B1-0295, correct", "B2-0591, 3, 3, Redundant" and "A2-0920, correct", the system may output the result shown as "B1-1138, 7, 8, Disorder", "A2-0087, 12, 13, Missing", "A2-0904, 5, 6, Missing", "B1-0990, correct", "A2-0789, 2, 5, Disorder", "B1-0295, correct", "B2-0591, 3, 3, Redundant" and "A2-0920, 4, 5, Selection". The evaluation tool will yield the following performance. • False Positive Rate (FPR) = 0.5 (=2/4) Notes: {"A2-0904, 5, 6, Missing", "A2-0920, 4, 5, Selection"} /{"A2-0904, correct", "B1-0090, correct", "B1-0295, correct", "A2-0920, correct"} • Detection-level • Accuracy =0.75 (=6/8) Notes: {"B1-1138, Disorder", "A2-0087, Missing", "B1-0990, correct", "A2-0789, Disorder", "B1-0295, correct", "B2-0591, Redundant"} / {"B1-1138, Disorder", "A2-0087, Missing", "A2-0904, Missing", "B1-0990, correct", "A2-0789, Disorder", "B1-0295, correct", "B2-0591, Redundant", "A2-0920, Selection".} • Precision = 0.67 (=4/6) Notes: {"B1-1138, Disorder", "A2-0087, Missing", "A2-0789, Disorder", "B2-0591, Redundant"} / {"B1-1138, Disorder", "A2-0087, Missing", "A2-0904, Missing", "A2-0789, Disorder", "B2-0591, Redundant", "A2-0920, Selection".} • Position-level • Accuracy =0.5 (=4/8) Notes: {"A2-0087, 12, 13, Missing", "B1-0990, correct", "B1-0295, correct", "B2-0591, 3, 3, Redundant"} / {"B1-1138, 7, 8, Disorder", "A2-0087, 12, 13, Missing", "A2-0904, 5, 6, Missing", "B1-0990, correct", "A2-0789, 2, 5, Disorder", "B1-0295, correct", "B2-0591, 3, 3, Redundant", "A2-0920, 4, 5, Selection"} • Precision = 0.33 (=2/6) Notes: {"A2-0087, 12, 13, Missing", "B2-0591, 3, 3, Redundant"} / {"B1-1138, 7, 8, Disorder", "A2-0087, 12, 13, Missing", "A2-0904, 5, 6, Missing", "A2-0789, 2, 5, Disorder", "B2-0591, 3, 3, Redundant", "A2-0920, 4, 5, Selection"} • Recall = 0.5 (=2/4) Notes: {"A2-0087, 12, 13, Missing", "B2-0591, 3, 3, Redundant"} / {"B1-1138, 7, 10, Disorder", "A2-0087, 12, 13, Missing", "A2-0789, 2, 3, Selection", "B2-0591, 3, 3, Redundant"} • F1=0.4 (=2*0.33*0.5/(0.33+0.5)) Table 2 summarizes the submission statistics for the participating teams. Of 13 registered teams, 6 teams submitted their testing results. In formal testing phase, each participant was allowed to submit at most three runs using different models or parameter settings. In total, we had received 18 runs. Table 3 shows the task testing results. The CYUT team achieved the lowest false positive rate of 0.082. Detection-level evaluations are designed to detect whether a sentence contains grammatical errors or not. A neutral baseline can be easily achieved by always reporting all testing errors are correct without errors. According to the test data distribution, the baseline system can achieve an accuracy level of 0.5. All systems achieved results slightly better than the baseline. The system result submitted by NCYU achieved the best detection accuracy of 0.607. We used the F1 score to reflect the tradeoff between precision and recall. In the testing results, NTOU provided the best error detection results, providing a high F1 score of 0.6754. For correction-level evaluations, the systems need to identify the error types in the given sentences. The system developed by NCYU provided the highest F1 score of 0.3584 for grammatical error identification. For position-level evaluations, CYUT achieved the best F1 score of 0.1742. Note that it is difficult to perfectly identify the error positions, partly because no word delimiters exist among Chinese words. Table 3 . Testing results of our Chinese grammatical error diagnosis task. Evaluation Results In summary, none of the submitted systems provided superior performance. It is a really difficult task to develop an effective computer-assisted learning tool for grammatical error diagnosis, especially for the CFL uses. In general, this research problem still has long way to go. Conclusions and Future Work This paper provides an overview of the NLP-TEA 2015 shared task for Chinese grammatical error diagnosis, including task design, data preparation, evaluation metrics, and performance evaluation results. Regardless of actual performance, all submissions contribute to the common effort to produce an effective Chinese grammatical diagnosis tool, and the individual reports in the shared task proceedings provide useful insight into Chinese language processing. We hope the data sets collected for this shared task can facilitate and expedite the future development of NLP tools for computer-assisted Chinese language learning. Therefore, all data sets with gold standards and evaluation tool are publicly available for research purposes at http://ir.itc.ntnu.edu.tw/lre/nlptea15cged.htm. We plan to build new language resources to improve existing techniques for computer-aided Chinese language learning. In addition, new data sets with the contextual information of target sentences obtained from CFL learners will be investigated for the future enrichment of this research topic.
1 2 3 4 5 6
Introduction Task Description Data Preparation Performance Metrics Evaluation Results Conclusions and Future Work
Data Preparation
The essay sec%on of the computer-based Test of Chinese as a Foreign Language (TOCFL)
Na%ve Chinese speakers were trained to manually annotate gramma%cal errors and provide error correc%ons corresponding to each
The essay sec%on of the computer-based Test of Chinese as a Foreign Language (TOCFL)
Na%ve Chinese speakers were trained to manually annotate gramma%cal errors and provide error correc%ons corresponding to each