Yuan et al., 2021; Katsumata and Komachi, 2020).

The use of several pre-trained model architectures, however, tells a different
story and it is generally the case that using multiple architectures improves
performance: the top 3 latest state-of-the-art systems all use at least 2
different pre-trained models (Qorib, Na, and Ng, 2022; Lai et al., 2022;
Tarnavskyi, Chernodub, and Omelianchuk, 2022). This suggests that different
pre-training tasks capture different aspects of natural language that
complement each other in different ways in GEC. In contrast, approaches that
rely on a single pre-trained model typically perform slightly worse than those
that combine architectures, although it is worth keeping in mind that there is
also a trade-off between model complexity and run-time which is seldom
reported (Omelianchuk et al., 2020; Sun et al., 2021).

Finally, adding more performance-boosting techniques also tends to result in
better performance, and the systems that incorporate the most techniques
typically score highest. Among these techniques, the use of model ensembling
or system combination (Section 4.2) mitigates the instability of neural models
and allows a final system to make use of the strengths of several other
systems. However, this comes at a cost to model complexity and run-time.

##  8 Future Challenges

While much progress has been made in the past decade, several important
challenges remain (Qorib and Ng, 2022). This section highlights some of them
and offers suggestions for future work.

##### Domain Generalisation

Robustness is an important attribute of any NLP system. In the case of GEC, we
not only want our systems to work well for language learners, but also native
speakers in different domains such as business emails, literature and
instruction manuals. Some efforts have been made in this direction, such as
the native web texts in CWEB (Flachs et al., 2020), scientific articles in
AESW (Daudaravicius et al., 2016) and conversational dialog in ErAConD (Yuan
et al., 2022), but more effort is needed to create new corpora that represent
a wider variety of domains. This is important because previous research has
shown that systems that perform well in one domain do not necessarily perform
well in other domains (Napoles, Nădejde, and Tetreault, 2019).

##### Personalised Systems

Related to domain generalisation is the fact that system performance is also
tied to the profiles of the users in the training data. For example, a system
trained on L2 English data produced by advanced L1 Spanish learners is
unlikely to perform as well on L2 English data produced by beginner L1
Japanese learners because of the mismatch in ability level and first language.
It is thus important to develop corpora and tools that can adapt to different
users (Chollampatt, Hoang, and Ng, 2016), since different ability levels and
L1s can significantly affect the distribution of errors that authors are
likely to make (Nadejde and Tetreault, 2019).

##### Feedback Comment Generation

GEC systems are currently trained to correct errors without explaining why a
correction was needed. This is insufficient in an educational context however,
where it is desirable for a system to explain the cause of an error such that
a user may learn from it and not make the same mistake again. Resources have
begun to emerge to support this endeavour, but much more work is needed to
generate robust feedback comments to support explainable GEC (Nagata, 2019;
Nagata, Inui, and Ishikawa, 2020; Hanawa, Nagata, and Inui, 2021; Nagata et
al., 2021).

##### Model Interpretability

Related to feedback generation, it is also important that model output is
interpretable by humans. For example, although a system may make a prediction
with high confidence, there is no guarantee that the prediction will be
consistent with human intuition. Researchers have thus begun to build systems
that estimate the quality of model output in an effort to provide more
confidence that a given prediction is correct (Chollampatt and Ng, 2018c; Liu
et al., 2021). Similarly, Kaneko et al. (2022) propose an example-based
approach, where a model additionally outputs similar corrections in different
contexts in order to add credibility to the notion that the model truly
understood the error.

##### Semantic Errors

One of the areas where state-of-the-art systems still underperform is semantic
errors, which include complex phenomenon such as collocations, idioms, multi-
word expressions and fluency edits. A lot of work in GEC has focused on
correcting function word errors, which typically have small confusion sets and
comprise a majority of error types, but this does not mean we can neglect the
correction of content word errors. Although there has been some work on
correcting collocations (Kochmar and Briscoe, 2014; Herbelot and Kochmar,
2016) and multi-word expressions (Mizumoto, Mita, and Matsumoto, 2015;
Taslimipoor, Bryant, and Yuan, 2022), semantic errors remain a notable area in
which GEC systems could improve.

##### Contextual GEC

To date, most GEC systems operate at the sentence level, and so do not perform
well on errors that require cross-sentence context or document-level
understanding. Although work has already been done to incorporate multi-
sentence context into GEC systems (Chollampatt, Wang, and Ng, 2019; Yuan and
Bryant, 2021; Mita et al., 2022), almost all current datasets expect sentence
tokenised input and so do not facilitate multi-sentence evaluation. Paragraph
or document-level datasets, like in the Arabic QALB shared tasks (Mohit et
al., 2014; Rozovskaya et al., 2015), should thus be developed to encourage
contextual GEC in the future.

##### System Combination

Although much recent work focuses on NMT for GEC, this does not mean that
other approaches have nothing to offer. Work on system combination has shown
that systems built with different approaches have complementary strengths and
weaknesses such that a combined system can achieve significantly improved
performance (Susanto, Phandi, and Ng, 2014; Han and Ng, 2021; Lin and Ng,
2021; Qorib, Na, and Ng, 2022). Better understanding of these strengths and
weaknesses, and when and how to combine approaches are promising areas of
research. One tool is ALLECS (Qorib, Moon, and Ng, 2023), which is a web-
interface tool to produce text corrections using GEC system combination
methods.

##### Training Data Selection

Current state-of-the-art systems rely on pre-training on a massive amount of
synthetic parallel corpora, however this is both computationally-expensive and
not environmentally friendly. It is also questionable whether so much training
data is really necessary, as humans are not exposed to training data on such a
massive scale, yet can still correct errors without issue. A more economical
approach to effective training data selection is thus an important research
question that will go a long way towards reducing training time and developing
more efficient GEC systems (Lichtarge, Alberti, and Kumar, 2020; Takahashi,
Katsumata, and Komachi, 2020; Mita and Yanaka, 2021; Rothe et al., 2021).

##### Unsupervised Approaches

The dependency on parallel corpora (both real and synthetic) is a major
limiting factor in GEC system development, in that it is both laborious and
time-consuming to train human annotators to manually correct errors, and also
surprisingly difficult to generate high-quality synthetic errors that reliably
imitate human error patterns. It is furthermore noteworthy that humans can
correct errors without access to a large corpus of erroneous examples and
instead rely on their knowledge of grammatical language in order to detect and
correct mistakes. It should thus be intuitive that a GEC system might be able
to do the same by interpreting deviations from grammatical data as anomalies
that need to be corrected. The success of such an unsupervised approach would
significantly hasten the development of multilingual GEC systems and also
eliminate the need to compile parallel corpora.

##### Multilingual GEC

Although most work on GEC has focused on English, work on other languages is
also beginning to take off as new resources become available; e.g. in German
(Boyd, 2018), Russian (Rozovskaya and Roth, 2019) and Czech (Náplava et al.,
2022). While it is important to encourage research into GEC systems for
specific languages, it is also important to remember that it is ultimately not
scalable to build a separate system for every language. It is desirable to
work towards a single multilingual system that can correct all languages
simultaneously like in machine translation (Katsumata and Komachi, 2020; Rothe
et al., 2021).

##### Spoken GEC

Another aspect of GEC that has seldom been explored in the literature is that
of spoken GEC. While progress has largely been hindered by a lack of available
data, researchers have recently begun to build systems capable of detecting
and correcting errors in learner speech (Knill et al., 2019; Caines et al.,
2020; Kyriakopoulos, Knill, and Gales, 2020; Lu, Gales, and Wang, 2020; Lu,
Bannò, and Gales, 2022). Compared to text-based GEC, additional challenges
include recognising non-native accented speech (possibly including non-
standard pronunciation), disfluency detection, and utterance segmentation.

##### Improved Evaluation

Finally, robust evaluation of GEC system output is still an unsolved problem
and current evaluation practices may actually hinder progress (Rozovskaya and
Roth, 2021). For example, almost all metrics to date require tokenised text,
yet end-users require untokenised text, and so there is a disconnect between
system capability and user expectation. Similarly, GEC systems are typically
trained to output a single best correction for a sentence, yet end-users may
prefer a short n-best list of possible corrections for each edit, like in most
spellcheckers. Ultimately, alternative answers and untokenised text are not
yet properly accounted for in GEC system evaluation, leaving room for new
metrics to drive the field towards better practices.

##  9 Conclusion

In this survey paper, we set out to provide a comprehensive overview of the
state of the art in the field of Grammatical Error Correction. Our main goal
was to summarise the progress that has been made since Leacock et al. (2014)
but also complement the work of Wang et al. (2021) with more in-depth and
recent coverage on various topics.

With this in mind, we first explored the nature of the task and illustrated
the inherent difficulties in defining an error according to the perceived
communicative intent of the author. We next alluded to how these difficulties
can manifest in human-annotated corpora, before introducing the most commonly
used benchmark corpora for English, several less commonly used corpora for
English, and new corpora for GEC systems in other languages, including Arabic,
Chinese, Czech, German and Russian. Research into GEC for non-English
languages has begun to take off in the last couple of years and will no doubt
continue to grow in the future.

We next characterised the evolution of approaches to GEC, from error-type
specific classifiers to state-of-the-art NMT and edit-based sequence-
labelling, and summarised some of the additional supplementary techniques that
are commonly used to boost performance, such as re-ranking, multi-task
learning and iterative decoding. We also described different methods of
artificial data generation and augmentation, which have become core components
of recent GEC systems, but also drew attention to the benefits of low-resource
GEC systems that may be less resource intensive and more easily extended to
other languages.

Robust evaluation is still an unsolved problem in GEC, but we introduced the
most commonly used metrics in the field, along with their strengths and
weaknesses, and listed previous attempts at both reference-based and
reference-less metrics that were designed to overcome various shortcomings. We
furthermore highlighted the difficulty in correlating human judgements with
metric performance in light of the highly subjective nature of the task.

Finally, we provided an analysis of very recent progress in the field,
including making observations about which techniques/resources seemed to
perform best (particularly in the context of model efficiency), before
concluding with several possibilities for future work. We hope that this
survey will serve as comprehensive resource for researchers who are new to the
field or who want to be kept apprised of recent developments.

###### Acknowledgements.

We thank the anonymous reviewers for their helpful comments. This research is
supported by both Cambridge University Press & Assessment and the National
Research Foundation, Singapore under its AI Singapore Programme (AISG Award
No: AISG-RP-2019-014).

## References

  * Abend and Rappoport (2013) Abend, Omri and Ari Rappoport. 2013.  Universal Conceptual Cognitive Annotation (UCCA).  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 228–238.
  * Alikaniotis and Raheja (2019) Alikaniotis, Dimitris and Vipul Raheja. 2019.  The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error Correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 127–133.
  * Alsufieva, Kisselev, and Freels (2012) Alsufieva, Anna A., Olesya V. Kisselev, and Sandra G. Freels. 2012.  Results 2012: Using Flagship Data to Develop a Russian Learner Corpus of Academic Writing.  _Russian Language Journal_ , 62:79–105.
  * Asano et al. (2019) Asano, Hiroki, Masato Mita, Tomoya Mizumoto, and Jun Suzuki. 2019.  The AIP-Tohoku System at the BEA-2019 Shared Task.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 176–182.
  * Asano, Mizumoto, and Inui (2017) Asano, Hiroki, Tomoya Mizumoto, and Kentaro Inui. 2017.  Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 343–348.
  * Awasthi et al. (2019) Awasthi, Abhijeet, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019.  Parallel Iterative Edit Models for Local Sequence Transduction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4260–4270.
  * Bahdanau, Cho, and Bengio (2015) Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural Machine Translation by Jointly Learning to Align and Translate.  In _Proceedings of the 3rd International Conference on Learning Representations_.
  * Belkebir and Habash (2021) Belkebir, Riadh and Nizar Habash. 2021.  Automatic Error Type Annotation for Arabic.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 596–606.
  * Bell, Yannakoudakis, and Rei (2019) Bell, Samuel, Helen Yannakoudakis, and Marek Rei. 2019.  Context is Key: Grammatical Error Detection with Contextual Word Representations.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 103–115.
  * Berend et al. (2013) Berend, Gábor, Veronika Vincze, Sina Zarrieß, and Richárd Farkas. 2013\.  LFG-based Features for Noun Number and Article Grammatical Errors.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 62–67.
  * van den Bosch and Berck (2013) van den Bosch, Antal and Peter Berck. 2013.  Memory-based Grammatical Error Correction.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 102–108.
  * Boyd (2018) Boyd, Adriane. 2018.  Using Wikipedia Edits in Low Resource Grammatical Error Correction.  In _Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text_ , pages 79–84.
  * Boyd et al. (2014) Boyd, Adriane, Jirka Hana, Lionel Nicolas, Detmar Meurers, Katrin Wisniewski, Andrea Abel, Karin Schöne, Barbora Štindlová, and Chiara Vettori. 2014.  The MERLIN corpus: Learner language and the CEFR.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 1281–1288.
  * Brockett, Dolan, and Gamon (2006) Brockett, Chris, William B. Dolan, and Michael Gamon. 2006.  Correcting ESL Errors Using Phrasal SMT Techniques.  In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_ , pages 249–256.
  * Brown et al. (2020) Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901.
  * Bryant (2019) Bryant, Christopher. 2019.  _Automatic annotation of error types for grammatical error correction_.  Ph.D. thesis, University of Cambridge.
  * Bryant and Briscoe (2018) Bryant, Christopher and Ted Briscoe. 2018.  Language Model Based Grammatical Error Correction without Annotated Training Data.  In _Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 247–253.
  * Bryant and Felice (2016) Bryant, Christopher and Mariano Felice. 2016.  Issues in preprocessing current datasets for grammatical error correction.  Technical Report UCAM-CL-TR-894, University of Cambridge, Computer Laboratory.
  * Bryant et al. (2019) Bryant, Christopher, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019\.  The BEA-2019 Shared Task on Grammatical Error Correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 52–75.
  * Bryant, Felice, and Briscoe (2017) Bryant, Christopher, Mariano Felice, and Ted Briscoe. 2017.  Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 793–805.
  * Bryant and Ng (2015) Bryant, Christopher and Hwee Tou Ng. 2015.  How Far are We from Fully Automatic High Quality Grammatical Error Correction?  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 697–707.
  * Burstein, Chodorow, and Leacock (2003) Burstein, Jill, Martin Chodorow, and Claudia Leacock. 2003.  CriterionSM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays.  In _Proceedings of the Fifteenth Conference on Innovative Applications of Artificial Intelligence_ , pages 3–10.
  * Caines et al. (2020) Caines, Andrew, Christian Bentz, Kate Knill, Marek Rei, and Paula Buttery. 2020\.  Grammatical error detection in transcriptions of spoken English.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2144–2162.
  * Cao, Yang, and Ng (2021) Cao, Hannan, Wenmian Yang, and Hwee Tou Ng. 2021.  Grammatical Error Correction with Contrastive Learning in Low Error Density Domains.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4867–4874.
  * Cao, Yang, and Ng (2023) Cao, Hannan, Wenmian Yang, and Hwee Tou Ng. 2023.  Mitigating exposure bias in grammatical error correction with data augmentation and reweighting.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_.
  * Chen et al. (2020a) Chen, Mengyun, Tao Ge, Xingxing Zhang, Furu Wei, and Ming Zhou. 2020a.  Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7162–7169.
  * Chen et al. (2020b) Chen, Ting, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b.  A Simple Framework for Contrastive Learning of Visual Representations.  In _Proceedings of the 37th International Conference on Machine Learning_ , pages 1597–1607.
  * Cho et al. (2014) Cho, Kyunghyun, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , pages 1724–1734.
  * Chodorow et al. (2012) Chodorow, Martin, Markus Dickinson, Ross Israel, and Joel Tetreault. 2012.  Problems in Evaluating Grammatical Error Detection Systems.  In _Proceedings of COLING 2012_ , pages 611–628.
  * Chodorow, Tetreault, and Han (2007) Chodorow, Martin, Joel Tetreault, and Na-Rae Han. 2007.  Detection of Grammatical Errors Involving Prepositions.  In _Proceedings of the Fourth ACL-SIGSEM Workshop on Prepositions_ , pages 25–30.
  * Choe et al. (2019) Choe, Yo Joong, Jiyeon Ham, Kyubyong Park, and Yeoil Yoon. 2019.  A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 213–227.
  * Chollampatt, Hoang, and Ng (2016) Chollampatt, Shamil, Duc Tam Hoang, and Hwee Tou Ng. 2016.  Adapting Grammatical Error Correction Based on the Native Language of Writers with Neural Network Joint Models.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1901–1911.
  * Chollampatt and Ng (2017) Chollampatt, Shamil and Hwee Tou Ng. 2017.  Connecting the Dots: Towards Human-Level Grammatical Error Correction.  In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 327–333.
  * Chollampatt and Ng (2018a) Chollampatt, Shamil and Hwee Tou Ng. 2018a.  A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)_ , pages 5755–5762.
  * Chollampatt and Ng (2018b) Chollampatt, Shamil and Hwee Tou Ng. 2018b.  A Reassessment of Reference-Based Grammatical Error Correction Metrics.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2730–2741.
  * Chollampatt and Ng (2018c) Chollampatt, Shamil and Hwee Tou Ng. 2018c.  Neural Quality Estimation of Grammatical Error Correction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2528–2539.
  * Chollampatt, Taghipour, and Ng (2016) Chollampatt, Shamil, Kaveh Taghipour, and Hwee Tou Ng. 2016.  Neural network translation models for grammatical error correction.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_ , pages 2768–2774.
  * Chollampatt, Wang, and Ng (2019) Chollampatt, Shamil, Weiqi Wang, and Hwee Tou Ng. 2019.  Cross-Sentence Grammatical Error Correction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 435–445.
  * Choshen and Abend (2018a) Choshen, Leshem and Omri Abend. 2018a.  Automatic Metric Validation for Grammatical Error Correction.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1372–1382.
  * Choshen and Abend (2018b) Choshen, Leshem and Omri Abend. 2018b.  Inherent Biases in Reference-based Evaluation for Grammatical Error Correction.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 632–642.
  * Choshen and Abend (2018c) Choshen, Leshem and Omri Abend. 2018c.  Reference-less Measure of Faithfulness for Grammatical Error Correction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 124–129.
  * Choshen et al. (2020) Choshen, Leshem, Dmitry Nikolaev, Yevgeni Berzak, and Omri Abend. 2020.  Classifying Syntactic Errors in Learner Language.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 97–107.
  * Choshen et al. (2021) Choshen, Leshem, Matanel Oren, Dmitry Nikolaev, and Omri Abend. 2021.  SERRANT: a syntactic classifier for English Grammatical Error Types.  _arXiv_ , 2104.02310.
  * Chowdhery et al. (2022) Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  PaLM: Scaling Language Modeling with Pathways.  _arXiv_ , 2204.02311.
  * Council of Europe (2001) Council of Europe. 2001.  _Common European Framework of Reference for Languages: Learning, Teaching, Assessment._
  * Coyne and Sakaguchi (2023) Coyne, Steven and Keisuke Sakaguchi. 2023.  An Analysis of GPT-3’s Performance in Grammatical Error Correction.  _arXiv_ , 2303.14342.
  * Crowston (2012) Crowston, Kevin. 2012.  Amazon Mechanical Turk: A Research Tool for Organizations and Information Systems Scholars.  In _Shaping the Future of ICT Research. Methods and Approaches_ , pages 210–221.
  * Dahlmeier and Ng (2011a) Dahlmeier, Daniel and Hwee Tou Ng. 2011a.  Correcting Semantic Collocation Errors with L1-induced Paraphrases.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 107–117.
  * Dahlmeier and Ng (2011b) Dahlmeier, Daniel and Hwee Tou Ng. 2011b.  Grammatical Error Correction with Alternating Structure Optimization.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 915–923.
  * Dahlmeier and Ng (2012a) Dahlmeier, Daniel and Hwee Tou Ng. 2012a.  A Beam-Search Decoder for Grammatical Error Correction.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 568–578.
  * Dahlmeier and Ng (2012b) Dahlmeier, Daniel and Hwee Tou Ng. 2012b.  Better Evaluation for Grammatical Error Correction.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 568–572.
  * Dahlmeier, Ng, and Wu (2013) Dahlmeier, Daniel, Hwee Tou Ng, and Siew Mei Wu. 2013.  Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English.  In _Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 22–31.
  * Dale, Anisimoff, and Narroway (2012) Dale, Robert, Ilya Anisimoff, and George Narroway. 2012.  HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task.  In _Proceedings of the Seventh Workshop on Building Educational Applications Using NLP_ , pages 54–62.
  * Daudaravicius et al. (2016) Daudaravicius, Vidas, Rafael E. Banchs, Elena Volodina, and Courtney Napoles. 2016\.  A Report on the Automatic Evaluation of Scientific Writing Shared Task.  In _Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 53–62.
  * De Felice (2008) De Felice, Rachele. 2008.  _Automatic error detection in non-native English_.  Ph.D. thesis, Oxford University.
  * De Felice and Pulman (2008) De Felice, Rachele and Stephen G. Pulman. 2008.  A Classifier-Based Approach to Preposition and Determiner Error Correction in L2 English.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pages 169–176.
  * Denkowski and Lavie (2014) Denkowski, Michael and Alon Lavie. 2014.  Meteor Universal: Language Specific Translation Evaluation for Any Target Language.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 376–380.
  * Devlin et al. (2019) Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Edunov et al. (2018) Edunov, Sergey, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding Back-Translation at Scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500.
  * Ehsan and Faili (2013) Ehsan, Nava and Heshaam Faili. 2013.  Grammatical and context-sensitive error correction using a statistical machine translation framework.  _Software: Practice and Experience_ , 43(2):187–206.
  * Fang et al. (2023) Fang, Tao, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023.  Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation.  _arXiv_.
  * Federmann (2010) Federmann, Christian. 2010.  Appraise: An Open-Source Toolkit for Manual Phrase-Based Evaluation of Translations.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)_ , pages 1731–1734.
  * Felice (2016) Felice, Mariano. 2016.  _Artificial error generation for translation-based grammatical error correction_.  Ph.D. thesis, University of Cambridge.
  * Felice and Briscoe (2015) Felice, Mariano and Ted Briscoe. 2015.  Towards a standard evaluation method for grammatical error detection and correction.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 578–587.
  * Felice, Bryant, and Briscoe (2016) Felice, Mariano, Christopher Bryant, and Ted Briscoe. 2016.  Automatic Extraction of Learner Errors in ESL Sentences Using Linguistically Enhanced Alignments.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 825–835.
  * Felice and Yuan (2014a) Felice, Mariano and Zheng Yuan. 2014a.  Generating artificial errors for grammatical error correction.  In _Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 116–126.
  * Felice and Yuan (2014b) Felice, Mariano and Zheng Yuan. 2014b.  To err is human, to correct is divine.  _XRDS_ , 21(1):22–27.
  * Felice et al. (2014) Felice, Mariano, Zheng Yuan, Øistein E. Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014.  Grammatical error correction using hybrid systems and type filtering.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 15–24.
  * Flachs, Lacroix, and Søgaard (2019) Flachs, Simon, Ophélie Lacroix, and Anders Søgaard. 2019.  Noisy Channel for Low Resource Grammatical Error Correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 191–196.
  * Flachs et al. (2020) Flachs, Simon, Ophélie Lacroix, Helen Yannakoudakis, Marek Rei, and Anders Søgaard. 2020.  Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8467–8478.
  * Flachs, Stahlberg, and Kumar (2021) Flachs, Simon, Felix Stahlberg, and Shankar Kumar. 2021.  Data Strategies for Low-Resource Grammatical Error Correction.  In _Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 117–122.
  * Gamon (2010) Gamon, Michael. 2010.  Using Mostly Native Data to Correct Errors in Learners’ Writing.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 163–171.
  * Gamon et al. (2008) Gamon, Michael, Jianfeng Gao, Chris Brockett, Alexandre Klementiev, William B. Dolan, Dmitriy Belenko, and Lucy Vanderwende. 2008.  Using Contextual Speller Techniques and Language Modeling for ESL Error Correction.  In _Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I_ , pages 449–456.
  * Ge, Wei, and Zhou (2018) Ge, Tao, Furu Wei, and Ming Zhou. 2018.  Fluency Boost Learning and Inference for Neural Grammatical Error Correction.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1055–1065.
  * Geertzen, Alexopoulou, and Korhonen (2013) Geertzen, Jeroen, Theodora Alexopoulou, and Anna Korhonen. 2013.  Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge Open Language Database (EFCAMDAT).  In _Selected Proceedings of the 31st Second Language Research Forum (SLRF)s_.
  * Gehring et al. (2017) Gehring, Jonas, Michael Auli, David Grangier, and Yann Dauphin. 2017.  A Convolutional Encoder Model for Neural Machine Translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 123–135.
  * Gong et al. (2022) Gong, Peiyuan, Xuebo Liu, Heyan Huang, and Min Zhang. 2022.  Revisiting Grammatical Error Correction Evaluation and Beyond.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6891–6902.
  * Goodfellow, Bengio, and Courville (2016) Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016.  _Deep Learning_.  http://www.deeplearningbook.org.
  * Goodfellow et al. (2014) Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative Adversarial Nets.  In _Advances in Neural Information Processing Systems_ , volume 27.
  * Gotou et al. (2020) Gotou, Takumi, Ryo Nagata, Masato Mita, and Kazuaki Hanawa. 2020.  Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2085–2095.
  * Graham, Baldwin, and Mathur (2015) Graham, Yvette, Timothy Baldwin, and Nitika Mathur. 2015.  Accurate Evaluation of Segment-level Machine Translation Metrics.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1183–1191.
  * Granger (1998) Granger, Sylviane. 1998.  The computer learner corpus: A versatile new source of data for SLA research.  In Sylviane Granger, editor, _Learner English on Computer_. pages 3–18.
  * Grundkiewicz and Junczys-Dowmunt (2014) Grundkiewicz, Roman and Marcin Junczys-Dowmunt. 2014.  The WikEd Error Corpus: A Corpus of Corrective Wikipedia Edits and its Application to Grammatical Error Correction.  In _Advances in Natural Language Processing – Lecture Notes in Computer Science_ , volume 8686, pages 478–490.
  * Grundkiewicz and Junczys-Dowmunt (2018) Grundkiewicz, Roman and Marcin Junczys-Dowmunt. 2018.  Near Human-Level Performance in Grammatical Error Correction with Hybrid Machine Translation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 284–290.
  * Grundkiewicz and Junczys-Dowmunt (2019) Grundkiewicz, Roman and Marcin Junczys-Dowmunt. 2019.  Minimally-Augmented Grammatical Error Correction.  In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_ , pages 357–363.
  * Grundkiewicz, Junczys-Dowmunt, and Gillian (2015) Grundkiewicz, Roman, Marcin Junczys-Dowmunt, and Edward Gillian. 2015.  Human Evaluation of Grammatical Error Correction Systems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 461–470.
  * Grundkiewicz, Junczys-Dowmunt, and Heafield (2019) Grundkiewicz, Roman, Marcin Junczys-Dowmunt, and Kenneth Heafield. 2019.  Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 252–263.
  * Hagiwara and Mita (2020) Hagiwara, Masato and Masato Mita. 2020.  GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 6761–6768.
  * Han, Chodorow, and Leacock (2006) Han, Na-Rae, Martin Chodorow, and Claudia Leacock. 2006.  Detecting errors in English article usage by non-native speakers.  _Natural Language Engineering_ , 12(2):115–129.
  * Han and Ng (2021) Han, W. and H. Ng. 2021.  Diversity-Driven Combination for Grammatical Error Correction.  In _2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI)_ , pages 972–979.
  * Hanawa, Nagata, and Inui (2021) Hanawa, Kazuaki, Ryo Nagata, and Kentaro Inui. 2021.  Exploring Methods for Generating Feedback Comments for Writing Learning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9719–9730.
  * Hanna and Bojar (2021) Hanna, Michael and Ondřej Bojar. 2021.  A Fine-Grained Analysis of BERTScore.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 507–517.
  * Heilman et al. (2014) Heilman, Michael, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. 2014.  Predicting Grammaticality on an Ordinal Scale.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 174–180.
  * Herbelot and Kochmar (2016) Herbelot, Aurélie and Ekaterina Kochmar. 2016.  ‘Calling on the classical phone’: a distributional model of adjective-noun errors in learners’ English.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 976–986.
  * Hoang, Chollampatt, and Ng (2016) Hoang, Duc Tam, Shamil Chollampatt, and Hwee Tou Ng. 2016.  Exploiting N-Best Hypotheses to Improve an SMT Approach to Grammatical Error Correction.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_ , page 2803–2809.
  * Hochreiter and Schmidhuber (1997) Hochreiter, Sepp and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ , 9(8):1735–1780.
  * Hoskere (2019) Hoskere, Jayakumar. 2019.  Everyday AI: beyond spell check, how Google Docs is smart enough to correct grammar.  _Google Blogs_.
  * Hotate, Kaneko, and Komachi (2020) Hotate, Kengo, Masahiro Kaneko, and Mamoru Komachi. 2020.  Generating Diverse Corrections with Local Beam Search for Grammatical Error Correction.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2132–2137.
  * Htut and Tetreault (2019) Htut, Phu Mon and Joel Tetreault. 2019.  The Unbearable Weight of Generating Artificial Errors for Grammatical Error Correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 478–483.
  * Huang et al. (2017) Huang, Yan, Jeroen Geertzen, Rachel Baker, Anna Korhonen, and Theodora Alexopoulou. 2017.  The EF Cambridge Open Language Database (efcamdat): Information for Users.  Technical report, Department of Theoretical and Applied Linguistics, University of Cambridge and EF Education First.
  * Islam and Magnani (2021) Islam, Md Asadul and Enrico Magnani. 2021.  Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3009–3015.
  * Jawahar, Sagot, and Seddah (2019) Jawahar, Ganesh, Benoît Sagot, and Djamé Seddah. 2019.  What Does BERT Learn about the Structure of Language?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657.
  * Jensen et al. (1983) Jensen, K., G. E. Heidorn, L. A. Miller, and Y. Ravin. 1983.  Parse Fitting and Prose Fixing: Getting a Hold on Ill-Formedness.  _American Journal of Computational Linguistics_ , 9(3-4):147–160.
  * Ji et al. (2017) Ji, Jianshu, Qinlong Wang, Kristina Toutanova, Yongen Gong, Steven Truong, and Jianfeng Gao. 2017.  A Nested Attention Neural Hybrid Model for Grammatical Error Correction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 753–762.
  * Jia, Wang, and Zhao (2013) Jia, Zhongye, Peilu Wang, and Hai Zhao. 2013.  Grammatical Error Correction as Multiclass Classification with Single Model.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 74–81.
  * Junczys-Dowmunt and Grundkiewicz (2014) Junczys-Dowmunt, Marcin and Roman Grundkiewicz. 2014.  The AMU System in the CoNLL-2014 Shared Task: Grammatical Error Correction by Data-Intensive and Feature-Rich Statistical Machine Translation.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 25–33.
  * Junczys-Dowmunt and Grundkiewicz (2016) Junczys-Dowmunt, Marcin and Roman Grundkiewicz. 2016.  Phrase-based Machine Translation is State-of-the-Art for Automatic Grammatical Error Correction.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1546–1556.
  * Junczys-Dowmunt et al. (2018) Junczys-Dowmunt, Marcin, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018\.  Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 595–606.
  * Kalchbrenner and Blunsom (2013) Kalchbrenner, Nal and Phil Blunsom. 2013.  Recurrent Continuous Translation Models.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1700–1709.
  * Kaneko et al. (2019) Kaneko, Masahiro, Kengo Hotate, Satoru Katsumata, and Mamoru Komachi. 2019.  TMU Transformer System Using BERT for Re-ranking at BEA 2019 Grammatical Error Correction on Restricted Track.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 207–212.
  * Kaneko et al. (2020) Kaneko, Masahiro, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020.  Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4248–4254.
  * Kaneko et al. (2022) Kaneko, Masahiro, Sho Takase, Ayana Niwa, and Naoaki Okazaki. 2022.  Interpretability for Language Learners Using Example-Based Grammatical Error Correction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7176–7187.
  * Kantor et al. (2019) Kantor, Yoav, Yoav Katz, Leshem Choshen, Edo Cohen-Karlik, Naftali Liberman, Assaf Toledo, Amir Menczel, and Noam Slonim. 2019.  Learning to combine Grammatical Error Corrections.  In _Proceedings of BEA_ , pages 139–148.
  * Katsumata and Komachi (2020) Katsumata, Satoru and Mamoru Komachi. 2020.  Stronger Baselines for Grammatical Error Correction Using a Pretrained Encoder-Decoder Model.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 827–832.
  * Ke and Ng (2019) Ke, Zixuan and Vincent Ng. 2019.  Automated Essay Scoring: A Survey of the State of the Art.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 6300–6308.
  * Kiyono et al. (2019) Kiyono, Shun, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019.  An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1236–1242.
  * Knill et al. (2019) Knill, K.M., M.J.F. Gales, P.P. Manakul, and A.P. Caines. 2019.  Automatic Grammatical Error Detection of Non-native Spoken Learner English.  In _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 8127–8131.
  * Kochmar, Andersen, and Briscoe (2012) Kochmar, Ekaterina, Øistein Andersen, and Ted Briscoe. 2012.  HOO 2012 Error Recognition and Correction Shared Task: Cambridge University Submission Report.  In _Proceedings of the Seventh Workshop on Building Educational Applications Using NLP_ , pages 242–250.
  * Kochmar and Briscoe (2014) Kochmar, Ekaterina and Ted Briscoe. 2014.  Detecting Learner Errors in the Choice of Content Words Using Compositional Distributional Semantics.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 1740–1751.
  * Korre, Chatzipanagiotou, and Pavlopoulos (2021) Korre, Katerina, Marita Chatzipanagiotou, and John Pavlopoulos. 2021.  ELERRANT: Automatic Grammatical Error Type Classification for Greek.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_ , pages 708–717.
  * Koyama et al. (2021) Koyama, Aomi, Kengo Hotate, Masahiro Kaneko, and Mamoru Komachi. 2021.  Comparison of Grammatical Error Correction Using Back-Translation Models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop_ , pages 126–135.
  * Koyama et al. (2020) Koyama, Aomi, Tomoshige Kiyuna, Kenji Kobayashi, Mio Arai, and Mamoru Komachi. 2020\.  Construction of an Evaluation Corpus for Grammatical Error Correction for Learners of Japanese as a Second Language.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 204–211.
  * Kunchukuttan, Chaudhury, and Bhattacharyya (2014) Kunchukuttan, Anoop, Sriram Chaudhury, and Pushpak Bhattacharyya. 2014.  Tuning a Grammar Correction System for Increased Precision.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 60–64.
  * Kwasny and Sondheimer (1981) Kwasny, Stan C. and Norman K. Sondheimer. 1981.  Relaxation Techniques for Parsing Grammatically Ill-Formed Input in Natural Language Understanding Systems.  _American Journal of Computational Linguistics_ , 7(2):99–108.
  * Kyriakopoulos, Knill, and Gales (2020) Kyriakopoulos, Konstantinos, Kate M. Knill, and Mark J. F. Gales. 2020.  Automatic Detection of Accent and Lexical Pronunciation Errors in Spontaneous Non-Native English Speech.  In _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_ , pages 3052–3056.
  * Lai et al. (2022) Lai, Shaopeng, Qingyu Zhou, Jiali Zeng, Zhongli Li, Chao Li, Yunbo Cao, and Jinsong Su. 2022.  Type-Driven Multi-Turn Corrections for Grammatical Error Correction.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3225–3236.
  * Leacock et al. (2014) Leacock, Claudia, Martin Chodorow, Michael Gamon, and Joel Tetreault. 2014.  _Automated Grammatical Error Detection for Language Learners_.
  * Leacock, Gamon, and Brockett (2009) Leacock, Claudia, Michael Gamon, and Chris Brockett. 2009.  User Input and Interactions on Microsoft Research ESL Assistant.  In _Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 73–81.
  * Lee (2004) Lee, John. 2004.  Automatic Article Restoration.  In _Proceedings of the Student Research Workshop at HLT-NAACL 2004_ , pages 31–36.
  * Lee and Seneff (2008) Lee, John and Stephanie Seneff. 2008.  Correcting Misuse of Verb Forms.  In _Proceedings of ACL-08: HLT_ , pages 174–182.
  * Lee and Lee (2014) Lee, Kyusong and Gary Geunbae Lee. 2014.  POSTECH Grammatical Error Correction System in the CoNLL-2014 Shared Task.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 65–73.
  * Lee, Tseng, and Chang (2018) Lee, Lung-Hao, Yuen-Hsien Tseng, and Li-Ping Chang. 2018.  Building a TOCFL Learner Corpus for Chinese Grammatical Error Diagnosis.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , pages 2298–2304.
  * Levenshtein (1966) Levenshtein, Vladimir I. 1966.  Binary Codes Capable of Correcting Deletions, Insertions and Reversals.  _Soviet Physics Doklady_ , 10:707–710.
  * Lewis et al. (2020) Lewis, Mike, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Li et al. (2019) Li, Ruobing, Chuan Wang, Yefei Zha, Yonghong Yu, Shiman Guo, Qiang Wang, Yang Liu, and Hui Lin. 2019.  The LAIX Systems in the BEA-2019 GEC Shared Task.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 159–167.
  * Lichtarge, Alberti, and Kumar (2020) Lichtarge, Jared, Chris Alberti, and Shankar Kumar. 2020.  Data Weighted Training Strategies for Grammatical Error Correction.  _Transactions of the Association for Computational Linguistics_ , 8:634–646.
  * Lichtarge et al. (2019) Lichtarge, Jared, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong. 2019.  Corpora Generation for Grammatical Error Correction.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3291–3301.
  * Lin and Ng (2021) Lin, Ruixi and Hwee Tou Ng. 2021.  System Combination for Grammatical Error Correction Based on Integer Programming.  In _Proceedings of RANLP_ , pages 829–834.
  * Liu et al. (2023) Liu, Pengfei, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  _ACM Comput. Surv._ , 55(9).
  * Liu et al. (2020) Liu, Xiaodong, Kevin Duh, Liyuan Liu, and Jianfeng Gao. 2020.  Very Deep Transformers for Neural Machine Translation.  _arXiv_ , 2008.07772.
  * Liu et al. (2021) Liu, Zhenghao, Xiaoyuan Yi, Maosong Sun, Liner Yang, and Tat-Seng Chua. 2021.  Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5441–5452.
  * Lu, Bannò, and Gales (2022) Lu, Yiting, Stefano Bannò, and Mark Gales. 2022.  On Assessing and Developing Spoken ’Grammatical Error Correction’ Systems.  In _Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)_ , pages 51–60.
  * Lu, Gales, and Wang (2020) Lu, Yiting, Mark J. F. Gales, and Yu Wang. 2020.  Spoken Language ’Grammatical Error Correction’.  In _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_ , pages 3840–3844.
  * Madnani, Tetreault, and Chodorow (2012) Madnani, Nitin, Joel Tetreault, and Martin Chodorow. 2012.  Exploring grammatical error correction with not-so-crummy machine translation.  In _Proceedings of the Seventh Workshop on Building Educational Applications Using NLP_ , pages 44–53.
  * Maeda, Kaneko, and Okazaki (2022) Maeda, Koki, Masahiro Kaneko, and Naoaki Okazaki. 2022.  IMPARA: Impact-Based Metric for GEC Using Parallel Data.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3578–3588.
  * Malmi et al. (2019) Malmi, Eric, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019.  Encode, Tag, Realize: High-Precision Text Editing.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5054–5065.
  * Mikolov et al. (2013) Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed Representations of Words and Phrases and their Compositionality.  In _Advances in Neural Information Processing Systems_ , volume 26.
  * Mita et al. (2020) Mita, Masato, Shun Kiyono, Masahiro Kaneko, Jun Suzuki, and Kentaro Inui. 2020.  A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 267–280.
  * Mita et al. (2019) Mita, Masato, Tomoya Mizumoto, Masahiro Kaneko, Ryo Nagata, and Kentaro Inui. 2019\.  Cross-Corpora Evaluation and Analysis of Grammatical Error Correction Models — Is Single-Corpus Evaluation Enough?  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1309–1314.
  * Mita et al. (2022) Mita, Masato, Keisuke Sakaguchi, Masato Hagiwara, Tomoya Mizumoto, Jun Suzuki, and Kentaro Inui. 2022.  Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond.  _arXiv_ , 2205.11484.
  * Mita and Yanaka (2021) Mita, Masato and Hitomi Yanaka. 2021.  Do Grammatical Error Correction Models Realize Grammatical Generalization?  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4554–4561.
  * Mizumoto et al. (2012) Mizumoto, Tomoya, Yuta Hayashibe, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2012.  The Effect of Learner Corpus Size in Grammatical Error Correction of ESL Writings.  In _Proceedings of COLING 2012: Posters_ , pages 863–872.
  * Mizumoto et al. (2011) Mizumoto, Tomoya, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011.  Mining Revision Log of Language Learning SNS for Automated Japanese Error Correction of Second Language Learners.  In _Proceedings of 5th International Joint Conference on Natural Language Processing_ , pages 147–155.
  * Mizumoto and Matsumoto (2016) Mizumoto, Tomoya and Yuji Matsumoto. 2016.  Discriminative Reranking for Grammatical Error Correction with Statistical Machine Translation.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1133–1138.
  * Mizumoto, Mita, and Matsumoto (2015) Mizumoto, Tomoya, Masato Mita, and Yuji Matsumoto. 2015.  Grammatical Error Correction Considering Multi-word Expressions.  In _Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications_ , pages 82–86.
  * Mohit et al. (2014) Mohit, Behrang, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014.  The First QALB Shared Task on Automatic Text Correction for Arabic.  In _Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)_ , pages 39–47.
  * Muennighoff et al. (2022) Muennighoff, Niklas, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022\.  Crosslingual Generalization through Multitask Finetuning.  _arXiv_ , 2211.01786.
  * Nadejde and Tetreault (2019) Nadejde, Maria and Joel Tetreault. 2019.  Personalizing Grammatical Error Correction: Adaptation to Proficiency Level and L1.  In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_ , pages 27–33.
  * Nagata (2019) Nagata, Ryo. 2019.  Toward a Task of Feedback Comment Generation for Writing Learning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3206–3215.
  * Nagata et al. (2021) Nagata, Ryo, Masato Hagiwara, Kazuaki Hanawa, Masato Mita, Artem Chernodub, and Olena Nahorna. 2021.  Shared Task on Feedback Comment Generation for Language Learners.  In _Proceedings of the 14th International Conference on Natural Language Generation_ , pages 320–324.
  * Nagata, Inui, and Ishikawa (2020) Nagata, Ryo, Kentaro Inui, and Shin’ichiro Ishikawa. 2020.  Creating Corpora for Research in Feedback Comment Generation.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 340–345.
  * Náplava and Straka (2019) Náplava, Jakub and Milan Straka. 2019.  Grammatical Error Correction in Low-Resource Scenarios.  In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_ , pages 346–356.
  * Náplava et al. (2022) Náplava, Jakub, Milan Straka, Jana Straková, and Alexandr Rosen. 2022.  Czech Grammar Error Correction with a Large and Diverse Corpus.  _Transactions of the Association for Computational Linguistics_ , 10:452–467.
  * Napoles, Nădejde, and Tetreault (2019) Napoles, Courtney, Maria Nădejde, and Joel Tetreault. 2019.  Enabling Robust Grammatical Error Correction in New Domains: Data Sets, Metrics, and Analyses.  _Transactions of the Association for Computational Linguistics_ , 7:551–566.
  * Napoles et al. (2015) Napoles, Courtney, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015.  Ground Truth for Grammatical Error Correction Metrics.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 588–593.
  * Napoles et al. (2016) Napoles, Courtney, Keisuke Sakaguchi, Matt Post, and Joel R. Tetreault. 2016.  GLEU Without Tuning.  _arXiv_ , 1605.02592.
  * Napoles, Sakaguchi, and Tetreault (2016) Napoles, Courtney, Keisuke Sakaguchi, and Joel Tetreault. 2016.  There’s No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2109–2115.
  * Napoles, Sakaguchi, and Tetreault (2017) Napoles, Courtney, Keisuke Sakaguchi, and Joel Tetreault. 2017.  JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 229–234.
  * Ng et al. (2014) Ng, Hwee Tou, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014.  The CoNLL-2014 Shared Task on Grammatical Error Correction.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 1–14.
  * Ng et al. (2013) Ng, Hwee Tou, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013.  The CoNLL-2013 Shared Task on Grammatical Error Correction.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 1–12.
  * Nicholls (2003) Nicholls, Diane. 2003.  The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT.  In _Proceedings of the Corpus Linguistics 2003 conference_ , pages 572–581.
  * Nivre et al. (2020) Nivre, Joakim, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4034–4043.
  * Omelianchuk et al. (2020) Omelianchuk, Kostiantyn, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020.  GECToR – Grammatical Error Correction: Tag, Not Rewrite.  In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 163–170.
  * Opitz and Maclin (1999) Opitz, David and Richard Maclin. 1999.  Popular Ensemble Methods: An Empirical Study.  _J. Artif. Int. Res._ , 11(1):169–198.
  * Papineni et al. (2002) Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Parnow, Li, and Zhao (2021) Parnow, Kevin, Zuchao Li, and Hai Zhao. 2021.  Grammatical Error Correction as GAN-like Sequence Labeling.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3284–3290.
  * Pennington, Socher, and Manning (2014) Pennington, Jeffrey, Richard Socher, and Christopher Manning. 2014.  GloVe: Global Vectors for Word Representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , pages 1532–1543.
  * Putra and Szabó (2013) Putra, Desmond Darma and Lili Szabó. 2013.  UdS at CoNLL 2013 Shared Task.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 88–95.
  * Qorib, Na, and Ng (2022) Qorib, Muhammad, Seung-Hoon Na, and Hwee Tou Ng. 2022.  Frustratingly easy system combination for grammatical error correction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1964–1974.
  * Qorib, Moon, and Ng (2023) Qorib, Muhammad Reza, Geonsik Moon, and Hwee Tou Ng. 2023.  ALLECS: A Lightweight Language Error Correction System.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_.
  * Qorib and Ng (2022) Qorib, Muhammad Reza and Hwee Tou Ng. 2022.  Grammatical Error Correction: Are We There Yet?  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 2794–2800.
  * Radford et al. (2019) Radford, Alec, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language Models are Unsupervised Multitask Learners.
  * Raffel et al. (2020) Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Raheja and Alikaniotis (2020) Raheja, Vipul and Dimitris Alikaniotis. 2020.  Adversarial Grammatical Error Correction.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3075–3087.
  * Rao, Yang, and Zhang (2020) Rao, Gaoqi, Erhong Yang, and Baolin Zhang. 2020.  Overview of NLPTEA-2020 shared task for Chinese grammatical error diagnosis.  In _Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications_ , pages 25–35.
  * Rei et al. (2017) Rei, Marek, Mariano Felice, Zheng Yuan, and Ted Briscoe. 2017.  Artificial Error Generation with Machine Translation and Syntactic Patterns.  In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 287–292.
  * Rei and Yannakoudakis (2016) Rei, Marek and Helen Yannakoudakis. 2016.  Compositional Sequence Labeling Models for Error Detection in Learner Writing.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1181–1191.
  * Rei and Yannakoudakis (2017) Rei, Marek and Helen Yannakoudakis. 2017.  Auxiliary Objectives for Neural Error Detection Models.  In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 33–43.
  * Reznicek et al. (2012) Reznicek, Marc, Anke Ludeling, Cedric Krummes, and Franziska Schwantuschke. 2012\.  Das FalkoHandbuch. Korpusaufbau und Annotationen Version 2.0.
  * van Rijsbergen (1979) van Rijsbergen, Cornelis J. 1979.  _Information Retrieval_ , 2nd edition.
  * Rosen (2016) Rosen, Alexandr. 2016.  Building and using corpora of non-native Czech.  In _Proceedings of the 16th ITAT: Slovenskočeský NLP workshop (SloNLP 2016)_ , volume 1649 of _CEUR Workshop Proceedings_ , pages 80–87.
  * Rothe et al. (2021) Rothe, Sascha, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021.  A Simple Recipe for Multilingual Grammatical Error Correction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 702–707.
  * Rozovskaya et al. (2015) Rozovskaya, Alla, Houda Bouamor, Nizar Habash, Wajdi Zaghouani, Ossama Obeid, and Behrang Mohit. 2015.  The Second QALB Shared Task on Automatic Text Correction for Arabic.  In _Proceedings of the Second Workshop on Arabic Natural Language Processing_ , pages 26–35.
  * Rozovskaya and Roth (2010a) Rozovskaya, Alla and Dan Roth. 2010a.  Generating Confusion Sets for Context-Sensitive Error Correction.  In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_ , pages 961–970.
  * Rozovskaya and Roth (2010b) Rozovskaya, Alla and Dan Roth. 2010b.  Training Paradigms for Correcting Errors in Grammar and Usage.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 154–162.
  * Rozovskaya and Roth (2011) Rozovskaya, Alla and Dan Roth. 2011.  Algorithm Selection and Model Adaptation for ESL Correction Tasks.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 924–933.
  * Rozovskaya and Roth (2013) Rozovskaya, Alla and Dan Roth. 2013.  Joint Learning and Inference for Grammatical Error Correction.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 791–802.
  * Rozovskaya and Roth (2014) Rozovskaya, Alla and Dan Roth. 2014.  Building a State-of-the-Art Grammatical Error Correction System.  _Transactions of the Association for Computational Linguistics_ , 2:419–434.
  * Rozovskaya and Roth (2019) Rozovskaya, Alla and Dan Roth. 2019.  Grammar Error Correction in Morphologically Rich Languages: The Case of Russian.  _Transactions of the Association for Computational Linguistics_ , 7:1–17.
  * Rozovskaya and Roth (2021) Rozovskaya, Alla and Dan Roth. 2021.  How Good (really) are Grammatical Error Correction Systems?  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2686–2698.
  * Rozovskaya, Roth, and Srikumar (2014) Rozovskaya, Alla, Dan Roth, and Vivek Srikumar. 2014.  Correcting Grammatical Verb Errors.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 358–367.
  * Sakaguchi et al. (2016) Sakaguchi, Keisuke, Courtney Napoles, Matt Post, and Joel Tetreault. 2016.  Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality.  _Transactions of the Association for Computational Linguistics_ , 4:169–182.
  * Sakaguchi and Van Durme (2018) Sakaguchi, Keisuke and Benjamin Van Durme. 2018.  Efficient Online Scalar Annotation with Bounded Support.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 208–218.
  * Sanh et al. (2022) Sanh, Victor, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask Prompted Training Enables Zero-Shot Task Generalization.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_.
  * Šebesta (2010) Šebesta, Karel. 2010.  Korpusy češtiny a osvojování jazyka [Corpora of Czech and Language Acquistion].  _Studie z aplikované lingvistiky/Studies in Applied Linguistics_ , 1:11–34.
  * Sennrich, Haddow, and Birch (2016) Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016.  Improving Neural Machine Translation Models with Monolingual Data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96.
  * Shannon (1948) Shannon, Claude E. 1948.  A Mathematical Theory of Communication.  _The Bell System Technical Journal_ , 27(3):379–423.
  * Sorokin (2022) Sorokin, Alexey. 2022.  Improved grammatical error correction by ranking elementary edits.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11416–11429.
  * Specia et al. (2020) Specia, Lucia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André F. T. Martins. 2020.  Findings of the WMT 2020 Shared Task on Quality Estimation.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 743–764.
  * Stahlberg, Bryant, and Byrne (2019) Stahlberg, Felix, Christopher Bryant, and Bill Byrne. 2019.  Neural Grammatical Error Correction with Finite State Transducers.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4033–4039.
  * Stahlberg and Kumar (2020) Stahlberg, Felix and Shankar Kumar. 2020.  Seq2Edits: Sequence Transduction Using Span-level Edit Operations.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5147–5159.
  * Stahlberg and Kumar (2021) Stahlberg, Felix and Shankar Kumar. 2021.  Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models.  In _Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 37–47.
  * Sun et al. (2015) Sun, Chengjie, Xiaoqiang Jin, Lei Lin, Yuming Zhao, and Xiaolong Wang. 2015.  Convolutional Neural Networks for Correcting English Article Errors.  In _Proceedings of the 4th National CCF Conference on Natural Language Processing and Chinese Computing_ , pages 102–110.
  * Sun et al. (2021) Sun, Xin, Tao Ge, Furu Wei, and Houfeng Wang. 2021.  Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5937–5947.
  * Sun and Wang (2022) Sun, Xin and Houfeng Wang. 2022.  Adjusting the Precision-Recall Trade-Off with Align-and-Predict Decoding for Grammatical Error Correction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 686–693.
  * Susanto, Phandi, and Ng (2014) Susanto, Raymond Hendy, Peter Phandi, and Hwee Tou Ng. 2014.  System Combination for Grammatical Error Correction.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 951–962.
  * Sutskever, Vinyals, and Le (2014) Sutskever, Ilya, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to Sequence Learning with Neural Networks.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_. pages 3104–3112.
  * Syvokon and Nahorna (2021) Syvokon, Oleksiy and Olena Nahorna. 2021.  UA-GEC: Grammatical Error Correction and Fluency Corpus for the Ukrainian Language.  _arXiv_ , 2103.16997.
  * Tajiri, Komachi, and Matsumoto (2012) Tajiri, Toshikazu, Mamoru Komachi, and Yuji Matsumoto. 2012.  Tense and Aspect Error Correction for ESL Learners Using Global Context.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 198–202.
  * Takahashi, Katsumata, and Komachi (2020) Takahashi, Yujin, Satoru Katsumata, and Mamoru Komachi. 2020.  Grammatical Error Correction Using Pseudo Learner Corpus Considering Learner’s Error Tendency.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 27–32.
  * Tarnavskyi, Chernodub, and Omelianchuk (2022) Tarnavskyi, Maksym, Artem Chernodub, and Kostiantyn Omelianchuk. 2022.  Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3842–3852.
  * Taslimipoor, Bryant, and Yuan (2022) Taslimipoor, Shiva, Christopher Bryant, and Zheng Yuan. 2022.  Improving grammatical error correction for multiword expressions.  In _Proceedings of the 18th Workshop on Multiword Expressions @LREC2022,_ , pages 9–15.
  * Tetreault, Foster, and Chodorow (2010) Tetreault, Joel, Jennifer Foster, and Martin Chodorow. 2010.  Using Parse Features for Preposition Selection and Error Detection.  In _Proceedings of the ACL 2010 Conference Short Papers_ , pages 353–358.
  * Tetreault and Chodorow (2008) Tetreault, Joel R. and Martin Chodorow. 2008.  The Ups and Downs of Preposition Error Detection in ESL Writing.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pages 865–872.
  * Trinh and Rozovskaya (2021) Trinh, Viet Anh and Alla Rozovskaya. 2021.  New Dataset and Strong Baselines for the Grammatical Error Correction of Russian.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4103–4111.
  * Vaswani et al. (2017) Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , volume 30.
  * Wan, Wan, and Wang (2020) Wan, Zhaohong, Xiaojun Wan, and Wenguang Wang. 2020.  Improving Grammatical Error Correction with Data Augmentation by Editing Latent Representation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2202–2212.
  * Wang, Li, and Lin (2017) Wang, Chuan, Ruobing Li, and Hui Lin. 2017.  Deep Context Model for Grammatical Error Correction.  In _Proceedings of the 7th ISCA Workshop on Speech and Language Technology in Education (SLaTE 2017)_ , pages 167–171.
  * Wang et al. (2019) Wang, Yiren, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019.  Multi-Agent Dual Learning.  In _Proceedings of International Conference on Learning Representations_.
  * Wang et al. (2021) Wang, Yu, Yuelin Wang, Kai Dang, Jie Liu, and Zhuo Liu. 2021.  A Comprehensive Survey of Grammatical Error Correction.  _ACM Trans. Intell. Syst. Technol._ , 12(5).
  * Wei et al. (2022) Wei, Jason, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned Language Models are Zero-Shot Learners.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_.
  * Wei et al. (2021) Wei, Jason, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021.  Frequency Effects on Syntactic Rule Learning in Transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 932–948.
  * White and Rozovskaya (2020) White, Max and Alla Rozovskaya. 2020.  A Comparative Study of Synthetic Data Generation Methods for Grammatical Error Correction.  In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 198–208.
  * Wu et al. (2023) Wu, Haoran, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023.  ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark.  _arXiv_ , 2303.13648.
  * Xiang et al. (2013) Xiang, Yang, Bo Yuan, Yaoyun Zhang, Xiaolong Wang, Wen Zheng, and Chongqiang Wei. 2013.  A Hybrid Model For Grammatical Error Correction.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 115–122.
  * Xie et al. (2016) Xie, Ziang, Anand Avati, Naveen Arivazhagan, Dan Jurafsky, and Andrew Y. Ng. 2016\.  Neural Language Correction with Character-Based Attention.  _arXiv_ , 1603.09727.
  * Xie et al. (2018) Xie, Ziang, Guillaume Genthial, Stanley Xie, Andrew Ng, and Dan Jurafsky. 2018.  Noising and Denoising Natural Language: Diverse Backtranslation for Grammar Correction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 619–628.
  * Xu et al. (2019) Xu, Shuyao, Jiehao Zhang, Jin Chen, and Long Qin. 2019.  Erroneous data generation for Grammatical Error Correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 149–158.
  * Yannakoudakis, Briscoe, and Medlock (2011) Yannakoudakis, Helen, Ted Briscoe, and Ben Medlock. 2011.  A New Dataset and Method for Automatically Grading ESOL Texts.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 180–189.
  * Yannakoudakis et al. (2018) Yannakoudakis, Helen, Øistein E Andersen, Ardeshir Geranpayeh, Ted Briscoe, and Diane Nicholls. 2018.  Developing an automated writing placement system for ESL learners.  _Applied Measurement in Education_ , 31(3):251–267.
  * Yannakoudakis et al. (2017) Yannakoudakis, Helen, Marek Rei, Øistein E. Andersen, and Zheng Yuan. 2017.  Neural Sequence-Labelling Models for Grammatical Error Correction.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2795–2806.
  * Yasunaga, Leskovec, and Liang (2021) Yasunaga, Michihiro, Jure Leskovec, and Percy Liang. 2021.  LM-Critic: Language Models for Unsupervised Grammatical Error Correction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7752–7763.
  * Yasunaga and Liang (2021) Yasunaga, Michihiro and Percy Liang. 2021.  Break-It-Fix-It: Unsupervised Learning for Program Repair.  In _International Conference on Machine Learning (ICML)_.
  * Yoshimoto et al. (2013) Yoshimoto, Ippei, Tomoya Kose, Kensuke Mitsuzawa, Keisuke Sakaguchi, Tomoya Mizumoto, Yuta Hayashibe, Mamoru Komachi, and Yuji Matsumoto. 2013.  NAIST at 2013 CoNLL Grammatical Error Correction Shared Task.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 26–33.
  * Yoshimura et al. (2020) Yoshimura, Ryoma, Masahiro Kaneko, Tomoyuki Kajiwara, and Mamoru Komachi. 2020.  SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6516–6522.
  * Yu, Lee, and Chang (2014) Yu, Liang-Chih, Lung-Hao Lee, and Li-Ping Chang. 2014.  Overview of Grammatical Error Diagnosis for Learning Chinese as a Foreign Language.  In _Proceedings of the 22nd International Conference on Computers in Education_ , pages 42–47.
  * Yuan et al. (2022) Yuan, Xun, Derek Pham, Sam Davidson, and Zhou Yu. 2022.  ErAConD: Error Annotated Conversational Dialog Dataset for Grammatical Error Correction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 76–84.
  * Yuan (2017) Yuan, Zheng. 2017.  _Grammatical error correction in non-native English_.  Ph.D. thesis, University of Cambridge.
  * Yuan and Briscoe (2016) Yuan, Zheng and Ted Briscoe. 2016.  Grammatical error correction using neural machine translation.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 380–386.
  * Yuan, Briscoe, and Felice (2016) Yuan, Zheng, Ted Briscoe, and Mariano Felice. 2016.  Candidate re-ranking for SMT-based grammatical error correction.  In _Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 256–266.
  * Yuan and Bryant (2021) Yuan, Zheng and Christopher Bryant. 2021.  Document-level grammatical error correction.  In _Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 75–84.
  * Yuan and Felice (2013) Yuan, Zheng and Mariano Felice. 2013.  Constrained Grammatical Error Correction using Statistical Machine Translation.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 52–61.
  * Yuan et al. (2019) Yuan, Zheng, Felix Stahlberg, Marek Rei, Bill Byrne, and Helen Yannakoudakis. 2019\.  Neural and FST-based approaches to grammatical error correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 228–239.
  * Yuan et al. (2021) Yuan, Zheng, Shiva Taslimipoor, Christopher Davis, and Christopher Bryant. 2021\.  Multi-Class Grammatical Error Detection for Correction: A Tale of Two Systems.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8722–8736.
  * Zaghouani et al. (2015) Zaghouani, Wajdi, Nizar Habash, Houda Bouamor, Alla Rozovskaya, Behrang Mohit, Abeer Heider, and Kemal Oflazer. 2015.  Correction Annotation for Non-Native Arabic Texts: Guidelines and Corpus.  In _Proceedings of The 9th Linguistic Annotation Workshop_ , pages 129–139.
  * Zaghouani et al. (2014) Zaghouani, Wajdi, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014.  Large Scale Arabic Error Annotation: Guidelines and Framework.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 2362–2369.
  * Zhang (2009) Zhang, Baolin. 2009.  Features and functions of the HSK dynamic composition corpus (in Chinese).  In _International Chinese Language Education_ , pages 71–79.
  * Zhang et al. (2022a) Zhang, Susan, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022a.  OPT: Open Pre-trained Transformer Language Models.  _arXiv_ , 2205.01068.
  * Zhang et al. (2020) Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  BERTScore: Evaluating Text Generation with BERT.  In _International Conference on Learning Representations_.
  * Zhang et al. (2022b) Zhang, Yue, Zhenghua Li, Zuyi Bao, Jiacheng Li, Bo Zhang, Chen Li, Fei Huang, and Min Zhang. 2022b.  MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3118–3130.
  * Zhang et al. (2022c) Zhang, Yue, Bo Zhang, Zhenghua Li, Zuyi Bao, Chen Li, and Min Zhang. 2022c.  SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2518–2531.
  * Zhao et al. (2019) Zhao, Wei, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019.  Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 156–165.
  * Zhao et al. (2018) Zhao, Yuanyuan, Nan Jiang, Weiwei Sun, and Xiaojun Wan. 2018.  Overview of the NLPCC 2018 shared task: Grammatical error correction.  In _CCF International Conference on Natural Language Processing and Chinese Computing_ , pages 439–445.
  * Zhao and Wang (2020) Zhao, Zewei and Houfeng Wang. 2020.  MaskGEC: Improving Neural Grammatical Error Correction via Dynamic Masking.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(01):1226–1233.
  * Zhou et al. (2020) Zhou, Wangchunshu, Tao Ge, Chang Mu, Ke Xu, Furu Wei, and Ming Zhou. 2020.  Improving Grammatical Error Correction with Machine Translation Pairs.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 318–328.