# Preregistering NLP research

Emiel van Miltenburg  Chris van der Lee  Emiel Krahmer
Tilburg Center for Cognition and Communnication (TiCC)
Tilburg University
Tilburg, The Netherlands
<EMAIL_ADDRESS>
###### Abstract

Preregistration refers to the practice of specifying what you are going to do,
and what you expect to find in your study, before carrying out the study. This
practice is increasingly common in medicine and psychology, but is rarely
discussed in NLP. This paper discusses preregistration in more detail,
explores how NLP researchers could preregister their work, and presents
several preregistration questions for different kinds of studies. Finally, we
argue in favour of _registered reports_ , which could provide firmer grounds
for _slow science_ in NLP research. The goal of this paper is to elicit a
discussion in the NLP community, which we hope to synthesise into a general
NLP preregistration form in future research.

##  1 Introduction

Scientific results are only as reliable as the methods that we use to obtain
those results. Recent years have seen growing concerns about the
reproducibility of scientific research, leading some to speak of a
‘reproducibility crisis’ (see Fidler and Wilcox 2018 for an overview of the
debate). Although the main focus of the debate has been on psychology (e.g.
through Open Science Collaboration 2015) and medicine Macleod et al. (2014),
there are worries about the reproducibility of Natural Language Processing
(NLP) research as well Fokkens et al. (2013); Cohen et al. (2018); Moore and
Rayson (2018); Branco et al. (2020). The reproducibility debate has led to
Munafò et al.’s (2017) Manifesto for reproducible science, where the authors
discuss the different threats to reproducible science, and different ways to
address these threats. We will first highlight some of their proposals, and
discuss their adoption rate in NLP. Our main observation is that
preregistration is rarely used. We believe this is an undesirable situation,
and devote the rest of this paper to argue for preregistration of NLP
research.

Munafò et al. recommend more methodological training, so that e.g. statistical
methods are applied correctly. In NLP, we see different researchers picking up
the gauntlet to teach others about statistics Dror et al. (2018, 2020),
achieving language-independence Bender (2011), or best practices in human
evaluation van der Lee et al. (2019, 2021). Moreover, every *ACL conference
offers tutorials on a wide range of different topics. While efforts to improve
methodology could be more systematic (e.g. by actively encouraging methodology
tutorials, and working towards community standards),111A more radical proposal
would be to _always_ host methodology-focused tutorials, and to invite
researchers to teach specific modules, similar to keynote talks. the
infrastructure is in place.

Munafò et al. also recommend to diversify peer review. Instead of only having
journals, that are responsible for both the evaluation and dissemination of
research, we can now also solicit peer feedback after publishing our work on a
platform like ArXiv or OpenReview. The NLP community is clearly ahead of the
curve in terms of the adoption of preprints, and actively discussing ways to
improve peer review (ACL Reviewing Committee 2020a, b; Rogers and Augenstein
2020). To improve the quality of the reviews themselves, ACL2020 featured a
tutorial on peer reviewing Cohen et al. (2020).

Another advice from Munafò et al. is to adopt reporting guidelines, so that
papers include all relevant details for others to reproduce the results. The
NLP community is rapidly adopting such guidelines, in the form of Dodge et
al.’s (2019) reproducibility checklist that authors for EMNLP2020 need to fill
in. Beyond reproducibility, we are also seeing more and more researchers
adopting Data statements Bender and Friedman (2018), Model cards Mitchell et
al. (2019), and Datasheets Gebru et al. (2018) for ethical reasons.

Data collection |  Have any data been collected for this study already?
---|---
Hypothesis |  What’s the main question being asked or hypothesis being tested in this study?
Dependent variable |  Describe the key dependent variable(s) specifying how they will be measured.
Conditions |  How many and which conditions will participants be assigned to?
Analyses |  Specify exactly which analyses you will conduct to examine the main question/hypothesis.
Outliers and Exclusions |  Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
Sample Size |  How many observations will be collected or what will determine sample size?
Other |  Anything else you would like to pre-register?
Research aim |  Specify the overall aim of the research.
Use of literature |  Specify the role of theory in your research design.
Rationale |  Elaborate if your research is conducted from a certain theoretical perspective.
Tradition |  Specify the type of tradition you work in: grounded theory, phenomenology, …
Data collection plan |  Describe your data collection plan freely. Be as explicit as possible.
Type of data collected |  Select the type(s) of data you will collect.
Type of sampling |  Indicate the type of sampling you will rely on: purposive, theoretical, convenience, snowball…
Rationale |  Indicate why you choose this particular type of sampling.
Sort of sample |  Pick the ideal composition of your sample: heterogenous, homogenous, …
Stopping rule |  Indicate what will determine to stop data collection: saturation, planning, resources, other.
Data collection script |  Upload your topic guide, observation script, focus group script, etc.
Table 1: Top: preregistration form from AsPredicted (https://aspredicted.org)
for quantitative research. Bottom: additional items from Haven and Grootel
(2019) for qualitative research. All text is quoted verbatim.

Munafò et al.’s final recommendation, preregistration, means that authors
should specify what they are going to do, and what they expect to find, before
carrying out their studies Nosek et al. (2018). The goal of preregistration is
to ensure that all hypotheses and research methods are made explicit before
researchers are confronted with the data. Otherwise, researchers end up in a
garden of forking paths, where all research decisions are made implicitly,
based on common sense and the available data Gelman and Loken (2013). This
negatively impacts the reliability and generalisability of any study. In other
words: preregistration allows us to distinguish between exploratory and
confirmatory research. Exploratory research does not require preregistration,
because the goal is to get a sense of what is possible. Any pattern you come
across during exploratory research, allows us to draw up hypotheses. For a
subsequent confirmatory study you could/should preregister to test those
hypotheses. By explicitly marking (parts of) your study as exploratory or
confirmatory, it is easier to understand the status of your results.

Compared to the work on reporting quality, there has been little talk of
preregistration in the NLP literature; the terms ‘preregister’ or
‘preregistration’ are hardly used in the ACL Anthology.222Looking for these
terms, we found four papers that mention preregistration: Cao et al. (2018)
and van der Lee et al. (2019) mention it, and van Miltenburg et al. (2018) and
Futrell and Levy (2019) share their own preregistration. For this reason, we
will focus on preregistration and its application in NLP research. The next
sections discuss how preregistration works (§2), propose preregistration
questions for NLP research (§3), discuss the idea of ‘registered reports’ as
an alternative pathway to publication (§4) and the overall feasibility of
preregistrations in NLP (§5).

##  2 How does preregistration work?

Before you begin, you enter the hypotheses, design, and analysis plan of your
study on a website like the Open Science Framework, AsPredicted, or
ResearchBox. These sites provide a time stamp; evidence that you indeed made
all the relevant decisions before carrying out the study. During your study,
you follow the preregistered plans as closely as possible. In an ideal world,
there would be an exact match between your plans and the actual study you
carried out. But there are usually unforeseen circumstances that force you to
change your study. This is fine, if the changes are clearly specified
(including the reasons for those changes) in your final report Nosek et al.
(2018).

A typical preregistration form. Table 1 shows questions from the
preregistration form from AsPredicted.333See https://osf.io/zab38/wiki/home/
for an overview of different forms. This form is geared towards hypothesis-
driven, experimental research where human participants are assigned to
different experimental conditions. Simmons et al. (2017) note that answers
should state exactly how the study will be executed, but also that it should
be short and easy to read.

What are your hypotheses/key assumptions?
---
What is the independent variable? (e.g. model architecture)
What is the dependent variable (e.g. output quality)
How will you measure the dependent variable?
Is there just one condition (corpus/task), or more?
What parameter settings will you use?
What data will you use, and how is it split in train/val/test?
Why this data? What are key properties of the data?
How will you analyse the results and test the hypotheses?
Table 2: Questions for analysis, experiments, and reproduction papers
(expanded in Appendix A).

Data collection, hypothesis, dependent variable. The form first asks whether
data collection has been carried out yet (ideally the answer should be no, but
see Appendix §A.1), and then asks researchers to make their main hypothesis
explicit so that it cannot be changed after the fact. Following the
hypothesis, researchers should describe their key dependent variables (i.e.
the main outcome variables) and how they will be measured. This includes
cutoff points that will be used to discretise continuous variables (e.g. to
divide participants in different groups).

Conditions, analyses, outliers and exclusions. Next, the form asks about the
design of the study, the analyses, and the process of determining outliers
(and whether those should be excluded). The answer needs to be detailed enough
so that other researchers are able to reproduce the study.

Sample size _and_ other. The form then asks how much data will be collected,
so as to prevent optional stopping (where researchers keep collecting data
until the results are in line with their preferred hypothesis).444Although it
is not necessary for the form, at this point it is good to justify the sample
size, e.g. by using a power analysis. Finally, the form allows researchers to
specify other aspects of the study they would like to preregister, such as
“secondary analyses, variables collected for exploratory purposes, [or]
unusual analyses.”

Qualitative research. Preregistration is not only suitable for quantitative
research; Haven and Grootel (2019) present a proposal to preregister
qualitative studies as well. Their suggestions are also presented in Table 1.
The authors argue that, although qualitative research differs in its goals
from quantitative research (developing theories rather than testing them), it
is still valuable to make your assumptions and research plans explicit before
carrying out your planned study. Because qualitative research is more flexible
than quantitative research, Haven and Grootel view qualitative
preregistrations as living documents; continuously updated to track the
research progress. This stimulates conscientiousness, and avoids sloppy
research. Public preregistrations also allow for immediate feedback.

##  3 Preregistration in NLP research

To determine what a preregistration for NLP research should look like, we need
to consider the different kinds of research contributions in NLP. For this, we
use the paper types proposed for COLING 2018.555https://coling2018.org/paper-
types/ These are: Computationally-aided linguistic analysis; NLP engineering
experiment paper; Reproduction/Resource/Position/Survey Paper. Of these,
position papers are less suitable for preregistration, since these are more
opinion/experience-driven, and the process of writing them cannot be
formalised. We treat the others below.

What do you aim to learn from the error analysis?
---
What do you know from the literature about system errors?
What kinds of errors do you expect to find?
How will you sample the outputs to analyse?
Do you also consider the input in your sampling strategy?
How do you plan to analyse the output?
How many judges will assess the output? Are they trained?
How is the reliability of the judges assessed?
Is there a fixed error categorisation scheme or not?
Table 3: Questions to ask before an error analysis.

Analysis, experiments, and reproduction papers typically have one or more
hypotheses, even though they may not always be marked as such.666Taking the
best papers from COLING 2018 as an example, Ruppenhofer et al. (2018,
analysis) test assumptions from the linguistics literature about affixoids,
Thompson and Mimno (2018, experiment) test which subsampling methods improve
the output generated by topic models, and Lan and Xu (2018, reproduction) test
whether the reported performance for different neural network models
generalises to other tasks. This means we can ask many of the same questions
for these studies as for experimental research. Table 2 provides a rough
overview of important questions to ask before carrying out your research.

If your study contains an error analysis, then you could ask the more
qualitatively oriented questions in Table 3. They acknowledge that you always
enter error analysis with some expectation (i.e. researcher bias) of what
kinds of mistakes systems are likely to make, and where those mistakes may be
found. The questions also stimulate researchers to go beyond the practice of
providing some ‘lemons’ alongside cherry-picked examples showing good
performance.

The main benefit of asking these questions beforehand is that they force
researchers to carefully consider their methodology, and they make
researchers’ expectations explicit. This also helps to identify unexpected
findings, or changes that were made to the research design during the study.

Resource papers are on the qualitative side of the spectrum, and as such the
questions from Haven and Grootel (2019), presented at the bottom of Table 1,
are generally appropriate for these kinds of papers as well. Particularly 1)
the original purpose for collecting the data, 2) sampling decisions (what
documents to include), and 3) annotation (what framework/perspective to use)
are important. Because the former typically influences the latter two, it is
useful to document how the goal of the study influenced decisions regarding
sampling and annotation, in case the study at some point pivots towards
another goal.

Survey papers should follow the PRISMA guidelines for structured reviews Moher
et al. (2009); Liberati et al. (2009). According to these guidelines,
researchers should state exactly where they searched for existing literature,
what search terms they used, and what criteria they used to select relevant
papers. This increases reproducibility, allows readers to find any gaps in the
survey, and avoids a biased presentation of the literature (i.e. only citing
researchers you know, or work that fits your preferred narrative). A recent
NLP example of a structured review is provided by Reiter (2018).

##  4 Registered reports

Registered reports “[split] conventional peer review in half” Chambers (2019).
First, authors submit a well-motivated research plan for review, before
carrying out the study (similar to a preregistration). This plan may go back-
and-forth between the authors and the reviewers, but once the plan is
accepted, the authors receive the guarantee that, if they carry out the study
according to plan, their work will be published. As with preregistration,
deviations from the original plan are allowed, but these should be indentified
in the final report. The main advantage of registered reports is that they
provide a means to avoid publication bias. Because studies aren’t judged on
the basis of their results, positive results are equally likely to be
published as negative results. As long as the study is deemed valuable a
priori, it should get published. An additional benefit of registered reports
is that reviews may actually correct flaws in the research design, meaning
that we reduce the chance of running an expensive study all for nothing. In
the case of NLP research, this may save a lot of energy (cf. Strubell et al.
2019). We are not aware of any NLP journals that offer registered reports, but
strongly encourage the NLP community to take steps in this direction.777Cf.
Mannarswamy and Roy (2018) regarding AI research.

##  5 Feasibility

Gelman and Loken (2013, 2014) touch upon the feasibility of preregistration,
noting that: [leftmargin=]“[f]or most of our own research projects this
strategy hardly seems possible: in our many applied research projects, we have
learned so much by looking at the data. Our most important hypotheses could
never have been formulated ahead of time.”

This certainly rings true for NLP as well. However, we should be careful about
conclusions that are drawn on the basis of pre-existing data. Gelman and Loken
(2013) note that in such cases, if it is feasible to collect more data, it is
good to follow up positive results with a pre-registered replication to
confirm your initial findings. One way to do this is to collect and evaluate
your model on a new test set (cf. Recht et al. 2019). This tells us to what
extent trained models generalise to unseen data. Another idea could be to
preregister the human evaluation (or error analysis) of the model output.

We believe that preregistration, and especially registered reports, could ease
the pressure to publish as soon as possible. If your analysis plan is accepted
for publication, you can take as long as you want to actually carry out the
study, without having to worry about being scooped. This provides new
opportunities for slow science in NLP (also see Min-Yen Kan’s keynote at
COLING 2018).

##  6 Questions about preregistration

Below we address some common questions about preregistration. We thank our
anonymous reviewers for raising some of these questions.

Is preregistration more work? In our experience, preregistration adds little
overhead to a research project. Especially if a project requires approval by
an Institutional Review Board (IRB), you need to write a description along
similar lines anyway. For projects not requiring IRB approval, it is good
practice to provide a model card Mitchell et al. (2019), data sheet Gebru et
al. (2018) or data statement Bender and Friedman (2018) with your model or
resource. Given the ethical aspects of NLP research, it is advisable to
consider all dimensions of your study before you carry it out. Moreover,
preregistration is a good way to start writing the paper before carrying out
the research, a practice advocated by Eisner (2010) to maximise the impact of
your work. Finally, it may be more work to prepare a registered report, but
this comes with the benefit of having a pre-approved methodology. Once the
project is completed, reviewers will not reject your paper based on
methodological choices.

Should I worry about being scooped? There is no need to worry. We already
discussed registered reports, where research proposals are provisionally
accepted before data collection starts. Otherwise, this worry has been
addressed through the existence of both public and private preregistrations. A
researcher can choose to keep a preregistration private until the research is
completed. They can make their preregistration public whenever they like, for
example to invite feedback from the community. In addition, preregistrations
are also time-stamped, and you can use these time stamps during the review
phase to show that you have had these ideas before similar work was
published.888The public/private distinction has been implemented by both the
Open Science Foundation and AsPredicted.org. The Open Science Foundation
allows for a 4-year embargo, during which the preregistration is kept private.
Aspredicted allows for preregistrations to be private indefinitely.

What about citing preregistrations? In some regards, the discussion about
preregistrations is similar to the discussion about preprints (i.e. papers on
ArXiv), thus similar questions arise. Both preregistrations and published
studies are being cited. For example, medical journals like BMC Public Health
also publish study protocols (similar to preregistrations), without any
results, that are also cited by others (e.g. work using a similar protocol).

What should we do with concurrent work? It may of course happen that multiple
researchers have similar ideas around the same time. We believe that it is
still valuable to publish multiple independent studies with similar results.
Even if they don’t provide any new insights (which is rare), they do provide
evidence towards the robustness of the findings. Where and how those findings
should be published is a separate discussion.999However, if there is value in
publishing the ‘first’ paper, there is probably also value in publishing the
‘second’ one. The same holds for the question of whether both studies should
be cited; good scholarship considers _all_ the available evidence.

How should we teach preregistration? Preregistration is already being
incorporated into Psychology courses (see, for example, Blincoe and Buchert
2020). It is relatively straightforward to implement as part of student
research proposals during applied courses in NLP: specify what you plan to do
exactly, and what you expect to find. It is often useful for students to have
an explicit format to think through their research plans, to make sure that
they make sense.

##  7 Limitations

Although preregistration is offered as a solution to improve our work, it does
not solve all of our problems. van ’t Veer and Giner-Sorolla (2016) mention
three limitations: 1\. Flexibility. It may be difficult or infeasible for
authors to foresee all possible outcomes, and as such there may be gaps in the
preregistration, which still allow for flexibility in the analysis. 2\. Fraud.
There is no way to prevent fraudulent researchers from, e.g., creating
multiple preregistrations, or falsely ‘preregistering’ studies that were
already run. At some point we just have to trust each other to do the right
thing, but increased transparency does make it harder to commit fraud. 3\.
Applicability. Preregistration may not be possible for all kinds of studies.
As discussed above, it has mainly been developed for quantitative studies
(particularly experiments), and there are proposals for the preregistration of
qualitative research Haven and Grootel (2019), although we have yet to see
whether this idea will catch on. Finally, Szollosi et al. (2020) argue that,
although preregistration might offer greater transparency, it does not by
itself improve scientific reasoning and theory development. Since large parts
of NLP are pre-theoretical (we have observed effects but do not have any
theoretical explanations for why these effects occur), one might reasonably
argue that we should focus on theory development first, before we can carry
out any meaningful experiments.

##  8 Conclusion

We have discussed how preregistration could benefit NLP research, and how
different kinds of contributions could be preregistered. We have also proposed
an initial list of questions to ask before carrying out NLP research (and see
Appendix A for example preregistration forms). With this paper, we hope to
encourage other NLP researchers to consider preregistering their work, so that
they will no longer get lost in the garden of forking paths. Still, there is
no silver bullet to cure sloppy science. Although preregistration is certainly
helpful, it does not guarantee high-quality research, and we do need to stay
critical about preregistered studies, and the way they are carried out.

## Acknowledgments

Thanks to the anonymous reviewers for their constructive feedback, and to all
the #NLProc Twitter people for discussion.

## References

  * ACL Reviewing Committee (2020a) ACL Reviewing Committee. 2020a.  Acl rolling review proposal.  Archived by the Internet Archive on June 20, 2020.
  * ACL Reviewing Committee (2020b) ACL Reviewing Committee. 2020b.  Short-term reform proposals for acl reviewing.  Adopted by the ACL Exec on June 8 as an initial step to improve reviewing. Archived by the Internet Archive on October 30, 2020.
  * Bender (2011) Emily M. Bender. 2011.  On achieving and evaluating language-independence in NLP.  _Linguistic Issues in Language Technology_ , 6(3):1–26.
  * Bender et al. (2018) Emily M. Bender, Leon Derczynski, and Pierre Isabelle, editors. 2018.  _Proceedings of the 27th International Conference on Computational Linguistics_. Association for Computational Linguistics, Santa Fe, New Mexico, USA.
  * Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604.
  * Blincoe and Buchert (2020) Sarai Blincoe and Stephanie Buchert. 2020.  Research preregistration as a teaching and learning tool in undergraduate psychology courses.  _Psychology Learning & Teaching_, 19(1):107–115.
  * Branco et al. (2020) António Branco, Nicoletta Calzolari, Piek Vossen, Gertjan Van Noord, Dieter van Uytvanck, João Silva, Luís Gomes, André Moreira, and Willem Elbers. 2020.  A shared task of a new, collaborative type to foster reproducibility: A first exercise in the area of language science and technology with REPROLANG2020.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 5539–5545, Marseille, France. European Language Resources Association.
  * Cao et al. (2018) Xuân-Nga Cao, Cyrille Dakhlia, Patricia Del Carmen, Mohamed-Amine Jaouani, Malik Ould-Arbi, and Emmanuel Dupoux. 2018.  BabyCloud, a technological platform for parents and researchers.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Chambers (2019) Chris Chambers. 2019.  What’s next for registered reports?  _Nature_ , 573:187–189.
  * Cohen et al. (2018) K. Bretonnel Cohen, Jingbo Xia, Pierre Zweigenbaum, Tiffany Callahan, Orin Hargraves, Foster Goss, Nancy Ide, Aurélie Névéol, Cyril Grouin, and Lawrence E. Hunter. 2018.  Three dimensions of reproducibility in natural language processing.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Cohen et al. (2020) Kevin Cohen, Karën Fort, Margot Mieskes, and Aurélie Névéol. 2020\.  Reviewing natural language processing research.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_ , pages 16–18, Online. Association for Computational Linguistics.
  * Dodge et al. (2019) Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019\.  Show your work: Improved reporting of experimental results.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2185–2194, Hong Kong, China. Association for Computational Linguistics.
  * Dror et al. (2018) Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018.  The hitchhiker’s guide to testing statistical significance in natural language processing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1383–1392, Melbourne, Australia. Association for Computational Linguistics.
  * Dror et al. (2020) Rotem Dror, Lotem Peled-Cohen, Segev Shlomov, and Roi Reichart. 2020.  _Statistical Significance Testing for Natural Language Processing_.  Human Language Technologies. Morgan & Claypool.
  * Eisner (2010) Jason Eisner. 2010.  Write the paper first.  Advice published on Jason Eisner’s academic webpage, at Johns Hopkins University.
  * Fidler and Wilcox (2018) Fiona Fidler and John Wilcox. 2018.  Reproducibility of Scientific Results.  In Edward N. Zalta, editor, _The Stanford Encyclopedia of Philosophy_ , winter 2018 edition. Metaphysics Research Lab, Stanford University.
  * Fokkens et al. (2013) Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen, and Nuno Freire. 2013.  Offspring from reproduction problems: What replication failure teaches us.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1691–1701, Sofia, Bulgaria. Association for Computational Linguistics.
  * Futrell and Levy (2019) Richard Futrell and Roger P. Levy. 2019.  Do RNNs learn human-like abstract word order preferences? In _Proceedings of the Society for Computation in Linguistics (SCiL) 2019_ , pages 50–59.
  * Gebru et al. (2018) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2018.  Datasheets for Datasets.  In _Proceedings of the 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning_ , Stockholm, Sweden.
  * Gelman and Loken (2013) Andrew Gelman and Eric Loken. 2013.  The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time.  _Department of Statistics, Columbia University_.
  * Gelman and Loken (2014) Andrew Gelman and Eric Loken. 2014.  The statistical crisis in science: data-dependent analysis–a" garden of forking paths"–explains why many statistically significant comparisons don’t hold up.  _American scientist_ , 102(6):460–466.
  * Haven and Grootel (2019) Tamarinde L. Haven and Dr. Leonie Van Grootel. 2019.  Preregistering qualitative research.  _Accountability in Research_ , 26(3):229–244.  PMID: 30741570.
  * Howcroft et al. (2020) David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Kan (2018) Min-Yen Kan. 2018.  "research fast and slow".  Keynote presented at COLING 2018, Santa Fe, NM, USA. Slides available through http://bit.ly/kan-coling18.
  * Lan and Xu (2018) Wuwei Lan and Wei Xu. 2018.  Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3890–3902, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Liberati et al. (2009) Alessandro Liberati, Douglas G. Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C. Gøtzsche, John P. A. Ioannidis, Mike Clarke, P. J. Devereaux, Jos Kleijnen, and David Moher. 2009.  The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration.  _PLOS Medicine_ , 6(7):1–28.
  * Macleod et al. (2014) Malcolm R Macleod, Susan Michie, Ian Roberts, Ulrich Dirnagl, Iain Chalmers, John P A Ioannidis, Rustam Al-Shahi Salman, An-Wen Chan, and Paul Glasziou. 2014\.  Biomedical research: increasing value, reducing waste. _Lancet_ , 383(9912):101–104.
  * Mannarswamy and Roy (2018) Sandya Mannarswamy and Shourya Roy. 2018.  Evolving ai from research to real life – some challenges and suggestions.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pages 5172–5179. International Joint Conferences on Artificial Intelligence Organization.
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model Cards for Model Reporting.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* ’19_ , pages 220–229, Atlanta, GA, USA. ACM Press.
  * Moher et al. (2009) David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G. Altman, and The PRISMA Group. 2009.  Preferred reporting items for systematic reviews and meta-analyses: The prisma statement.  _PLOS Medicine_ , 6(7):1–6.
  * Moore and Rayson (2018) Andrew Moore and Paul Rayson. 2018.  Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for target dependent sentiment analysis.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1132–1144, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Munafò et al. (2017) Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. 2017.  A manifesto for reproducible science.  _Nature Human Behaviour_ , 1(1):0021.
  * Nosek et al. (2018) Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018\.  The preregistration revolution.  _Proceedings of the National Academy of Sciences_ , 115(11):2600–2606.
  * Open Science Collaboration (2015) Open Science Collaboration. 2015.  Estimating the reproducibility of psychological science.  _Science_ , 349(6251).
  * Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019.  Do ImageNet classifiers generalize to ImageNet? In _Proceedings of Machine Learning Research_ , volume 97, pages 5389–5400, Long Beach, California, USA. PMLR.
  * Reiter (2018) Ehud Reiter. 2018.  A structured review of the validity of BLEU.  _Computational Linguistics_ , 44(3):393–401.
  * Roettger (2020) Timo Roettger. 2020.  Preregistration in experimental linguistics: Applications, challenges, and limitations.  Preprint available at: https://psyarxiv.com/vc9hu/. DOI: https://doi.org/10.31234/osf.io/vc9hu.
  * Rogers and Augenstein (2020) Anna Rogers and Isabelle Augenstein. 2020.  What can we do to improve peer review in NLP? In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1256–1262, Online. Association for Computational Linguistics.
  * Ruppenhofer et al. (2018) Josef Ruppenhofer, Michael Wiegand, Rebecca Wilm, and Katja Markert. 2018.  Distinguishing affixoid formations from compounds.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3853–3865, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Shimorina and Belz (2021) Anastasia Shimorina and Anya Belz. 2021.  The human evaluation datasheet 1.0: A template for recording details of human evaluation experiments in nlp.
  * Simmons et al. (2017) Joe Simmons, Leif Nelson, and Uri Simonsohn. 2017.  How to properly preregister a study.  _Data Colada_.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Szollosi et al. (2020) Aba Szollosi, David Kellen, Danielle J Navarro, Richard Shiffrin, Iris van Rooij, Trisha Van Zandt, and Chris Donkin. 2020.  Is preregistration worthwhile?  _Trends in cognitive sciences_ , 24(2):94–95.
  * Thompson and Mimno (2018) Laure Thompson and David Mimno. 2018.  Authorless topic models: Biasing models away from known structure.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3903–3914, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021\.  Human evaluation of automatically generated text: Current trends and best practice guidelines.  _Computer Speech & Language_, 67:101151.
  * van der Lee et al. (2019) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019.  Best practices for the human evaluation of automatically generated text.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 355–368, Tokyo, Japan. Association for Computational Linguistics.
  * van Miltenburg et al. (2018) Emiel van Miltenburg, Ákos Kádár, Ruud Koolen, and Emiel Krahmer. 2018\.  DIDEC: The Dutch image description and eye-tracking corpus.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3658–3669, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * van ’t Veer and Giner-Sorolla (2016) Anna Elisabeth van ’t Veer and Roger Giner-Sorolla. 2016.  Pre-registration in social psychology—a discussion and suggested template.  _Journal of Experimental Social Psychology_ , 67:2–12.  Special Issue: Confirmatory.

##  Appendix A Preregistration forms

This appendix provides preregistration forms for different kinds of paper
types. These forms are preliminary, and they are mainly meant as a starting
point for discussions of preregistration in NLP. We are happy to admit that
there may be flaws in this appendix (either in the forms or in our reasoning).
Future work should investigate whether these forms are complete (i.e. limit
_researcher degrees of freedom_ as much as possible) and appropriate for
different kinds of NLP research.

###  A.1 Preface: data availability in NLP

Preregistration is a means to avoid hindsight bias, because you have to
specify your expectations upfront, when your perspective is not yet colored by
your experience with the data. But for NLP studies it is unclear what ‘the
data’ is. We can distinguish three kinds of data: 1. The
training/validation/test sets, 2. The model output, 3. Human judgments.

In an ideal situation, preregistration would occur before any kind of data has
been obtained. The problem is that this is often not the case; there are many
canonical datasets for which the data is publicly available. Of course one
could collect an additional test set (as we suggested above), but the
community often judges new approaches based on their performance for
established datasets. So what should we do? Still preregister! Arguably the
training, validation, and test data is usually not central to the work. What
matters is how a particular system performs. So even if we don’t usually find
ourselves in the ideal situation where none of the data is available yet, it
is typically fine to preregister your study if the train/eval/test data is
available but system outputs and evaluation scores are not. When authors are
transparent in their data sharing policy, we can reconstruct the timeline of
events before and after the preregistration, to see how much their knowledge
about the data may have influenced them.

###  A.2 Computationally aided linguistic analysis

This paper type corresponds to several different setups, ranging from
experiments with human subjects, to corpus analyses to see if particular
generalisations from the literature hold up. Preregistration has been
discussed from a linguistics perspective by Roettger (2020). For experiments
with human participants, readers may refer to the standard preregistration
forms from AsPredicted (see our Table 1), OSF, or the questions from
Roettger’s Figure 1.

For more corpus-oriented studies (e.g. Ruppenhofer et al. 2018), we should
consider a mix of the quantitative and qualitative questions from our Table 1.
Usually these kinds of studies do require some data collection, so authors
should ask:

  1. 1.

What is the goal of this study?

  2. 2.

What are the main questions/hypotheses?

  3. 3.

What kind of data will be collected?

  4. 4.

How will this data be collected?

  5. 5.

What sampling strategy will be used? Why?

  6. 6.

How much data are you planning to collect? (Is there any target or stopping
criterion?)

  7. 7.

How will the data be analysed?

    1. (a)

If automatic: what analysis tool will you use, and how will it be configured?

    2. (b)

If manual: what is the background of the annotators? How will you ensure
reliability and validity of the analysis?

  8. 8.

What statistical tests will be used, if any?

  9. 9.

Anything else you’d like to preregister?

###  A.3 NLP Engineering experiment paper

NLP engineering experiments are like experiments in the social sciences,
except that the subjects are NLP models and the performance data is model
output. So the standard social science questions do not need to be modified
that much to fit NLP experiments.

  1. 1.

What is the goal of your study?

  2. 2.

What are your hypotheses/key assumptions?

  3. 3.

What are the (in)dependent variables?

  4. 4.

How will these variables be measured?

  5. 5.

Is there just one condition, or more?

  6. 6.

What software libraries will you use?

  7. 7.

What hardware will you use?

  8. 8.

What parameter settings will you use?

  9. 9.

What data set will you use?

  10. 10.

If the data set does not already exist, see §A.6. If it does:

    1. (a)

How familiar are you with the data?

    2. (b)

To what extent are your hypotheses informed by yourself or others interacting
with this data? To what extent does this hinder the generalisability of your
approach?

    3. (c)

Are you planning to collect additional data to validate your approach?

  11. 11.

Why this data? What are its key properties?

  12. 12.

How is the data split in train/val/test?

  13. 13.

How will you analyse the results and test the hypotheses?

    1. (a)

If automatic: what metric(s) (including implementation) will you use, and how
will they be configured?

    2. (b)

If human judgments: see §A.8.1.

  14. 14.

Will you carry out an error analysis? If so, see §A.8.2.

  15. 15.

Anything else you’d like to preregister?

###  A.4 Position paper

Position papers typically do not need to be preregistered, since they often do
not provide any new data, but rely on the author’s experience. These kinds of
papers also usually signal that they are more opinionated than other kinds of
papers.

###  A.5 Reproduction paper

For a reproduction paper, the questions are a mix of the questions above
(§A.3) with reproduction-specific questions.

  1. 1.

What results do you aim to reproduce?

  2. 2.

What kinds of experiments does this involve?

  3. 3.

What is the goal?

    1. (a)

What constitutes a successful reproduction?

    2. (b)

What constitutes an unsuccessful reproduction?

    3. (c)

What is the margin of error?

  4. 4.

Do you expect to be successful? Why (not)?

  5. 5.

How are you planning to reproduce the original results?

    1. (a)

Will you use the same soft/hardware?

    2. (b)

Will you use the same data?

    3. (c)

Will you use the same codebase?

    4. (d)

If human participants are used: will you target the same demographic, and use
the same experimental settings?

    5. (e)

Will you contact the authors?

    6. (f)

How much time do authors have to respond to your queries?

    7. (g)

How much time/effort are you willing to spend?

  6. 6.

Will you carry out an error analysis? If so, see §A.8.2.

  7. 7.

Anything else you’d like to preregister?

###  A.6 Resource paper

It is at least a bit unexpected to promote preregistration for resource
papers. After all, if all you do is data collection, then there are no
hypotheses to test. But since the goal of this appendix is to provide a
starting point for discussion, we are taking the stance that no study is free
from biases or initial expectations. As such, it is useful to at least
document what you aim to collect, for what reasons, and how you are planning
to do so. Once the project is completed, if you

  1. 1.

What is the goal of this study?

  2. 2.

What kind of data will be collected?

  3. 3.

How will this data be collected?

  4. 4.

What is the intended application for the data you plan to collect?

  5. 5.

What sampling strategy will be used? Why?

  6. 6.

How much data are you planning to collect? (Is there any target or stopping
criterion?)

  7. 7.

How will the data be analysed?

    1. (a)

If automatic: what analysis tool will you use, and how will it be configured?

    2. (b)

If manual: what is the background of the annotators? How will you ensure
reliability and validity of the analysis?

  8. 8.

What properties should the data have?

  9. 9.

How will you ensure that the data will have those properties?

  10. 10.

Anything else you’d like to preregister?

###  A.7 Survey paper

We would recommend that authors follow the PRISMA guidelines Moher et al.
(2009); Liberati et al. (2009)) for their surveys. This requires authors to
develop a review protocol, which means authors should answer the following
questions before initiating their study:

  1. 1.

What is the goal of this study?

  2. 2.

What is the rationale behind this study?

  3. 3.

What questions do you hope to answer?

  4. 4.

What types of articles are relevant to answer your question?

    1. (a)

What are the inclusion criteria?

    2. (b)

What are the exclusion criteria?

    3. (c)

What languages are included?

  5. 5.

How will you decide which articles are relevant? (e.g. judging by the
title/abstract)

  6. 6.

What search engines will you use?

  7. 7.

What search queries will you use?

  8. 8.

What are the variables of interest?

  9. 9.

How will you synthesize the results?

  10. 10.

How will you ensure the reliability and validity of your study?

  11. 11.

Anything else you’d like to preregister?

###  A.8 Other kinds of preregistrations

Instead of preregistering a full study, one might also preregister part of a
study, e.g. a human evaluation or error analysis.

####  A.8.1 Human evaluation

Human evaluation studies often do not report all the necessary details to
reproduce their work Howcroft et al. (2020). Thus Shimorina and Belz (2021)
developed a datasheet for recording all the necessary details. This datasheet
can mostly be filled in before the study is carried out. A selection of their
questions is provided below (see the paper for more details and additional
questions).

  1. 1.

What type of input(s) does the system have?

  2. 2.

What type of output does the system produce?

  3. 3.

What task is the system supposed to carry out?

  4. 4.

What languages are involved?

  5. 5.

How many systems/outputs per system are being evaluated?

  6. 6.

How are the outputs selected?

  7. 7.

What is the statistical power of the sample size?

  8. 8.

What kind of evaluators are being used?

  9. 9.

What training is given to the evaluators?

  10. 10.

What is the background if the evaluators?

  11. 11.

How are responses collected?

  12. 12.

What quality assurance measures are used?

  13. 13.

What do evaluators see when carrying out evaluations?

  14. 14.

How free are evaluators regarding when and how quickly they are supposed to
evaluate the results.

  15. 15.

Can evaluators provide feedback or not?

  16. 16.

What are the experimental conditions like?

  17. 17.

What type of quality is assessed in the evaluation?

  18. 18.

How is this quality assessed?

  19. 19.

How are the responses processed?

  20. 20.

What are the ethical implications of your work?

####  A.8.2 Error analysis

Error analysis is similar to human evaluation, except that it is typically
more qualitatively oriented. This does not mean that there cannot be a
quantitative component (e.g. counting the number of errors, comparing this
number between different systems), but often systems are also just analysed by
themselves, and we just want to know what future researchers still ought to
improve about the system.

  1. 1.

What is the goal of the error analysis?

  2. 2.

What type of input(s) does the system have?

  3. 3.

What type of output does the system produce?

  4. 4.

What task is the system supposed to carry out?

  5. 5.

What languages are involved?

  6. 6.

What do you know from the literature about system errors?

  7. 7.

When does something count as an error?

  8. 8.

What kinds of errors do you expect to find?

  9. 9.

How many outputs will you analyse?

  10. 10.

How will you sample the outputs to analyse?

  11. 11.

Do you also consider the input in your sampling strategy?

  12. 12.

How do you plan to analyse the output?

  13. 13.

How many judges will assess the output?

  14. 14.

What is the background of the judges?

  15. 15.

What training do the judges receive?

  16. 16.

How is the reliability of the judges assessed?

  17. 17.

How will their responses be processed?

  18. 18.

Is there a fixed error categorisation scheme or not?