# Evaluation for Change

Rishi Bommasani
Stanford University
<EMAIL_ADDRESS>
###### Abstract

Evaluation is the central means for assessing, understanding, and
communicating about NLP models. In this position paper, we argue evaluation
should be more than that: it is a force for driving change, carrying a
sociological and political character beyond its technical dimensions. As a
force, evaluation’s power arises from its adoption: under our view, evaluation
succeeds when it achieves the desired change in the field. Further, by framing
evaluation as a force, we consider how it competes with other forces. Under
our analysis, we conjecture that the current trajectory of NLP suggests
evaluation’s power is waning, in spite of its potential for realizing more
pluralistic ambitions in the field. We conclude by discussing the legitimacy
of this power, who acquires this power and how it distributes. Ultimately, we
hope the research community will more aggressively harness evaluation for
change.

##  1 Introduction

Evaluation plays a defining role in NLP research; in fact, evaluation has a
very rich history. While this genealogy can be traced in many ways, since this
piece (roughly) coincides with the 5th anniversary of the passing of one NLP’s
beloved pioneers and the first recipient of the ACL Lifetime Achievement
Award, we look to Aravind Joshi’s legacy. Best known for grammar formalism and
discourse (see Webber, 2018), his research journey reflects broader field-wide
trends towards evaluation. In early works (e.g. Joshi, 1969; Joshi et al.,
1972; Grosz et al., 1983), evaluation went entirely unmentioned. Yet, over
time, Aravind’s work involved more evaluation (e.g. Joshi and Schabes, 1989),
implicitly building new norms for evaluation in grammar formalism and
discourse (Miltsakaki et al., 2004; Prasad et al., 2008, 2014). Liberman
(2005) cites Joshi’s standards for evaluation in conveying Joshi’s signature
belief in multidisciplinary approaches to human language.

Joshi’s life and 5 decades of scholarship teaches us evaluation is important,
but how should we reason about evaluation? Here, we present two perspectives
that frame evaluation in considerably different ways. Under the first account,
evaluation is technical in nature, functioning as a lens to study models. The
motivation for this lens may depend on the specific evaluation, stakeholder,
or both: evaluation may allow us to derive scientific insight. Or it can
transparently document technology for broader audiences (e.g. practitioners,
colleagues in other fields, policymakers, the public). Regardless, to
determine if an evaluation is successful, under this account, the lens must
yield the desired understanding about models.

In this work, we argue for a second perspective, which we believe is partially
acknowledged but considerably less salient than the first perspective. Under
our second account, evaluation is political in nature, functioning as a force
to drive change. In contrast to the first account, this means evaluation
pushes the research community in some direction, possibly referring to a
specific social or scientific objective, with the emphasis being on future
model development more so than existing models. Critically, under this
account, to determine if an evaluation is successful, the force must yield the
desired change in the community. By separating these two accounts, our goal is
neither to suggest they are at odds nor that they are meaningfully separable,
but to shed conceptual clarity on the merits of power-centric analysis.

In pushing for this position of viewing evaluation as a force, we explore what
this force influences, what other forces it competes with, how it accrues
power, whether its power is legitimate, and who it empowers. Motivated by the
growing impact of language technology and our field, the abundant discord on
the status quo, and the uncertainty on what lies ahead, we believe
evaluation’s potential for change presents a vital path forward.

##  2 Evaluation as a Force

If evaluation is a force, what domain does it act upon? And where does its
power come from?

### Domain.

We will restrict our scope to how evaluation influences NLP research.
Specifically, evaluation concretizes desired behavior for systems, thereby
communicating an objective for model design. This allows for the community to
coordinate on goals for modeling research. For this goal-setting to succeed,
future research should then go on to make progress on the proposed evaluation.
That is, successful evaluation requires that the evaluation be prioritized,
redistributing research attention such that it is allocated towards making
progress on the evaluation.

### Adoption constructs power.

As this suggests, the adoption of an evaluation (by others) generates its
power and determines its success. It is in this sense that our account for
evaluation success deviates from a purely technical/intrinsic
characterization. Most evaluations are concrete instantiations of a broader
agenda: for these evaluations to be effective, they must shift power, namely
towards addressing this agenda and materially making progress. In spite of
this, we generally find that evaluations in NLP research do not even mention
how adoption will arise, and if evaluation creators will take any overt
actions to accelerate adoption.

### Accelerating adoption.

If the power of evaluations come from adoption, and evaluation creators are
incentivized to accrue such power to advance their broader agenda, are there
ways to accelerate adoption? We observe at least two such approaches, though
they have not been considered in this way to our knowledge. As a softer means
for acquiring adoption/power, evaluations may be used as shared tasks (e.g.
SemEval; see Parra Escartín et al., 2017; Nissim et al., 2017) or be built as
part of workshops/conferences (e.g. BIG-bench; see WELM, 2021; Srivastava et
al., 2022), which leans into the relationship between coordinating research
and convening researchers. More aggressively, explicit competitions with
prizes or other stronger incentives can more directly drive adoption, perhaps
most famously in the Netflix Prize, which remarkably accelerated and shifted
research on recommender systems (see Hallinan and Striphas, 2016).

### Authority as a standard.

As evaluations accrue influence, they eventually become reified as high-status
standards like ImageNet, WMT, and SQuAD (Dotan and Milli, 2020; Dehghani et
al., 2021). While it is difficult to directly assess the power these
evaluations have (e.g. how would research have changed counterfactually in
their absence; see Liu et al., 2021), strong norms emerge for modeling work to
evaluate on these standards. And, consequently, improvements on these
evaluations function as stand-ins for more fundamental progress (Liao et al.,
2021; Raji et al., 2021). In fact, their authority is made clear in how
serious improvements were seen as watershed moments, ushering in new
paradigms. Famous examples include the performance of AlexNet (Krizhevsky et
al., 2012) on ImageNet, which initiated the deep learning revolution, and
Transformers (Vaswani et al., 2017) on WMT, which, by outperforming
specialized machine translation approaches by a considerable margin, marked
the dawn of the current dominance of Transformers.

### Related work.

This work is not the first to bring questions of power, values, reflection,
and change to the fore in relation to evaluation/benchmarking (Spärck Jones
and Galliers, 1995; Welty et al., 2019; Dotan and Milli, 2020; Ethayarajh and
Jurafsky, 2020; Linzen, 2020; Scheuerman et al., 2021; Kiela et al., 2021;
Dehghani et al., 2021; Bowman and Dahl, 2021; Raji et al., 2021; Koch et al.,
2021; Denton et al., 2021; Paullada et al., 2021; Liu et al., 2021; Hutchinson
et al., 2021; Jacobs and Wallach, 2021; Birhane et al., 2022; Liang et al.,
2022b, inter alia). Prior work establishes that evaluations embed values,
carry influence, encode broader power structures, and the nature of evaluation
as ranking aligns with broader themes of hierarchy. They make clear how other
disciplines can provide guidance on what we see in NLP, but also how our
evaluation practices are distinctive (e.g. competitive tendencies in in
benchmarking, differences in standards for measure validity).

While we draw significant inspiration from these works, our work also
significantly diverges in its objective. Rather than trying to make visible
the tacit assumptions, norms, and infrastructure that animate evaluation’s
power, we instead set our sights on how evaluation’s power can animate change.
In this regard, our work more closely mirrors the aesthetic of Abebe et al.
(2020), as can be seen in the similar titles.

##  3 Competing Forces

Having argued for where evaluation draws power from, how powerful is it? While
difficult to state in absolute terms, we instead consider what other forces
are in play and how they interact/compete.

### Coexisting forces.

NLP research is a fabric stitched through myriad social interactions:
conversations with colleagues, talks at conferences, academic Twitter,
scholarship from adjacent disciplines, and much more. In our view, most of
these interactions are poorly conceptualized as forces: while they exert
influence, they are generally diffuse rather than concentrated and lack strong
directionality. For this short-form analysis, we juxtapose evaluation with the
force of resources. By resources, we refer to assets like money, compute, and
engineering support, choosing to treat them as monolithic (rather than
disaggregating) for this short-form analysis.

### Language models.

Given the central position language models occupy in modern NLP, we take
language models as a case study to consider the forces of evaluation and
resources. Our thesis is resources, to a far greater extent than evaluation,
govern research on language models, with second-order effects for broader NLP
research given the pervasive dependence on language models in the field at
present. Beyond the fact that influential language models have near-
exclusively been developed by resource-rich institutions, resource-allocation
mindsets appear to drive decision-making in their development. Namely, the use
of scaling laws (e.g. Kaplan et al., 2020; Hoffmann et al., 2022) suggests
development is framed as an efficient resource allocation problem. Evaluation
does play a small role here: scaling laws relate resources (x-axis) with
evaluated model performance (y-axis). But the evaluation scope is narrow:
scaling laws generally center accuracy for a single task (generally upstream
language model perplexity), with predictability of this relationship being the
principal concern (Ganguli et al., 2022, cf. Wei et al. (2022)).

In contrast, we argue evaluation does not currently exert similar overarching
influence for language model development. While influential language models
generally come from resource-rich institutions, they generally do not report
evaluations on the same benchmarks. Liang et al. (2022b) make this precise,
showing that 30 prominent language models only evaluate on 17.9% of the
scenarios they introduce in HELM. In fact, even for datasets that are most
widely evaluated on across language modeling works at the time, Liang et al.
(2022b) find that RTE is the unique dataset evaluated for in more than 50% of
the 32 works they consider that introduce language models (e.g. GPT-3, GPT-
NeoX, BLOOM, PaLM, Gopher, OPT, GLM).111This is likely a direct side effect of
RTE being the unique dataset in both GLUE and SuperGLUE (Wang et al., 2019a,
b). Given this status quo (i.e. language models not even being evaluated on
the same benchmarks), evaluation currently fails to achieve the widespread
adoption to drive change, though recent high-profile evaluation efforts (e.g.
BIG-bench, the HuggingFace Evaluate library, HELM) may change this.

### Contrasting properties.

Which forces orient NLP research is material: different forces profile
differently. Resources are distributed very unevenly, so resources orienting
progress implies a narrow and particular subset of the community expresses
out-sized influence in shaping the field’s trajectory. Further, by the nature
of how these resource disparities came to be, these resource-rich actors tend
to have specific incentives (e.g. commercial interest) and demographics (e.g.
poor diversity), potentially causing them to advance more conservative
agendas. In contrast, we believe evaluation structurally is better equipped to
enable broader participation (e.g. BIG-bench) and, critically, pluralism.
Different values can be simultaneously foregrounded in evaluations (e.g. HELM
(Liang et al., 2022b) highlights values/desiderata such as accuracy,
robustness, fairness, uncertainty, and efficiency). For example, insofar as
scaling laws drive language model development, greater pluralism would be
achieved if scaling laws were studied, fit, and applied for a broader array of
evaluation targets than just upstream accuracy/perplexity.

##  4 Legitimacy

Since evaluation accrues power, is this power legitimate? And who does this
power distribute to?

### Legitimacy.

As Liang et al. (2022b) note, evaluations are generally built by a small
number of researchers, but then potentially go on to orient work in the
broader research community. Consequently, in arguing for the greater use of
evaluation as a means for shifting power, we should question whether this
implicitly recommends value imposition: imposing values of the few onto the
many. However, we further recognize that the power of an evaluation is not
principally determined by the evaluation creators, but is instead more
directly contingent on the evaluation’s adoption. Consequently, for this power
to emerge requires the consensual action of the early adopters, who choose to
use the evaluation. To an extent, this (voluntary) choice suggests that the
power of evaluation is generally and, at least, initially legitimate.222As a
matter of scope, we do not broach whether other forces in NLP research are
similarly legitimate.

If the power of evaluation is legitimate, then what does this imply when
evaluations are shown to have issues with respect to their validity,
reliability, relevance, or appropriateness (Gururangan et al., 2018; Kaushik
and Lipton, 2018; Ethayarajh, 2020; Blodgett et al., 2021; Aribandi et al.,
2021; Birhane and Prabhu, 2021, inter alia)? Here, we recognize that while the
initial adoption of an evaluation is in most cases clearly legitimate, the
subsequent sustained adoption can be more complicated. In particular, we
emphasize that evaluations tend to exhibit strong inertia: once an evaluation
is widely adopted, it is hard for the evaluation to lose this adoption or for
other evaluations to eclipse it (e.g. due to reviewing norms; Dehghani et al.,
2021), even when there are strong reasons to demote or deprecate the
evaluation (Peng et al., 2021). Most directly, we point to the strong norms of
comparison in NLP, whereby model developers are expected to compare their
models to prior models in head-to-head comparisons. While generally a useful
norm, this does promote a certain conservatism, especially when prior models
(i.e. those that are to be compared to) are not public or laborious to re-
evaluate on new datasets, as new models can most easily be compared to old
models on the evaluations used in prior work. In this regard, paradigms where
evaluations are continuously updated and refreshed (e.g. the evaluation rounds
in ANLI (Nie et al., 2020) and versions in HELM (Liang et al., 2022b);
inherently dynamic evaluations like DynaBench (Kiela et al., 2021)) may more
directly ensure the sustained power of specific evaluations is legitimate.

### Distribution of power.

Even if an evaluation’s power is acquired legitimately, we should further
question how the power distributes over different members of the community,
especially as other forces (especially resources) are inequitably distributed.
Koch et al. (2021) show the distribution of evaluation developers is also
uneven, aligning strongly with institutional privilege (e.g. elite academic
institutions like Stanford and Princeton, massive commercial organizations
like Microsoft and Google). In part, this is likely a byproduct of the fact
that evaluations themselves can be quite resource-intensive, especially when
this scale is a virtue: ImageNet (Deng et al., 2009), especially for its time,
was exceedingly costly in both money and time; large-scale model evaluation on
HELM costs almost $40$k USD in addition to $20$k A100 GPU hours (Liang et al.,
2022b).

With that said, we have significant optimism that evaluation can realize more
pluralistic visions. Specifically, (i) the rise of foundation models in NLP
has shifted the field towards few-shot evaluations (Brown et al., 2020; Bragg
et al., 2021), which means evaluations need not include large-scale training
subsets which constituted much of the cost for evaluations historically (e.g.
80%, or 80k+, of the examples in SQuAD (Rajpurkar et al., 2016) were allocated
for training). This suggests that their development should be more broadly
accessible (Bommasani et al., 2021, §4.4), though the dynamics of their
adoption are less clear. Further, (ii) the practice of community-driven
evaluation design has been successfully implemented in several instances: the
EleutherAI LM Harness Gao et al. (2021), GEM (Gehrmann et al., 2021), GEMv2
(Gehrmann et al., 2022), BIG-Bench (Srivastava et al., 2022), the Hugging Face
Evaluate library (von Werra et al., 2022), with examples like Universal
Dependencies (UD; Nivre et al., 2016; de Marneffe et al., 2021) even pre-
dating them for many years. In most cases, these efforts did not push a very
clear directional change/agenda in research priorities (UD as a partial
exception), but we believe future efforts could more explicitly exert power
while learning from these prior efforts. Finally, (iii) the community has
grown to more properly recognize and value evaluation-type contributions (e.g.
the NeurIPS datasets and benchmarks track, cf. Rogers (2020)). That is, while
we argue evaluation’s power is currently waning relative to resources,
suggesting a trend towards less pluralism, we simultaneously believe the
conditions are ripe for renewed commitment to evaluation to reverse this
trajectory.

##  5 Conclusion

Evaluation wields power: we believe the community is largely aware of this,
yet we foreground this power to understand how evaluation drives change. This
perspective leads us to three conclusions: (i) adoption imbues evaluation with
its power, (ii) evaluation’s power relative to other competing social forces
appears to be diminishing, and yet (iii) evaluation has attractive qualities,
especially under current conditions, as a force for change relative to other
forces with growing power.

Overall, we hope the community reflects on the mantra "evaluation for change"
and the Gandhian maxim "build evaluations for the change you wish to see in
the world".333Remark: The associated quote that this riffs off of was never
actually said by Mahatma Gandhi, in spite of this popular misconception. But
we believe its essence aligns well with Gandhi’s character nonetheless.

## Limitations

This work puts forth a position: by the nature of a position paper, the work
is deliberately intended to be evocative and opinionated, in some places not
having unequivocal evidence for certain claims. This presents a clear
limitation: the analysis presented may diverge from the realities of NLP at
present or in the future, namely if the assumptions/conditions presented
themselves prove to be untrue in practice. Nonetheless, we believe centering
power and change, and understanding evaluation as a political and sociological
phenomenon, is likely to be useful under all conditions.

Further, in understanding the qualities of evaluation relative to other social
forces, we directly suggest that evaluation is more readily operationalized in
more pluralistic ways than other key forces (primarily resources). While
initial efforts indicate the potential for such holistic approaches that
reflect many different desiderata (Liang et al., 2022b) as well as
participatory approaches that permit contribution from different entities
(e.g. Srivastava et al., 2022), it is still unclear how much adoption such
approaches will get, and therefore how much power they will acquire. That is,
the extent to which evaluation can realize this pluralistic vision still
largely remains an unresolved aspiration than a readily realizable certainty.
And, conversely, we do note that while current practices potentially put
pluralism and resources at odds, they may be mutually compatible in other
regimes (e.g. decentralized training through the pooling of shared/volunteered
compute (Yuan et al., 2022), open-source software development (Wolf et al.,
2020; Gao et al., 2021; von Werra et al., 2022)).

Finally, we do not discuss other forces that we believe have not exhibited
strong influence on NLP research thus far, in favor of allocating focus to
evaluation and resources, which have had clear influence. To enumerate some of
these other (potential) forces, we specifically note (i) research norms, (ii)
policy and regulation, and (iii) auditing/advocacy. For (i), we note that
while the NLP research community has many established norms (e.g.
reproducibility checklists, peer review guidelines, conference organization
structure, policies on respectful conduct), most of these do not
directly/significantly influence what research topics different researchers
work on. We do note that is possible in the future that certain norms (e.g.
the access to training data or model checkpoints; Liang et al., 2022a) would
influence what research is conducted (e.g. we may have not seen as much work
on the learning dynamics of language models and/or memorization of training
data due to the relative inaccessibility of intermediary checkpoints and
training data until recently). For (ii), we note that policy and regulatory
efforts have had little to no salient impact on the deployment of most
language technologies, let alone NLP research, to our knowledge. With that
said, much as efforts like GDPR and privacy legislation has impacted
scientific research on privacy (e.g. work that operationalizes the right to be
forgotten as in Ginart et al., 2019), similar trends could occur in NLP
research (e.g. in response to the EU AI Act). Akin to (ii), for (iii), we also
have seen fairly little impact from auditing/advocacy work on NLP research to
our knowledge. But, much as work on auditing/advocacy around face recognition
(Buolamwini and Gebru, 2018; Raji and Buolamwini, 2019; Raji et al., 2020,
inter alia) influenced research in the computer vision community, we could see
similar trends in NLP (e.g. in response to auditing/advocacy intervention
around language models).

## Ethics Statement

We do not find serious risks or ethical concerns with this work. We do note
this work advances a specific position, which we clearly identify. It should
not be assumed there is consensus in the community (or beyond) on any account
for evaluation, let alone the account on power that we espouse. In this
regard, we actively solicit response and interrogation of the positions
presented in this work, especially given myriad relevant analyses of
evaluation/measurement/benchmarking exist in other parts of AI, computer
science, linguistics, and other disciplines.

## Acknowledgements

I would like to thank Alex Hanna, Ben Recht, Chris Manning, Chris Potts,
Claire Cardie, Dan Jurafsky, Deb Raji, Emily Denton, Henrik Kugelberg, Jacob
Andreas, Jacob Steinhardt, John Hewitt, Judy Shen, Kawin Ethayarajh, Nelson
Liu, Rediet Abebe, Rohan Taori, Sam Bowman, Stella Biderman, Tal Linzen, Yann
Dubois, and Yoav Goldberg for being specific inspirations, whose writings and
thoughts helped develop my current position on evaluation, with special thanks
to Percy Liang. I would like to thank the CRFM community; the experience of
designing and building HELM (Liang et al., 2022b) in particular helped sharpen
my belief in this philosophy towards evaluation. I am supported by the NSF
Graduate Research Fellowship Program under grant number DGE-1655618.

## References

  * Abebe et al. (2020) Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G. Robinson. 2020.  Roles for computing in social change.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , FAT* ’20, page 252–260, New York, NY, USA. Association for Computing Machinery.
  * Aribandi et al. (2021) Vamsi Aribandi, Yi Tay, and Donald Metzler. 2021.  How reliable are model diagnostics? In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1778–1785, Online. Association for Computational Linguistics.
  * Birhane et al. (2022) Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022.  The values encoded in machine learning research.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22, page 173–184, New York, NY, USA. Association for Computing Machinery.
  * Birhane and Prabhu (2021) Abeba Birhane and Vinay Uday Prabhu. 2021.  Large image datasets: A pyrrhic win for computer vision? In _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1536–1546.
  * Blodgett et al. (2021) Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021.  Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1004–1015, Online. Association for Computational Linguistics.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher Rè, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.  On the opportunities and risks of foundation models.  _ArXiv_.
  * Bowman and Dahl (2021) Samuel R. Bowman and George Dahl. 2021.  What will it take to fix benchmarking in natural language understanding? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4843–4855, Online. Association for Computational Linguistics.
  * Bragg et al. (2021) Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021.  FLEX: Unifying evaluation for few-shot NLP.  In _Advances in Neural Information Processing Systems_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ , volume 81 of _Proceedings of Machine Learning Research_ , pages 77–91. PMLR.
  * de Marneffe et al. (2021) Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021.  Universal Dependencies.  _Computational Linguistics_ , 47(2):255–308.
  * Dehghani et al. (2021) Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. 2021.  The benchmark lottery.  _ArXiv_ , abs/2107.07002.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255.
  * Denton et al. (2021) Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, and Hilary Nicole. 2021\.  On the genealogy of machine learning datasets: A critical history of imagenet.  _Big Data & Society_, 8(2):20539517211035955.
  * Dotan and Milli (2020) Ravit Dotan and Smitha Milli. 2020.  Value-laden disciplinary shifts in machine learning.  _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_.
  * Ethayarajh (2020) Kawin Ethayarajh. 2020.  Is your classifier actually biased? measuring fairness under uncertainty with bernstein bounds.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2914–2919, Online. Association for Computational Linguistics.
  * Ethayarajh and Jurafsky (2020) Kawin Ethayarajh and Dan Jurafsky. 2020.  Utility is in the eye of the user: A critique of NLP leaderboards.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4846–4853, Online. Association for Computational Linguistics.
  * Ganguli et al. (2022) Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, and Jack Clark. 2022.  Predictability and surprise in large generative models.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22, page 1747–1764, New York, NY, USA. Association for Computing Machinery.
  * Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021.  A framework for few-shot language model evaluation.  _Version v0. 0.1. Sept_.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021.  The GEM benchmark: Natural language generation, its evaluation and metrics.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 96–120, Online. Association for Computational Linguistics.
  * Gehrmann et al. (2022) Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna V. Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh D. Dhole, Khyathi Raghavi Chandu, Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qinqin Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja vStajner, Sébastien Montella, Shailza, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin P. Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Yi Xu, Yisi Sang, Yixin Liu, and Yufang Hou. 2022.  Gemv2: Multilingual nlg benchmarking in a single line of code.  _ArXiv_ , abs/2206.11249.
  * Ginart et al. (2019) Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. 2019.  Making ai forget you: Data deletion in machine learning.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Grosz et al. (1983) Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1983.  Providing a Unified Account of Definite Noun Phrases in Discourse.  In _21st Annual Meeting of the Association for Computational Linguistics_ , pages 44–50, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hallinan and Striphas (2016) Blake Hallinan and Ted Striphas. 2016.  Recommended for you: The netflix prize and the production of algorithmic culture.  _New Media & Society_, 18(1):117–137.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. 2022.  An empirical analysis of compute-optimal large language model training.  In _Advances in Neural Information Processing Systems_.
  * Hutchinson et al. (2021) Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021.  Towards accountability for machine learning datasets: Practices from software engineering and infrastructure.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 560–575, New York, NY, USA. Association for Computing Machinery.
  * Jacobs and Wallach (2021) Abigail Z. Jacobs and Hanna Wallach. 2021.  Measurement and fairness.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 375–385, New York, NY, USA. Association for Computing Machinery.
  * Joshi (1969) Aravind K. Joshi. 1969.  Properties of Formal Grammars with Mixed Type of Rules and their Linguistic Relevance.  In _International Conference on Computational Linguistics COLING 1969: Preprint No. 47_ , Sånga Säby, Sweden.
  * Joshi et al. (1972) Aravind K. Joshi, S. Rao Kosaraju, and H.M. Yamada. 1972.  String adjunct grammars: I. local and distributed adjunction.  _Information and Control_ , 21(2):93–116.
  * Joshi and Schabes (1989) Aravind K. Joshi and Yves Schabes. 1989.  An evaluation of lexicalization in parsing.  In _Speech and Natural Language: Proceedings of a Workshop Held at Cape Cod, Massachusetts, October 15-18, 1989_.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _ArXiv_ , abs/2001.08361.
  * Kaushik and Lipton (2018) Divyansh Kaushik and Zachary C. Lipton. 2018.  How much reading does reading comprehension require? a critical investigation of popular benchmarks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5010–5015, Brussels, Belgium. Association for Computational Linguistics.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021.  Dynabench: Rethinking benchmarking in NLP.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4110–4124, Online. Association for Computational Linguistics.
  * Koch et al. (2021) Bernard Koch, Emily Denton, Alex Hanna, and Jacob Gates Foster. 2021.  Reduced, reused and recycled: The life of a dataset in machine learning research.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_ , volume 25. Curran Associates, Inc.
  * Liang et al. (2022a) Percy Liang, Rishi Bommasani, Kathleen A. Creel, and Rob Reich. 2022a.  The time is now to develop community norms for the release of foundation models.
  * Liang et al. (2022b) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R’e, Diana Acosta-Navas, Drew A. Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha, Niladri S. Chatterji, O. Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas F. Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022b.  Holistic evaluation of language models.  _ArXiv_ , abs/2211.09110.
  * Liao et al. (2021) Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. 2021.  Are we learning yet? a meta review of evaluation failures across machine learning.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Liberman (2005) Mark Liberman. 2005.  Franklin Medal to Aravind Joshi.  _Language Log_.
  * Linzen (2020) Tal Linzen. 2020.  How can we accelerate progress towards human-like linguistic generalization? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5210–5217, Online. Association for Computational Linguistics.
  * Liu et al. (2021) Nelson F. Liu, Tony Lee, Robin Jia, and Percy Liang. 2021.  Can small and synthetic benchmarks drive modeling innovation? a retrospective study of question answering modeling approaches.  ArXiv:2102.01065.
  * Miltsakaki et al. (2004) Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004.  The Penn Discourse Treebank.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Nissim et al. (2017) Malvina Nissim, Lasha Abzianidze, Kilian Evang, Rob van der Goot, Hessel Haagsma, Barbara Plank, and Martijn Wieling. 2017.  Sharing is caring: The future of shared tasks.  _Computational Linguistics_ , 43(4):897–904.
  * Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016.  Universal Dependencies v1: A multilingual treebank collection.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1659–1666, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Parra Escartín et al. (2017) Carla Parra Escartín, Wessel Reijers, Teresa Lynn, Joss Moorkens, Andy Way, and Chao-Hong Liu. 2017.  Ethical considerations in NLP shared tasks.  In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_ , pages 66–73, Valencia, Spain. Association for Computational Linguistics.
  * Paullada et al. (2021) Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021.  Data and its (dis)contents: A survey of dataset development and use in machine learning research.  _Patterns_ , 2(11):100336.
  * Peng et al. (2021) Kenneth L Peng, Arunesh Mathur, and Arvind Narayanan. 2021.  Mitigating dataset harms requires stewardship: Lessons from 1000 papers.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Prasad et al. (2008) Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008.  The Penn Discourse TreeBank 2.0. In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08)_ , Marrakech, Morocco. European Language Resources Association (ELRA).
  * Prasad et al. (2014) Rashmi Prasad, Bonnie Webber, and Aravind Joshi. 2014.  Reflections on the Penn Discourse TreeBank, comparable corpora, and complementary annotation.  _Computational Linguistics_ , 40(4):921–950.
  * Raji and Buolamwini (2019) Inioluwa Deborah Raji and Joy Buolamwini. 2019.  Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’19, page 429–435, New York, NY, USA. Association for Computing Machinery.
  * Raji et al. (2021) Inioluwa Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021.  AI and the everything in the whole wide world benchmark.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Raji et al. (2020) Inioluwa Deborah Raji, Timnit Gebru, Margaret Mitchell, Joy Buolamwini, Joonseok Lee, and Emily Denton. 2020.  Saving face: Investigating the ethical concerns of facial recognition auditing.  In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’20, page 145–151, New York, NY, USA. Association for Computing Machinery.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Rogers (2020) Anna Rogers. 2020.  Peer review in nlp: resource papers.
  * Scheuerman et al. (2021) Morgan Klaus Scheuerman, Alex Hanna, and Emily Denton. 2021.  Do datasets have politics? disciplinary values in computer vision dataset development.  _Proc. ACM Hum.-Comput. Interact._ , 5(CSCW2).
  * Spärck Jones and Galliers (1995) Karen Spärck Jones and Julia R. Galliers. 1995.  _Evaluating Natural Language Processing Systems: An Analysis and Review_.  Number 1083 in Lecture Notes in Computer Science. Springer Verlag, Berlin.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek B Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Annasaheb Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmuller, Andrew M. Dai, Andrew D. La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakacs, Bridget R. Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Stephen Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, C’esar Ferri Ram’irez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Tatiana Ramirez, Clara Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Daniel H Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Gonz’alez, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, D. Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth P. Donoway, Ellie Pavlick, Emanuele Rodolà, Emma FC Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fan Xia, Fatemeh Siar, Fernando Mart’inez-Plumed, Francesca Happ’e, François Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-L’opez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Han Sol Kim, Hannah Rashkin, Hanna Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hubert Wong, Ian Aik-Soon Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, John Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, J. Brooker Simon, James Koppel, James Zheng, James Zou, Jan Koco’n, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Narain Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jenni Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Oluwadara Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Jane W Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jorg Frohberg, Jos Rozen, José Hernández-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Ochieng’ Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Luca Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Col’on, Luke Metz, Lutfi Kerem cSenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Madotto Andrea, Maheen Saleem Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, M Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew Leavitt, Matthias Hagen, M’aty’as Schubert, Medina Baitemirova, Melissa Arnaud, Melvin Andrew McElrath, Michael A. Yee, Michael Cohen, Mi Gu, Michael I. Ivanitskiy, Michael Starritt, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Monica Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, T MukundVarma, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas S. Roberts, Nicholas Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W. Chang, Peter Eckersley, Phu Mon Htut, Pi-Bei Hwang, P. Milkowski, Piyush S. Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, QING LYU, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ram’on Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib J. Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Sam Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi S. Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo hwan Lee, Spencer Bradley Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Rose Biderman, Stephanie C. Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq A. Ali, Tatsuo Hashimoto, Te-Lin Wu, Theo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, T. N. Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler O’Brien Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, W Vossen, Xiang Ren, Xiaoyu F Tong, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yang Song, Yasaman Bahri, Ye Ji Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yu Hou, Yushi Bai, Zachary Seid, Zhao Xinran, Zhuoye Zhao, Zi Fu Wang, Zijie J. Wang, Zirui Wang, Ziyi Wu, Sahib Singh, and Uri Shaham. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _ArXiv_ , abs/2206.04615.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * von Werra et al. (2022) Leandro von Werra, Lewis Tunstall, Abhishek Thakur, Alexandra Sasha Luccioni, Tristan Thrush, Aleksandra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, Helen Ngo, Omar Sanseviero, Mario vSavsko, Albert Villanova, Quentin Lhoest, Julien Chaumond, Margaret Mitchell, Alexander M. Rush, Thomas Wolf, and Douwe Kiela. 2022.  Evaluate&evaluation on the hub: Better best practices for data and model measurements.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _International Conference on Learning Representations_.
  * Webber (2018) Bonnie Webber. 2018.  Obituary: Aravind K. Joshi.  _Computational Linguistics_ , 44(3):387–392.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_.  Survey Certification.
  * WELM (2021) WELM. 2021.  Workshop on Enormous Language Models (WELM).
  * Welty et al. (2019) Chris Welty, Praveen K. Paritosh, and Lora Aroyo. 2019.  Metrology for ai: From benchmarks to instruments.  _ArXiv_ , abs/1911.01875.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yuan et al. (2022) Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Re, and Ce Zhang. 2022.  Decentralized training of foundation models in heterogeneous environments.  In _Advances in Neural Information Processing Systems_.


# World Knowledge in Multiple Choice Reading Comprehension

Adian Liusie
ALTA Institute, Cambridge University
<EMAIL_ADDRESS>
&Vatsal Raina11footnotemark: 1
ALTA Institute, Cambridge University
<EMAIL_ADDRESS>
Mark Gales
ALTA Institute, Cambridge University
<EMAIL_ADDRESS>
Equal Contribution

###### Abstract

Recently it has been shown that without any access to the contextual passage,
multiple choice reading comprehension (MCRC) systems are able to answer
questions significantly better than random on average. These systems use their
accumulated "world knowledge" to directly answer questions, rather than using
information from the passage. This paper examines the possibility of
exploiting this observation as a tool for test designers to ensure that the
form of "world knowledge" is acceptable for a particular set of questions. We
propose information-theory based metrics that enable the level of "world
knowledge" exploited by systems to be assessed. Two metrics are described: the
expected number of options, which measures whether a passage-free system can
identify the answer to a question using world knowledge; and the contextual
mutual information, which measures the importance of context for a given
question. We demonstrate that questions with low expected number of options,
and hence answerable by the shortcut system, are often similarly answerable by
humans without context. This highlights that the general knowledge ‘shortcuts’
could be equally used by exam candidates, and that our proposed metrics may be
helpful for future test designers to monitor the quality of questions.

##  1 Introduction

Reading comprehension (RC) exams are used extensively in a wide range of
language competency examinations (Alderson, 2000), and have become a
ubiquitous assessment method to probe how well candidates can read a passage
and understand the text’s core meaning. A fundamental assumption of RC exams
is that to answer any of the questions correctly, one has to read the passage,
comprehend its meaning, and identify the relevant information of a given
question. However, recent work has shown that multiple-choice machine reading
comprehension (MCMRC) systems without access to the passage can achieve
reasonable performance (Pang et al., 2022), showing that the models may be
doing less comprehension than anticipated.

Figure 1: Output probabilities of our model (trained with contexts omitted) on
real RACE++ (Liang et al., 2019) examples. ‘Effective number of options’ is a
metric that captures the model’s confidence. Figure 2: Model architecture.

In this paper we analyse this phenomena and for several standard MCMRC
datasets (Liang et al., 2019; Huang et al., 2019; Yu et al., 2020) verify that
passage-free baselines are able to achieve performance significantly better
than random. We show that a subset of questions can be answered accurately and
confidently without access to the contextual passage, where further analysis
shows this is partly due to the presence of low-quality distractors, i.e.
options that can be eliminated using only the question. As an example, given
the question “Mina’s sister’s name is:", one can eliminate any options that
use a traditionally male name (see Figure 1). This highlights a potential
‘shortcut’ candidates could use to answer questions while bypassing the
context. Our work raises awareness to this potential flaw, and proposes a
simple solution to catch questions that can be answered without comprehension.
The proposed metrics might be a useful tool for future multiple-choice RC test
designers to ensure that all questions truly assess reading comprehension
ability.

Machine reading comprehension (MRC) is a highly researched area, with state-
of-the-art (SoTA) systems (Zhang et al., 2021; Yamada et al., 2020; Zaheer et
al., 2020; Wang et al., 2022) often approaching or even exceeding human level
performance on public benchmarking leaderboards (Clark et al., 2018; Lai et
al., 2017; Trischler et al., 2017; Yang et al., 2018). Existing work has
analysed the robustness of MRC systems, where researchers have questioned
whether systems fully leverage context and whether they accomplish the
underlying comprehension task (Sugawara et al., 2020; Rajpurkar et al., 2016;
Kaushik and Lipton, 2018; Jia and Liang, 2017; Si et al., 2019). Most notably
Kaushik and Lipton (2018) show that for a range of question-answering tasks,
passage-only systems can often achieve remarkable performance, which has been
observed in the MCRC setting (Pang et al., 2022).

Most existing work has discussed model robustness, demonstrating that for some
tasks it is possible to obtain high average system performance with no context
information. In contrast, this paper focuses on the attributes of individual
questions and options, identifying questions where "world knowledge" can be
leveraged, and the extent to which this knowledge can be leveraged. This could
be a useful tool to enable test designers to monitor the questions being
proposed, and whether alternative distractors or questions should be
considered.

##  2 Multiple choice reading comprehension

Multiple-choice reading comprehension is a popular task where given a context
passage $C$ and question $Q$, the correct answer must be deduced from a set of
answer options $\\{O\\}$. Current SoTA MRC systems are dominated by pre-
trained language models (PrLMs) based on the transformer encoder architecture
(Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020).

Model Architecture Our question-answering system follows the standard MCMRC
architecture of Figure 2 (Yu et al., 2020; Raina and Gales, 2022). Each option
is individually encoded along with the question and the context into a score,
and a softmax layer converts the 4 options’ scores into a probability
distribution. At inference, the predicted answer is the option with the
greatest probability.

‘No Context’ Shortcut System A requirement for good MCRC questions is that
information from both the question and the context passage must be used to
determine the correct answer. To probe whether this is an issue for MCMRC, we
train systems using ‘context free’ inputs (similar to Pang et al. (2022)). The
standard set-up (Figure 2) is still followed, however the input is now altered
to exclude the context, as shown in Figure 3.

Figure 3: System inputs for shortcut system.

Effective Number of Options Consider the output probability distribution of
the predicted answer, ${\tt P}(y)$. One can determine the entropy, ${\cal
H}(Y)$, which can be converted into the more interpretable effective number of
options, ${\cal N}(Y)$, a value bounded between 1 and the maximum number of
options:

$\displaystyle{\cal N}(Y)=2^{{\cal H}(Y)},\hskip 5.0pt{\cal H}(Y)=-\sum_{y\in
Y}{\tt P}(y)\log_{2}{\tt P}(y)$ (1)

For well designed questions, one would expect systems with missing information
(i.e. the ‘shortcut’ models) to have no information of what the answer is.
This would correspond to a uniform distribution output (the distribution of
maximum entropy), with an effective number of options equal to the total
number of answer options. However, if the effective number of options is
significantly lower than the total number of answer options, then this implies
that prior information stored during training can be used to answer the
question, without comprehension.

Mutual Information To probe how much information is gained by the context, one
can additionally look at an approximation of mutual information of the
context. This looks at how much the entropy decreases between the ‘no context’
shortcut system and the ‘context’ baseline system .

$\displaystyle\\!\\!{\cal I}(Y;C|Q,\\{O\\})={\cal H}(Y|Q,\\{O\\})-{\cal
H}(Y|Q,\\{O\\},C)$ (2)

An alternative approach would be to use random contexts (Creswell and
Shanahan, 2022) however we consider the stricter ‘no context’ setting.

##  3 Experiments

Data We consider three popular MCMRC datasets: RACE++ (Lai et al., 2017),
COSMOSQA (Huang et al., 2019) and ReClor (Yu et al., 2020). RACE++ is a
dataset of English comprehension questions for Chinese high school students,
COSMOSQA is a large scale commonsense-based reading comprehension dataset,
while ReClor is a challenging logical reasoning dataset at a graduate student
level. All datasets have 4 options per question, one of which being the
correct answer.

| TRN | DEV | EVL
---|---|---|---
RACE++ | 100,388 | 5,599 | 5,642
COSMOS | 25,262 | 2,985 | –
ReClor | 4,638 | 500 | 1000
Table 1: Dataset statistics

Training An ELECTRA-
large111https://huggingface.co/docs/transformers/model_doc/electra model is
fine-tuned on the training split TRN, hyper-parameters are tuned on the
developement set DEV, and performance reported on the test split EVL for
RACE++ (DEV splits are used for COSMOS and ReClor due to unavailability of the
EVL splits). All model parameters (transformer and classifier) are updated
during fine-tuning. Additionally, models are trained and evaluated using the
‘no context’, as described in Section 2. Final hyperparameters are given in
Appendix B.1. Three seeds are trained, and ensemble accuracy is used as the
default metric when reporting performance.222code for experiments available
at:
https://github.com/adianliusie/MCRC

###  3.1 Results

Context-Free Performance We compare the performance of the baseline ‘standard’
system against the shortcut ‘no context’ systems for the various MCMRC
datasets. Table 2 illustrates that the shortcut systems achieve high
performance across all MCMRC datasets, all above 50%, significantly above the
expected random performance of 25%. Further, we find that the shortcut rules
can generalise across domains, most notably seen with the 54% performance when
training the shortcut system on RACE and evaluating on COSMOS. This highlights
that the shortcut performance cannot be explained purely by dataset bias, but
that there is a skill, unrelated to comprehension, that the systems are
meaningfully leveraging.

Training data | RACE++ | COSMOS | ReClor
---|---|---|---
| – | 25.00 | 25.00 | 25.00
RACE++ | stan. | 85.01 | 70.05 | 48.60
no con. | 57.32 | 54.04 | 34.80
COSMOS | stan. | 66.81 | 84.49 | 41.20
no con. | 38.73 | 68.51 | 27.80
ReClor | stan. | 52.69 | 41.68 | 69.80
no con. | 31.27 | 33.13 | 51.80
Table 2: Cross-performance of systems on RACE++, COSMOSQA and ReClor (standard
vs no context).

RACE++ Effective Number of Options Figure 4 presents the count and accuracy
plots of the effective number of options (bin width of 0.2) for both the
standard and shortcut systems on RACE++ (see Appendix for other datasets). The
counts plot show the number of questions within the bin range, while accuracy
refers to the accuracy over all the examples within the bin. Since the systems
are slightly overconfident333For both models, the mean of the maximum
probability is 5% above the overall accuracy., the systems’ output
probabilities are calibrated using temperature annealing (Guo et al., 2017)
(see Appendix B.3).

Figure 4: Distribution of effective number of options and corresponding
(binned) accuracy.

The baseline system has high certainty for most points, which is somewhat
expected given the baseline’s high accuracy. However the shortcut system,
without any contextual information, has a significant number of examples in
the very low entropy region. This shows that for a subset of questions, the
system can confidently answer questions correctly without doing any
comprehension at all. In other cases, the shortcut system can leverage some
information from the question and can reduce the number of effective options
to between 2-3, which implies that certain poor distractors can be eliminated
by the question alone. We also show that for both models, there is a clear
linear relationship between uncertainty and accuracy, illustrating that the
context-free system’s use of world knowledge is sensible and that it leverages
meaningful task information (see Appendix D for low-entropy examples). This
confirms that the systems are well calibrated and that the effective number of
options is a good measure of actual model uncertainty.

Mutual Information To further look at the influence of context, the mutual
information (MI) between prediction and context was approximated for each
example using Equation 2. Examples with a high MI are questions where the
model is certain of the answer with context, but is uncertain without context
- a desired property for comprehension questions. Figure 5 shows the counts
when all the examples are ordered by MI (see Equation 2) along with both the
baseline and shortcut system accuracies. We note that the count distribution
has a mix of high and low MI questions, which shows that the benefit of
context is not a system-wide property but instead varies over questions. The
accuracy of the baseline system increases considerably when context is useful,
while accuracy falls for the shortcut system. It is interesting that a small
fraction of questions have negative MI. Though MI should always be positive,
negative values can be observed since models are only approximations of the
true underlying distributions. The low accuracy of the shortcut model on
negative MI questions occurs when standard world knowledge is not consistent
with the information in the context.

Figure 5: Distribution of counts and corresponding accuracy when points are
sorted by MI approximation.

Human Evaluation of Metrics We perform human evaluation to judge the practical
use of our metrics. We select 100 questions with lowest and highest entropy,
and three volunteer graduate students independently answer the questions
without access to the context. We further select 50 questions with lowest and
highest MI, and get our volunteers to first answer questions without context,
then with context, and calculate the accuracy increase. All questions are
shuffled, and volunteers attempt to best answer all questions. We find that
our metrics are very effective in measuring their desired properties. Without
context, humans are often able to answer the questions that the shortcut
systems answer confidently, with humans achieving an average accuracy of 92%
on the 100 lowest entropy and 32% on the 100 highest entropy examples
respectively. Further, for high MI questions humans get a performance boost of
71% when context is provided, and only 22% for low mutual information
questions.

| low ent. | high ent. | high MI | low MI
---|---|---|---|---
human | $91.7_{\pm 1.9}$ | $31.7_{\pm 2.9}$ | $\Delta 69.3_{\pm 0.9}$ | $\Delta 24.7_{\pm 5.0}$
system | $99.0_{\pm 0.0}$ | $24.3_{\pm 6.2}$ | $\Delta 68.0_{\pm 0.9}$ | $\Delta 3.3_{\pm 4.7}$
Table 3: Human and system ‘no context’ accuracy on lowest and highest entropy
questions as well as human and system change in accuracies on lowest and
highest mutual information questions.

##  4 Conclusions

For popular MCMRC datasets, systems can achieve reasonably high performance
without performing any comprehension. Without passage information, ‘shortcut’
systems can confidently determine some correct answer options, eliminate some
unlikely distractors, and use general knowledge to gain information. Rather
than focusing on average system performance, our work analyses individual
question’s reliance on world knowledge. We propose a metric based on the
shortcut systems to automatically flag questions that are answerable without
comprehension. We further provide evidence that the flagged questions are
answerable by humans without any context. Lastly, using an approximation of
the mutual information, we show that the importance of context varies over the
questions in the dataset, and reason that high MI questions can be thought of
as candidates for high-quality questions that truly measure comprehension
abilities.

##  5 Limitations

We propose an approach that can automatically flag questions that can be
answered without contextual information. However, the remaining questions are
not necessarily high-quality questions, since many other aspects make up
question quality. Second, the experiments are conducted using only the Electra
model, though it is expected similar trends will be picked up by alternative
transformer-based language models. Further, exams might be aimed at a level
where a lack of specific knowledge may be assumed. Our work does not consider
variable candidate knowledge levels, and our evaluation was only done by
highly educated (we’d like to think) graduate students. Finally, we
acknowledge that our human evaluation was limited in size and questions,
however it is clearly demonstrated that for low ‘shortcut entropy’ questions,
comprehension is not necessarily required.

##  6 Acknowledgements

This research is funded by the EPSRC (The Engineering and Physical Sciences
Research Council) Doctoral Training Partnership (DTP) PhD studentship and
supported by Cambridge Assessment, University of Cambridge and ALTA.

##  7 Ethics Statement

There are no serious ethical concerns with this work. The human volunteers all
performed the human evaluation tasks willingly without any coercion. The human
evaluation took 2 hours per person.

## References

  * Alderson (2000) J. Charles. Alderson. 2000.  _Assessing Reading_ , 1 edition.  Cambridge University Press,, Cambridge :.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _ArXiv_ , abs/1803.05457.
  * Creswell and Shanahan (2022) Antonia Creswell and Murray Shanahan. 2022.  Faithful reasoning using large language models.  _arXiv preprint arXiv:2208.14271_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International conference on machine learning_ , pages 1321–1330. PMLR.
  * Huang et al. (2019) Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019.  Cosmos qa: Machine reading comprehension with contextual commonsense reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2391–2401.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _EMNLP_.
  * Kaushik and Lipton (2018) Divyansh Kaushik and Zachary C Lipton. 2018.  How much reading does reading comprehension require? a critical investigation of popular benchmarks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5010–5015.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and E. Hovy. 2017.  Race: Large-scale reading comprehension dataset from examinations.  In _EMNLP_.
  * Liang et al. (2019) Yichan Liang, Jianheng Li, and Jian Yin. 2019.  A new multi-choice reading comprehension dataset for curriculum learning.  In _Proceedings of The Eleventh Asian Conference on Machine Learning_ , volume 101 of _Proceedings of Machine Learning Research_ , pages 742–757, Nagoya, Japan. PMLR.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _International conference on machine learning_ , abs/1907.11692.
  * Pang et al. (2022) Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. 2022.  QuALITY: Question answering with long input texts, yes! In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5336–5358, Seattle, United States. Association for Computational Linguistics.
  * Raina and Gales (2022) Vatsal Raina and Mark Gales. 2022.  Answer uncertainty and unanswerability in multiple-choice machine reading comprehension.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1020–1034, Dublin, Ireland. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Si et al. (2019) Chenglei Si, Shuohang Wang, Min-Yen Kan, and Jing Jiang. 2019.  What does bert learn from multiple-choice reading comprehension datasets?  _arXiv preprint arXiv:1910.12391_.
  * Sugawara et al. (2020) Saku Sugawara, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2020.  Assessing the benchmarking capacity of machine reading comprehension datasets.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8918–8927.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  Newsqa: A machine comprehension dataset.  In _Rep4NLP@ACL_.
  * Wang et al. (2022) Siyuan Wang, Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. 2022.  Logic-driven context extension and data augmentation for logical reasoning of text.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1619–1629.
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020\.  Luke: Deep contextualized entity representations with entity-aware self-attention.  In _EMNLP_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _EMNLP_.
  * Yu et al. (2020) Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020.  Reclor: A reading comprehension dataset requiring logical reasoning.  In _International Conference on Learning Representations_.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big bird: Transformers for longer sequences.  _ArXiv_ , abs/2007.14062.
  * Zhang et al. (2021) Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2021.  Retrospective reader for machine reading comprehension.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 14506–14514.

##  Appendix Appendix A Additional Results

###  Appendix A.1 COSMOSQA

Figure Appendix A.1: Distribution of effective number of options and binned
accuracy for COSMOSQA. Figure Appendix A.2: Distribution of counts and
corresponding accuracy when points are sorted by MI approximation for
COSMOSQA.

We repeat the entropy plot (Figure Appendix A.1) for COSMOSQA and find similar
trends to those seen in RACE++. The shortcut no-context system has a very flat
distribution with a substantial number of questions answerable without
context, with the effective number of options again having a clean linear
relationship with accuracy. The repeated mutual information plot (Figure
Appendix A.2) for COSMOSQA also has the same trend seen in RACE++, validating
that our findings are more general that just for RACE++.

###  Appendix A.2 ReClor

Figure Appendix A.3: Distribution of effective number of options and binned
accuracy for ReClor. Figure Appendix A.4: Distribution of counts and
corresponding accuracy when points are sorted by MI approximation for ReClor.

ReClor show roughly the same trends, however the questions of ReClor are much
more challenging than in either RACE++ and COSMOSQA, and so we notice that the
counts distribution is pushed considerably to the higher entropy side.
Additionally, since ReClor is much smaller than RACE++ and COSMOSQA (see Table
1), the curves are less smooth and largely suffer from noise.

###  Appendix A.3 Other Shortcuts

We also consider other shortcut approaches, such as having context and options
(i.e. missing question) and only options (Figure Appendix A.5). Performance of
the systems is shown in Table Appendix A.1.

Figure Appendix A.5: System inputs for alternative shortcut systems. Training data | RACE++ | COS. | ReClor
---|---|---|---
| – | 25.00 | 25.00 | 25.00
RACE++ | {O} | 41.76 | 21.44 | 34.00
Q+{O} | 57.32 | 54.04 | 34.80
{O}+C | 68.20 | 54.61 | 46.00
Q+{O}+C | 85.01 | 70.05 | 48.60
COSMOS | {O} | 29.95 | 57.39 | 25.20
Q+{O} | 38.73 | 68.51 | 27.80
{O}+C | 52.41 | 78.96 | 40.40
Q+{O}+C | 66.81 | 84.49 | 41.20
ReClor | {O}. | 26.07 | 18.29 | 49.00
Q+{O} | 31.27 | 33.13 | 51.80
{O}+C | 39.83 | 36.88 | 68.40
Q+{O}+C | 52.69 | 41.68 | 69.80
Table Appendix A.1: Cross-performance of systems on RACE++, COSMOSQA and
ReClor using accuracy.

##  Appendix Appendix B Model Information

###  B.1 Training Details

For all systems, deep ensembles of 3 models are trained with the large
444Configuration at: https://huggingface.co/google/electra-large-
discriminator/blob/main/config.json. ELECTRA PrLM as a part of the multiple-
choice MRC architecture depicted in Figure 2. Each model has 340M parameters.
Grid search was performed for hyperparameter tuning with the initial setting
of the hyperparameter values dictated by the baseline systems from Yu et al.
(2020); Raina and Gales (2022). Apart from the default values used for various
hyperparamters, the grid search was performed for the maximum number of epochs
$\in\\{2,5,10\\}$; learning rate $\in\\{2e-7,2e-6,2e-5\\}$; batch size
$\in\\{2,4\\}$. For RACE++, training was performed for 2 epochs at a learning
rate of 2e-6 with a batch size of 4 and inputs truncated to 512 tokens. For
systems trained on ReClor the final hyperparameter settings included training
for 10 epochs at a learning rate of 2e-6 with a batch size of 4 and inputs
truncated to 512 tokens. For COSMOSQA, training was performed for 5 epochs at
a learning rate of 2e-6 with a batch size of 4 and inputs truncated to 512
tokens. Cross-entropy loss was used at training time with models built using
NVIDIA A100 graphical processing units with training time under 3 hours per
model for ReClor, 5 hours for COSMOSQA and 4 hours for RACE++. All
hyperparameter tuning was performed by training on TRN and selecting values
that achieved optimal performance on DEV. For fairness, the ‘shortcut’ systems
(omitting various forms of the input) for each dataset were trained with the
same hyperparameter settings as their corresponding baseline systems.

###  B.2 Evaluation Details

For each dataset, the systems are trained on the training split and
hyperparameter tuned on the development split. For RACE++, systems are
evaluated on the held out test data, but for COSMOSQA and ReClor, the
evaluations are performed on the development split because their test splits
have their labels hidden.

###  B.3 Calibration

The trained models were calibrated post-hoc using single parameter temperature
annealing (Guo et al., 2017). Uncalibrated, model probabilities are determined
by applying the softmax to the output logit scores $s_{i}$:

$P(y=k;\bm{\theta})\propto\exp(s_{k})$ (3)

where $k$ denotes a possible output class for a prediction $y$. Temperature
annealing ‘softens’ the output probability distribution by dividing all logits
by a single parameter $T$ before the softmax.

$P_{CAL}(y=k;\bm{\theta})\propto\exp(s_{k}/T)$ (4)

As the parameter $T$ does not change the relative rankings of the logits, the
model’s prediction will be unchanged and so temperature scaling does not
affect the model’s accuracy. The parameter $T$ is chosen such that the
accuracy of the system is equal to the mean of the maximum probability (which
would be expected for a calibrated system).

##  Appendix Appendix C Licenses

This section details the license agreements of the scientific artifacts used
in this work. The dataset COSMOSQA (Huang et al., 2019) has BSD 3-Clause
License. The datasets RACE++ (Lai et al., 2017) and ReClor (Yu et al., 2020)
are freely available with the limitation on the latter that it can only be
used for non-commercial research purposes. Huggingface transformer models are
released under: Apache License 2.0. All the scientific aritfacts are
consistent with their intended uses.

##  Appendix Appendix D Low Entropy Examples