Vatsal Raina, Adian Liusie, Mark Gales ††thanks: This research is funded by
the EPSRC (The Engineering and Physical Sciences Research Council) Doctoral
Training Partnership (DTP) PhD studentship and supported by Cambridge
Assessment, University of Cambridge and ALTA.

# Analyzing Multiple-Choice Reading and Listening Comprehension Tests

###### Abstract

Multiple-choice reading and listening comprehension tests are an important
part of language assessment. Content creators for standard educational tests
need to carefully curate questions that assess the comprehension abilities of
candidates taking the tests. However, recent work has shown that a large
number of questions in general multiple-choice reading comprehension datasets
can be answered without comprehension, by leveraging world knowledge instead.
This work investigates how much of a contextual passage needs to be read in
multiple-choice reading based on conversation transcriptions and listening
comprehension tests to be able to work out the correct answer. We find that
automated reading comprehension systems can perform significantly better than
random with partial or even no access to the context passage. These findings
offer an approach for content creators to automatically capture the trade-off
between comprehension and world knowledge required for their proposed
questions.

Index Terms: machine reading comprehension, listening comprehension, multiple-
choice, automatic speech recognition, world knowledge

##  1 Introduction

Multiple-choice reading and listening comprehension tests serve as essential
tools for evaluating language proficiency in educational settings [1]. In
particular, multiple-choice questions permit fast and automated objective
assessment of candidates' abilities. The creation of these standardized tests
necessitates the careful selection of questions that accurately assess
candidates' comprehension abilities. It is of interest for content creators to
develop a framework to categorize the quality of questions used in assessment
across several criteria such as complexity and diversity [2].

However, recent work [3] has identified an issue within general multiple-
choice reading comprehension datasets sourced from real tests — many questions
can be answered correctly without language learners truly comprehending the
passage, merely by relying on prior world knowledge. This work builds upon the
concept of world knowledge in reading comprehension and aims to explore the
extent to which contextual passages must be read/heard in multiple-choice
reading/listening tests based on conversation transcriptions and listening
comprehension assessments to deduce the correct answer. For example, a
candidate may be able to deduce the correct answer to a large number of the
comprehension questions by only reading the first sentence. Typically language
learners may not understand the whole context and only partially comprehend
the sentences. Figure 1 demonstrates three multiple-choice questions with
varying degrees of required comprehension. Full comprehension, when the whole
passage must be read in order to determine the correct answer. Partial
comprehension, when the correct answer can be deduced from reading only a
small part of the context. Finally, zero comprehension in the extreme case
where the correct answer can be deduced without reading the context at all and
by using world knowledge instead. For instance, in the zero comprehension
example in Figure 1, without any need to read the context it is obvious that
the answer is sick children as the question asks about charities.

Figure 1: Example questions that can be answered with full, partial and zero
comprehension respectively.

Information about the extent of comprehension required in reading and
listening tests can act as a core component in the question assessment
framework [2, 4]. The degree of comprehension required can vary across the
nature of the comprehension dataset. In this work, we consider a range of
publicly available datasets that are very different in nature including
commonsense-based reasoning, logical reasoning and multi-turn dialogue, speech
transcriptions. We make the following contributions in this work:

  * •

Portability of world knowledge and partial comprehension systems from standard
multiple-choice reading comprehension to dialogue and speech.

  * •

A thorough investigation of the degree of partial comprehension from zero
comprehension (world knowledge) to full comprehension.

We emphasize the need for content creators to carefully and explicitly
consider the extent of comprehension required for the questions they generate
in order to better capture how language learners may interact with the
deployed questions in tests.

##  2 Related work

[3] indicates world knowledge is prevalent in several standard multiple-choice
reading comprehension systems, reinforcing whether machine reading
comprehension systems fully leverage the context for the desired comprehension
task [5, 6, 7, 8]. [3] further introduces two performance metrics, effective
number of options and mutual information of the context, to assess the extent
to which world knowledge is used in these reading comprehension systems. We
extend the work on world knowledge to investigate the spectrum between zero
comprehension to full comprehension of real multiple-choice comprehension
questions for text-based, dialogue-based and speech-based contexts.

Previous work investigated automated approaches to assess the quality of
comprehension questions. [2] present a framework to assess the quality of
generated multiple-choice questions for comprehension. Four main qualities are
identified: grammatical fluidity, answerability, diversity and complexity. Our
work on assessing the extent to which the context needs to be read acts as an
extension to this framework to capture the comprehensibility of the generated
questions.

Due to the lack of appropriately annotated speech corpora, several works
investigate porting text-based systems for listening comprehension tasks. [9]
explores applying a text-based question answering system on the TOEFL
listening comprehension multiple-choice test from [10]. [11] further
investigates the transfer learning style approach for extractive comprehension
from SQuAD 2.0 [12] to a proprietary spoken question answering task, with a
particular focus on the impact of automatic speech recognition (ASR) errors.
Our approach ports systems from a multiple-choice reading comprehension task
to a multiple-choice listening comprehension task to identify the extent to
which comprehension of the context is required.

##  3 Multiple-choice comprehension

###  3.1 Task

Multiple-choice comprehension is a common assessment technique to assess the
comprehension abilities of candidates in standardized tests [13]. Given a
context passage, $C$ and a question, $Q$, the correct answer must be deduced
from a discrete set of $N$ answer options, $\\{O\\}$. Hence, it is required to
deduce the correct answer by comprehending the question and using the context
passage as the information source to identify which answer option is the most
suitable.

###  3.2 Machine comprehension

Figure 2: The architecture for multiple-choice machine comprehension with
context, $C$, question, $Q$ and $N$ options, $\\{O\\}$.

Machine comprehension performs the comprehension task using automated systems.
Machine reading and listening comprehension for multiple-choice tests is a
well researched area with state-of-the-art systems [14, 15, 16, 17] competing
and out-performing humans on public benchmarks [18, 19, 20, 21].

In this work, the machine comprehension system's architecture replicates the
standard multiple-choice machine reading comprehension systems from [22, 23]
and depicted in Figure 2. Each option is separately encoded with the question
and the context to generate a score. A softmax layer converts the scores
associated with each option into a probability distribution where at inference
time the predicted answer is taken to be the option with the greatest
probability. The parameters of the core transformer [24] encoder and the
linear layer are shared across all options. Hence, there is no requirement for
the number of options at training and inference time to match.

###  3.3 World knowledge

It is expected that information must be used from both the context passage and
the question to determine the correct answer. If the answer can be deduced
without the context, it suggests `world knowledge' [3] is sufficient to answer
the question. We train a context-free system where the context is omitted to
determine the extent to which world knowledge can be leveraged for
comprehension. Table 1 summarizes the main differences between the standard
and context-free systems where [CLS] and [SEP] denote classification and
separation tokens respectively.

Table 1: Format for multiple-choice comprehension systems. System | Format
---|---
Standard | [CLS]$<$$C$$>$[SEP]$<$$Q$$>$$<$$O_{i}$$>$[SEP]
Context-free | [CLS]$<$$Q$$>$$<$$O_{i}$$>$[SEP]

###  3.4 Partial context

Language learners often can shortcut reading the whole context passage in
comprehension tasks and still correctly answer the question. Hence, we devise
a simple approach to investigate the extent to which a context must be
comprehended in order to determine the correct answer to standard multiple-
choice questions. A standard system (see Table 1) trained with the full
context is taken and applied at inference time to questions with only partial
access to the context. After applying tokenization of the context, only
$\tau$% of the context tokens are retained and input to the standard system.
$\tau$ can be varied to determine how much of the context is necessary for
comprehension.

##  4 Experiments

###  4.1 Data

Several multiple-choice reading/listening comprehension datasets are used in
this work including: RACE++ [25], ReClor [22], COSMOSQA [26], DREAM [27] and
IBM-Debater [28].

Table 2: Dataset statistics. Relevant examples are underlined. | TRN | DEV | EVL | #options
---|---|---|---|---
RACE++ | 100,388 | 5,599 | 5,642 | 4
COSMOSQA | 25,262 | 2,985 | – | 4
ReClor | 4,638 | 500 | 1000 | 4
DREAM | 6,116 | 2,040 | 2,041 | 3
IBM-Debater | – | – | 200 | 2

RACE++ is a dataset of English reading comprehension questions for Chinese
high school students. The questions are collected at three levels: middle
school, high school and college level, corresponding to increasing levels of
complexity.

COSMOSQA is a large scale commonsense-based reading comprehension dataset with
four options per question. For this work, 2,985 examples from the development
set is used.

ReClor is a logical reasoning dataset at a graduate student level with four
options per question. This is a challenging dataset as graduate students
achieve an accuracy of 63%. 500 examples from the development split are used
for this work (the test set is hidden).

DREAM is a multiple-choice (three options) reading comprehension dataset that
focuses on dialogue understanding. These dialogue are multi-turn and multi-
party. It contains 10,197 questions and 6,444 dialogues, which were collected
from English-as-a-foreign-language examinations. This work uses the 2,041
questions from the test split. The context is constructed by concatenating all
dialogues into a single text.

IBM-Debater consists of 200 spontaneous speeches arguing for or against 50
controversial topics. The dataset is structured to form a multiple-choice
listening comprehension task by formulating each speech as a question that is
aimed at confirming or rejecting the argument in a speech. Hence, each
question has a binary class label with the transcribed speech acting as the
context. The transcriptions are available as both manual and automatic speech
recognition transcriptions.

###  4.2 Training details and hyperparameters

Two systems are trained on the large RACE++ training dataset (see Table 1): 1.
A standard multiple-choice reading comprehension system with access to the
context; 2. A context-free system without access to the context. Both systems
are deep ensembles of 3 models that specifically use the large 111Model
configuration at: https://huggingface.co/google/electra-
largediscriminator/blob/main/config.json ELECTRA [29] pre-trained language
model in the form of the multiple-choice machine comprehension architecture of
Figure 2.

Each model has 340M parameters. Grid search was performed for hyperparameter
tuning of the standard system with the initial setting of the hyperparameter
values by the systems from [23]. Apart from the default values used for
various hyperparameters, the grid search was performed for the maximum number
of epochs $\in\\{2,5,10\\}$; learning rate $\in\\{2e-7,2e-6,2e-5\\}$; batch
size $\in\\{2,4\\}$. Training was performed for 2 epochs at a learning rate of
2e-6 with a batch size of 4 and inputs truncated to 512 tokens at both
training and inference time. Cross-entropy loss was used at training time with
models built using NVIDIA A100 graphical processing units with training time
under 4 hours per model. The context-free system had its hyperparameters
selected to be identical to the standard system.

###  4.3 Assessment

Accuracy is used as the standard performance metric for inference on all
datasets. The evaluation process aims to assess two aspects of the multiple-
choice questions in each dataset: 1. the ability to use world knowledge in
order to determine the correct answer and consequently the effective number of
options per question; 2. the extent to which the context must be read/listened
to determine the correct answer. The former is assessed by comparing the
accuracy of a context-free comprehension system against a standard multiple-
choice comprehension system while the latter is assessed by varying the amount
of context available to a standard multiple-choice reading comprehension
system at test time.

##  5 Results

Multiple-choice questions are assessed for comprehensibility in terms of both
world knowledge and partial access to the context.

###  5.1 World knowledge

(a) Text

(b) Dialogue

(c) Speech

Figure 3: Accuracy with partial context access. Points are plotted at 10%
intervals.

Table 3 presents the prevalence of world knowledge across a range of reading
and listening comprehension datasets. As both the standard and the context-
free systems are trained on the RACE++ dataset, Table 3 further presents the
portability of the systems to different forms of reading/listening
comprehension.

Table 3: Accuracy of standard and context-free systems trained on RACE++ in-domain and out-of-domain. | Standard | Context-free | Random
---|---|---|---
RACE++ | 86.8 | 59.1 | 25.0
COSMOSQA | 73.2 | 52.8 | 25.0
ReClor | 48.8 | 38.0 | 25.0
DREAM | 86.0 | 46.1 | 33.3
IBM-manual | 65.0 | 50.0 | 50.0
IBM-ASR | 62.0 | 50.0 | 50.0

As in [3], the reading comprehension datasets of RACE++, COSMOSQA and ReClor
observe significant presence of world knowledge. In particular, the context-
free system on RACE++ achieves an accuracy of 59.1% despite having no access
to the contextual passage that is more than double the accuracy of a random
baseline. The ported context-free system also out-performs the 25% random
baseline for commonsense reasoning and logical reasoning for COSMOSQA and
ReClor respectively. Note, ReClor is a more challenging reading comprehension
dataset than COSMOSQA and RACE++ [22], confirmed by the standard RACE++
trained system getting an accuracy of 73.2% on COSMOSQA but 48.8% on ReClor.
Systems trained directly on COSMOSQA, ReClor observe a similar pattern [3].

From Table 3, both the context-free and the standard systems port across well
to dialogues in the DREAM dataset. As before, the DREAM dataset demonstrates
the presence of world knowledge as the context-free system surpasses the
random baseline of 33% to achieve 46%. It is further interesting to observe
the standard system ported from RACE++ gets an accuracy of 86%, which
approaches the state-of-the-art performance of standard systems trained on
DREAM [30].

However, the context-free system performs randomly on the speech
transcriptions from the IBM-Debater dataset. This is an expected result as the
speeches are reformulated into listening comprehension questions by posing
whether the speech is pro or con a specific controversial topic (see Section
4.1). As the speeches are balanced for each topic, it is impossible to use
world knowledge for a context-free system to deduce the argument in the speech
without listening to it. The standard system, with access to the speech
transcription, gets an accuracy of 65% with manual transcriptions and 62% with
ASR transcriptions, comparable to [28]. Hence, the presence of ASR errors
leads to a small drop in performance for binary classification.

###  5.2 Partial information access

This section investigates to what extent the context passage must be read or
listened. Figure 3 presents the accuracy with partial access to the context,
varying from zero to full access, for text, dialogue and speech-based
comprehension questions.

All results are presented using the standard system trained on RACE++. Hence,
the accuracy with 0% access to the context on the plots differs in performance
from the context-free system applied to the datasets from Table 2 \- the
context-free system's performance can expect to be an upperbound of
performance with world knowledge as the system has explicitly been trained to
try and deduce the correct answer without using the context. It is notable
from Figure 3 that both the text-based and dialogue based reading
comprehension datasets all start above the random line while the speech-based
listening comprehension dataset begins at random accuracy, agreeing with Table
3.

Figure 3 depicts that the text-based reading comprehension datasets increase
linearly (approximately) with increasing access to the context passage. Such a
linear relationship indicates that information required to deduce the correct
answer is evenly distributed throughout the context passage. A similar
behaviour is observed with DREAM, though the slow start indicates that
information may be more disjoint in order to deduce the correct answer as
emphasized in the original release of the DREAM dataset [27]. In contrast, a
very different shape is observed for the speech transcriptions: there is a
sharp increase on the IBM-Debater dataset with increased access to the speech
and then the performance plateaus. Such a shape suggests the information is
front-heavy where it is possible to deduce the side of the argument made in a
speech using the first sentence.

Table 4: Accuracy on IBM with 20% access to the context. | Manual | ASR
---|---|---
Beginning [0-20%] | 64.5 | 65.5
Random | 58.0 | 57.0
End [80-100%] | 52.5 | 55.5

Table 4 further investigates the extent to which information is unevenly
distributed in the IBM-Debater speeches. From Figure 3, 20% is used as an
appropriate operating point to compare the performance with access to only the
beginning extract of the context against the end and random extracts. For both
the manual and the ASR transcriptions the performance is the highest for the
beginning 20% and lowest for the end 20%, confirming the information to deduce
the correct answer is concentrated at the beginning of the context. Future
work should consider evaluating how performance varies with access to the
easiest vs the most difficult sentences as the easiest sections mimic the
parts of the context a language learner understands 222Initial experiments
with sentence complexity based on standard vocabulary levels did not observe a
statistically significant difference between the easiest and most difficult
20% according to text readability..

Content creators are encouraged to plot similar characteristic graphs for
newly proposed questions to gauge the degree of comprehension required by
language learners.

##  6 Conclusions

This work highlights the trade-off between contextual comprehension and world
knowledge in multiple-choice reading and listening comprehension tests. We
found that automated reading comprehension systems perform significantly
better than random, even with limited access to the context passage. These
findings provide content creators with an approach to capture the balance
between comprehension and world knowledge in their questions. We further
investigated to what extent a context needs to be read before the correct
answer can be deduced, finding that it is possible to answer some questions
across several reading/listening comprehension datasets with only access to a
fraction of the context. Overall, our findings guide content creators in
constructing more valid and reliable assessments, ensuring accurate evaluation
of language proficiency.

##  7 Limitations

A limitation for the IBM-Debater dataset is that the contexts have been
truncated to 512 tokens prior to any experiments despite the average length
being approximately 1000 tokens to use the standard pretrained language model
finetuned on RACE++.

## References

  * [1] J. C. Alderson, _Assessing Reading_ , 1st ed. Cambridge :: Cambridge University Press,, 2000.
  * [2] V. Raina and M. Gales, ``Multiple-choice question generation: Towards an automated assessment framework,'' _arXiv preprint arXiv:2209.11830_ , 2022\.
  * [3] A. Liusie, V. Raina, and M. Gales, ````world knowledge'' in multiple choice reading comprehension,'' in _Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER)_. Dubrovnik, Croatia: Association for Computational Linguistics, May 2023, pp. 49–57. [Online]. Available: https://aclanthology.org/2023.fever-1.5
  * [4] A. Liusie, V. Raina, A. Mullooly, K. Knill, and M. J. F. Gales, ``Camchoice: A corpus of multiple choice questions and candidate response distributions,'' 2023\.
  * [5] S. Sugawara, P. Stenetorp, K. Inui, and A. Aizawa, ``Assessing the benchmarking capacity of machine reading comprehension datasets,'' in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, 2020, pp. 8918–8927.
  * [6] D. Kaushik and Z. C. Lipton, ``How much reading does reading comprehension require? a critical investigation of popular benchmarks,'' in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018, pp. 5010–5015.
  * [7] R. Jia and P. Liang, ``Adversarial examples for evaluating reading comprehension systems,'' in _EMNLP_ , 2017.
  * [8] C. Si, S. Wang, M.-Y. Kan, and J. Jiang, ``What does bert learn from multiple-choice reading comprehension datasets?'' _arXiv preprint arXiv:1910.12391_ , 2019.
  * [9] Y.-A. Chung, H.-Y. Lee, and J. Glass, ``Supervised and unsupervised transfer learning for question answering,'' in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 2018, pp. 1585–1594.
  * [10] B.-H. Tseng, S.-s. Shen, H.-Y. Lee, and L.-S. Lee, ``Towards machine comprehension of spoken content: Initial toefl listening comprehension test by machine,'' _Interspeech 2016_ , pp. 2731–2735, 2016.
  * [11] V. Raina and M. J. Gales, ``An initial investigation of non-native spoken question-answering,'' _arXiv preprint arXiv:2107.04691_ , 2021.
  * [12] P. Rajpurkar, R. Jia, and P. Liang, ``Know what you don’t know: Unanswerable questions for squad,'' in _ACL_ , 2018.
  * [13] P. FRIZELLE, C. O'NEILL, and D. V. M. BISHOP, ``Assessing understanding of relative clauses: a comparison of multiple-choice comprehension versus sentence repetition,'' _Journal of Child Language_ , vol. 44, no. 6, p. 1435–1457, 2017.
  * [14] Z. Zhang, J. Yang, and H. Zhao, ``Retrospective reader for machine reading comprehension,'' in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 16, 2021, pp. 14 506–14 514.
  * [15] I. Yamada, A. Asai, H. Shindo, H. Takeda, and Y. Matsumoto, ``Luke: Deep contextualized entity representations with entity-aware self-attention,'' in _EMNLP_ , 2020.
  * [16] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontañón, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, ``Big bird: Transformers for longer sequences,'' _ArXiv_ , vol. abs/2007.14062, 2020\.
  * [17] S. Wang, W. Zhong, D. Tang, Z. Wei, Z. Fan, D. Jiang, M. Zhou, and N. Duan, ``Logic-driven context extension and data augmentation for logical reasoning of text,'' in _Findings of the Association for Computational Linguistics: ACL 2022_ , 2022, pp. 1619–1629.
  * [18] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, ``Think you have solved question answering? try arc, the ai2 reasoning challenge,'' _ArXiv_ , vol. abs/1803.05457, 2018.
  * [19] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, ``Race: Large-scale reading comprehension dataset from examinations,'' in _EMNLP_ , 2017.
  * [20] A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman, ``Newsqa: A machine comprehension dataset,'' in _Rep4NLP@ACL_ , 2017.
  * [21] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, ``Hotpotqa: A dataset for diverse, explainable multi-hop question answering,'' in _EMNLP_ , 2018.
  * [22] W. Yu, Z. Jiang, Y. Dong, and J. Feng, ``Reclor: A reading comprehension dataset requiring logical reasoning,'' in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=HJgJtT4tvB
  * [23] V. Raina and M. Gales, ``Answer uncertainty and unanswerability in multiple-choice machine reading comprehension,'' in _Findings of the Association for Computational Linguistics: ACL 2022_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 1020–1034. [Online]. Available: https://aclanthology.org/2022.findings-acl.82
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [25] Y. Liang, J. Li, and J. Yin, ``A new multi-choice reading comprehension dataset for curriculum learning,'' in _Proceedings of The Eleventh Asian Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, W. S. Lee and T. Suzuki, Eds., vol. 101. Nagoya, Japan: PMLR, 17–19 Nov 2019, pp. 742–757. [Online]. Available: http://proceedings.mlr.press/v101/liang19a.html
  * [26] L. Huang, R. Le Bras, C. Bhagavatula, and Y. Choi, ``Cosmos qa: Machine reading comprehension with contextual commonsense reasoning,'' in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019, pp. 2391–2401.
  * [27] K. Sun, D. Yu, J. Chen, D. Yu, Y. Choi, and C. Cardie, ``Dream: A challenge data set and models for dialogue-based reading comprehension,'' _Transactions of the Association for Computational Linguistics_ , vol. 7, pp. 217–231, 2019.
  * [28] S. Mirkin, G. Moshkowich, M. Orbach, L. Kotlerman, Y. Kantor, T. Lavee, M. Jacovi, Y. Bilu, R. Aharonov, and N. Slonim, ``Listening comprehension over argumentative content,'' in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 719–724. [Online]. Available: https://aclanthology.org/D18-1078
  * [29] K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, ``Electra: Pre-training text encoders as discriminators rather than generators,'' in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=r1xMH1BtvB
  * [30] Y. Zhang and H. Yamana, ``Hrca+: Advanced multiple-choice machine reading comprehension method,'' in _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , 2022, pp. 6059–6068.