# A Quantitative Review on
Language Model Efficiency Research

Meng Jiang, Hy Dang, and Lingbo Tong
Department of Computer Science and Engineering
University of Notre Dame
Notre Dame, IN 46556
{mjiang2, hdang<EMAIL_ADDRESS>

###### Abstract

Language models (LMs) are being scaled and becoming powerful. Improving their
efficiency is one of the core research topics in neural information processing
systems. Tay et al. provided a comprehensive overview of efficient
Transformers that have become an indispensable staple in the field of NLP.
However, in the section of _On Evaluation_ , they left an open question “which
fundamental efficient Transformer one should consider,” answered by “still a
mystery,” because “many research papers select their own benchmarks.”
Unfortunately, there was not quantitative analysis about the performances of
Transformers on any benchmarks. Moreover, state space models (SSMs) have
demonstrated their abilities of modeling long-range sequences with non-
attention mechanisms, which were not discussed in the prior review. This
article makes a _meta analysis_ on the results from a set of papers on
efficient Transformers as well as those on SSMs. It provides a quantitative
review on LM efficiency research and gives suggestions for future research.

##  1 Introduction

Language models are trained to learn the underlying distribution of words in a
given language. A Transformer model is a neural network that learns context
and thus meaning by tracking relationships in sequential data like the words
in natural language (Vaswani et al., 2017). It has been observed that when the
Transformers are scaled to have billions of parameters, the Transformer-based
language models exhibit amazing performance on various language tasks (Wei et
al., 2022). Meanwhile, the efficiency of the language models, in terms of time
and memory complexity, has attracted great attention from research, because it
would address the bottleneck of training and deploying such large-scale
models. Tay et al. (2022a) wrote a survey paper that provided a taxonomy of
efficient Transformer models, characterizing them by their technical
innovation and primary use case. The review was comprehensive, but
unfortunately, readers who really wanted to learn or do _language model
efficiency research_ would not be able to find answers to the following
questions:

  * •

Q1: What were the state-of-the-art efficient language models?

  * •

Q2: Were the results (i.e., performance measures) reported and confirmed by
multiple sources? Were there _inconsistent_ results, i.e., significantly
different measured performances of one type of solutions reported by different
sources?

  * •

Q3: Most studies claimed that they were the best solution when the papers were
submitted or accepted. Were these claims correct?

That’s because the solid answers would require a _quantitative_ analysis
across papers in this research field. The existing survey, in _On Evaluation_
section on page 21, discussed a few NLP benchmarks such as GLUE,
NaturalQuestions, and TriviaQA; and it argued that “many research papers
select their own benchmarks to showcase the abilities of the proposed model,”
leaving it “a mystery to which fundamental efficient Transformer block one
should consider using” (Tay et al., 2022a).

As dozens of related studies are being performed and published, some language
tasks and datasets have been commonly selected as test scenarios to
evaluate/compare model efficiency. Moreover, non-attention models such as
state space models (SSMs) (Gu et al., 2022b) have been proposed to address the
long range modeling problem and evaluated on the NLP benchmarks. Neither their
technical innovations or experimental results were discussed in the existing
survey. A broad, comprehensive, quantitative review is needed in language
model efficiency research. And this article presents the work. It briefly
describes different types of efficient language models, followed by tasks,
datasets, and evaluation metrics. Then it provides a comprehensive
quantitative meta analysis that aims to integrate the results from prior
research to answer the aforementioned questions.

Reducing time and/or memory complexity would inevitably sacrifice a bit non-
efficiency performance like accuracy. When the complexities of a set of models
were reduced to a certain level, one could claim that the model that achieved
the highest accuracy would be the most efficient solution. It sacrificed the
accuracy the least, so people hypothesize that when the models achieved the
same accuracy, this model would have the least complexity. Past empirical
studies unanimously performed efficiency evaluation based on this hypothesis.
Our key observations from a meta analysis on these empirical results are
listed as follows:

  1. 1.

Most empirical studies compared their proposed model against others on
multiple tasks, and usually claimed theirs is the best one. However, the meta
analysis identifies different winning approaches for different tasks and even
different datasets. It fixes the one-sided understanding that researchers
would have from learning only one or a few empirical studies.

  2. 2.

More than half of the results were reported by at least two sources. However,
it is impossible to tell from the papers whether the numbers were
reproduced/confirmed or just re-used from previous work. Meanwhile,
inconsistent results were found on almost every task, caused by various
settings of hyperparameters (e.g., model sizes, configurations) and
reproductions.

  3. 3.

In quite a few studies, the proposed models were evaluated on a small subset
of the tasks, and they actually did _not_ perform better than those that were
published earlier and not cited.

Based on this quantitative review, we offer a few suggestions for future
research:

  1. 1.

Researchers are suggested to investigate as many suitable baselines as
possible, write clearly if the numbers in experimental results were re-used
from prior work or reproduced, and report and analyze any inconsistent results
that are identified compared with related studies.

  2. 2.

Researchers in this field need a community and need a public collection of
leaderboards.

  3. 3.

We hope that this quantitative review is helpful, and if it is, it should be
continuously updated. Quantitative reviews are needed in flourishing research
fields.

##  2 Efficient Language Model Architectures

The core ability of language models is modeling and learning sequential
dependencies which play an essential role in natural language. Besides
traditional recurrent models (e.g., LSTM), there are two popular types of
model architectures: Transformer models and state space models, corresponding
to fully self-attention and non-attention mechanisms.

###  2.1 Transformers

The multi-headed self-attention in Transformers delivers improved parallelism,
enabling more efficient processing of input sequences than recurrent models.
Tay et al. (2022a) provided a taxonomy of Transformer models in its Figure 2
and Section 3.1. The taxonomy has six categories based on core techniques that
improve the memory complexity of the self-attention mechanism: (1)
fixed/factorized patterns, e.g., Sparse Transformer (Child et al., 2019), (2)
learnable patterns, e.g., Reformer (Kitaev et al., 2020), (3) low
rank/kernels, e.g., Linear Transformer (Wang et al., 2020a), (4) recurrence,
e.g., Transformer-XL (Dai et al., 2019), (5) memory/downsampling, e.g.,
Charformer (Tay et al., 2022b), and (6) sparse attention, e.g., Switch
Transformer (Fedus et al., 2022). Some Transformer models adopt multiple core
techniques. Compressive Transformer (Rae et al., 2019) uses both
memory/downsampling and recurrence to compress the Transformer parameters.
Longformer (Beltagy et al., 2020) combines the pattern and downsampling
approaches to reduce the memory complexity.

Table 1: Most popular tasks, datasets, and evaluation metrics for language
model efficiency research

###  2.2 State Space Models

State space models refer to a class of probabilistic graphical models that
describes the probabilistic dependence between the latent state variable and
the observed measurement. They deal with dynamic time series problems in
control engineering and thus have the potential to deal with long-range
dependency (LRD) problems in modeling natural language. Gu et al. (2021)
introduce the Linear State-Space Layer (LSSL), which utilizes an implicit
state to map a 1-dimensional input sequence to an output sequence through the
simulation of a linear continuous-time state-space representation in discrete-
time. LSSLs have been employed to simulate continuous processes, handle
missing data, and adapt to different timescales. However, this work has been
limited due to prohibitive computation and memory requirements induced by the
state representation. This has led to the introduction of the Structured State
Space sequence model (S4), which solves the critical computational bottleneck
(Gu et al., 2022b). The S4 model transforms the state matrices by breaking
them down into a low-rank and normal terms. It computes truncated generating
functions in frequency space, making it simpler to evaluate. S4 significantly
advances the state-of-the-art for low-rank decomposition (LRD), long-range
arena (LRA) tasks, and speech classification with long sequences.

S4 achieves high performance, but the diagonal-plus-low-rank structure
requires several reduction steps and linear algebraic techniques to compute
state space output efficiently, making S4 difficult to analyze. A recent paper
proposes a simpler Diagonal State Space (DSS) (Gupta et al., 2022) model that
enforces diagonal state matrices, making it easier to formulate, implement,
and analyze while being as expressive as general state spaces. Hasani et al.
(2022) use a diagonal plus low-rank decomposition of the state transition
matrix introduced in S4, and a few simplifications, the LTC-based structural
state-space model, dubbed Liquid-S4, achieves the new state-of-the-art
generalization across sequence modeling tasks with long-term dependencies such
as text, image, and audio.

Recently, Smith et al. (2022) introduce a new state space layer – the S5 layer
– which builds on the S4 layer but simplifies it in two main ways. First, S5
uses one multi-input multi-output SSM instead of the bank of many independent
single-input single-output SSMs in S4. Second, S5 uses an efficient parallel
scan instead of the convolutional and frequency-domain approach used by S4.
The resulting S5 layer has the same computational complexity as S4 but
operates purely recurrently and in the time domain. The final S5 layer has
many desirable properties, including linear complexity in the sequence length,
the ability to handle time-varying SSMs and irregularly sampled observations,
and state-of-the-art performance on a variety of long-range sequence modeling
tasks.

Table 2: Selected results on enwik8, a language modeling task dataset. Smaller BPC (bits per character) is better. Bold numbers: claimed as the best results where they were proposed. Background colors: one color indicates one set of inconsistent results. Complete results are in Table 7. Model | BPC $\downarrow$ |  Sources
---|---|---
LN HyperNetworks (Ha et al., 2016) | 1.34 |  Table 2 in (Dai et al., 2019), Table 4 in (Rae et al., 2020)
Locality-Sensitive Hashing (Kitaev et al., 2020) | 1.33 |  Table 5 in (Wang et al., 2021)
LN HM-LSTM (Chung et al., 2016) | 1.32 |  Table 2 in (Dai et al., 2019), Table 4 in (Rae et al., 2020)
RHN (Zilly et al., 2017) | 1.27 |  Table 2 in (Dai et al., 2019), Table 4 in (Rae et al., 2020)
Large mLSTM / mLSTM (Krause et al., 2016) | 1.24 |  Table 2 in (Dai et al., 2019) / Table 4 in (Rae et al., 2020)
Cluster-Former (#C=512) (Wang et al., 2021) | 1.22 |  Table 5 in (Wang et al., 2021)
T12 (Al-Rfou et al., 2019) | 1.11 |  Table 4 in (Wang et al., 2021)
64L Transformer / 64L Transf. (Al-Rfou et al., 2019) | 1.06 |  Table 2 in (Dai et al., 2019) / Table 4 in (Rae et al., 2020)
Transformer-XL / XFM-XL (Dai et al., 2019) | 1.06 |  Table 5 in (Wang et al., 2021), Table 5 in (Ma et al., 2023)
Reformer (Kitaev et al., 2020) | 1.05 |  Table 4 in (Zhu et al., 2021)
Adaptive (Sukhbaatar et al., 2019) | 1.02 |  Table 5 in (Wang et al., 2021)
MEGA (Ma et al., 2023) | 1.02 |  Table 5 in (Ma et al., 2023)
24L Transformer-XL / 24L TXL (Dai et al., 2019) | 0.99 |  Table 2 in (Dai et al., 2019) / Table 4 in (Rae et al., 2020)
Adaptive Transf. (Sukhbaatar et al., 2019) | 0.98 |  Table 4 in (Rae et al., 2020)
24L Compressive Transformer (Rae et al., 2020) | 0.97 |  Table 4 in (Rae et al., 2020)
Table 3: Selected results of empirical studies on NQ (long answer), a popular question answering dataset. Bigger F1 score is better. Results are reported on two evaluation sets: Dev and Test. Model | F1 $\uparrow$ |  Sources
---|---|---
On Dev: |  |
DecAtt (Parikh et al., 2016) \+ DocReader (Chen et al., 2017) | 54.8 |  Table 2 in (Zhang et al., 2021), Table 2 in (Wang et al., 2021)
BERT-large / BERT-large / BERT-joint (Alberti et al., 2019) | 64.7 |  Table 2 in (Ainslie et al., 2020) / Table 2 in (Zhang et al., 2021) / Table 2 in (Wang et al., 2021)
Sparse Transformer / Sparse Attention (Jaszczur et al., 2021) | 74.5 |  Table 2 in (Zhang et al., 2021) / Table 2 in (Wang et al., 2021)
RikiNet-RoBERTa / RikiNet / RikiNet (Liu et al., 2020) | 75.3 |  Table 2 in (Wang et al., 2021) / Table 2 in (Ainslie et al., 2020) / Table 2 in (Zhang et al., 2021)
Reformer / Locality-Sensitive Hashing (Kitaev et al., 2020) | 75.5 |  Table 2 in (Zhang et al., 2021) / Table 2 in (Wang et al., 2021)
RikiNet-ensemble (Liu et al., 2020) | 75.9 |  Table 2 in (Zhang et al., 2021)
Cluster-Former (Wang et al., 2021) | 76.5 |  Table 2 in (Zhang et al., 2021), Table 2 in (Wang et al., 2021)
ReflectionNet-ensemble (Wang et al., 2020b) | 77.0 |  Table 2 in (Zhang et al., 2021)
Poolingformer (Zhang et al., 2021) | 77.5 |  Table 2 in (Zhang et al., 2021)
ETC-large (lifting from RoBERTa) (Ainslie et al., 2020) | 78.2 |  Table 2 in (Ainslie et al., 2020)
On Test: |  |
RikiNet-v2 / RikiNet-ensemble (Liu et al., 2020) | 76.1 |  Table 3 in (Zaheer et al., 2020) / Table 2 in (Zhang et al., 2021)
ReflectionNet (Liu et al., 2020) | 77.1 |  Table 3 in (Zaheer et al., 2020)
ReflectionNet-ensemble (Liu et al., 2020) | 77.2 |  Table 2 in (Zhang et al., 2021)
ETC (official) (Ainslie et al., 2020) | 77.78 |  Table 5 in (Ainslie et al., 2020)
BIGBIRD-ETC (Zaheer et al., 2020) | 77.8 |  Table 3 in (Zaheer et al., 2020), Table 2 in (Zhang et al., 2021)
Cluster-Former-ensemble / Cluster-Former (Wang et al., 2021) | 78 |  Table 3 in (Wang et al., 2021) / Table 2 in (Zhang et al., 2021)
Poolingformer-ensamble (Zhang et al., 2021) | 79.8 |  Table 2 in (Zhang et al., 2021)
Table 4: Selected results on arXiv for document summarization. Bigger R-L (ROUGE-L) is better. Model | R-L $\uparrow$ |  Sources
---|---|---
Long-Doc-Seq2Seq / Discourse-aware (Cohan et al., 2018) | 31.80 |  Table 4 in (Zaheer et al., 2020) / Table 11 in (Beltagy et al., 2020)
Extr-Abst-TLM (Subramanian et al., 2019) | 38.03 |  Table 4 in (Zaheer et al., 2020), Table 11 in (Beltagy et al., 2020), Table 4 in (Zhang et al., 2021)
Sent-PTR(Subramanian et al., 2019) | 38.06 |  Table 4 in (Zaheer et al., 2020), Table 4 in (Zhang et al., 2021)
Dancer / Dancer / Dancer RUM (Gidiotis and Tsoumakas, 2020) | 38.44 |  Table 4 in (Zaheer et al., 2020) / Table 4 in (Zhang et al., 2021) / Table 5 in (Jaszczur et al., 2021)
Pegasus (Zhang et al., 2020) | 38.83 |  Table 4 in (Zaheer et al., 2020), Table 5 in (Jaszczur et al., 2021), Table 11 in (Beltagy et al., 2020), Table 4 in (Zhang et al., 2021)
Pegasus (Re Eval) (Zhang et al., 2020) | 39.17 |  Table 4 in (Zaheer et al., 2020)
Dancer / Dancer PEGASUS (Gidiotis and Tsoumakas, 2020) | 40.56 |  Table 4 in (Zhang et al., 2021) / Table 5 in (Jaszczur et al., 2021)
BIGBIRD-Pegasus / BIGBIRD-Pegasus / BigBird (seqlen:4096) / BigBird (Zaheer et al., 2020) | 41.77 |  Table 4 in (Zaheer et al., 2020) / Table 5 in (Jaszczur et al., 2021) / Table 11 in (Beltagy et al., 2020) / Table 4 in (Zhang et al., 2021)
LED-large (seqlen: 16384) / LED16k (Beltagy et al., 2020) | 41.83 |  Table 11 in (Beltagy et al., 2020) / Table 4 in (Zhang et al., 2021)
Poolingformer16k (Zhang et al., 2021) | 42.69 |  Table 4 in (Zhang et al., 2021)
Table 5: Complete results on GLUE QQP (Quora Question Pairs2) for natural language understanding. Bigger (dev) Accuracy/F1 score is better. Results are sorted by dev F1. Model | Acc |  F1 $\uparrow$ |  Sources
---|---|---|---
BERT (Devlin et al., 2018) | 71.2 | - |  Table 16 in (Zaheer et al., 2020)
Nystromformer (Xiong et al., 2021) | - | 86.3 |  Table 2 in (Xiong et al., 2021)
BERT-base(Devlin et al., 2018) | - | 87.3 |  Table 2 in (Xiong et al., 2021)
DyConv(Wu et al., 2019) | 84.2 | 88.2 |  Table 5 in (Tay et al., 2020a)
BIGBIRD(Zaheer et al., 2020) | 88.6 | - |  Table 16 in (Zaheer et al., 2020)
BERT-base (16GB) (Devlin et al., 2018) | - | 89.6 |  Table 6 in (Ma et al., 2021)
Linformer-128 (16GB) (Wang et al., 2020a) | - | 90.2 |  Table 6 in (Ma et al., 2021)
T5-Base+(Raffel et al., 2020) | 88.3 | 91.2 |  Table 5 in (Tay et al., 2020a)
Luna-128 (160GB) (Ma et al., 2021) | - | 91.3 |  Table 6 in (Ma et al., 2021)
XLNet (Yang et al., 2019) | 91.4 | - |  Table 16 in (Zaheer et al., 2020)
Syn (D+V) (Tay et al., 2020a) | 88.6 | 91.5 |  Table 5 in (Tay et al., 2020a)
RoBERTa (Liu et al., 2019) | 91.9 | - |  Table 16 in (Zaheer et al., 2020)
ROBERTABase / RoBERTa-base (160GB) (Liu et al., 2019) | - | 91.9 |  Table 3 in (Dai et al., 2020) / Table 6 in (Ma et al., 2021)
Transformer (L24H1024) (Vaswani et al., 2017) | 89.6 | 92.2 |  Table 2 in (Dai et al., 2020)
ROBERTALarge (Liu et al., 2019) | - | 92.2 |  Table 3 in (Dai et al., 2020)
XLNetLarge (Yang et al., 2019) | - | 92.3 |  Table 3 in (Dai et al., 2020)
ELECTRALarge (Clark et al., 2020) | - | 92.4 |  Table 3 in (Dai et al., 2020)
Funnel Transformer (B10-10-10H1024) (Dai et al., 2020) | 89.8 | 92.4 |  Table 3 in (Dai et al., 2020)
Table 6: Selected results on LRA-Retrieval. All inconsistent results and SSMs’
results are selected.

Model Acc $\uparrow$ Sources Linformer (Wang et al., 2020a) 52.27 Table 1 in
(Ma et al., 2021), Table 2 in (Ma et al., 2023), Table 10 in (Gu et al.,
2022b) Linformer (Wang et al., 2020a) 53.09 Table 1 in (Hasani et al., 2022)
Reformer (Kitaev et al., 2020) 53.4 Table 1 in (Hasani et al., 2022), Table 10
in (Gu et al., 2022b), Table 2 in (Ma et al., 2023), Table 1 in (Ma et al.,
2021) Performer in (Choromanski et al., 2020) 53.82 Table 1 in (Ma et al.,
2021), Table 2 in (Ma et al., 2023), Table 10 in (Gu et al., 2022b), Table 1
in (Hasani et al., 2022) XFM / Transformer / Transformer / Transformer /
Transformer (Vaswani et al., 2017) 57.46 Table 2 in (Ma et al., 2023) / Table
1 in (Hasani et al., 2022) / Table 1 in (Ma et al., 2021) / Table 10 in (Gu et
al., 2022b) / Table 2 in (Smith et al., 2022) Performer in (Choromanski et
al., 2020) 78.62 Table 3 in (Xiong et al., 2021) Reformer (Kitaev et al.,
2020) 78.64 Table 1 in (Zhu et al., 2021), Table 3 in (Xiong et al., 2021)
Transformer (re-impl) / XFM (re-impl) (Vaswani et al., 2017) 79.14 Table 1 in
(Ma et al., 2021) / Table 2 in (Ma et al., 2023) Luna-256 (Ma et al., 2021)
79.29 Table 10 in (Gu et al., 2022b), Table 2 in (Smith et al., 2022), Table 1
in (Hasani et al., 2022) Standard (Vaswani et al., 2017) 79.35 Table 3 in
(Xiong et al., 2021) Linformer (Wang et al., 2020a) 79.37 Table 3 in (Xiong et
al., 2021), Table 1 in (Zhu et al., 2021) Luna-256 (Ma et al., 2021) 79.56
Table 1 in (Ma et al., 2021), Table 2 in (Ma et al., 2023) Nystromformer
(Xiong et al., 2021) 79.56 Table 3 in (Xiong et al., 2021), Table 1 in (Hasani
et al., 2022), Table 10 in (Gu et al., 2022b) Nystromformer (Xiong et al.,
2021) 81.29 Table 1 in (Zhu et al., 2021) Performer in (Choromanski et al.,
2020) 81.7 Table 1 in (Zhu et al., 2021) Full Attention (Vaswani et al., 2017)
82.3 Table 1 in (Zhu et al., 2021) S4-v2 / S4 (updated) (Gu et al., 2022b)
90.9 Table 2 in (Ma et al., 2023) / Table 10 in (Gu et al., 2022b) S4-v2 (re-
impl) (Gu et al., 2022b) 90.94 Table 2 in (Ma et al., 2023) Liquid-S4 /
Liquid-S4-PB 91.2 Table 2 in (Smith et al., 2022) / Table 1 in (Hasani et al.,
2022) MEGA (Ma et al., 2023) 91.25 Table 2 in (Ma et al., 2023), Table 2 in
(Smith et al., 2022) S5 (Smith et al., 2022) 91.4 Table 2 in (Smith et al.,
2022), Table 1 in (Hasani et al., 2022)

##  3 Meta Analysis

###  3.1 How to Read Our Results

Table 1 summarizes the NLP tasks that at least three studies have used to
evaluate the efficiency of (proposed) language models, along with their
typical datasets and evaluation metrics. Appendix 6 has more concrete
descriptions. Results are presented from Table 2 to Table 24. The results of
the tables in this section are selected from those in Appendix, due to page
limit. They have three columns: model “mentions”, evaluation metric/score, and
“Sources” for where the score was collected from, including the table number
and paper citation. A score may be found in multiple sources. If the sources
mention the model (with a citation in the first column) by different names,
the names and sources are separated by “/” in the first and third columns,
respectively. If the mentions are the same, the sources are separated by “,”
instead.

We highlight a few things in the tables. First, the score is bold if the
source claimed the proposed model achieved the best performance. So we can
find answers to Q1 and Q3 in the Introduction. Second, some model mentions are
highlighted with background colors such as  ,  ,  ,  ,  ,  , and  , if there
are inconsistent results (i.e., different evaluation scores) for the same
model citation in different sets of sources. These extractions are helpful for
Q2.

All the tables are sorted by evaluation scores, from the worst to the best. In
the main sections, we select one dataset per task and select entries to make a
table. An entry is selected if it satisfies any condition: (1) the score is
bold for being claimed as the best; (2) the score is confirmed by more than
one sources; (3) the model mentions are of background colors, associated with
inconsistent results.

###  3.2 Results on Language Modeling

Table 2 contains results from five studies on the enwik8 dataset. _Compressive
Transformer_ achieved the smallest BPC (0.97). Adaptive Transformer in 2019
was the second best (0.98), better than Reformer in 2020 (1.05), Cluster-
Former in 2021 (1.22), and MEGA in 2023 (1.02). There are four sets of
highlighted results: although referencing to the same paper (which helps link
the different model mentions), the results are different for the same dataset,
e.g., Locality-Sensitive Hashing (1.33) and Reformer (1.05), Transformer-
XL/XFM-XL (1.06) and 24L Transformer-XL/24L TXL (0.99), Adaptive (1.02) and
Adaptive Transf. (0.98). But there was no discussion in any paper addressing
the differences. The reason may be different mode configurations.

Table 8 in Appendix has results on the wikitext dataset. _Routing Transformer_
achieved the best PPL (15.8) but was not evaluated on enwik8. Compressive
Transformer was ranked at the 2nd. As many as five different PPL scores were
reported on the Adaptive Transformer model, citing the same paper (Baevski and
Auli, 2018), which were worse than the PPL of proposed models in all the
studies.

###  3.3 Results on Question Answering

Table 3 presents the results of various studies on NQ (long answer). Among the
models evaluated, ETC-large achieved the highest performance on the Dev set
with an F1-score of 78.2 in 2020, followed by Poolingformer in 2021 (77.5).
Both models integrated neural memory modules, suggesting the potential of this
approach. However, while Poolingformer (2021) claimed to be state-of-the-art
in its original paper, it did not compare its performance with ETC-large
(2020). On the Test set, Poolingformer-ensemble achieved the highest F1-score
(79.8), followed by Cluster-Former (78.0), BIGBIRD-ETC (77.8), and ETC
(77.78). The full meta results are in Table 18 in Appendix 7.

Table 19 gives results from studies conducted on NQ (short answer).
ReflectionNet emerged as the best-performing model on both the Dev set (63.4)
and the Test set (64.1), where it significantly outperformed all other models,
including RikiNet (61.3) and Cluster-Former (60.9 on the Test set). The reason
for the outstanding performance ReflectionNet is that it targeted particularly
on the challenge of no-answer condition on the NQ dataset.

Upon comparing results on NQ long answer and short answer, it is evident that
the best-performing models on the two sub tasks are different. For example,
ETC and PoolingFormer outperformed all other models on NQ long answer, but was
exceeded by ReflectionNet on NQ short answer. Similarly, ReflectionNet was the
state-of-the-art method on NQ short answer but was not on NQ long answer.
Interestingly, while the NQ long answer is generally considered more
challenging than the short answer as it requires the model to understand the
context of the question and return a longer piece of text, models generally
achieve a higher F1 score on NQ long answer compared to NQ short answer.

Table 17 presents the results of studies on TriviaQA. BIGBIRD achieved the
best performance on all three sub-tasks with F1 scores of 79.5, 84.5, and 92.4
on Dev, Test, and Test Verified, respectively. The rankings on Test and Test
Verified are consistent, with BIGBIRD-ETC being the best followed by Fusion-
in-Decoder, SpanBERT, and Longformer. The scores on Test Verified are
generally higher, indicating that identifying the absence of the answer in the
contextual passage is still a challenge.

Additionally, it was observed that Switch (published in 2022), the most recent
model in Table 17, only compared its accuracy with T5 in the original paper,
making its comparison with other models unavailable. Furthermore, during the
meta analysis, we discovered that approximately half of the papers on TriviaQA
reported only accuracy, while the other half reported only F1 score. We
recommend that researchers report both results to facilitate comparisons and
enable a more comprehensive evaluation of the models.

To summarize, we found that memory-based models such as Poolingformer, ETC,
and BIGBIRD showed the best performance on QA. In addition, there is a trend
of model integration where models are combining multiple techniques to achieve
better results. For example, Poolingformer incorporates both sliding window
patterns and global memory modules, while BIGBIRD-ETC builds upon ETC and uses
various techniques, including global memory, random attention, and local
sliding windows.

One unique aspect of the QA task is its well-standardized nature in comparison
to other tasks like GLUE. The QA task benefits from a clearly defined dev/test
set split established by the original paper. In addition, papers introducing
results on NQ and TriviaQA rarely contain unambiguous model names (the
highlights in our QA tables indicates variations of the same model rather than
conflicted results). This illustrates the importance of having a public
benchmark for evaluation purposes. It is recommended that other NLP tasks
follow a similar approach to ensure consistency and replicability.

###  3.4 Results on Summarization

Table 4 presents results from studies conducted on arXiv. Poolingformer16k
achieved the highest ROUGE-L score (42.69), followed by LED-large (41.83) and
BIGBIRD-Pegasus (41.77). Since Poolingformer is known as memory-based
transformer, LED (or referred as Longformer) is a variation of sparse
transformer, and BIGBIRD belongs to both categories, the results indicate the
effectiveness of both memory-based and sparse-based approaches on this task.
Full results can be found in Table 9.

During the analysis, we observe that Table 4 reports varying results for the
same model. For instance, the reported performance of Dancer differs across
multiple papers, with scores ranging from 38.44 to 40.56. Similarly, there are
discrepancies in the reported scores for Pegasus, with values of 38.83 and
39.17. These differences in performance may be due to variations in the
dataset splits and/or hyperparameter settings used in the experiments.

###  3.5 Results on GLUE

Table 5 presents the results obtained on the GLUE-QQP benchmark dataset. The
Funnel Transformer with configuration (B10-10-10H1024) achieved the highest
performance, with accuracy (Acc) and F1 score values of 89.8 and 92.4,
respectively. Similarly, Funnel Transformers, which is a Down Sampling
Transformer approach, yielded the best results among 5 out of 8 benchmark
datasets, indicating the effectiveness of this approach. The results for the
remaining GLUE benchmark datasets can be found in the Appendix section,
specifically Table 10 to Table 16.

ELECTRALarge emerged as the second-best model on the GLUE benchmark datasets,
demonstrating competitive performance compared to Funnel Transformers.
ELECTRALarge secured the second position in 6 out of 8 GLUE benchmark datasets
and achieved the state-of-the-art result on GLUE STS-B with a Pearson
correlation coefficient (SC-P) of 92.6, as illustrated in Table 15.

It is worth noting that while multiple papers reference the same study, the
reported results for the same dataset differ. For instance, different papers
report varying scores for BERT-base on the GLUE QQP benchmark dataset, with
values of 87.3 and 89.6 being mentioned.

Furthermore, not all proposed models evaluated their performance on all GLUE
benchmark datasets. Among the studies reviewed, only two papers, focusing on
Synthesizer and Funnel Transformer, respectively, assessed their results
across most of the GLUE benchmark datasets, encompassing eight datasets for
Synthesizer and nine datasets for Funnel Transformer. Additionally, during
this investigation of the GLUE dataset, it was observed that despite claiming
superior results compared to other baselines, Synthesizer did not compare its
performance against earlier models such as XLNet or ROBERTA.

Considering the diverse characteristics of the GLUE datasets and the
experimental settings for each model, direct comparisons of all achieved
results pose challenges. For example, different models underwent pre-training
on distinct datasets. Therefore, to enable more meaningful comparisons among
models on the GLUE datasets, it is suggested to incorporate pre-training on
the same datasets or include all recent models in the comparisons.

###  3.6 Results on Long Range Arena

Table 6 presents the results obtained from seven studies conducted on the LRA-
Retrieval dataset. The remaining LRA benchmark datasets, namely ListOps, Text,
Image, and Pathfinder, are provided in the Appendix. Specifically focusing on
the LRA-Retrieval dataset, S5 achieved the highest performance with a score of
91.4, while MEGA secured the second-best for the same dataset with a score of
91.25.

The Appendix includes Table 20, which showcases the results achieved on the
LRA-ListOps dataset. MEGA attained the highest accuracy (63.14), followed by
Liquid-S4/Liquid S4-PB as the second-best performer (62.75). Similarly, for
the other benchmark datasets within the LRA category, including Text (Table
21), Image (Table 23), and Pathfinder (Table 24), MEGA achieved state-of-the-
art results across all of them, with no reported conflicts in evaluations.
These outcomes highlight the promising performance of the MEGA model within
the LRA benchmark.

In addition to MEGA, the S4-variants and S4 models also demonstrated strong
performance in LRA tasks, achieving competitive results compared to baselines.
This underscores the robust performance of State Space Models.

Based on the results, the most recent models, namely MEGA and SSMs, exhibit
strong performance on the LRAs-benchmark dataset. According to (Ma et al.,
2023), the multi-dimensional damped EMA utilized in MEGA can be considered a
simplified variant of a state space model, establishing a close relationship
between MEGA and S4. However, the difference lies in MEGA not relying on the
HiPPO framework for parameter initialization, distinguishing it from S4 or
S4D.

It is worth noting that despite referencing the same paper, different studies
report varying results for the same dataset. For example, on the LRA-Retrieval
benchmark dataset, Nystromformer’s performance differs across different
papers, with reported scores of 79.56 and 81.29. Similarly, the scores for
XFM/Transformer (57.46), Transformer (re-impl) / XFM (re-impl) (79.14),
Standard (79.35), and Full Attention (82.3) also exhibit discrepancies.
However, the paper does not provide any discussion or analysis addressing
these differences.

Despite these variations, the overall results indicate significant improvement
over Transformer models by utilizing recent models such as SSMs and MEGA to
capture long-range dependencies. These observations present alternative
approaches for analyzing the long-range context.

##  4 Discussions

Key observations and suggestions have been summarized in the Introduction
section. In this section, we discuss a possible action item to implement the
suggestions and its limitations.

Indirect evaluation and comparison are not desired. The real objective should
be time and space complexity when the focus is model efficiency. We should not
compare their marginal differences on accuracy-based performance when the
complexity of the models was not quantitatively reported or analyzed. The
comparison would be meaningless when the model complexities were not
comparable. Therefore, when resources and budget allow, we propose to organize
a competition to have teams develop models for two or three selected tasks
which do not have to cover all the tasks in this review. The competition
organization will provide a platform so the model submissions will be
performed and evaluated by the same hardware and software. We will specific a
few complexity-based metrics to create leaderboads for the tasks. The
submissions will be considered for the leaderboard, only when they achieve
“satisfactory performance” on the corresponding task, which is defined as a
threshold of accuracy/error-based metrics. The shortness of the direct
evaluation and comparison is the financial cost of development of a great
number of models, especially when there is very little room for the top models
to optimize their time and space complexity.

##  5 Conclusion

This was the first literature review on language model efficiency research
that used a quantitative method and meta analysis. It offered a set of
integrated comparative results that would not be observed without such an
effort. It covered both Transformer-based models and the emergent state space
models. The meta analysis finally gave suggestions for future research.

#### Limitations

This quantitative review was developed upon the taxonomy-based literature
review by Tay et al. (2022a). Per Transformer-based models, our review covers
only the ones that have been studied and categorized in the previous review.
We add a set of recent work on state space models. However, the paper
collection was limited to April 1, 2023. We will continuously update this
quantitative review when new methods or even new types of methods emerge.

## References

  * Ainslie et al. [2020] Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang.  Etc: Encoding long and structured inputs in transformers.  _arXiv preprint arXiv:2004.08483_ , 2020.
  * Al-Rfou et al. [2019] Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones.  Character-level language modeling with deeper self-attention.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 3159–3166, 2019.
  * Alberti et al. [2019] Chris Alberti, Kenton Lee, and Michael Collins.  A bert baseline for the natural questions.  _arXiv preprint arXiv:1901.08634_ , 2019.
  * Baevski and Auli [2018] Alexei Baevski and Michael Auli.  Adaptive input representations for neural language modeling.  _arXiv preprint arXiv:1809.10853_ , 2018.
  * Bai et al. [2018a] Shaojie Bai, J Zico Kolter, and Vladlen Koltun.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  _arXiv preprint arXiv:1803.01271_ , 2018a.
  * Bai et al. [2018b] Shaojie Bai, J Zico Kolter, and Vladlen Koltun.  Trellis networks for sequence modeling.  _arXiv preprint arXiv:1810.06682_ , 2018b.
  * Beltagy et al. [2020] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ , 2020.
  * Bradbury et al. [2016] James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher.  Quasi-recurrent neural networks.  _arXiv preprint arXiv:1611.01576_ , 2016.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. [2017] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.  Reading wikipedia to answer open-domain questions.  _arXiv preprint arXiv:1704.00051_ , 2017.
  * Cheng et al. [2023] Lei Cheng, Ruslan Khalitov, Tong Yu, Jing Zhang, and Zhirong Yang.  Classification of long sequential data using circular dilated convolutional neural networks.  _Neurocomputing_ , 518:50–59, 2023.
  * Child et al. [2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.  Generating long sequences with sparse transformers.  _arXiv preprint arXiv:1904.10509_ , 2019.
  * Choromanski et al. [2020] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.  Rethinking attention with performers.  _arXiv preprint arXiv:2009.14794_ , 2020.
  * Chung et al. [2016] Junyoung Chung, Sungjin Ahn, and Yoshua Bengio.  Hierarchical multiscale recurrent neural networks.  _arXiv preprint arXiv:1609.01704_ , 2016.
  * Clark and Gardner [2017] Christopher Clark and Matt Gardner.  Simple and effective multi-paragraph reading comprehension.  _arXiv preprint arXiv:1710.10723_ , 2017.
  * Clark et al. [2020] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_ , 2020.
  * Cohan et al. [2018] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian.  A discourse-aware attention model for abstractive summarization of long documents.  _arXiv preprint arXiv:1804.05685_ , 2018.
  * Dai et al. [2019] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, 2019.
  * Dai et al. [2020] Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le.  Funnel-transformer: Filtering out sequential redundancy for efficient language processing.  _Advances in neural information processing systems_ , 33:4271–4282, 2020.
  * Dauphin et al. [2017] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier.  Language modeling with gated convolutional networks.  In _International conference on machine learning_ , pages 933–941. PMLR, 2017.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Du et al. [2022] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al.  Glam: Efficient scaling of language models with mixture-of-experts.  In _International Conference on Machine Learning_ , pages 5547–5569. PMLR, 2022.
  * Erkan and Radev [2004] Günes Erkan and Dragomir R Radev.  Lexrank: Graph-based lexical centrality as salience in text summarization.  _Journal of artificial intelligence research_ , 22:457–479, 2004.
  * Fedus et al. [2022] William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _The Journal of Machine Learning Research_ , 23(1):5232–5270, 2022.
  * Gidiotis and Tsoumakas [2020] Alexios Gidiotis and Grigorios Tsoumakas.  A divide-and-conquer approach to the summarization of long documents.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:3029–3040, 2020.
  * Grave et al. [2016] Edouard Grave, Armand Joulin, and Nicolas Usunier.  Improving neural language models with a continuous cache.  _arXiv preprint arXiv:1612.04426_ , 2016.
  * Graves [2013] Alex Graves.  Generating sequences with recurrent neural networks.  _arXiv preprint arXiv:1308.0850_ , 2013.
  * Gu et al. [2021] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré.  Combining recurrent, convolutional, and continuous-time models with linear state space layers.  _Advances in neural information processing systems_ , 34:572–585, 2021.
  * Gu et al. [2022a] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré.  On the parameterization and initialization of diagonal state space models.  _Advances in Neural Information Processing Systems_ , 35:35971–35983, 2022a.
  * Gu et al. [2022b] Albert Gu, Karan Goel, and Christopher Re.  Efficiently modeling long sequences with structured state spaces.  In _International Conference on Learning Representations_ , 2022b.
  * Gupta et al. [2022] Ankit Gupta, Albert Gu, and Jonathan Berant.  Diagonal state spaces are as effective as structured state spaces.  _Advances in Neural Information Processing Systems_ , 35:22982–22994, 2022.
  * Ha et al. [2016] David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ , 2016.
  * Hasani et al. [2022] Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus.  Liquid structural state-space models.  _arXiv preprint arXiv:2209.12951_ , 2022.
  * Izacard and Grave [2020] Gautier Izacard and Edouard Grave.  Leveraging passage retrieval with generative models for open domain question answering.  _arXiv preprint arXiv:2007.01282_ , 2020.
  * Jaszczur et al. [2021] Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, and Jonni Kanerva.  Sparse is enough in scaling transformers.  _Advances in Neural Information Processing Systems_ , 34:9895–9907, 2021.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, 2017\.
  * Joshi et al. [2020] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy.  Spanbert: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77, 2020.
  * Kalchbrenner et al. [2016] Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu.  Neural machine translation in linear time.  _arXiv preprint arXiv:1610.10099_ , 2016.
  * Katharopoulos et al. [2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In _International Conference on Machine Learning_ , pages 5156–5165. PMLR, 2020.
  * Kitaev et al. [2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  _arXiv preprint arXiv:2001.04451_ , 2020.
  * Knol [2017] Bryon Knol.  cmix v13, 2017.
  * Krause et al. [2016] Ben Krause, Liang Lu, Iain Murray, and Steve Renals.  Multiplicative lstm for sequence modelling.  _arXiv preprint arXiv:1609.07959_ , 2016.
  * Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466, 2019.
  * Lee-Thorp et al. [2021] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon.  Fnet: Mixing tokens with fourier transforms.  _arXiv preprint arXiv:2105.03824_ , 2021.
  * Lioutas and Guo [2020] Vasileios Lioutas and Yuhong Guo.  Time-aware large kernel convolutions.  In _International Conference on Machine Learning_ , pages 6172–6183. PMLR, 2020.
  * Liu et al. [2020] Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan.  Rikinet: Reading wikipedia pages for natural question answering.  _arXiv preprint arXiv:2004.14560_ , 2020.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Ma et al. [2021] Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer.  Luna: Linear unified nested attention.  _Advances in Neural Information Processing Systems_ , 34:2441–2453, 2021.
  * Ma et al. [2023] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer.  Mega: moving average equipped gated attention.  In _International Conference on Learning Representations_ , 2023.
  * Mujika et al. [2017] Asier Mujika, Florian Meier, and Angelika Steger.  Fast-slow recurrent neural networks.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Nenkova and Vanderwende [2005] Ani Nenkova and Lucy Vanderwende.  The impact of frequency on summarization.  _Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005_ , 101, 2005.
  * Pan et al. [2019] Lin Pan, Rishav Chakravarti, Anthony Ferritto, Michael Glass, Alfio Gliozzo, Salim Roukos, Radu Florian, and Avirup Sil.  Frustratingly easy natural question answering.  _arXiv preprint arXiv:1909.05286_ , 2019.
  * Parikh et al. [2016] Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit.  A decomposable attention model for natural language inference.  _arXiv preprint arXiv:1606.01933_ , 2016.
  * Peng et al. [2021] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong.  Random feature attention.  _arXiv preprint arXiv:2103.02143_ , 2021.
  * Rae et al. [2018] Jack Rae, Chris Dyer, Peter Dayan, and Timothy Lillicrap.  Fast parametric learning with activation memorization.  In _International Conference on Machine Learning_ , pages 4228–4237. PMLR, 2018.
  * Rae et al. [2019] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap.  Compressive transformers for long-range sequence modelling.  _arXiv preprint arXiv:1911.05507_ , 2019.
  * Rae et al. [2020] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap.  Compressive transformers for long-range sequence modelling.  In _International Conference on Learning Representations_ , 2020.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Romero et al. [2022] David W Romero, David M Knigge, Albert Gu, Erik J Bekkers, Efstratios Gavves, Jakub M Tomczak, and Mark Hoogendoorn.  Towards a general purpose cnn for long range dependencies in nd.  _arXiv preprint arXiv:2206.03398_ , 2022.
  * Rothe et al. [2020] Sascha Rothe, Shashi Narayan, and Aliaksei Severyn.  Leveraging pre-trained checkpoints for sequence generation tasks.  _Transactions of the Association for Computational Linguistics_ , 8:264–280, 2020.
  * Roy et al. [2021] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier.  Efficient content-based sparse attention with routing transformers.  _Transactions of the Association for Computational Linguistics_ , 9:53–68, 2021.
  * Santoro et al. [2018] Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap.  Relational recurrent neural networks.  _Advances in neural information processing systems_ , 31, 2018.
  * See et al. [2017] Abigail See, Peter J Liu, and Christopher D Manning.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_ , 2017.
  * Smith et al. [2022] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman.  Simplified state space layers for sequence modeling.  _arXiv preprint arXiv:2208.04933_ , 2022.
  * Song et al. [2020] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  Mpnet: Masked and permuted pre-training for language understanding.  _Advances in Neural Information Processing Systems_ , 33:16857–16867, 2020.
  * Subramanian et al. [2019] Sandeep Subramanian, Raymond Li, Jonathan Pilault, and Christopher Pal.  On extractive and abstractive neural document summarization with transformer language models.  _arXiv preprint arXiv:1909.03186_ , 2019.
  * Sukhbaatar et al. [2019] Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin.  Adaptive attention span in transformers.  _arXiv preprint arXiv:1905.07799_ , 2019.
  * Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ , 27, 2014.
  * Tay et al. [2020a] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng.  Synthesizer: Rethinking self-attention in transformer models.  _ArXiv_ , abs/2005.00743, 2020a.
  * Tay et al. [2020b] Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan.  Sparse sinkhorn attention.  In _International Conference on Machine Learning_ , pages 9438–9447. PMLR, 2020b.
  * Tay et al. [2020c] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler.  Long range arena: A benchmark for efficient transformers.  _arXiv preprint arXiv:2011.04006_ , 2020c.
  * Tay et al. [2021a] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng.  Synthesizer: Rethinking self-attention for transformer models.  In _International conference on machine learning_ , pages 10183–10192. PMLR, 2021a.
  * Tay et al. [2021b] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler.  Long range arena: A benchmark for efficient transformers.  In _International Conference on Learning Representations_ , 2021b.
  * Tay et al. [2022a] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey.  _ACM Computing Surveys_ , 55(6):1–28, 2022a.
  * Tay et al. [2022b] Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler.  Charformer: Fast character transformers via gradient-based subword tokenization.  In _International Conference on Learning Representations_ , 2022b.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2021] Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, and Jingjing Liu.  Cluster-former: Clustering-based sparse transformer for question answering.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3958–3968, 2021.
  * Wang et al. [2020a] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_ , 2020a.
  * Wang et al. [2020b] Xuguang Wang, Linjun Shou, Ming Gong, Nan Duan, and Daxin Jiang.  No answer is better than wrong answer: A reflection model for document level machine reading comprehension.  _arXiv preprint arXiv:2009.12056_ , 2020b.
  * Wei et al. [2022] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022.
  * Wiseman et al. [2017] Sam Wiseman, Stuart M Shieber, and Alexander M Rush.  Challenges in data-to-document generation.  _arXiv preprint arXiv:1707.08052_ , 2017.
  * Wu et al. [2019] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli.  Pay less attention with lightweight and dynamic convolutions.  _arXiv preprint arXiv:1901.10430_ , 2019.
  * Xiong et al. [2021] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh.  Nyströmformer: A nyström-based algorithm for approximating self-attention.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 14138–14148, 2021.
  * Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32, 2019.
  * Ye et al. [2019] Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang.  Bp-transformer: Modelling long-range context via binary partitioning.  _arXiv preprint arXiv:1911.04070_ , 2019.
  * Yu et al. [2021] Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, and Michael Zeng.  Kg-fid: Infusing knowledge graph in fusion-in-decoder for open-domain question answering.  _arXiv preprint arXiv:2110.04330_ , 2021.
  * Zaheer et al. [2020] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al.  Big bird: Transformers for longer sequences.  _Advances in neural information processing systems_ , 33:17283–17297, 2020.
  * Zhang et al. [2021] Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen.  Poolingformer: Long document modeling with pooling attention.  In _International Conference on Machine Learning_ , pages 12437–12446. PMLR, 2021.
  * Zhang et al. [2020] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR, 2020.
  * Zhu et al. [2021] Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro.  Long-short transformer: Efficient transformers for language and vision.  _Advances in Neural Information Processing Systems_ , 34:17723–17736, 2021.
  * Zhu and Soricut [2021] Zhenhai Zhu and Radu Soricut.  H-transformer-1d: Fast one-dimensional hierarchical attention for sequences.  _arXiv preprint arXiv:2107.11906_ , 2021.
  * Zilly et al. [2017] Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnık, and Jürgen Schmidhuber.  Recurrent highway networks.  In _International conference on machine learning_ , pages 4189–4198, 2017.

##  6 Appendix: Evaluation Methods on Language Model Efficiency

In this section, also as shown in Table 1, we summarize the NLP tasks that at
least three studies have used to evaluate the efficiency of (proposed)
language models, followed by the description of their typical datasets and
evaluation metrics.

###  6.1 Tasks

#### Language Modeling

is a task that involves predicting the likelihood of a sequence of words in a
language. The goal of language modeling is to learn the underlying patterns
and structure of a language, which can be used to generate new text or to
improve the performance of other NLP tasks.

#### Question Answering

involves answering a question posed in natural language. The goal is to build
an automated system that can understand natural language questions and provide
accurate answers.

#### Summarization

involves creating a condensed version of a longer piece of text while
retaining its main ideas and important information. The goal is to make large
amounts of information more accessible and easier to understand, while also
saving time and effort.

#### Natural Language Understanding / Inference

provides the fundamental abilities in applications in machine translation,
text categorization, speech recognition, and large-scale content analysis.

#### Long-Range Sequence Modeling

tests the abilities of learning dependencies in long-context scenarios, such
as byte-level language modeling and byte-level document classification.

###  6.2 Datasets

#### enwik8

consists of the first 100 million bytes of the English Wikipedia, which
contains a diverse range of text, including articles, tables, and lists. It
includes a wide range of linguistic phenomena, including spelling and
grammatical errors, proper names, technical jargon, and more.

#### wikitext

contains over 100 million words from a larger and more diverse set of
Wikipedia articles (named wikitext-103), processed for language modeling
tasks.

#### TriviaQA

consists of a large collection of trivia questions and their corresponding
answers, sourced from a variety of sources, including quiz bowl competitions,
trivia websites, and more. Many of the questions require deep comprehension
and reasoning skills, rather than simple keyword matching.

#### NQ

consists of over 300,000 real, anonymized, naturally occurring questions and
answers (NaturalQuestions), sampled from a wide range of sources, such as
community forums, how-to sites, and news articles. The questions are diverse
and cover a wide range of topics, from history to pop culture.

#### arXiv

is a big collection of research papers in the fields of physics, mathematics,
computer science, and other related disciplines, which is commonly used for
document summarization.

#### GLUE

short for General Language Understanding Evaluation, is a benchmark dataset
consisting nine different tasks that cover a range of natural language
inference (NLI) or understanding (NLU) tasks such as sentiment analysis,
natural language inference, paraphrase detection, and more:

  * •

CoLA (Corpus of Linguistic Acceptability) involves determining whether a given
sentence is grammatically correct or not.

  * •

SST-2 (Stanford Sentiment Treebank) involves determining the sentiment
(positive or negative) of a given sentence.

  * •

MRPC (Microsoft Research Paraphrase Corpus) involves determining whether two
given sentences are semantically equivalent or not.

  * •

STS-B (Semantic Textual Similarity Benchmark) involves determining the degree
of semantic similarity between two given sentences.

  * •

QQP (Quora Question Pairs) involves determining whether two given questions
are semantically equivalent or not.

  * •

MNLI (Multi-Genre Natural Language Inference) involves determining the
relationship between a given pair of sentences (entailment, contradiction, or
neutral).

  * •

QNLI (Question NLI) involves determining the relationship between a given pair
of sentences (entailment or not).

  * •

RTE (Recognizing Textual Entailment) involves determining whether a given
sentence entails another given sentence or not.

  * •

WNLI (Winograd NLI) involves resolving pronoun references in a given sentence
to determine whether it entails another given sentence or not.

#### LRA

short for Long-Range Arena, is a benchmark dataset consisting multiple tasks
to evaluate sequence models on long-context scenarios:

  * •

ListOps is designed to investigate the abilities of modeling hierarchically
structured data in a long-context scenario, specifically, the parsing ability
of neural models.

  * •

Text refers to byte-level text classification, dealing with compositionality
as it is required to.

  * •

Retrieval refers to byte-level document retrieval, learning compressed text
representations.

  * •

Image refers to image classification, requiring to learn the 2D spatial
relations between input pixels, while presented as a 1D sequence of symbols.

  * •

Pathfinder refers to learn long-range spatial dependencies, as a synthetic
visual task motivated by cognitive psychology. We skip the extreme version
(Path-X) and focus on text modeling tasks.

###  6.3 Evaluation Metrics

#### BPC

short for Bits per Character, measures the average number of bits required to
encode each character in a generated text sequence. It is used for evaluating
the performance of language models on language modeling tasks. The lower the
BPC score, the better the model is at generating text that closely matches the
distribution of the training data.

#### Perplexity

measures how well a probability model predicts a sample. In simpler terms, it
measures how well a language model can predict the next word in a sequence
given the previous words. A lower perplexity indicates that the language model
is better at predicting the next word in a sequence.

#### Accuracy

measures how many predictions of a language model are correct, which can be
applied for a variety of tasks. So it’s used in both GLUE and LRA benchmarks,
as well as some other tasks.

#### F1

is a combined measure of precision and recall for binary classification.
Question answering, retrieval, and sentiment prediction use F1 score for
evaluation, for being considered as the classification tasks. The F1 score
ranges from 0 to 1, with 1 being the best possible score.

#### ROUGE-L

measures the overlap between the generated summary and the reference summary,
where the reference summary is a human-written summary of the same document.
Specifically, it measures the Longest common subsequence between the generated
and reference summaries.

#### MCC

stands for Matthews Correlation Coefficient, and it measures the correlation
between the predicted and actual labels in a binary classification task. It is
particularly useful when dealing with imbalanced datasets where one class is
significantly more prevalent than the other.

#### SC-P/SC-S

stands for Spearman’s rank correlation coefficient on population or samples.
It measures the strength of the association between two variables, ranging
from -1 to 1.

##  7 Appendix: Extra Results from Meta Analysis

#### Results of empirical studies on _language modeling_

Table 7 presents the BPC of 24 models by integrating results from four
research papers on the enwik8 dataset. Seven of the results are confirmed or
re-used by more than one study. Important prior work that contributed a better
results was ignored in some of the studies. And there are at least four sets
of inconsistent results, i.e., different values for the same citation.
Selected entries are discussed in main content of this paper.

Table 7: Results of empirical studies on enwik8, a dataset for language modeling task. Smaller BPC (bits per character) is better. Bold numbers: claimed as the best results where they were proposed. Background colors: one color indicates one set of inconsistent results. Model | BPC $\downarrow$ |  Sources
---|---|---
7L LSTM [Graves, 2013] | 1.67 |  Table 4 in [Rae et al., 2020]
Sliding window | 1.34 |  Table 5 in [Wang et al., 2021]
LN HyperNetworks [Ha et al., 2016] | 1.34 |  Table 2 in [Dai et al., 2019], Table 4 in [Rae et al., 2020]
Locality-Sensitive Hashing [Kitaev et al., 2020] | 1.33 |  Table 5 in [Wang et al., 2021]
LN HM-LSTM [Chung et al., 2016] | 1.32 |  Table 2 in [Dai et al., 2019], Table 4 in [Rae et al., 2020]
ByteNet [Kalchbrenner et al., 2016] | 1.31 |  Table 4 in [Rae et al., 2020]
Sparse Attention [Child et al., 2019] | 1.29 |  Table 5 in [Wang et al., 2021]
RHN [Zilly et al., 2017] | 1.27 |  Table 2 in [Dai et al., 2019], Table 4 in [Rae et al., 2020]
FS-LSTM-4 [Mujika et al., 2017] | 1.25 |  Table 2 in [Dai et al., 2019]
Large mLSTM / mLSTM [Krause et al., 2016] | 1.24 |  Table 2 in [Dai et al., 2019] / Table 4 in [Rae et al., 2020]
cmix v13 [Knol, 2017] | 1.23 |  Table 2 in [Dai et al., 2019]
Cluster-Former (#C=512) [Wang et al., 2021] | 1.22 |  Table 5 in [Wang et al., 2021]
T12 [Al-Rfou et al., 2019] | 1.11 |  Table 4 in [Wang et al., 2021]
64L Transformer / 64L Transf. [Al-Rfou et al., 2019] | 1.06 |  Table 2 in [Dai et al., 2019] / Table 4 in [Rae et al., 2020]
Transformer-XL / XFM-XL [Dai et al., 2019] | 1.06 |  Table 5 in [Wang et al., 2021], Table 5 in [Ma et al., 2023]
Reformer [Kitaev et al., 2020] | 1.05 |  Table 4 in [Zhu et al., 2021]
Adaptive [Sukhbaatar et al., 2019] | 1.02 |  Table 5 in [Wang et al., 2021]
MEGA [Ma et al., 2023] | 1.02 |  Table 5 in [Ma et al., 2023]
BP-Transformer [Ye et al., 2019] | 1.02 |  Table 5 in [Wang et al., 2021]
Longformer [Tay et al., 2021b] | 1.00 |  Table 5 in [Wang et al., 2021]
24L Transformer-XL / 24L TXL [Dai et al., 2019] | 0.99 |  Table 2 in [Dai et al., 2019] / Table 4 in [Rae et al., 2020]
Adaptive Transf. [Sukhbaatar et al., 2019] | 0.98 |  Table 4 in [Rae et al., 2020]
24L Compressive Transformer [Rae et al., 2020] | 0.97 |  Table 4 in [Rae et al., 2020]

Table 8 presents the perplexity of 28 models by integrating results from five
research papers on the wikitext dataset. Nine of the results are confirmed or
re-used by more than one study. This table also reflects missing important
prior work and inconsistent results.

#### Results of empirical studies on _document summarization_

Table 9 presents the ROUGE-L values of 19 models by integrating results from
three research papers on the arXiv dataset. Eight of the results are confirmed
or re-used by more than one study.

Table 8: Results of empirical studies on wikitext, a dataset for language modeling task. Smaller PPL (perplexity) is better. Bold numbers: claimed as the best results where they were proposed. Background colors: one color indicates one set of inconsistent results. Model | PPL $\downarrow$ |  Sources
---|---|---
LSTM | 48.7 |  Table 5 in [Rae et al., 2020], Table 1 in [Dai et al., 2019]
Temporal CNN / TCN [Bai et al., 2018a] | 45.2 |  Table 5 in [Rae et al., 2020] / Table 1 in [Dai et al., 2019]
LSTMs / LSTM + Neural cache [Grave et al., 2016] | 40.8 |  Table 2 in [Roy et al., 2021] / Table 1 in [Dai et al., 2019]
GLU CNN / GCNN-14 / GCNN-14 [Dauphin et al., 2017] | 37.2 |  Table 8 in [Gu et al., 2022b] / Table 5 in [Rae et al., 2020] / Table 1 in [Dai et al., 2019]
AWD-QRNN / Quasi-RNN / QRNNs / QRNN [Bradbury et al., 2016] | 33 |  Table 8 in [Gu et al., 2022b] / Table 5 in [Rae et al., 2020] / Table 2 in [Roy et al., 2021] / Table 1 in [Dai et al., 2019]
RMC [Santoro et al., 2018] | 31.9 |  Table 5 in [Rae et al., 2020]
Hebbian + Cache [Rae et al., 2018] | 29.9 |  Table 1 in [Dai et al., 2019]
LSTM + Hebb. [Rae et al., 2018] | 29.2 |  Table 8 in [Gu et al., 2022b], Table 5 in [Rae et al., 2020]
TrellisNet [Bai et al., 2018b] | 29.19 |  Table 8 in [Gu et al., 2022b]
Transformer [Baevski and Auli, 2018] | 26.2 |  Table 1 in [Peng et al., 2021]
Dynamic Conv [Wu et al., 2019] | 25 |  Table 8 in [Gu et al., 2022b]
RFA [Peng et al., 2021] | 23.5 |  Table 1 in [Peng et al., 2021]
TaLK Conv [Lioutas and Guo, 2020] | 23.3 |  Table 8 in [Gu et al., 2022b]
S4 [Gu et al., 2022b] | 21.28 |  Table 8 in [Gu et al., 2022b], Table 5 in [Ma et al., 2023]
Sliding window | 20.8 |  Table 5 in [Wang et al., 2021]
Locality-Sensitive Hashing [Kitaev et al., 2020] | 20.8 |  Table 5 in [Wang et al., 2021]
Adaptive Transformer [Sukhbaatar et al., 2019] | 20.6 |  Table 2 in [Roy et al., 2021]
Transformer [Baevski and Auli, 2018] | 20.51 |  Table 8 in [Gu et al., 2022b]
Sparse Attention [Child et al., 2019] | 20.5 |  Table 5 in [Wang et al., 2021]
Adaptive Input [Baevski and Auli, 2018] | 20.5 |  Table 1 in [Dai et al., 2019]
Clusterformer [Wang et al., 2021] | 20.2 |  Table 4 in [Wang et al., 2021]
Local Transformer [Vaswani et al., 2017] | 19.8 |  Table 2 in [Roy et al., 2021]
Transformer / Adaptive Input [Baevski and Auli, 2018] | 18.7 |  Table 5 in [Rae et al., 2020] / Table 2 in [Roy et al., 2021]
XFM-adaptive [Baevski and Auli, 2018, Al-Rfou et al., 2019] | 18.66 |  Table 5 in [Ma et al., 2023]
XFM-XL / 18L TransformerXL / TransformerXL / TransformerXL Large [Dai et al., 2019] | 18.3 |  Table 5 in [Ma et al., 2023] / Table 5 in [Rae et al., 2020] / Table 2 in [Roy et al., 2021] / Table 1 in [Dai et al., 2019]
MEGA [Ma et al., 2023] | 18.07 |  Table 5 in [Ma et al., 2023]
Compressive Transformer [Rae et al., 2020] | 17.1 |  Table 5 in [Rae et al., 2020]
Routing Transformer [Roy et al., 2021] | 15.8 |  Table 2 in [Roy et al., 2021]
Table 9: Results of empirical studies on arXiv, a dataset for document summarization task. Bigger R-L (ROUGE-L) is better. Bold numbers: claimed as the best results where they were proposed. Background colors: one color indicates one set of inconsistent results. Model | R-L $\uparrow$ |  Sources
---|---|---
Pntr-Gen-Seq2Seq [See et al., 2017] | 25.16 |  Table 4 in [Zaheer et al., 2020]
Attn-Seq2Seq [Sutskever et al., 2014] | 25.56 |  Table 4 in [Zaheer et al., 2020]
Transformer [Vaswani et al., 2017] | 25.58 |  Table 4 in [Zaheer et al., 2020]
LSA [Wiseman et al., 2017] | 25.67 |  Table 4 in [Zaheer et al., 2020]
SumBasic [Nenkova and Vanderwende, 2005] | 26.30 |  Table 4 in [Zaheer et al., 2020]
LexRank [Erkan and Radev, 2004] | 28.99 |  Table 4 in [Zaheer et al., 2020]
Transformer + RoBERTa[Rothe et al., 2020] | 29.53 |  Table 4 in [Zaheer et al., 2020]
Transformer + Pegasus [Zhang et al., 2020] | 30.14 |  Table 4 in [Zaheer et al., 2020]
Long-Doc-Seq2Seq / Discourse-aware [Cohan et al., 2018] | 31.80 |  Table 4 in [Zaheer et al., 2020] / Table 11 in [Beltagy et al., 2020]
Extr-Abst-TLM [Subramanian et al., 2019] | 38.03 |  Table 4 in [Zaheer et al., 2020], Table 11 in [Beltagy et al., 2020], Table 4 in [Zhang et al., 2021]
Sent-PTR[Subramanian et al., 2019] | 38.06 |  Table 4 in [Zaheer et al., 2020], Table 4 in [Zhang et al., 2021]
Dancer / Dancer / Dancer RUM [Gidiotis and Tsoumakas, 2020] | 38.44 |  Table 4 in [Zaheer et al., 2020] / Table 4 in [Zhang et al., 2021] / Table 5 in [Jaszczur et al., 2021]
Pegasus [Zhang et al., 2020] | 38.83 |  Table 4 in [Zaheer et al., 2020], Table 5 in [Jaszczur et al., 2021], Table 11 in [Beltagy et al., 2020], Table 4 in [Zhang et al., 2021]
Pegasus (Re Eval) [Zhang et al., 2020] | 39.17 |  Table 4 in [Zaheer et al., 2020]
Dancer / Dancer PEGASUS [Gidiotis and Tsoumakas, 2020] | 40.56 |  Table 4 in [Zhang et al., 2021] / Table 5 in [Jaszczur et al., 2021]
Terraformer [Jaszczur et al., 2021] | 41.21 |  Table 5 in [Jaszczur et al., 2021]
BIGBIRD-Pegasus / BIGBIRD-Pegasus / BigBird (seqlen:4096) / BigBird [Zaheer et al., 2020] | 41.77 |  Table 4 in [Zaheer et al., 2020] / Table 5 in [Jaszczur et al., 2021] / Table 11 in [Beltagy et al., 2020] / Table 4 in [Zhang et al., 2021]
LED-large (seqlen: 16384) / LED16k [Beltagy et al., 2020] | 41.83 |  Table 11 in [Beltagy et al., 2020] / Table 4 in [Zhang et al., 2021]
Poolingformer16k [Zhang et al., 2021] | 42.69 |  Table 4 in [Zhang et al., 2021]

#### Results of empirical studies on _natural language understanding /
inference_

Table 5, Table 10, Table 11, Table 13, Table 12, Table 14, Table 15, and Table
16 present the results on eight natural language understanding and/or
inference tasks in the GLUE benchmark:

  * •

QQP: Quora Question Pairs2, evaluated by dev accuracy and F1 score;

  * •

MNLI: Multi-Genre Natural Language Inference, evaluated by accuracy on match
(m) and mismatch (mm);

  * •

SST-2: Stanford Sentiment Treebank, about sentiment analysis, evaluated by
accuracy;

  * •

QNLI: Question-answering Natural Language Inference, evaluated by dev
accuracy;

  * •

COLA: Corpus of Linguistic Acceptability, evaluated by dev Matthew’s
correlation coefficient (MCC);

  * •

RTE: Recognizing Textual Entailment, evaluated by accuracy;

  * •

STS-B: (Semantic Textual Similarity Benchmark, evaluated by dev Spearman
correlation P (SC-P) and dev Spearman correlation S (SC-S);

  * •

MRPC: Microsoft Research Paraphrase Corpus, evaluated by dev accuracy and F1
score.

#### Results of empirical studies on _question answering_

There are two datasets that are commonly used to evaluate the efficiency of
language models. TriviaQA is a challenging reading comprehension dataset
containing over 650K question-answer-evidence triples, which was collected by
University of Washington and published in 2017 Joshi et al. [2017]. The other
is Natural Questions (NQ), a benchmark for question answering research
collected by Google and published in 2019 Kwiatkowski et al. [2019].

Table 17 presents the quantitative results on the TriviaQA dataset. The
evaluation metrics are accuracy and F1 score. Different studies may use
different metrics. Towards evaluation, the studies may use the
development/validation set (Dev), test set (Test), or verified test set (Test
Verified). And the results across different sets are not comparable.

Table 18 and Table 19 present the quantitative results on the NQ datasets,
long answers and short answers, respectively. The studies use the Dev and Test
sets for evaluation. The studies were not as many as those on other tasks, and
no inconsistent results were observed.

#### Results of empirical studies on _long-range arena_

Table 20, Table 21, Table 22, Table 23, and Table 24 present the results on
five tasks in the LRA benchmark, such as ListOps, Text, Retrieval, Image, and
Pathfinder, all evaluated by accuracy. Table 6 has results on the LRA-
Retrieval dataset, which focuses on

Table 10: Results of empirical studies on GLUE MNLI (Multi-Genre Natural
Language Inference). Bigger (dev) Accuracy on match (m) / Accuracy on mismatch
(mm) is better.

Model Acc (m) $\uparrow$ Acc (mm) Sources DyConv [Wu et al., 2019] 73.8 75.1
Table 5 in [Tay et al., 2020a] Nystromformer [Xiong et al., 2021] 80.9 82.2
Table 2 in [Xiong et al., 2021] BERT-base[Devlin et al., 2018] 82.4 82.4 Table
2 in [Xiong et al., 2021] BERT [Devlin et al., 2018] 84.6 83.4 Table 16 in
[Zaheer et al., 2020] T5-Base [Raffel et al., 2020] 84.7 85 Table 5 in [Tay et
al., 2020a] Syn (R+V) [Tay et al., 2020a] 85 84.6 Table 5 in [Tay et al.,
2020a] XLNet [Yang et al., 2019] 86.8 - Table 16 in [Zaheer et al., 2020]
BIGBIRD [Zaheer et al., 2020] 87.5 87.3 Table 16 in [Zaheer et al., 2020]
ROBERTABase / RoBERTa [Liu et al., 2019] 87.6 - Table 3 in [Dai et al., 2020]
/ Table 16 in [Zaheer et al., 2020] MPNetBase [Song et al., 2020] 88.5 - Table
3 in [Dai et al., 2020] Transformer [Vaswani et al., 2017] 89.4 - Table 2 in
[Dai et al., 2020] ROBERTALarge [Liu et al., 2019] 90.2 - Table 3 in [Dai et
al., 2020] XLNetLarge [Yang et al., 2019] 90.8 - Table 3 in [Dai et al., 2020]
ELECTRALarge [Clark et al., 2020] 90.9 - Table 3 in [Dai et al., 2020] Funnel
Transformer (B10-10-10H1024) [Dai et al., 2020] 91.1 - Table 3 in [Dai et al.,
2020]

Table 11: Results of empirical studies on GLUE SST-2 (Stanford Sentiment
Treebank), a dataset for sentiment analysis. Bigger accuracy is better.

Model Acc $\uparrow$ Sources BERT-base [Devlin et al., 2018] 90 Table 2 in
[Xiong et al., 2021] DyConv [Wu et al., 2019] 90.6 Table 5 in [Tay et al.,
2020a] Nystromformer [Xiong et al., 2021] 91.4 Table 2 in [Xiong et al., 2021]
Syn (D+V) [Tay et al., 2020a] 92.4 Table 5 in [Tay et al., 2020a]
Linformer-128 (16GB) [Wang et al., 2020a] 92.4 Table 6 in [Ma et al., 2021]
BERT-base [Devlin et al., 2018] 92.7 Table 6 in [Ma et al., 2021]
T5-Base+[Raffel et al., 2020] 92.9 Table 5 in [Tay et al., 2020a] BERT [Devlin
et al., 2018] 93.5 Table 16 in [Zaheer et al., 2020] BIGBIRD [Zaheer et al.,
2020] 94.6 Table 16 in [Zaheer et al., 2020] Luna-128 (160GB) [Ma et al.,
2021] 94.6 Table 6 in [Ma et al., 2021] XLNet [Yang et al., 2019] 94.7 Table
16 in [Zaheer et al., 2020] Transformer (L24H1024) [Vaswani et al., 2017] 94.8
Table 2 in [Dai et al., 2020] ROBERTABase / RoBERTa / RoBERTa-base (160GB)
[Liu et al., 2019] 94.8 Table 3 in [Dai et al., 2020] / Table 16 in [Zaheer et
al., 2020] / Table 6 in [Ma et al., 2021] MPNetBase [Song et al., 2020] 95.4
Table 3 in [Dai et al., 2020] ROBERTALarge [Liu et al., 2019] 96.4 Table 3 in
[Dai et al., 2020] Funnel Transformer (B10-10-10H1024) [Dai et al., 2020] 96.8
Table 3 in [Dai et al., 2020] ELECTRALarge [Clark et al., 2020] 96.9 Table 3
in [Dai et al., 2020] XLNetLarge [Yang et al., 2019] 97 Table 3 in [Dai et
al., 2020]

Table 12: Results of empirical studies on GLUE COLA (Corpus of Linguistic
Acceptability). Bigger (dev) Matthew’s correlation coefficient (MCC) is
better.

Model MCC $\uparrow$ Sources DyConv [Wu et al., 2019] 33.9 Table 5 in [Tay et
al., 2020a] BERT [Devlin et al., 2018] 52.1 Table 16 in [Zaheer et al., 2020]
Syn (R+V) [Tay et al., 2020a] 53.3 Table 5 in [Tay et al., 2020a] T5-Base+
[Raffel et al., 2020] 54.3 Table 5 in [Tay et al., 2020a] BIGBIRD [Zaheer et
al., 2020] 58.5 Table 16 in [Zaheer et al., 2020] XLNet [Yang et al., 2019]
60.2 Table 16 in [Zaheer et al., 2020] ROBERTABase / RoBERTa [Liu et al.,
2019] 63.6 Table 3 in [Dai et al., 2020] / Table 16 in [Zaheer et al., 2020]
MPNetBase [Song et al., 2020] 65 Table 3 in [Dai et al., 2020] Transformer
(L24H1024) [Vaswani et al., 2017] 66.5 Table 2 in [Dai et al., 2020]
ROBERTALarge [Liu et al., 2019] 68 Table 3 in [Dai et al., 2020] XLNetLarge
[Yang et al., 2019] 69 Table 3 in [Dai et al., 2020] ELECTRALarge [Clark et
al., 2020] 69.1 Table 3 in [Dai et al., 2020] Funnel Transformer
(B10-10-10H1024) [Dai et al., 2020] 88.7 Table 3 in [Dai et al., 2020]

Table 13: Results of empirical studies on GLUE QNLI (Question-answering
Natural Language Inference). Bigger (dev) accuracy is better.

Model Acc $\uparrow$ Sources DyConv [Wu et al., 2019] 84.4 Table 5 in [Tay et
al., 2020a] BERT-base [Devlin et al., 2018] 88.4 Table 6 in [Ma et al., 2021]
Nystromformer [Xiong et al., 2021] 88.7 Table 2 in [Xiong et al., 2021] BERT-
base [Devlin et al., 2018] 90.3 Table 2 in [Xiong et al., 2021] Linformer-128
(16GB) [Wang et al., 2020a] 90.4 Table 6 in [Ma et al., 2021] BERT [Devlin et
al., 2018] 90.5 Table 16 in [Zaheer et al., 2020] T5-Base [Raffel et al.,
2020] 91.7 Table 5 in [Tay et al., 2020a] XLNet [Yang et al., 2019] 91.7 Table
16 in [Zaheer et al., 2020] BIGBIRD [Zaheer et al., 2020] 92.2 Table 16 in
[Zaheer et al., 2020] Luna-128 (160GB) [Ma et al., 2021] 92.2 Table 6 in [Ma
et al., 2021] Syn (R+V) [Tay et al., 2020a] 92.3 Table 5 in [Tay et al.,
2020a] ROBERTABase / RoBERTa / RoBERTa-base [Liu et al., 2019] 92.8 Table 3 in
[Dai et al., 2020] / Table 16 in [Zaheer et al., 2020] / Table 6 in [Ma et
al., 2021] MPNetBase [Song et al., 2020] 93.3 Table 3 in [Dai et al., 2020]
Transformer (L24H1024)[Vaswani et al., 2017] 94.1 Table 2 in [Dai et al.,
2020] ROBERTALarge [Liu et al., 2019] 94.7 Table 3 in [Dai et al., 2020]
XLNetLarge [Yang et al., 2019] 94.9 Table 3 in [Dai et al., 2020] ELECTRALarge
[Clark et al., 2020] 95 Table 3 in [Dai et al., 2020] Funnel Transformer
(B10-10-10H1024) [Dai et al., 2020] 95.1 Table 3 in [Dai et al., 2020]

Table 14: Results of empirical studies on GLUE RTE (Recognizing Textual
Entailment).

Model Acc $\uparrow$ Sources DyConv [Wu et al., 2019] 58.1 Table 5 in [Tay et
al., 2020a] BERT [Devlin et al., 2018] 66.4 Table 16 in [Zaheer et al., 2020]
XLNet [Yang et al., 2019] 74 Table 16 in [Zaheer et al., 2020] BIGBIRD [Zaheer
et al., 2020] 75 Table 16 in [Zaheer et al., 2020] ROBERTABase / RoBERTa [Liu
et al., 2019] 78.7 Table 3 in [Dai et al., 2020] / Table 16 in [Zaheer et al.,
2020] T5-Base+ [Raffel et al., 2020] 79.1 Table 5 in [Tay et al., 2020a] Syn
(R+V) [Tay et al., 2020a] 81.2 Table 5 in [Tay et al., 2020a] Transformer
(L24H1024) [Vaswani et al., 2017] 84.5 Table 2 in [Dai et al., 2020] MPNetBase
[Song et al., 2020] 85.2 Table 3 in [Dai et al., 2020] XLNetLarge [Yang et
al., 2019] 85.9 Table 3 in [Dai et al., 2020] ROBERTALarge [Liu et al., 2019]
86.6 Table 3 in [Dai et al., 2020] ELECTRALarge [Clark et al., 2020] 88 Table
3 in [Dai et al., 2020] Funnel Transformer (B10-10-10H1024) [Dai et al., 2020]
89.5 Table 3 in [Dai et al., 2020]

Table 15: Results of empirical studies on GLUE STS-B (Semantic Textual
Similarity Benchmark). Bigger (dev) Spearman correlation P (SC-P) / Spearman
correlation S (SC-S) is better.

Model SC-P $\uparrow$ SC-S Sources DyConv [Wu et al., 2019] 60.7 63.1 Table 5
in [Tay et al., 2020a] BERT [Devlin et al., 2018] 85.8 - Table 16 in [Zaheer
et al., 2020] BIGBIRD [Zaheer et al., 2020] 87.8 - Table 16 in [Zaheer et al.,
2020] T5-Base [Raffel et al., 2020] 89.1 88.9 Table 5 in [Tay et al., 2020a]
Syn (R+V) [Tay et al., 2020a] 89.3 88.9 Table 5 in [Tay et al., 2020a] XLNet
[Yang et al., 2019] 89.5 - Table 16 in [Zaheer et al., 2020] MPNetBase [Song
et al., 2020] 90.9 - Table 3 in [Dai et al., 2020] ROBERTABase / RoBERTa [Liu
et al., 2019] 91.2 - Table 3 in [Dai et al., 2020] / Table 16 in [Zaheer et
al., 2020] Transformer (L24H1024) [Vaswani et al., 2017] 91.5 - Table 2 in
[Dai et al., 2020] Funnel Transformer (B10-10-10H1024) [Dai et al., 2020] 92.1
- Table 3 in [Dai et al., 2020] ROBERTALarge [Liu et al., 2019] 92.4 - Table 3
in [Dai et al., 2020] XLNetLarge [Yang et al., 2019] 92.5 - Table 3 in [Dai et
al., 2020] ELECTRALarge [Clark et al., 2020] 92.6 - Table 3 in [Dai et al.,
2020]

Table 16: Results of empirical studies on GLUE MRPC (Microsoft Research Paraphrase Corpus). Bigger (dev) accuracy/F1 is better. The results are sorted by F1 score. Model | Acc |  F1 $\uparrow$ |  Sources
---|---|---|---
DyConv[Wu et al., 2019] | 72.5 | 75.1 |  Table 5 in [Tay et al., 2020a]
Syn (D+V) [Tay et al., 2020a] | 87.7 | 84.8 |  Table 5 in [Tay et al., 2020a]
T5-Base [Raffel et al., 2020] | 88.7 | 85 |  Table 5 in [Tay et al., 2020a]
Nystromformer [Xiong et al., 2021] | - | 88.1 |  Table 2 in [Xiong et al., 2021]
XLNet [Yang et al., 2019] | - | 88.2 |  Table 16 in [Zaheer et al., 2020]
BERT-base [Devlin et al., 2018] | - | 88.4 |  Table 2 in [Xiong et al., 2021]
BERT [Devlin et al., 2018] | - | 88.9 |  Table 16 in [Zaheer et al., 2020]
Transformer (L24H1024) [Vaswani et al., 2017] | - | 90 |  Table 2 in [Dai et al., 2020]
ROBERTABase / RoBERTa-base [Liu et al., 2019] | - | 90.2 |  Table 3 in [Dai et al., 2020] / Table 16 in [Zaheer et al., 2020]
XLNetLarge [Yang et al., 2019] | - | 90.8 |  Table 3 in [Dai et al., 2020]
ELECTRALarge [Clark et al., 2020] | - | 90.8 |  Table 3 in [Dai et al., 2020]
ROBERTALarge [Liu et al., 2019] | - | 90.9 |  Table 3 in [Dai et al., 2020]
Funnel Transformer (B10-10-10H1024) [Dai et al., 2020] | - | 90.9 |  Table 3 in [Dai et al., 2020]
MPNetBase [Song et al., 2020] | - | 91.5 |  Table 3 in [Dai et al., 2020]
BIGBIRD [Zaheer et al., 2020] | - | 91.5 |  Table 16 in [Zaheer et al., 2020]
Table 17: Results of empirical studies on TriviaQA, a popular question answering dataset. Bigger accuracy/F1 is better. Results are reported on three evaluation sets: Dev, Test, and Test Verified. Model |  Acc $\uparrow$ |  F1 $\uparrow$ |  Sources
---|---|---|---
On Dev: |  |  |
T5-Base [Raffel et al., 2020] | 24.5 | - |  Table 5 in [Fedus et al., 2022]
T5-Large [Raffel et al., 2020] | 29.5 | - |  Table 5 in [Fedus et al., 2022]
Switch-Base [Fedus et al., 2022] | 30.7 | - |  Table 5 in [Fedus et al., 2022]
Switch-Large [Fedus et al., 2022] | 36.9 | - |  Table 5 in [Fedus et al., 2022]
Switch-C [Fedus et al., 2022] | 47.5 | - |  Table 5 in [Du et al., 2022]
GPT-3 Zero-shot [Brown et al., 2020] | 64.3 | - |  Table 11 in [Du et al., 2022]
GPT-3 One-shot [Brown et al., 2020] | 68 | - |  Table 11 in [Du et al., 2022]
GPT-3 64-shot [Brown et al., 2020] | 71.2 | - |  Table 11 in [Du et al., 2022]
GLaM Zero-shot [Du et al., 2022] | 71.3 | - |  Table 11 in [Du et al., 2022]
GLaM One-shot [Du et al., 2022] | 75.8 | - |  Table 11 in [Du et al., 2022]
RoBERTa [Liu et al., 2019] | - | 74.3 |  Table 2 in [Zaheer et al., 2020]
Longformer [Beltagy et al., 2020] | - | 75.2 |  Table 2 in [Zaheer et al., 2020]
BIGBIRD-ETC [Zaheer et al., 2020] | - | 78.7 |  Table 2 in [Zaheer et al., 2020]
BIGBIRD-ITC [Zaheer et al., 2020] | - | 79.5 |  Table 2 in [Zaheer et al., 2020]
On Test: |  |  |
KG-FiD (large) [Yu et al., 2021] | 69.8 | - |  Table 5 in [Du et al., 2022]
GPT-3 64-shot [Brown et al., 2020] | 71.2 | - |  Table 5 in [Du et al., 2022]
GLaM One-shot [Du et al., 2022] | 75 | - |  Table 5 in [Du et al., 2022]
RoBERTa-base [Liu et al., 2019] | - | 74.3 |  Table 7 in [Beltagy et al., 2020]
Longformer-base [Beltagy et al., 2020] | - | 75.2 |  Table 7 in [Beltagy et al., 2020]
Longformer-large [Beltagy et al., 2020] | - | 77.3 |  Table 8 in [Beltagy et al., 2020], Table 3 in [Zaheer et al., 2020]
SpanBERT [Joshi et al., 2020] | - | 79.1 |  Table 3 in [Zaheer et al., 2020]
Fusion-in-Decoder [Izacard and Grave, 2020] | - | 84.4 |  Table 3 in [Zaheer et al., 2020]
BIGBIRD-ETC [Zaheer et al., 2020] | - | 84.5 |  Table 3 in [Zaheer et al., 2020]
On Test Verified: |  |  |
Longformer [Beltagy et al., 2020] | - | 85.3 |  Table 3 in [Zaheer et al., 2020]
SpanBERT [Joshi et al., 2020] | - | 86.6 |  Table 3 in [Zaheer et al., 2020]
Fusion-in-Decoder [Izacard and Grave, 2020] | - | 90.3 |  Table 3 in [Zaheer et al., 2020]
BIGBIRD-ETC [Zaheer et al., 2020] | - | 92.4 |  Table 3 in [Zaheer et al., 2020]
Table 18: Results of empirical studies on NQ (long answer), a popular question answering dataset. Bigger F1 score is better. Results are reported on two evaluation sets: Dev and Test. Model | F1 $\uparrow$ |  Sources
---|---|---
On Dev: |  |
DrQA [Chen et al., 2017] | 46.1 |  Table 2 in [Wang et al., 2021]
DocumentQA [Clark and Gardner, 2017] | 46.1 |  Table 2 in [Zhang et al., 2021]
DecAtt [Parikh et al., 2016] \+ DocReader [Chen et al., 2017] | 54.8 |  Table 2 in [Zhang et al., 2021], Table 2 in [Wang et al., 2021]
BERT-base [Devlin et al., 2018] | 63.4 |  Table 2 in [Ainslie et al., 2020]
BERT-large / BERT-large / BERT-joint [Alberti et al., 2019] | 64.7 |  Table 2 in [Ainslie et al., 2020] / Table 2 in [Zhang et al., 2021] / Table 2 in [Wang et al., 2021]
BERT-wwm+SQuAD2 [Pan et al., 2019] | 68.2 |  Table 2 in [Wang et al., 2021]
ETC-base [Ainslie et al., 2020] | 72.5 |  Table 2 in [Ainslie et al., 2020]
Sparse Transformer / Sparse Attention [Jaszczur et al., 2021] | 74.5 |  Table 2 in [Zhang et al., 2021] / Table 2 in [Wang et al., 2021]
Sliding Window | 75.3 |  Table 2 in [Wang et al., 2021]
RikiNet-RoBERTa / RikiNet / RikiNet [Liu et al., 2020] | 75.3 |  Table 2 in [Wang et al., 2021] / Table 2 in [Ainslie et al., 2020] / Table 2 in [Zhang et al., 2021]
Reformer / Locality-Sensitive Hashing [Kitaev et al., 2020] | 75.5 |  Table 2 in [Zhang et al., 2021] / Table 2 in [Wang et al., 2021]
ReflectionNet [Wang et al., 2020b] | 75.9 |  Table 2 in [Zhang et al., 2021]
RikiNet-ensemble [Liu et al., 2020] | 75.9 |  Table 2 in [Zhang et al., 2021]
Cluster-Former [Wang et al., 2021] | 76.5 |  Table 2 in [Zhang et al., 2021], Table 2 in [Wang et al., 2021]
ReflectionNet-ensemble [Wang et al., 2020b] | 77.0 |  Table 2 in [Zhang et al., 2021]
Poolingformer [Zhang et al., 2021] | 77.5 |  Table 2 in [Zhang et al., 2021]
ETC-large (lifting from RoBERTa) [Ainslie et al., 2020] | 78.2 |  Table 2 in [Ainslie et al., 2020]
On Test: |  |
DocumentQA [Clark and Gardner, 2017] | 45.7 |  Table 2 in [Zhang et al., 2021]
DecAtt [Parikh et al., 2016] \+ DocReader [Chen et al., 2017] | 55 |  Table 2 in [Zhang et al., 2021]
BERT-joint [Alberti et al., 2019] | 66.2 |  Table 2 in [Zhang et al., 2021]
RikiNet-v2 / RikiNet-ensemble [Liu et al., 2020] | 76.1 |  Table 3 in [Zaheer et al., 2020] / Table 2 in [Zhang et al., 2021]
ReflectionNet [Liu et al., 2020] | 77.1 |  Table 3 in [Zaheer et al., 2020]
ReflectionNet-ensemble [Liu et al., 2020] | 77.2 |  Table 2 in [Zhang et al., 2021]
ETC (official) [Ainslie et al., 2020] | 77.78 |  Table 5 in [Ainslie et al., 2020]
BIGBIRD-ETC [Zaheer et al., 2020] | 77.8 |  Table 3 in [Zaheer et al., 2020], Table 2 in [Zhang et al., 2021]
Cluster-Former-ensemble / Cluster-Former [Wang et al., 2021] | 78 |  Table 3 in [Wang et al., 2021] / Table 2 in [Zhang et al., 2021]
Poolingformer-esamble [Zhang et al., 2021] | 79.8 |  Table 2 in [Zhang et al., 2021]
Table 19: Results of empirical studies on NQ (short answer), a popular question answering dataset. Bigger F1 score is better. Results are reported on two evaluation sets: Dev and Test. Model | F1 $\uparrow$ |  Sources
---|---|---
On Dev: |  |
DecAtt [Parikh et al., 2016] \+ DocReader [Chen et al., 2017] | 31.4 |  Table 2 in [Wang et al., 2021], Table 2 in [Zhang et al., 2021]
DocumentQA [Clark and Gardner, 2017] | 35.7 |  Table 2 in [Zhang et al., 2021]
DrQA [Chen et al., 2017] | 35.7 |  Table 2 in [Wang et al., 2021]
BERT-base [Devlin et al., 2018] | 47.5 |  Table 2 in [Ainslie et al., 2020]
ETC-base [Ainslie et al., 2020] | 52.2 |  Table 2 in [Ainslie et al., 2020]
BERT-large / BERT-large / BERT-joint [Alberti et al., 2019] | 52.7 |  Table 2 in [Ainslie et al., 2020] / Table 2 in [Wang et al., 2021] / Table 2 in [Zhang et al., 2021]
Sparse Transformer / Sparse Attention [Child et al., 2019] | 56.1 |  Table 2 in [Wang et al., 2021] / Table 2 in [Zhang et al., 2021]
Reformer / Locality-Sensitive Hashing [Kitaev et al., 2020] | 56.4 |  Table 2 in [Zhang et al., 2021] / Table 2 in [Wang et al., 2021]
Sliding Window | 56.4 |  Table 2 in [Wang et al., 2021]
Cluster-Former (#C=512) [Wang et al., 2021] | 57.1 |  Table 2 in [Wang et al., 2021], Table 2 in [Zhang et al., 2021]
BERT-wwm+SQuAD2 [Pan et al., 2019] | 57.2 |  Table 2 in [Wang et al., 2021]
ETC-large (lifting from RoBERTa) [Ainslie et al., 2020] | 58.5 |  Table 2 in [Ainslie et al., 2020]
Poolingformer [Zhang et al., 2021] | 58.6 |  Table 2 in [Zhang et al., 2021]
RikiNet / RikiNet / RikiNet-RoBERTa [Liu et al., 2020] | 59.3 |  Table 2 in [Ainslie et al., 2020] / Table 2 in [Wang et al., 2021] / Table 2 in [Zhang et al., 2021]
RikiNet-ensemble [Liu et al., 2020] | 61.1 |  Table 2 in [Zhang et al., 2021]
ReflectionNet [Wang et al., 2020b] | 61.3 |  Table 2 in [Zhang et al., 2021]
ReflectionNet-ensemble [Wang et al., 2020b] | 63.4 |  Table 2 in [Zhang et al., 2021]
On Test: |  |
DecAtt [Parikh et al., 2016] \+ DocReader [Chen et al., 2017] | 31.5 |  Table 2 in [Zhang et al., 2021]
DocumentQA [Clark and Gardner, 2017] | 35.1 |  Table 2 in [Zhang et al., 2021]
BERT-joint [Alberti et al., 2019] | 52.1 |  Table 2 in [Zhang et al., 2021]
ETC (official) [Ainslie et al., 2020] | 57.86 |  Table 5 in [Ainslie et al., 2020]
BIGBIRD-ETC [Zaheer et al., 2020] | 57.9 |  Table 3 in [Zaheer et al., 2020], Table 2 in [Zhang et al., 2021]
Cluster-Former / Cluster-Former-ensemble [Wang et al., 2021] | 60.9 |  Table 3 in [Wang et al., 2021] / Table 2 in [Zhang et al., 2021]
RikiNet-v2 / RikiNet-ensemble [Liu et al., 2020] | 61.3 |  Table 3 in [Zaheer et al., 2020] / Table 2 in [Zhang et al., 2021]
Poolingformer-esamble [Zhang et al., 2021] | 61.6 |  Table 2 in [Zhang et al., 2021]
ReflectionNet / ReflectionNet-ensemble [Wang et al., 2020b] | 64.1 |  Table 3 in [Zaheer et al., 2020] / Table 2 in [Zhang et al., 2021]
Table 20: Results of empirical studies on LRA-ListOps. Bigger accuracy is
better.

Model Acc $\uparrow$ Sources Local Attention / Local Attention / Local Attn.
[Tay et al., 2020c] 15.82 Table 1 in [Ma et al., 2021] / Table 10 in [Gu et
al., 2022b] / Table 1 in [Hasani et al., 2022] Linear Trans. [Katharopoulos et
al., 2020] 16.13 Table 1 in [Ma et al., 2021], Table 10 in [Gu et al., 2022b],
Table 1 in [Hasani et al., 2022] Linformer [Wang et al., 2020a] 16.13 Table 1
in [Hasani et al., 2022] Sparse Trans. / Sparse Trans. / Sparse Transformer
[Child et al., 2019] 17.07 Table 1 in [Ma et al., 2021] / Table 10 in [Gu et
al., 2022b] / Table 1 in [Hasani et al., 2022] Performer in [Choromanski et
al., 2020] 18.01 Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023],
Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al., 2022], Table 3 in
[Xiong et al., 2021] Reformer [Kitaev et al., 2020] 19.05 Table 3 in [Xiong et
al., 2021] Performer in [Choromanski et al., 2020] 32.78 Table 1 in [Zhu et
al., 2021] Sinkhorn Trans. [Tay et al., 2020b] 33.67 Table 1 in [Ma et al.,
2021], Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al., 2022] FNet
[Lee-Thorp et al., 2021] 35.33 Table 10 in [Gu et al., 2022b], Table 1 in
[Hasani et al., 2022] Longformer [Beltagy et al., 2020] 35.63 Table 1 in [Ma
et al., 2021], Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al.,
2022] Linformer [Wang et al., 2020a] 35.7 Table 1 in [Ma et al., 2021], Table
2 in [Ma et al., 2023], Table 10 in [Gu et al., 2022b] BigBird [Zaheer et al.,
2020] 36.05 Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023], Table
10 in [Gu et al., 2022b], Table 1 in [Hasani et al., 2022] XFM / Transformer /
Transformer / Transformer / Transformer [Vaswani et al., 2017] 36.37 Table 2
in [Ma et al., 2023] / Table 1 in [Hasani et al., 2022] / Table 1 in [Ma et
al., 2021] / Table 10 in [Gu et al., 2022b] / Table 2 in [Smith et al., 2022]
Reformer [Kitaev et al., 2020] 36.44 Table 1 in [Zhu et al., 2021] Synthesizer
[Tay et al., 2021a] 36.99 Table 1 in [Ma et al., 2021], Table 10 in [Gu et
al., 2022b] Standard [Vaswani et al., 2017] 37.1 Table 3 in [Xiong et al.,
2021] Transformer (re-impl) / XFM (re-impl) [Vaswani et al., 2017] 37.11 Table
1 in [Ma et al., 2021] / Table 2 in [Ma et al., 2023] Full Attention [Vaswani
et al., 2017] 37.13 Table 1 in [Zhu et al., 2021] Nystromformer [Xiong et al.,
2021] 37.15 Table 3 in [Xiong et al., 2021], Table 1 in [Hasani et al., 2022],
Table 10 in [Gu et al., 2022b] Linformer [Wang et al., 2020a] 37.25 Table 3 in
[Xiong et al., 2021] Luna-256 [Ma et al., 2021] 37.25 Table 10 in [Gu et al.,
2022b], Table 2 in [Ma et al., 2023], Table 1 in [Hasani et al., 2022]
Reformer [Kitaev et al., 2020] 37.27 Table 1 in [Hasani et al., 2022], Table
10 in [Gu et al., 2022b], Table 2 in [Ma et al., 2023], Table 1 in [Ma et al.,
2021] Nystromformer [Xiong et al., 2021] 37.34 Table 1 in [Zhu et al., 2021]
Linformer [Wang et al., 2020a] 37.38 Table 1 in [Zhu et al., 2021] Luna-256
[Ma et al., 2021] 37.98 Table 1 in [Ma et al., 2021], Table 2 in [Ma et al.,
2023] Luna-128 [Ma et al., 2021] 38.01 Table 1 in [Ma et al., 2021]
Transformer-LS [Zhu et al., 2021] 38.36 Table 1 in [Zhu et al., 2021] CCNN
[Romero et al., 2022] 43.6 Table 2 in [Smith et al., 2022] CDIL [Cheng et al.,
2023] 44.05 Table 1 in [Hasani et al., 2022] H-Trans.-1D / H-Transformer-1D
[Zhu and Soricut, 2021] 49.53 Table 2 in [Smith et al., 2022] / Table 1 in
[Hasani et al., 2022] DSS [Gupta et al., 2022] 57.6 Table 1 in [Hasani et al.,
2022] S4-v2 (re-impl) [Gu et al., 2022b] 59.1 Table 2 in [Ma et al., 2023]
S4-v2 / S4 (updated) [Gu et al., 2022b] 59.6 Table 2 in [Ma et al., 2023] /
Table 10 in [Gu et al., 2022b] S4D-LegS [Gu et al., 2022a] 60.47 Table 2 in
[Smith et al., 2022], Table 1 in [Hasani et al., 2022] S4D-Lin [Gu et al.,
2022a] 60.52 Table 1 in [Hasani et al., 2022] S5 [Smith et al., 2022] 62.15
Table 2 in [Smith et al., 2022], Table 1 in [Hasani et al., 2022] Liquid-S4 /
Liquid-S4-PB [Hasani et al., 2022] 62.75 Table 2 in [Smith et al., 2022] /
Table 1 in [Hasani et al., 2022] MEGA [Ma et al., 2023] 63.14 Table 2 in [Ma
et al., 2023], Table 2 in [Smith et al., 2022]

Table 21: Results of empirical studies on LRA-Text. Bigger accuracy is better.

Model Acc $\uparrow$ Sources Local Attention / Local Attention / Local Attn.
[Tay et al., 2020c] 52.98 Table 1 in [Ma et al., 2021] / Table 10 in [Gu et
al., 2022b] / Table 1 in [Hasani et al., 2022] Linformer [Wang et al., 2020a]
53.94 Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023], Table 10 in
[Gu et al., 2022b] Linformer [Wang et al., 2020a] 55.91 Table 3 in [Xiong et
al., 2021] Reformer [Kitaev et al., 2020] 56.1 Table 1 in [Hasani et al.,
2022], Table 10 in [Gu et al., 2022b], Table 2 in [Ma et al., 2023], Table 1
in [Ma et al., 2021] Linformer [Wang et al., 2020a] 56.12 Table 1 in [Zhu et
al., 2021] Sinkhorn Trans. [Tay et al., 2020b] 61.2 Table 1 in [Ma et al.,
2021], Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al., 2022]
Synthesizer [Tay et al., 2021a] 61.68 Table 1 in [Ma et al., 2021], Table 10
in [Gu et al., 2022b] Longformer [Beltagy et al., 2020] 62.85 Table 1 in [Ma
et al., 2021], Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al.,
2022] Sparse Trans. / Sparse Trans. / Sparse Transformer [Child et al., 2019]
63.58 Table 1 in [Ma et al., 2021] / Table 10 in [Gu et al., 2022b] / Table 1
in [Hasani et al., 2022] Performer in [Choromanski et al., 2020] 63.81 Table 3
in [Xiong et al., 2021] BigBird [Zaheer et al., 2020] 64.02 Table 1 in [Ma et
al., 2021], Table 2 in [Ma et al., 2023], Table 10 in [Gu et al., 2022b],
Table 1 in [Hasani et al., 2022] XFM / Transformer / Transformer / Transformer
/ Transformer [Vaswani et al., 2017] 64.27 Table 2 in [Ma et al., 2023] /
Table 1 in [Hasani et al., 2022] / Table 1 in [Ma et al., 2021] / Table 10 in
[Gu et al., 2022b] / Table 2 in [Smith et al., 2022] Luna-256 [Ma et al.,
2021] 64.57 Table 10 in [Gu et al., 2022b], Table 2 in [Smith et al., 2022],
Table 1 in [Hasani et al., 2022] Reformer [Kitaev et al., 2020] 64.88 Table 1
in [Zhu et al., 2021], Table 3 in [Xiong et al., 2021] Standard [Vaswani et
al., 2017] 65.02 Table 3 in [Xiong et al., 2021] FNet [Lee-Thorp et al., 2021]
65.11 Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al., 2022]
Performer in [Choromanski et al., 2020] 65.21 Table 1 in [Zhu et al., 2021]
Transformer (re-impl) / XFM (re-impl) [Vaswani et al., 2017] 65.21 Table 1 in
[Ma et al., 2021] / Table 2 in [Ma et al., 2023] Full Attention [Vaswani et
al., 2017] 65.35 Table 1 in [Zhu et al., 2021] Performer in [Choromanski et
al., 2020] 65.4 Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023],
Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al., 2022] Nystromformer
[Xiong et al., 2021] 65.52 Table 3 in [Xiong et al., 2021], Table 1 in [Hasani
et al., 2022], Table 10 in [Gu et al., 2022b] Nystromformer [Xiong et al.,
2021] 65.75 Table 1 in [Zhu et al., 2021] Luna-256 [Ma et al., 2021] 65.78
Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023] Linear Trans.
[Katharopoulos et al., 2020] 65.9 Table 1 in [Ma et al., 2021], Table 10 in
[Gu et al., 2022b], Table 1 in [Hasani et al., 2022] Linformer [Wang et al.,
2020a] 65.9 Table 1 in [Hasani et al., 2022] Transformer-LS [Zhu et al., 2021]
68.4 Table 1 in [Zhu et al., 2021] DSS [Gupta et al., 2022] 76.6 Table 1 in
[Hasani et al., 2022] H-Trans.-1D / H-Transformer-1D [Zhu and Soricut, 2021]
78.69 Table 2 in [Smith et al., 2022] / Table 1 in [Hasani et al., 2022] CCNN
[Romero et al., 2022] 84.08 Table 2 in [Smith et al., 2022] S4-v2 (re-impl)
[Gu et al., 2022b] 86.53 Table 2 in [Ma et al., 2023] CDIL [Cheng et al.,
2023] 86.78 Table 1 in [Hasani et al., 2022] S4-v2 / S4 (updated) [Gu et al.,
2022b] 86.82 Table 2 in [Ma et al., 2023] / Table 10 in [Gu et al., 2022b]
S4-LegS [Gu et al., 2022a] 86.82 Table 2 in [Smith et al., 2022], Table 1 in
[Hasani et al., 2022] S4D-Inv [Gu et al., 2022a] 87.34 Table 1 in [Hasani et
al., 2022] Liquid-S4 / Liquid-S4-PB 89.02 Table 2 in [Smith et al., 2022] /
Table 1 in [Hasani et al., 2022] S5 [Smith et al., 2022] 89.31 Table 2 in
[Smith et al., 2022], Table 1 in [Hasani et al., 2022] MEGA [Ma et al., 2023]
90.43 Table 2 in [Ma et al., 2023], Table 2 in [Smith et al., 2022]

Table 22: Results of empirical studies on LRA-Retrieval. Bigger accuracy is
better.

Model Acc $\uparrow$ Sources Linformer [Wang et al., 2020a] 52.27 Table 1 in
[Ma et al., 2021], Table 2 in [Ma et al., 2023], Table 10 in [Gu et al.,
2022b] Linear Trans. [Katharopoulos et al., 2020] 53.09 Table 1 in [Ma et al.,
2021], Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al., 2022]
Linformer [Wang et al., 2020a] 53.09 Table 1 in [Hasani et al., 2022] Local
Attention / Local Attention / Local Attn. [Tay et al., 2020c] 53.39 Table 1 in
[Ma et al., 2021] / Table 10 in [Gu et al., 2022b] / Table 1 in [Hasani et
al., 2022] Reformer [Kitaev et al., 2020] 53.4 Table 1 in [Hasani et al.,
2022], Table 10 in [Gu et al., 2022b], Table 2 in [Ma et al., 2023], Table 1
in [Ma et al., 2021] Performer in [Choromanski et al., 2020] 53.82 Table 1 in
[Ma et al., 2021], Table 2 in [Ma et al., 2023], Table 10 in [Gu et al.,
2022b], Table 1 in [Hasani et al., 2022] Sinkhorn Trans. [Tay et al., 2020b]
53.83 Table 1 in [Ma et al., 2021], Table 10 in [Gu et al., 2022b], Table 1 in
[Hasani et al., 2022] Synthesizer [Tay et al., 2021a] 54.67 Table 1 in [Ma et
al., 2021], Table 10 in [Gu et al., 2022b] Longformer [Beltagy et al., 2020]
56.89 Table 1 in [Ma et al., 2021], Table 10 in [Gu et al., 2022b], Table 1 in
[Hasani et al., 2022] XFM / Transformer / Transformer / Transformer /
Transformer [Vaswani et al., 2017] 57.46 Table 2 in [Ma et al., 2023] / Table
1 in [Hasani et al., 2022] / Table 1 in [Ma et al., 2021] / Table 10 in [Gu et
al., 2022b] / Table 2 in [Smith et al., 2022] BigBird [Zaheer et al., 2020]
59.29 Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023], Table 10 in
[Gu et al., 2022b], Table 1 in [Hasani et al., 2022] Sparse Trans. / Sparse
Trans. / Sparse Transformer [Child et al., 2019] 59.59 Table 1 in [Ma et al.,
2021] / Table 10 in [Gu et al., 2022b] / Table 1 in [Hasani et al., 2022] FNet
[Lee-Thorp et al., 2021] 59.61 Table 10 in [Gu et al., 2022b], Table 1 in
[Hasani et al., 2022] H-Trans.-1D / H-Transformer-1D [Zhu and Soricut, 2021]
63.99 Table 2 in [Smith et al., 2022] / Table 1 in [Hasani et al., 2022]
Performer in [Choromanski et al., 2020] 78.62 Table 3 in [Xiong et al., 2021]
Reformer [Kitaev et al., 2020] 78.64 Table 1 in [Zhu et al., 2021], Table 3 in
[Xiong et al., 2021] Transformer (re-impl) / XFM (re-impl) [Vaswani et al.,
2017] 79.14 Table 1 in [Ma et al., 2021] / Table 2 in [Ma et al., 2023]
Luna-256 [Ma et al., 2021] 79.29 Table 10 in [Gu et al., 2022b], Table 2 in
[Smith et al., 2022], Table 1 in [Hasani et al., 2022] Standard [Vaswani et
al., 2017] 79.35 Table 3 in [Xiong et al., 2021] Linformer [Wang et al.,
2020a] 79.37 Table 3 in [Xiong et al., 2021], Table 1 in [Zhu et al., 2021]
Luna-256 [Ma et al., 2021] 79.56 Table 1 in [Ma et al., 2021], Table 2 in [Ma
et al., 2023] Nystromformer [Xiong et al., 2021] 79.56 Table 3 in [Xiong et
al., 2021], Table 1 in [Hasani et al., 2022], Table 10 in [Gu et al., 2022b]
Nystromformer [Xiong et al., 2021] 81.29 Table 1 in [Zhu et al., 2021]
Performer in [Choromanski et al., 2020] 81.7 Table 1 in [Zhu et al., 2021]
Transformer-LS [Zhu et al., 2021] 81.95 Table 1 in [Zhu et al., 2021] Full
Attention [Vaswani et al., 2017] 82.3 Table 1 in [Zhu et al., 2021] CDIL
[Cheng et al., 2023] 85.36 Table 1 in [Hasani et al., 2022] DSS [Gupta et al.,
2022] 87.6 Table 1 in [Hasani et al., 2022] S4-v2 / S4 (updated) [Gu et al.,
2022b] 90.9 Table 2 in [Ma et al., 2023] / Table 10 in [Gu et al., 2022b]
S4-LegS [Gu et al., 2022a] 90.9 Table 2 in [Smith et al., 2022], Table 1 in
[Hasani et al., 2022] S4-v2 (re-impl) [Gu et al., 2022b] 90.94 Table 2 in [Ma
et al., 2023] S4D-Inv [Gu et al., 2022a] 91.09 Table 1 in [Hasani et al.,
2022] Liquid-S4 / Liquid-S4-PB 91.2 Table 2 in [Smith et al., 2022] / Table 1
in [Hasani et al., 2022] MEGA [Ma et al., 2023] 91.25 Table 2 in [Ma et al.,
2023], Table 2 in [Smith et al., 2022] S5 [Smith et al., 2022] 91.4 Table 2 in
[Smith et al., 2022], Table 1 in [Hasani et al., 2022]

Table 23: Results of empirical studies on LRA-Image. Bigger accuracy is
better.

Model Acc $\uparrow$ Sources Performer in [Choromanski et al., 2020] 37.07
Table 3 in [Xiong et al., 2021] Linformer [Wang et al., 2020a] 37.84 Table 3
in [Xiong et al., 2021] Reformer [Kitaev et al., 2020] 38.07 Table 1 in
[Hasani et al., 2022], Table 10 in [Gu et al., 2022b], Table 2 in [Ma et al.,
2023], Table 1 in [Ma et al., 2021] Standard [Vaswani et al., 2017] 38.2 Table
3 in [Xiong et al., 2021] Linformer [Wang et al., 2020a] 38.56 Table 1 in [Ma
et al., 2021], Table 2 in [Ma et al., 2023], Table 10 in [Gu et al., 2022b]
FNet [Lee-Thorp et al., 2021] 38.67 Table 10 in [Gu et al., 2022b], Table 1 in
[Hasani et al., 2022] BigBird [Zaheer et al., 2020] 40.83 Table 1 in [Ma et
al., 2021], Table 2 in [Ma et al., 2023], Table 10 in [Gu et al., 2022b],
Table 1 in [Hasani et al., 2022] Sinkhorn Trans. [Tay et al., 2020b] 41.23
Table 1 in [Ma et al., 2021], Table 10 in [Gu et al., 2022b], Table 1 in
[Hasani et al., 2022] Local Attention / Local Attention / Local Attn. [Tay et
al., 2020c] 41.46 Table 1 in [Ma et al., 2021] / Table 10 in [Gu et al.,
2022b] / Table 1 in [Hasani et al., 2022] Nystromformer [Xiong et al., 2021]
41.58 Table 3 in [Xiong et al., 2021], Table 1 in [Hasani et al., 2022], Table
10 in [Gu et al., 2022b] Synthesizer [Tay et al., 2021a] 41.61 Table 1 in [Ma
et al., 2021], Table 10 in [Gu et al., 2022b] Longformer [Beltagy et al.,
2020] 42.22 Table 1 in [Ma et al., 2021], Table 10 in [Gu et al., 2022b],
Table 1 in [Hasani et al., 2022] Linear Trans. [Katharopoulos et al., 2020]
42.34 Table 1 in [Ma et al., 2021], Table 10 in [Gu et al., 2022b], Table 1 in
[Hasani et al., 2022] Linformer [Wang et al., 2020a] 42.34 Table 1 in [Hasani
et al., 2022] XFM / Transformer / Transformer / Transformer / Transformer
[Vaswani et al., 2017] 42.44 Table 2 in [Ma et al., 2023] / Table 1 in [Hasani
et al., 2022] / Table 1 in [Ma et al., 2021] / Table 10 in [Gu et al., 2022b]
/ Table 2 in [Smith et al., 2022] Performer in [Choromanski et al., 2020]
42.77 Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023], Table 10 in
[Gu et al., 2022b], Table 1 in [Hasani et al., 2022] Transformer (re-impl) /
XFM (re-impl) [Vaswani et al., 2017] 42.94 Table 1 in [Ma et al., 2021] /
Table 2 in [Ma et al., 2023] Reformer [Kitaev et al., 2020] 43.29 Table 3 in
[Xiong et al., 2021] Sparse Trans. / Sparse Trans. / Sparse Transformer [Child
et al., 2019] 44.24 Table 1 in [Ma et al., 2021] / Table 10 in [Gu et al.,
2022b] / Table 1 in [Hasani et al., 2022] H-Trans.-1D / H-Transformer-1D [Zhu
and Soricut, 2021] 46.05 Table 2 in [Smith et al., 2022] / Table 1 in [Hasani
et al., 2022] Luna-256 [Ma et al., 2021] 47.38 Table 10 in [Gu et al., 2022b],
Table 2 in [Smith et al., 2022], Table 1 in [Hasani et al., 2022] Luna-256 [Ma
et al., 2021] 47.86 Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023]
CDIL [Cheng et al., 2023] 66.91 Table 1 in [Hasani et al., 2022] DSS [Gupta et
al., 2022] 85.8 Table 1 in [Hasani et al., 2022] S5 [Smith et al., 2022] 88
Table 2 in [Smith et al., 2022], Table 1 in [Hasani et al., 2022] S4-v2 (re-
impl) [Gu et al., 2022b] 88.48 Table 2 in [Ma et al., 2023] S4-v2 / S4
(updated) [Gu et al., 2022b] 88.65 Table 2 in [Ma et al., 2023], Table 10 in
[Gu et al., 2022b] S4-LegS [Gu et al., 2022a] 88.65 Table 2 in [Smith et al.,
2022], Table 1 in [Hasani et al., 2022] CCNN [Romero et al., 2022] 88.9 Table
2 in [Smith et al., 2022] S4-FouT [Gu et al., 2022a] 89.07 Table 1 in [Hasani
et al., 2022] Liquid-S4 / Liquid-S4-PB 89.5 Table 2 in [Smith et al., 2022] /
Table 1 in [Hasani et al., 2022] MEGA [Ma et al., 2023] 90.44 Table 2 in [Ma
et al., 2023], Table 2 in [Smith et al., 2022]

Table 24: Results of empirical studies on LRA-Pathfinder. Bigger accuracy is
better.

Model Acc $\uparrow$ Sources Local Attention / Local Attention / Local Attn.
[Tay et al., 2020c] 66.63 Table 1 in [Ma et al., 2021] / Table 10 in [Gu et
al., 2022b] / Table 1 in [Hasani et al., 2022] Sinkhorn Trans. [Tay et al.,
2020b] 67.45 Table 1 in [Ma et al., 2021], Table 10 in [Gu et al., 2022b],
Table 1 in [Hasani et al., 2022] Linformer [Wang et al., 2020a] 67.6 Table 3
in [Xiong et al., 2021] Reformer [Kitaev et al., 2020] 68.5 Table 1 in [Hasani
et al., 2022], Table 10 in [Gu et al., 2022b], Table 2 in [Ma et al., 2023],
Table 1 in [Ma et al., 2021] H-Trans.-1D / H-Transformer-1D [Zhu and Soricut,
2021] 68.78 Table 2 in [Smith et al., 2022] / Table 1 in [Hasani et al., 2022]
Reformer [Kitaev et al., 2020] 69.36 Table 3 in [Xiong et al., 2021]
Synthesizer [Tay et al., 2021a] 69.45 Table 1 in [Ma et al., 2021], Table 10
in [Gu et al., 2022b] Longformer [Beltagy et al., 2020] 69.71 Table 1 in [Ma
et al., 2021], Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al.,
2022] Performer in [Choromanski et al., 2020] 69.87 Table 3 in [Xiong et al.,
2021] Nystromformer [Xiong et al., 2021] 70.94 Table 3 in [Xiong et al.,
2021], Table 1 in [Hasani et al., 2022], Table 10 in [Gu et al., 2022b] XFM /
Transformer / Transformer / Transformer / Transformer [Vaswani et al., 2017]
71.4 Table 2 in [Ma et al., 2023], Table 1 in [Hasani et al., 2022], Table 1
in [Ma et al., 2021], Table 10 in [Gu et al., 2022b], Table 2 in [Smith et
al., 2022] Sparse Trans. / Sparse Trans. / Sparse Transformer [Child et al.,
2019] 71.71 Table 1 in [Ma et al., 2021] / Table 10 in [Gu et al., 2022b] /
Table 1 in [Hasani et al., 2022] Transformer (re-impl) / XFM (re-impl)
[Vaswani et al., 2017] 71.83 Table 1 in [Ma et al., 2021] / Table 2 in [Ma et
al., 2023] Standard [Vaswani et al., 2017] 74.16 Table 3 in [Xiong et al.,
2021] BigBird [Zaheer et al., 2020] 74.87 Table 1 in [Ma et al., 2021], Table
2 in [Ma et al., 2023], Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et
al., 2022] Linear Trans. [Katharopoulos et al., 2020] 75.3 Table 1 in [Ma et
al., 2021], Table 10 in [Gu et al., 2022b], Table 1 in [Hasani et al., 2022]
Linformer [Wang et al., 2020a] 75.3 Table 1 in [Hasani et al., 2022] BigBird
[Zaheer et al., 2020] 75.87 Table 10 in [Gu et al., 2022b] Linformer [Wang et
al., 2020a] 76.34 Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023],
Table 10 in [Gu et al., 2022b] Performer in [Choromanski et al., 2020] 77.05
Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023], Table 10 in [Gu et
al., 2022b], Table 1 in [Hasani et al., 2022] Luna-256 [Ma et al., 2021] 77.72
Table 10 in [Gu et al., 2022b], Table 2 in [Smith et al., 2022], Table 1 in
[Hasani et al., 2022] FNet [Lee-Thorp et al., 2021] 77.8 Table 10 in [Gu et
al., 2022b], Table 1 in [Hasani et al., 2022] Luna-256 [Ma et al., 2021] 78.55
Table 1 in [Ma et al., 2021], Table 2 in [Ma et al., 2023] Luna-128 [Ma et
al., 2021] 78.89 Table 1 in [Ma et al., 2021] DSS [Gupta et al., 2022] 84.1
Table 1 in [Hasani et al., 2022] CCNN [Romero et al., 2022] 91.51 Table 2 in
[Smith et al., 2022] CDIL [Cheng et al., 2023] 91.7 Table 1 in [Hasani et al.,
2022] S4-v2 (re-impl) [Gu et al., 2022b] 94.01 Table 2 in [Ma et al., 2023]
S4-v2 / S4 (updated) [Gu et al., 2022b] 94.2 Table 2 in [Ma et al., 2023] /
Table 10 in [Gu et al., 2022b] S4-LegS [Gu et al., 2022a] 94.2 Table 2 in
[Smith et al., 2022], Table 1 in [Hasani et al., 2022] S4-FouT [Gu et al.,
2022a] 94.46 Table 1 in [Hasani et al., 2022] Liquid-S4 / Liquid-S4-PB 94.8
Table 2 in [Smith et al., 2022], Table 1 in [Hasani et al., 2022] S5 [Smith et
al., 2022] 95.33 Table 2 in [Smith et al., 2022], Table 1 in [Hasani et al.,
2022] MEGA [Ma et al., 2023] 96.01 Table 2 in [Ma et al., 2023], Table 2 in
[Smith et al., 2022]