# Advancing State of the Art in Language Modeling

David Herel
FEE, CTU
CIIRC CTU
<EMAIL_ADDRESS>
Tomas Mikolov
CIIRC CTU
<EMAIL_ADDRESS>

###### Abstract

Generalization is arguably the most important goal of statistical language
modeling research. Publicly available benchmarks and papers published with an
open-source code have been critical to advancing the field. However, it is
often very difficult, and sometimes even impossible, to reproduce the results
fully as reported in publications. In this paper, we propose a simple
framework that should help advance the state of the art in language modeling
in terms of generalization. We propose to publish not just the code, but also
probabilities on dev and test sets with future publications so that one can
easily add the new model into an ensemble. This has crucial advantages: it is
much easier to determine whether a newly proposed model is actually
complementary to the current baseline. Therefore, instead of inventing new
names for the old tricks, the scientific community can advance faster.
Finally, this approach promotes diversity of ideas: one does not need to
create an individual model that is the new state of the art to attract
attention; it will be sufficient to develop a new model that learns patterns
which other models do not. Thus, even a suboptimal model can be found to have
value. Remarkably, our approach has yielded new state-of-the-art results
across various language modeling benchmarks up to 10%.

Advancing State of the Art in Language Modeling

David Herel FEE, CTU CIIRC CTU<EMAIL_ADDRESS>Tomas Mikolov CIIRC CTU
<EMAIL_ADDRESS>

##  1 Introduction

Statistical language modeling has been around since the work of Claude Shannon
in 1950 (Shannon, 1951). For decades, it was believed that the state of the
art in language modeling are the n-gram techniques. While dozens of more
advanced approaches have been developed (including neural language models), it
was believed that those alternatives only work on small datasets, and with
increasing amount of data, the improvements over n-grams vanish (Goodman,
2001).

Figure 1: This visualization illustrates the concept of ensemble application
on a language modeling dataset. It enables easy identification of models that
are either obsolete or complementary to the current state-of-the-art. Notably,
the models do not necessarily have to be state-of-the-art themselves; they
simply need to complement the existing state-of-the-art models effectively.

This has changed and neural language models are the mainstream techniques now.
However, despite the almost exponential growth of interest in neural language
modeling in the last decade, it is still very difficult to advance the state
of the art. Many top language models today are simply too large and expensive
to be explored efficiently in academia, especially their training phase. While
there is a range of benchmarks that are more suitable for research purposes
due to the limited training data, it is still very hard to reproduce the
state-of-the-art results. Many projects are published with open-source code
that doesn’t work as advertised in the papers, or with rather complex
implementations that are very difficult to extend. Since the best published
results often use several tricks together, any new idea faces a significant
barrier which may prevent its publication: it may simply be too hard to show
its value and impress the reviewers.

In this paper, we propose to address this issue. We suggest that researchers
should publish not just their papers with code, but also with probabilities
per valid and test sets. It is then straightforward to see if a newly proposed
model is complementary to the current state of the art by simply adding it
into an ensemble using a linear combination. The weights of all models in the
ensemble would be optimized on the valid set.

Following this approach, we report new state-of-the-art results on the Penn
TreeBank, WikiText-2 and Wikitext-103 datasets. We find that some of the best
neural language modeling architectures are in fact complementary, mainly the
LSTMs and Transformers. Thus, the findings in this paper may convince the
research community that there is value in diversity of ideas: we do not need
to search for the best single model, but rather use a combination of
complementary models. At the same time, our approach can be elegantly used to
detect ideas that are simply reinventing a new name for already known tricks,
and thus save time for researchers who are mainly interested in advancing the
best possible results.

This work brings several key contributions:

  1. 1.

Simplified Ensemble Integration: Our framework has the potential to
significantly advance statistical language modeling, simplifying the
integration of new models into an ensemble and accelerating scientific
progress.

  2. 2.

New State-of-the-art: Our ensemble approach has yielded up to a 10%
improvement on various language modeling benchmarks, underscoring the value of
model diversity.

  3. 3.

Open-Sourced Models: We make all our reproduced results
available111https://github.com/DavidHerel/sota_lm, enabling straightforward
testing of new models to assess their compatibility with the state-of-the-art.
We shift the focus from producing identical state-of-the-art models to
introducing complementary models that enhance ensemble performance.

  4. 4.

Promoting Ensemble Approaches over Single Models: The scientific community has
traditionally prioritized developing single models, overlooking the true
potential of ensemble state-of-the-art. This was primarily due to the
difficulty in reproducing results, hindering the creation of ensembles.
However, our work has transformed this dynamic, enabling the community to
progress more rapidly and improve the true state-of-the-art, beyond just
individual model performance.

##  2 Related work

Statistical language modeling, a domain that can trace its origins back to the
seminal work of Claude Shannon in 1950, has seen a remarkable evolution over
the decades. Shannon’s work laid the foundation for the use of probabilistic
models in predicting the likelihood of a given sequence of words in a
language, setting the stage for the development of n-gram models (Shannon,
1951). For a significant period, these n-gram techniques, which predict the
next word in a sequence based on the previous ’n’ words, were the gold
standard in language modeling. They were used extensively in a variety of
applications, from speech recognition to machine translation.

The rise of neural language models marked a pivotal shift. These models, which
use neural networks to predict the next word in a sequence, challenged the
long-standing belief that advanced alternatives to n-gram models were only
effective on smaller datasets, and their performance would decline as the size
of the data increased (Goodman, 2001).

Today, the scenario has evolved, and neural language models have become the
standard techniques. This shift can be credited to several pivotal factors:

  * •

The availability of open-source code, with the first neural language modeling
toolkit published in 2010 Mikolov and Zweig (2012).

  * •

The presence of open benchmarks, including the Modified Penn TreeBank
benchmark from 2010 in the RNNLM project (Mikolov and Zweig, 2012), as well as
datasets like Wikitext-103 Mikolov and Zweig (2012).

  * •

Advances in algorithms and hyper-parameter tuning, such as effectively
handling large vocabularies Vaswani et al. (2017), gradient clipping Mikolov
et al. (2010), utilization of large models with dropouts Dai et al. (2019);
Gal and Ghahramani (2016).

  * •

Increased computational power.

However, the community subsequently shifted towards evaluating single models
on these datasets. This raises the question of why we should limit ourselves
to single models, and what precisely constitutes a single model. Even a large
language model with dropouts can be viewed as an ensemble of sorts. To remain
relevant, researchers must continually produce new state-of-the-art models, as
this is the only way to persuade others. We propose publishing valid and test
word probabilities along with code and paper, enabling easy integration into
ensembles to determine whether a model introduces innovations to the state-of-
the-art or merely repackages existing techniques.

Ensemble learning, a widely adopted technique for enhancing model performance,
involves harnessing the strengths of multiple weaker models Sagi and Rokach
(2018); Aniol et al. (2019). Typically, ensemble learning entails considering
model weights or combining diverse outputs, much like our approach of
combining word probabilities. Mix-of-Experts (MoE) represents a specific
ensemble approach that combines predictions from multiple specialized sub-
models to improve overall performance. MoE has found success in diverse
domains, including natural language processing and computer vision Jacobs et
al. (1991); Shazeer et al. (2017). However, we demonstrate that it’s possible
to achieve state-of-the-art results through a simple linear combination of
models.

##  3 Experiments

In today’s competitive landscape, a model must significantly surpass the
prevailing top-performing models to gain recognition. However, when we delve
into ensemble setups, the dynamics change. Here, a model may not shine
individually, but its unique functionality can bolster the overall ensemble’s
performance.

To determine which models enhance language modeling results, we’ve executed
rigorous tests on the three most distinguished language modeling datasets.
These datasets span from 1 million to 103 million words, as illustrated in
Figure 2. This broad spectrum allows us to illustrate how an increase in data
volume can shift model partitioning.

Subsequently, we re-trained and re-evaluated all the models evaluated on these
datasets and integrated them into a robust ensemble.

###  3.1 Datasets

Figure 2: This graph illustrates the number of training tokens present in each
dataset, specifically 1M, 2M, and 103M Mikolov and Zweig (2012); Merity et al.
(2016). It provides a visual representation of the increase in token numbers
across the datasets, enabling an analysis of how the application of different
language model architectures evolves in response to increasing data volume.

Experiments are conducted on widely accepted language modeling benchmarks.
Namely, Penn TreeBank Mikolov and Zweig (2012), Wikitext-2Merity et al. (2016)
and Wikitext-103Merity et al. (2016). To understand the impact of increased
data volume on model partitioning, models are evaluated across datasets
ranging from 1 million to 103 million words.

  * •

Penn TreeBank Mikolov and Zweig (2012) is a word-level language modeling
dataset comprised of Wall Street Journal articles. It stands as one of the
most recognized and frequently utilized corpora for evaluating language
modeling models. The Penn TreeBank Corpus was segmented as follows: sections
0-20 served as the training data with 930k tokens, sections 21-22 were
designated as the validation data with 74k tokens, and sections 23-24 were
allocated as the test data with 82k tokens. All words beyond the 10K
vocabulary were assigned to a unique token (unknown word) across all Penn
TreeBank datasets, thereby eliminating Out-Of-Vocabulary (OOV) words.

  * •

WikiText-2 Merity et al. (2016) is a robust word-level language modeling
dataset. This dataset, derived from validated Good and Featured articles on
Wikipedia, is more than twice the size of the preprocessed Penn TreeBank (PTB)
dataset. Additionally, it offers a significantly larger vocabulary and
maintains the original case, punctuation, and numbers - elements that are
entirely removed in PTB. Given its comprehensive nature, consisting of full-
length articles, the dataset is perfectly tailored for models designed to
capitalize on long-term dependencies.

  * •

WikiText-103 Merity et al. (2016) surpasses the original Penn TreeBank by
being a hundred times larger. With the same foundation in validated Good and
Featured articles on Wikipedia and identical preprocessing cleaning as in
WikiText-2, it brings a breadth of 103 million training words and a vocabulary
spanning 267k words.

###  3.2 Models

In our efforts to build a comprehensive ensemble of models, we’ve included a
wide variety of models for each dataset. This includes Transformers, Recurrent
Neural Networks (RNNs), Feed-Forward Neural Networks (FNNs), Convolutional
Neural Networks (CNNs), N-gram based models and others.

We’ve gone through the process of retraining and reproducing every possible
model that had previously been evaluated on these datasets. These models were
then added to our ensemble. Our goal in doing this was to determine which
models genuinely add value to the state-of-the-art and which ones don’t
contribute anything new and are therefore not useful in the larger context.

####  3.2.1 Transformer-based models

In terms of models based on the Transformer architecture Vaswani et al.
(2017), we’ve selected a variety of options. First, we’ve chosen Transformer-
XL Dai et al. (2019), a model that facilitates learning dependencies that
extend beyond a fixed length. We’ve also incorporated ShortFormer Press et al.
(2021), a model designed to work effectively with shorter inputs.
Additionally, we’ve included Mega model Ma et al. (2023), a Transformer-based
model that utilizes a more efficient attention mechanism. Finally, we’ve
chosen the DeLight model Mehta et al. (2020), notable for its performance
despite using significantly fewer parameters than a standard Transformer
model.

####  3.2.2 RNN-based models

Regarding Recurrent Neural Networks (RNNs) Elman (1990), we’ve utilized a
variety of models. We’ve made use of an LSTM-based Hochreiter and Schmidhuber
(1997) network with several modifications, popularly known as AWD-LSTM Merity
et al. (2017). Additionally, we’ve incorporated an LSTM model equipped with
associative memory, known as FWM Schlag et al. (2021). We’ve also included
models that are based on GRU Cho et al. (2014), specifically the EGRU model
Subramoney et al. (2023). The EGRU model is inspired by biological neuron
dynamics, leading to sparse and discrete communication between RNN units. This
results in a backward pass with backpropagation through time (BPTT) that is
both computationally sparse and efficient.

####  3.2.3 CNN-based models

Our ensemble includes models based on Convolutional Neural Networks (CNN). We
used QRNN Bradbury et al. (2016), a neural sequence modeling method that
alternates between convolutional layers, which work simultaneously across
different time steps, and a basic recurrent pooling function that works
simultaneously across different channels. We also used Gated Convolutional
Networks (GCN) Dauphin et al. (2016) that create a limited context approach
through stacked convolutions. This approach is more efficient because it
allows for parallel processing over sequential tokens.

####  3.2.4 N-gram Models

We selected a 5-gram model with KneserNey smoothing Ney et al. (1994) for our
N-gram model. This smoothing method has consistently shown top performance and
is widely used.

####  3.2.5 Additional Architectures

We incorporated a new architecture known as Structured State Spaces (S4) Gu et
al. (2021), which outperforms specific Transformers architecture. We also
integrated a method called Nearest Neighbor Language Models Khandelwal et al.
(2019). In this approach, the language model is combined linearly with a
k-nearest neighbor model.

####  3.2.6 Exclusion of Dynamic Evaluation and Cache Models

In the development of our ensemble, we’ve consciously opted not to include
cache models or those utilizing dynamic evaluation. This decision stems from
the fact that not all models we reproduced have dynamic evaluation
implemented. Adding this feature ourselves would present significant
challenges, given the numerous unique codebases we would have to modify. This
could introduce potential inconsistencies or errors and result in sub-optimal
outcomes.

Moreover, our focus lies in enhancing existing models within their original
design constraints. Incorporating dynamic evaluation deviates from the models’
original design and exceeds the scope of our current project. By excluding
dynamic evaluation and cache models, we ensure a consistent approach to our
ensemble creation.

###  3.3 Linear combination of models

Our approach to ensemble modeling involves a strategic linear combination of
multiple models to enhance the accuracy of predictions. We employ a method of
soft voting, which involves a weighted average of the predicted word
probabilities. This process enables us to integrate the strengths of
individual models, striking a balance that leverages the unique capabilities
of each model.

In this model, individual predictions, denoted by
$P_{j}(w_{i}|w_{1}...w_{i-1})$, are weighted by a specific weight, $m_{j}$,
which is a positive value and sums up to one. This is used to calculate the
final probability of a word, $w_{i}$, following the formula:

$P(w_{i}|w_{1}...w_{i-1})=\sum_{j=0}^{K}m_{j}P_{j}(w_{i}|w_{1}...w_{i-1})$

where $K$ represents the total number of ensemble models.

The determination of weights, $m_{j}$, is a critical step in our process. We
achieve this by minimizing the cross-entropy, $H(W)$, on the validation set,
following the formula:

$H(W)=-\frac{1}{N}\sum_{i=1}^{N}\log(P(w_{i}|w_{1}...w_{i-1}))$

where $N$ is the total number of words. Here, the softmax function is applied
to ensure that the weights, $m_{j}$, are positive and that their total sum is
equal to one.

By minimizing cross entropy, we are directly minimizing the perplexity, a
measure of how well a probability distribution or probability model predicts a
sample, as defined by Jurafsky and Martin (2009):

$PP(W)=e^{H(W)}$

This process allows us to evaluate the performance of our ensemble model and
iteratively improve the weighting and combination of models to achieve optimal
results.

##  4 Results

###  4.1 Penn TreeBank Results

(a) Penn TreeBank | (b) WikiText-2
---|---
Model |  Weight $\uparrow$ |  Validation $\downarrow$ |  Test $\downarrow$ |  Weight $\uparrow$ |  Validation $\downarrow$ |  Test $\downarrow$
QRNN Bradbury et al. (2016) |  0.00 |  60.38 |  58.43 |  0.00 |  69.23 |  66.61
KnerserNey-5gram Ney et al. (1994) |  0.01 |  148.41 |  141.46 |  0.01 |  233.93 |  219.27
Fast Weights Schlag et al. (2021) |  0.11 |  58.49 |  56.45 |  0.03 |  69.51 |  66.49
EGRU Subramoney et al. (2023) |  0.15 |  61.21 |  57.18 |  0.22 |  69.40 |  67.20
AWD-LSTM-MOS Yang et al. (2017) |  0.19 |  56.04 |  54.00 |  0.17 |  63.92 |  61.54
Transformer-XL Dai et al. (2019) |  0.20 |  57.93 |  55.41 |  0.16 |  67.41 |  64.85
AWD-LSTM-DOC Takase et al. (2018) |  0.34 |  54.12 |  52.38 |  0.41 |  60.27 |  58.01
Ensemble of All |  1 |  48.92 |  47.31 |  1 |  55.40 |  53.73
Table 1: Perplexity on validation and test sets for Penn TreeBank and
WikiText-2 dataset Mikolov and Zweig (2012); Merity et al. (2016). First
column corresponds to the model name. Second column refers to the weight the
model receives in the ensemble. Valid and test refers to the obtained
perplexity on validation and test set. A downward arrow ($\downarrow$)
indicates that lower is better.  (c) WikiText-103
---
Model | Weight $\uparrow$ | Validation $\downarrow$ | Test $\downarrow$
Delight Mehta et al. (2020) | 0.00 | 42.77 | 44.42
GCN Dauphin et al. (2016) | 0.00 | 34.34 | 35.81
AdaptiveInputs Baevski and Auli (2018) | 0.03 | 18.15 | 18.86
S4 Gu et al. (2021) | 0.06 | 20.47 | 21.14
ShortFormer Press et al. (2021) | 0.09 | 17.47 | 18.15
MEGA Ma et al. (2023) | 0.29 | 17.17 | 18.09
kNN LM Khandelwal et al. (2019) | 0.53 | 14.39 | 14.54
Ensemble of All | 1 | 13.11 | 13.29
Table 2:  Perplexity on validation and test sets for WikiText-103 (WT-103)
dataset Merity et al. (2016). First column corresponds to the model name.
Second column refers to the weight the model receives in the ensemble. Valid
and test refers to the obtained perplexity on validation and test set. A
downward arrow ($\downarrow$) indicates that lower is better. Since previous
model authors did not train and evaluate their models on larger datasets like
WT-103, we had to identify models that were, leading to the creation of the
ensemble with different models.

Our analysis of the Penn TreeBank dataset from Table 1 reveals several
intriguing findings. Firstly, we note that the QRNN model Bradbury et al.
(2016), despite achieving a perplexity of 58.43 on the test data, does not
contribute any weight to our ensemble. In contrast, the KneserNey 5-gram model
Ney et al. (1994), which exhibits perplexity more than twice as high, is
assigned a higher weight in the ensemble. This discrepancy can be attributed
to the distinct behaviors of these models; the 5-gram model operates
differently from the QRNN, which is constructed on neural network foundations
combining RNN and CNN architectures. Since these architectural components are
already represented in our ensemble, it appears that the QRNN does not provide
any additional benefits.

Furthermore, we observe that models based on Transformers Vaswani et al.
(2017) and RNN Elman (1990) architectures dominate the ensemble’s weights,
indicating their superior performance in this small-scale language modeling
task. Interestingly, even though the AWD-LSTM-MOS Yang et al. (2017) model
exhibits better perplexity scores compared to Transformer-XL Dai et al.
(2019), the latter is assigned a higher weight in the ensemble. This disparity
can be attributed to the complementarity of Transformer-XL within the
ensemble, as it demonstrates a distinct behavior from the LSTM-based models.

Incorporating these diverse models into our ensemble leads to a substantial
improvement in perplexity, surpassing the state-of-the-art AWD-LSTM-DOC Takase
et al. (2018) by over 10%.

###  4.2 WikiText-2 Results

Results obtained on the Wiki-Text 2 dataset, which is two times larger than
the previously used Penn TreeBank dataset, highlight the prominence of the
AWD-LSTM-DOC model Takase et al. (2018), followed closely by EGRU Subramoney
et al. (2023), a distinct variant of LSTM Hochreiter and Schmidhuber (1997),
as we can see in Table 1.

Interestingly, despite KnerserNey exhibiting more than a threefold higher
perplexity compared to QRNN, KneserNey has a much higher contribution to our
ensemble whereas QRNN receives 0 weight. This phenomenon mirrors our
observations on the Penn TreeBank dataset, where n-gram models exhibit
distinctly different behavior from neural networks. Consequently, they retain
their significance within the ensemble, despite their significantly lower
performance.

The combination of these diverse models in our ensemble leads to a substantial
improvement in perplexity, surpassing the state-of-the-art AWD-LSTM-DOC Takase
et al. (2018) model by more than 8%.

###  4.3 Wikitext-103 Results

Figure 3: This figure illustrates the complementary nature of the models
within the ensemble. It showcases the predicted probabilities for each word in
the sentence ’will be available in minimum denominations’ as determined by
each model in the ensemble. Notably, one of the RNN-based models, EGRU
Subramoney et al. (2023), surpasses other models with its predictions for the
words ’be’ and ’in’, by roughly 0.2. On the other hand, the Transformer-XL
model Dai et al. (2019) surpasses other models predictions for the word
’minimum’ by more than 0.5. This underscores the value of having diverse
models in the ensemble, justifying why both Transformer-XL and EGRU are
assigned substantial weights of 0.2 and 0.15 respectively in the ensemble on
the Penn TreeBank test set.

Wikitext-3 represents the largest dataset from which we were able to assemble
a substantial ensemble of models. With its dataset size being 100 times larger
than the Penn TreeBank, our findings on this corpus reveal intriguing
insights, as can be seen in Table 2.

As most previous models were not trained or evaluated on larger datasets like
WT-103, we sought models that were trained specifically on this dataset. Due
to the complexity of finding accurate hyperparameters across various
codebases, we chose to train and evaluate only those models with provided
hyperparameters, enabling us to replicate the results.

Remarkably, the kNN LM Khandelwal et al. (2019) model emerges as the dominant
contributor, receiving more than half of the ensemble’s weight. This
underscores the value of leveraging kNN neighbors in language modeling,
demonstrating its complementarity with other model types.

Following closely is MEGA Ma et al. (2023), which receives nearly 0.3 of the
ensemble weight. This result suggests that an alternative attention mechanism,
distinct from the conventional approach, plays a complementary role in our
ensemble. Additionally, our exploration of newer architectures, such as S4 Gu
et al. (2021), demonstrates that it still contributes with a weight of 0.06,
although Transformer-based models significantly outweigh it.

In contrast, models based on CNN, specifically GCN Dauphin et al. (2016), and
the Delight model Dauphin et al. (2016), do not perform well in this context,
receiving zero weight in our ensemble.

With a substantial reduction in perplexity of over 9%, our ensemble of the
best models has achieved significant improvements compared to the state-of-
the-art kNN LM Khandelwal et al. (2019).

###  4.4 Complementarity in Model Ensemble

This section offers practical examples that highlight how different models
within the ensemble complement each other. We conducted a series of tests on
the Penn TreeBank test set Mikolov and Zweig (2012) to find out the biggest
differences in word probabilities between the models. These findings shine a
light on the strength of using an ensemble approach. In situations where one
model might have difficulty, another model steps up to give a more precise
prediction, boosting the overall precision of the ensemble. A specific example
of this phenomenon on a sample sentence is presented in Figure 3.

Figure 3 displays how the models in the ensemble work together. It presents
the predicted probabilities for each word in the sentence ’will be available
in minimum denominations’ as determined by each model in the ensemble.
Interestingly, EGRU model Subramoney et al. (2023) does better than others in
predicting the words ’be’ and ’in’ by about 0.2. On the other hand, the
Transformer-XL model Dai et al. (2019) is better at predicting the word
’minimum’ by over 0.5. This shows why it is beneficial to have different types
of models in the ensemble and why both Transformer-XL and EGRU have larger
weights in the ensemble on the Penn TreeBank test set. The image also shows
that the predictions between models can change for each word, with some
increasing and some decreasing, e.g. for the word ’minimum’.

In an effort to provide a more comprehensive understanding of the
complementarity of the models in the ensemble, further examples and detailed
explanations are available in Appendix A. These additional examples not only
support the findings presented in this section but also provide deeper
insights into the dynamic interplay between different models in the ensemble.

##  5 Conclusion

In conclusion, our framework has significantly advanced the state of the art
in statistical language modeling. By advocating for the publication of code
and probabilities on development and test sets, we have simplified the
integration of new models into an ensemble. This has led to a more efficient
determination of a model’s complementarity to the current state of the art,
thereby accelerating scientific progress.

Our approach has yielded remarkable results, with up to a 10% improvement on
various language modeling benchmarks, including the Penn TreeBank, WikiText-2,
and Wikitext-103 datasets. This underscores the value of model diversity and
the potential contribution of suboptimal models when combined effectively.

Crucially, our approach has also addressed a longstanding issue within the
scientific community: the traditional focus on developing single models, which
often overlooks the immense potential of ensemble state-of-the-art. This was
primarily due to the challenges in reproducing results, which impeded the
creation of ensembles. Our work has radically transformed this dynamic,
enabling the community to progress more rapidly and enhance the true state-of-
the-art, beyond just individual model performance.

We have made all our reproduced results on these datasets available, allowing
for easy testing of new models to determine their true complementarity to the
state-of-the-art222https://github.com/DavidHerel/sota_lm. It is no longer
necessary to produce a new state-of-the-art model that behaves identically to
others; instead, it is sufficient to produce a complementary model, which
despite potentially underperforming on its own still enhances the ensemble’s
performance.

We believe that our approach, promoting transparency and reproducibility, can
revolutionize the field by lowering barriers for new ideas and eliminating
redundant efforts. We encourage the research community to adopt this framework
for more rapid and diverse advancements in language modeling.

## Limitations

While our approach has demonstrated significant advancements, it is important
to acknowledge certain limitations. Firstly, the use of ensembles inherently
increases the complexity of the code base. This increased complexity could
potentially make it harder to maintain the code.

Our models were trained on data up to 100M words, due to the computational
power available to us. This, however, does not limit the applicability of our
approach to larger datasets. We encourage future work, especially by groups
with greater computational resources, to apply our framework to larger
benchmarks.

## Acknowledgement

We would like to thank Daniela Hradilova for the useful discussions and
suggestions. Our work is part of the RICAIP project that has received funding
from the European Union’s Horizon 2020 research and innovation programme under
grant agreement No. 857306.

## References

  * Aniol et al. (2019) Anna Aniol, Marcin Pietron, and Jerzy Duda. 2019.  Ensemble approach for natural language question answering problem.  In _2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW)_ , pages 180–183.
  * Baevski and Auli (2018) Alexei Baevski and Michael Auli. 2018.  Adaptive input representations for neural language modeling.  _CoRR_ , abs/1809.10853.
  * Bradbury et al. (2016) James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2016.  Quasi-recurrent neural networks.  _CoRR_ , abs/1611.01576.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _CoRR_ , abs/1406.1078.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019.  Transformer-xl: Attentive language models beyond a fixed-length context.  _CoRR_ , abs/1901.02860.
  * Dauphin et al. (2016) Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2016.  Language modeling with gated convolutional networks.  _CoRR_ , abs/1612.08083.
  * Elman (1990) J. L. Elman. 1990.  Finding structure in time.  _Cognitive Science_ , 14:179–211.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  A theoretically grounded application of dropout in recurrent neural networks.
  * Goodman (2001) Joshua Goodman. 2001.  A bit of progress in language modeling.  _CoRR_ , cs.CL/0108005.
  * Gu et al. (2021) Albert Gu, Karan Goel, and Christopher Ré. 2021.  Efficiently modeling long sequences with structured state spaces.  _CoRR_ , abs/2111.00396.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Comput._ , 9(8):1735–1780.
  * Jacobs et al. (1991) Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991.  Adaptive mixtures of local experts.  _Neural Computation_ , 3(1):79–87.
  * Jurafsky and Martin (2009) Daniel Jurafsky and James H. Martin. 2009.  _Speech and Language Processing (2Nd Edition)_.  Prentice-Hall, Inc., Upper Saddle River, NJ, USA.
  * Khandelwal et al. (2019) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019.  Generalization through memorization: Nearest neighbor language models.  _CoRR_ , abs/1911.00172.
  * Ma et al. (2023) Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. 2023.  Mega: Moving average equipped gated attention.
  * Mehta et al. (2020) Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2020.  Delight: Very deep and light-weight transformer.  _CoRR_ , abs/2008.00623.
  * Merity et al. (2017) Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017.  Regularizing and optimizing LSTM language models.  _CoRR_ , abs/1708.02182.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.  Pointer sentinel mixture models.  _CoRR_ , abs/1609.07843.
  * Mikolov et al. (2010) Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010.  Recurrent neural network based language model.  volume 2, pages 1045–1048.
  * Mikolov and Zweig (2012) Tomas Mikolov and Geoffrey Zweig. 2012.  Context dependent recurrent neural network language model.  In _2012 IEEE Spoken Language Technology Workshop (SLT)_ , pages 234–239.
  * Ney et al. (1994) Hermann Ney, Ute Essen, and Reinhard Kneser. 1994.  On structuring probabilistic dependences in stochastic language modelling.  _Computer Speech & Language_, 8(1):1–38.
  * Press et al. (2021) Ofir Press, Noah A. Smith, and Mike Lewis. 2021.  Shortformer: Better language modeling using shorter inputs.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5493–5505, Online. Association for Computational Linguistics.
  * Sagi and Rokach (2018) Omer Sagi and Lior Rokach. 2018.  Ensemble learning: A survey.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 8:e1249.
  * Schlag et al. (2021) Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. 2021.  Learning associative inference using fast weight memory.  In _International Conference on Learning Representations_.
  * Shannon (1951) Claude Elwood Shannon. 1951.  Prediction and entropy of printed english.  _Bell System Technical Journal_ , 30:50–64.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _CoRR_ , abs/1701.06538.
  * Subramoney et al. (2023) Anand Subramoney, Khaleelulla Khan Nazeer, Mark Schöne, Christian Mayr, and David Kappel. 2023.  Efficient recurrent architectures through activity sparsity and sparse back-propagation through time.
  * Takase et al. (2018) Sho Takase, Jun Suzuki, and Masaaki Nagata. 2018.  Direct output connection for a high-rank language model.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4599–4609, Brussels, Belgium. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _CoRR_ , abs/1706.03762.
  * Yang et al. (2017) Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2017.  Breaking the softmax bottleneck: A high-rank RNN language model.  _CoRR_ , abs/1711.03953.

##  Appendix A Detailed Analysis of Model Interplay in Ensemble

This Appendix section offers practical examples that highlight how different
models within the ensemble complement each other. We conducted a series of
tests on the Penn TreeBank test set Mikolov and Zweig (2012) to find out the
biggest differences in word probabilities between the models. These findings
shine a light on the strength of using an ensemble approach. In situations
where one model might have difficulty, another model steps up to give a more
precise prediction, boosting the overall trustworthiness of the ensemble.

Figure 4: The figure presents the predicted probabilities for each word in the
sentence ’pacific telesis group stopped trading’ as determined by each model
in the ensemble on the Penn TreeBank test set. The complementarity of the
models within the ensemble is evident. For instance, the EGRU Subramoney et
al. (2023) model significantly outperforms other models in predicting the
words ’telesis’ and ’group’, while the Transformer-XL Dai et al. (2019) excels
in predicting the word ’trading’.

Figure 5: This figure depicts the predicted probabilities for the sentence
’the bills will be dated’ as determined by each model in the ensemble on the
Penn TreeBank test set. Despite the general similarity in predictions among
the models, the Transformer-XL Dai et al. (2019) model displays a unique
behavior in predicting the word ’dated’, demonstrating an increased
probability compared to the other models.