# SNNLP: Energy-Efficient Natural Language Processing Using Spiking Neural
Networks

R. Alexander Knipper  Kaniz Mishty  Mehdi Sadi  and Shubhra Kanti Karmaker
Santu Manuscript received October 27, 2023; revised XXX XX, 202X. This work
was supported in part by the National Science Foundation (NSF) under Grant
Number CRII-2153394. The authors R. Alexander Knipper and Shubhra Kanti
Karmaker Santu are with the Department of Computer Science and Software
Engineering, Auburn University, Auburn, AL 36849, USA. E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>The authors Kaniz Mishty and Mehdi
Sadi are with the Department of Electrical and Computer Engineering, Auburn
University, Auburn, AL 36849, USA. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As spiking neural networks receive more attention, we look toward applications
of this computing paradigm in fields other than computer vision and signal
processing. One major field, underexplored in the neuromorphic setting, is
Natural Language Processing (NLP), where most state-of-the-art solutions still
heavily rely on resource-consuming and power-hungry traditional deep learning
architectures. Therefore, it is compelling to design NLP models for
neuromorphic architectures due to their low energy requirements, with the
additional benefit of a more human-brain-like operating model for processing
information. However, one of the biggest issues with bringing NLP to the
neuromorphic setting is in properly encoding text into a spike train so that
it can be seamlessly handled by both current and future SNN architectures. In
this paper, we compare various methods of encoding text as spikes and assess
each method’s performance in an associated SNN on a downstream NLP task,
namely, sentiment analysis. Furthermore, we go on to propose a new method of
encoding text as spikes that outperforms a widely-used rate-coding technique,
Poisson rate-coding, by around 13% on our benchmark NLP tasks. Subsequently,
we demonstrate the energy efficiency of SNNs implemented in hardware for the
sentiment analysis task compared to traditional deep neural networks,
observing an energy efficiency increase of more than 32x during inference and
60x during training while incurring the expected energy-performance tradeoff.

As we see typical Natural Language Processing (NLP) architectures requiring
vast amounts of computational power (e.g. ChatGPT, LLaMA, etc.), the issue of
energy efficiency in NLP becomes more apparent than ever. Spiking neural
networks (SNNs) offer vast energy savings under a new computing paradigm,
which has its strengths in processing sequences. However, while most NLP
technologies process language as a sequence, there is not a well-defined way
to convert these text sequences into binary “0/1” spike trains for use in the
energy-efficient paradigm introduced by SNNs. In this paper, we recommend a
method of encoding text as a spike train that outperforms other typically used
methods by roughly 13% on our benchmark NLP tasks while using less energy,
allowing NLP to more readily be used in this energy-efficient environment.

Spiking Neural Networks, Low-Energy Computing, Natural Language Processing

##  1 Introduction

As we continue into the data-driven age, we find that more and more
applications are leveraging machine learning techniques in some capacity.
However, today’s machine learning approaches bring up a glaring issue: these
methods are incredibly energy-consuming, using countless floating-point
operations to process large amounts of data. As we look toward implementing
these machine learning (ML) applications in IoT devices, we need to reduce the
energy consumption on learning tasks in order to increase the battery life of
such devices.

Common approaches to reducing energy consumption to date mainly exist in
improving the underlying hardware, while still adhering to the typical Von-
Neumann architecture. However, a more promising approach lies in the
Neuromorphic architecture, where the goal is to create biologically-inspired
computing chips for machine learning that deviates from the standard Von-
Neumann approach [1]. This biologically-inspired approach places data and
processing power in the same place—much like how the brain operates, reducing
the overhead in moving data to/from a processor for intensive calculations
like those done in typical ML approaches.

Using these biologically-inspired Spiking Neural Networks (SNNs), we can
perform similar processing to typical Artificial Neural Networks (ANNs) using
a network of neurons that exclusively communicate using binary “0/1” pulses.
Furthermore, due to the nature of these ‘spiking neurons’, spiking neural
networks end up modeling the dependencies in input data over time, which
aligns itself incredibly well with processing sequential data. With that in
mind, natural language processing (NLP) presents itself as a promising
candidate for implementations in spiking hardware, especially with the energy-
intensive networks that currently dominate the state-of-the-art.

Current SNN research efforts focus mainly on two areas: computer vision and
information processing [2]. As a result of these efforts, spiking neural
networks were proven both viable and useful in terms of achieving a comparable
performance (with some expected performance degradation) to real-valued
counterpart neural networks. Additionally, the spiking neural networks exhibit
much lower energy consumption than their real-valued counterparts. However,
many research areas, including NLP, are still underexplored in the spiking
neural network setting and hence, it is compelling to study popular NLP tasks
in this neuromorphic computing setting.

As text data is sequential in nature, NLP tasks in general lend themselves
well to the processing paradigm of current spiking neural network
architectures, which involve a time dimension and take in spike trains as
inputs. However, one of the largest issues with bringing NLP to the
neuromorphic setting is in properly encoding text into a spike train so it can
be seamlessly handled by current and future SNN architectures. Some typical
methods that can be leveraged to do so include binarized word embeddings [3]
and rate-coding techniques [4], but each method has its drawbacks.

In this paper, we explore the viability of processing text using spiking
neural networks by applying them to a popular NLP problem: sentiment
classification. In doing so, we aim to answer four major questions:

  1. 1.

Which works better for NLP tasks, directly using a binary embedding or rate-
coding a floating-point embedding?

  2. 2.

If we make rate-coding a deterministic process as opposed to a stochastic one,
do we see higher accuracy on NLP tasks?

  3. 3.

On NLP tasks, do we see the same energy-accuracy tradeoff that is reported in
previous SNN literature?

  4. 4.

How far can we decrease the latency cost of SNNs while keeping performances
competitive with traditional ANNs?

Coupled with those major questions, we offer four major findings, as follows:

  1. 1.

While both binary embeddings and rate-coded floating-point embeddings perform
similarly on downstream NLP tasks, directly using the binary word embedding
offers slightly improved performance.

  2. 2.

We propose a method that makes rate-coding into a deterministic process,
offering significantly improved performance ($13\%$ increased accuracy) on
downstream NLP tasks compared to a typical stochastic method, Poisson rate-
coding.

  3. 3.

We observe a similar energy-performance tradeoff to results seen in previous
SNN literature [2, 5], further grounding our results.

  4. 4.

Latency can be reduced to between $9.3\times$ and $17.5\times$ without
significantly affecting accuracy, and can be reduced to between $3.7\times$
and $7\times$ when taking a performance hit of roughly $7$-$10\%$.

To be more specific, we show that, given an appropriate spike formatting for
input text, emerging SNN training techniques are applicable to make inferences
on text data. We then subsequently report the energy efficiency of SNN
architectures applied to these tasks (Fig. 1). In summary, the paper is
organized as follows: Section 2 presents some background information and
related work. Section 3 explores different methods of encoding text for use in
a spiking neural network. Section 4 describes our methods and the energy model
we use. Section 5 describes our experimental setup, and section 6 presents our
results on the sentiment classification problems, discussing our findings.
Finally, we conclude the paper in Section 8, discussing the limitations of our
work and some possible future directions.

Figure 1: Overall encoding/training pipeline for both SNN and ANN. Text gets
encoded into a continuous representation, converted into a spike train (for
SNN usage), and sent to the downstream neural network for training/testing.
During training/testing, operation counts are recorded so we can calculate
appropriate energy numbers for comparison.

##  2 Related Work

###  2.1 Spiking Neural Networks

Spiking neural networks promise incredibly low-powered hardware [1], adhering
to a neuromorphic paradigm as opposed to the more energy-intensive Von-Neumann
approaches we see in everyday commercial use. However, at present, spiking
neural networks are difficult to train, lacking typical backpropagation
methods and commonly-used activation functions. Furthermore, there is still a
considerable gap in performance between a spiking neural network and its real-
valued artificial neural network counterpart. To address this, three main
approaches emerged, each aiming for a trained SNN with comparable performance
to a similar ANN:

####  2.1.1 Conversion Approaches

The most-studied approach has been the conversion approach, where an ANN is
trained with backpropagation, and its learned weights are mapped to an SNN
with the same architecture. So far, this approach has achieved the most
consistent performance [5].

####  2.1.2 Unsupervised Learning Approaches

Unsupervised learning in SNNs is mainly explored in applications or
modifications of Spike Time-Dependent Plasticity (STDP) [4], and is perhaps
the second most-studied approach for SNN training.

####  2.1.3 Supervised Learning Approaches

Arguably the most difficult approach for learning in SNNs is supervised
learning, mostly due to the inability to calculate gradients from the spiking
activation function. However, active research is proposing an alternate,
“surrogate” gradient approximation function. The most notable work in this
area is SLAYER [6], which treats the spiking activations probabilistically,
and approximates the gradient using the estimated probability density function
of the spiking activation. In this work, we focus mainly on supervised
learning and utilize SLAYER for backpropagation during SNN training.

###  2.2 Natural Language Processing

####  2.2.1 Text Encoding Methods

In natural language processing, one of the key techniques applied in nearly
every approach taken is to encode the input text into a continuous
n-dimensional ‘feature vector’ called an embedding [7], mapping each token’s
(character, word, sentence, etc.) symbolic representation to a
multidimensional latent vector space. One of the most important properties of
this vector space is that it not only encodes the syntactic information of
words but also encodes the proper semantic information, allowing related
meanings to be extracted easily. Many word embedding techniques have been
proposed so far, e.g., Word2Vec [8], GloVe [9], FastText [10], etc.

Going beyond word embeddings, NLP researchers have also proposed methods that
encode larger language constructs, such as phrases [10] or sentences [11, 12].
These representations for larger language constructs have gained more traction
in NLP research as the needs for more dense representations of text have
become more apparent.

####  2.2.2 Embedding Binarization

While binarization efforts fall under efficient NLP, it is also a relatively
underexplored field. However, one piece of notable work includes the proposal
of a hash-based clustering technique for learning binary embeddings [13],
where the authors concatenated the binary codes of the closest centroids for
each word. Another notable method is to transform an existing real-valued
embedding to a binary embedding using an auto-encoder [3], and yet another
method is to learn correlations between one-hot encoded context and target
blocks [14]. Furthermore, our own previous work explored a method of learning
compressed binary embeddings from scratch using a genetic algorithm [15].

##  3 Background on Spike-Aware Text Encoding

In order to make proper use of a spiking neural network for natural language
processing, we need to first encode our input text into a spike train. This
can be done by converting already-existing feature vectors created for any
text token (character, word, sentence, etc.) to binary spikes, either
deterministically or non-determinstically. In this work, we focus on comparing
both deterministic and non-deterministic conversion approaches, assessing the
accuracy and energy consumption of each.

###  3.1 Word Embeddings

In the real-valued domain, word embeddings typically capture semantic and
syntactic information by observing word co-occurrences and either predicting
the target word given its context, predicting the context given a target word,
or observing the ratios between global co-occurrences [7, 9]. To use a word
embedding vector in an SNN, we need to convert said word vector from the real-
valued domain to the spiking domain. This involves utilizing either a
binarization method or a rate-coding approach.

When using a binarization method, like the near-lossless binarization method
proposed in [3], one can convert a vector from a multi-hundred dimensional
vector of floating-point numbers to a binary vector, comprised of ”0/1” bits
(commonly, embeddings are made up of 64, 128, 256, or 512 bits). This allows
for a reduction in both the size and complexity of the word embeddings being
used as input. While this method typically requires an autoencoder to generate
the binary embeddings, this conversion is an operation that only needs to be
done _once_ , so the energy cost incurred can be ignored under the assumption
that the word embeddings being used were already converted in advance.

On the other hand, rate-coding is a well-known method of encoding floating-
point information as a spike train [4, 16, 17, 18]. Using this method,
floating-point numbers are easily converted to a series of spikes based on a
Poisson process, where, for a number of timesteps $k$:

$Y_{i}=\begin{cases}1&\text{if }X_{i}<p_{i}\\\ 0&\text{if }X_{i}\geq
p_{i}\end{cases}$ (1)

Given that $X\sim\text{Binomial}(k,p)$. This process is repeated for all items
in the input vector $v$, generating a spike train for each of the input
vector’s feature dimensions.

In this work, we explore and compare both binarization and rate-coding
approaches at the word level to gauge the reliability of each. Furthermore, we
put forth an additional rate-coding technique that aims to increase the
fidelity of the downstream SNN’s input.

###  3.2 Sentence Embeddings

Unlike word embeddings, sentence embeddings typically utilize a Transformer
[19] model to encode sentences as a single vector representation, training to
maximize the ability to seamlessly transfer sentence representations between
different NLP tasks. With these representations, each sentence’s relative
meaning is also encoded, allowing these embeddings to be used in a similar
manner to word embeddings, but at the sentence level. However, due to their
reliance on the underlying Transformer architecture, sentence embeddings tend
to inherently consume more energy than word embeddings in return for increased
information density, information accuracy, and generalizability.

Similar to word embeddings, sentence embeddings can also be converted for use
in an SNN via binarization techniques or rate-coding techniques. However, it
is important to know that the majority of sentence embeddings generate their
sentence embeddings _each time_ a new dataset is introduced, as opposed to
only being calculated once. This makes sentence embeddings far more expensive
energy-wise compared to word embeddings. As such, we make use of sentence
embeddings in this work purely to help illustrate the potential performance of
spike-aware text encoding approaches.

In this work, we make use of Universal Sentence Encoder (USE) [12] to encode
each of our input sentences into a single 512-dimension floating-point vector,
allowing for a compact representation of each text sample. Once again, in
order to use these embedded representations in an SNN, we need to convert this
floating-point vector into a binary vector using either a binarization method
or a rate-coding method. In this work, we opt to utilize various rate-coding
approaches, as binarization methods typically involve an autoencoder _in
addition to_ the Transformer model being used to generate the embedding in the
first place, which is far too expensive energy-wise.

##  4 Sentiment/Emotion Classification using SNN

To demonstrate an SNN’s effectiveness in performing natural language
processing tasks, we applied it to a simple, widely-used task called sentiment
classification. By extension, we also demonstrate various possible methods of
preprocessing text data for use in an SNN. As described in Section 5, we
choose to test the SNN on both a 2-class sentiment classification task and a
6-class emotion classification task to demonstrate its effectiveness and
benchmark its performance.

###  4.1 Network Type

While sentiment classification tasks are currently solvable with a high
accuracy ($>94\%$) using state-of-the-art neural architectures like
Transformers [19], SNNs do not yet have these complex architectures reliably
available to them. Therefore, in order to perform an apple-to-apple
comparison, we opt to implement both the spiking and real-valued neural
network architectures using a fully-connected, feedforward neural network on
both of these tasks, and subsequently, report each downstream SNN’s accuracy
and energy consumption compared to the equivalent, real-valued “shadow”
network.

###  4.2 Encoding Methods

We test various encoding methods in order to find the best way to properly
utilize the time-based property of the network. Namely, we utilize the
following method-model combinations, both at the sentence and word-level (with
the exception of SNN-bin, which is only used at the word-level):

  * •

ANN \- This is a typical ANN using regular floating-point embeddings

  * •

SNN-rate \- This is a custom, deterministically rate-coded embedding being
used in a downstream SNN

  * •

SNN-rate-rand \- This is a Poisson rate-coded embedding being used in a
downstream SNN

  * •

SNN-bin \- This is a binarized embedding being used in a downstream SNN

For our comparisons, we use an inference window of 50ms with the exception of
SNN-bin, which instead takes in a fixed-length context of 512 for the 2-class
task and 32 for the 6-class task. In this method, the input text is encoded at
the word-level and formatted into spike trains as-is, so a sufficiently-long
context is required to represent all of the words in a given input sequence.

Our other methods (ANN, SNN-rate, and SNN-rate-rand) leverage the average of
all word embeddings in the input sequence, which allows us to get an
approximation of a sentence-level embedding without the associated energy
cost. Once we have this averaged embedding, we then encode it with our rate-
coding methods for use in our downstream SNNs. As explained in section 3.1,
the Poisson rate-coding process is a rather popular encoding method, and
results in a stochastic set of spike trains that roughly approximate the
floating-point number that was encoded. However, this approach raises a
question: What if we deterministically encode the spikes such that the
floating-point value being encoded is represented with higher fidelity? Would
this higher-fidelity spiking rate result in a corresponding higher
performance?

Our custom rate-coding method, which we refer to as “SNN-rate” in this work,
aims to address that question by modifying the inherently stochastic Poisson
rate-coding method (referred to as “SNN-rate-rand”) to instead generate
deterministic spike trains for the given embedding. We generate our
deterministic encoding as follows:

$V_{i}(t)=V_{i}(t-1)+x_{i}(t)$ (2) $Y_{i}(t)=\begin{cases}1&\text{if
}V_{i}(t)\geq\alpha_{i}\\\ 0&\text{otherwise }\end{cases}$ (3)

Where $x_{i}$ is the floating-point value being encoded, $V_{i}(t)$ is the
membrane potential of the neuron at timestep $t$, and $\alpha_{i}$ is the
membrane threshold of the neuron. Essentially, we modify the standard Leaky
Integrate-and-Fire (LIF) neuron model in the following ways:

  * •

Make the neuron take in a floating-point value: the self-accumulation
parameter

  * •

Every timestep, perform the accumulation operation using _only_ the value
stored as the self-accumulation parameter, which we refer to as the self-
accumulation operation

It is important to note that this modified “Self-Accumulate-and-Fire” (SAF)
neuron model (Fig. 2) only takes as input the self-accumulation parameter,
$x_{i}$, with the clock, $t$, telling the neuron how many times to perform the
self-accumulation operation. Furthermore, when the neuron receives a new
floating-point input, that input value becomes the neuron’s new self-
accumulation parameter, $x_{i}$, for all future self-accumulation operations.

Figure 2: Illustrated “Self-Accumulate-and-Fire” (SAF) neuron. Takes in a
floating-point value and self-accumulates it over time, spiking similarly to a
standard LIF neuron.

The SAF neuron’s output is then recorded as our rate-coded spike train and is
sent directly to the rest of the SNN. This modified “Self-Accumulate-and-Fire”
(SAF) neuron, designed to enhance the integration of rate-coding behavior into
the SNN model’s architecture, facilitates our deterministic rate-coding
process. This approach lends itself well to the spiking paradigm, particularly
when formulated as a modified neuron model.

###  4.3 Training Method

While using the various spike-aware text encodings discussed in Section 3, we
opt to train our downstream SNNs using SLAYER [6], which allows the SNN to
update its weights during training using a surrogate gradient for
backpropagation. We additionally train two “baseline” ANNs for each task: 1) A
regular ANN using the floating-point sentence embeddings, and 2) An ANN using
floating-point word embeddings.

For all networks, we maintain their network architectures to be as identical
as possible. All networks utilize an input size eqivalent to the size of the
embedding technique being used. The input sizes used for each embedding method
are as follows:

  * •

Floating-point & rate-coded sentence embeddings: $512$

  * •

Floating-point & rate-coded word embeddings: $200$

  * •

Binarized word embeddings: $64$

All networks have two hidden layers, sized at 256 and 128 neurons
respectively. Finally, we size the output layer to match the number of output
classes in each benchmark task, i.e. 2 neurons for sentiment classification
and 6 neurons for emotion classification.

All networks use a learning rate ($\eta$) of $1e^{-5}$, and train for 50
epochs. The ANNs use cross-entropy loss as their loss function, while the SNNs
use the spike rate as their loss function. Namely, the SNNs aim for the
correct output class’ neuron to spike for a majority of the inference window
and all other output class’ neurons to spike far less.

Furthermore, the SNNs all use the following neuron parameters:

  * •

threshold: $1.25$

  * •

current_decay: $0.25$

  * •

voltage_decay: $0.03$

  * •

tau_grad: $0.03$

  * •

scale_grad: $3$

Where the threshold and decay values determine the general behavior of the
Leaky Integrate and Fire (LIF) neuron, that is, when the neuron fires and how
much membrane potential is going to decay off per timestep. Additionally, the
tau_grad and scale_grad values adjust the time constant and scale of the
gradient, respectively.

###  4.4 Energy Model

The energy efficiency of an SNN comes from two main features. First, there is
less computational demand due to the event-driven binary activity of the
neurons. In an SNN, a computation is performed only if an incoming spike is
received. The neuron’s membrane potential $V$ at time $t$ can be expressed as
$V(t)=V(t-1)+L+\sum_{i=1}^{N}w_{i}\sum_{k=1}^{\infty}\delta_{i}(t-k)$ [1, 5].
The parameter $w_{i}$ represents the weight of synapse $i$, $\delta$ is the
delta function, $L$ is the leak term and $N$ is the number of synaptic
weights. This equation implies that in the presence of a spike at time-stamp
$t$, the synaptic weight is simply added with the existing membrane potential.
These computations can be mapped as Accumulation (AC) operations and performed
by an Adder/Accumulator circuit. On the other hand, a typical RNN/ANN neuron
requires one/two Multiplication and one Accumulation (MAC) operation. Second,
the number of both on and off-chip memory accesses is much smaller for an SNN
compared to an ANN. In a regular ANN, most of the on-chip memory is occupied
by the large intermediate activation; sometimes, we even need to go to the
off-chip to store the internal states. As a result, there are more off-chip
accesses to fetch the network parameters and activations. In addition, ANNs
require frequent on-chip accesses (depending on the size of the computation
array unit and on-chip memory size) to store the partial sums of the output
activations. However, in an SNN during inference, we only need to fetch the
synaptic weights and initial inputs from off-chip once.

To model the total system energy consumed by an SNN in operation, we dissect
the total energy into two parts: (1) energy associated with the computation
and (2) energy associated with the memory access. This paper only models the
energy required for the computations (case 1), and modeling the system energy
is considered future work as it involves detailed modeling of the hardware’s
memory system hierarchy.

We count the number of operations per layer per sample (for each NN) to
estimate the computation energy of the SNN and ANN networks [20]. An ANN layer
$l$ with $N$ neurons will require ($N_{l-1}N_{l}$) MAC operations to produce
its output. If $E_{MAC}$ is the energy cost of a single MAC operation, the
total cumulative energy consumed ($E_{ANN_{total}}$) in the computation of $L$
layers is modeled as follows:

$E_{ANN_{total}}=\sum_{l=1}^{L}(N_{l-1}N_{l})E_{MAC}$ (4)

A spiking neuron on specialized SNN hardware will perform an Accumulate (AC)
operation per synapse if a spike is received. An SNN layer $l$ with $N$
neurons upon the reception of a spike needs to perform $N_{l-1}N_{l}$ AC
operations. However, depending on the fraction of neuron firing rate, $S$, not
all the neurons will remain perpetually active [20]. Therefore, the total
energy consumed in the computation of $L$ layers over $K$ timesteps is modeled
as:

$E_{SNN_{total}}=\sum_{t=1}^{K}\sum_{l=1}^{L}(N_{l-1}S_{l-1}N_{l})E_{AC}$ (5)

$S$ is always $<<$1, making SNNs computation-efficient compared to ANNs. In
addition, an AC operation consumes less than half of the energy a MAC
operation consumes, as shown in Table 6.

##  5 Experimental Setup

In this section, we describe how we evaluate and compare each implemented SNN
and its counterpart real-valued ANN.

###  5.1 Dataset

For our experiments, we used the IMDb Movie Review dataset [21] for our simple
2-class sentiment classification problem and the CARER dataset [22] for our
multiclass (6 labels) emotion classification problem. The datasets are split
into training, validation, and testing sets as indicated in Table 1, and the
tasks associated with each dataset are briefly described below:

  * •

IMDb Movie Review: Given a string of text from a movie review, decide whether
the text is expressing positive or negative sentiment.

  * •

CARER: Given a tweet as text, decide which of the 6 emotions - sadness, joy,
love, anger, fear, or surprise - are being exhibited in the text.

Table 1: Number of training, validation, and inference examples used for each evaluation dataset Dataset | Training | Validation | Inference | Total
---|---|---|---|---
IMDb | 30,000 | 5,000 | 15,000 | 50,000
CARER | 12,000 | 2,000 | 6,000 | 20,000

###  5.2 Implementation Details

For all models, we test the same model architecture on both problems, using
the layer sizes described in Section 4.3. We train all models for 50 epochs,
recording their convergence in Figure 3.

Figure 3: All model training curves. The top two graphs depict curves for
sentence-level methods and the bottom two graphs show word-level methods. The
left-side graphs show models trained for the IMDB 2-class task, where the
right-side graphs show models trained for the 6-class emotion classification
task.

###  5.3 Evaluation Metrics

When evaluating our models for comparison, we report two main performance
metrics: inference accuracy and the mean reciprocal rank (MRR). We report
accuracy numbers for both tasks, but only report the MRR for the emotion
classification task, as it having more than two output classes means we can
benefit from knowing where the desired class sits in the overall predicted
ranking.

With the MRR measurements, we aim to examine how close the correct class was
to the top spot if we had ranked the output neurons in terms of how many times
they actually fired. For a given sample set, we define MRR as:

$MRR=\frac{1}{|S|}\sum_{i=1}^{|S|}\frac{1}{rank_{i}}$ (6)

where $S$ is the collection of testing samples and $rank_{i}$ indicates the
ordinal position of the correct class label for a particular sample $i$.

##  6 Experimental Results and Analysis

###  6.1 Model Performance

In this section, we focus on answering our first two questions: 1) Which works
better for NLP, binary embeddings or rate-coded floating-point embeddings? 2)
If we make rate-coding a deterministic process as opposed to a stochastic one,
do we see a performance improvement? In addressing these questions, we test
our SNN on two NLP tasks:

  * •

IMDb Review dataset: 2-class sentiment classification

  * •

CARER dataset: 6-class emotion classification

We outline our results on both tasks in Tables 2, 3, 4, and 5. Note that we do
not record the mean reciprocal rank (MRR) for the 2-class sentiment
classification problem, as it is not meaningful for a binary classification
task.

As shown in Tables 2, 3, 4, and 5, we demonstrate the expected performance gap
between the ANN and our SNNs. Furthermore, both the word-level and sentence-
level models are largely competitive with each other, with the sentence-level
models performing better on 2-class classification (Table 2) and word-level
models performing better with multi-class classification (Table 5).

Additionally, it appears that at the word-level, we see similar performances
from Poisson rate-coded spike trains and spike trains generated directly from
a binary word embedding (Tables 4 and 5). In all cases, we observe slightly
improved performance when using a binary word embedding as opposed to using a
Poisson rate-coded spike train. However, we see a significant performance
increase across all measures, embeddings, and tasks when the rate-coded spike
trains are generated deterministically. This also exhibits a similarly
significant improvement over using the binary embedding directly.

It is important to note that most state-of-the-art results on these tasks are
achieved using Transformer architectures, like BERT [23] and XL-NET [24],
which are more complex (and energy-intensive) models than fully-connected
neural networks. Such architectures are currently being researched, and stable
implementations are not yet available for neuromorphic hardware.

Overall, our results demonstrate the following conclusions:

  * •

We observe the expected performance degradation going from an ANN to its SNN
counterpart, which is justified by the energy improvements explored in Section
6.2.

  * •

We observe a similar performance between Poisson rate-coded and binary
embedding spike trains, with the binary embedding being slightly more
effective than typical Poisson rate-coding in both tasks.

  * •

We observe a significant increase in performance when using deterministically-
encoded spike trains as opposed to Poisson rate-coded spike trains. By
extension, we see a similarly significant improvement using the
deterministically-encoded spike trains over the binary embedding-based spike
trains as well.

Table 2: Sentence-level 2-class sentiment classification:
Accuracy Model |  | Training
---
Accuracy
| Inference
---
Accuracy
ANN | 0.859 | 0.867
SNN-rate | 0.836 | 0.840
SNN-rate-rand | 0.659 | 0.656
Table 3: Sentence-level 6-class emotion classification:
Accuracy & MRR Model |  | Training
---
Accuracy
| Inference
---
Accuracy
| Inference
---
MRR
ANN | 0.617 | 0.633 | 0.778
SNN-rate | 0.534 | 0.522 | 0.703
SNN-rate-rand | 0.410 | 0.395 | 0.609
Table 4: Word-level 2-class sentiment classification:
Accuracy Model |  | Training
---
Accuracy
| Inference
---
Accuracy
ANN | 0.821 | 0.824
SNN-rate | 0.788 | 0.788
SNN-rate-rand | 0.628 | 0.631
SNN-bin | 0.650 | 0.664
Table 5: Word-level 6-class emotion classification:
Accuracy & MRR Model |  | Training
---
Accuracy
| Inference
---
Accuracy
| Inference
---
MRR
ANN | 0.574 | 0.572 | 0.734
SNN-rate | 0.542 | 0.536 | 0.711
SNN-rate-rand | 0.424 | 0.432 | 0.633
SNN-bin | 0.446 | 0.432 | 0.635

###  6.2 Energy Efficiency

In this subsection, we demonstrate the energy improvement of our implemented
word-level, SNN-based architecture over an equivalent, conventional ANN for
performing sentiment and emotion classification on the IMDb Movie Review and
CARER datasets, respectively. We design an FP32 Multiplier-Accumulator unit
and an FP32 Accumulator unit to carry out the GEMM operations of ANNs and the
accumulation operations of SNN tasks. We synthesize and perform PNR (place and
route) of the design using Synopsys 14-nm Standard Cell Library [25] at a 1GHz
clock frequency. Each unit’s energy consumption per operation is listed below
in Table 6.

Table 6: FP MAC and ACC module energy Module | Tech. (nm) | Energy (pj)
---|---|---
FP Multiplier-Accumulator (FP-MAC) | 14 | 57.14
FP Accumulator (FP-ACC) | 14 | 25.63
Figure 4: SNN vs ANN number of arithmetic operations of _Sentiment
Classification_ during inference (a) & training (c) and _Emotion
Classification_ during inference (b) & training (d). Figure 5: Improvement of
SNN in computation energy over ANN. _Sentiment Classification_ during
inference (a) and training (c). _Emotion Classification_ during inference (b)
and training (d).

In Fig. 4, we plot the layer-wise and total computations per sample for the
ANN and SNN architectures while performing the sentiment classification
((a)-inference, ((c)-training) and emotion classification ((b)-inference,
(d)-training) tasks. As shown, ANNs appear to require an order of magnitude
more operations than SNNs. Overall, the total number of computations in ANNs
for sentiment and emotion classification are $\sim 15\times$ of SNNs during
inference (Fig. 4 (a)-(b)). As the SNNs run for 50 timesteps per sample
(equating roughly 50ms) during inference, and this “inference window” can be
adjusted, this value will increase/decrease along with the configured
inference window size. Additionally, to compare training costs on a per-epoch
basis, we averaged the number of computations per sample over all 50 epochs,
which is reported in Fig. 4 (c)-(d). As shown, ANNs require even more
operations to train their parameters: $\sim 27\times$ more than SNNs.
Furthermore, we again emphasize that MAC operations dominate the ANNs while
SNNs are dominated by AC operations, further increasing our energy
improvements.

To showcase this improvement, we estimate the energy consumption of the ANN
and SNN for our two use cases using equations 4 & 5, and the energy values
from Table 6. As shown in Fig. 5, the SNN achieves roughly $32\times$ more
energy efficiency during inference (Fig. 5 (a)-(b)) and roughly $60\times$
during training (Fig. 5 (c)-(d)) compared to the ANN for sentiment and emotion
classification tasks. However, we do observe calculations taking place in the
SNNs’ input layer (Layer 0), whereas the ANNs’ input layers do not have any
extra calculations. This is due to the input layer being comprised of spiking
neurons - since the input is a spike train, it will naturally follow the Leaky
Integrate-and-Fire (LIF) behavior, which incurs extra operations. Nonetheless,
despite the extra layers’ worth of operations and energy consumption, SNNs
still achieve exceptional energy savings compared to ANNs - roughly $32\times$
during inference and $60\times$ during training.

Overall, this means that our SNN-NLP architectures achieve the expected energy
improvements in return for the previously observed performance loss incurred.

###  6.3 Latency Impact

Furthermore, we measure the amount of time taken to perform inference on both
the ANN and the SNN to determine the effect on the per-sample inference
latency. In its current configuration, the SNNs take $50ms$ to infer each
sample for both tasks, although that is a value that can be adjusted, as
explored in section 7. The ANNs, on the other hand, take about $2.69ms$ per
sample for the sentiment classification task and $1.429ms$ per sample for the
emotion classification task. This means the SNNs have an approximate
$18.53\times$ increase in inference latency over the associated ANN for
2-class sentiment classification and a $34.99\times$ increase for multi-class
emotion classification. Overall, our observed latency increase for SNN-NLP
conforms with previously-reported SNN latency increases over regular neural
networks for image recognition tasks [5]. Furthermore, this latency increase
is justified for low-power applications in edge computing.

##  7 Effects of Inference Window Size

In an SNN, we can adjust how long the model runs for, that is, the _inference
window_ of the model. In essence, it dictates how many timesteps’ worth of
spikes are sent to the model as input, as well as how many timesteps’ worth of
output spikes the model can make. As common sense would dictate, making the
inference window larger would result in a performance increase at the cost of
increased latency and energy consumption. Conversely, decreasing the inference
window would result in a performance decrease with an associated reduction in
both latency and energy consumption. In this section, we aim to both confirm
that common sense assumption _and_ find an acceptable performance/latency
tradeoff for our downstream SNN models.

Figure 6: Accuracy based on inference window size for the IMDb 2-class
sentiment classification task. The top graph depicts performance at the
sentence-level and the bottom graph depicts performances at the word-level.
Figure 7: Accuracy based on inference window size for the Twitter Emotion
6-class sentiment classification task. The top graph depicts performance at
the sentence-level and the bottom graph depicts performances at the word-
level. Figure 8: Mean Reciprocal Rank (MRR) based on inference window size for
the Twitter Emotion 6-class sentiment classification task. The top graph
depicts performance at the sentence-level and the bottom graph depicts
performances at the word-level.

For our experiments, we vary the inference window between the following
values: 100ms, 50ms (The inference window used in all other results), 25ms,
and 10ms, where each millisecond is counted as a single timestep. Our results
are shown in Fig. 6, 7, and 8. As shown, we see the expected performance
degradation as we reduce the size of the inference window, with performance
dropping off more sharply between 25ms and 10ms. As a result, we recommend
that the inference window for most downstream models stays between 10 and
25ms, as this achieves a performance-latency tradeoff where SNNs remain
competitive with ANNs ($\sim 7$-$10\%$ less accuracy), and latency drops to
$\sim 3.7$-$7\times$ that of equivalent ANNs.

##  8 Conclusions & Future Directions

In this work, we explore the viability of spiking neural networks for
performing natural language processing tasks, describing various encoding
methods and testing various SNNs on two downstream (sentiment/emotion)
classification tasks. Overall, we present the following major contributions:

  * •

At the word-level, we demonstrate that both Poisson rate-coding and directly
using the binary embedding perform similarly.

  * •

By making rate-coding a deterministic process, we see a significant
improvement in accuracy over Poisson rate-coding (roughly $13\%$). This
narrows the performance gap between the SNNs and ANNs to roughly $3.6\%$ at
the word-level, further underscoring SNNs’ viability on NLP tasks.

  * •

We observe the expected energy-accuracy tradeoff between the ANN and the SNN,
as seen in previous SNN literature [2, 5], further grounding our results.

  * •

We successfully reduce the SNNs’ latency to $9.3-17.5x$ that of the ANNs
without significantly affecting performance, and further reduce the latency to
$3.7-7x$ when taking an additional performance hit or roughly $7-10\%$.

These results show that applying SNNs to NLP tasks is not only viable, but it
can be _useful_ , especially in limited-resource environments where more
energy-efficient methods are required. Additionally, we demonstrate that the
SNNs are extremely power-efficient, achieving an approximate $32\times$ and
$60\times$ energy savings during inference and training, respectively.
Furthermore, this energy efficiency is achieved _while_ narrowing the
performance gap between ANNs and SNNs to under 5% accuracy.

###  8.1 Limitations

This work is limited in a few ways, which we aim to address as we move forward
with this direction. The main limitations of this work are as follows:

  * •

We mainly test our SNNs on text classification tasks; more research is needed
to verify the generalizability of SNNs to other, more complex NLP tasks (e.g.,
language translation) and architectures (e.g., Transformer, BERT).

  * •

While for NLP tasks the SNNs result in a vast increase in energy efficiency,
this comes at a slight cost to both model performance and inference latency.
However, this energy-latency trade-off is well-known for SNNs [2, 5].

###  8.2 Future Directions

For further work, we plan to _fully_ leverage the time-series nature of SNNs
and test its ability to process arbitrarily long sequences of text using a
series of embedded data. To that end, we also plan to test SNNs on a wider
variety of tasks to further cement its effectiveness and usefulness, and test
more methods of encoding text into spikes in an energy-efficient manner.
Finally, we plan to incorporate the memory-access-related energy in our energy
model and evaluate the system-level energy improvements of SNNs over ANNs.

###  8.3 Reproducibility

All code, data, and experiments contained in this paper are available on
GitHub at https://github.com/alexknipper/SNNLP.

## References

  * [1] C.-S. Poon and K. Zhou, “Neuromorphic silicon neurons and large-scale neural networks: Challenges and opportunities,” _Frontiers in Neuroscience_ , vol. 5, 2011.
  * [2] J. D. Nunes, M. Carvalho, D. Carneiro, and J. S. Cardoso, “Spiking neural networks: A survey,” _IEEE Access_ , vol. 10, 2022.
  * [3] J. Tissier, C. Gravier, and A. Habrard, “Near-lossless binarization of word embeddings,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, pp. 7104–7111, Jul. 2019. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/4692
  * [4] P. Diehl and M. Cook, “Unsupervised learning of digit recognition using spike-timing-dependent plasticity,” _Frontiers in Computational Neuroscience_ , vol. 9, 2015.
  * [5] A. Sengupta, Y. Ye, R. Wang, C. Liu, and K. Roy, “Going deeper in spiking neural networks: Vgg and residual architectures,” _Frontiers in Neuroscience_ , vol. 13, 2019.
  * [6] S. B. Shrestha and G. Orchard, “SLAYER: spike layer error reassignment in time,” _CoRR_ , vol. abs/1810.08646, 2018.
  * [7] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in Neural Information Processing Systems_ , C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, Eds., vol. 26. Curran Associates, Inc., 2013.
  * [8] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” _preprint arXiv:1301.3781_ , 2013.
  * [9] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _Proceedings of the conference on empirical methods in natural language processing (EMNLP)_ , 2014.
  * [10] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” _Transactions of the Association for Computational Linguistics_ , vol. 5, pp. 135–146, 2017.
  * [11] A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, “Supervised learning of universal sentence representations from natural language inference data,” _CoRR_ , vol. abs/1705.02364, 2017.
  * [12] D. Cer, Y. Yang, S. Kong, and N. Hua, “Universal sentence encoder,” _CoRR_ , vol. abs/1803.11175, 2018.
  * [13] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov, “Fasttext.zip: Compressing text classification models,” _CoRR_ , vol. abs/1612.03651, 2016.
  * [14] Y. Liang, C. K. Ryali, B. Hoover, L. Grinberg, S. Navlakha, M. J. Zaki, and D. Krotov, “Can a fruit fly learn word embeddings?” _CoRR_ , vol. abs/2101.06887, 2021.
  * [15] R. A. Knipper, M. M. Hassan, M. Sadi, and S. K. Karmaker Santu, “Analogy-guided evolutionary pretraining of binary word embeddings,” in _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , Y. He, H. Ji, S. Li, Y. Liu, and C.-H. Chang, Eds. Online only: Association for Computational Linguistics, Nov. 2022, pp. 683–693. [Online]. Available: https://aclanthology.org/2022.aacl-main.52
  * [16] M. Kiselev, “Rate coding vs. temporal coding - is optimum between?” in _2016 International Joint Conference on Neural Networks (IJCNN)_ , 2016, pp. 1355–1359.
  * [17] C. Lee, S. S. Sarwar, P. Panda, G. Srinivasan, and K. Roy, “Enabling spike-based backpropagation for training deep neural network architectures,” _Frontiers in Neuroscience_ , vol. 14, 2020. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fnins.2020.00119
  * [18] Y. Kim, H. Park, A. Moitra, A. Bhattacharjee, Y. Venkatesha, and P. Panda, “Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 71–75.
  * [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _CoRR_ , vol. abs/1706.03762, 2017.
  * [20] B. Yin, F. Corradi, and S. M. Bohté, “Effective and efficient computation with multiple-timescale spiking recurrent neural networks,” in _International Conference on Neuromorphic Systems 2020_ , 2020, pp. 1–8.
  * [21] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Portland, Oregon, USA: Association for Computational Linguistics, Jun. 2011, pp. 142–150.
  * [22] E. Saravia, H.-C. T. Liu, Y.-H. Huang, J. Wu, and Y.-S. Chen, “CARER: Contextualized affect representations for emotion recognition,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 3687–3697.
  * [23] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” _CoRR_ , vol. abs/1810.04805, 2018.
  * [24] Z. Yang, Z. Dai, Y. Yang, J. G. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” _CoRR_ , vol. abs/1906.08237, 2019.
  * [25] “Synopsys.” [Online]. Available: https://www.synopsys.com//

[]R. Alexander Knipper is a PhD student in the Department of Computer Science
and Software Engineering at Auburn University, Auburn, AL. He received a BS
degree from Indiana Institute of Technology in Fort Wayne, IN in 2020. His
current research work is focused on developing conversational systems and
working with Natural Language Processing in Neuromorphic architectures.

[]Kaniz Mishty received the B.S. degree in Electronics and Communication
Engineering from Khulna University of Engineering and Technology, Bangladesh,
in 2018. She is currently working towards her Ph.D. degree in ECE at Auburn
University, AL, USA. Her research interests are energy efficient AI hardware
design and AI/ML in CAD. She interned with Apple Inc. in Summer ’22, 23 and
Qualcomm in ’21 on AI application in SoC and custom circuit design.

[]Mehdi Sadi (S’12-M’17) is an Assistant Professor at the Department of
Electrical and Computer Engineering at Auburn University, Auburn, AL. Dr. Sadi
earned his PhD in ECE from University of Florida, Gainesville, in 2017, MS
from University of California at Riverside, USA in 2011 and BS from Bangladesh
University of Engineering and Technology in 2010. Prior to joining Auburn
University, he was a Senior R&D SoC Design Engineer at Intel Corporation in
Oregon. Dr. Sadi‘s research focus is on developing algorithms/CAD techniques
for implementation, design, reliability, and security of AI hardware. He was
the recipient of SRC best in session award, Intel Xeon Design Group
recognition awards, and National Science Foundation CRII award.

[]Shubhra Kanti Karmaker Santu is currently an Assistant Professor in the
Department of Computer Science and Software Engineering (CSSE) at Auburn
University, Alabama. With a broad interest in the field of Artificial
Intelligence and Data Science, his primary research focus lies at the
intersection of Natural Language Processing (NLP) and Information Retrieval
(IR). He brings significant experience in NLP/IR research from both academia
(Ph.D. from UIUC, Postdoc experience from MIT) and industry internships
(Microsoft Research, Yahoo Research, @WalmartLabs). As a researcher, he has
published 28 research papers at premier venues, including ACL, EMNLP, SIGIR,
WWW, COLING, CoNLL, AACL, CIKM, IUI, ACM TIST, and ACM Computing Surveys. His
research projects have been funded by multiple agencies, including the
National Science Foundation (NSF), Air Force Office of Scientific Research
(AFOSR), Army Research Office (ARO) and US Dept. of Agriculture (USDA). For
external service, he currently serves as an action editor for the ACL Rolling
Review Initiative (ARR) and as ARR’s communication chair. Dr. Karmaker served
as the tutorial chair for CIKM 2022. He has also served regularly as a program
committee member for ACL and SIGIR-sponsored conferences for the last five
years.


# How not to Lie with a Benchmark:
Rearranging NLP Leaderboards

Tatiana Shavrina
Sberbank of Russia, SberDevices
Moscow, Russia
AI Research Institute (AIRI)
Moscow, Russia
<EMAIL_ADDRESS>
Valentin Malykh
Huawei Noah’s Ark lab
Moscow, Russia
Kazan Federal University
Kazan, Russia
<EMAIL_ADDRESS>

###### Abstract

Comparison with a human is an essential requirement for a benchmark for it to
be a reliable measurement of model capabilities. Nevertheless, the methods for
model comparison could have a fundamental flaw - the arithmetic mean of
separate metrics is used for all tasks of different complexity, different size
of test and training sets.

In this paper, we examine popular NLP benchmarks’ overall scoring methods and
rearrange the models by geometric and harmonic mean (appropriate for averaging
rates) according to their reported results. We analyze several popular
benchmarks including GLUE, SuperGLUE, XGLUE, and XTREME. The analysis shows
that e.g. human level on SuperGLUE is still not reached, and there is still
room for improvement for the current models.

##  1 Introduction

The benchmarking approach has a rich history throughout computer science and
is now the leading method in machine learning progress validation. In the
field of Natural Language Processing (NLP), there exist at least 898
benchmarks111according to https://paperswithcode.com/area/natural-language-
processing, the most prominent being GLUE, SuperGLUE, XGLUE, etc., created
within a single paradigm.

The SOTA-centricity of the language model benchmarking has been since
criticized as misleading 222https://hackingsemantics.xyz/2019/leaderboards/:
the battle for percentage fractions leads to extensive improvements – an
increase in the data volume and number of parameters of models, – rather than
an intensive improvement in their architectures. The existing solutions
suggest Pareto efficiency between the estimated accuracy and computational
costs (Dodge et al., 2019) and draw attention to the question of model utility
versus overall score (Ethayarajh and Jurafsky, 2021). The recent work
(Dehghani et al., 2021) has also stated that many factors, other than
fundamental algorithmic superiority, can lead to a method being evaluated as
superior, for example, various subsamples of the benchmark tasks’ result in
different leaderboard arrangements. This forms the conception of the
"benchmark lottery", describing the fragility of the main model evaluation
instruments.

We state that the delicate question of a general model assessment should not
be solved by a simplistic yet incorrect method – the arithmetic mean over all
the results. Stated method does not take into account the scatter of results
on various tasks, the differing size of the task test sets (e.g. in SuperGLUE
they differ a hundred times, compare 146 test samples in Winograd Schema and
10’000 test samples in ReCoRd(Wang et al., 2019a)), different susceptibility
to leaks (Elangovan et al., 2021), including the year of creation (Recognizing
Textual Entailment data was collected in 2005 (Dagan et al., 2005), while
BoolQ or CommitmentBank data was collected in 2019(Clark et al., 2019), (De
Marneffe et al., 2019)).

In this article we present an analysis of the NLP benchmarks’ results, using
not the arithmetic mean, but the other metrics: geometric mean and harmonic
mean. As F1 (harmonic mean) is frequently used to normalize Precision and
Recall as they are fractions, optimizing the classifier threshold to maximize
F1 leads to a better balance between metrics than the arithmetic mean because
it penalizes systems more for the smaller values(Sasaki et al., 2007). The
geometric mean, as noted in the (Fleming and Wallace, 1986), is the preferred
metric to the arithmetic mean in computing performance benchmarks when it
comes to normalized values and percentages.

The results change the usual idea of the models’ order on leaderboards: humans
still occupy the first place in the intellectual task solving, and the best
results (1.5-2% worse than humans) belong to DeBerta(He et al., 2020),
T5+Meena (Raffel et al., 2020) and McAlbert+DKM
models333https://www.iflytek.com/news/2118. Thus, the contribution of this
paper is two-fold:

  1. 1.

we present the reviewed model evaluation technique, core for benchmark design,

  2. 2.

we re-arrange the currently existing leaderboards of most popular benchmarks.

##  2 Previous Work

Evaluation and comparison of NLP models beget a rich history, rising with the
Turing test (Turing, 2009). The next step in the development and assessment of
intelligent systems belongs to the ML-benchmark methodology, which aims to
bring the solution of the Natural Language Understanding problem closer -
General Language Understanding Evaluation(Wang et al., 2019b).

As stated in (Wang et al., 2019a), "Lacking a fair criterion with which to
weight the contributions of each task to the overall score, we opt for the
simple approach of weighing each task equally, and for tasks with multiple
metrics, first averaging those metrics to get a task score." All the GLUE-
based benchmarks follow this methodology.

However, apart from the GLUE format, other benchmarks have provided several
alternatives to evaluate the overall model contribution. KILT, a Benchmark for
Knowledge Intensive Language Tasks (Petroni et al., 2020), avoids calculating
the overall result, and also do not compare the result with the human level,
but only provides metrics for individual tasks. DecaNLP (McCann et al., 2018)
makes a rating using not the average, but the sum of points for all tasks.
This approach allows balancing the contributions of different tasks to the
overall metric.

##  3 Method

We arrange the NLP benchmark results using the publicly available model scores
for all the tasks to calculate new overall scores.

Other available options from Pythagorean means - harmonic mean and geometric
mean - can also be considered: we have centered our research around 2 simple
statistics that are widely used for averaging fractions (lun Chou, 1969) or
normalized values (Fleming and Wallace, 1986) among the possible alternatives.
We did not consider other measures of central tendency, like median and mode,
as the averaged samples more often consist of about 10 measurements, and on
them such metrics can give the same results for competing systems.

  * •

The arithmetic mean (AM) desribed in eq. 1 is calculated as the sum of the
task scores (Xs) divided by the total number of tasks, referred to as N.

  * •

The geometric mean (GM) desribed in eq. 2 is calculated as the N-th root of
the product of all task scores (with the above conditions), where N is the
number of values.

  * •

The harmonic mean (HM) desribed in eq. 3 is calculated as the number of values
N divided by the sum of the reciprocal of the values

${\displaystyle
AM={\frac{1}{n}}\sum_{i=1}^{n}x_{i}={\frac{x_{1}+x_{2}+\cdots+x_{n}}{n}}}$ (1)
${\displaystyle
GM=\left(\prod_{i=1}^{n}x_{i}\right)^{\frac{1}{n}}={\sqrt[n]{x_{1}x_{2}\cdots
x_{n}}}}$ (2)

${\displaystyle
HM=\frac{N}{{\frac{1}{x_{1}}}+{\frac{1}{x_{2}}}+\cdots+{\frac{1}{x_{N}}}}}$
(3)

The sections below present the results of the leaderboard re-weighting as of
May 2021.

###  3.1 Reevaluating the Benchmarks

The GLUE (11 tasks for English), SuperGLUE (10 tasks for English), XGLUE (11
tasks for 19 languages), and XTREME (4 tasks for 40 languages) provide
different scoring metrics for each task, including Accuracy, F1, Matthew’s
correlation coefficient, Exact Match, while the overall score is calculated by
their simple average. In cases like these, the geometric mean is appropriate
when the data contains values with different units of measure (lun Chou,
1969).

The harmonic mean of the task results as a better overall metric has the same
grounding as introduction of the F-1 measure over precision and recall (Sasaki
et al., 2007): the harmonic mean is more intuitive than the arithmetic mean
when computing a mean of ratios. Given the set of metrics with a large
scatter, the harmonic mean will be less than the arithmetic mean, penalizing
the system more for the errors made.

The harmonic mean is the appropriate mean if the data is comprised of rates,
while the geometric mean is used as an unbiased estimation when working with
normalized ratios, for example, in finance (Dittmann and Maug, 2008) or
computing benchmarks (Fleming and Wallace, 1986).

However, their applicability to a better summarization of the model
performance to a single number has been widely discussed, see (Smith, 1988),
discussing performance computing:

  * •

the harmonic mean is considered the appropriate metric to summarize benchmark
results expressed as rates,

  * •

while geometric mean is applicable in case of the use of performance numbers
that are normalized with respect to one of the results being compared (see 4),

  * •

and arithmetic mean should not be used as a summarizing metric with rates,
making it the worst choice for results accumulation.

###  3.2 GLUE

GLUE benchmark (Wang et al., 2019b) combines 11 tasks in various text
classification and question answering.

Overall score: average of all the task results. If task has 2 main metrics,
these metrics are averaged, then added to the overall average.

Human evaluation: collected on reported human performance numbers from
original datasets, not exceeding 200 examples (heavily criticised in (Nangia
and Bowman, 2019)). The human baseline performance on the diagnostic set was
provided by the project authors with the help of six NLP researchers
annotating 50 randomly selected sentence pairs.

Rearranging the scores: the results of geometric and harmonic mean
rearrangement are presented in Tab. 1. GLUE benchmark seem to be the most
reordered of all the ratings considered: the best result by geometric and
harmonic means belongs to humans, DeBerta and McAlbert+DKM got a 1 point
demotion, and the other models got severely rearranged their places.

N | Name | AM | HM | GM | CoLA | SST-2 |  | MRPC
---
Mean
| STS-B
---
Mean
| QQP
---
Mean
| MNLI
---
m
| MNLI
---
mm
QNLI
16 | Human | 87.10 | 86.16 | 86.91 | 66.40 | 97.80 | 83.55 | 92.65 | 69.95 | 92.00 | 92.80 | 91.20
1 | DeBERTa | 90.80 | 84.78 | 86.25 | 71.50 | 97.50 | 93.00 | 92.75 | 83.50 | 91.90 | 91.60 | 99,20
2 |  | Mac
---
Albert
+DKM
90.70 | 84.70 | 86.13 | 74.80 | 97.00 | 93.55 | 92.70 | 82,65 | 91.30 | 91.10 | 97.80
6 | T5 | 90.30 | 84.48 | 85.92 | 71.60 | 97.50 | 91.60 | 92.95 | 82.85 | 92.20 | 91.90 | 96.90
4 | PING-AN | 90.60 | 84.26 | 85.83 | 73.50 | 97.20 | 93.00 | 92.70 | 83.55 | 91.60 | 91.30 | 97.50
5 | ERNIE | 90,40 | 84.27 | 85.75 | 74,40 | 97.50 | 92.45 | 92.80 | 83.05 | 91.40 | 91.00 | 96.60
Table 1: Top results of ranking GLUE benchmark with geometric mean. N –
original model rank on the leaderboard. MNLI m and MNLI mm correspond to
MultiNLI Matched & MultiNLI Mismatched, other task abbreviations correspond to
their GLUE leaderboard designations accordingly.

###  3.3 SuperGLUE

SuperGLUE (Wang et al., 2019a) is the sophisticated version of the GLUE
benchmark, combining 10 tasks with a higher demand for higher intellectual
abilities. Task data must is available under various licenses that allow use
and redistribution for research purposes.

Overall score: average of all the task results. If task has 2 main metrics,
these metrics are averaged, then added to the overall average.

Human evaluation: ready-made estimates for WiC, MultiRC, RTE, and ReCoRD
datasets, the other tasks being evaluated by the project creators with the
help of crowdworker annotators through Amazon’s Mechanical Turk.

Rearranging the scores: Re-weighting the results using the geometric mean and
harmonic mean again makes significant changes to the original ranking: the
top-3 result (human) is ranked top-1, the DeBerta and T5 models are shifted
down 1 position, PAI ALbert and Nezha Plus models swap their places, see Tab.
2.

N | Model | AM | HM | GM | BoolQ |  | CB
---
Mean
COPA |  | MRC
---
| RCD
---
RTE | WiC | WSC | AX-b |  | AX-g
---
mean
3 | Human | 89,80 | 87,96 | 88,73 | 89,00 | 97,35 | 100,00 | 66,85 | 91,50 | 93,60 | 80,00 | 100,00 | 76,6 | 99,5
1 | DeBERTa | 90,30 | 86,89 | 87,60 | 90,40 | 96,65 | 98,40 | 75,95 | 94,30 | 93,20 | 77,50 | 95,90 | 66,7 | 93,55
2 |  | T5+
---
Meena
90,20 | 86,42 | 87,10 | 91,30 | 96,70 | 97,40 | 75,65 | 93,85 | 92,70 | 77,90 | 95,90 | 66,5 | 89,35
4 | T5 | 89,30 | 85,89 | 86,57 | 91,20 | 95,35 | 94,80 | 75,70 | 93,75 | 92,50 | 76,90 | 93,80 | 65,6 | 92,3
6 |  | PAI
---
Albert
86,10 | 85,24 | 85,78 | 88,10 | 94,40 | 91,80 | 69,65 | 88,65 | 88,80 | 74,10 | 93,20 | 75,6 | 98,75
5 |  | Nezha
---
plus
86,70 | 81,30 | 82,29 | 87,80 | 95,20 | 93,60 | 69,85 | 89,85 | 89,10 | 74,60 | 93,20 | 58,00 | 80,75
Table 2: Top results of ranking SuperGLUE benchmark with geometric mean. N –
original model rank on the leaderboard MRC stands for MultiRC averaged metric,
RCD - ReCoRD averaged metric.

###  3.4 XTREME

The XTREME benchmark (Hu et al., 2020)covers 40 typologically diverse
languages from 12 language families and includes 9 tasks that require analysis
of different levels of syntax or semantics.

Overall score: 2-step averaging: 1) calculating average for each task on all
languages 2) calculating average on all tasks.

Human evaluation: 2-step averaging:

  1. 1.

step 1: ready-made estimates from the original datasets taken and extrapolated
to all unestimated languages; besides, for some datasets there were no
original estimates provided (POS) and an empirical estimate of 97% was taken
based on (Manning, 2011); no estimates for NER and sentence retrieval tasks;

  2. 2.

step 2: all the task results averaged together.

Rearranging the scores: the results of applying the geometric mean and
harmonic mean did not change the current ranking of the models - the quality
spread between them is high enough for the metrics averaging them to retain
the current order.

N | Model | AM | HM | GM |  | Sentence-pair
---
Classification
| Structured
---
Prediction
| Question
---
Answering
| Sentence
---
Retrieval
1 | Human | 93,30 | 93,13 | 93,21 | 95,10 | 97,00 | 87,80 | 0.00001
2 | VECO | 81,10 | 81,27 | 81,70 | 88,60 | 75,40 | 72,40 | 92,10
3 | ERNIE-M | 80,90 | 81,11 | 81,52 | 87,90 | 75,60 | 72,30 | 91,90
4 | T-ULRv2 | 80,70 | 80,91 | 81,25 | 88,80 | 75,40 | 72,90 | 89,30
5 | Anonymous3 | 79,90 | 80,12 | 80,50 | 88,20 | 74,60 | 71,70 | 89,00
6 | Polyglot | 77,80 | 78,02 | 78,56 | 87,80 | 72,90 | 67,40 | 88,30
Table 3: Top results of ranking XTREME benchmark with geometric mean. N –
original model rank on the leaderboard; the averaged task scores are shown by
the column markings.

###  3.5 XGLUE

The XGLUE benchmark(Liang et al., 2020)consists of 11 problems in 19 languages
and evaluates the performance of multilingual pre-trained systems in terms of
their ability to cross-language understanding and natural language generation.

Overall score: 2-step averaging: 1) calculating average for each task on all
languages 2) calculating average on all tasks.

Human evaluation: not provided.

Rearranging the scores: Since the human level is not measured in the
benchmark, we can only compare the 2 present models with each other. Tab. 4
shows the results - the difference in the quality of the models is large
enough to preserve their ranking on all averaging metrics.

N | Model | AM | HM | GM | NER | POS | NC | MLQA | XNLI | PAWS-X | QADSM | WPR | QAM
---|---|---|---|---|---|---|---|---|---|---|---|---|---
1 | FILTER | 80.10 | 79.61 | 79.86 | 82.60 | 81.60 | 83.50 | 76.20 | 83.90 | 93.80 | 71.40 | 74.70 | 73.40
2 |  | Unicoder
---
Baseline
76.10 | 75.45 | 75.80 | 79.70 | 79.60 | 83.50 | 66.00 | 75.30 | 90.10 | 68.40 | 73.90 | 68.90
N | Model | AM | HM | GM | QG | NTG |  |  |  |  |  |  |
1 |  | Unicoder
---
Baseline
10.70 | 10.65 | 9.10 | 10.60 | 10.70 |  |  |  |  |  |  |
2 | MP-Tune | 8.70 | 8.70 | 7.10 | 8.10 | 9.40 |  |  |  |  |  |  |
Table 4: Top results of ranking XGLUE benchmark with geometric mean. N –
original model rank on the leaderboard; the first 2 rows correspond to NLU
tasks, the last 2 rows - to the NLG tasks.

##  4 Results and Discussion

The results show that the ranking of results within a single leaderboard can
fluctuate significantly. So, in GLUE, the first place in terms of the harmonic
and geometric mean belongs to the result occupying the 16th line in the
arithmetic mean. In SuperGLUE, the permutation is not so striking - the third
result is on the 1st place. On the XTREME and XGLUE benchmarks system ranking
is preserved.

Since all three averaging metrics considered are subject to different biases,
we present the statistical measurements of the top-3 SuperGLUE results in Tab.
5. Human results have the highest total points for all tasks (as in the
DecaNLP methodology), while the standard deviation and variance are greater
than top-2 and top-3 models.

Model | AM | GM | HM | Sum | Var | Std
---|---|---|---|---|---|---
Human | 89,8 | 88,73 | 87,96 | 894,40 | 130,31 | 11,42
DeBerta | 90,3 | 87,60 | 86,89 | 882,55 | 117,46 | 10,84
T5 + Meena | 90,2 | 87,10 | 86,42 | 877,25 | 112,39 | 10,60
Table 5: Measuring the statistics of the top-3 SuperGLUE results. Sum is a sum
of all the task scores, Var and Std are variance and standard deviation on the
task scores respectively. Notable results are highlighted in bold.

The following topics remain debatable and need special attention of the
community:

  1. 1.

Different metrics for obtaining the average value (arithmetic, geometric,
harmonic) have different restrictions on the accepted values (for example, not
every one can take negative or zero values). At the same time, metrics that
take zero and negative values are actively used in measuring various skills -
MCC metric on SuperGLUE diagnostics can be negative, other metrics can be
equal to or greater than zero, and they are averaged altogether.

  2. 2.

Correct averaging of the overall score for multilingual benchmarks creates
additional problems while performing the averaging operation in 2 stages: for
all languages and all tasks.

  3. 3.

In addition to the problem of the main averaging metric, we left outside of
the scope the problem that was also discovered within the framework of this
study: human benchmark scores on various tasks were obtained in a very
different way.

##  5 Conclusion

In this paper we present an alternative method to arrange the popular NLP
benchmark results, elaborating on several task evaluation. We analyze popular
benchmark averaging methods and provide new insight into model comparison.
Namely, we obtain the following results:

  * •

for popular benchmarks GLUE and SuperGLUE we can conclude that their overall
score is subject to bias due to outliers; the alternative arrengement methods
end with significantly different ordering of the results;

  * •

rebuilding leaderboards using other metrics (geometric or harmonic mean)
allows one to conclude that human result is the first in the rankings;

  * •

in XGLUE leaderboard human result is obtained by extrapolation from one
language to others, while in practice the level of problem-solving by native
speakers of different languages varies;

  * •

the last finding could be extended to other multilingual benchmarks also.

We believe that an unbiased generalization of the benchmark scores is a
necessity for the community to target language modelling. The tracking of the
complex process of model improvement over time can require the improvement of
the benchmark design itself.

The expansion towards multilingualism and multimodality of the new benchmark
practices plus new models makes the stated problem more urgent and we hope our
help to foster research in this direction.

## References

  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  _arXiv preprint arXiv:1905.10044_.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The pascal recognising textual entailment challenge.  In _Machine Learning Challenges Workshop_ , pages 177–190. Springer.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _proceedings of Sinn und Bedeutung_ , volume 23, pages 107–124.
  * Dehghani et al. (2021) Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. 2021.  The benchmark lottery.
  * Dittmann and Maug (2008) Ingolf Dittmann and Ernst Maug. 2008.  Biases and error measures: How to compare valuation methods.
  * Dodge et al. (2019) Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019\.  Show your work: Improved reporting of experimental results.
  * Elangovan et al. (2021) Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021.  Memorization vs. generalization: quantifying data leakage in nlp performance evaluation.  _arXiv preprint arXiv:2102.01818_.
  * Ethayarajh and Jurafsky (2021) Kawin Ethayarajh and Dan Jurafsky. 2021.  Utility is in the eye of the user: A critique of nlp leaderboards.
  * Fleming and Wallace (1986) Philip J. Fleming and John J. Wallace. 1986.  How not to lie with statistics: The correct way to summarize benchmark results.  _Commun. ACM_ , 29(3):218–221.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _International Conference on Machine Learning_ , pages 4411–4421. PMLR.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020.  Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation.  _arXiv preprint arXiv:2004.01401_.
  * lun Chou (1969) Ya lun Chou. 1969.  Statistical analysis: With business and economic applications.
  * Manning (2011) Christopher D Manning. 2011.  Part-of-speech tagging from 97% to 100%: is it time for some linguistics?  In _International conference on intelligent text processing and computational linguistics_ , pages 171–189. Springer.
  * McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018.  The natural language decathlon: Multitask learning as question answering.  _arXiv preprint arXiv:1806.08730_.
  * Nangia and Bowman (2019) Nikita Nangia and Samuel R Bowman. 2019.  Human vs. muppet: A conservative estimate of human performance on the glue benchmark.  _arXiv preprint arXiv:1905.10425_.
  * Petroni et al. (2020) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2020.  Kilt: a benchmark for knowledge intensive language tasks.  _arXiv preprint arXiv:2009.02252_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Sasaki et al. (2007) Yutaka Sasaki et al. 2007.  The truth of the f-measure. 2007.
  * Smith (1988) James E. Smith. 1988.  Characterizing computer performance with a single number.  _Communications of the ACM_ , 31(10):1202–1206.
  * Turing (2009) Alan M Turing. 2009.  Computing machinery and intelligence.  In _Parsing the turing test_ , pages 23–65. Springer.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019b.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019_.

_

## Appendix

See tables below

###  5.1 Appendix 1

Table 6: SuperGLUE benchmark results normalized to human level: geometric mean is the main metric Rank |  | SuperGLUE
---
Model
| HM
---
| GM
---
| AM
---
BoolQ |  | CB
---
mean
COPA |  | MultiRC
---
Mean
| ReCoRD
---
mean
RTE | WiC | WSC | AX-b |  | AX-g
---
mean
3 | Human | 87,9 (1) | 88,7 (1) | 89,8 (1) | 89 (1) | 97,35 (1) | 100 (1) | 66,85 (1) | 91,5 (1) | 93,6 (1) | 80 (1) | 100 (1) | 76,6 (1) | 99,5 (1) |
1 | DeBERTa | 0,985 | 0,987 | 0,989 | 1,016 | 0,993 | 0,984 | 1,136 | 1,031 | 0,996 | 0,969 | 0,959 | 0,871 | 0,940 |
2 | T5 + Meena | 0,979 | 0,982 | 0,984 | 1,026 | 0,993 | 0,974 | 1,132 | 1,026 | 0,990 | 0,974 | 0,959 | 0,868 | 0,898 |
4 | T5 | 0,973 | 0,976 | 0,978 | 1,025 | 0,979 | 0,948 | 1,132 | 1,025 | 0,988 | 0,961 | 0,938 | 0,856 | 0,928 |
Rank |  | GLUE
---
Model
| HM
---
| GM
---
| AM
---
CoLA | SST-2 |  | MRPC
---
Mean
| STS-B
---
Mean
| QQP
---
Mean
MNLI-m | MNLI-mm | QNLI | RTE | WNLI | AX
16 | Human | 86,1 (1) | 86,9 (1) | 87,1 (1) | 66,4 (1) | 97,8 (1) | 83,55 (1) | 92,65 (1) | 69,95 (1) | 92 (1) | 92,8 (1) | 91,2 (1) | 93,6 (1) | 95,9 (1) | 76,6 (1)
1 | DeBERTa | 0,995 | 1,004 | 1,012 | 1,077 | 0,997 | 1,113 | 1,001 | 1,194 | 0,999 | 0,987 | 1,088 | 0,996 | 0,985 | 0,695
2 | McAlbert | 0,993 | 1,002 | 1,011 | 1,127 | 0,992 | 1,120 | 1,001 | 1,182 | 0,992 | 0,982 | 1,072 | 0,983 | 0,985 | 0,687
6 | T5 | 0,991 | 1,000 | 1,008 | 1,078 | 0,997 | 1,096 | 1,003 | 1,184 | 1,002 | 0,990 | 1,063 | 0,991 | 0,985 | 0,693

###  5.2 Appendix 2

Table 7: GLUE results with all metrics for the tasks, including tasks with double metrics.  N | Model |  | GM
---
| AM
---
CoLA | SST-2 |  | MRPC
---
F1
| MRPC
---
Acc
| MRPC
---
Mean
| STS-B
---
Prs
| STS-B
---
Spr
| STS-B
---
Mean
| QQP
---
F1
| QQP
---
Acc
| QQP
---
Mean
| MNLI
---
m
| MNLI
---
mm
QNLI | RTE | WNLI | AX
16 | Human | 86,906 | 87,1 | 66,4 | 97,8 | 86,3 | 80,8 | 83,55 | 92,7 | 92,6 | 92,65 | 59,5 | 80,4 | 69,95 | 92 | 92,8 | 91,2 | 93,6 | 95,9 | -
1 | DeBERTa | 86,248 | 90,8 | 71,5 | 97,5 | 94 | 92 | 93 | 92,9 | 92,6 | 92,75 | 76,2 | 90,8 | 83,5 | 91,9 | 91,6 | 99,2 | 93,2 | 94,5 | 53,2
2 |  | HFL
---
iFLYTEK
86,127 | 90,7 | 74,8 | 97 | 94,5 | 92,6 | 93,55 | 92,8 | 92,6 | 92,7 | 74,7 | 90,6 | 82,65 | 91,3 | 91,1 | 97,8 | 92 | 94,5 | 52,6
6 | T5 | 85,915 | 90,3 | 71,6 | 97,5 | 92,8 | 90,4 | 91,6 | 93,1 | 92,8 | 92,95 | 75,1 | 90,6 | 82,85 | 92,2 | 91,9 | 96,9 | 92,8 | 94,5 | 53,1
4 | PING-AN | 85,827 | 90,6 | 73,5 | 97,2 | 94 | 92 | 93 | 93 | 92,4 | 92,7 | 76,1 | 91 | 83,55 | 91,6 | 91,3 | 97,5 | 91,7 | 94,5 | 51,2
5 | ERNIE | 85,754 | 90,4 | 74,4 | 97,5 | 93,5 | 91,4 | 92,45 | 93 | 92,6 | 92,8 | 75,2 | 90,9 | 83,05 | 91,4 | 91 | 96,6 | 90,9 | 94,5 | 51,7

###  5.3 Appendix 3

Table 8: SuperGLUE results with all metrics for the tasks, including tasks with double metrics.  Rank | Model |  | GM
---
| AM
---
BoolQ |  | CB
---
F1
| CB
---
Acc
| CB
---
mean
COPA |  | Multi
---
RC F1
| Multi
---
RC Acc
| Multi
---
RC Mean
| ReCoRD
---
F1
| ReCoRD
---
Acc
| ReCoRD
---
mean
RTE | WiC | WSC
3 | Human | 88,729 | 89,8 | 89 | 95,8 | 98,9 | 97,35 | 100 | 81,8 | 51,9 | 66,85 | 91,7 | 91,3 | 91,5 | 93,6 | 80 | 100
1 | DeBERTa | 87,601 | 90,3 | 90,4 | 95,7 | 97,6 | 96,65 | 98,4 | 88,2 | 63,7 | 75,95 | 94,5 | 94,1 | 94,3 | 93,2 | 77,5 | 95,9
2 | T5 + Meena | 87,097 | 90,2 | 91,3 | 95,8 | 97,6 | 96,7 | 97,4 | 88,3 | 63 | 75,65 | 94,2 | 93,5 | 93,85 | 92,7 | 77,9 | 95,9
4 | T5 | 86,567 | 89,3 | 91,2 | 93,9 | 96,8 | 95,35 | 94,8 | 88,1 | 63,3 | 75,7 | 94,1 | 93,4 | 93,75 | 92,5 | 76,9 | 93,8
6 | PAI Albert | 85,784 | 86,1 | 88,1 | 92,4 | 96,4 | 94,4 | 91,8 | 84,6 | 54,7 | 69,65 | 89 | 88,3 | 88,65 | 88,8 | 74,1 | 93,2
5 |  | NEZHA
---
Plus
82,294 | 86,7 | 87,8 | 94,4 | 96 | 95,2 | 93,6 | 84,6 | 55,1 | 69,85 | 90,1 | 89,6 | 89,85 | 89,1 | 74,6 | 93,2