#  Sarcasm Detection Framework Using Emotion and Sentiment Features

Oxana Vitman, Yevhen Kostiuk, Grigori Sidorov, Alexander Gelbukh

(October 2022)

###### Abstract

Sarcasm detection is an essential task that can help identify the actual
sentiment in user-generated data, such as discussion forums or tweets. Sarcasm
is a sophisticated form of linguistic expression because its surface meaning
usually contradicts its inner, deeper meaning. Such incongruity is the
essential component of sarcasm, however, it makes sarcasm detection quite a
challenging task. In this paper, we propose a model, which incorporates
emotion and sentiment features to capture the incongruity intrinsic to
sarcasm. We use CNN and pre-trained Transformer to capture context features.
Our approach outperformed previous state-of-the-art results on four datasets
from social networking platforms and online media.

##  1 Introduction

The world’s use of online communications is expanding quickly as social media
has become the most important source of news and public opinion about almost
every daily topic. The most popular application of social media analysis is
detecting consumer sentiment to help businesses and online merchants to
address the needs of their clients, including handling and resolving
complaints.

However, not always emotions or sentiment are expressed directly. Frequently
social media users tend to make their posts or messages sarcastic in order to
have better response from the other users and stimulate the virality of social
media content. Moreover, negative sarcastic tweets attract substantially
higher social media responses when compared to actual negative tweets [15].
Thus, sarcasm identification in online communications, discussion forums, and
e-commerce websites have become essential for fake news detection, sentiment
analysis, opinion mining, and hate speech detection. However, sarcasm
detection appears to be quite challenging task due to it’s sophisticated
nature.

The surface meaning of sarcasm frequently contrasts with the underlying deeper
meaning, making it a particularly complex kind of verbal expression. Although,
this incongruity is the essential component of sarcasm, the intention may also
be to appear humorous, make fun of someone, or show contempt. As a result,
sarcasm is seen as an extremely sophisticated and intelligent language
construct that presents a number of difficulties for the perception of
emotions.

The following example of a tweet illustrates the above mentioned nuances: _“On
our 6 am walk, my daughter asked me where the moon goes each morning. I let
her know it’s in heaven, visiting daddy’s freedom”_.

On the surface, this statement seems to indicate that the speaker is enjoying
his morning walk with his daughter and telling her amusing stories. However,
close examination of the speaker’s emotion and sentiment reveals that the
speaker is unhappy and experiencing some unpleasant emotions at the time of
speaking.

This is where incongruity between sentiment and emotions plays an important
role. Sentiment, emotion and sarcasm are highly interconnected, and one helps
in the understanding of the others. We propose a model, which utilizes
sentiment and emotion detection and learns their dependencies related to
sarcasm. We hypothesized that learning a pattern of contradiction between
surface sentiment and intended sentiment is a key component in sarcasm
detection.

##  2 Related Work

While sarcasm detection is still a relatively new field, it has recently drawn
more attention from researchers due to the rapid growth of social media and
the need for sentiment analysis therein. A number of approaches have been
developed and studied in this regard. In this section, we are going to pay
attention to works implementing Transformers, Neural Networks or focusing on a
relation between sentiment, emotion and sarcasm.

The research [19] proposed a model based on BERT fine-tuned on related
intermediate tasks, such as sentiment classification and emotion detection,
before fine-tuning it on the target task. Experimental results on three
datasets that have different characteristics showed that the intermediate task
transfer learning model outperforms many previous models.

The study [3] proposed two novel deep neural network models for sarcasm
detection. Models add to the BERT architecture two components, namely
affective feature embedding and contextual feature embedding. The two models
are different in the way the two components are combined and the input to the
affective feature embedding component.

The paper [8] proposed a neural network semantic model composed of Convolution
Neural Network (CNN), followed by a Long Short Term Memory (LSTM) network and
a Deep Neural Network (DNN). The proposed model outperforms Support Vector
Machine applied to the same dataset.

In the work [2], the authors focus on detecting sarcasm in textual
conversation from social media websites and online forums. They developed a
model consisting of a multi-headed self-attention module and gated recurrent
units. Multi-head self-attention module helps in identifying important
sarcastic keywords from the input, and the recurrent units learn long-range
dependencies between these keywords to classify the input text more
accurately. Proposed approach achieved state-of-the-art results on multiple
datasets from social media platforms and online networks.

In the paper [5], authors have also hypothesized that sarcasm is closely
related to sentiment and emotion. They proposed two attention mechanisms,
where the first segment learns the relationship between the different parts of
the sentence, and the second segment focuses within the same part of the
sentence across the modalities. Finally, representations from both attentions
are merged for multi-task classification task. The research proved that
emotions and sentiment help to improve the effect of satire detection,
however, the experiments were performed on MUStArd dataset [4], which consists
of audiovisual utterances accompanied by its context, thus providing
additional cues for sarcasm detection task.

##  3 Description of Datasets

We conducted experiments on four benchmark datasets: two Reddit [11]
subreddits datasets: SARC/movies and SARC/technology, a subset of Internet
Argument Corpus-V2 [20] (IAC-V2), and Twitter [18]. All of them have been
widely used in evaluating sarcasm detection. The details are shown in Table 1.

Table 1: Statistics of the experimental data. | Train | Test
---|---|---
Datasets | Sarcastic |  Non-sarcastic | Sarcastic |  Non-sarcastic
SARC/movies | 2,533 |  2,707 | 641 |  669
SARC/technology | 2,738 |  1,815 | 677 |  462
IAC_V2 | 2,616 |  2,600 | 644 |  660
Twitter (Ptacek et al., 2014) | 22,323 |  25,785 | 5,648 |  6,379

###  3.1 Reddit

Khodak et al. [11] collected SARC, a corpus comprising of 600,000 sarcastic
comments on Reddit. We use two subreddits: SARC/movies and SARC/technology for
sarcasm detection. The subreddits contain news and discussions concerning
films and development, application and related issues of technology,
respectively.

###  3.2 Internet Argument Corpus

Internet Argument Corpus (IAC) [20] is collected from online political debates
forum. IAC-V2 [1] is the subset of the Internet Argument Corpus. IAC-V2
divides sarcasm into three sub-types, (i.e., general sarcasm, hyperbole, and
rhetorical questions). We use the largest subset (general sarcasm) in our
experiments.

###  3.3 Twitter

We use Twitter dataset provided by Tomáš Ptáček [18], who collected a self-
annotated corpus of tweets with the _#sarcasm_ hashtag. We used English
balanced version.

##  4 Proposed Method

###  4.1 Overview of the Proposed Approach

The model consists of four blocks: Sarcasm Pre-Trained Transformer (SarcPTT),
Emotion Detection Pre-Trained Transformer (EmoDPTT), Sentiment Detection Pre-
Trained Transformer (SentDPTT), and CNN block. EmoDPTT and SentDPTT are used
as feature extractors and are not trainable during model fitting. Other
modules (CNN and SarcPTT) are trainable. The detailed overview of the blocks
is presented in the following subsections.

The general flow is presented in Figure 1. The input text is tokenized via
transformer tokenizer $\\{CLS,T_{1},...,SEP\\}$ and is passed through SarcPTT.
The output of this step is a last hidden state, but only vector representation
$V_{CLS}$ of the first token ($CLS$) is used for further processing. After
that, the output is passed through Fully Connected Network and new vector
$V_{l}$ is obtained.

The non-trainable blocks (EmoDPTT and SentDPTT) are used as feature
extractors. After passing the input text tokenized via transformer tokenizers
($\\{CLS_{e},T_{1}^{e},...,SEP_{e}\\}$ and
$\\{CLS_{s},T_{1}^{s},...,SEP_{s}\\}$ respectively) through the models, the
following features are obtained:

  * •

EmoDPTT vector representation of the first token of the last hidden state:
$U_{CLS}$;

  * •

Probability distribution of dimension $28$ of EmoDPTT labels, e.g., amusement,
relief, disgust, neutral, etc.: $\\{EL_{1},EL_{2},...,EL_{28}\\}$;

  * •

SentDPTT vector representation of the first token of last hidden state :
$S_{CLS}$;

  * •

Probability distribution of dimension $2$ of SentDPTT labels, i.e., positive
and negative sentiments $\\{SL_{1},SL_{2}\\}$.

Similarly to the SarcPTT output, text representations were passed through
Fully Connected Networks, and new vectors were obtained: $S_{l}$ and $U_{l}$.
Probability distributions of labels were concatenated and passed through Fully
Connected Network, producing vector $D_{l}$ as output.

For passing data through CNN block, the input text
$T=\\{W_{1},W_{2},...,W_{N}\\}$ was tokenized using $nltk$
library111https://www.nltk.org/, where each $W_{i}$ represents a tokenized
word. The text $T$ was passed through CNN, and text representations were
obtained as vector $C$. Vector $C$ was passed through Fully Connected Network,
transforming it into vector $C_{l}$.

As a final step, all the output feature vectors
$V_{l},U_{l},D_{l},S_{l},C_{l}$ were concatenated and passed through Fully
Connected Network and Softmax transformation, obtaining the prediction.

We made our code available at
GitHub222https://github.com/Wittmann9/SarcasmTuneUntrainableSentEmo.

Figure 1: Proposed architecture of the model.

###  4.2 Sarcasm Pre-Trained Transformer

We merged all training parts of the datasets and then we pre-trained RoBERTa
model on it using masked language modeling objective.

###  4.3 Emotion Detection Pre-Trained Transformer

As an emotion feature extractor, we used BERT
model333https://huggingface.co/bhadresh-savani/bert-base-go-emotion pre-
trained on GoEmotions dataset [6] on multi-label (28) emotion classification
task. The input text was passed through this model and corresponding vector
representation of CLS token of the last hidden state was obtained, as well as
labels’ distribution.

###  4.4 Sentiment Detection Pre-Trained Transformer

As a sentiment feature extractor, we used SiEBERT model [9]. It is a fine-
tuned on sentiment classification task RoBERTa-large model [13]. The model was
trained on 15 datasets from different text sources (reviews, tweets, etc.). We
passed the input text through the model and corresponding vector
representation of CLS token of last hidden state was obtained, as well as
labels’ distribution.

###  4.5 CNN

We used the CNN architecture following [12], see Figure 2. For processing the
input text through CNN, we built a vocabulary dictionary for each dataset
separately to create an embedding layer. We used Glove Common
Crawl444https://nlp.stanford.edu/projects/glove/ pre-trained vectors (42
billion words version). For words with no pre-trained vectors, we checked
their stemmed versions. If no pre-trained embedding were found, random vector
was initialized. Then, the input text was encoded into the embedding matrix of
shape $(N_{W},300)$, where $N_{W}$ is a number of words in the input text and
300 is an embedding dimension.

Figure 2: CNN block.

We then used convolutions with different filter sizes to extract feature maps
from the embedding matrix. Next, we applied the ReLU activation and max-over-
time-pooling to reduce each feature map to a single scalar. Then we
concatenated these scalars into a vector and obtained vector representations
for the texts.

##  5 Description of Baselines

Since prior studies lack uniform datasets, we run baseline comparison
experiments using our datasets and either open source code or reproduced code.

###  5.1 NBOW

Neural bag-of-words (NBOW) baseline takes an average of word vectors in the
given text as sentence representation and feeds it into standard logistic
regression model. As word vector representations we used GloVe 100-dimensional
vectors [16] pre-trained on Wikipedia and an archive of English newswire text
named Gigaword 5, the version contains 6 billion tokens.

###  5.2 CNN

We used CNN configuration following [12]. The overall architecture is as
follows. The padded embedded sentences are processed via the CNN cells. Next,
the ReLU activation function and Max Pooling are applied. The concatenated
outputs from the previous step are processed by linear layers to produce the
distribution of the classes.

###  5.3 CNN-LSTM-DNN

This model [8] is a combination of CNN, LSTM, and DNN. It stacks two layers of
convolution and two LSTM layers, then passes the output to a DNN for
prediction.

###  5.4 SAWS

SAWS [14] model proposes a self-attention mechanism of weighted snippets to
model the incongruity between sentence snippets.

###  5.5 ELMo

This model [10] uses character-level vector representations of words, based on
embeddings from ELMo[17] language model architecture. Subsequently, word
embeddings passed on to a BiLSTM, the output hidden states are max-pooled and
fed to the 2-layer feed-forward network. The output of this step is then fed
to the final layer of the model, which performs the binary classification.

##  6 Experiments

We implemented our model using Pytorch Lightning [7] and transformers library
[21]. For each dataset, we have experimented with different sets of
hyperparameters. The best performing hyperparameters are presented in the
Table 2.

Table 2: Best performing hyperparameters for various datasets. | Datasets
---|---
Parameter | SARC/movies | SARC/technology | IAC_V2 |  Twitter (Ptacek et al., 2014)
max_length | 18 | 14 | 16 |  16
max_epochs | 12 | 30 | 20 |  20
lr | 1e(-5) | 1e(-5) | 1e(-5) |  1e(-5)
batch_size | 8 | 4 | 32 |  32

##  7 Results and Discussion

We used precision, recall, macro-weighted F1, and accuracy scores as the
evaluation metrics. We emphasized F1 score as the main metric in our analysis.
Table 3 reports the results of our method as well as baselines.

Table 3: Results of the experiments. Datasets | Model | Precision | Recall | Acc. | F1
---|---|---|---|---|---
| NBOW | 0.60 | 0.60 | 0.60 | 0.60
| CNN | 0.65 | 0.65 | 0.64 | 0.65
| CNN-LSTM-DNN | 0.58 | 0.58 | 0.59 | 0.58
| SAWS | 0.65 | 0.65 | 0.65 | 0.65
| ELMo | 0.72 | 0.57 | 0.68 | 0.64
SARC/movies | Our Model | 0.72 | 0.74 | 0.73 | 0.73
| NBOW | 0.63 | 0.64 | 0.64 | 0.64
| CNN | 0.67 | 0.67 | 0.66 | 0.66
| CNN-LSTM-DNN | 0.65 | 0.58 | 0.56 | 0.61
| SAWS | 0.65 | 0.65 | 0.66 | 0.65
| ELMo | 0.76 | 0.81 | 0.73 | 0.78
SARC/technology | Our Model | 0.77 | 0.84 | 0.75 | 0.80
| NBOW | 0.72 | 0.71 | 0.72 | 0.71
| CNN | 0.71 | 0.70 | 0.70 | 0.70
| CNN-LSTM-DNN | 0.63 | 0.75 | 0.65 | 0.68
| SAWS | 0.75 | 0.75 | 0.75 | 0.75
| ELMo | 0.78 | 0.74 | 0.77 | 0.76
IAC-V2 | Our Model | 0.86 | 0.84 | 0.85 | 0.85
| NBOW | 0.75 | 0.75 | 0.75 | 0.75
| CNN | 0.80 | 0.80 | 0.80 | 0.80
| CNN-LSTM-DNN | 0.77 | 0.82 | 0.80 | 0.80
| SAWS | 0.81 | 0.81 | 0.81 | 0.81
| ELMo | 0.83 | 0.86 | 0.85 | 0.84
Twitter | Our Model | 0.93 | 0.94 | 0.94 | 0.93

For each dataset, we trained each baseline model and our model. We observed
that the basic models (i.e., NBOW, CNN, and CNN-LSTM-DNN) perform poorly
compared to other models. Those models simply encode input text to capture
local semantic information, which is insufficient to derive global context or
recognize incongruity between words.

However, out of three above-mentioned models, CNN showed higher results on
SARC/movies and SARC/technology subreddit. One of the reasons leading to such
result could be in the relative shortness of texts in those datasets. The
statistics are presented in Table 4. Particularly, median and mean for
SARC/movies dataset $(Med=10,\mu_{i}=12.24)$ are significantly lower than
median and mean for IAC-V2 dataset respectively $(Med=39,\mu_{i}=17.61)$.

Table 4: Statistics of the datasets. | Metrics
---|---
Dataets | Median | Mean | Variance | Max | Min
SARC/movies | 10 | 12.24 | 8.814760348 | 138 | 1
SARC/technology | 12 | 13.88 | 9.330058949 | 103 | 1
IAC-V2 | 39 | 50.63 | 36.05135226 | 212 | 10
Twitter | 17 | 17.61 | 6.257155903 | 64 | 1

SAWS and ELMo models outperformed basic models. Compared to the previous
baselines, SAWS and ELMo models are built to capture more sophisticated
patterns, such as text fragments incongruity and complex morpho-syntactic
features. However, the ELMo model is almost always a few points ahead of SAWS,
showing that purely character-based input and subsequently obtained contextual
embeddings capture more useful sarcastic information.

For all datasets, our model outperformed all baselines for all metrics and
achieved state-of-the-art performance. The best improvement of the F1 metric
was achieved on IAC-V2 and Twitter dataset. Specifically, the F1 metric is 9%
higher, compared with the previous state-of-the-art version. For SARC/movies
dataset, F1 metric is improved by 8%, and for SARC/technology dataset F1
metric is improbed by 2%.

We used different transformer models fitted on emotion and sentiment detection
tasks to obtain contextualized and more deep features. This allows the model
to learn complex patterns from different perspectives, e.g. “emotional” and
“sentiment”.

Furthermore, the CNN block of our model utilizes Glove embeddings in order to
capture semantic relationships of words and general context. Our model shows
that modeling dependencies between emotion, sentiment and sarcasm is an
important feature for sarcasm detection task.

Interestingly, all models show its best results on Twitter dataset, and its
performance decreases when the length of the input text is relatively long
(IAC-V2) or short (SARC/movies). It suggests that more ideas should be
investigated for texts of particular length.

##  8 Conclusion and Future Work

In this paper, we introduced a novel sarcasm detection method, which
incorporates both emotion and sentiment features. Proposed approach contains
four components: Sarcasm Pre-Trained Transformer (SarcPTT), Emotion Detection
Pre-Trained Transformer (EmoDPTT), Sentiment Detection Pre-Trained Transformer
(SentDPTT), and CNN block.

EmoDPTT and SentDPTT are used as feature extractors and are not trainable
during model fitting. These models are used to highlight the parts of the
sentence which provide crucial cues for sarcasm detection. CNN module extracts
semantic relationships of words and general context features. SarcPTT is pre-
tprained on train chunks of datasets, so the model learns sarcasm patterns.
The output from each block is passed through a fully-connected network and
then through a linear layer to get the final classification score.

Experiments are conducted on four datasets: SARC/movies, SARC/technology,
IAC-V2, and Twitter. They show significant improvement over the state-of-the-
art models using all evaluation metrics. Our results show that emotion and
sentiment features along with contextual knowledge play a crucial role in the
performance of sarcasm detection.

Even though our model outperformed all the baselines, all the methods are
struggling on too short or too long texts. In our future works we are planning
to address this challenge.

## References

  * [1] Abbott, R., Ecker, B., Anand, P., Walker, M.: Internet argument corpus 2.0: An sql schema for dialogic social media and the corpora to go with it. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). pp. 4445–4452 (2016)
  * [2] Akula, R., Garibay, I.: Interpretable multi-head self-attention architecture for sarcasm detection in social media. Entropy 23(4), 394 (2021)
  * [3] Babanejad, N., Davoudi, H., An, A., Papagelis, M.: Affective and contextual embedding for sarcasm detection. In: Proceedings of the 28th International Conference on Computational Linguistics. pp. 225–243. International Committee on Computational Linguistics, Barcelona, Spain (Online) (Dec 2020). https://doi.org/10.18653/v1/2020.coling-main.20, https://aclanthology.org/2020.coling-main.20
  * [4] Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., Poria, S.: Towards multimodal sarcasm detection (an _obviously_ perfect paper). In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Florence, Italy (7 2019)
  * [5] Chauhan, D.S., Dhanush, S., Ekbal, A., Bhattacharyya, P.: Sentiment and emotion help sarcasm? a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4351–4360 (2020)
  * [6] Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., Ravi, S.: GoEmotions: A Dataset of Fine-Grained Emotions. In: 58th Annual Meeting of the Association for Computational Linguistics (ACL) (2020)
  * [7] Falcon, W., others: Pytorch lightning. GitHub. Note: https://github. com/PyTorchLightning/pytorch-lightning 3(6) (2019)
  * [8] Ghosh, A., Veale, T.: Fracking sarcasm using neural network. In: Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis. pp. 161–169 (2016)
  * [9] Hartmann, J., Heitmann, M., Siebert, C., Schamp, C.: More than a feeling: Accuracy and application of sentiment analysis. International Journal of Research in Marketing (2022)
  * [10] Ilić, S., Marrese-Taylor, E., Balazs, J., Matsuo, Y.: Deep contextualized word representations for detecting sarcasm and irony. In: Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. pp. 2–7. Association for Computational Linguistics, Brussels, Belgium (Oct 2018). https://doi.org/10.18653/v1/W18-6202, https://aclanthology.org/W18-6202
  * [11] Khodak, M., Saunshi, N., Vodrahalli, K.: A large self-annotated corpus for sarcasm. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan (May 2018), https://aclanthology.org/L18-1102
  * [12] Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL. pp. 1746–1751 (2014), http://aclweb.org/anthology/D/D14/D14-1181.pdf
  * [13] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [14] Pan, H., Lin, Z., Fu, P., Wang, W.: Modeling the incongruity between sentence snippets for sarcasm detection. In: ECAI 2020, pp. 2132–2139. IOS Press (2020)
  * [15] Peng, W., Adikari, A., Alahakoon, D., Gero, J.: Discovering the influence of sarcasm in social media responses. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 9(6), e1331 (2019)
  * [16] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532–1543 (2014)
  * [17] Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 2227–2237. Association for Computational Linguistics, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/N18-1202, https://aclanthology.org/N18-1202
  * [18] Ptáček, T., Habernal, I., Hong, J.: Sarcasm detection on Czech and English Twitter. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. pp. 213–223. Dublin City University and Association for Computational Linguistics, Dublin, Ireland (Aug 2014), https://aclanthology.org/C14-1022
  * [19] Savini, E., Caragea, C.: Intermediate-task transfer learning with BERT for sarcasm detection. Mathematics 10(5) (2022). https://doi.org/10.3390/math10050844, https://www.mdpi.com/2227-7390/10/5/844
  * [20] Walker, M., Tree, J.E.F., Anand, P., Abbott, R., King, J.: A corpus for research on deliberation and debate. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12). pp. 812–817 (2012)
  * [21] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45. Association for Computational Linguistics, Online (Oct 2020), https://www.aclweb.org/anthology/2020.emnlp-demos.6