11institutetext: Instituto Superior Técnico, Universidade de Lisboa, Lisboa,
Portugal 22institutetext: INESC-ID, Lisboa, Portugal 22email:
<EMAIL_ADDRESS>
# Quantifying Valence and Arousal in Text
with Multilingual Pre-trained
Transformers

Gonçalo Azevedo Mendes 1122 0000-0002-0595-3367 Bruno Martins 1122
0000-0002-3856-2936

###### Abstract

The analysis of emotions expressed in text has numerous applications. In
contrast to categorical analysis, focused on classifying emotions according to
a pre-defined set of common classes, dimensional approaches can offer a more
nuanced way to distinguish between different emotions. Still, dimensional
methods have been less studied in the literature. Considering a valence-
arousal dimensional space, this work assesses the use of pre-trained
Transformers to predict these two dimensions on a continuous scale, with input
texts from multiple languages and domains. We specifically combined multiple
annotated datasets from previous studies, corresponding to either emotional
lexica or short text documents, and evaluated models of multiple sizes and
trained under different settings. Our results show that model size can have a
significant impact on the quality of predictions, and that by fine-tuning a
large model we can confidently predict valence and arousal in multiple
languages. We make available the code, models, and supporting data.

###### Keywords:

Transformer-Based Multilingual Language Models Emotion Analysis in Text
Predicting Valence and Arousal

##  1 Introduction

The task of analyzing emotions expressed in text is commonly modeled as a
classification problem, representing affective states (e.g., Ekman’s six basic
emotions [22]) as specific classes. The alternative approach of dimensional
emotion analysis focuses on rating emotions according to a pre-defined set of
dimensions, offering a more nuanced way to distinguish between different
emotions [7]. Emotional states are represented on a continuous numerical
space, with the most common dimensions defined as valence and arousal. In
particular, valence describes the pleasantness of a stimulus, ranging from
negative to positive feelings. Arousal represents the degree of excitement
provoked by a stimulus, from calm to excited. The Valence-Arousal (VA) space
[4] corresponds to a 2-dimensional space to which a text sequence can be
mapped.

This study proposes using pre-trained multilingual Transformer models to
predict valence and arousal ratings in text from different languages and
domains. Models pre-trained on huge amounts of data from multiple languages
can be fine-tuned to different types of downstream tasks with relatively small
datasets in one or few languages, and still obtain reliable results on
different languages [43]. While previous research focused on monolingual VA
prediction as regression from text, this study compiled 34 publicly available
psycho-linguistic datasets, from different languages, into a single uniform
dataset. We then eva luated multilingual DistilBERT [48] and XLM-RoBERTa [17]
models, to understand the impact of model size and training conditions on the
ability to correctly predict affective ratings from textual contents.

Experimental results show that multilingual VA prediction is possible with a
single Transformer model, particularly when considering the larger XLM-RoBERTa
model. Even if performance differs across languages, most results improve or
stay in line with the results from previous research focused on predicting
these affective ratings on a single language. The code, models, and data used
in this study are available on a GitHub repository111
https://www.github.com/gmendes9/multilingual_va_prediction .

The rest of the paper is organized as follows: Section 2 presents related
work, while Section 3 describes the models considered for predicting valence
and arousal. Section 4 describes the corpora used for model training and
evaluation. Section 5 presents our findings and compares the results. Finally,
Section 6 summarizes the main findings and discusses possibilities for future
work.

##  2 Related Work

Since Russel [47] first proposed a two-dimensional model of emotions, based on
valence and arousal, much research has been done on dimensional emotion
analysis. Most relevant to this study are the main lexicons [5, 38, 49, 55,
60] and corpora [6, 8, 42, 66] annotated according to these dimensions, used
in previous work. Still, while several NLP and IR studies have addressed
dimensional emotion extraction, most previous work has focused on categorical
approaches [1].

Trying to predict valence and arousal has long been a relevant topic, both at
the word-level [11, 20, 26, 45, 51, 62, 67] and at the sentence/text-level [7,
9, 32, 36, 41, 44, 52, 59, 63]. Recchia et al. used pointwise mutual
information coupled with k-NN regression to estimate valence and arousal for
words [45]. Hollis et al. resorted to linear regression modelling [26]. Sedoc
et al. combined distributional approaches with signed spectral clustering
[51]. Du and Zhang explored the use of CNNs [20]. Wu et al. used a densely
connected LSTM network and word features to identify emotions on the VA space
for words and phrases [62]. More recently, Buechel et al. proposed a method
for creating arbitrarily large emotion lexicons in 91 languages, using a
translation model, a target language embedding model, and a multitask learning
feed-forward neural network [11]. This last work is interesting when compared
to ours, as it is one of the few attempts to predict VA at a multilingual
level, if only for individual words.

Paltoglou et al. attempted text-level VA prediction by resorting to affective
dictionaries, as supervised machine learning techniques were inadequate for
the small dataset used in their tests [41]. Preoţiuc-Pietro et al. compiled a
corpus of Facebook posts and built a bag-of-words (BoW) linear regression
prediction model [44]. Similarly, Buechel and Hahn used BoW representations in
conjunction with TF-IDF weights [7, 9]. More recently, several studies have
compared CNNs and RNNs, amongst other neural architectures [32, 52, 59, 63].
For instance, Lee et al. explored different methods for prediction, ranging
from linear regression to multiple neural network architectures [36]. This
last study explored the use of a BERT model, but differs from our work as the
data is not multilingual. The present work follows the steps of some of the
aforementioned studies leveraging deep learning, aiming to build a single
multilingual model capable of predicting affective ratings for valence and
arousal.

##  3 Models for Predicting Valence and Arousal from Text

We address the prediction of valence and arousal scores as text-based
regression, using pre-trained multilingual models adapted from the Huggingface
library [61]. In particular, we use DistilBERT [48] and XLM-RoBERTa [17]
models.

The multilingual DistilBERT model, consisting of 134M parameters, is based on
a 6 layer Transformer encoder, with 12 attention heads and a hidden state size
of 768. The model can train two times faster with only a slight performance
decrease (approx. 5%), compared to a multilingual BERT-base model with 25%
more parameters. As for XLM-RoBERTa, we used both the base (270M parameters)
and large (550M parameters) versions. The base version is a 12 layer
Transformer, with 12 attention heads and a hidden state size of 768. The large
version uses 24 layers, 16 attention heads, and a hidden state size of 1024.

Both these models are pre-trained on circa 100 different languages, which will
likely enable the generalization to languages for which there are no annotated
data in terms of valence and arousal ratings. These models are fine-tuned for
the task at hand with a regression head on top, consisting of a linear layer
on top of the pooled representation from the Transformer (i.e., the
representation of the first token in the input sequence).

The regression head produces two outputs, which are processed through a hard
sigmoid activation function, forcing the predicted values on both dimensions
to respect the target interval between zero and one.

Three loss functions were initially compared for model training, namely the
Mean Squared Error (MSE), the Concordance Correlation Coefficient Loss (CCCL),
and a recently proposed Robust Loss (RL) function [3]. In all these cases, the
models are trained with the sum of the loss for the valence and arousal
predictions, equally weighting both affective dimensions.

MSE is the most used loss function in regression problems and can be defined
as the mean of the squared differences between predicted ($\hat{y}$) and
ground-truth ($y$) values, as shown in Equation 1.

$\mathrm{MSE}=\frac{1}{N}\sum_{i=0}^{N}(y_{i}-\hat{y_{i}})^{2}.$ (1)

The CCCL corresponds to a correlation-based function, evaluating the ranking
agreement between the true and predicted values, within a batch of instances.
It varies from the Pearson correlation by penalizing the score in proportion
to the deviation if the predictions shift in value. Atmaja and Akagi [56]
compared this function to the MSE and Mean Absolute Error (MAE) loss functions
for the task of predicting emotional ratings from speech signals using LSTM
neural networks, suggesting that this loss yields a better performance than
error-based functions. The CCCL follows Equation 2, where $\rho_{y\hat{y}}$
represents the Pearson correlation coefficient between $y$ and $\hat{y}$,
$\sigma$ represents the standard deviation, and $\mu$ the mean value. Notice
that the correlation ranges from -1 to 1, and thus we use one minus the
correlation as the loss.

$\mathrm{CCCL}=1-\frac{2\rho_{y\hat{y}}\sigma_{y}\sigma_{\hat{y}}}{\sigma_{y^{2}}+\sigma_{\hat{y}^{2}}+(\mu_{y}-\mu_{\hat{y}})^{2}}.$
(2)

The RL function generalizes some of the most common robust loss functions
(e.g., the Huber loss), that reduce the influence of outliers [3], being
described in its general form through Equation 3. In this function, $x$ is the
variable being minimized, corresponding to the difference between true and
predicted values (i.e., $x_{i}=y_{i}-\hat{y_{i}}$). The function involves two
parameters that tune its shape, namely $\alpha\in\mathbb{R}$ that controls the
robustness, and a scale parameter $c>0$ which controls the size of its
quadratic bowl.

$\mathrm{RL}=\frac{1}{N}\sum_{i=0}^{N}\begin{cases}\frac{1}{2}(x_{i}/c)^{2}&\text{
if }\alpha=2\\\ \textup{log}\left(\frac{1}{2}(x_{i}/c)^{2}+1\right)&\text{ if
}\alpha=0\\\ 1-\textup{exp}\left(-\frac{1}{2}(x_{i}/c)^{2}\right)&\text{ if
}\alpha=\infty\\\
\frac{|\alpha-2|}{\alpha}\left(\left(\frac{(x_{i}/c)^{2}}{|\alpha-2|}+1\right)^{\alpha/2}-1\right)&\text{otherwise}.\end{cases}$
(3)

A lower value of $\alpha$ implies penalizing minor errors at the expense of
larger ones, while a higher value of $\alpha$ allows more inliers while
increasing the penalty for outliers. We used the adaptive form of this robust
loss function, where the parameter $\alpha$ is optimized and tuned during
model training via stochastic gradient descent, as explained in the original
paper [3].

We also tested two hybrid loss functions derived from the previous ones,
combining their different properties and merits. While the MSE and the RL
functions analyze results at the instance level, the CCCL function does the
same at the batch level. With this in mind, one hybrid loss function combines
the CCCL and the MSE functions, while the other combines the CCCL with the RL
function, in both cases through a simple addition.

##  4 Resources

We collected 34 different public datasets to form a large corpus of annotated
data for the emotional dimensions of valence and arousal, with the intent to
build the largest possible multilingual dataset. The original datasets
comprise 13 different languages, which represent up to 2.5 billion native
speakers worldwide222https://www.cia.gov/the-world-
factbook/countries/world/#people-and-society333https://www.ethnologue.com/.
There are two types of datasets described on Table 1, namely word and short
text datasets, respectively associating valence and arousal ratings to either
individual words or short text sequences. All of these datasets were manually
annotated by humans, either via crowdsourcing or by experienced
linguists/psychologists, according to the Self-Assessment Manikin (SAM) method
[4]. In addition, several lexicons relate to the Affective Norms for English
Words (ANEW) resource, corresponding to either adaptations to other languages
or extensions in terms of the number of words [5]. ANEW was the first lexicon
providing real-valued scores for the emotional dimensions of valence and
arousal. It is important to note that this lexicon is excluded from our corpus
for being part of larger datasets that were included, such as the one from
Warriner et al. [60].

Table 1: Dataset characterization. $\mu_{\tiny{\textrm{length}}}$ represents the mean text length of each instance, in terms of the number of words. $\mu$ and $\sigma$ represent the mean and standard deviation, in the emotional ratings, respectively. |  |  |  | Arousal | Valence
---|---|---|---|---|---
Dataset | Language | Items | $\mu_{\tiny{\textrm{length}}}$ | $\mu{}$ | $\sigma{}$ |  | $\mu{}$ | $\sigma{}$ |
EmoBank [8, 10] | English | 10062 | 23.27 | 0.51 | 0.06 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0 0.2222 0.0016 0.3333 0.0395 0.4444 0.5866 0.5556 1.0 0.6667 0.1083 0.7778 0.0129 0.8889 0.0026 1.0 0.0 | 0.49 | 0.09 |  10 -0.05 1.05 0.0 0.0005 0.1111 0.004 0.2222 0.0525 0.3333 0.1507 0.4444 0.4579 0.5556 1.0 0.6667 0.1177 0.7778 0.021 0.8889 0.0032 1.0 0.0
IEMOCAP [12] | English | 10039 | 19.22 | 0.56 | 0.22 |  10 -0.05 1.05 0.0 0.017 0.1111 0.1827 0.2222 0.6293 0.3333 0.3892 0.4444 0.0 0.5556 0.829 0.6667 1.0 0.7778 0.9717 0.8889 0.4501 1.0 0.0312 | 0.48 | 0.17 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0572 0.2222 0.0739 0.3333 0.3536 0.4444 0.19 0.5556 1.0 0.6667 0.5849 0.7778 0.1566 0.8889 0.1073 1.0 0.0044
Facebook Posts [44] | English | 2894 | 28.15 | 0.29 | 0.25 |  10 -0.05 1.05 0.0 1.0 0.1111 0.8179 0.2222 0.2243 0.3333 0.2968 0.4444 0.1689 0.5556 0.4459 0.6667 0.2361 0.7778 0.0369 0.8889 0.0501 1.0 0.0 | 0.53 | 0.15 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0572 0.2222 0.0739 0.3333 0.3536 0.4444 0.19 0.5556 1.0 0.6667 0.5849 0.7778 0.1566 0.8889 0.1073 1.0 0.0044
EmoTales [24] | English | 1395 | 17.91 | 0.55 | 0.12 |  10 -0.05 1.05 0.0 0.0069 0.1111 0.0 0.2222 0.0361 0.3333 0.1015 0.4444 0.4596 0.5556 1.0 0.6667 0.5009 0.7778 0.2048 0.8889 0.0516 1.0 0.0052 | 0.49 | 0.15 |  10 -0.05 1.05 0.0 0.0057 0.1111 0.0684 0.2222 0.1977 0.3333 0.2966 0.4444 0.4791 0.5556 1.0 0.6667 0.3308 0.7778 0.116 0.8889 0.0437 1.0 0.0
ANET [6] | English | 120 | 31.96 | 0.66 | 0.22 |  10 -0.05 1.05 0.0 0.0 0.1111 0.1471 0.2222 0.2059 0.3333 0.2353 0.4444 0.1765 0.5556 0.3529 0.6667 0.3824 0.7778 0.8824 0.8889 1.0 1.0 0.1471 | 0.52 | 0.33 |  10 -0.05 1.05 0.0 0.32 0.1111 1.0 0.2222 0.32 0.3333 0.0 0.4444 0.04 0.5556 0.28 0.6667 0.36 0.7778 0.08 0.8889 0.84 1.0 0.76
PANIG [16] | German | 619 | 9.12 | 0.47 | 0.12 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0 0.2222 0.1881 0.3333 0.7525 0.4444 1.0 0.5556 0.6337 0.6667 0.3465 0.7778 0.1337 0.8889 0.0099 1.0 0.0 | 0.40 | 0.22 |  10 -0.05 1.05 0.0 0.0345 0.1111 0.4885 0.2222 1.0 0.3333 0.523 0.4444 0.2356 0.5556 0.2414 0.6667 0.2931 0.7778 0.3276 0.8889 0.1264 1.0 0.0
COMETA sentences [15] | German | 120 | 16.75 | 0.48 | 0.15 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0882 0.2222 0.4706 0.3333 0.5588 0.4444 0.6176 0.5556 1.0 0.6667 0.6176 0.7778 0.1765 0.8889 0.0 1.0 0.0 | 0.50 | 0.20 |  10 -0.05 1.05 0.0 0.0312 0.1111 0.1562 0.2222 0.375 0.3333 1.0 0.4444 0.375 0.5556 0.4375 0.6667 0.5938 0.7778 0.5938 0.8889 0.1875 1.0 0.0
COMETA stories [15] | German | 64 | 90.17 | 0.53 | 0.15 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0 0.2222 0.3333 0.3333 0.5333 0.4444 1.0 0.5556 1.0 0.6667 0.8667 0.7778 0.3333 0.8889 0.2 1.0 0.0 | 0.56 | 0.21 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0 0.2222 0.3571 0.3333 0.6429 0.4444 0.4286 0.5556 0.6429 0.6667 0.2857 0.7778 1.0 0.8889 0.4286 1.0 0.0714
CVAT [66] | Mandarin | 2969 | 58.00 | 0.48 | 0.13 |  10 -0.05 1.05 0.0 0.0069 0.1111 0.0446 0.2222 0.2491 0.3333 0.6286 0.4444 0.8537 0.5556 1.0 0.6667 0.4686 0.7778 0.1211 0.8889 0.0206 1.0 0.0 | 0.48 | 0.17 |  10 -0.05 1.05 0.0 0.01 0.1111 0.1605 0.2222 0.7742 0.3333 0.9582 0.4444 0.6304 0.5556 1.0 0.6667 0.9666 0.7778 0.4348 0.8889 0.0301 1.0 0.0
CVAI [63] | Mandarin | 1465 | 29.53 | 0.51 | 0.12 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0066 0.2222 0.0773 0.3333 0.521 0.4444 0.9073 0.5556 1.0 0.6667 0.468 0.7778 0.223 0.8889 0.0265 1.0 0.0044 | 0.32 | 0.06 |  10 -0.05 1.05 0.0 0.0024 0.1111 0.0694 0.2222 0.5104 0.3333 1.0 0.4444 0.1973 0.5556 0.0049 0.6667 0.0 0.7778 0.0 0.8889 0.0 1.0 0.0
ANPST [28] | Polish | 718 | 28.16 | 0.48 | 0.13 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0619 0.2222 0.2143 0.3333 0.6476 0.4444 1.0 0.5556 0.8333 0.6667 0.4381 0.7778 0.1857 0.8889 0.0381 1.0 0.0 | 0.47 | 0.22 |  10 -0.05 1.05 0.0 0.0588 0.1111 0.7143 0.2222 1.0 0.3333 0.8571 0.4444 0.7395 0.5556 0.7479 0.6667 0.6303 0.7778 0.8739 0.8889 0.4118 1.0 0.0
MAS [42] | Portuguese | 192 | 8.94 | 0.52 | 0.17 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0 0.2222 0.0645 0.3333 1.0 0.4444 0.629 0.5556 0.3548 0.6667 0.2903 0.7778 0.5161 0.8889 0.2419 1.0 0.0 | 0.49 | 0.28 |  10 -0.05 1.05 0.0 0.5 0.1111 0.8095 0.2222 0.1429 0.3333 0.0 0.4444 0.4286 0.5556 1.0 0.6667 0.0714 0.7778 0.619 0.8889 0.6429 1.0 0.119
Yee [65] | Cantonese | 292 |  | 0.40 | 0.15 |  10 -0.05 1.05 0.0 0.0 0.1111 0.1807 0.2222 0.9036 0.3333 1.0 0.4444 0.6627 0.5556 0.4217 0.6667 0.1446 0.7778 0.1446 0.8889 0.0602 1.0 0.0 | 0.58 | 0.17 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0682 0.2222 0.0568 0.3333 0.1136 0.4444 0.4432 0.5556 1.0 0.6667 0.6023 0.7778 0.375 0.8889 0.2841 1.0 0.0341
Ćoso et al. [18] | Croatian | 3022 |  | 0.45 | 0.15 |  10 -0.05 1.05 0.0 0.0014 0.1111 0.0962 0.2222 0.6099 0.3333 1.0 0.4444 0.9162 0.5556 0.7527 0.6667 0.4808 0.7778 0.2184 0.8889 0.0481 1.0 0.0 | 0.51 | 0.21 |  10 -0.05 1.05 0.0 0.0505 0.1111 0.4218 0.2222 0.4625 0.3333 0.3567 0.4444 0.5358 0.5556 1.0 0.6667 0.8453 0.7778 0.5635 0.8889 0.2785 1.0 0.0
Moors et al. [40] | Dutch | 4299 |  | 0.52 | 0.14 |  10 -0.05 1.05 0.0 0.0 0.1111 0.02 0.2222 0.1146 0.3333 0.7616 0.4444 1.0 0.5556 0.9245 0.6667 0.6278 0.7778 0.3276 0.8889 0.1137 1.0 0.0036 | 0.49 | 0.18 |  10 -0.05 1.05 0.0 0.0231 0.1111 0.1742 0.2222 0.4261 0.3333 0.422 0.4444 0.5681 0.5556 1.0 0.6667 0.4624 0.7778 0.3105 0.8889 0.114 1.0 0.0
Verheyen et al. [57] | Dutch | 1000 |  | 0.52 | 0.17 |  10 -0.05 1.05 0.0 0.0 0.1111 0.1066 0.2222 0.3279 0.3333 0.668 0.4444 0.7992 0.5556 1.0 0.6667 0.5205 0.7778 0.4795 0.8889 0.1803 1.0 0.0164 | 0.50 | 0.20 |  10 -0.05 1.05 0.0 0.0 0.1111 0.3086 0.2222 0.8642 0.3333 0.8148 0.4444 0.8272 0.5556 1.0 0.6667 0.858 0.7778 0.8086 0.8889 0.3148 1.0 0.0062
NRC-VAD [38] | English | 19971 |  | 0.50 | 0.17 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0727 0.2222 0.3846 0.3333 0.8113 0.4444 1.0 0.5556 0.8168 0.6667 0.5278 0.7778 0.3332 0.8889 0.1953 1.0 0.0574 | 0.50 | 0.22 |  10 -0.05 1.05 0.0 0.0443 0.1111 0.3251 0.2222 0.3673 0.3333 0.506 0.4444 0.7804 0.5556 1.0 0.6667 0.7426 0.7778 0.4277 0.8889 0.2219 1.0 0.0
Warriner et al. [60] | English | 13915 |  | 0.40 | 0.11 |  10 -0.05 1.05 0.0 0.0008 0.1111 0.0417 0.2222 0.5034 0.3333 1.0 0.4444 0.7965 0.5556 0.3944 0.6667 0.1397 0.7778 0.024 0.8889 0.0015 1.0 0.0 | 0.51 | 0.16 |  10 -0.05 1.05 0.0 0.0069 0.1111 0.1156 0.2222 0.309 0.3333 0.4064 0.4444 0.6551 0.5556 1.0 0.6667 0.6394 0.7778 0.2968 0.8889 0.0714 1.0 0.0
Scott et al. [50] | English | 5553 |  | 0.45 | 0.14 |  10 -0.05 1.05 0.0 0.0226 0.1111 0.209 0.2222 0.2372 0.3333 0.2341 0.4444 0.4187 0.5556 1.0 0.6667 0.4963 0.7778 0.2524 0.8889 0.1571 1.0 0.0 | 0.51 | 0.19 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0304 0.2222 0.4578 0.3333 1.0 0.4444 0.9426 0.5556 0.7387 0.6667 0.3883 0.7778 0.1533 0.8889 0.0385 1.0 0.0
Söderholm et al. [54] | Finnish | 420 |  | 0.50 | 0.13 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0216 0.2222 0.0935 0.3333 0.6547 0.4444 1.0 0.5556 0.5683 0.6667 0.4245 0.7778 0.223 0.8889 0.036 1.0 0.0 | 0.50 | 0.25 |  10 -0.05 1.05 0.0 0.3188 0.1111 0.6957 0.2222 0.6667 0.3333 0.3188 0.4444 0.4203 0.5556 0.913 0.6667 1.0 0.7778 0.9275 0.8889 0.5362 1.0 0.0
Eilola et al. [21] | Finnish | 210 |  | 0.36 | 0.19 |  10 -0.05 1.05 0.0 0.3902 0.1111 1.0 0.2222 0.8049 0.3333 0.5854 0.4444 0.878 0.5556 0.6829 0.6667 0.5854 0.7778 0.1707 0.8889 0.0244 1.0 0.0 | 0.44 | 0.26 |  10 -0.05 1.05 0.0 0.1321 0.1111 1.0 0.2222 0.4717 0.3333 0.0189 0.4444 0.6604 0.5556 0.4906 0.6667 0.283 0.7778 0.5094 0.8889 0.3962 1.0 0.0
FAN [39] | French | 1031 |  | 0.41 | 0.13 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0741 0.2222 0.6235 0.3333 1.0 0.4444 0.7006 0.5556 0.4568 0.6667 0.216 0.7778 0.0957 0.8889 0.0154 1.0 0.0 | 0.56 | 0.17 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0854 0.2222 0.1179 0.3333 0.2602 0.4444 0.7683 0.5556 1.0 0.6667 0.7114 0.7778 0.5569 0.8889 0.1951 1.0 0.0081
FEEL [25] | French | 835 |  | 0.56 | 0.17 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0688 0.2222 0.3807 0.3333 0.367 0.4444 0.422 0.5556 0.7982 0.6667 1.0 0.7778 0.6606 0.8889 0.133 1.0 0.0 | 0.43 | 0.20 |  10 -0.05 1.05 0.0 0.0091 0.1111 0.3545 0.2222 1.0 0.3333 0.7045 0.4444 0.3227 0.5556 0.3227 0.6667 0.5591 0.7778 0.4364 0.8889 0.0409 1.0 0.0
BAWL-R [58] | German | 2902 |  | 0.44 | 0.17 |  10 -0.05 1.05 0.0 0.0131 0.1111 0.1985 0.2222 0.7109 0.3333 1.0 0.4444 0.7898 0.5556 0.6423 0.6667 0.4686 0.7778 0.2423 0.8889 0.0832 1.0 0.0 | 0.51 | 0.21 |  10 -0.05 1.05 0.0 0.0058 0.1111 0.3779 0.2222 0.5581 0.3333 0.6008 0.4444 0.5523 0.5556 1.0 0.6667 0.8314 0.7778 0.7519 0.8889 0.2287 1.0 0.0
ANGST [49] | German | 1034 |  | 0.52 | 0.16 |  10 -0.05 1.05 0.0 0.0 0.1111 0.019 0.2222 0.381 0.3333 0.881 0.4444 0.9905 0.5556 1.0 0.6667 0.9476 0.7778 0.5762 0.8889 0.1286 1.0 0.0 | 0.51 | 0.24 |  10 -0.05 1.05 0.0 0.0506 0.1111 0.7089 0.2222 0.6962 0.3333 0.4367 0.4444 0.2532 0.5556 1.0 0.6667 0.7215 0.7778 0.8987 0.8889 0.5127 1.0 0.0
LANG [29] | German | 1000 |  | 0.39 | 0.20 |  10 -0.05 1.05 0.0 0.0282 0.1111 1.0 0.2222 0.5927 0.3333 0.5121 0.4444 0.4919 0.5556 0.625 0.6667 0.504 0.7778 0.246 0.8889 0.0323 1.0 0.0 | 0.50 | 0.13 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0081 0.2222 0.2769 0.3333 0.4247 0.4444 0.3817 0.5556 1.0 0.6667 0.4919 0.7778 0.1048 0.8889 0.0 1.0 0.0
Italian ANEW [23] | Italian | 1121 |  | 0.52 | 0.19 |  10 -0.05 1.05 0.0 0.0 0.1111 0.1031 0.2222 0.5471 0.3333 1.0 0.4444 0.8655 0.5556 0.6682 0.6667 0.6547 0.7778 0.7937 0.8889 0.3408 1.0 0.009 | 0.51 | 0.26 |  10 -0.05 1.05 0.0 0.0872 0.1111 0.9282 0.2222 0.5128 0.3333 0.241 0.4444 0.2205 0.5556 0.5128 0.6667 0.8308 0.7778 1.0 0.8889 0.5436 1.0 0.0
Xu et al. [64] | Mandarin | 11310 |  | 0.52 | 0.14 |  10 -0.05 1.05 0.0 0.0 0.1111 0.007 0.2222 0.1516 0.3333 0.6134 0.4444 0.9202 0.5556 1.0 0.6667 0.6646 0.7778 0.3145 0.8889 0.0841 1.0 0.0047 | 0.52 | 0.16 |  10 -0.05 1.05 0.0 0.0113 0.1111 0.098 0.2222 0.19 0.3333 0.2008 0.4444 0.297 0.5556 1.0 0.6667 0.3795 0.7778 0.1997 0.8889 0.0741 1.0 0.0
CVAW [36, 66] | Mandarin | 5512 |  | 0.50 | 0.18 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0918 0.2222 0.5008 0.3333 0.659 0.4444 0.8951 0.5556 1.0 0.6667 0.6033 0.7778 0.3369 0.8889 0.2443 1.0 0.023 | 0.44 | 0.21 |  10 -0.05 1.05 0.0 0.0335 0.1111 0.2662 0.2222 1.0 0.3333 0.71 0.4444 0.4119 0.5556 0.452 0.6667 0.5465 0.7778 0.3613 0.8889 0.1532 1.0 0.0
ANPW_R [27] | Polish | 4905 |  | 0.39 | 0.11 |  10 -0.05 1.05 0.0 0.0 0.1111 0.016 0.2222 0.6193 0.3333 1.0 0.4444 0.6514 0.5556 0.3204 0.6667 0.0891 0.7778 0.0183 0.8889 0.0 1.0 0.0 | 0.50 | 0.16 |  10 -0.05 1.05 0.0 0.006 0.1111 0.1027 0.2222 0.396 0.3333 0.3705 0.4444 0.4953 0.5556 1.0 0.6667 0.5792 0.7778 0.2725 0.8889 0.0564 1.0 0.0
NAWL [46] | Polish | 2902 |  | 0.34 | 0.13 |  10 -0.05 1.05 0.0 0.0606 0.1111 0.5316 0.2222 0.9919 0.3333 1.0 0.4444 0.7537 0.5556 0.4361 0.6667 0.1184 0.7778 0.0121 0.8889 0.0013 1.0 0.0 | 0.53 | 0.20 |  10 -0.05 1.05 0.0 0.0 0.1111 0.2597 0.2222 0.4699 0.3333 0.4235 0.4444 0.4791 0.5556 1.0 0.6667 0.7635 0.7778 0.6213 0.8889 0.2767 1.0 0.0216
Portuguese ANEW [53] | Portuguese | 1034 |  | 0.49 | 0.14 |  10 -0.05 1.05 0.0 0.0039 0.1111 0.0502 0.2222 0.2394 0.3333 1.0 0.4444 0.9112 0.5556 0.8533 0.6667 0.5985 0.7778 0.3012 0.8889 0.0347 1.0 0.0 | 0.50 | 0.23 |  10 -0.05 1.05 0.0 0.0113 0.1111 0.5876 0.2222 0.8475 0.3333 0.4859 0.4444 0.4124 0.5556 1.0 0.6667 0.7401 0.7778 0.8079 0.8889 0.4972 1.0 0.0
S.-Gonzalez et al. [55] | Spanish | 14031 |  | 0.70 | 0.22 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0047 0.2222 0.0136 0.3333 0.0411 0.4444 0.1277 0.5556 0.5568 0.6667 1.0 0.7778 0.9168 0.8889 0.4825 1.0 0.3414 | 0.72 | 0.16 |  10 -0.05 1.05 0.0 0.0 0.1111 0.064 0.2222 0.1804 0.3333 0.207 0.4444 0.2145 0.5556 0.392 0.6667 0.7517 0.7778 1.0 0.8889 0.5824 1.0 0.389
Kapucu et al. [30] | Turkish | 2031 |  | 0.50 | 0.11 |  10 -0.05 1.05 0.0 0.0 0.1111 0.0031 0.2222 0.1207 0.3333 0.5345 0.4444 0.8777 0.5556 1.0 0.6667 0.5392 0.7778 0.0956 0.8889 0.0125 1.0 0.0 | 0.49 | 0.20 |  10 -0.05 1.05 0.0 0.0729 0.1111 0.3794 0.2222 0.2598 0.3333 0.2897 0.4444 0.7439 0.5556 1.0 0.6667 0.4262 0.7778 0.372 0.8889 0.1963 1.0 0.0

Overall, merging the 34 datasets allowed us to build a large multilingual VA
dataset, consisting of 128,987 independently annotated instances (i.e., 30,657
short texts and 98,330 words). The textual sequences were left unchanged from
the source datasets. As for the valence and arousal ratings, we took the mean
annotated values when ratings were obtained from multiple individuals, and
normalized the scores between 0 and 1. The normalization was performed
according to the equation
$z_{i}=(x_{i}-\textrm{min}(x))/(\textrm{max}(x)-\textrm{min}(x))$, in which
$z_{i}$ denotes the normalized value, $x_{i}$ the original value, and min and
max denote the extremes of the scales in which the original scores were rated
on.

Table 1 presents a statistical characterization for the short text datasets in
its first half, followed by the word datasets. Each entry describes the
dataset source language, the dataset size, and the mean number of words (this
last variable in the case of the short texts). An exploratory analysis of the
VA ratings supports a better understanding of the score distributions. In
turn, Figure 1 presents the distribution of the ratings for the entire merged
dataset, as well as for its two subsets (i.e., words and short texts). The
ratings were plotted on the two-dimensional valence-arousal space, and they
are visualized with the help of a kernel density estimate. The individual
distributions of the two dimensions are displayed on the margins. The analysis
of the resulting merged dataset leads to the conclusion that there is a
quadratic relationship between the two emotional dimensions, with a tendency
for increased arousal on high and low valence values, and abundant low arousal
scores in the middle of the valence scale. A similar pattern was previously
observed in several different studies in Psychology, such as in the original
ANEW study and its extensions [5, 18, 28, 33, 39, 42, 64].

Figure 1: Distribution of dataset instances in the valence-arousal space. Each
dimensions’ distribution is shown with a histogram on the corresponding axis.
An orange trend line shows the quadratic relation between valence and arousal.

##  5 Experimental Evaluation

Each of the individual original datasets were randomly split in half and
combined with the others to form two subsets of data equally representative of
all the datasets, later used for 2-fold cross-validation. For each
configuration, two models were separately trained on each fold, and then
separately used to make predictions for the instances in the other fold
(containing instances not seen during training), with final evaluation metrics
computed on the complete set of results (the predictions from the models
trained on each fold were joined, and the metrics were computed over the full
set of predictions). Hyperparameters were defined through an initial set of
tests and kept constant for all models. The batch size was fixed at 16, and
models were trained during 10 epochs. AdamW was the chosen optimizer, and we
used it together with a linear learning rate schedule with warm-up. The
learning rate was set at $6\cdot 10^{-6}$, with a warm-up ratio of $1\cdot
10^{-1}$. We experimented with various model and loss function combinations,
namely by using the three differently-sized pre-trained Transformer models, as
well as the loss functions presented in Section 3.

Three different evaluation metrics were used to assess and compare model
performance, namely the Mean Absolute Error (MAE), the Root Mean Squared Error
(RMSE), and the Pearson correlation coefficient ($\rho$). The MAE, as detailed
by Equation 4, corresponds to the sum of absolute errors between observations
$x_{i}$ and predictions $y_{i}$.

$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}|x_{i}-y_{i}|.$ (4)

The RMSE, as shown by Equation 5, is the square root of the mean square of the
differences between observations $x_{i}$ and predictions $y_{i}$.

$\mathrm{RMSE}=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(x_{i}-y_{i})^{2}}.$ (5)

Finally, the Pearson correlation coefficient, given by Equation 6, is used to
assess the presence of a linear relationship between the ground truth $x$ and
the predicted results given by $y$.

$\rho=\frac{\sum_{i=1}^{N}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{N}(x_{i}-\bar{x})^{2}(y_{i}-\bar{y})^{2}}}.$
(6)

While the first two metrics should be minimized, the latter is best when it is
closer to one, i.e., the value denoting a perfect correlation.

###  5.1 Results with Different Models and Loss Functions

Table 2 summarizes the results for the different combinations of model size
and loss function. The single thing that affects evaluation metrics the most
is the size of the pre-trained Transformer model being used. The best
performing model was the large version of XLM-RoBERTa, respectively returning
on average 9% and 20% better correlation results than XLM-RoBERTa-base and
DistilBERT. For each model, we compared the five loss functions, highlighting
in bold the best performing one for each metric, and evaluating separately for
valence and arousal. In short, the choice of loss function has less impact on
the quality of the results. For the best model, we see differences in
correlation of up to 2% between best and worst performing loss functions.
Comparatively, in the error metrics, these differences can be of up to 12%. As
such, looking to identify the best model/loss-function combination, we gave
more relevance to the error metrics. We identified the MSE loss function as
the best performing one, adding to the fact that this loss function is also
the simplest of the set of functions that were tested. Consequently, further
results are presented for that model/loss pair.

Table 2: Comparison between different models and loss functions. Model | Loss | $\mathrm{\rho_{V}}$ | $\mathrm{\rho_{A}}$ | $\mathrm{RMSE_{V}}$ | $\mathrm{RMSE_{A}}$ | $\mathrm{MAE_{V}}$ | $\mathrm{MAE_{A}}$
---|---|---|---|---|---|---|---
| MSE | 0.663 | 0.594 | 0.138 | 0.132 | 0.102 | 0.101
| CCCL | 0.657 | 0.590 | 0.150 | 0.146 | 0.111 | 0.111
| RL | 0.668 | 0.598 | 0.138 | 0.132 | 0.101 | 0.101
| MSE+CCCL | 0.657 | 0.590 | 0.149 | 0.145 | 0.110 | 0.111
DistilBERT | RL+CCCL | 0.664 | 0.591 | 0.147 | 0.144 | 0.109 | 0.110
| MSE | 0.757 | 0.646 | 0.121 | 0.125 | 0.088 | 0.095
| CCCL | 0.757 | 0.653 | 0.136 | 0.144 | 0.101 | 0.110
| RL | 0.757 | 0.657 | 0.122 | 0.125 | 0.088 | 0.095
| MSE+CCCL | 0.757 | 0.655 | 0.135 | 0.141 | 0.099 | 0.108
XLM RoBERTa base | RL+CCCL | 0.757 | 0.657 | 0.134 | 0.141 | 0.099 | 0.107
| MSE | 0.810 | 0.695 | 0.109 | 0.120 | 0.079 | 0.091
| CCCL | 0.817 | 0.698 | 0.117 | 0.132 | 0.085 | 0.099
| RL | 0.802 | 0.689 | 0.114 | 0.122 | 0.083 | 0.092
| MSE+CCCL | 0.815 | 0.699 | 0.121 | 0.135 | 0.089 | 0.103
XLM RoBERTa large | RL+CCCL | 0.813 | 0.694 | 0.119 | 0.133 | 0.087 | 0.100

When analyzing the results, it is possible to break them down into two
categories: predicting valence and arousal for individual words or, on the
other hand, for short texts (see Table 3). Our models are more accurate at
predicting word-level scores, although this is also a more straightforward
problem with less ambiguity. An essential fact to take from the results is the
greater difficulty in predicting the affective dimension of arousal. Previous
research has also stated that human ratings themselves varied much more in
annotating arousal when compared to the valence dimension [41].

Table 3: Comparing VA prediction on words or short texts using the XLM-RoBERTa-large model and considering the MSE loss function for training. Dataset | $\mathrm{\rho_{V}}$ | $\mathrm{\rho_{A}}$ | $\mathrm{RMSE_{V}}$ | $\mathrm{RMSE_{A}}$ | $\mathrm{MAE_{V}}$ | $\mathrm{MAE_{A}}$
---|---|---|---|---|---|---
All data | 0.810 | 0.695 | 0.109 | 0.120 | 0.079 | 0.091
Words | 0.833 | 0.686 | 0.107 | 0.116 | 0.078 | 0.090
Short texts | 0.682 | 0.711 | 0.115 | 0.132 | 0.082 | 0.093

###  5.2 Results per Language and Dataset

Further analysis focused on the results of predicting ratings for each of the
original datasets, with results summarized on Table 4.

For most word datasets, compared in the bottom half of Table 4, our best model
performed to high standards, showing a correlation between predicted values
and the ground truth of around $0.8$ for valence and $0.7$ for arousal. As a
comparison, when evaluating correlation on Warriner’s dataset [60], our work
achieved $\rho_{V}=0.84$ and $\rho_{A}=0.65$, while Hollis [26] achieved
$\rho_{V}=0.80$ and $\rho_{A}=0.63$. Although good scores are observed for
most datasets, we can also identify some outliers, like in the case of the
dataset from Kapucu et al. [30].

As for the short text datasets, compared in the top half of Table 4,
performance varies more significantly, with an overall lower correlation and a
higher error. A particular case is the COMETA stories dataset [15], which
shows a correlation close to zero. The COMETA dataset is a database of
conceptual metaphors, in which half of the text instances contain metaphors
while the other half corresponds to their literal counterparts. The obtained
results indicate that even the best model does not cope well with metaphorical
phrasing. Comparing our model to the method from Preoţiuc-Pietro et al. [44],
the correlation values we obtained for the Facebook Posts dataset were
$\rho_{V}=0.80$ and $\rho_{A}=0.78$, while they achieved $\rho_{V}=0.65$ and
$\rho_{A}=0.85$ (i.e., we have better results for valence, and worse for
arousal). In [66], Yu et al. predict VA on the CVAT dataset using the ratings
obtained for the CVAW words. They obtained correlation results of
$\rho_{V}=0.54$ and $\rho_{A}=0.16$, while our approach obtained
$\rho_{V}=0.89$ and $\rho_{A}=0.62$. In subsequent research, the same team
tried to predict VA ratings with different neural network approaches,
including a model based on BERT, for which they obtained $\rho_{V}=0.87$ and
$\rho_{A}=0.58$ on the same dataset [36].

It should be noted that all previous comparisons against other studies are
merely indicative, given that the experimental conditions (e.g., the data
splits used for training and evaluation) were very different.

Table 4: Evaluation results for the short texts (top) and words (bottom) datasets, using the XLM-RoBERTa-large model and considering the MSE loss. Dataset | Language | $\mathrm{\rho_{V}}$ | $\mathrm{\rho_{A}}$ | $\mathrm{RMSE_{V}}$ | $\mathrm{RMSE_{A}}$ | $\mathrm{MAE_{V}}$ | $\mathrm{MAE_{A}}$
---|---|---|---|---|---|---|---
EmoBank | English | 0.736 | 0.440 | 0.061 | 0.071 | 0.044 | 0.052
IEMOCAP | English | 0.469 | 0.656 | 0.159 | 0.173 | 0.126 | 0.132
Facebook Posts | English | 0.797 | 0.776 | 0.098 | 0.176 | 0.075 | 0.124
EmoTales | English | 0.560 | 0.405 | 0.127 | 0.123 | 0.095 | 0.091
ANET | English | 0.920 | 0.859 | 0.135 | 0.111 | 0.095 | 0.087
PANIG | German | 0.597 | 0.563 | 0.181 | 0.111 | 0.137 | 0.085
COMETA sent. | German | 0.853 | 0.598 | 0.103 | 0.120 | 0.074 | 0.096
COMETA stories | German | 0.072 | 0.042 | 0.254 | 0.160 | 0.206 | 0.130
CVAT | Mandarin | 0.890 | 0.623 | 0.082 | 0.105 | 0.062 | 0.085
CVAI | Mandarin | 0.517 | 0.720 | 0.068 | 0.089 | 0.053 | 0.071
ANPST | Polish | 0.868 | 0.607 | 0.113 | 0.111 | 0.082 | 0.089
MAS | Portuguese | 0.935 | 0.694 | 0.115 | 0.124 | 0.082 | 0.100
Yee | Cantonese | 0.875 | 0.718 | 0.090 | 0.121 | 0.069 | 0.099
Ćoso et al. | Croatian | 0.784 | 0.646 | 0.133 | 0.120 | 0.096 | 0.093
Moors et al. | Dutch | 0.776 | 0.653 | 0.116 | 0.125 | 0.081 | 0.098
Verheyen et al. | Dutch | 0.791 | 0.637 | 0.130 | 0.137 | 0.096 | 0.109
NRC-VAD | English | 0.858 | 0.754 | 0.111 | 0.124 | 0.086 | 0.097
Warriner et al. | English | 0.843 | 0.655 | 0.101 | 0.114 | 0.078 | 0.090
Scott et al. | English | 0.884 | 0.636 | 0.095 | 0.117 | 0.067 | 0.092
Söderholm et al. | Finnish | 0.645 | 0.492 | 0.188 | 0.138 | 0.147 | 0.109
Eilola et al. | Finnish | 0.807 | 0.534 | 0.164 | 0.191 | 0.117 | 0.161
FAN | French | 0.755 | 0.605 | 0.116 | 0.112 | 0.086 | 0.087
FEEL | French | 0.823 | 0.664 | 0.131 | 0.131 | 0.096 | 0.103
BAWL-R | German | 0.749 | 0.629 | 0.139 | 0.133 | 0.101 | 0.105
ANGST | German | 0.837 | 0.738 | 0.135 | 0.117 | 0.092 | 0.089
LANG | German | 0.802 | 0.696 | 0.100 | 0.144 | 0.074 | 0.115
Italian ANEW | Italian | 0.846 | 0.644 | 0.138 | 0.148 | 0.099 | 0.118
Xu et al. | Mandarin | 0.882 | 0.754 | 0.078 | 0.098 | 0.055 | 0.077
CVAW | Mandarin | 0.904 | 0.666 | 0.094 | 0.136 | 0.071 | 0.108
ANPW_R | Polish | 0.846 | 0.689 | 0.093 | 0.088 | 0.065 | 0.069
NAWL | Polish | 0.828 | 0.581 | 0.111 | 0.122 | 0.081 | 0.096
Portuguese ANEW | Portuguese | 0.893 | 0.779 | 0.106 | 0.103 | 0.074 | 0.081
S.-Gonzalez et al. | Spanish | 0.808 | 0.689 | 0.100 | 0.095 | 0.074 | 0.072
Kapucu et al. | Turkish | 0.571 | 0.373 | 0.165 | 0.127 | 0.125 | 0.101

We performed a similar comparison to evaluate the result quality in distinct
languages, grouping prediction results by language. It was possible to
conclude that our best model yields good results in most languages. The most
challenging languages in terms of word prediction are Finnish and Turkish,
with the model seemingly excelling at Portuguese, Mandarin, and English, to
name a few. The lower scores observed for Finnish and Turkish can be explained
by the small sample of training data in those languages, respectively 0.48%
and 1.57% of the entire dataset, as well as by the complex morphology and
productive compounding associated with these languages, as found by Buechel et
al. [11].

As for the short texts, compared in detail in Table 5, the most challenging
language was German. On this subject, we note that the German training sample
contains the metaphorical instances of the COMETA dataset, which can explain
the gap in the results for this language. Predicting valence in English also
proved demanding. If analyzed in detail, the results are heavily influenced by
the IEMOCAP dataset, which makes up for 46% of the English short text corpus.
IEMOCAP is a particular dataset, created through the video recording of actors
performing scripts designed to contain select emotions [12]. We used the
transcriptions of the audio, which is annotated for valence and arousal in the
dataset. Contrarily to all other datasets, these instances were annotated from
videos, which can portray a large range of sentiments for the same textual
script, depending on aspects such as posture and intonation of the actors.
This implies that annotations range over a broader scope too, which likely
affects the quality of the prediction results.

Stemming from these last conclusions, we performed one more separate
experiment. Considering the same training setting, we trained the model with a
combined dataset not containing the two seemingly troublesome datasets, COMETA
stories and IEMOCAP. Compared to previous results, the Pearson’s $\rho$ for
valence increased from $0.8095$ to $0.8423$, and arousal’s correlation
increased from $0.6974$ to $0.7107$. Performance gains were observed for all
tested languages. In particular, valence and arousal correlation values for
German short texts increased 13% and 7%, and most noticeably for English they
increased 31% and 11%, respectively. This took the scores obtained for these
two languages, which are well represented in the training instances, to levels
akin to most other languages, and explained the previously noticed discrepancy
in the evaluations.

Table 5: Evaluation results for individual languages on the short text datasets, using the XLM-RoBERTa-large model and considering the MSE loss function. Language | $\mathrm{\rho_{V}}$ | $\mathrm{\rho_{A}}$ | $\mathrm{RMSE_{V}}$ | $\mathrm{RMSE_{A}}$ | $\mathrm{MAE_{V}}$ | $\mathrm{MAE_{A}}$
---|---|---|---|---|---|---
English | 0.592 | 0.719 | 0.118 | 0.138 | 0.085 | 0.096
Mandarin | 0.892 | 0.657 | 0.077 | 0.100 | 0.059 | 0.080
German | 0.619 | 0.533 | 0.179 | 0.117 | 0.133 | 0.090
Portuguese | 0.935 | 0.694 | 0.115 | 0.124 | 0.082 | 0.100
Polish | 0.868 | 0.607 | 0.113 | 0.111 | 0.082 | 0.089

###  5.3 Results in Zero-Shot Settings

With the previous results in mind, a question remained on whether our best
model could generalize well to other languages in which it was not trained on.
For that purpose, two other XLM-RoBERTa-large models were fine-tuned under the
same training setup. Specifically, these models were trained with all the data
from the merged dataset except for either the Polish or the Portuguese
instances. These instances were saved for subsequent zero-shot evaluations,
separately focusing on each of these languages. This trial aimed to assert
whether the proposed approach can generalize to a language not used for
training. Polish and Portuguese were chosen for this purpose, as both these
languages are represented in our dataset, simultaneously with word and short
text instances. Despite being reasonably popular languages, they are not as
extensively present as English, and thus they allow us to adequately simulate
the scenario of testing the proposed model on a new language not seen during
training, and also not seen extensively during the model pre-training stage
(i.e., the DiltilBERT and XML-RoBERTa models, despite being multilingual, have
seen much more English training data in comparison to other languages).

We can compare the results of these zero-shot experiments, presented in Table
6, with the results obtained for the Polish and Portuguese subsets of
predictions presented previously in Table 4. When comparing correlation and
error metrics, we found overall worse results. However, the difference is not
significant, and the results are in fact higher than some of the observed
results for other languages on which the model was fine-tuned on. The zero-
shot performance for both languages shows promising prospects for the
application of the proposed approach to different languages without available
emotion corpora.

Table 6: Zero-shot evaluation for Polish (PL) and Portuguese (PT) data, using the XLM-RoBERTa-large model and considering the MSE loss function. Training on | Predicting on | $\mathrm{\rho_{V}}$ | $\mathrm{\rho_{A}}$ | $\mathrm{RMSE_{V}}$ | $\mathrm{RMSE_{A}}$ | $\mathrm{MAE_{V}}$ | $\mathrm{MAE_{A}}$
---|---|---|---|---|---|---|---
All |  | 0.839 | 0.648 | 0.101 | 0.103 | 0.072 | 0.080
All excl. PL | Any PL input | 0.818 | 0.618 | 0.111 | 0.135 | 0.080 | 0.108
All |  | 0.895 | 0.756 | 0.108 | 0.107 | 0.075 | 0.084
All excl. PT | Any PT input | 0.886 | 0.735 | 0.112 | 0.112 | 0.079 | 0.088
All |  | 0.833 | 0.631 | 0.100 | 0.102 | 0.071 | 0.079
All excl. PL | PL words | 0.814 | 0.647 | 0.111 | 0.135 | 0.079 | 0.108
All |  | 0.893 | 0.779 | 0.106 | 0.103 | 0.074 | 0.081
All excl. PT | PT words | 0.906 | 0.777 | 0.102 | 0.107 | 0.071 | 0.084
All |  | 0.868 | 0.607 | 0.113 | 0.111 | 0.082 | 0.089
All excl. PL | PL short texts | 0.860 | 0.487 | 0.113 | 0.135 | 0.085 | 0.108
All |  | 0.935 | 0.694 | 0.115 | 0.124 | 0.082 | 0.100
All excl. PT | PT short texts | 0.923 | 0.627 | 0.155 | 0.135 | 0.121 | 0.109

##  6 Conclusions and Future Work

This paper presented a bi-dimensional and multilingual model to predict real-
valued emotion ratings from instances of text. First, a multi-language emotion
corpus of words and short texts was assembled. This goes in contrast to most
previous studies, which focused solely on words or texts in a single language.
The corpus, consisting of 128,987 instances, features annotations for the
psycho-linguistic dimensions of Valence and Arousal (VA), spanning 13
different languages. Subsequently, DistilBERT and XLM-RoBERTa models were
trained for VA prediction using the multilingual corpus. The evaluation
methodology used Pearson’s $\rho$ and two error metrics to assess the results.
Overall, the predicted ratings showed a high correlation with human ratings,
and the results are in line with those of previous monolingual predictive
approaches. Additionally, this research highlights the challenge of predicting
arousal to the same degree of confidence of predicting valence from text. In
sum, the evaluation of our best model showed competitive results against
previous approaches, having the advantage of generalization to different
languages and different types of text.

An interesting idea to explore in future work concerns applying uncertainty
quantification444https://mapie.readthedocs.io/en/latest/ to the predicted
ratings, for instance as explained by Angelopoulos and Bates [2]. Instead of
predicting a single pair of values for the valence and arousal ratings, the
aim would be to predict a high confidence interval of values in which valence
and arousal are contained. Future work can also address the study of data
augmentation methods (e.g., based on machine translation), in an attempt to
further improve result quality and certainty.

Another interesting direction for future work concerns extending the work
reported in this paper to consider multimodal emotion estimation. Instead of
the models considered here, we can consider fine-tuning a large multilingual
vision-and-language model555https://huggingface.co/laion/CLIP-ViT-H-14-frozen-
xlm-roberta-large-laion5B-s13B-b90k such as CLIP [13], combining the textual
datasets together with affective image datasets like the International
Affective Picture System (IAPS) [35], the Geneva Affective PicturE Database
(GAPED) [19], the Nencki Affective Picture System (NAPS) [37], the Open
Affective Standardized Image Set (OASIS) [34], or others [14, 31].

## Acknowledgements

This research was supported by the European Union’s H2020 research and
innovation programme, under grant agreement No. 874850 (MOOD), as well as by
the Portuguese Recovery and Resilience Plan (RRP) through project
C645008882-00000055 (Responsible.AI), and by Fundação para a Ciência e
Tecnologia (FCT), through the INESC-ID multi-annual funding with reference
UIDB/50021/2020, and through the projects with references DSAIPA/DS/0102/2019
(DEBAQI) and PTDC/CCI-CIF/32607/2017 (MIMU).

## References

  * [1] Alm, C.O., Roth, D., Sproat, R.: Emotions from text: Machine learning for text-based emotion prediction. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (2005)
  * [2] Angelopoulos, A.N., Bates, S.: A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 (2021)
  * [3] Barron, J.T.: A general and adaptive robust loss function. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)
  * [4] Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25(1), 49–59 (1994)
  * [5] Bradley, M.M., Lang, P.J.: Affective Norms for English Words (ANEW): Instruction manual and affective ratings. Technical report C-1, The center for research in psychophysiology, University of Florida (1999)
  * [6] Bradley, M.M., Lang, P.J.: Affective Norms for English Text (ANET): Affective ratings of text and instruction manual. Techical Report D-1, University of Florida (2007)
  * [7] Buechel, S., Hahn, U.: Emotion analysis as a regression problem — dimensional models and their implications on emotion representation and metrical evaluation. In: Proceedings of the European Conference on Artificial Intelligence (2016)
  * [8] Buechel, S., Hahn, U.: EmoBank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In: Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (2017)
  * [9] Buechel, S., Hahn, U.: A flexible mapping scheme for discrete and dimensional emotion representations: Evidence from textual stimuli. In: Proceedings of the Annual Meeting of the Cognitive Science Society (2017)
  * [10] Buechel, S., Hahn, U.: Readers vs. writers vs. texts: Coping with different perspectives of text understanding in emotion annotation. In: Proceedings of the Linguistic Annotation Workshop (2017)
  * [11] Buechel, S., Rücker, S., Hahn, U.: Learning and evaluating emotion lexicons for 91 languages. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (2020)
  * [12] Busso, C., Bulut, M., Lee, C.C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J.N., Lee, S., Narayanan, S.S.: IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation 42(4), 335–359 (2008)
  * [13] Carlsson, F., Eisen, P., Rekathati, F., Sahlgren, M.: Cross-lingual and multilingual CLIP. In: Proceedings of the Language Resources and Evaluation Conference (2022)
  * [14] Carretié, L., Tapia, M., López-Martín, S., Albert, J.: EmoMadrid: An emotional pictures database for affect research. Motivation and Emotion 43(6) (2019)
  * [15] Citron, F.M.M., Lee, M., Michaelis, N.: Affective and psycholinguistic norms for German conceptual metaphors (COMETA). Behavior Research Methods 52(3), 1056–1072 (2020)
  * [16] Citron, F.M., Cacciari, C., Kucharski, M., Beck, L., Conrad, M., Jacobs, A.M.: When emotions are expressed figuratively: Psycholinguistic and Affective Norms of 619 Idioms for German (PANIG). Behavior Research Methods 48(1), 91–111 (2016)
  * [17] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (2020)
  * [18] Ćoso, B., Guasch, M., Ferré, P., Hinojosa, J.A.: Affective and concreteness norms for 3,022 Croatian words. Quarterly Journal of Experimental Psychology 72(9), 2302–2312 (2019)
  * [19] Dan-Glauser, E.S., Scherer, K.R.: The Geneva Affective Picture Database (GAPED): A new 730-picture database focusing on valence and normative significance. Behavior research methods 43(2) (2011)
  * [20] Du, S., Zhang, X.: Aicyber’s system for IALP 2016 shared task: Character-enhanced word vectors and boosted neural networks. In: Proceedings of the International Conference on Asian Language Processing (2016)
  * [21] Eilola, T.M., Havelka, J.: Affective norms for 210 British English and Finnish nouns. Behavior Research Methods 42(1), 134–140 (2010)
  * [22] Ekman, P.: An argument for basic emotions. Cognition & Emotion 6(3-4), 169–200 (1992)
  * [23] Fairfield, B., Ambrosini, E., Mammarella, N., Montefinese, M.: Affective norms for Italian words in older adults: age differences in ratings of valence, arousal and dominance. PloS One 12(1), e0169472 (2017)
  * [24] Francisco, V., Hervás, R., Peinado, F., Gervás, P.: EmoTales: Creating a corpus of folk tales with emotional annotations. Language Resources and Evaluation 46(3), 341–381 (2012)
  * [25] Gilet, A.L., Grühn, D., Studer, J., Labouvie-Vief, G.: Valence, arousal, and imagery ratings for 835 French attributes by young, middle-aged, and older adults: The French Emotional Evaluation List (FEEL). European Review of Applied Psychology 62(3), 173–181 (2012)
  * [26] Hollis, G., Westbury, C., Lefsrud, L.: Extrapolating human judgments from skip-gram vector representations of word meaning. Quarterly Journal of Experimental Psychology 70, 1–45 (06 2016)
  * [27] Imbir, K.K.: Affective Norms for 4900 Polish Words Reload (ANPW_R): Assessments for valence, arousal, dominance, origin, significance, concreteness, imageability and, age of acquisition. Frontiers in Psychology 7, 1081 (2016)
  * [28] Imbir, K.K.: The Affective Norms for Polish Short Texts (ANPST) database properties and impact of participants’ population and sex on affective ratings. Frontiers in Psychology 8, 855 (2017)
  * [29] Kanske, P., Kotz, S.A.: Leipzig affective norms for German: A reliability study. Behavior Research Methods 42(4), 987–991 (2010)
  * [30] Kapucu, A., Kılıç, A., Özkılıç, Y., Sarıbaz, B.: Turkish emotional word norms for arousal, valence, and discrete emotion categories. Psychological Reports 124(1), 188–209 (2021)
  * [31] Kim, H.R., Kim, Y.S., Kim, S.J., Lee, I.K.: Building emotional machines: Recognizing image emotions through deep neural networks. IEEE Transactions on Multimedia 20(11) (2018)
  * [32] Kratzwald, B., Ilić, S., Kraus, M., Feuerriegel, S., Prendinger, H.: Deep learning for affective computing: Text-based emotion recognition in decision support. Decision Support Systems 115, 24–35 (2018)
  * [33] Kron, A., Goldstein, A., Lee, D., Gardhouse, K., Anderson, A.: How are you feeling? revisiting the quantification of emotional qualia. Psychological Science 24(8), 1503–1511 (2013)
  * [34] Kurdi, B., Lozano, S., Banaji, M.R.: Introducing the Open Affective Standardized Image Set (OASIS). Behavior Research Methods 49(2) (2017)
  * [35] Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International Affective Picture System (IAPS): Affective ratings of pictures and instruction manual. Tech. rep., NIMH Center for the Study of Emotion and Attention (2005)
  * [36] Lee, L.H., Li, J.H., Yu, L.C.: Chinese EmoBank: Building valence-arousal resources for dimensional sentiment analysis. Transactions on Asian and Low-Resource Language Information Processing 21(4), 1–18 (2022)
  * [37] Marchewka, A., Żurawski, Ł., Jednoróg, K., Grabowska, A.: The Nencki Affective Picture System (NAPS): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database. Behavior Research Methods 46(2) (2014)
  * [38] Mohammad, S.: Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (2018)
  * [39] Monnier, C., Syssau, A.: Affective Norms for French words (FAN). Behavior Research Methods 46(4), 1128–1137 (2014)
  * [40] Moors, A., De Houwer, J., Hermans, D., Wanmaker, S., van Schie, K., Harmelen, A.L., De Schryver, M., Winne, J., Brysbaert, M.: Norms of valence, arousal, dominance, and age of acquisition for 4,300 Dutch words. Behavior Research Methods 45(1), 169–177 (2013)
  * [41] Paltoglou, G., Theunis, M., Kappas, A., Thelwall, M.: Predicting emotional responses to long informal text. IEEE Transactions on Affective Computing 4(1), 106–115 (2013)
  * [42] Pinheiro, A.P., Dias, M., Pedrosa, J., Soares, A.P.: Minho Affective Sentences (MAS): Probing the roles of sex, mood, and empathy in affective ratings of verbal stimuli. Behavior Research Methods 49(2), 698–716 (2017)
  * [43] Pires, T., Schlinger, E., Garrette, D.: How multilingual is multilingual BERT? In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (2019)
  * [44] Preoţiuc-Pietro, D., Schwartz, H.A., Park, G., Eichstaedt, J., Kern, M., Ungar, L., Shulman, E.: Modelling valence and arousal in Facebook posts. In: Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (2016)
  * [45] Recchia, G., Louwerse, M.: Reproducing affective norms with lexical co-occurrence statistics: Predicting valence, arousal, and dominance. Quarterly Journal of Experimental Psychology 68, 1–41 (2014)
  * [46] Riegel, M., Wierzba, M., Wypych, M., Żurawski, Ł., Jednoróg, K., Grabowska, A., Marchewka, A.: Nencki Affective Word List (NAWL): the cultural adaptation of the Berlin Affective Word List–Reloaded (BAWL-R) for Polish. Behavior Research Methods 47(4), 1222–1236 (2015)
  * [47] Russell, J.: A circumplex model of affect. Journal of Personality and Social Psychology 39, 1161–1178 (1980)
  * [48] Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)
  * [49] Schmidtke, D.S., Schröder, T., Jacobs, A.M., Conrad, M.: ANGST: Affective norms for German sentiment terms derived from the affective norms for English words. Behavior Research Methods 46(4), 1108–1118 (2014)
  * [50] Scott, G.G., Keitel, A., Becirspahic, M., Yao, B., Sereno, S.C.: The Glasgow norms: Ratings of 5,500 words on nine scales. Behavior Research Methods 51(3), 1258–1270 (2019)
  * [51] Sedoc, J., Preoţiuc-Pietro, D., Ungar, L.: Predicting emotional word ratings using distributional representations and signed clustering. In: Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (2017)
  * [52] Shad Akhtar, M., Ghosal, D., Ekbal, A., Bhattacharyya, P., Kurohashi, S.: A Multi-task Ensemble Framework for Emotion, Sentiment and Intensity Prediction. arXiv preprint arXiv:1808.01216 (2018)
  * [53] Soares, A., Comesaña, M., Pinheiro, A., Simões, A., Frade, S.: The adaptation of the Affective Norms for English Words (ANEW) for European Portuguese. Behavior Research Methods 44(1), 256–269 (2012)
  * [54] Söderholm, C., Häyry, E., Laine, M., Karrasch, M.: Valence and arousal ratings for 420 Finnish nouns by age and gender. PloS One 8(8), e72859 (2013)
  * [55] Stadthagen-Gonzalez, H., Imbault, C., Pérez Sánchez, M.A., Brysbaert, M.: Norms of valence and arousal for 14,031 Spanish words. Behavior Research Methods 49(1), 111–123 (2017)
  * [56] Tris Atmaja, B., Akagi, M.: Evaluation of error- and correlation-based loss functions for multitask learning dimensional speech emotion recognition. Journal of Physics: Conference Series 1896, 012004 (2021)
  * [57] Verheyen, S., De Deyne, S., Linsen, S., Storms, G.: Lexicosemantic, affective, and distributional norms for 1,000 Dutch adjectives. Behavior Research Methods 52(3), 1108–1121 (2020)
  * [58] Vo, M.L.H., Conrad, M., Kuchinke, L., Urton, K., Hofmann, M.J., Jacobs, A.M.: The Berlin Affective Word List Reloaded (BAWL-R). Behavior Research Methods 41(2), 534–538 (2009)
  * [59] Wang, J., Yu, L.C., Lai, K., Zhang, X.: Tree-structured regional CNN-LSTM model for dimensional sentiment analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing 28, 581–591 (2020)
  * [60] Warriner, A., Kuperman, V., Brysbaert, M.: Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods 45(4), 1191–1207 (2013)
  * [61] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A.: Transformers: State-of-the-art natural language processing. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (2020)
  * [62] Wu, C., Wu, F., Huang, Y., Wu, S., Yuan, Z.: THU_NGN at IJCNLP-2017 task 2: Dimensional sentiment analysis for Chinese phrases with deep LSTM. In: Proceedings of the International Joint Conference on Natural Language Processing (2017)
  * [63] Xie, H., Lin, W., Lin, S., Wang, J., Yu, L.C.: A multi-dimensional relation model for dimensional sentiment analysis. Information Sciences 579, 832–844 (2021)
  * [64] Xu, X., Li, J., Chen, H.: Valence and arousal ratings for 11,310 simplified Chinese words. Behavior Research Methods 54(1), 26–41 (2022)
  * [65] Yee, L.: Valence, arousal, familiarity, concreteness, and imageability ratings for 292 two-character Chinese nouns in Cantonese speakers in Hong Kong. PloS One 12(3), e0174569 (2017)
  * [66] Yu, L.C., Lee, L.H., Hao, S., Wang, J., He, Y., Hu, J., Lai, K.R., Zhang, X.: Building chinese affective resources in valence-arousal dimensions. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (2016)
  * [67] Yu, L.C., Wang, J., Lai, K.R., Zhang, X.: Pipelined neural networks for phrase-level sentiment intensity prediction. IEEE Transactions on Affective Computing 11(3), 447–458 (2020)