# Persian Emotion Detection using ParsBERT and Imbalanced Data Handling
Approaches

Amirhossein Abaskohi1, Nazanin Sabri2, Behnam Bahrak1
1School of Electrical and Computer Engineering
College of Engineering, University of Tehran, Tehran, Iran
{amir.abaskohi<EMAIL_ADDRESS>
2School of Computer Science University of California, San Diego, La Jolla, CA
92093
<EMAIL_ADDRESS>
###### Abstract

Emotion recognition is one of the machine learning applications which can be
done using text, speech, or image data gathered from social media spaces.
Detecting emotion can help us in different fields, including opinion mining.
With the spread of social media, different platforms like Twitter have become
data sources, and the language used in these platforms is informal, making the
emotion detection task difficult. EmoPars and ArmanEmo are two new human-
labeled emotion datasets for the Persian language. These datasets, especially
EmoPars, are suffering from inequality between several samples between two
classes. In this paper, we evaluate EmoPars and compare them with ArmanEmo.
Throughout this analysis, we use data augmentation techniques, data re-
sampling, and class-weights with Transformer-based Pretrained Language
Models(PLMs) to handle the imbalance problem of these datasets. Moreover,
feature selection is used to enhance the models’ performance by emphasizing
the text’s specific features. In addition, we provide a new policy for
selecting data from EmoPars, which selects the high-confidence samples; as a
result, the model does not see samples that do not have specific emotion
during training. Our model reaches a Macro-averaged F1-score of 0.81 and 0.76
on ArmanEmo and EmoPars, respectively, which are new state-of-the-art results
in these benchmarks.111Code is publicly available at
https://github.com/AmirAbaskohi/Persian-Emotion-Detection-using-ParsBERT-and-
Imbalanced-Data-Handling-Approaches

##  1 Introduction

In people’s social and professional lives, emotional expression and detection
are crucial due to the fact that systems can change their responses and
behavioral patterns in response to human emotions, improving the naturalness
of interactionsTzirakis et al. (2019). They strongly influence the variety and
quality of our experiences and social interactions, and they are intimately
tied to our cognitive and communicative abilitiesDolan (2002). Social
networks, for example, have evolved into mediums for users to express their
emotionsTettegah and Noble (2015). Additionally, they are valuable tools to
keep up with the news. Furthermore, finding new social trends based on
stakeholder groups’ or the general public’s expectations, attitudes, and
dispositions is the aim of opinion research. Opinion research is often used in
policymaking to more accurately predict the effects of proposed changes and to
better communicate the projected advantages and drawbacksSobkowicz et al.
(2012). As a result, we use the data from these platforms for emotion
recognition for opinion mining.

Sentiment analysis tools are helpful if we just want to check the polarity of
a sentence or documentAndalibi and Buss (2020). In this paper, we want to go
deeper, trying to detect a social media post’s emotion since sentiment
analysis can find whether a sample is positive or not, whereas emotion is a
more complex concept. With emotion recognition, we can get more granular data,
bringing out the real emotions of the user who created that specific post.

The precision with which people can gauge the emotions of others varies
greatlyGendron et al. (2014) and this is a reason why automated systems have
problems in this task. Automating the identification of facial emotions from
video, spoken expressions from audio, written expressions from text, and
physiology, as measured by wearables, has received the most attention to date.
The technology generally performs best when it integrates several modalities
into the environmentTzirakis et al. (2017). However, gathering enough
synchronous multi-modal data is not easy, and as a result, there is a need for
improving single-modal models. Here our focus is using text data for emotion
recognition.

One of the usages of emotion recognition is in detecting psychological
disorders. Severe social interaction deficiencies are a hallmark of borderline
personality disorder (BPD), which may be related to abnormalities in emotion
perceptionMinzenberg et al. (2006); Domes et al. (2009). Furthermore, emotion
recognition is essential for society’s satisfaction evaluation. This
evaluation can be used in different situations like an election or even a
recent pandemic, COVID-19Esmin et al. (2012); Shyry et al. (2020); Meléndez et
al. (2020). Moreover, emotion recognition can be used in examining the well-
being of people under different circumstances. For instance, in Schlegel et
al. (2021), emotion recognition was used as a predictor of well-being during
the COVID-19 pandemic. Consequently, emotion detection is a helpful research
line, and using social media, like Twitter, whose data type is textual data,
plays an important role here since they provide a massive amount which makes
the data collection pipeline a lot easier. However, the mentioned papers are
in English, and in this paper, we propose an emotion recognition model to use
in the areas above. Despite several pieces of research in English, in Persian,
mining emotions from text and other media needs more research as the provided
datasets for this task in Persian is no specific model for these datasets is
provided. EmoParsSabri et al. (2021) and ArmanEmoMirzaee et al. (2022) are two
famous textual emotion datasets. Although some methods for detecting emotions
are provided in Mirzaee et al. (2022), in this paper, we try to improve those
approaches by solving the imbalance dataset issue and providing a feature-
selecting method.

In order to achieve a high F1-score in Persian emotion detection, we attempt
to use transformer architectureVaswani et al. (2017) and various methods for
handling imbalanced data such as undersampling for the major class, suitable
class weights, F1-Cross-Entropy loss function, and data augmentation
techniques. Our method reaches a state-of-the-art result in both EmoParsSabri
et al. (2021) and ArmanEmoMirzaee et al. (2022) benchmarks. In general, the
main contributions of this paper are:

  * •

Introducing new data sampling on the EmoPars dataset to improve the results on
this dataset. Our strategy aims to use more data in contrast to the
methodology presented in Mirzaee et al. (2022), which eliminates 96% of the
tweets in this dataset.

  * •

Improving the model’s fairness in EmoPars benchmark by using random
undersampling for major class and Easy Data Augmentation(synonym replacement,
word addition, word elimination, and shuffling)to decrease the differences in
the number of samples in classes and using proper class weights and F1-Cross-
Entropy loss function.

  * •

Introducing a feature extraction method to help the model to pay more
attention to specific features in the data, which has much information about
the emotion of the sample, such as hashtags and emojis.

##  2 Related Works

In this section, we first discuss emotion detection studies that have been
previously investigated, especially in Persian texts. We then provide a brief
review of approaches for handling imbalanced datasets. Finally, we will look
at the feature extraction approaches in English and Persian texts.

###  2.1 Emotion Detection Methods In Texts

People still prefer using text to communicate their thoughts regarding other
people, objects, or eventsWARSI (2021). Due to the nature of the data, it is
still more challenging to extract emotion from the text since the whole text,
including metaphors and sarcasm, should be consideredSkórzewski (2022).
Sometimes several emotions might be present in a single paragraph.
Additionally, some words in texts have many meanings, while others express the
same feeling with multiple words.

There are several methods for detecting textual emotions. Affective computing
includes tasks like emotion recognition, and diverse academics have divided
the computational techniques employed in this field into several categories.
These researchers’ methodologies may be broadly divided into four groups:
machine learning method, lexicon-based method, keyword-based method, and
hybrid methodAndalibi and Buss (2020). Machine learning methods got more
popular due to the adequate performance of word embedding and transformer
architecture in sentiment analysis tasks, and this was not just in English.

Emotion recognition in English has been investigated in several studies. In
speech emotion recognition, various approaches, including self-supervised,
multilingual fusion, and knowledge transfer, were usedNeumann et al. (2018);
Sarkar and Etemad (2020); Iosifov et al. (2022). In text emotion recognition,
using the psychological frameworks offered by Russell (1980) and Ekman (1992b)
to map documents, phrases, and words to a set of emotions was among the first
approaches. The problem with traditional methodologies was then overcome by
machine learning-based approaches, leading to notable improvements in text
emotional recognitionZhang et al. (2016). In text analysis research, deep
learning-based methods have emerged as the cutting edge. They have
demonstrated the capacity to pick up on the key emotion characteristics, which
has significantly improved text emotion identification performanceAlswaidan
and Menai (2020). Convolutional Neural Networks(CNNs) and Long Short Term
Memories(LSTMs) are among these approachesShrivastava et al. (2019); Mahto and
Yadav (2022); Alla et al. (2022). With the introduction of Transformer
architecture, transformer-based, especially BERT-based models, were used for
emotion recognition on text data and multi-modal scenariosLi et al. (2020);
Acheampong et al. (2021).

Although emotion recognition in Persian has been studied in speech and text
domainsHamidi and Mansoorizade (2012); Pourebrahim et al. (2021); Yazdani et
al. (2021), very few studies have been conducted on emotion detection from
Persian textSadeghi et al. (2021). One of the reasons for this is the few
Persian datasets available for emotion detection. After the introduction of
EmoParsSabri et al. (2021), the study of this topic has started with more
effort. This dataset includes Persian Tweets and will be introduced in Section
3.1. However, emotion detection is not investigated in the provided paper, and
just the data and its statistics are provided. ArmanEmoMirzaee et al. (2022)
is another dataset that includes other data sources like Instagram. In
addition to introducing a new dataset, they performed transformer-based
approaches including ParsBERTMehrdad Farahani and Manthouri (2021) and XLM-
RoBERTaConneau et al. (2020) to evaluate their dataset and EmoPars dataset.
These models will be used in this paper as well. Moreover, we will investigate
different approaches for solving the imbalance problem of these datasets and
improving the performance of the model as well.

In Persian emotion detection, Sadeghi et al.Sadeghi et al. (2021) used a
hybrid approach by combining the cognitive features and Word2VecMikolov et al.
(2013) embedding. The emotional construction, keywords, and parts of speech
are utilized to construct the cognitive features. Then the features are passed
to a Gated Recurrent Unit (GRU)Cho et al. (2014) network for classification.
However, this method needs human supervision which makes it hard to use in
other scenarios in this paper our goal is to have a more automatic approach.
Since we are dealing with imbalanced data sets illustrated in Section 3.1, we
need to refer to previous work on how to deal with these datasets.

###  2.2 Handling Imbalanced Datasets

Unbalanced datasets have become increasingly important in machine learning in
recent years since if the data set is imbalanced, then you get a high accuracy
just by predicting the majority class. However, you fail to capture the
minority class. Hence, the model does not give good results in production and
unseen data since it would completely ignore the minority class in favor of
the majority class. The ratio between the majority and minority classes can
range from 100:1 to 1000:1Makki et al. (2019), meaning that more people belong
to the majority class than the minority class. As a consequence, some helpful
information about the data itself, which could be necessary for building rule-
based classifiers, may be lost.

In data-level approaches, Majority Weighted Minority Oversampling Technique
(MWMOTE)Barua et al. (2012) has been introduced to use oversampling and
weights for major classes to solve the negative effect of imbalanced data in
20 real-world datasets. In another study, Neighbourhood Balanced Bagging
(NBBag)Błaszczyński and Stefanowski (2015) was introduced. In this approach,
sampling probabilities of examples are modified according to the class
distribution in their neighborhood. Two versions are considered: the first one
keeps a larger size of bootstrap samples by hybrid over-sampling, and the
other reduces this size with stronger under-sampling.

In algorithm-level approaches, defining class weights for machine learning
models has been used in random forest algorithms to overcome imbalanced data
problems in medical dataZhu et al. (2018). Furthermore, different loss
functions have been used to increase the model performance by concentrating
more on minor data in the process of learningZhang et al. (2020); Li et al.
(2021). Also, data augmentation methods have been studied for solving the
imbalanced data problemdos Santos Tanaka and Aranha (2019); Jiang and Ge
(2020); Abaskohi et al. (2022).

###  2.3 Feature Extraction

Specific features of social media texts like emojis and hashtags usually carry
valuable informationAlfano et al. (2021). In English, Sari et al. (2014) and
Lemmens et al. (2020) used these features to improve the result of their model
by giving them separately to the model. This method has been used in Persian
for rumor detection, and verification Mahmoodabad et al. (2018); Jahanbakhsh-
Nagadeh et al. (2021). In this paper, in addition to using emojis and
hashtags, we used Part of Speech(POS) and misspelled words. We believe
misspelled words are usually used to grab more attention, which is a reason
for having a piece of helpful information. Furthermore, as the language models
used in our experiments do not follow the formal sentence structure used in
the language models’ pre-training phase, POS tagging helps the model better
understand the sentence structure.

##  3 Methods and Data

This section explains the properties of the dataset utilized in this work. In
addition, we will contrast this dataset with other Persian datasets currently
accessible for emotion recognition. Finally, we will outline our process to
develop a superior generic model for emotion recognition.

###  3.1 Datasets

Emotion detection, as one of the complex tasks in natural language processing,
not only has challenges in detecting tweets but also the data, as the vital
part of the deep learning model, has its challenges.

EmoParsSabri et al. (2021), is a dataset of 30,000 Persian Tweets labeled with
Ekman’s six basic emotions (Anger, Fear, Happiness, Sadness, Hatred, and
Wonder)Ekman (1992a). The first publicly available Persian emotion dataset
provides enough information for further data analysis. Although other
available datasets like ArmanEmoMirzaee et al. (2022) believe each sentence
can only contain one emotion, it seems emotion detection is a multi-label
taskBhowmick et al. (2009); Deng and Ren (2020); Fei et al. (2021). In
contrast, the EmoPars dataset considers this task as a multi-label task. In
this paper, we use multi-label training in one of our experiments on EmoPars;
however, using multiple binary classifiers reaches better results since in
this approach handling data imbalance is easier and the model has a simple
task as well since it faces just two classes.

EmoPars uses information obtained from Twitter’s official developer APISabri
et al. (2021). They gather tweets using multiple keywords, including articles
and prepositions, from various topics to avoid data being skewed by any topic
or debate. Include at least a few tweets from each period in your tweets. As
in the published dataset, the vote counts have been provided in the range of
[0, 5] for classification; various policies can be used to create a
classification (whether binary, multi-label, or multi-class). Here we try to
find out the best policy for classification.

ArmanEmoMirzaee et al. (2022) is another dataset whose data is gathered from
Twitter, Instagram, and customers’ comments on Digikala (an online shopping
platform), and it consists of approximately 2.55 million raw samples(1.5
million from Twitter, 1 million from Instagram, and 50K from Digikala). Unlike
EmoPars, they removed samples whose lengths felt outside of a specific range
or included specific user IDs or links in the annotation step. Furthermore,
they controlled the class balance and considered only using samples with one
emotion and removed samples with multiple or no emotions.

###  3.2 Data Selection Policy

Data annotation is not a simple task, especially in emotion detection, where
some emotions are debatable, and detecting whether they have a specific
emotion is more challenging and may result in incorrectly labeled data. The
voting method used in this paper’s datasets, EmoPars and ArmanEmo, can cause
some disagreement in the data. Although ArmanEmo tries to handle this problem
by removing samples with no emotion or multiple emotions, EmoPars does not.

One possible solution might be using a threshold to annotate samples.
Additionally, tweets close to this level can result in other issues. Using a
specific threshold to decide whether a tweet contains an emotion can be
helpful but finding this threshold may be challenging. In addition, tweets
near this threshold may cause different problems.

Although we have tried different threshold values, and the results are
discussed in Section 4, we are introducing a new strategy for selecting
valuable data for training our model. In this approach, the tweet is not
considered when two or three individuals out of 5 annotators agree that a
tweet contains a particular emotion.

We have also selected thirty tweets from each emotion’s label and labeled them
ourselves. The results in Table 1 show us that our approach is acceptable. In
general, there is no confidence in labels two and three. Nonetheless, about
Wonder emotion, our general idea is applicable; the crucial point here is the
hardness of predicting Wonder label. It is important to note that fewer than
thirty tweets with the label "5" for each emotion. Because of this, we have
included this label in all tweets.

Emotion | Labels
---|---
0 | 1 | 2 | 3 | 4 | 5
Anger | Predicted 0 | 23 | 21 | 17 | 15 | 9 | 3
Predicted 1 | 7 | 9 | 13 | 15 | 21 | 21
Fear | Predicted 0 | 24 | 23 | 18 | 16 | 10 | 1
Predicted 1 | 6 | 7 | 12 | 14 | 20 | 1
Sadness | Predicted 0 | 23 | 28 | 27 | 18 | 10 | 6
Predicted 1 | 7 | 2 | 3 | 12 | 20 | 22
Happiness | Predicted 0 | 28 | 25 | 18 | 16 | 12 | 1
Predicted 1 | 2 | 5 | 12 | 14 | 18 | 10
Wonder | Predicted 0 | 27 | 28 | 21 | 17 | 15 | 1
Predicted 1 | 3 | 2 | 9 | 13 | 15 | 7
Hatred | Predicted 0 | 22 | 20 | 15 | 16 | 10 | 1
Predicted 1 | 8 | 10 | 15 | 14 | 20 | 12
Table 1: The result of labeling thirty tweets of each label of each emotion to
check whether removing tweets with two or three agreed annotators is an
acceptable approach.

###  3.3 Data Augmentation

One of the main challenges in solving emotion detection using the EmoPars
dataset is the unbalanced data. Figure 1 shows a big difference between 0 and
1 labels in each emotion. This unbalanced data will result in biased results.
There are several approaches to solve this issue.

Figure 1: An overview of the number of data samples in each emotion. There are
1 to 100 different labels for each emotion. With this ratio, the model will
predict 0 in most cases.

Although there is a massive difference in our case, it may not only not help
us but also weaken the result, as was shown inAquino et al. (2017). In
addition, our analysis of the length of the tweets which can be seen in Figure
2, shows us that creating 100 new views of the initial sample may create
tweets in which the sentiment of the sentence is different or tweets with the
same sentiment.

Mutation-based data augmentation, which uses synonym replacement, word
elimination, word addition, and word swap, has been shown to improve the
models’ results in various tasksKafle et al. (2017); Parida and Motlicek
(2019); Shorten et al. (2021); Abaskohi et al. (2022). We used this data
augmentation to create more samples of label 1 of each emotion to decrease the
difference between classes.

Figure 2: Length of each tweet by words. As most samples have less than 20
words, data augmentation would create another view of the initial tweet.

In our data augmentation, we used NLPAugMa (2019) python module’s
ContextualWordEmbsAug and RandomWordAug methods. This module uses a language
model for word insertion and synonym replacement. We used ParsBERTMehrdad
Farahani and Manthouri (2021) as the language model in this section. Due to
the different ratios between 0 and 1 classes in each emotion, the number of
augmented samples per initial sample is different per emotion. Table 2 shows
data augmentation hyper-parameters used for each emotion.

Emotion | Hyper-parameters | # Augmented Samples
---|---|---
Swap p | Replacement p | Insertion p | Deletion p
Anger | 0.6 | 0.6 | 0.3 | 0.3 | 10
Fear | 0.6 | 0.6 | 0.5 | 0.4 | 20
Happiness | 0.6 | 0.6 | 0.5 | 0.4 | 20
Hatred | 0.6 | 0.6 | 0.3 | 0.3 | 10
Sadness | 0.6 | 0.6 | 0.3 | 0.3 | 10
Wonder | 0.6 | 0.6 | 0.4 | 0.4 | 15
Table 2: Hyper-parameters for mutation-based data augmentation are used in the
approach. The number of augmented samples per initial sample varies for each
emotion as the ratio between the number of samples in each class is different.
"p" here stands for probability.

###  3.4 Model

This section presents baseline models for EmoPars transfer learning-based
emotion classification. Since labels are expensive, supervised learning is
challenging to apply to NLP issues, including emotion detection. Transfer
learning is helpful in this situation. In recent years, state-of-the-art
performance in various NLP tasks has resulted from transfer learning from pre-
trained deep neural Language Models (LMs) towards downstream language
issuesYing et al. (2019); Durrani et al. (2021).

We use the ParsBERTMehrdad Farahani and Manthouri (2021) pre-trained language
model for Persian in most of our experiments to find the best solution. A
monolingual language model called ParsBERT is built on the BERT
architectureDevlin et al. (2019). The ParsBERT model surpasses the
multilingual BERTPires et al. (2019) and earlier models in some Persian NLP
downstream tasks, such as text categorization and sentiment analysis,
according to research by Farahani et al.Mehrdad Farahani and Manthouri (2021).
ParsBERT has been trained on a broader and more varied set of pre-trained
Persian datasets than the multilingual BERT, making it lighter in weight.
Although, due to the unbalanced data, we used other strategies to increase the
model’s performance. We will introduce them in the following sections.

In addition, to compare the quality of the EmoPars and ArmanEmo (a human-
labeled emotion dataset of more than 7000 Persian sentences labeled for seven
categories)Mirzaee et al. (2022), we used XLM-RoBERTaConneau et al. (2020) and
XLM-EMOBianchi et al. (2022).

A multilingual language model called XLM-RoBERTa has already been trained
using Masked Language Modeling (MLM). XLM-RoBERTa performs better than
multilingual BERT since it has more training data. XLM-RoBERTa is trained on
100 languages was.

A multilingual emotion prediction model called XLM-EMO was developed using
social media data. The model performs admirably in a zero-shot scenario,
indicating that it is helpful for low-resource languages.

####  3.4.1 Data Pre-processing

The data pre-processing method used in our approach was like what was used in
ArmanEmo paperMirzaee et al. (2022). We used the Parsivar toolkitMohtaj et al.
(2018) in this manner. Along with various character refining procedures,
Parsivar does several rule-based space correction operations (including word,
punctuation, and affix spacing) (such as removing stretching letters).
However, its normalization guidelines are not exhaustive. In addition to
normalization, we did the following pre-processings:

  * •

If there is an English word in the sentence; first we check if the word exists
in a single token to single token dictionary provided by Meta222It is
available at https://dl.fbaipublicfiles.com/arrival/dictionaries.tar.gz. If
the word does not exist in the dictionary, we use transliteration provided by
Polyglot module333https://polyglot.readthedocs.io/en/latest/polyglot.html.
Information from other languages is not considered in our approach.

  * •

Removing letters from Persian words that are frequently purposely misspelled
in order to emphasize certain words more strongly in casual writing.

  * •

Removing from the text any Arabic diacritics that the Parsivar normalizer did
not remove.

  * •

After completing the aforementioned stages, we removed any last non-Persian
characters.

  * •

Removing the "#" from the text’s hashtags while retaining the information they
contain.

####  3.4.2 Loss Function

The loss function greatly impacts a model’s performance. The influence of the
different loss functions has been investigated in Abdel-Salam (2022) for the
sarcasm detection task. In their results, they found that F1-Cross-Entropy and
Recall-Cross-Entropy may help that to reach a good F1-Sarcastic. Equation 1
and 2 shows how this loss functions work respectively. In these equations,
$F1_{c}$ and $Recall_{c}$ are the F1-Score and Recall corresponding to a
specific class c, respectively.

$\centering\sum_{c=1}^{C}-(1-F1_{c})\times
N_{c}\times\log(P_{c})\@add@centering$ (1)

$\centering\sum_{c=1}^{C}-(1-Recall_{c})\times
N_{c}\times\log(P_{c})\@add@centering$ (2)

Although in the mentioned task, Recall-Cross-Entropy reaches better results,
in our task, F1-Cross-Entropy showed better results.

####  3.4.3 Feature Extraction

Although considering the whole context gives the sample’s sentiment and
emotion, some specific parts of the samples, like hashtags or emojis, include
more information. Considering these features, as well as the embedding of the
sample, showed improvement in previous studiesHernández-Farías et al. (2015);
Yaghoobian et al. (2021); Nandwani and Verma (2021). After extracting some
lexical-based features, using the splitting token "</s></s>", all text
features are simply added to the pre-processed tweets. Figure 3 shows how this
process works.

Figure 3: We use different feature extractors, and then we add the extracted
features to the pre-processed tweet using the "</s></s>" token. The input
sentence translation in English is: "There’s nothing wrong with saying that
you took the exam at that time :))). My health does not matter <broken-heart-
emoji> <neutral-face-emoji>. How about
yours?#postponement_of_safe_conditions".

The feature extraction methods we used for this task are:

  * •

Emoji: After examining a sizable amount of Twitter data, emoji are a notable
multi-model feature that depicts human emotions.

  * •

Parts of Speech(POS): Additionally, changing the sentence structure sometimes
is used to show the writer’s emotion. As a result, POS tags are utilized as a
crucial component.

  * •

Misspelled Words: Occasionally, incorrectly spelled and capitalized words that
are not at the start of a phrase might increase the emotion.

  * •

Hashtags: Hashtags show the main point of the sentence more clearly. Using
them as a particular feature would be helpful.

####  3.4.4 Undersampling and Class Weight

In Section 3.3, we used data augmentation to decrease the big difference
between the number of samples between the two classes. In addition, we
mentioned that due to the vast difference, creating nearly 100 views of a
sentence using augmentation methods not only does not help us but may also
weaken the performance of the model. Other methods can help us to do, in
addition to data augmentation, to make the number of overcome the unbalanced
dataset problem.

Instead of increasing the number of samples for the minor class, we can
decrease the number of samples in the significant class, which is called
undersampling. Undersampling is a technique to balance uneven datasets by
keeping all of the data in the minority class and decreasing the size of the
majority class. This method has been investigated in previous papersZhu and
Hovy (2007); Kapadi and Luz (2022). To do undersampling, we selected the
tweets randomly with a preference for shorter tweets(using
$\frac{1}{TweetLength}$ as the weight of our selection).

In addition to undersampling, class weights can be used in the training
approach. Most machine learning algorithms are not very useful with partial
class data. However, we can modify the current training algorithm to consider
the skewed distribution of the classes. This can be achieved by giving weight
to both the majority and minority classes. The difference in weights will
influence the classification of the classes during the training phase. The
purpose is to penalize the misclassification made by the minority class by
setting a higher class weight and simultaneously reducing the weight for the
majority class.

We decreased our dataset imbalance using data augmentation, undersampling, and
class weights. Table 3 shows information about the final dataset to the model.
In addition to these methods, feature extraction introduced in Section 3.4.3
and the F1-Cross-Entropy loss function mentioned in Section 3.4.2 were also
helpful.

Emotion | # of 0 Samples | # of 1 Samples | Weights
---|---|---|---
Anger | 2590 | 13986 | 5-1
Fear | 1520 | 12768 | 8-1
Happiness | 1440 | 14562 | 9-1
Hatred | 1750 | 12263 | 7-1
Sadness | 2990 | 13643 | 4-1
Wonder | 1935 | 14512 | 7-1
Table 3: The information of the final dataset is provided to the model. Class
weights are reported in "weight of the 0 class-weight of the 1 class".

##  4 Experiments

In this section, we report our results in different experiments. We compare
two famous Persian emotion datasets with our approach. In addition, on
EmoPars, we evaluate the effectiveness of our method through different
experiments, including multi-label training and different threshold in
selecting tweets.

In our experiments, 10% of the datasets have been used for evaluation and the
rest for training. In addition, batch size 16, 5 epochs, the learning rate of
1e-4, and a max length of 256 were utilized in these experiments.

###  4.1 Training With Using Class Weights Only

Because no baseline has been provided in the EmoPars paper and the analysis in
the ArmanEmo paper on this dataset is not fair, we first trained ParsBERT
without using any of the other methods. The only method that has been used to
handle imbalanced datasets was using class weights. The class weight was set
based on the number of samples, where the ratio of the number of significant
class samples by the number of minor class samples was used as the weight of
the minor class, and the weight of the major class was one.

Emotion | Accuracy | Precision | Recall | F1-score
---|---|---|---|---
Anger | 0.38 | 0.15 | 0.39 | 0.21
Fear | 0.49 | 0.24 | 0.49 | 0.32
Happiness | 0.51 | 0.26 | 0.51 | 0.34
Hatred | 0.42 | 0.17 | 0.42 | 0.25
Sadness | 0.38 | 0.15 | 0.38 | 0.21
Wonder | 0.45 | 0.2 | 0.45 | 0.28
Table 4: Baseline results for evaluating the effectiveness of our approach.
Class weights are used to reach good results for this result in addition to
data preprocessing.

Table 4 shows the baseline results for EmoPars dataset. The interesting fact
about these results might be that although happiness has the best results,
sadness does not show this fact. This is because people use more specific
words in happiness, while in sadness, the words can conflict with those in
hatred. Another reason may be the larger number of samples in class zero in
sadness than happiness, resulting in using a larger class weight for class
one.

###  4.2 Multi-label Training

As mentioned in Section 3.1, emotion detection is multi-label task which
EmoPars considers this fact. One of the first experiments we did was using the
ParsBERT model for multi-label training of the model. We used Hamming Loss to
evaluate the model. Equation 3 shows how hamming loss is calculated. In this
formula, $N$ is the number of samples, $L$ is the number of labels, and
$\oplus$ denotes exclusive-or.

$\centering HL=\frac{1}{NL}\sum_{l=1}^{L}\sum_{i=1}^{N}Y_{i,l}\oplus
X_{i,l}\@add@centering$ (3)

Threshold | Test Result
---|---
Hamming Loss | Hamming Score
1 | 0.4653 | 0.4022
2 | 0.1848 | 0.3265
3 | 0.0383 | 0.803
4 | 0.0054 | 0.9685
5 | 0.0005 | 0.9971
Table 5: The result of multi-label training with ParsBERT. The results are
reported based on different thresholds—the tweets with votes more equal to or
more than the threshold considered having the emotion. Figure 4: An overview
of hamming loss of ParsBERT that was trained on the EmoPars per epoch.

Table 5 shows the result of multi-label training with different thresholds on
ParsBERT. The hamming score shows the accuracy of the model. Although we have
a high score on the last two thresholds, we should consider this result
unreliable since we face imbalanced data, which Figure 1 depicts. When the
considered threshold is bigger, the number of samples in one class(class zero)
is higher. As a result, we have a more hamming score. Even though multi-label
is more efficient, we have to use multiple binary classifiers to detect the
emotions in one sample. Moreover, Figure 4 illustrates loss per each batch for
every five epochs which is less smooth for thresholds 4 and 5.

###  4.3 Removing Indeterminate Data

One of our ideas to evaluate the EmoPars dataset properly was removing
indeterminate data explained in Section 3.2. Before applying our method for
handling imbalanced data, we need to check the effect of the provided policy.

Emotion | Precision | Recall | F1-score | Support
---|---|---|---|---
Anger | 1 | 0.25 | 0.33 | 2357
Fear | 0.43 | 0.31 | 0.36 | 2785
Happiness | 0.28 | 0.44 | 0.34 | 2615
Hatred | 0.57 | 0.2 | 0.29 | 2425
Sadness | 0.76 | 0.22 | 0.35 | 2305
Wonder | 0.57 | 0.17 | 0.27 | 2501
Table 6: The result of ParsBERT on the EmoPars after deleting samples with
labels two and three and using class weights. Support shows the number of
samples used for evaluation. Accuracy in all emotions was 0.99 except for
Happiness which was 0.98.

Table 6 shows the effect of removing samples with labels two and three. As the
results show, F1-score has been increased to 0.1, which is a noticeable
improvement in this imbalanced dataset.

###  4.4 Impact Of Our Approach

Since we were facing with imbalanced data, we used some approaches to solve
this issue which is introduced in Sections 3.3, 3.4.1, 3.4.2, 3.4.3, and
3.4.4. The combination of all of these methods resulted in significant
improvement in the result of our method.

Emotion | Accuracy | Precision | Recall | F1-score
---|---|---|---|---
Anger | 0.81 | 0.7 | 0.78 | 0.74
Fear | 0.79 | 0.65 | 0.81 | 0.72
Happiness | 0.81 | 0.68 | 0.84 | 0.75
Hatred | 0.83 | 0.75 | 0.74 | 0.75
Sadness | 0.87 | 0.81 | 0.81 | 0.81
Wonder | 0.86 | 0.8 | 0.78 | 0.79
Table 7: The results of ParsBERT on the EmoPars using F1-Cross-Entropy, Data Augmentation, Undersampling, Class Weights, and Feature Extraction. Model | Test F1-score | Macro-F1
---|---|---
Hatred | Happiness | Sadness | Wonder | Anger | Fear
XLM-EMO | 0.78 | 0.77 | 0.82 | 0.82 | 0.8 | 0.75 | 0.79
XLM-RoBERTa | 0.79 | 0.79 | 0.84 | 0.85 | 0.81 | 0.79 | 0.81
ParsBERT | 0.74 | 0.72 | 0.81 | 0.79 | 0.74 | 0.72 | 0.76
Table 8: Macro-averaged F1-score and each emotion F1-score for different
models on EmoPars.

Table 7 shows our approach has approximately double the F1-score in each
emotion. We believe this result is due to using a small amount of each method
for handling the imbalanced data. Using just data augmentation can make
redundant samples, and under-sampling can decrease the number of samples.
Nevertheless, using both of them and adding more attention by feature
extraction can help us more. Moreover, the F1-Cross-Entropy tries to increase
the F1-score instead of Accuracy, which will help us to reach higher precision
and recall.

Figure 5: An overview of the loss value while employing our strategy and a comparison to training the model without it. Train Set | Test Set
---|---
EmoPars | ArmanEmo
Arman Emo | 0.76 | 0.81
EmoPars | 0.78 | 0.74
Table 9: Macro-averaged F1-score for comparison between ArmanEmo and EmoPars
datasets.

Figure 5 shows the influence of our approach on training. Feature extraction
has a significant impact on the initial loss of the model. Then, the loss
function helps the model. In addition, since the model sees more samples in
the positive class, it can reach better results.

###  4.5 Comparing Datasets

One of the experiments in Mirzaee et al. (2022) compared the provided dataset,
ArmanEmo, and the EmoPars dataset. However, their data selection policy was
not fair and accurate. They just used 919 samples from EmoPars, a small amount
compared to 4700 samples in ArmanEmo.

Here, we trained the model on EmoPars with more samples and used the same
model and imbalanced data handling approaches. Table 9 shows the results of
our comparison. Unlike the results provided in ArmanEmo, training on one
dataset results in a better F1-score on the same dataset. However, results
show that EmoPars is noisier and harder to learn, which can be because of
including multiple emotions in one sample.

###  4.6 Comparison Among Different Models

Another experiment that we have done was evaluating different models on
EmoPars. All the experiments till this section were done using the ParsBERT
model. Table 8 shows the results of the three models we have used: XLM-
EMOBianchi et al. (2022), XLM-RoBERTaBianchi et al. (2022), and
ParsBERTMehrdad Farahani and Manthouri (2021). XLM-RoBERTa and XLM-EMO have
higher values due to having a higher number of parameters.

##  5 Conclusion

This paper uses an approach for handling imbalanced data on EmoPars, an
emotion recognition dataset. Since EmoPars produces imbalanced multi-label
data labeled with voting technique, selecting definite data is essential. Our
experiment on EmoPars shows that removing uncertain samples increased the
F1-score of the method up to 51%. However, this could not help with imbalanced
data, which results in a biased model. As a result, we used data augmentation
(including swapping words, synonym substitution, word deletion, and word
addition), undersampling, and F1-Cross-Entropy loss function to solve this
issue. In addition, we used a feature extraction technique that adds emojis,
hashtags, POS tags, and misspelled words at the end of the sentence to improve
the performance. Our model reaches a Macro-F1-score of 0.76 on the EmoPars.
Furthermore, our introduced method reaches a Macro-F1-score of 0.81 on
ArmanEmoMirzaee et al. (2022).

## References

  * Abaskohi et al. (2022) Amirhossein Abaskohi, Arash Rasouli, Tanin Zeraati, and Behnam Bahrak. 2022.  UTNLP at SemEval-2022 task 6: A comparative analysis of sarcasm detection using generative-based and mutation-based data augmentation.  In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_ , pages 962–969, Seattle, United States. Association for Computational Linguistics.
  * Abdel-Salam (2022) Reem Abdel-Salam. 2022.  reamtchka at SemEval-2022 task 6: Investigating the effect of different loss functions for sarcasm detection for unbalanced datasets.  In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_ , pages 896–906, Seattle, United States. Association for Computational Linguistics.
  * Acheampong et al. (2021) Francisca Adoma Acheampong, Henry Nunoo-Mensah, and Wenyu Chen. 2021.  Transformer models for text-based emotion detection: a review of bert-based approaches.  _Artificial Intelligence Review_ , 54(8):5789–5829.
  * Alfano et al. (2021) Mark Alfano, Ritsaart Reimann, Ignacio Quintana, Marc Cheong, and Colin Klein. 2021\.  The affiliative use of emoji and hashtags in the black lives matter movement: A twitter case study.
  * Alla et al. (2022) Keerthi Reddy Alla, Nikhil Kandibanda, Priyanka Katta, Abhinav Muthavarapu, and Swarna Kuchibhotla. 2022.  Emotion detection from text using lstm.  In _Proceedings of Sixth International Congress on Information and Communication Technology_ , pages 545–553. Springer.
  * Alswaidan and Menai (2020) Nourah Alswaidan and Mohamed El Bachir Menai. 2020.  A survey of state-of-the-art approaches for emotion recognition in text.  _Knowledge and Information Systems_ , 62(8):2937–2987.
  * Andalibi and Buss (2020) Nazanin Andalibi and Justin Buss. 2020.  The human in emotion recognition on social media: Attitudes, outcomes, risks.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , pages 1–16.
  * Aquino et al. (2017) N Romero Aquino, Matheus Gutoski, Leandro T Hattori, and Heitor S Lopes. 2017.  The effect of data augmentation on the performance of convolutional neural networks.  _Braz. Soc. Comput. Intell_.
  * Barua et al. (2012) Sukarna Barua, Md Monirul Islam, Xin Yao, and Kazuyuki Murase. 2012.  Mwmote–majority weighted minority oversampling technique for imbalanced data set learning.  _IEEE Transactions on knowledge and data engineering_ , 26(2):405–425.
  * Bhowmick et al. (2009) Plaban Kr Bhowmick, Anupam Basu, Pabitra Mitra, and Abhishek Prasad. 2009.  Multi-label text classification approach for sentence level news emotion analysis.  In _International conference on pattern recognition and machine intelligence_ , pages 261–266. Springer.
  * Bianchi et al. (2022) Federico Bianchi, Debora Nozza, and Dirk Hovy. 2022.  XLM-EMO: Multilingual emotion prediction in social media text.  In _Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis_, pages 195–203, Dublin, Ireland. Association for Computational Linguistics.
  * Błaszczyński and Stefanowski (2015) Jerzy Błaszczyński and Jerzy Stefanowski. 2015.  Neighbourhood sampling in bagging for imbalanced data.  _Neurocomputing_ , 150:529–542.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014\.  On the properties of neural machine translation: Encoder-decoder approaches.  _arXiv preprint arXiv:1409.1259_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Deng and Ren (2020) Jiawen Deng and Fuji Ren. 2020.  Multi-label emotion detection via emotion-specified feature extraction and emotion correlation learning.  _IEEE Transactions on Affective Computing_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dolan (2002) Raymond J Dolan. 2002.  Emotion, cognition, and behavior.  _science_ , 298(5596):1191–1194.
  * Domes et al. (2009) Gregor Domes, Lars Schulze, and Sabine C Herpertz. 2009.  Emotion recognition in borderline personality disorder—a review of the literature.  _Journal of personality disorders_ , 23(1):6–19.
  * dos Santos Tanaka and Aranha (2019) Fabio Henrique Kiyoiti dos Santos Tanaka and Claus Aranha. 2019.  Data augmentation using gans.  _Proceedings of Machine Learning Research XXX_ , 1:16.
  * Durrani et al. (2021) Nadir Durrani, Hassan Sajjad, and Fahim Dalvi. 2021.  How transfer learning impacts linguistic knowledge in deep nlp models?  _arXiv preprint arXiv:2105.15179_.
  * Ekman (1992a) Paul Ekman. 1992a.  Are there basic emotions?
  * Ekman (1992b) Paul Ekman. 1992b.  An argument for basic emotions.  _Cognition & emotion_, 6(3-4):169–200.
  * Esmin et al. (2012) Ahmed AA Esmin, Roberto L De Oliveira Jr, and Stan Matwin. 2012.  Hierarchical classification approach to emotion recognition in twitter.  In _2012 11th International Conference on Machine Learning and Applications_ , volume 2, pages 381–385. IEEE.
  * Fei et al. (2021) Chengwei Fei, Rong Liu, Zihao Li, Tianmin Wang, and Faisal N Baig. 2021.  Machine and deep learning algorithms for wearable health monitoring.  In _Computational intelligence in healthcare_ , pages 105–160. Springer.
  * Gendron et al. (2014) Maria Gendron, Debi Roberson, Jacoba Marietta van der Vyver, and Lisa Feldman Barrett. 2014.  Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture.  _Emotion_ , 14(2):251.
  * Hamidi and Mansoorizade (2012) Mina Hamidi and Muharram Mansoorizade. 2012.  Emotion recognition from persian speech with neural network.  _International Journal of Artificial Intelligence & Applications_, 3(5):107.
  * Hernández-Farías et al. (2015) Irazú Hernández-Farías, José-Miguel Benedí, and Paolo Rosso. 2015.  Applying basic features from sentiment analysis for automatic irony detection.  In _Iberian Conference on Pattern Recognition and Image Analysis_ , pages 337–344. Springer.
  * Iosifov et al. (2022) Ievgen Iosifov, Olena Iosifova, Oleh Romanovskyi, Volodymyr Sokolov, and Ihor Sukailo. 2022.  Transferability evaluation of speech emotion recognition between different languages.  In _International Conference on Computer Science, Engineering and Education Applications_ , pages 413–426. Springer.
  * Jahanbakhsh-Nagadeh et al. (2021) Zoleikha Jahanbakhsh-Nagadeh, Mohammad-Reza Feizi-Derakhshi, and Arash Sharifi. 2021\.  A semi-supervised model for persian rumor verification based on content information.  _Multimedia Tools and Applications_ , 80(28):35267–35295.
  * Jiang and Ge (2020) Xiaoyu Jiang and Zhiqiang Ge. 2020.  Data augmentation classifier for imbalanced fault classification.  _IEEE Transactions on Automation Science and Engineering_ , 18(3):1206–1217.
  * Kafle et al. (2017) Kushal Kafle, Mohammed Yousefhussien, and Christopher Kanan. 2017.  Data augmentation for visual question answering.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 198–202.
  * Kapadi and Luz (2022) Tarif Kapadi and Saturnino Luz. 2022.  Natural language processing of electronic patient records to predict psychiatric inpatients at risk of early readmission to hospital using predictive models derived through machine learning.  _BJPsych Open_ , 8(S1):S6–S6.
  * Lemmens et al. (2020) Jens Lemmens, Ben Burtenshaw, Ehsan Lotfi, Ilia Markov, and Walter Daelemans. 2020\.  Sarcasm detection using an ensemble approach.  In _proceedings of the second workshop on figurative language processing_ , pages 264–269.
  * Li et al. (2021) Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. 2021.  Autobalance: Optimized loss functions for imbalanced data.  _Advances in Neural Information Processing Systems_ , 34:3163–3177.
  * Li et al. (2020) Qingbiao Li, Chunhua Wu, Zhe Wang, and Kangfeng Zheng. 2020.  Hierarchical transformer network for utterance-level emotion recognition.  _Applied Sciences_ , 10(13):4447.
  * Ma (2019) Edward Ma. 2019.  Nlp augmentation.  https://github.com/makcedward/nlpaug.
  * Mahmoodabad et al. (2018) Sajjad Dehghani Mahmoodabad, Saeed Farzi, and Danial Bidekani Bakhtiarvand. 2018\.  Persian rumor detection on twitter.  In _2018 9th international symposium on telecommunications (IST)_ , pages 597–602. IEEE.
  * Mahto and Yadav (2022) Dashrath Mahto and Subhash Chandra Yadav. 2022.  Hierarchical bi-lstm based emotion analysis of textual data.  _Bulletin of the Polish Academy of Sciences. Technical Sciences_ , 70(3).
  * Makki et al. (2019) Sara Makki, Zainab Assaghir, Yehia Taher, Rafiqul Haque, Mohand-Said Hacid, and Hassan Zeineddine. 2019.  An experimental study with imbalanced classification approaches for credit card fraud detection.  _IEEE Access_ , 7:93010–93022.
  * Mehrdad Farahani and Manthouri (2021) Marzieh Farahani Mehrdad Farahani, Mohammad Gharachorloo and Mohammad Manthouri. 2021.  Parsbert: Transformer-based model for persian language understanding.  _Neural Processing Letters_.
  * Meléndez et al. (2020) Juan C Meléndez, Encarnacion Satorres, Maria Reyes-Olmedo, Iraida Delhom, Elena Real, and Yaiza Lora. 2020.  Emotion recognition changes in a confinement situation due to covid-19.  _Journal of Environmental Psychology_ , 72:101518.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Minzenberg et al. (2006) Michael J Minzenberg, John H Poole, and Sophia Vinogradov. 2006.  Social-emotion recognition in borderline personality disorder.  _Comprehensive psychiatry_ , 47(6):468–474.
  * Mirzaee et al. (2022) Hossein Mirzaee, Javad Peymanfard, Hamid Habibzadeh Moshtaghin, and Hossein Zeinali. 2022.  Armanemo: A persian dataset for text-based emotion detection.  _arXiv preprint arXiv:2207.11808_.
  * Mohtaj et al. (2018) Salar Mohtaj, Behnam Roshanfekr, Atefeh Zafarian, and Habibollah Asghari. 2018.  Parsivar: A language processing toolkit for Persian.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Nandwani and Verma (2021) Pansy Nandwani and Rupali Verma. 2021.  A review on sentiment analysis and emotion detection from text.  _Social Network Analysis and Mining_ , 11(1):1–19.
  * Neumann et al. (2018) Michael Neumann et al. 2018.  Cross-lingual and multilingual speech emotion recognition on english and french.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5769–5773. IEEE.
  * Parida and Motlicek (2019) Shantipriya Parida and Petr Motlicek. 2019.  Abstract text summarization: A low resource challenge.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5994–5998.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Pourebrahim et al. (2021) Yousef Pourebrahim, Farbod Razzazi, and Hossein Sameti. 2021.  Parallel shared hidden layers auto-encoder as a cross-corpus transfer learning approach for unsupervised persian speech emotion recognition.  _Signal Processing and Renewable Energy_ , 5(4):83–106.
  * Russell (1980) James A Russell. 1980.  A circumplex model of affect.  _Journal of personality and social psychology_ , 39(6):1161.
  * Sabri et al. (2021) Nazanin Sabri, Reyhane Akhavan, and Behnam Bahrak. 2021.  EmoPars: A collection of 30K emotion-annotated Persian social media texts.  In _Proceedings of the Student Research Workshop Associated with RANLP 2021_ , pages 167–173, Online. INCOMA Ltd.
  * Sadeghi et al. (2021) Seyedeh S Sadeghi, Hasan Khotanlou, and M Rasekh Mahand. 2021.  Automatic persian text emotion detection using cognitive linguistic and deep learning.  _Journal of AI and Data Mining_ , 9(2):169–179.
  * Sari et al. (2014) Yuita Arum Sari, Evy Kamilah Ratnasari, Siti Mutrofin, and Agus Zainal Arifin. 2014\.  User emotion identification in twitter using specific features: hashtag, emoji, emoticon, and adjective term.  _Jurnal Ilmu Komputer dan Informasi_ , 7(1):18–23.
  * Sarkar and Etemad (2020) Pritam Sarkar and Ali Etemad. 2020.  Self-supervised ecg representation learning for emotion recognition.  _IEEE Transactions on Affective Computing_.
  * Schlegel et al. (2021) Katja Schlegel, Helene M von Gugelberg, Lisa M Makowski, Danièle A Gubler, and Stefan J Troche. 2021.  Emotion recognition ability as a predictor of well-being during the covid-19 pandemic.  _Social psychological and personality science_ , 12(7):1380–1391.
  * Shorten et al. (2021) Connor Shorten, Taghi M Khoshgoftaar, and Borko Furht. 2021.  Text data augmentation for deep learning.  _Journal of big Data_ , 8(1):1–34.
  * Shrivastava et al. (2019) Kush Shrivastava, Shishir Kumar, and Deepak Kumar Jain. 2019.  An effective approach for emotion detection in multimedia text data using sequence based convolutional neural network.  _Multimedia Tools and Applications_ , 78(20):29607–29639.
  * Shyry et al. (2020) S Prayla Shyry, K Kaja Kartheek, and KN RR Aravind. 2020.  Election prediction with automated speech emotion recognition.  In _2020 4th International Conference on Trends in Electronics and Informatics (ICOEI)(48184)_ , pages 709–715. IEEE.
  * Skórzewski (2022) Paweł Skórzewski. 2022.  Using book dialogues to extract emotions from texts.  In _Language and Technology Conference_ , pages 244–255. Springer.
  * Sobkowicz et al. (2012) Pawel Sobkowicz, Michael Kaschesky, and Guillaume Bouchard. 2012.  Opinion mining in social media: Modeling, simulating, and forecasting political opinions in the web.  _Government information quarterly_ , 29(4):470–479.
  * Tettegah and Noble (2015) Sharon Tettegah and Safiya Noble. 2015.  _Emotions, technology, and design_.  Academic Press.
  * Tzirakis et al. (2017) Panagiotis Tzirakis, George Trigeorgis, Mihalis A Nicolaou, Björn W Schuller, and Stefanos Zafeiriou. 2017.  End-to-end multimodal emotion recognition using deep neural networks.  _IEEE Journal of selected topics in signal processing_ , 11(8):1301–1309.
  * Tzirakis et al. (2019) Panagiotis Tzirakis, Stefanos Zafeiriou, and Björn Schuller. 2019.  Real-world automatic continuous affect recognition from audiovisual signals.  In _Multimodal Behavior Analysis in the Wild_ , pages 387–406. Elsevier.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * WARSI (2021) MJ WARSI. 2021.  Text messaging: A linguistic phenomenon.  _Interdisciplinary Journal of Linguistics_ , 14:13–23.
  * Yaghoobian et al. (2021) Hamed Yaghoobian, Hamid R Arabnia, and Khaled Rasheed. 2021.  Sarcasm detection: A comparative study.  _arXiv preprint arXiv:2107.02276_.
  * Yazdani et al. (2021) Ali Yazdani, Hossein Simchi, and Yasser Shekofteh. 2021.  Emotion recognition in persian speech using deep neural networks.  In _2021 11th International Conference on Computer Engineering and Knowledge (ICCKE)_ , pages 374–378. IEEE.
  * Ying et al. (2019) Wenhao Ying, Rong Xiang, and Qin Lu. 2019.  Improving multi-label emotion classification by integrating both general and domain knowledge.  In _Proceedings of the 5th Workshop on Noisy User-Generated Text (W-NUT 2019)_ , page 316.
  * Zhang et al. (2016) Fan Zhang, Hua Xu, Jiushuo Wang, Xiaomin Sun, and Junhui Deng. 2016.  Grasp the implicit features: Hierarchical emotion classification based on topic model and svm.  In _2016 International joint conference on neural networks (IJCNN)_ , pages 3592–3599. IEEE.
  * Zhang et al. (2020) Jiajia Zhang, Mengce Zheng, Jiehui Nan, Honggang Hu, and Nenghai Yu. 2020.  A novel evaluation metric for deep learning-based side channel analysis and its extended application to imbalanced data.  _IACR Transactions on Cryptographic Hardware and Embedded Systems_ , pages 73–96.
  * Zhu and Hovy (2007) Jingbo Zhu and Eduard Hovy. 2007.  Active learning for word sense disambiguation with methods for addressing the class imbalance problem.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 783–790.
  * Zhu et al. (2018) Min Zhu, Jing Xia, Xiaoqing Jin, Molei Yan, Guolong Cai, Jing Yan, and Gangmin Ning. 2018.  Class weights random forest algorithm for processing class imbalanced medical data.  _IEEE Access_ , 6:4641–4652.


# A Reliable, Self-Adaptive Face Identification Framework via Lyapunov
Optimization

Dohyeon Kim  Naver Webtoon
Seongnam, Republic of Korea
<EMAIL_ADDRESS>Joongheon Kim  Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Jae young Bang  Quandary Peak Research
Los Angeles, CA, USA
<EMAIL_ADDRESS>
###### Abstract

Realtime face identification (FID) from a video feed is highly computation-
intensive, and may exhaust computation resources if performed on a device with
a limited amount of resources (e.g., a mobile device). In general, FID
performs better when images are sampled at a higher rate, minimizing false
negatives. However, performing it at an overwhelmingly high rate exposes the
system to the risk of a queue overflow that hampers the system’s reliability.
This paper proposes a novel, queue-aware FID framework that adapts the
sampling rate to maximize the FID performance while avoiding a queue overflow
by implementing the Lyapunov optimization. A preliminary evaluation via a
trace-based simulation confirms the effectiveness of the framework.

##  I Introduction

Realtime face identification (FID) is an active area of research in the
artificial intelligence domain that has widely been adopted by the industry.
As it matures, today, we are witnessing it spreading into our daily lives and
being implemented even in mobile applications that often have limitations in
the amount of computation resources available.

The computation resource limitation puts the system reliability and the FID
performance in a trade-off relationship. An FID system samples images from a
video feed and inserts them into a queue. The system concurrently retrieves
the images and executes an FID algorithm to detect faces that appear in those
images. With a higher sampling rate, the system would less likely to miss
faces that do appear in the feed but not in the sampled images. However, an
overwhelmingly high sampling rate could also cause a queue overflow, resulting
in unexpected system behaviors. On the other hand, while sampling at a lower
rate would help prevent an overflow, the system would more likely to fail to
identify faces.

In this paper, we present a queue-aware FID framework that automatically
adapts the sampling rate to the computation resource availability via the
Lyapunov optimization in order to achieve maximum FID performance (fewest
false negatives) while avoiding a queue overflow. We do not develop new FID
algorithms; we exploit the existing ones and provide a self-adaptive framework
on which the algorithms can be implemented and executed in a reliable fashion.

Figure 1: The proposed FID framework using OpenFace.

##  II Stochastic Frame Rate Adaptation

###  II-A Lyapunov optimization framework

The theory of stochastic optimization [1] aims at optimizing a time-average
utility subject to queue stability when the objective function and the queue
stability constraints are in a trade-off relationship. Stochastic optimization
models the queue stability using the Lyapunov drift. It takes actions that
minimizes the Lyapunov drift while pursuing the minimization of a time-average
objective function with the gap of $O(1/V)$ under queue stability with the
bound of $O(V)$ where $V$ is defined as a trade-off between utility and
stability. Neely discusses the details of the theory in his book [1], and
there are a number of applications that implement the theory [2, 3, 4, 5, 6,
7, 8, 9].

###  II-B Reference face identification (FID) platform

Fig. 1 depicts our reference FID framework. A system that implements the
framework will have a video source (e.g., a camera) that constantly feeds a
stream of images (frames) into the framework, and the framework stores the
images in a queue in which they wait until the framework processes them. The
framework concurrently conducts face identification on the images in the queue
using the OpenFace library in the following four steps that correspond to
those illustrated in Fig. 1: (1) input image loading (Input Image), (2) face
recognition and preprocessing (Detect, Transform, and Crop), (3) neural
network forwarding (Deep Neural Network, Representation), and (4)
classification.

The FID framework automatically controls the frame rate of the video feed in
order to maximize FID performance while avoiding a queue overflow. In this
paper, we define the _FID performance_ as following: the performance with a
frame rate $f(t)$ at time $t$ is defined as
$S(f(t))\triangleq\frac{\alpha(f(t))}{\beta(t)}$ where $\alpha(f(t))$ and
$\beta(t)$ respectively denote the number of identified faces with given
$f(t)$ and the total number of faces appeared in the original video feed, at
time $t$. $S(f(t))=1$ if all faces are identified in $f(t)$ whereas
$S(f(t))=0$ if no face is identified at all in $f(t)$. Higher $f(t)$ generally
leads to higher $S(f(t))$; that lowers the chance of having faces that quickly
pass by between two frames, unidentified (false negatives). However, in
reality, because it is likely for the FID system to have a limited amount of
queue storage and finite computation power, an overwhelmingly high $f(t)$
could cause a queue overflow and result in unreliable system behaviors. The
framework continuously monitors the queue status and adapts the rate at which
the frames from the video feed are inserted into the queue to the resource
availability.

Algorithm 1 Frame rate control via Lyapunov optimization

1:$Q(t)\leftarrow 0$ and $t\leftarrow 0$

2:while $t\leq T$ do // $T$: operation time

3: Observe $Q(t)$ and $\mathcal{T}^{*}\leftarrow-\infty$

4: for $f(t)\in\mathcal{F}$ do $\mathcal{T}\leftarrow V\cdot
S(f(t))-Q(t)\cdot\lambda(f(t))$

5: if $\mathcal{T}\geq\mathcal{T}^{*}$ then
$\mathcal{T}^{*}\leftarrow\mathcal{T}$ and $f^{*}(t)\leftarrow f(t)$

6: end if

7: end for

8:end while

###  II-C Frame rate control via Lyapunov optimization

This section describes how we model the time-average optimal frame rate using
the Lyapunov optimization [1] on which our FID framework controls $f(t)$
based. An FID system that implements our framework continuously adjusts $f(t)$
to the outputs of the optimal frame rate model.

We first model the arrival queue dynamics as
$Q(t+1)=\max\left\\{Q(t)-\mu(t),0\right\\}+\lambda(f(t))$ where $Q(t)$,
$\mu(t)$, and $\lambda(f(t))$ respectively denote the queue-backlog size, the
number of images departing from the queue, and the number of images arriving
in the queue with $f(t)$.We then formulate the mathematical program for
maximizing the time-average system performance $S(f(t))$ as
$\max:\lim_{t\rightarrow\infty}\sum_{\tau=0}^{t-1}\nolimits S(f(\tau))$ and
the queue stability constraint (to avoid a queue overflow) as
$\lim_{t\rightarrow\infty}\frac{1}{t}\sum_{\tau=0}^{t-1}\nolimits
Q(t)<\infty$. According to the Lyapunov optimization theory [1], this program
can be re-formulated as following where $f^{*}(t)$ is time-average optimal
frame rate for observed $Q(t)$:

$f^{*}(t)\leftarrow\arg\max_{f(t)\in\mathcal{F}}\nolimits\left\\{V\cdot
S(f(t))-Q(t)\cdot\lambda(f(t))\right\\}$

##  III Implementation and Evaluation

We evaluated our framework via a trace-based simulation, which showed that the
framework maximizes the FID performance while not having any queue overflow.
The evaluation was preliminary, and we made an assumption that maximizing the
number of frames that the framework processes would also maximize the FID
performance. We designed the simulation to mimic an FID system with a
threshold of 10 frames/sec at which a queue-divergence would occur. We then
varied the frame rate from 1 to 10, with and without our framework. Fig. 2
depicts the four simulation results: (1–red) the queue eventually overflows
with a fixed frame rate of 10, (2–black / 3–blue) the queue stabilizes at
certain points depending on the given $V$, and (4–green) the queue is stable
but the FID performance is the lowest at 1 frames/sec. Our framework does not
require a predetermined frame rate since it self-adapts the rate to the queue
status on-the-fly.

Figure 2: Evaluation results: queue dynamics.

##  IV Concluding Remarks

This paper proposes a reliable FID framework that achieves maximum FID
performance by self-adapting the frame rate to the queue-backlog status using
a unique Lyapunov optimization model. There are other system objectives such
as time to identify a face or energy consumption of an FID system that would
also be crucial for an FID system in practice, and we see an interesting
research opportunity in creating new optimization models that target those
objectives.

## Acknowledgment

This research was supported by National Research Foundation of Korea
(2017R1A4A1015675). J. Kim is a corresponding author. This paper was presented
at ACM Symposium on Operating Systems Principles (SOSP) Workshop on AI Systems
(AISys), Shanghai, China, October 2017.

## References

  * [1] M. Neely, Stochastic network optimization with application to communication and queueing systems. Morgan & Claypool, 2010.
  * [2] J. Kim, G. Caire, and A. F. Molisch, “Quality-aware streaming and scheduling for device-to-device video delivery,” IEEE/ACM Transactions on Networking, vol. 24, no. 4, pp. 2319–-2331, August 2016.
  * [3] J. Koo, J. Yi, J. Kim, M. A. Hoque, and S. Choi, “Seamless dynamic adaptive streaming in LTE/Wi-Fi integrated network under smartphone resource constraints,” IEEE Trans. on Mobile Computing, vol. 18, no. 7, pp. 1647-–1660, July 2019.
  * [4] M. Choi, J. Kim, and J. Moon, “Wireless video caching and dynamic streaming under differentiated quality requirements,” IEEE Journal on Selected Areas in Communications, vol. 36, no. 6, pp. 1245-–1257, June 2018.
  * [5] N.-N. Dao, D.-N. Vu, W. Na, J. Kim, and S. Cho, “SGCO: Stabilized green crosshaul orchestration for dense IoT offloading services,” IEEE Journal on Selected Areas in Communications, vol. 36, no. 11, pp. 2538-–2548, November 2018.
  * [6] M. Choi, J. Kim, and J. Moon, “Dynamic power allocation and user scheduling for power-efficient and delay-constrained multiple access networks,” IEEE Trans. on Wireless Communications, vol. 18, no. 10, pp. 4846–4858, October 2019.
  * [7] M. Choi, A. No, M. Ji, and J. Kim, “Markov decision policies for dynamic video delivery in wireless caching networks,” IEEE Trans. on Wireless Communications, vol. 18, no. 12, pp. 5705–5718, December 2019.
  * [8] M. Choi, A. F. Molisch, and J. Kim, “Joint distributed link scheduling and power allocation for content delivery in wireless caching networks,” IEEE Trans. on Wireless Communications, vol. 19, no. 12, pp. 7810–7824, December 2020.
  * [9] S. Jung, J. Kim, and J.-H. Kim, “Intelligent active queue management for stabilized QoS guarantees in 5G mobile networks,” IEEE Systems Journal, vol. 15, no. 3, pp. 4293–4302, September 2021.