# Pre-Trained Model Representations and their Robustness against Noise for
Speech Emotion Analysis

###### Abstract

Pre-trained model representations have demonstrated state-of-the-art
performance in speech recognition, natural language processing, and other
applications. Speech models, such as Bidirectional Encoder Representations
from Transformers (BERT) and Hidden units BERT (HuBERT), have enabled
generating lexical and acoustic representations to benefit speech recognition
applications. We investigated the use of pre-trained model representations for
estimating dimensional emotions, such as activation, valence, and dominance,
from speech. We observed that while valence may rely heavily on lexical
representations, activation and dominance rely mostly on acoustic information.
In this work, we used multi-modal fusion representations from pre-trained
models to generate state-of-the-art speech emotion estimation, and we showed a
${100\%}$ and ${30\%}$ relative improvement in concordance correlation
coefficient ($CCC$) on valence estimation compared to standard acoustic and
lexical baselines. Finally, we investigated the robustness of pre-trained
model representations against noise and reverberation degradation and noticed
that lexical and acoustic representations are impacted differently. We
discovered that lexical representations are more robust to distortions
compared to acoustic representations, and demonstrated that knowledge
distillation from a multi-modal model helps to improve the noise-robustness of
acoustic-based models.

Index Terms—  Speech emotion, robustness, multi-modal

##  1 Introduction

Speech-based emotion recognition models aim to recognize the emotional state
of a speaker in real-time. Current human-machine interaction systems can
recognize the words said by the speaker but fail to acknowledge the expressed
emotion. Reliable and robust speech-based emotion models can help improve
human-computer interaction and health/wellness applications, such as voice
assistants [1, 2], clinical mental health diagnoses [3, 4], and/or therapy
treatments [5].

Emotion recognition research has pursued two definitions of emotion: (1)
discrete emotions [6, 7], and (2) dimensional emotions [8]. Discrete emotions
are categorized as fear, anger, joy, sadness, etc., and the emotion lexicon
size varies from six [6] to twenty-seven categories [9]. Variation in discrete
emotion definitions results in annotation complexities, difficulty in
realizing consistent emotion labels, and failure to include rare and/or
complex emotional states. Dimensional emotion represents the emotion space
using a three-dimensional model of (i) Activation (reflects vocal energy),
(ii) Valence (indicates negativity or positivity), and (iii) Dominance
(specifies how strong or meek one may sound).

Prior work have shown that combining lexical- and acoustic- based
representations can boost model performance on emotion recognition from speech
[10, 11, 12, 13]. Recent studies have demonstrated that pre-trained model
representations can generate state-of-the-art emotion recognition performance
[13, 14], which can substantially improve valence estimation performance [12,
15]. For real world applications, it is imperative to assess the robustness of
speech emotion models against background distortions. The effect of noise in
speech emotion recognition and robustness of features and modeling approaches
have been studied in [16, 17], however, prior art have not investigated noise
robustness of pre-trained model representations.

In this work we demonstrate:
(1) Fusion of multiple lexical and acoustic representations obtained from pre-
trained models results in state-of-the-art dimensional emotion estimation
performance on a large publicly available real-world data-set.
(2) Pre-trained model representations (both lexical and acoustic) demonstrate
better robustness against noise and channel degradation compared to low-level
acoustic features.
(3) Lexical embeddings are more robust to acoustic distortions than their
acoustic counterparts.

##  2 Data

All experiments are run using the publicly available MSP-Podcast dataset 1.6
[18] containing speech from English speakers. The speech segments contain
single speaker utterances with a duration of 3 to 11 seconds. The data
contains manually assigned valence, activation and dominance scores (7-point
likert scores) from multiple graders. The data split is shown in Table 1. To
make our results comparable to literature [11, 12], we report results on
Eval1.3 and Eval1.6 (see Table 1). A subset of the data has manual
transcriptions which was used to perform the ASR-analyis presented in Section
4.

To analyze the robustness of the emotion models, we add noise and
reverberation to the Eval1.6 set. Noise (reflecting home appliance noise) was
added at 3 SNR levels: 20 to 30 dB (avg. 20 dB), 10 to 20 dB (avg. 15 dB), and
0 to 10 dB (avg. 5 dB). The SNR level for each file was randomly selected from
a uniform distribution. Hence, we created three additional evaluations sets
one for each of the three SNR levels (see Table 1). The noise-added data was
artificially reverberated to simulate realistic indoor conditions. For
performing noise-aware training, we added similar distortions to 9% of the
files from the training set ”Train+noise” in Table 1.

Table 1: MSP-podcast data split and noise-degraded sets. Split | hours | Description
---|---|---
Train1.6 | 85 | Training set
Valid1.6 | 15 | Validation set
Eval1.3 | 22 | Podcast1.3 evaluation set
Eval1.6 | 25 | Podcast1.6 evaluation set
Train+noise | 93 | Training set with 9% noisy data
$Eval1.6_{25dB}$ | 25 | Eval1.6 + noise within 20-30 dB
$Eval1.6_{15dB}$ | 25 | Eval1.6 + noise within 10-20 dB
$Eval1.6_{5dB}$ | 25 | Eval1.6 + noise within 0-10 dB

##  3 Representations & Acoustic Features

###  3.1 Acoustic Features

As a baseline acoustic feature we use 40-dimensional mel-filterbank energies
appended with pitch, pitch-delta, and voicing features, resulting in a
43-dimensional $MFBF_{0}$ feature.

###  3.2 Acoustic Representations from HuBERT

We explore embeddings generated from HuBERT large, a pre-trained acoustic
model [19], which was pre-trained on 60,000 hours of speech from the Libri-
light dataset with 24 transformer layers and 1024 embedding dimensions. In our
study, we extracted the following embeddings:
(1) $HUBERT_{L}$ embeddings from the $24^{th}$ layer of the pre-trained HuBERT
large model (no fine tuning).
(2) $HUBERT_{A}$ embeddings from the $HUBERT_{L}$ model ($24^{th}$ layer)
fine-tuned on an automatic speech recognition (ASR) task using 100 hours of
Librispeech data.

###  3.3 Lexical Representations from BERT

We investigated the use of pre-trained BERT [20] embeddings for emotion model
training. As ground truth transcriptions are only available for a subset of
speech segments in MSP Podcast data, we rely on two ASR models to estimate
transcripts from speech: $HUBERT_{A}$ and an in-house ASR system. Given
transcribed speech, we obtained 768-dimensional embeddings from the $12^{th}$
layer of the pre-trained BERT model.

##  4 Analysis of text-data from ASR models

We evaluated ASR performance on a portion of the MSP-Podcast data that
contained manual transcription. We used two separate ASR systems (an in-house
ASR system and the $HUBERT_{A}$ system) and Figure 1 shows how the word error
rate (WER) varied for different ranges (low, neutral and high) of valence,
activation, and dominance by treating each dimension as independent of the
others. Figure 1 show that for higher values of valence and activation the WER
increases, while the same is observed at lower dominance. The increase in WER
at high valence, high activation or low dominance indicate that ASR systems
made more mistakes under such conditions. Figure 1 indicates that BERT
embeddings from ASR transcriptions may be impacted by varying WERs at
different emotion levels.

Fig. 1: WER variation with low, neutral and high values of Valence, Activation
and Dominance, obtained from transcriptions generated by the ASR models (WER:
word error rate, WIL: word information lost, MER: match error rate)

##  5 Emotion Estimation Model

###  5.1 Model Architectures

All models presented here use the same basic architecture (see Figure 2) for
supervised dimensional emotion estimation: a time-convolutional (TC) layer
with skip connection followed by 2-layer GRUs whose output is projected to a
128-dimensional embedding layer, followed by an output layer.
Uni-Modal Input: The TC-GRU model takes one modality as input (any one of the
features described in Section 3). The input feature dimension and time
resolution may vary based on the feature type and its modality.
Multi-Modal Input: The lexical-acoustic fusion model uses a combination of
$HuBERT_{L}$ and $HuBERT_{A}$ embeddings (2048 dimension) which are projected
to 128 dimension as shown in Figure 3. The resulting 128 dimensional
embeddings are concatenated with embeddings obtained from TC-GRU models
trained with BERT embeddings from $HuBERT_{A}$ and in-house ASR
transcriptions, respectively.

Fig. 2: The basic TC-GRU network

Fig. 3: Multi-modal emotion estimation model

###  5.2 Model Training

For uni- and multi- modal inputs, 2-layer TC-GRU models were trained, where
the performance on a held-out Valid1.6 set (see Table 1) was used for model
selection. Concordance correlation coefficient ($CCC$) [21] is used as the
loss function ($L_{ccc}$) (see (1)), where ${L_{ccc}}$ is a combination
($\alpha=1/3$ and $\beta=1/3$) of $CCC$’s obtained from each of the three
dimensional emotions. $CCC$ is defined by (2), where ${\mu_{x}}$ and
${\mu_{y}}$ are the means, ${\sigma_{x}^{2}}$ and ${\sigma_{y}^{2}}$ are the
corresponding variances for the estimated and ground-truth variables, and
${\rho}$ is the correlation coefficient between those variables. The models
are trained with a mini-batch size of 32 and a learning rate of 0.0005.

$L_{ccc}=-(\alpha CCC_{v}+\beta CCC_{a}+(1-\alpha-\beta)CCC_{d})$ (1)
$CCC=\frac{2\rho\sigma_{x}\sigma_{y}}{\sigma_{x}^{2}+\sigma_{y}^{2}+(\mu_{x}-\mu_{y})^{2}}.$
(2)

###  5.3 Distillation

To investigate if knowledge can be distilled from the multi-modal models into
the simpler uni-modal models, we explored distilling the representations
learned by the former into the latter using a teacher-student framework. Let
the embedding from the multi-modal teacher network $T$ for a speech sample
${i}$ is ${E_{T,i}}$, the learned embedding from the student model $S$ for
sample $i$ be ${\hat{E}_{S,i}}$, the ground truth label be ${l_{k,i}}$ (where
${k\in\\{a,v,d\\}}$), and the estimated label from $T$ are
${\tilde{l}_{k,i}}$. We propose the distillation loss as:

$L_{dis,k}=CCC_{E_{T,i},\hat{E}_{S,i}}\cdot\gamma_{i},$ (3)
$\gamma_{i}=1-\frac{1}{M}\left(\frac{\sum_{k}|l_{k,i}-\tilde{l}_{k,i}|}{3}\right).$
(4)

In (3) the distillation loss is computed using the $CCC$ (see 2) between the
teacher (${E_{T,i}}$) and the student (${\hat{E}_{S,i}}$) embeddings, and is
weighted by the residual error ($|l_{k,i}-\tilde{l}_{k,i}|$) between the
target (${l_{k,i}}$) and the estimated (${\tilde{l}_{k,i}}$) labels of the
teacher network. We explored $L_{2}$ and $cosine$ distance loss for
distillation and found the proposed $CCC$ based loss to perform better. Note
that $M$ represents the dynamic range of the target labels ${l_{k,i}}$ in the
dataset. $\gamma_{i}$ provides a confidence measure for each sample, where it
is closer to 1 for samples that have low residual error from the teacher
network and closer to 0 when the residuals are high. The final loss is defined
as:

$L_{k}=\kappa(L_{ccc,k}+L_{CE,k})+\lambda L_{dis,k},$ (5)

where the values of $\kappa$ and $\lambda$ were empirically selected to be
0.001 and 1, to emphasize more on the distillation loss.

##  6 Results

###  6.1 Emotion estimation

We trained unimodal TC-GRU emotion models with the following inputs: (i)
${MFBF_{0}}$ based acoustic features, embeddings extracted from (ii)
$HuBERT_{L}$ and (iii) $HuBERT_{A}$, and (iv) lexical embeddings from BERT
($BERT_{HuB}$) using transcriptions generated by $HuBERT_{A}$ (note that BERT
embeddings from in-house ASR demonstrated similar performance as
$BERT_{HuB}$). We trained the multi-modal model (Figure 3) using speech
embeddings ($HuBERT_{L}$ and $HuBERT_{A}$ as input, combined with the $BERT$
embeddings. The performance of the uni- and multi- modal models for
dimensional emotion estimation is shown in Table 2. Table 2 shows that
features extracted from any given choice of pre-trained model (whether lexical
or acoustic), significantly improves valence estimation relative to
${MFBF_{0}}$. $BERT$ and $HuBERT_{L}$ based models demonstrated a $70\%$ and
$60\%$ relative improvement in $CCC$ on valence estimation compared to
${MFBF_{0}}$, respectively. Multi-modal fusion further improved valence
estimation relative to all uni-modal models (see Table 2), where we observe a
$100\%$, $40\%$ and $44\%$ relative improvement in $CCC$ on valence estimation
compared to the ${MFBF_{0}}$, $BERT$ and $HuBERT_{L}$ based models,
respectively. Table 2 shows that lexical representations may not be sufficient
by themselves for estimating activation and dominance, however, through multi-
modal fusion they achieve the best overall performance.

###  6.2 Model Robustness to Acoustic Distortions

We investigated the robustness of the models presented above, to acoustic
distortions unseen during training. Table 3 shows that noise degradation
impacts the performance of $MFBF_{0}$ more severely than the pre-trained model
representations. We observe that for valence, lexical embeddings are
relatively robust to noise than the acoustic embeddings.

Additionally, we explored noise-aware training using the Train+noise set
(Table 1). We observed improvement across all dimensions for $MFBF_{0}$
feature, while the noise-aware training had little or no impact on the BERT-
based models. BERT embeddings demonstrated robust performance under acoustic
distortions. Multi-modal fusion demonstrated overall the best state-of-the-art
performance. These findings indicate that the representations learned from the
pre-trained models demonstrate robustness against acoustic distortions.

Next, we investigated if representations learned by the multi-modal fusion
model can be distilled (using the the procedure outlined in section 5.3) into
the $MFBF_{0}$ and $HuBERT_{L}$ based models, to improve their robustness.
Tables 4 and 3 show that knowledge distillation from the multi-modal model
helped to improve the robustness of both the models trained with $MFBF_{0}$
and $HuBERT_{L}$ features.

Table 2: Dimensional emotion estimation $CCC$ from uni-modal models and multi-modal fusion model. System | Eval1.3 | Eval1.6
---|---|---
act | val | dom | act | val | dom
${MFBF_{0}}$ | 0.73 | 0.33 | 0.63 | 0.71 | 0.33 | 0.64
$BERT_{HuB}$ | 0.33 | 0.56 | 0.29 | 0.30 | 0.55 | 0.27
$HuBERT_{A}$ | 0.73 | 0.53 | 0.64 | 0.70 | 0.52 | 0.62
$HuBERT_{L}$ | 0.76 | 0.54 | 0.68 | 0.74 | 0.53 | 0.66
Multi-modal | 0.78 | 0.68 | 0.69 | 0.75 | 0.66 | 0.67
Table 3: Emotion estimation $CCC$ from models trained on Train1.6, Trian+noise and evaluated in noisy conditions. System | Train1.6 | Train+noise
---|---|---
act | val | dom | act | val | dom
${MFBF_{0}}$ |  |  |  |  |  |
$Eval1.6_{25dB}$ | 0.67 | 0.31 | 0.58 | 0.69 | 0.33 | 0.61
$Eval1.6_{15dB}$ | 0.64 | 0.23 | 0.56 | 0.69 | 0.30 | 0.60
$Eval1.6_{5dB}$ | 0.45 | 0.11 | 0.41 | 0.60 | 0.23 | 0.53
$BERT_{Hub}$ |  |  |  |  |  |
$Eval1.6_{25dB}$ | 0.31 | 0.55 | 0.27 | 0.29 | 0.55 | 0.26
$Eval1.6_{15dB}$ | 0.31 | 0.54 | 0.27 | 0.29 | 0.54 | 0.26
$Eval1.6_{5dB}$ | 0.29 | 0.51 | 0.26 | 0.28 | 0.51 | 0.26
$HuBERT_{A}$ |  |  |  |  |  |
$Eval1.6_{25dB}$ | 0.68 | 0.52 | 0.58 | 0.69 | 0.54 | 0.59
$Eval1.6_{15dB}$ | 0.65 | 0.51 | 0.57 | 0.68 | 0.53 | 0.58
$Eval1.6_{5dB}$ | 0.55 | 0.48 | 0.47 | 0.63 | 0.50 | 0.51
$HuBERT_{L}$ |  |  |  |  |  |
$Eval1.6_{25dB}$ | 0.72 | 0.51 | 0.61 | 0.71 | 0.51 | 0.61
$Eval1.6_{15dB}$ | 0.68 | 0.50 | 0.57 | 0.70 | 0.50 | 0.60
$Eval1.6_{5dB}$ | 0.52 | 0.44 | 0.38 | 0.64 | 0.45 | 0.55
Multi-modal |  |  |  |  |  |
$Eval1.6_{25dB}$ | 0.72 | 0.66 | 0.62 | 0.72 | 0.66 | 0.62
$Eval1.6_{15dB}$ | 0.68 | 0.64 | 0.59 | 0.70 | 0.64 | 0.59
$Eval1.6_{5dB}$ | 0.53 | 0.59 | 0.51 | 0.57 | 0.57 | 0.43

##  7 Conclusions

In this work, we investigated lexical and acoustic representations from pre-
trained models and explored their robustness under varying acoustic
conditions. We observed that lexical

Table 4: Emotion estimation ($CCC$) from uni-modal models after knowledge distillation from multi-modal model System | ${MFBF_{0}}$ | $HuBERT_{L}$
---|---|---
act | val | dom | act | val | dom
$Eval1.6_{25dB}$ | 0.70 | 0.35 | 0.60 | 0.73 | 0.55 | 0.63
$Eval1.6_{15dB}$ | 0.68 | 0.31 | 0.58 | 0.70 | 0.53 | 0.61
$Eval1.6_{5dB}$ | 0.54 | 0.22 | 0.46 | 0.54 | 0.47 | 0.43

representations were helpful for accurate estimation of valence but not for
activation and dominance. Low-level acoustic features, such as $MFBF_{0}$ are
found to be susceptible to noise degradation, however, noise-aware training
helped to improve its robustness. Models trained with $HuBERT_{L}$ embeddings
were relatively robust compared to $MFBF_{0}$, especially for valence
estimation across SNR levels, however activation and dominance estimation
suffered at low SNR levels. Multi-modal fusion helped generate better
representations providing the state-of-the-art performance that was robust
against all SNR conditions. Representations learned from multi-modal fusion
and distilling such information to acoustic only systems demonstrated
improvement in performance for those systems under noisy conditions. In the
future, we plan to investigate how domain mismatch can impact lexical features
and valence estimation.

## References

  * [1] Vikramjit Mitra, Sue Booker, Erik Marchi, David S Farrar, Ute D Peitz, Bridget Cheng, Ermine Teves, Anuj Mehta, and Devang Naik,  “Leveraging acoustic cues and paralinguistic embeddings to detect expression from voice,”  arXiv preprint arXiv:1907.00112, 2019.
  * [2] Vasudha Kowtha, Vikramjit Mitra, Chris Bartels, Erik Marchi, Sue Booker, William Caruso, Sachin Kajarekar, and Devang Naik,  “Detecting emotion primitives from speech and their use in discerning categorical emotions,”  in ICASSP. IEEE, 2020, pp. 7164–7168.
  * [3] DS Sahana and L Girish,  “Automatic drug reaction detection using sentimental analysis,”  International Journal of Advanced Research in Computer Engineering & Technology (IJARCET), vol. 4, no. 5, 2015.
  * [4] Brian Stasak, Julien Epps, Nicholas Cummins, and Roland Goecke,  “An investigation of emotional speech in depression classification.,”  in Interspeech, 2016, pp. 485–489.
  * [5] Jared B Torre and Matthew D Lieberman,  “Putting feelings into words: Affect labeling as implicit emotion regulation,”  Emotion Review, vol. 10, pp. 116–124, 2018.
  * [6] Paul Ekman,  “An argument for basic emotions,”  Cognition & emotion, vol. 6, no. 3-4, pp. 169–200, 1992.
  * [7] Robert Plutchik,  “The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice,”  American scientist, pp. 344–350, 2001.
  * [8] James A Russell and Albert Mehrabian,  “Evidence for a three-factor theory of emotions,”  Journal of research in Personality, vol. 11, pp. 273–294, 1977\.
  * [9] Alan S Cowen and Dacher Keltner,  “Self-report captures 27 distinct categories of emotion bridged by continuous gradients,”  Proceedings of the National Academy of Sciences, vol. 114, no. 38, pp. E7900–E7909, 2017.
  * [10] Saurabh Sahu, Vikramjit Mitra, Nadee Seneviratne, and Carol Y Espy-Wilson,  “Multi-modal learning for speech emotion recognition: An analysis and comparison of asr outputs with ground truth transcription.,”  in Interspeech, 2019, pp. 3302–3306.
  * [11] Ayoub Ghriss, Bo Yang, Viktor Rozgic, Elizabeth Shriberg, and Chao Wang,  “Sentiment-aware automatic speech recognition pre-training for enhanced speech emotion recognition,”  arXiv preprint arXiv:2201.11826, 2022.
  * [12] Sundararajan Srinivasan, Zhaocheng Huang, and Katrin Kirchhoff,  “Representation learning through cross-modal conditional teacher-student training for speech emotion recognition,”  arXiv preprint arXiv:2112.00158, 2021.
  * [13] Shamane Siriwardhana, Andrew Reis, Rivindu Weerasekera, and Suranga Nanayakkara,  “Jointly fine-tuning ”BERT-like” self supervised models to improve multimodal speech emotion recognition,”  arXiv preprint arXiv:2008.06682, 2020.
  * [14] Leonardo Pepino, Pablo Riera, and Luciana Ferrer,  “Emotion recognition from speech using wav2vec 2.0 embeddings,”  Proc. Interspeech, pp. 3400–3404, 2021.
  * [15] Vikramjit Mitra, Sherry Chien Hsiang-Yun, Vasudha Kowtha, Joseph Yitan Cheng, and Erdrin Azemi,  “Speech emotion: Investigating model representations, multi-task learning and knowledge distillation,”  in Proc. of Interspeech, 2022.
  * [16] Seong-Gyun Leem, Daniel Fulford, Jukka-Pekka Onnela, David Gard, and Carlos Busso,  “Not all features are equal: Selection of robust features for speech emotion recognition in noisy environments,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6447–6451.
  * [17] Andreas Triantafyllopoulos, Gil Keren, Johannes Wagner, Ingmar Steiner, and Björn Schuller,  “Towards robust speech emotion recognition using deep residual networks for speech enhancement,”  Proc. Interspeech 2019, pp. 1691–1695, 2019.
  * [18] Soroosh Mariooryad, Reza Lotfian, and Carlos Busso,  “Building a naturalistic emotional speech corpus by retrieving expressive behaviors from existing speech corpora,”  in Fifteenth Annual Conference of the International Speech Communication Association, 2014.
  * [19] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [20] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,”  in Proceedings of NAACL-HLT, 2019, pp. 4171–4186.
  * [21] I Lawrence and Kuei Lin,  “A concordance correlation coefficient to evaluate reproducibility,”  Biometrics, pp. 255–268, 1989.