11institutetext: Department of Physics, Federal University of Technology,
Akure, Ondo State, Nigeria. 22institutetext: Michael and Cecilia Ibru
University, Ughelli, Delta State, Nigeria.

# Bifurcation analysis and chaos control of periodically driven discrete
fractional order memristive Duffing Oscillator

Samuel Ogunjo Corresponding author: 11<EMAIL_ADDRESS>Ibiyinka Fuwape
1122<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Discrete fractional order chaotic systems extends the memory capability to
capture the discrete nature of physical systems. In this research, the
memristive discrete fractional order chaotic system is introduced. The
dynamics of the system was studied using bifurcation diagrams and phase space
construction. The system was found chaotic with fractional order
$0.465<n<0.562$. The dynamics of the system under different values makes it
useful as a switch. Controllers were developed for the tracking control of the
two systems to different trajectories. The effectiveness of the designed
controllers were confirmed using simulations.
PACS: 05.45.Tp; 05.45.-a

##  1 Introduction

The study of chaos arose from the study of integer order systems lorenz1963 .
This was soon extended to discrete systems to explain the behaviour in natural
systems and models especially population models fuwapebifurcation . Many
methods used in integer order and discrete systems have been developed over
the years to study observational data such as climate Fuwape2017 , economic
and financial data Lahmiri2017  and communication systems Fuwape2016 .
Although it has the capability to capture memory effect in models, fractional
order systems have not gained attention over the years due to inadequate
solution technique fuwapebifurcation . New developments in recent times such
as computation methods and computing powers have brought attention to
fractional order systems Shukla2017 .

There exist situations where it is desired that systems be chaotic. This can
be achieved through chaotification or anti-control qi . At other times when
the existence of chaos in a system is undesirable, the methods of chaos
control can be used to remove the effect of chaos in such system. Chaos
control involves obtaining a chaotic, periodic or stationary behaviour in a
chaotic system by the application of tiny perturbations or controllers
fuwapebifurcation ; Boccaletti2000 . Synchronization is a form of control
whereby the trajectory of a chaotic system is made to track the trajectory of
another system. Different forms of synchronization such as projective
synchronization, function projective synchronization, phase synchronization,
lag synchronization, complete synchronization, anticipated synchronization
have been proposed ojo2013comparison . Synchronization of chaotic systems has
been found useful in secure communication ojo2012synchronization ; Essimbi2012
.

The practical application of chaos to secure communication involves building
electronic circuits for its implementation. Chua chua1971  proposed a
nonlinear device, the memristor, as the fourth circuit element in addition to
the resistor, capacitor and inductor. The memristor is considered as a
nonlinear resistor with memory Danca2016 . Novel chaotic systems which have
been developed based on the memristor include the two component circuit
Tchitnga2012 , memristor biomembranes Volkov2016 , memristive oscillatory
systems Talukdar2012 , and Muthuswamy and Chua system muthuswamy . Memristors
have practical potential applications in high performance computing, dynamic
memory elements and neural synapses Sabarathinam .

Discrete fractional order implementation of systems have been investigated
including: Extension of integer order chaotic systems have been carried for
systems such as sine and standard maps wu2014discrete , Logistic map
wu2014chaos , Chua system Agarwal2013  and Henon map hu2014discrete ;
liu2016chaotic . In this paper, we extend the study of the Duffing oscillators
to the discrete fractional order model. By introducing a memristor into the
system, the different responses of the system to varying parameters were
investigated using bifurcation diagrams. Control of the memristive discrete
fractional order model was also carried out using the method of active
control.

##  2 Methods

###  2.1 Memristors

Different forms of flux dependent rate of change of charge have been proposed.
In this work, we study a flux controlled memristor of the form

$\phi(q)=\omega_{0}^{2}q+\beta q^{3}$ (1)

The memductance is obtained as

$M(q)=\frac{d\phi(q)}{dq}=\omega_{0}^{2}+3\beta q^{2}$ (2)

where $\omega_{0}^{2}$ and $\beta$ are constants.

###  2.2 Fractional order systems

The fractional sum of order $\nu<0$ can be defined as atici2009

$\Delta_{a}^{-\nu}x(t):=\frac{1}{\Gamma(\nu)}\sum_{s=a}^{t-\nu}(t-\sigma(s))^{(\nu-1)x(s)},\,\,t\in\\{a,a+1,a+2,\ldots\\}$
(3)

$\sigma(s)=s+1$, and $t^{(\nu)}$ is the falling function defined as

$t^{(\nu)}=\frac{\Gamma(t+1)}{\Gamma(t+1-\nu)}$ (4)

This definition was extended by Abdeljawad2011  for the $\nu$-order
($\nu>0,\nu\not\in\mathbb{N}$) Caputo-like delta difference as

$\Delta_{C}^{\nu}x(t):=\frac{1}{\Gamma(n-\nu)}\sum_{s=a}^{t-(n-\nu)}(t-\sigma(s))^{(n-\nu-1)}\Delta^{n}_{s}x(s)$
(5)

where $n=[\nu]+1$.

Using this transformation, a discrete fractional system can be written as
fuwapebifurcation

$\Delta^{\nu}_{a}x(t)=f(t+v-1,x(t+\nu-1)$ (6)

This has a solution given by

$x(t)=x_{0}+\frac{1}{\Gamma(a)}\sum_{s=1-\nu}^{t-\nu}(t-s-1)^{(\nu-1)}f(s+\nu-1,x(s+v-1)$
(7)

###  2.3 System description

Duffing oscillator is defined as

$m\ddot{x}+c\dot{x}+kx+ax^{3}=f\sin(\varpi t)$ (8)

where $m$, $c$, $k$, and $a$ are system constants. Substituting Equation 2 and
rewriting as a two dimensional system, the integer order Duffing oscillator
can be written as

$\begin{split}\dot{x}&=y\\\ \dot{y}&=f_{1}\sin(\varpi
t)-M(q)x-f_{2}y\end{split}$ (9)

where $f_{1}=\frac{f}{m}$, $3\beta=\frac{a}{m}$, $\omega_{0}^{2}=\frac{k}{m}$,
$M(q)=\omega_{0}^{2}+3\beta x^{2}$ and $f_{2}=\frac{c}{m}$. System 9 can be
written in fractional order form as

$\begin{split}\frac{d^{n}x}{dt^{n}}&=y\\\
\frac{d^{n}y}{dt^{n}}&=f_{1}\sin(\varpi t)-M(q)x-f_{2}y\end{split}$ (10)

where $n$ is the order of the system. Using the transformation described in
section 2.2, the discrete fractional form of the memristive Duffing oscillator
is written as

$\begin{split}\Delta^{\nu}_{a}x&=y(t+\nu-1)\\\
\Delta^{\nu}_{a}y&=f_{1}\sin(\varpi
t)-M(q)x(t+\nu-1)-f_{2}y(t+\nu-1)\end{split}$ (11)

Sabarathinam et al Sabarathinam  transformed equation 9 into a four
dimensional integer order system for analysis.

##  3 Results and discussion

###  3.1 Bifurcation analysis

The bifurcation analysis of the discrete fractional order Duffing oscillator
with respect to different system parameters was investigated and the results
presented in Figure 1. The system behaviour with respect to the fractional
order, $n$, shows a reverse period doubling route to chaos (Figure 1a).
Chaotic regions were found in the region $0.465<n<0.562$ with transition to
period-2 and period-3 bifurcation at $n=0.86$ and $n=0.57$ respectively
(Figure 1b). A period doubling route to chaos was also observed when the
parameter $\omega_{0}^{2}$ was varied. The system was found to be chaotic in
only small positive region $0.517<\omega_{0}^{2}<0.566$. Using system
parameters $(f_{1},n,3\beta,f_{2},\varpi)=(0.5,0.5,1,0.4,0.0002)$, the Duffing
oscillator with periodic forcing was found to be chaotic when
$-0.43<\omega_{0}^{2}<-0.232$. By varying the parameter $f_{2}$, the system
was found to be chaotic when $0.288<f_{2}<0.45$ (Figure 1c). The parameter
$\varpi$ presents an interesting bifurcation structure in the region
$0<\varpi<1$. In the small region $0<\varpi<0.01$, the system was found to be
chaotic with a reverse period doubling bifurcation. In Figure 2, the system
showed periodic behaviour in the region $0.01<\varpi<0.2$ beyond which it
showed unusual chaotic behaviour. Thus, we can conclude that the system
follows a period doubling route to chaos when $0<\varpi<0.01$ but a
quasiperiodic route when $\varpi>0.01$). The bifurcation diagram obtained by
varying the memristor in the system is shown in Figure 3 and the phase space
when the fractional order is varied is presented in Figure 4. The behaviour of
the memristor at different values suggest possible applications as a switch.

###  3.2 Chaos control

According to Fuwape and Ogunjo fuwapebifurcation , tracking control is
obtained when the individual components of a system followed different
predefined rules. The aim is to control the $x$ and $y$ component of System
(11) to follow different trajectories. Rewriting the memristive discrete
fractional order system (System 11) and adding controllers, we have

$\begin{split}\Delta^{\nu}_{a}x&=y(t+\nu-1)+u_{1}\\\
\Delta^{\nu}_{a}y&=f_{1}\sin(\varpi
t)-M(x)x(t+\nu-1)-f_{2}y(t+\nu-1)+u_{2}\end{split}$ (12)

The error term is written as

$\begin{split}\dot{e}_{1}&=\dot{x}-\dot{c}_{1}\\\
\dot{e}_{2}&=\dot{y}-\dot{c}_{2}\end{split}$ (13)

where $c(c_{1},c_{2})$ are user defined functions. If $c_{1}\neq c_{2}$, mixed
tracking is obtained. Substituting Equation 13 into Equation 11, we obtain

$\begin{split}\dot{e}_{1}&=e_{2}+c_{2}-c_{1}\\\ \dot{e}_{2}&=f_{1}\sin(\varpi
t)-\omega_{0}^{2}e_{1}-\omega_{0}^{2}c_{1}-3\beta
x^{3}-f_{2}e_{2}-f_{2}c_{2}-c_{2}\end{split}$ (14)

Eliminating nonlinear terms in $e_{1},\,e_{2}$ gives the subcontroller
functions

$\begin{split}u_{1}&=-c_{2}+c_{1}+v_{1}\\\ u_{2}&=-f_{1}\sin(\varpi
t)+\omega_{0}^{2}c_{1}+3\beta x^{3}+f_{2}c_{2}+c_{2}+v_{2}\end{split}$ (15)

where the parameters $v_{i}$ will be obtained later. Substituting Equation 15
into Equation 14, we obtain

$\begin{pmatrix}\dot{e}_{1}\\\ \dot{e}_{2}\\\
\end{pmatrix}=\begin{pmatrix}0&-1\\\ \omega_{0}^{2}&f_{2}\\\
\end{pmatrix}\begin{pmatrix}e_{1}\\\ e_{2}\\\
\end{pmatrix}+\begin{pmatrix}v_{1}\\\ v_{2}\\\ \end{pmatrix}$ (16)

The method of active control requires that a constant matrix $\mathbb{K}$ is
chosen which will control the error dynamics such that the feedback matrix
becomes

$\begin{pmatrix}v_{1}\\\ v_{2}\\\
\end{pmatrix}=\mathbb{K}\begin{pmatrix}e_{1}\\\ e_{2}\\\ \end{pmatrix}$

Thus, a matrix of the form

$\mathbb{K}=\begin{pmatrix}\lambda_{1}&-1\\\
\omega_{0}^{2}&\lambda_{2}+f_{2}\\\ \end{pmatrix}$ (17)

To verify the effectiveness of the tracking controllers designed above,
numerical simulations were carried out. The $x$ component was made to track
and exponential function while the $y$ component tracked a sine wave. The
result obtained is shown in Figure 5. The trajectories of the $x$ and $y$
component were found to track the desired set functions, hence, we conclude
that tracking control of the system has been achieved.

##  4 Conclusion

Recently, there has been renewed interest in fractional order systems. In this
research, we extend the chaotic fractional order Duffing system to a discrete-
fractional order system with a memristor. Bifurcation diagrams were produced
to show the different behaviour of the system under different parameter
changes. The possibility of control the two state space of the discrete
fractional order memristive Duffing oscillator to two different trajectory was
investigated. The effectiveness of the controllers were demonstrated by
numerical simulations.

Figure 1: Bifurcation diagram for y-component with (a) fractional order, $n$
with parameters
$(f_{1},\omega_{0}^{2},3\beta,f_{2},\varpi)=(0.9,-0.35,1.5,0.4,0.0002)$. (b)
$\omega_{0}^{2}$ with system parameters
$(f_{1},n,3\beta,f_{2},\varpi)=(0.5,0.5,1,0.4,0.0002)$ (c) parameter, $f_{2}$
using $(f_{1},\omega_{0}^{2},3\beta,n,\varpi)=(0.5,-0.35,1,0.5,0.0002)$; and
(d) $\varpi$ with system parameters
$(f_{1},\omega_{0}^{2},3\beta,f_{2},n)=(0.5,-0.35,1,0.4,0.5)$

Figure 2: Bifurcation diagram for y-component with $\varpi$ in the range
$0<\varpi<1$

Figure 3: Bifurcation diagram for y-component with the memristor $M(q)$

Figure 4: $x-y$ phase space realization of the driven Duffing oscillator
obtained with parameter
$(f_{1},\omega_{0}^{2},3\beta,f_{2},\varpi)=(0.9,-0.35,1.5,0.4,0.0002)$ and
$0<n<1$.

Figure 5: Tracking control of the driven Duffing oscillator obtained with
parameter
$(f_{1},\omega_{0}^{2},3\beta,f_{2},\varpi,n)=(0.9,-0.35,1.5,0.4,0.0002,0.5)$.
The $x$ component (red line) is made to track the function $A\sin(wt)$ and the
$y$ component (black line) tracks the function $Ae^{kt}$, where $A,w,k$ are
constants.

## References

  * (1) T Abdeljawad Comput. Math. with Appl. 62 1602 (2011).
  * (2) R P Agarwal, A M A El-Sayed, S M Salman Advances in Difference Equations 2013 320 (2013).
  * (3) F M Atici, P W Eloe Proc. Am. Math. Soc. 137 981 (2009)
  * (4) S Boccaletti Physics Reports 329 103 (2000).
  * (5) L O Chua IEEE Transactions on Circuit Theory 18 507 (1971)
  * (6) M F Danca, W K Tang, G Chen Chaos, Solitons & Fractals 84 31 (2016).
  * (7) I A Fuwape, S Ogunjo International Journal of Dynamics and Control In press 1 (2018)
  * (8) I A Fuwape, S T Ogunjo, J B Dada, G A Ashidi, I Emmanuel Journal of Atmospheric and Solar-Terrestrial Physics 149 46 (2016).
  * (9) I A Fuwape, S T Ogunjo, S S Oluyamo, A B Rabiu Theoretical and Applied Climatology 130 119 (2017)
  * (10) T Hu Applied Mathematics 5 2243 (2014)
  * (11) S Lahmiri Physica A Statistical Mechanics and its Applications 466 389 (2017).
  * (12) Y Liu Indian Journal of Physics 90 313 (2016)
  * (13) E N Lorenz Journal of the Atmospheric Sciences 20 130 (1963).
  * (14) A D Mengue, B Z Essimbi Nonlinear Dynamics 70 1241 (2012).
  * (15) B Muthuswamy, L O Chua Int J Bifurcat Chaos 20 1567 (2010)
  * (16) K S Ojo, A N Njah, S T Ogunjo Pramana 80 825 (2013).
  * (17) K S Ojo, S T Ogunjo Journal of Nigerian Association of Mathematical Physics 21 35 (2012)
  * (18) G Qi, G Chen Phys. Lett. A. 352 386 (2006)
  * (19) S Sabarathinam, C K Volos, K Thamilmaran Nonlinear Dynamics 87 37 (2017)
  * (20) M K Shukla, B B Sharma AEU - International Journal of Electronics and Communications 78 265 (2017).
  * (21) A Talukdar, A Radwan, K Salama Microelectronics Journal 43 169 (2012).
  * (22) R Tchitnga, H B Fotsin, B Nana, P H Louodop Fotso, P Woafo Chaos, Solitons & Fractals 45 306 (2012).
  * (23) A Volkov, V Markin Memristors in Biomembranes.  In: Advances in Biomembranes and Lipid Self-Assembly (2016).
  * (24) G C Wu, D Baleanu Signal processing 102 96 (2014)
  * (25) G C Wu, D Baleanu, S D Zeng Physics Letters A 378 484 (2014)


# Domain Adversarial for Acoustic Emotion Recognition

Mohammed Abdelwahab, and Carlos Busso M. Abdelwahab and C. Busso are with the
Department of Electrical and Computer Engineering, The University of Texas at
Dallas, Richardson TX, 75013 USA e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>received March 26, 2018; revised xxx xx, 2018.

###### Abstract

The performance of speech emotion recognition is affected by the differences
in data distributions between train (source domain) and test (target domain)
sets used to build and evaluate the models. This is a common problem, as
multiple studies have shown that the performance of emotional classifiers drop
when they are exposed to data that does not match the distribution used to
build the emotion classifiers. The difference in data distributions becomes
very clear when the training and testing data come from different domains,
causing a large performance gap between validation and testing performance.
Due to the high cost of annotating new data and the abundance of unlabeled
data, it is crucial to extract as much useful information as possible from the
available unlabeled data. This study looks into the use of adversarial
multitask training to extract a common representation between train and test
domains. The primary task is to predict emotional attribute-based descriptors
for arousal, valence, or dominance. The secondary task is to learn a common
representation where the train and test domains cannot be distinguished. By
using a gradient reversal layer, the gradients coming from the domain
classifier are used to bring the source and target domain representations
closer. We show that exploiting unlabeled data consistently leads to better
emotion recognition performance across all emotional dimensions. We visualize
the effect of adversarial training on the feature representation across the
proposed deep learning architecture. The analysis shows that the data
representations for the train and test domains converge as the data is passed
to deeper layers of the network. We also evaluate the difference in
performance when we use a shallow neural network versus a _deep neural
network_ (DNN) and the effect of the number of shared layers used by the task
and domain classifiers.

###### Index Terms:

Speech emotion recognition, adversarial training, unlabeled adaptation of
acoustic emotional models.

##  I Introduction

In many practical applications for speech emotion recognition systems, the
testing data (target domain) is different from the labeled data used to train
the models (source domain). The mismatch in data distribution leads to a
performance degradation of the trained models [1, 2, 3, 4, 5]. Therefore, it
is vital to develop more robust systems that are more resilient to changes in
train and test conditions [6, 7, 8, 9]. One approach to ensure that models
perform well on the target domain is to use training data drawn from the same
distribution. However, this approach can be expensive, since it requires
enough data with emotional labels to build models specific to a new target
domain. A more practical approach is to use available labeled data from
similar domains along with unlabeled data from the target domain, creating
models that generalize well to the new testing conditions without the need to
annotate extra data with emotional labels. This study proposes an elegant
solution for the problem of mismatch in train-test data distributions based on
domain adversary training.

Figure 1: Formulation of the problem of data mismatch between train (source
domain) and test (target domain) datasets. Our solution uses unlabeled data
from the target domain to reduce the differences between domains.

We formulate the machine-learning problem as follows. We have a _source_
domain(s) with annotated emotional data, which is used to train the models,
and a large _target_ domain with unlabeled data (see Fig. 1). The testing data
come from the target domain. Due to the prohibitive cost of annotating new
data every time we change the target domain and the abundance of unlabeled
data, we aim to use the unlabeled target data to extract useful information,
reducing the differences between source and target domains. The envisioned
system generalizes better and is more robust by maximizing the performance
using a shared data representation for source and target domains. The key
principle in our approach is to find a consistent feature representation for
the source and target domains. Common approaches to address this problem
include feature transformation methods, where the representation of the source
data is mapped into a representation that resembles the features in the target
domain [10], and finding a common representation between the domains, such
that the features are invariant across domains [11, 12, 13]. The common
domain-invariant features do not necessarily contain useful information about
the main task. Therefore, it is vital to constrain the learned common feature
representation to ensure that it is discriminative with respect to the main
task.

This paper explores the idea of finding a common representation between the
source and target domains, such that data from the domains become
indistinguishable while maintaining the critical information used in emotion
recognition. This work is inspired by the work of Ganin et al. [14], which
proposed training an adversarial multitask network. The approach searches the
feature space for a representation that accurately describes the data from
either domain, containing the relevant information to accurately classify the
main task, in our study, the prediction of emotional attributes (arousal,
valence, and dominance). The discriminative and domain-invariant features are
learned by aligning the data distributions from the domains through back-
propagation. This approach allows our framework to use unlabeled target data
to learn a flexible representation.

This paper shows that adversarial training using unlabeled training data
benefits emotion recognition. By using the abundant unlabeled target data, we
gain on average 27.3% relative improvement in _concordance correlation
coefficient_ (CCC) compared to just training with the source data. We evaluate
the effect of adversarial training by visualizing the similarity of the data
representation learned by the network from both domains. The visualization
shows that adversarial training aligns the data distributions as expected,
reducing the gap between source and target domains. The study also shows the
effect of the number of shared layers between the domain classifier and the
emotion predictor on the performance of the system. The size of the source
domain is an important factor that determines the optimal number of shared
layers in the network. This novel framework for emotion recognition provides
an appealing approach to effectively leverage unlabeled data from a new
domain, generalizing the models and improving the classification performance.

This paper is organized as follows. Section II discusses previous work on
speech emotion recognition, with emphasis on frameworks that aim to reduce the
mismatch between train and test datasets. Section III presents our proposed
model, providing the motivation and the details of the proposed framework.
Section IV presents the experiment evaluation including the databases, network
structure and acoustic features. Section V presents the experimental results
and the analysis of the main findings. Section VI finalizes the study with
conclusions and future research directions.

##  II Related Work

The key challenge in speech emotion recognition is to build classifiers that
perform well under various conditions. The cross-corpora evaluation in Shami
and Verhels [15] demonstrated the drop in classification performance observed
when training on one emotional corpus and testing on another. Other studies
have shown similar results [16, 17, 18, 5]. Several approaches have been
proposed to solve this problem. Shami and Verhels [15] proposed to include
more variability in the training data by merging emotional databases. They
demonstrated that it is possible to achieve classification performance
comparable to within-corpus results. More recently, Chang and Scherer [19]
showed that data from other domains can improve the within-corpus performance
of a neural network. They employed _deep convolutional generative adversarial
network_ (DCGAN) to extract and learn useful feature representation from
unlabeled data from a different domain. This led to better generalization
compared to a model that did not make use of unlabeled data.

The main approach to attenuate the mismatch between train and test conditions
is to minimize the differences in the feature space between both domains.
Zhang et al. [20] showed that by separately normalizing the features of each
corpus, it is possible to minimize cross-corpus variability. Hassan et al.
[21] increased the weight of the train data that matches the test data
distribution by using _kernel mean matching_ (KMM), _Kullback-Leibler
importance estimation procedure_ (KLEIP), and _unconstrained least-squares
importance fitting_ (uLSIF). Zong et al. [8] used _least square regression_
(LSR) to mitigate the projected mean and covariance differences between source
data and unlabeled target samples while learning the regression coefficient
matrix. Because the learned coefficient matrix depends on the samples selected
from the target domain, multiple matrices were estimated and used to test new
samples. Each matrix was used to predict an emotional label for a test sample,
combining the results with the majority vote rule.

Studies have also explored mapping both train and test domains to a common
space, where the feature representation is more robust to the variations
between the domains. Deng et al. [11] used auto-encoders to find a common
feature representation across the domains. They trained an auto-encoder such
that it minimized the reconstruction error on both domains. Building upon this
work, Mao et al. [22] proposed to learn a shared feature representation across
domains by constraining their model to share the class priors across domains.
Sagha et al. [23], also motivated by the work of Deng et al. [11], used
_principal component analysis_ (PCA) along with _kernel canonical correlation
analysis_ (KCCA) to find views with the highest correlation between the source
and target corpora. First, they used PCA to represent the feature space of the
source and target data. Then, the features for the source and target domains
were projected using the PCA in both domains. Finally, they used KCCA to
select the top $N$ dimensions that maximized the correlation between the
views. Inspired by universum learning where unlabeled data is used to
regularize the training process for _support vector machine_ (SVM), Deng et
al. [24] proposed adding an universum loss to the reconstruction loss of an
auto-encoder. The added loss function was defined as the addition of the
$L_{2}$-margin loss and the $\epsilon$-insensitive loss, making use of both
labeled and unlabeled data. This approach aimed to learn auto-encoder
classifier has low reconstruction and classification errors on both domains.

Song et al. [25] proposed a couple of methods based on _non-negative matrix
factorization_ (NMF) that utilized data from both train and test domains. The
proposed methods aimed to represent a matrix formed by data from both domains
as two non-negative matrices whose product is an approximation of the original
matrix. The factorization was regularized by _maximum mean discrepancy_ (MMD)
to ensure that the differences in the feature distributions of the two corpora
were minimized. The proposed methods aim to learn a robust low dimensional
feature representation using either unlabeled data or labels as hard
constraints on the problem. Abdelwahab and Busso [6] proposed creating an
ensemble of classifiers, where each classifier focuses on a different feature
space (each classifier maximized the performance for a given emotion
category). The features were selected over the labeled data from the target
domain obtained with active learning. This semisupervised approach learned
discriminant features for the target domain, increasing the robustness against
shift in the data distributions between domains.

Instead of finding a common representation between domains, Deng et al. [26]
trained a sparse auto-encoder on the target data and used it to reconstruct
the source data. This approach used feature transformation in a way that
exploits the underlying structure in emotional speech learned from the target
data. Deng et al. [27] used two denoising auto-encoders. The first auto-
encoder was trained on the target data and the second auto-encoder was trained
on the source data, but it was constrained to be close to the first auto-
encoder. The second auto-encoder was then used to reconstruct the data from
both source and target domains.

Our proposed approach takes an innovative formulation with respect to previous
work relying on _domain adversarial neural network_ (DANN) [14]. While
Shinohara [28] showed that the use of DANN can increase the robustness of a
_automatic speech recognition_ (ASR) system against certain types of noise,
this framework has not been used for speech emotion recognition, which is an
important contribution as this framework can reduce the mismatch between train
and test sets in a principled way. DANN relies on adversarial training for
domain adaptation to learn a flexible representation during the training of
the emotion classifier. As the training data changes, both the emotion and
domain classifiers readjust their weights to find the new representation that
satisfies all conditions. The domain classifier can be considered as a
regularizer that prevents the main classifier from over-fitting to the source
domain. The final learned representation performs well on the target domain
without sacrificing the performance on the source domain.

##  III Proposed Approach

###  III-A Motivation

We present an unsupervised approach to reduce the mismatch between source and
target domains by creating a discriminative feature representation that
leverages unlabeled data from the target domain.

We aim to learn a common representation between the source and target domains,
where samples from both domains are indistinguishable to a domain classifier.
This approach is useful because all the knowledge learned while training the
classifier on the source domain is directly applicable to the target domain
data. We learn the representation by using a gradient reversal layer where the
gradient produced by the domain classifier is multiplied by a negative value
when it is propagated back to the shared layers. Changing the sign of the
gradient causes the feature representation of the samples from both domains to
converge, reducing the gap between domains. Ideally, the performance of the
domain classifiers should be at random level where both domains “look” the
same. When such a representation is learned, the data distributions of both
domains are aligned. This approach leads to a large performance improvement in
the target domain, as demonstrated by our experimental evaluation (see Section
V). A key feature of this framework is that it is unsupervised, since it does
not require labeled data from the target domain. However, this framework would
continue to be useful when labeled data from the target domain is available,
working as a semi-supervised approach.

###  III-B Domain Adversarial Neural Network for Emotion Recognition

Figure 2: Architecture of the _domain adversarial neural network_ (DANN)
proposed for emotion recognition. The network has three parts: a feature
representation common to both tasks, a task classifier layer, and a domain
classifier layer.

Ganin et al. [14], inspired by the recent work on _generative adversarial
networks_ (GAN) [29], proposed the _domain adversarial neural network_ (DANN).
The network is trained using labeled data from the source domain and unlabeled
data from the target domain. The network learns two classifiers: the main
classification task, and the domain classifier, which determines whether the
input sample is from the source or target domains. Both classifiers share the
first few layers that determine the representation of the data used for
classification. The approach introduced a gradient reversal layer between the
domain classifier and the feature representation layers. The layer passes the
data during forward propagation and inverts the sign of the gradient during
backward propagation. The network attempts to minimize the task classification
error and find a representation that maximizes the error of the domain
classifier. By considering these two goals, the model ensures a discriminative
representation for the main recognition task that makes the samples from
either domain indistinguishable.

Figure 2 shows an example structure of the DANN network. The network is fed
labeled source data and unlabeled target data in equal proportions. In our
formulation, we propose to predict emotional attribute descriptors as the
primary goal. We train the primary recognition task with the source data, for
which we have emotional labels. For the domain classifier, we train the
classifier with data from the source and target domains. Notice that the
domain classifier does not require emotional label, so we can rely on
unlabeled data from the target domain. The classifiers are trained in
parallel. The network’s objective is defined as:

$\begin{split}&E(\theta_{f},\theta_{y},\theta_{d})=\frac{1}{n}\sum_{i=1}^{n}L_{y}^{i}(\theta_{f},\theta_{y})\\\
&-\lambda\Big{(}\frac{1}{n}\sum_{i=1}^{n}L_{d}^{i}(\theta_{f},\theta_{d})+\frac{1}{m}\sum_{i=1}^{m}L_{d}^{i}(\theta_{f},\theta_{d})\Big{)}\end{split}$
(1)

where $\theta_{f}$ represents the parameters of the shared layers providing
the regularized feature representation, $\theta_{y}$ represents the parameters
of the layers associated with the main prediction task, and $\theta_{d}$
represents the parameters of the layers of the domain classifier ($n$ is the
number of labeled training samples, $m$ is the number of unlabeled training
samples). The optimization process consists of two loss functions. $L_{y}^{i}$
is the prediction loss for the main task, and $L_{d}^{i}$ is the domain
classification loss. The prediction loss and the domain classification loss
compete against each other in an adversarial manner. The parameter $\lambda$
is a regularization multiplier that controls the tradeoff between the losses.
This is a minimax problem. It attempts to find a saddle point parametrized by
$\hat{\theta}_{f},\hat{\theta}_{y},\hat{\theta}_{d}$,

$(\hat{\theta}_{f},\hat{\theta}_{y})=\operatorname*{arg\,min}_{\theta_{f},\theta_{y}}E(\theta_{f},\theta_{y},\hat{\theta}_{d})$
(2)
$\hat{\theta}_{d}=\operatorname*{arg\,max}_{\theta_{d}}E(\hat{\theta}_{f},\hat{\theta}_{y},\theta_{d})$
(3)

At the saddle point, the classification loss on the source domain is minimized
and the domain classification loss is maximized. The maximization is achieved
by introducing a gradient reversal layer that changes the sign of the gradient
going from the domain classification layers to the feature representation
layers (see white layer in Fig. 2). The updates taken on the feature
representation parameters are in opposite direction to the gradient. With this
approach, stochastic gradient descent tries to make the features similar
across domains, so what is learned from the source domain remains effective
for the target domain without loss in performance.

The simple concept of choosing a representation that confuses a competent
domain classifier leads to models that perform better in the target domain
without impacting the performance in the source domain. This approach is
particularly useful for emotion recognition, as most annotated corpora come
from studio settings that greatly differs from real world testing conditions.
This unsupervised approach uses available unlabeled data to align the
distributions of both domains. The aligned distributions lead to a common
representation, causing the domain classifier’s performance to drop to random
chance levels. The common indistinguishable representation retains
discriminative information learned during the training of the models with data
from the source domain. We improve the classifier’s performance on the target
domain without having to collect new annotated data. This is an important
contribution in this field, taking us one step closer toward robust speech
emotion classifiers that generalize well in most testing conditions.

##  IV Experimental Evaluation

We define the main task as a regression problem to estimate the emotional
content conveyed in speech described by the emotional attributes arousal (calm
versus activated), valence (negative versus positive), and dominance (weak
versus strong). This section describes the databases (Section IV-A), the
acoustic features (Section IV-B) and the specific network structures (Section
IV-C) used in the experimental evaluation.

###  IV-A Emotional Databases

The experimental evaluation consider a multi-corpus setting with three
databases. The source domain (test set) corresponds to two databases: the USC-
IEMOCAP [30] and MSP-IMPROV [31] corpora. The target domain corresponds to the
MSP-Podcast [32] database.

####  IV-A1 The USC-IEMOCAP Corpus

The USC-IEMOCAP database is an audiovisual corpus recorded from ten actors
during dyadic interactions [30]. It has approximately 12 hours of recordings
with detailed motion capture information carefully synchronized with audio
(this study only uses the audio). The goal of the data collection was to
elicit natural emotions within a controlled setting. This goal was achieved
with two elicitation frameworks: emotional scripts, and improvisation of
hypothetical scenarios. These approaches allowed the actors to express
spontaneous emotional behaviors driven by the context, as opposed to read
speech displaying prototypical emotions [33]. Several dyadic interactions were
recorded and manually segmented into turns. Each turn was annotated with
emotional labels by at least two evaluators across emotional attributes
(valence, arousal, dominance). Dimensional attributes take integer values that
range from one to five. The dimensional attribute of an utterance is the
average of the values given by the annotators. We linearly map the scores
between $-3$ and 3.

####  IV-A2 The MSP-IMPROV Corpus

The MSP-IMPROV database is a multimodal emotional database recorded from
actors interacting in dyadic sessions [31]. The recordings were carefully
designed to promote natural emotional behaviors, while maintaining control
over lexical and emotional contents. The corpus relied on a novel elicitation
scheme, where two actors improvised scenarios that lead one of them to utter
target sentences. For each of these target sentences, four emotional scenarios
were created to contextualize the sentence to elicit happy, angry, sad and
neutral reactions, respectively. The approach allows the actor to express
emotions as dictated by the scenarios, avoiding prototypical reactions that
are characteristic of other acted emotional corpus. Busso et al. [31] showed
that the target sentences occurring within these improvised dyadic
interactions were perceived more natural than read renditions of the same
sentences. The MSP-IMPROV corpus includes not only the target sentences, but
also other sentences during the improvisations that led one of the actors to
utter the target sentence. It also includes the natural interactions between
the actors during the breaks.

The corpus consists of 8,438 turns of emotional sentences recorded from 12
actors (over 9 hours). The sessions were manually segmented into speaking
turns, which were annotated with emotional labels using perceptual evaluations
conducted with crowdsourcing [34]. Each turn was annotated by at least five
evaluators, who annotated the emotional content in terms of the dimensional
attributes arousal, valence, and dominance. Dimensional attributes take
integer values that range from one to five. The consensus label assigned to
each speech turn is the average value of the scores provided by the
evaluators, which we linearly map between $-3$ and 3.

####  IV-A3 The MSP-Podcast Corpus

The _MSP-Podcast_ corpus is an extensive collection of natural speech from
multiple speakers appearing in Creative Commons licensed recordings downloaded
from audio-sharing websites [32]. Some of the key aspects of the corpus are
the different conditions in which the recordings are collected, large number
of speakers, and a large variety of natural content from spontaneous
conversations conveying emotional behaviors. The audio was preprocessed
removing portions that contain noise, music or overlapped speech. The
recordings were then segmented into speaking turns creating a big audio
repository with sentences that are between 2.75 seconds and 11 seconds.
Emotional models trained with existing databases are then used to retrieve
speech turns with target emotional content. The candidate segments were
annotated with emotional labels using an improved version of the crowdsourcing
framework proposed by Burmania et al. [34].

Each speech segment was annotated by at least five raters, who provided scores
for the emotional dimensions arousal, valence and dominance using seven-likert
scales. The consensus scores are the average scores assigned by the
evaluators, which are shifted such that they are in the range between $-3$ and
3. The collection of this corpus is an ongoing effort. This study uses 14,227
labeled sentences. From this set, we use 4,283 labeled sentences coming from
50 speakers as our test set, which is consistently used across conditions. For
the within corpus evaluation (i.e., training and testing in the same domain),
we define a development set with 1,860 sentences from 10 speakers, and a train
set with the remainder of the corpus (8,084 sentences). This study also uses
73,209 unlabeled sentences from the audio repository of segmented speech
turns. The unlabeled segments are used to train the domain classifier in the
DANN approach.

###  IV-B Acoustic Features

The acoustic features correspond to the set proposed for the INTERSPEECH 2013
Computational Paralinguistics Challenge (ComParE) [35]. This feature set
includes 6,373 acoustic features extracted at the sentence level (one feature
vector per sentence). First, it extracts 65 frame-by-frame _low-level
descriptors_ (LLDs) which includes various acoustic characteristics such as
_Mel-frequency cepstral coefficients_ (MFCCs), fundamental frequency, and
energy. The externalization of emotion is conveyed through different aspects
of speech production so including these LLDs is important to capture emotional
cues. After estimating LLDs, a list of functions are estimated for each LLD,
which are referred to as _high-level descriptors_ (HLD) features. These HLDs
include standard deviation, minimum, maximum, and ranges. The acoustic
features are extracted using OpenSMILE [36].

We separately normalize the features from each domain (i.e., corpus) to have
zero mean and a unit standard deviation. The mean and the variance of the data
is calculated considering only the values of the features within the 5% and
95% quantiles to avoid outliers skewing the values. After normalization, we
ignore any value greater than 10 times its standard deviation by setting their
values to zero.

###  IV-C Network Structure

As discussed in Section III-B, the DANN approach has three main components:
the domain classifier layers, task classifier layers, and feature
representation layers. The _domain classifier layers_ are implemented with two
layers across all the experimental evaluation. The _task classifier layers_
are also implemented with two layers, except for the shallow network described
below. The number of layers in the _feature representation layers_ is a
parameter of the network that is set on the development set. We consider
different number of shared layers, evaluating the performance of the system
with one, two, three or four layers.

We also study whether a simple shallow network can achieve similar performance
compared to the deep network. In the shallow network, the _task classifier
layer_ and the _feature representation layer_ are implemented with one layer
each. We implement the _domain classifier layer_ with two layers.

###  IV-D Baseline Systems

We establish two baselines. The first baseline is a network trained and
validated only on the source data. This condition creates a mismatch between
the train and test conditions. The second baseline corresponds to within
corpus evaluations, where the models are trained and tested with data from the
target domain. This model assumes that training data from the target domain is
available, so it corresponds to the ideal condition. The parameters of the
networks are optimized using the development set. The baselines are
implemented with similar architectures, serving as a fair comparison with the
proposed method (e.g., number of layers, number of nodes). The key difference
with the DANN models is the lack of the _domain classification layers_ , where
the feature representation layers only consider the primary classification
task.

###  IV-E Implementation

We train the networks using Keras [37] with Tensorflow as back-end [38]. We
use batch normalization and dropout. The dropout rates are $p$=0.2 for the
input layer and $p$=0.5 for the rest of the layers. We further regularize the
models using max-norm on the weights of value four and a clip norm on the
gradient of value ten. The loss function used for the main regression task is
the _mean square error_ (MSE). The loss function for the domain classification
task is the categorical cross-entropy. We use Adam as an optimizer with a
learning rate of 5$e^{-4}$ [39]. We train the models for 100 epochs with a
batch size of 256 sentences. A parameter of the DANN model is $\lambda$ in
Equation 1, which controls the tradeoff between the task and domain
classification losses. We follow a similar approach to the one proposed by
Ganin et al. [14], where $\lambda$ is initialized equal to zero for the first
ten training epochs. Then, we slowly increase its value until reaching
$\lambda=1$ by the end of the training. We train each model twenty times to
reduce the effect of initialization on the performance of the classifiers. We
report the average performance across the trails.

In adversarial training, we need unlabeled data to train the domain
classifier. We randomly select samples from the unlabeled portion of the
target domain to be fed to the domain classifier. The number of selected
samples from the audio repository of unlabeled speech turns is equal to the
number of samples in the source domain, keeping the balance in the training
set of the domain classification task.

##  V Experimental Results

The performance results for the baseline models and the DANN models are
reported in terms of the _root mean square error_ (RMSE), _Pearson’s
correlation coefficient_ (PR), and _concordance correlation coefficient_ (CCC)
between the ground-truth labels and the estimated values. While we presented
PR and RMSE, the analysis focus on CCC as the performance metric, which
combines _mean square error_ (MSE) and PR. CCC is defined as:

$\rho_{c}=\frac{2\rho\sigma_{x}\sigma_{y}}{\sigma_{x}^{2}+\sigma_{y}^{2}+(\mu_{x}-\mu_{y})^{2}}$
(4)

where $\rho$ is the Pearson’s correlation coefficient, $\sigma_{x}$ and
$\sigma_{y}$, and $\mu_{x}$ and $\mu_{y}$ are the standard deviations and
means of the predicted score $x$ and the ground truth label $y$, respectively.

###  V-A Number of Layers For the Shared Feature Representation

We first study the effect of the number of shared layers between the domain
classifier and the primary regression task on the DANN model’s performance
(e.g., _feature representation layers_ in Fig. 2). The domain and task
classifier layers are implemented with two layers each. We vary the number of
shared layers between the classifiers and observe how the changes in feature
representation affect the regression performance. This evaluation is conducted
exclusively on the validation set of the target domain (e.g., MSP-Podcast
corpus).

Table I shows the average RMSE, PR, and CCC for the models trained with one,
two, three and four shared layers. For arousal, we consistently observe better
performance (lower RMSE and higher PR and CCC) with one shared layer between
the domain and task classifiers. For the CCC values, the differences are
statistically significant for all the cases, except when the source domain is
the MSP-IMPROV corpus and the DANN model is implemented with two layers (one-
tailed t-test over the average across the twenty trails, asserting
significance if $p$-value $<0.05$). The performance degrades as more shared
layers are added. For valence and dominance, the number of shared layers that
provides the best performance varies from one corpus to another. In most
cases, two or three shared layers provide the best performance. Based on these
results on the validation set, we set the number of shared layers for the
feature representation networks to one for arousal, two for valence and three
for dominance.

TABLE I: Performance of DANN framework implemented with different number of shared layers for the domain and task classifiers [_Iem_ : USC-IEMOCAP corpus, _Imp_ : MSP-IMPROV corpus, _All_ : All corpora combined] | Arousal | Valence | Dominance
---|---|---|---
src | n | RMSE | PR | CCC | RMSE | PR | CCC | RMSE | PR | CCC
Iem | 1 | 1.26 | .412 | .365 | 1.57 | .140 | .135 | 1.30 | .151 | .124
2 | 1.29 | .379 | .332 | 1.64 | .152 | .144 | 1.27 | .160 | .135
3 | 1.32 | .353 | .305 | 1.67 | .150 | .140 | 1.27 | .217 | .181
4 | 1.31 | .347 | .296 | 1.67 | .147 | .135 | 1.27 | .192 | .161
Imp | 1 | 1.62 | .497 | .284 | 1.58 | .171 | .137 | 0.80 | .485 | .448
2 | 1.57 | .506 | .303 | 1.47 | .203 | .176 | 0.82 | .514 | .476
3 | 1.71 | .399 | .218 | 1.48 | .202 | .173 | 0.87 | .478 | .403
4 | 1.73 | .382 | .202 | 1.43 | .210 | .193 | 0.85 | .442 | .383
All | 1 | 1.37 | .432 | .341 | 1.56 | .181 | .165 | 1.01 | .395 | .356
2 | 1.41 | .396 | .313 | 1.56 | .173 | .160 | 1.05 | .345 | .309
3 | 1.46 | .369 | .286 | 1.58 | .183 | .170 | 1.06 | .339 | .308
4 | 1.45 | .371 | .282 | 1.56 | .161 | .149 | 1.06 | .307 | .270

###  V-B Regression Performance of the DANN Model

This section presents the regression results achieved by the DANN model, and
the baseline models. Table II lists the performance for the within-corpus
evaluation where the models are trained and tested with data from the MSP-
Podcast corpus (referred to as _target_ on the table), and the cross-corpus
evaluations, where the models are trained with other corpora (referred to as
_src_ on the table). These baseline results are compared with our proposed
DANN method (referred to as _dann_ on the table). The results are tabulated in
terms of emotional dimensions and networks structures. These values are the
average of twenty trails to account for different initializations and set of
unlabeled data used to train the DANN models. For the rows denoted “All
Databases”, we combine all the source domain together (IEMOCAP and MSP-IMPROV
corpora), treating them as a single source domain. To determine statistical
differences between the _src_ and _dann_ conditions, we use the one-tailed
t-test over the twenty trails, asserting significance if $p$-value $<0.05$. We
highlight in bold when the difference between these conditions is
statistically significant.

To visualize the results in Table II, we create figures showing the average
performance under different conditions (Figs. 3-6). We use statistical
significance tests to compare different conditions (values obtained from Table
II). We test the hypothesis that population means for matched conditions are
different using one-tailed z-test. We assert significance if $p$-value
$<0.05$. We use an asterisk in the figures to indicate if there is a
statistically significant difference relative to the baseline model trained
with the source domain.

TABLE II: Performance of the proposed DANN approach across different structures and emotional dimensions. Training Data | Arousal
---|---
Source | Approach | deep | shallow
RMSE | PR | CCC | RMSE | PR | CCC
MSP-Podcast | target | 0.67 | .786 | .776 | 0.66 | .787 | .767
USC-IEMOCAP | src | 1.01 | .515 | .452 | 1.00 | .510 | .434
dann | 1.10 | .503 | .489 | 1.07 | .520 | .503
MSP-IMPROV | src | 1.57 | .551 | .267 | 1.53 | .555 | .263
dann | 1.48 | .607 | .381 | 1.46 | .614 | .381
All Databases | src | 1.20 | .496 | .386 | 1.18 | .506 | .378
dann | 1.18 | .555 | .499 | 1.18 | .551 | .486
Training Data | Valence
Source | Approach | deep | shallow
RMSE | PR | CCC | RMSE | PR | CCC
MSP-Podcast | target | 1.08 | .327 | .294 | 1.03 | .344 | .283
USC-IEMOCAP | src | 1.30 | .202 | .198 | 1.19 | .227 | .207
dann | 1.44 | .218 | .215 | 1.34 | .267 | .255
MSP-IMPROV | src | 1.42 | .142 | .122 | 1.29 | .154 | .127
dann | 1.43 | .163 | .161 | 1.37 | .180 | .178
All Databases | src | 1.33 | .214 | .201 | 1.16 | .245 | .228
dann | 1.39 | .299 | .294 | 1.33 | .272 | .267
Training Data | Dominance
Source | Approach | deep | shallow
RMSE | PR | CCC | RMSE | PR | CCC
MSP-Podcast | target | 0.57 | .718 | .697 | 0.57 | .723 | .704
USC-IEMOCAP | src | 0.95 | .437 | .393 | 0.87 | .461 | .426
dann | 1.10 | .437 | .401 | 1.03 | .497 | .457
MSP-IMPROV | src | 0.86 | .563 | .368 | 0.87 | .520 | .353
dann | 0.89 | .565 | .456 | 0.88 | .578 | .472
All Databases | src | 0.85 | .481 | .418 | 0.81 | .493 | .437
dann | 0.92 | .526 | .499 | 0.90 | .550 | .516
Figure 3: Average concordance correlation coefficient across conditions. The
DANN models provide significant improvements over the baseline model trained
with the source domain.

Figure 3 shows the average concordance correlation coefficient across
emotional dimensions, training sources, trails, and networks structures (three
emotional dimensions $\times$ twenty trails $\times$ three sources $\times$
five structures = 900 matched conditions). The average performance for the
within-corpus evaluation (target) is close to double the performance for the
cross-corpus evaluations (source). This result demonstrates the importance of
minimizing the mismatch between train and test conditions in emotion
recognition. The figure shows that the proposed DANN approach greatly improves
the performance, achieving 6.6% gain compared to the systems trained with the
source domains. As highlighted by the asterisk, the improvement is
statistically significant. The proposed approach reduces the gap between
within-corpus and cross-corpus emotion recognition results, effectively using
unlabeled data from the target domain.

Figure 4: Average concordance correlation coefficient across conditions for
each emotional dimension.

Figure 4 shows the average concordance correlation coefficient for each
emotional dimension (twenty trails $\times$ three sources $\times$ five
structures = 300 matched conditions). On average, the figure shows that models
trained using DANN consistently outperform models trained with the source
data. The asterisk denotes that the difference is statistically significant
across all emotional dimensions. The relative improvements over the source
models are 22.8% for arousal, 33.4% for valence and 15.5% for dominance. In
general, the values for CCC are lower for valence, validating findings from
previous studies, which indicated that acoustic features are less
discriminative for this emotional attribute [40].

Figure 5: Average concordance correlation coefficient across condition for
each source domain. The solid line represents the within corpus performance.

Figure 5 shows the average concordance correlation coefficient per source
domain (three emotional dimensions $\times$ twenty trails $\times$ five
structures = 300 matched conditions). The figure shows the within-corpus
performance (target) with a solid horizontal line. The results consistently
show significant improvements when using DANN. The relative improvements in
performance over training with the source domain are 7.7% for the USC-IEMOCAP
corpus, 36.4% for the MSP-IMPROV corpus, and 25% when we combine all the
corpora. Figure 5 also shows that, on average, combining all the sources into
one domain improves the performance of the systems in recognizing emotions.
Adding variability during the training of the models is important, as also
demonstrated by Shami and Verhels [15]. DANN models also benefit from adding
variability. By leveraging the added data representations, DANN models are
able to find a common representation between the domains without sacrificing
the critical features relevant for learning the primary task.

Figure 6: Average concordance correlation coefficient across conditions for
deep and shallow structures.

Figure 6 compares the performance for deep and shallow networks (see Section
IV-C). The figure summarizes the results for the within-corpus evaluations
(target), cross-corpus evaluations (source) and with our proposed DANN model
(three emotional dimensions $\times$ twenty trails $\times$ three sources =
180 matched conditions). For the target models, we observe significant
improvements when using deep structures over shallow structures. However, the
differences are not statistically significant for the source and DANN models.

(a) Feature representation using the DANN model

(b) Feature representation using the baseline source model

Figure 7: Feature representation at the last shared layer for arousal models
trained on USC-IEMOCAP corpus. The figures are created with the t-SNE toolkit.

(a) Layer 1 - DANN

(b) Layer 2 - DANN

(c) Layer 3 - DANN

(d) Layer 4 - DANN

(e) Layer 1 - Source

(f) Layer 2 - Source

(g) Layer 3 - Source

(h) Layer 4 - Source

Figure 8: Feature representation at each layer using the DANN model, and the
baseline DNN trained with the source domain. The figures are created with the
t-SNE toolkit. The example corresponds to models for dominance with four
shared layers, using the MSP-IMPROV corpus as the source domain. Figures
(a)-(d) report the results for DANN model. Figures (e)-(h) report the results
for DNN trained with the source domain.

###  V-C Data Representation

The results in Tables I and II demonstrate the benefits of using the proposed
DANN framework. This section aims to understand the key aspect of the DANN
approach by visualizing the feature representation created during the training
process.

We use the t-SNE toolkit [41] to create 2D projections of the feature
distributions at different layers of the networks. Figure 7 shows the
distributions of the data from the source domain (blue circles) and the target
domain (red crosses) after projecting them in the feature representation
created by two models. This example corresponds to the models trained for
arousal using the USC-IEMOCAP corpus as the source domain (as explained in
Section V-A, the DANN model for arousal has only one shared layer as a feature
representation). Figure 7(a) shows the data representation at the shared layer
of the DANN model. Figure 7(b) shows the data representation at the equivalent
layer of the DNN model trained with the source domain (i.e., the USC-IEMOCAP
database). By using adversarial domain training, the feature distribution for
samples from both domains are almost indistinguishable, demonstrating that the
proposed approach is able to find a common representation. Without adversarial
domain training, in contrast, there are large regions in the feature space
where it is easy to separate samples from the source and target domains.
Figure 7(b) suggests the presence of a source-target mismatch which affects
the performance of the emotion classifier.

We also explore the feature representation when the DANN model is trained with
four shared layers using the t-SNE toolkit. The objective of this evaluation
is to visualize the distribution of the data in each of the shared layers.
This evaluation is implemented using the models for dominance, using the MSP-
IMPROV corpus as the source domain. Figures 8(a)-8(d) show the changes in the
data representation for each of the four shared layers in the DANN model. At
the first layer (Fig. 8(a)), the feature representation for the source data
(blue circles) and the target data (red crosses) are dissimilar enough for the
domain classifier to be able to distinguish them. While the difference between
domains in the feature representation has decreased at the second layer (Fig.
8(b)), there are still some regions dominated by samples from one of the two
domains. At the third layer (Fig. 8(c)), the feature representation of the
domains are similar enough to confuse the domain classifier. The common
representation is maintained in the fourth layer (Fig. 8(d)), where the data
from the target domain is indistinguishable from the data from the source
domain. This final representation is used by the emotion regression to predict
the emotional state of the target speech. For comparison, we also trained a
baseline model with six layers, matching the combined number of shared and
task classifier layers in the DANN model. Figures 8(e)-8(h) show the feature
representation of the corresponding first four layers of this model. In the
DANN model, the data representation of the samples from the source and target
domains become more similar in deeper layers. This trend is not observed in
the model trained with only the source data. The DANN model effectively
reduces the mismatch in the feature representation across domains, which leads
to significant improvements in the regression models.

##  VI Conclusions

This study proposed an appealing framework for emotion recognition that
exploits available unlabeled data from the target domain. The proposed
approach relies on _domain adversarial neural network_ (DANN), which creates a
flexible and discriminant feature representation that reduces the gap in the
feature space between the source and target domains. By using adversarial
training, we learned domain invariant representations that can effectively
discriminate the primary regression task. The model aims to find a balanced
representation that aligns the domain distributions, while retaining crucial
information for the primary regression task. The proposed adversarial training
leads to significant improvements in the emotion recognition classifier’s
performance over models exclusively trained with data from the source domain,
which was demonstrated by the experimental evaluation. We visualized the data
representation of both domains by projecting the features into the shared
layers of the proposed DANN model. The results showed that the model converged
to a common representation, where the source and target domains became
indistinguishable. The experimental evaluation also showed that the amount of
labeled data from the source domain plays a small role in determining how many
shared layers are needed between the domain and regression tasks. Since the
number of shared layers has a strong impact on the system’s performance, it is
vital to identify the optimal number of shared layers, given a specific source
domain.

One challenging aspect in using the proposed approach is the difficulty of
training adversarial networks. For example, Shinohara [28] noted that in ASR
problems, the improvements of DANN were large for some types of noises, but
less effective for others. They suggested that tuning the parameters could
lead to better results. We also observed that the framework failed to converge
for certain parameters, which is common in minimax problems. When properly
trained, however, this powerful framework can elegantly solve one of the most
important problems in speech emotion recognition: reducing the mismatch
between train and test domains.

In the case of multiple sources, our approach seems to work well when multiple
sources are combined, treating them as one. This approach forces the network
to learn a representation that is common across all the source domains. We
hypothesize that a better approach is to use asymmetric transformations, where
the model learns multiple possible representations for the test data, creating
one representation for each source. During testing, the network chooses the
most useful representation for each data point. Another alternative approach
is to transform the available sources to match the target domains. Finally,
this unsupervised approach can be easily extended to the cases where limited
labeled data from the target domain is available (semi-supervised approach),
creating a flexible framework to create emotion recognition systems that can
generalize across domain.

## Acknowledgment

This study was funded by the National Science Foundation (NSF) CAREER grant
IIS-1453781.

## References

  * [1] C. Busso, M. Bulut, and S.S. Narayanan,  “Toward effective automatic recognition systems of emotion in speech,”  in Social emotions in nature and artifact: emotions in human and human-computer interaction, J. Gratch and S. Marsella, Eds., pp. 110–127. Oxford University Press, New York, NY, USA, November 2013.
  * [2] Y. Kim and E. Mower Provost,  “Say cheese vs. smile: Reducing speech-related variability for facial emotion recognition,”  in ACM International Conference on Multimedia (MM 2014), Orlando, FL, USA, November 2014, pp. 27–36.
  * [3] D. Ververidis and C. Kotropoulos,  “Automatic speech classification to five emotional states based on gender information,”  in European Signal Processing Conference (EUSIPCO 2004), Vienna, Austria, September 2004, pp. 341–34.
  * [4] T. Vogt and E. André,  “Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition,”  in IEEE International Conference on Multimedia and Expo (ICME 2005), Amsterdam, The Netherlands, July 2005, pp. 474–477.
  * [5] S. Parthasarathy and C. Busso,  “Jointly predicting arousal, valence and dominance with multi-task learning,”  in Interspeech 2017, Stockholm, Sweden, August 2017, pp. 1103–1107.
  * [6] M. Abdelwahab and C. Busso,  “Ensemble feature selection for domain adaptation in speech emotion recognition,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), New Orleans, LA, USA, March 2017, pp. 5000–5004.
  * [7] M. Abdelwahab and C. Busso,  “Incremental adaptation using active learning for acoustic emotion recognition,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), New Orleans, LA, USA, March 2017, pp. 5160–5164.
  * [8] Y. Zong, W. Zheng, T. Zhang, and X. Huang,  “Cross-corpus speech emotion recognition based on domain-adaptive least-squares regression,”  IEEE Signal Processing Letters, vol. 23, no. 5, pp. 585–589, May 2016.
  * [9] M. Abdelwahab and C. Busso,  “Supervised domain adaptation for emotion recognition from speech,”  in International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2015), Brisbane, Australia, April 2015, pp. 5058–5062.
  * [10] T. Rahman and C. Busso,  “A personalized emotion recognition system using an unsupervised feature adaptation scheme,”  in International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2012), Kyoto, Japan, March 2012, pp. 5117–5120.
  * [11] J. Deng, R Xia, Z. Zhang, Y. Liu, and B. Schuller,  “Introducing shared-hidden-layer autoencoders for transfer learning and their application in acoustic emotion recognition,”  in International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2014), Florence, Italy, May 2014, pp. 4818–4822.
  * [12] Y. Zhang, Y. Liu, F. Weninger, and B. Schuller,  “Multi-task deep neural network with shared hidden layers: Breaking down the wall between emotion representations,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), New Orleans, LA, USA, March 2017, pp. 4490–4494.
  * [13] X. Glorot, A. Bordes, and Y. Bengio,  “Domain adaptation for large-scale sentiment classification: A deep learning approach,”  in International conference on machine learning (ICML 2011), Bellevue, WA, USA, June-July 2011, pp. 513–520.
  * [14] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky,  “Domain-adversarial training of neural networks,”  Journal of Machine Learning Research, vol. 17, no. 59, pp. 1–35, April 2016.
  * [15] M. Shami and W. Verhelst,  “Automatic classification of expressiveness in speech: A multi-corpus study,”  in Speaker Classification II, C. Müller, Ed., vol. 4441 of Lecture Notes in Computer Science, pp. 43–56. Springer-Verlag Berlin Heidelberg, Berlin, Germany, August 2007.
  * [16] A. Austermann, N. Esau, L. Kleinjohann, and B. Kleinjohann,  “Fuzzy emotion recognition in natural speech dialogue,”  in IEEE International Workshop on Robot and Human Interactive Communication, Nashville, TN, USA, August 2005, pp. 317–322.
  * [17] B. Schuller, B. Vlasenko, F. Eyben, M. Wöllmer, A. Stuhlsatz, A. Wendemuth, and G. Rigoll,  “Cross-corpus acoustic emotion recognition: Variances and strategies,”  IEEE Transactions on Affective Computing, vol. 1, no. 2, pp. 119–131, July-Dec 2010.
  * [18] L. Vidrascu and L. Devillers,  “Anger detection performances based on prosodic and acoustic cues in several corpora,”  in Second International Workshop on Emotion: Corpora for Research on Emotion and Affect, International conference on Language Resources and Evaluation (LREC 2008), Marrakech, Morocco, May 2008, pp. 13–16.
  * [19] J. Chang and S Scherer,  “Learning representations of emotional speech with deep convolutional generative adversarial networks,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), New Orleans, LA, USA, March 2017, pp. 2746–2750.
  * [20] Z. Zhang, F. Weninger, M. Wollmer, and B. Schuller,  “Unsupervised learning in cross-corpus acoustic emotion recognition,”  in IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU 2011), Waikoloa, HI, USA, December 2011, pp. 523–528.
  * [21] A. Hassan, R. Damper, and M. Niranjan,  “On acoustic emotion recognition: compensating for covariate shift,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 21, no. 7, pp. 1458–1468, July 2013.
  * [22] Q. Mao, W. Xue, Q. Rao, F. Zhang, and Y. Zhan,  “Domain adaptation for speech emotion recognition by sharing priors between related source and target classes,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2016), Shanghai, China, March 2016, pp. 2608–2612.
  * [23] H. Sagha, J. Deng, M. Gavryukova, J. Han, and B. Schuller,  “Cross lingual speech emotion recognition using canonical correlation analysis on principal component subspace,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2016), Shanghai, China, March 2016, pp. 5800–5804.
  * [24] J. Deng, X. Xu, Z. Zhang, S. Frühholz, and B. Schuller,  “Universum autoencoder-based domain adaptation for speech emotion recognition,”  IEEE Signal Processing Letters, vol. 24, no. 4, pp. 500–504, April 2017.
  * [25] P. Song, W. Zheng, S. Ou, X. Zhang, Y. Jin, J. Liu, and Y. Yu,  “Cross-corpus speech emotion recognition based on transfer non-negative matrix factorization,”  Speech Communication, vol. 83, pp. 34–41, October 2016.
  * [26] J. Deng, Z. Zhang, E. Marchi, and B. Schuller,  “Sparse autoencoder-based feature transfer learning for speech emotion recognition,”  in Affective Computing and Intelligent Interaction (ACII 2013), Geneva, Switzerland, September 2013, pp. 511–516.
  * [27] J. Deng, Z. Zhang, F. Eyben, and B. Schuller,  “Autoencoder-based unsupervised domain adaptation for speech emotion recognition,”  IEEE Signal Processing Letters, vol. 21, no. 9, pp. 1068–1072, September 2014.
  * [28] Y. Shinohara,  “Adversarial multi-task learning of deep neural networks for robust speech recognition,”  in Interspeech 2016, San Francisco, CA, USA, September 2016, pp. 2369–2372.
  * [29] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio,  “Generative adversarial nets,”  in Advances in neural information processing systems (NIPS 2014), Montreal, Canada, December 2014, vol. 27, pp. 2672–2680.
  * [30] C. Busso, M. Bulut, C.C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J.N. Chang, S. Lee, and S.S. Narayanan,  “IEMOCAP: Interactive emotional dyadic motion capture database,”  Journal of Language Resources and Evaluation, vol. 42, no. 4, pp. 335–359, December 2008.
  * [31] C. Busso, S. Parthasarathy, A. Burmania, M. AbdelWahab, N. Sadoughi, and E. Mower Provost,  “MSP-IMPROV: An acted corpus of dyadic interactions to study emotion perception,”  IEEE Transactions on Affective Computing, vol. 8, no. 1, pp. 67–80, January-March 2017.
  * [32] R. Lotfian and C. Busso,  “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,”  IEEE Transactions on Affective Computing, vol. To appear, 2018.
  * [33] C. Busso and S.S. Narayanan,  “Recording audio-visual emotional databases from actors: a closer look,”  in Second International Workshop on Emotion: Corpora for Research on Emotion and Affect, International conference on Language Resources and Evaluation (LREC 2008), Marrakech, Morocco, May 2008, pp. 17–22.
  * [34] A. Burmania, S. Parthasarathy, and C. Busso,  “Increasing the reliability of crowdsourcing evaluations using online quality assessment,”  IEEE Transactions on Affective Computing, vol. 7, no. 4, pp. 374–388, October-December 2016.
  * [35] B. Schuller, S. Steidl, A. Batliner, A. Vinciarelli, K. Scherer, F. Ringeval, M. Chetouani, F. Weninger, F. Eyben, E. Marchi, M. Mortillaro, H. Salamin, A. Polychroniou, F. Valente, and S. Kim,  “The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism,”  in Interspeech 2013, Lyon, France, August 2013, pp. 148–152.
  * [36] F. Eyben, M. Wöllmer, and B. Schuller,  “OpenSMILE: the Munich versatile and fast open-source audio feature extractor,”  in ACM International conference on Multimedia (MM 2010), Florence, Italy, October 2010, pp. 1459–1462.
  * [37] F. Chollet,  “Keras: Deep learning library for Theano and TensorFlow,” https://keras.io/, April 2017.
  * [38] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D.G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng,  “TensorFlow: A system for large-scale machine learning,”  in Symposium on Operating Systems Design and Implementation (OSDI 2016), Savannah, GA, USA, November 2016, pp. 265–283.
  * [39] T. Dozat,  “Incorporating Nesterov momentum into Adam,”  in Workshop track at International Conference on Learning Representations (ICLR 2015), San Juan, Puerto Rico, May 2015, pp. 1–4.
  * [40] C. Busso and T. Rahman,  “Unveiling the acoustic properties that describe the valence dimension,”  in Interspeech 2012, Portland, OR, USA, September 2012, pp. 1179–1182.
  * [41] L. Van Der Maaten,  “Accelerating t-SNE using tree-based algorithms,”  Journal of machine learning research, vol. 15, no. 1, pp. 3221–3245, October 2014.