# The Frobenius-Euler function and its applications

Serkan Araci University of Gaziantep, Faculty of Science and Arts, Department
of Mathematics, 27310 Gaziantep, TURKEY<EMAIL_ADDRESS>,  Deyao Gao Yuren
Lab., No. 8 Tongsheng Road, Changsha, P. R. China<EMAIL_ADDRESS>and
Mehmet Acikgoz University of Gaziantep, Faculty of Science and Arts,
Department of Mathematics, 27310 Gaziantep, TURKEY<EMAIL_ADDRESS>
###### Abstract.

In the present paper, we deal with Fourier-transformation of Frobenius-Euler
polynomials. We shall give its applications by using infinite series. Our
applications possess interesting properties which we state in this paper.

2010 Mathematics Subject Classification. 11S80, 11B68.

Keywords and phrases. Frobenius-Euler numbers and polynomials, Fourier
transformation, infinite series.

##  1\. Introduction

The ordinary Frobenius-Euler numbers are defined by means of the following
generating function:

(1.1)
$\sum_{n=0}^{\infty}H_{n}\left(u\right)\frac{t^{n}}{n!}=e^{H\left(u\right)t}=\frac{1-u}{e^{t}-u}\text{.}$

where, in the umbral calculus, $H^{n}\left(u\right)$ is symbolically replaced
by $H_{n}\left(u\right)$ in the formal series expansion of

$e^{tH\left(u\right)}=\sum_{n=0}^{\infty}H_{n}\left(u\right)\frac{t^{n}}{n!}\text{.}$

From expression of this definition, we state the following

(1.2)
$\left(H\left(u\right)+1\right)^{n}-uH_{n}\left(u\right)=\left\\{\begin{array}[]{cc}1-u&\text{if
}n=0,\\\ 0&\text{if }n\in\mathbb{N},\end{array}\right.$

where $\mathbb{N}$ denotes the set of positive integers.

From (1.2), we note that

$H_{0}(u)=1,H_{1}\left(u\right)=-\frac{1}{1-u},H_{2}\left(u\right)=\frac{1+u}{\left(1-u\right)^{2}},\cdots.$

The Frobenius Euler polynomials are also introduced as

(1.3)
$e^{xt}\frac{1-u}{e^{t}-u}=\sum_{n=0}^{\infty}\frac{t^{n}}{n!}H_{n}\left(x,u\right)\text{.}$

By (1.1) and (1.3), we can find the following

$H_{n}\left(x,u\right)=\sum_{l=0}^{n}\binom{n}{l}x^{n-l}H_{l}\left(u\right)\text{.}$

By expression of (1.2), it is not difficult to show that the recurrence
relation for the Frobenius-Euler numbers as follows:

(1.4)
$\sum_{l=0}^{n}\binom{n}{l}H_{l}\left(u\right)-uH_{n}\left(u\right)=\left(1-u\right)\delta_{0,n}$

where $\delta_{m,n}$ is the Kronecker delta, is defined by

$\delta_{m,n}=\left\\{\begin{array}[]{cc}\text{ }1,&\text{if }m=n\\\ 0,&\text{
if }m\neq n.\end{array}\right.$

Thus, we easily procure the following:

(1.5) $H_{n}\left(1,u\right)-uH_{n}\left(u\right)=0\text{ }\left(\text{for
}n\in\mathbb{N}\right)\text{.}$

Thus, we arrive the following lemma.

######  Lemma 1.

For $n\in\mathbb{N}$, we have

$H_{n}\left(1,u\right)=uH_{n}\left(u\right)\text{.}$

Substituting $u=-1$ in the above lemma, it leads to

$H_{n}\left(-1,u\right)=E_{n}\left(1\right)=-E_{n}$

where $E_{n}$ is called Euler numbers, as is well-known, Euler numbers are
defined by the following generating function:

$\sum_{n=0}^{\infty}E_{n}\frac{t^{n}}{n!}=\frac{2}{e^{t}+1}$

(for more informations on this subjects, see[1-20]).

Recently, Fourier transformation of the special functions have been studied by
many mathematicians (cf., [1], [2], [4], [13], [14], [16]). In [16], Luo gave
Fourier expansions of Apostol-Bernoulli and Apostol-Euler polynomials and
derived some integral representations of Apostol-Bernoulli and Apostol-Euler
polynomials by using Fourier expansions. After, Bayad [1] introduced as
theoretical identities of the Fourier transformation of the Apostol-Bernoulli,
Apostol-Euler and Apostol-Genocchi polynomials. Next, T. Kim also defined the
Euler function which is Fourier transformation of Euler polynomials. We easily
see that Kim’s method is different from Bayad and Luo. Actually, Kim’s paper
[3, pp. 131-136] motivated us to write this paper. Thus, we also give Fourier
transformation of Frobenius-Euler function by using Kim’s method. In this
paper, we also show that this function is related to Lerch trancendent
$\Phi\left(z,s,a\right)$.

##  2\. On the Frobenius-Euler function

In this section, we consider Frobenius-Euler function by using infinite
series. For $m\in\mathbb{N}$, the Fourier transformation of Frobenius-Euler
function is introduced as

(2.1)
$H_{m}\left(x,u\right)=\sum_{n=-\infty}^{\infty}a_{n}^{\left(m\right)}\left(u\right)e^{\left(2n+1\right)\pi
ix}\text{, }\left(a_{n}^{\left(m\right)}\left(u\right)\in\mathbb{C}\right)$

where $\mathbb{C}$ denotes the set of complex numbers and

(2.2)
$a_{n}^{\left(m\right)}\left(u\right)=\int_{0}^{1}H_{m}\left(x,u\right)e^{-\left(2n+1\right)ix\pi}dx\text{.}$

By applying some technical method on (2.2), we procure the following

$\displaystyle a_{n}^{\left(m\right)}\left(u\right)$ $\displaystyle=$
$\displaystyle\left[\frac{H_{m+1}\left(x,u\right)}{m+1}e^{-\left(2n+1\right)ix\pi}\right]_{0}^{1}-\frac{\left(2n+1\right)\pi
i}{m+1}\int_{0}^{1}H_{m+1}\left(x,u\right)e^{-\left(2n+1\right)ix\pi}dx$
$\displaystyle=$
$\displaystyle-\frac{u+1}{m+1}H_{m+1}\left(u\right)-\frac{\left(2n+1\right)\pi
i}{m+1}a_{n}^{\left(m+1\right)}\left(u\right)\text{.}$

So from above, it leads to the following

$a_{n}^{\left(m+1\right)}\left(u\right)=\left[a_{n}^{\left(m\right)}+\left(u+1\right)\frac{H_{m+1}\left(u\right)}{m+1}\right]\frac{m+1}{\left(\left(2n+1\right)\pi
i\right)}\text{.}$

By continuing this process, becomes as follows:

(2.3a) $\displaystyle
a_{n}^{\left(m\right)}\left(u\right)=\left[a_{n}^{\left(1\right)}\left(u\right)+\left(u+1\right)\frac{H_{2}\left(u\right)}{2}\right]\frac{m!}{\left(\left(2n+1\right)\pi
i\right)^{m-1}}$
$\displaystyle+\left(u+1\right)\left[\frac{1}{\left(2n+1\right)\pi
i}H_{m}\left(u\right)+\frac{m}{\left(\left(2n+1\right)\pi
i\right)^{2}}H_{m-1}\left(u\right)+...+\frac{m!}{4!\left(\left(2n+1\right)\pi
i\right)^{m-3}}\right]\text{.}$

We want to note that

$\lim_{u\rightarrow-1}a_{n}^{\left(m\right)}\left(u\right)=a_{n}^{\left(m\right)}\left(-1\right):=a_{n}^{\left(m\right)}$

where $a_{n}^{\left(m\right)}\in\mathbb{C}$ is defined by Kim in [3] as
follows:

$a_{n}^{\left(m\right)}=\frac{m!}{\left(\left(2n+1\right)\pi
i\right)^{m-1}}a_{n}^{\left(1\right)}\left(u\right)\text{.}$

By using (2.1) and (2.3a), we readily derive the following

$H_{m}\left(x,u\right)=\sum_{n=-\infty}^{\infty}\left\\{\begin{array}[]{c}\left[a_{n}^{\left(1\right)}\left(u\right)+\left(u+1\right)\frac{H_{2}\left(u\right)}{2}\right]\frac{m!}{\left(\left(2n+1\right)\pi
i\right)^{m-1}}+\left(u+1\right)\\\ \times\left[\frac{1}{\left(2n+1\right)\pi
i}H_{m}\left(u\right)+\frac{m}{\left(\left(2n+1\right)\pi
i\right)^{2}}H_{m-1}\left(u\right)+...+\frac{m!}{4!\left(\left(2n+1\right)\pi
i\right)^{m-3}}\right]\end{array}\right\\}e^{\left(2n+1\right)\pi ix}\text{.}$

From this, we can state the following

$\displaystyle
H_{m}\left(x,u\right)=\sum_{n=-\infty}^{\infty}\left[a_{n}^{\left(1\right)}\left(u\right)+\left(u+1\right)\frac{H_{2}\left(u\right)}{2}\right]\frac{m!e^{\left(2n+1\right)\pi
ix}}{\left(\left(2n+1\right)\pi i\right)^{m-1}}$
$\displaystyle+\left(u+1\right)\sum_{n=-\infty}^{\infty}\left[\frac{1}{\left(2n+1\right)\pi
i}H_{m}\left(u\right)+\frac{m}{\left(\left(2n+1\right)\pi
i\right)^{2}}H_{m-1}\left(u\right)+...+\frac{m!}{4!\left(\left(2n+1\right)\pi
i\right)^{m-3}}H_{4}\left(u\right)\right]e^{\left(2n+1\right)\pi ix}$

After some calculations on the above equation, we have the following

$\displaystyle H_{m}\left(x,u\right)$ $\displaystyle=$
$\displaystyle\sum_{n=-\infty}^{\infty}\left[\frac{u+1}{u-1}\frac{1}{\left(2n+1\right)\pi
i}+\frac{2}{\left(\left(2n+1\right)\pi
i\right)^{2}}+\frac{1}{2}\left(\frac{u+1}{1-u}\right)^{2}\right]\frac{m!e^{\left(2n+1\right)\pi
ix}}{\left(\left(2n+1\right)\pi i\right)^{m-1}}$
$\displaystyle+\left(u+1\right)\sum_{n=-\infty}^{\infty}\left[\sum_{k=0}^{m-4}\frac{1}{\left(\left(2n+1\right)\pi
i\right)^{k+1}}\frac{d^{k}}{dx^{k}}H_{m}\left(x,u\right)\mid_{x=0}\right]e^{\left(2n+1\right)\pi
ix}$

As a result, we conclude the following theorem.

######  Theorem 1.

For $m\in\mathbb{N}$ and $0\leq x<1$, we have

$\displaystyle
H_{m}\left(x,u\right)=m!\sum_{n=-\infty}^{\infty}\left[\frac{u+1}{u-1}\frac{1}{\left(2n+1\right)\pi
i}+\frac{2}{\left(\left(2n+1\right)\pi
i\right)^{2}}+\frac{1}{2}\left(\frac{u+1}{1-u}\right)^{2}\right]\frac{e^{\left(2n+1\right)\pi
ix}}{\left(\left(2n+1\right)\pi i\right)^{m-1}}$
$\displaystyle+\left(u+1\right)\sum_{k=0}^{m-4}\frac{1}{\left(2\pi
i\right)^{k+1}}\frac{d^{k}}{dx^{k}}H_{m}\left(x,u\right)\mid_{x=0}\sum_{n=-\infty}^{\infty}\frac{e^{\left(2n+1\right)\pi
ix}}{\left(n+\frac{1}{2}\right)^{k+1}}\text{.}$

Considering generating functions of Euler and Frobenius-Euler polynomials, we
reach the following corollary.

######  Corollary 1.

Taking $u=-1$, we have Fourier transformation of Euler function, which is
defined by Kim in [3] as follows:

$H_{m}\left(x,-1\right)=E_{m}\left(x\right)=2m!\sum_{n=-\infty}^{\infty}\frac{e^{\left(2n+1\right)\pi
ix}}{\left(\left(2n+1\right)\pi i\right)^{m+1}}\text{ }\left(\text{for }0\leq
x<1\right)\text{.}$

The Lerch trancendent $\Phi\left(z,s,a\right)$ is the analytic continuation of
the series

(2.5)
$\Phi\left(z,s,a\right)=\sum_{n=0}^{\infty}\frac{z^{n}}{\left(n+a\right)^{s}}$

which converges for $a\in\mathbb{C}\backslash\mathbb{Z}_{0}^{-}$,
$s\in\mathbb{C}$ when $\left|z\right|<1$; $\Re\left(s\right)>1$ when
$\left|z\right|=1$ where
$\mathbb{Z}_{0}^{-}=\mathbb{Z}^{-}\cup\left\\{0\right\\}$,
$\mathbb{Z}^{-}=\left\\{-1,-2,-3,...\right\\}$. Lerch trancendent
$\Phi\left(z,s,a\right)$ is the proportional not only Riemann zeta funtion,
Hurwitz zeta function, the Dirichlet’s eta function but also Dirichlet beta
function, the Legendre chi function, the polylogarithm, the Lerch zeta
function (for details, see [15], [17]).

We now want to indicate that
$\sum_{n=-\infty}^{\infty}\frac{e^{\left(2n+1\right)\pi
ix}}{\left(\left(2n+1\right)\pi i\right)^{m}}$ is closely related to Lerch
trancendent $\Phi\left(z,s,a\right)$. So, we compute as follows:

$\displaystyle\sum_{n=-\infty}^{\infty}\frac{e^{\left(2n+1\right)\pi
ix}}{\left(\left(2n+1\right)\pi i\right)^{m}}$ $\displaystyle=$
$\displaystyle\frac{1}{\left(2\pi
i\right)^{m}}\sum_{n=-\infty}^{\infty}\frac{e^{\left(2n+1\right)\pi
ix}}{\left(n+\frac{1}{2}\right)^{m}}$ $\displaystyle=$
$\displaystyle\frac{1}{\left(2\pi
i\right)^{m}}\sum_{n=-\infty}^{-1}\frac{e^{\left(2n+1\right)\pi
ix}}{\left(n+\frac{1}{2}\right)^{m}}+\frac{1}{\left(2\pi
i\right)^{m}}\sum_{n=0}^{\infty}\frac{e^{\left(2n+1\right)\pi
ix}}{\left(n+\frac{1}{2}\right)^{m}}$

After some applications on the above equation, we procure the following

(2.6) $\sum_{n=-\infty}^{\infty}\frac{e^{\left(2n+1\right)\pi
ix}}{\left(\left(2n+1\right)\pi i\right)^{m}}=-\frac{e^{\pi ix}}{\left(\pi
i\right)^{m}}+\frac{\left(-1\right)^{m}e^{\pi ix}}{\left(2\pi
i\right)^{m}}\Phi\left(e^{-2\pi ix},m,-\frac{1}{2}\right)\text{.}$

By using Theorem 1 and (2.6), we give the following theorem.

######  Theorem 2.

The following equality holds true:

$\displaystyle H_{m}\left(x,u\right)=m!\frac{u+1}{u-1}\left(-\frac{e^{\pi
ix}}{\left(\pi i\right)^{m}}+\frac{\left(-1\right)^{m}e^{\pi ix}}{\left(2\pi
i\right)^{m}}+\Phi\left(e^{-2\pi ix},m,-\frac{1}{2}\right)\right)$
$\displaystyle+2m!\left(-\frac{e^{\pi ix}}{\left(\pi
i\right)^{m+1}}+\frac{\left(-1\right)^{m+1}e^{\pi ix}}{\left(2\pi
i\right)^{m+1}}+\Phi\left(e^{-2\pi ix},m+1,-\frac{1}{2}\right)\right)$
$\displaystyle+\frac{1}{2}\left(\frac{u+1}{u-1}\right)^{2}\left(-\frac{e^{\pi
ix}}{\left(\pi i\right)^{m-1}}+\frac{\left(-1\right)^{m-1}e^{\pi
ix}}{\left(2\pi i\right)^{m-1}}+\Phi\left(e^{-2\pi
ix},m-1,-\frac{1}{2}\right)\right)$
$\displaystyle+\left(u+1\right)\sum_{k=0}^{m-4}\frac{d^{k}}{dx^{k}}H_{m}\left(x,u\right)\mid_{x=0}\left(-\frac{e^{\pi
ix}}{\left(\pi i\right)^{k+1}}+\frac{\left(-1\right)^{k+1}e^{\pi
ix}}{\left(2\pi i\right)^{k+1}}+\Phi\left(e^{-2\pi
ix},k+1,-\frac{1}{2}\right)\right)\text{.}$

For $u=-1$ on the above theorem, we have the following corollary.

######  Corollary 2.

The following identity

$E_{m}\left(x\right)=2m!\left(-\frac{e^{\pi ix}}{\left(\pi
i\right)^{m+1}}+\frac{\left(-1\right)^{m+1}e^{\pi ix}}{\left(2\pi
i\right)^{m+1}}+\Phi\left(e^{-2\pi ix},m+1,-\frac{1}{2}\right)\right)$

is true.

Setting $x=1$ in Theorem 1, we obtain

(2.7) $\displaystyle
H_{m}\left(1,u\right)=-m!\sum_{n=-\infty}^{\infty}\left[\frac{u+1}{u-1}\frac{1}{\left(2n+1\right)\pi
i}+\frac{2}{\left(\left(2n+1\right)\pi
i\right)^{2}}+\frac{1}{2}\left(\frac{u+1}{1-u}\right)^{2}\right]\frac{1}{\left(\left(2n+1\right)\pi
i\right)^{m-1}}$
$\displaystyle-\left(u+1\right)\sum_{k=0}^{m-4}\frac{1}{\left(2\pi
i\right)^{k+1}}\frac{d^{k}}{dx^{k}}H_{m}\left(x,u\right)\mid_{x=0}\sum_{n=-\infty}^{\infty}\frac{1}{\left(n+\frac{1}{2}\right)^{k+1}}\text{.}$

By expressions of (2.7) and Lemma 1, we easily see the following corollary.

######  Corollary 3.

The following identity holds true:

$\displaystyle
uH_{m}\left(u\right)=-m!\sum_{n=-\infty}^{\infty}\left[\frac{u+1}{u-1}\frac{1}{\left(2n+1\right)\pi
i}+\frac{2}{\left(\left(2n+1\right)\pi
i\right)^{2}}+\frac{1}{2}\left(\frac{u+1}{1-u}\right)^{2}\right]\frac{1}{\left(\left(2n+1\right)\pi
i\right)^{m-1}}$
$\displaystyle-\left(u+1\right)\sum_{k=0}^{m-4}\frac{1}{\left(2\pi
i\right)^{k+1}}\frac{d^{k}}{dx^{k}}H_{m}\left(x,u\right)\mid_{x=0}\sum_{n=-\infty}^{\infty}\frac{1}{\left(n+\frac{1}{2}\right)^{k+1}}\text{.}$

Now, by using Kim’s method in [3], we discover the following

(2.8a)
$\displaystyle\frac{1}{1-ue^{-t}}=\sum_{n=0}^{\infty}u^{n}e^{-nt}=\sum_{n=0}^{\infty}u^{n}\left(e^{-t}\right)^{n}=\sum_{n=0}^{\infty}u^{n}\left(\sum_{k=0}^{\infty}\left(-1\right)^{k}\frac{t^{k}}{k!}\right)^{n}$
$\displaystyle=\sum_{n=0}^{\infty}\left(\sum_{a_{1}+a_{2}+...+a_{n}=n}\frac{n!}{\left(a_{1}\right)!\left(a_{2}\right)!...}\frac{\left(-1\right)^{a_{1}+2a_{2}+...}}{\left(1!\right)^{a_{1}}\left(2!\right)^{a_{2}}...}\right)t^{a_{1}+2a_{2}+...}$

Let $p\left(i,j\right):a_{1}+2a_{2}+...=i,a_{1}+a_{2}+...=j$, from expression
of (2.8a), we compute as follows:

$\displaystyle\frac{1}{1-ue^{-t}}=\sum_{m=0}^{\infty}\left(\sum_{n=0}^{m}u^{n}\sum_{p\left(m,n\right)}\frac{n!}{\left(a_{1}\right)!\left(a_{2}\right)!...\left(a_{m}\right)!}\frac{\left(-1\right)^{a_{1}+2a_{2}+...+ma_{m}}}{\left(1!\right)^{a_{1}}\left(2!\right)^{a_{2}}...\left(m!\right)^{a_{m}}}\right)t^{a_{1}+2a_{2}+...+ma_{m}}$
$\displaystyle=\sum_{m=0}^{\infty}\left(-1\right)^{m}\left(\sum_{n=0}^{m}n!u^{n}\sum_{p\left(m,n\right)}\frac{m!}{\left(a_{1}\right)!\left(a_{2}\right)!...\left(a_{m}\right)!}\frac{\left(-1\right)^{m}}{\left(1!\right)^{a_{1}}\left(2!\right)^{a_{2}}...\left(m!\right)^{a_{m}}}\right)\frac{t^{m}}{m!}$
(2.9a)
$\displaystyle=\sum_{m=0}^{\infty}\left[\left(-1\right)^{m}\sum_{n=0}^{m}n!u^{n}s_{2}\left(m,n\right)\right]\frac{t^{m}}{m!}\text{.}$
where $s_{2}\left(m,n\right)$ is the second kind stirling number.

Via the definition of Frobenius-Euler numbers, we readily derive the following

$\displaystyle\frac{1}{1-ue^{-t}}$ $\displaystyle=$
$\displaystyle\frac{u^{-1}}{u^{-1}-1}\frac{1-u^{-1}}{e^{-t}-u^{-1}}$
$\displaystyle=$
$\displaystyle\frac{1}{1-u}\sum_{m=0}^{\infty}\left(-1\right)^{m}H_{m}\left(u^{-1}\right)\frac{t^{m}}{m!}\text{.}$

By comparing the coefficients of $\frac{t^{n}}{n!}$ on the both sides of
(2.9a) and (2), we reach the following theorem.

######  Theorem 3.

The following equality holds true:

$\frac{1}{1-u}H_{m}\left(u^{-1}\right)=\sum_{n=0}^{m}n!u^{n}s_{2}\left(m,n\right)\text{.}$

######  Corollary 4.

Substituting $u=-1$ in the above theorem, we get the following, which is
defined by Kim [3]

$E_{m}=2\sum_{n=0}^{m}n!\left(-1\right)^{n}s_{2}\left(m,n\right)\text{.}$

## References

  * [1] A. Bayad, Fourier expansions for Apostol-Bernoulli, Apostol-Euler and Apostol Genocchi polynomials, Mathematics of Computation, Volume 80, Number 276, October 2011, Pages 2219–2221.
  * [2] J. Choi, D. S. Kim, T. Kim and Y. H. Kim, A note on Some identities of Frobeniu-Euler Numbers and Polynomials, International Journal of Mathematics and Mathematical Sciences, Volume 2012, Article ID 861797, 9 pages.
  * [3] T. Kim, Note on the Euler numbers and polynomials, Advanced Studies in Contemporary Mathematics, vol. 17, no. 2, pp. 131–136, 2008.
  * [4] T. Kim and B. Lee, Some Identities of the Frobenius-Euler polynomials, Abstract and Applied Analysis, Volume 2009, Article ID 639439, 7 pages.
  * [5] T. Kim, Euler Numbers and Polynomials Associated with Zeta Functions, Abstract and Applied Analysis 2008 (2008), Article ID 581582, 11 pages.
  * [6] T. Kim, $q$-Volkenborn integration, Russ. J. Math. Phys. 9 (2002), 288-299.
  * [7] T. Kim, $q$-Bernoulli numbers and polynomials associated with Gaussian binomial coefficients, Russ. J. Math. Phys. 15 $\left(\text{2008}\right),$ 51-57.
  * [8] T. Kim, $q$-extension of the Euler formula and trigonometric functions, Russian J. Math. Phys. 14 (2007), 275-278.
  * [9] T. Kim, On Euler-Barnes multiple zeta functions, Russian J. Math. Phys. 10 (2003), 261-267.
  * [10] T. Kim, $q$-Euler numbers and polynomials associated with $p$-adic $q$-integrals, J. Nonlinear Math. Phys., 14 (2007), no. 1, 15–27.
  * [11] T. Kim, New approach to $q$-Euler polynomials of higher order, Russ. J. Math. Phys., 17 (2010), no. 2, 218–225.
  * [12] T. Kim, Some identities on the $q$-Euler polynomials of higher order and $q$-Stirling numbers by the fermionic $p$-adic integral on $\mathbb{Z}_{p}$, Russ. J. Math. Phys., 16 (2009), no.4, 484–491.
  * [13] T. Kim, Identities involving Frobenius-Euler polynomials arising from non-linear differential equatiaons, arXiv:1201.5088v1 [math.NT].
  * [14] Y. Simsek, Y. Osman, V. Kurt, On interpolation functions of the twisted generalized Frobenius-Euler numbers, Adv. Stud. Contemp. Math. 15 (2007), 187-194.
  * [15] H. M. Srivastava and J. Choi, Series Associated with the Zeta and Related Functions, Kluwer Academic Publishers, Dordrecht, The Netherlands, 2001.
  * [16] Q-M. Luo, Fourier expansions and integral representations for the Apostol-Bernoulli and Apostol-Euler polynomials, Math. Comp., Volume 78 (2009), No. 268, 2193–2208.
  * [17] M. Acikgoz and Y. Simsek, On multiple interpolation functions of the Nörlund-Type $q$-Euler polynomials, Abstract and Applied Analysis, Vol. 2009, Article ID 382574, 14 pages.
  * [18] S. Araci, M. Acikgoz, K. H. Park and H. Jolany, On the unification of two families of multiple twisted type polynomials by using $p$-adic $q$-integral on $\mathbb{Z}_{p}$ at $q=-1$, to appear in Bulletin of the Malaysian Mathematical Sciences and Society.
  * [19] S. Araci, D. Erdal and J. J. Seo, A study on the fermionic $p$-adic $q$-integral representation on $\mathbb{Z}_{p}$ associated with weighted $q$-Bernstein and $q$-Genocchi polynomials, Abstract and Applied Analysis, Volume 2011, Article ID 649248, 10 pages.
  * [20] S. Araci, M. Acikgoz and K. H. Park, A note on the $q$-analogue of Kim’s $p$-adic $\log$ gamma type functions associated with $q$-extension of Genocchi and Euler numbers with weight $\alpha$, Bulletin of the Korean Mathematical Society (accepted for publication).


[affiliation=1]AdhamIbrahim [affiliation=1,2]ShadyShehata
[affiliation=1,3]AjinkyaKulkarni [affiliation=1]MukhtarMohamed
[affiliation=1,2,4]MuhammadAbdul-Mageed

# What Does it Take to Generalize SER Model Across Datasets? A Comprehensive
Benchmark

###### Abstract

Speech emotion recognition (SER) is essential for enhancing human-computer
interaction in speech-based applications. Despite improvements in specific
emotional datasets, there is still a research gap in SER's capability to
generalize across real-world situations. In this paper, we investigate
approaches to generalize the SER system across different emotion datasets. In
particular, incorporate 11 emotional speech datasets and illustrate a
comprehensive benchmark on the SER task. We also address the challenge of
imbalanced data distribution using oversampling methods when combining SER
datasets for training. Furthermore, we explore various evaluation protocols
for adeptness in the generalization of SER. Building on this, we explore the
potential of Whisper for SER, emphasizing the importance of thorough
evaluation. Our approach is designed to advance SER technology by integrating
speaker-independent methods.

###### keywords:

speech emotion recognition, human-computer interaction, computational
paralinguistics

##  1 Introduction

Emotions are intrinsic characteristics of human communications expressed
through speech modality and plays crucial role in human-machine interfaces
[1]. Over the past decades, research has been focused on various feature
extraction methods along with machine learning. This includes traditional
approaches like Mel-frequency cepstral coefficients (MFCCs)[2], linear
prediction cepstral coefficients (LPCCs)[3], and prosodic features[4]. Deep
learning techniques such as deep neural network DNN[5], recurrent neural
network (RNN)[6], and convolutional neural networks (CNN)[7] have also gained
prominence. The last three years have seen a considerable exploration of
HuBERT[8] and Wav2Vec2[9] models in SER [10, 9], demonstrating their potential
for robust emotion recognition.

IEMOCAP [11] is considered the benchmark dataset for discrete speech emotion
recognition. An extensive evaluation of SER systems has been conducted using
IEMOCAP, employing various machine learning methods such as support vector
machines (SVM), LSTM, CNN, and ensemble learning [12, 13]. However, the use of
Self-supervised learning (SSL) speech models (e.g., Hubert [8] and Wav2Vec2
[9]) has emerged as the state-of-the-art approach in SER, producing the best
performance on the IEMOCAP dataset [9]. Another very important acted dataset
is (RAVDESS) [14]. RAVDESS was also involved in benchmarking many different
SER systems [15, 16]. A dataset closely similar to RAVDESS is CREMA-D [17],
which comprises 12 distinct spoken sentences performed by 91 different actors,
resulting in a total of 7442 unique audio files.

Among the more compact SER databases available, the Surrey Audio-Visual
Expressed Emotion (SAVEE) [18] and the Toronto Emotional Speech Set (TESS)
[19] stand out. Both datasets have only been benchmarked using traditional
machine learning techniques (SVM, LSTM, CNN) [20, 21, 22]. To the best of our
knowledge, these datasets have yet to be tested with newer transformer-based
models. MELD dataset has a unique conversational setup of 13,000 utterances
from 1,433 dialogues from the TV series Friends. This compilation aims to
address the complexities of emotion recognition within conversational
contexts, a task known for its challenges. Other SER datasets include ASVP-
ESD[23], EmoV-DB[24], EmoFilm[25], JL-Corpus[26], and ESD[27].
Despite the availability of various datasets for speech emotion recognition,
there's a consensus that SER still trails behind other tasks [28, 29, 30].
This shortfall is attributed to the absence of a large, universal dataset
capable of bridging the significant disparities among existing datasets and
facilitating adaptation to real-world scenarios. A recent study [16]
investigated the generalization capabilities and the possibility of merging
different SER datasets by applying a set of cross-validation experiments,
considering both single datasets and combinations of them, on RAVDESS, TESS,
CREMA-D, and IEMOCAP. Their results confirm that SER models do not generalize
well across datasets (training and testing on different datasets) and that
merging datasets can mitigate this problem, as SER performance improves with
access to a larger and more varied collection of data points.
A key challenge in speech emotion recognition stems from the substantial
variance among available datasets, attributed to differences in setup (acted,
natural, and elicited), recording quality, and subjective emotion perception
by speakers and annotators. Consequently, most systems struggle to generalize
across datasets, hindering performance in real-world applications [16].
Moreover, emotional states influence spoken sentences, compounded by personal
attributes like age and gender, posing obstacles for SER generalization and
hindering the development of speech-independent systems, particularly in tasks
like speech emotion recognition (SER). This challenge is underscored by human
behavior, as demonstrated in an experiment by Schuller et al. [31] where the
accuracy of 12 participants in recognizing expressed emotions decreased from
87.3% with familiar speakers to 64.7% when identifying emotions from speakers
they had not previously encountered.
Due to variations in emotion classes, emotion labeling, amount of emotional
data per dataset, SER more arduous task to generalize. In this paper, we
present novel comprehensive benchmarking results across 11 SER datasets and
various experimentations showcasing the vital aspects of generalization and
performance improvement of SER.

##  2 Methodology

Figure 1: Whisper based speech emotion recognition system

In this section, we highlight our primary contributions: Our SER system based
on Whisper, the combination of different datasets, and evaluation strategy.

###  2.1 Model Architecture

We illustrate the SER model architecture, which we use throughout this work
for various experiments across various SER datasets. This architecture aligns
with the current state-of-the-art (SoTA) and is representative of the current
best practices in speech emotion recognition [10].
Our SER first uses Whisper-based feature extraction [32] to map speech into
latent representation through a transformer-based encoder-decoder network as
shown in Fig 1. We use the base version of Whisper utilizing both the encoder
and decoder to extract fixed-size embedding and fine-tune the whole model with
the rest of the network. Thereafter, this latent representation as features is
given to feed-forward of fully connected layers: starting from a size of 4096,
it sequentially narrows down through layers of 2048, 1024, and 512, with each
layer followed by a ReLU activation function to add non-linearity. This setup
culminates in an output layer tailored to the number of target classes, making
it suitable for classification purposes. To our knowledge, Whisper-based SER
has not been explored for ASR tasks. We implement a basic five-layer feed-
forward neural network, where the last layer projects into the different
emotion labels. Cross-entropy serves as our loss function, and we maintain a
constant learning rate of 0.00001. We train all variants of SER on NVIDIA
A100-SXM GPUs equipped with 40GB of memory.

###  2.2 Datasets Preparation

Emotional databases in speech emotion recognition are classified into acted,
elicited, and natural categories, each with unique characteristics and
challenges. Acted databases consist of recordings from trained actors
simulating specific emotions, offering control but relying on the actors'
skills[33]. Elicited databases capture genuine emotional responses under
designed scenarios requiring ethical consent, while natural databases compile
authentic interactions from everyday life, presenting a wide authenticity
spectrum but limited emotional range[34].

Our study conducts a detailed analysis of individual and combined datasets to
understand the impact of similar emotions on model accuracy and generalization
on 11 datasets in the SER domain: IEMOCAP[11]: Approximately 12 hours from 10
speakers, with scripted and improvised dialogues across seven emotion classes
(anger, happiness, sadness, excited, frustrated, and neutral). MELD[35]:
13,708 utterances over 12 hours from the Friends TV series, categorized into
seven emotions (anger, disgust, sadness, joy, neutral, surprise, fear) with 10
main speakers and around 290 secondary speakers. ASVP-ESD[23]: A multi-lingual
corpus from movies, YouTube, and real interactions, focusing on English speech
with seven emotions, totaling 6.5 hours from main and secondary speakers.
EmoV-DB[24]: Covers five emotions with five speakers (four in English, one in
French), totaling 9.5 hours. TESS[19]: Features two actresses, covering seven
emotion classes over 2800 files in 1.6 hours. EmoFilm[25]: Samples in English,
Italian, and Spanish from 43 movies, focusing on five emotions over 20
minutes. SAVEE[18]: British-English corpus with 480 utterances from four
speakers, spanning seven emotions in 30 minutes. RAVDESS[14]: Focuses on the
speech part with 1440 utterances from 24 actors, covering eight emotions over
1.5 hours. CREMA-D[17]: Audio-visual dataset with 7442 stimuli from 91 actors,
spanning six emotions over 5.3 hours. JL_corpus[26]: Contains 2400 sentences
from four speakers, covering 10 emotions over 1.4 hours. ESD[27]: Offers 350
parallel utterances in five emotion categories from 10 native English
speakers, with over 29 hours recorded.

Table 1: Distribution of emotion samples across 11 SER datasets | Dataset Name |
---|---|---
Emotion | ASVP-ESD | CREMA-D | EmoF | ESD | IEMOCAP | JL | MELD | TESS | emov_db | savee | ravdess | Total
angry | 585 | 1271 | 77 | 3500 | 1103 | 240 | 1607 | 400 | 1268 | 60 | 192 | 10303
disgust | 612 | 1271 | - | - | - | - | 301 | 400 | 522 | 45 | 184 | 3335
anxious | - | - | - | - | - | 240 | - | - | - | - | - | 240
apologetic | - | - | - | - | - | 240 | - | - | - | - | - | 240
assertive | - | - | - | - | - | 240 | - | - | - | - | - | 240
concerned | - | - | - | - | - | 240 | - | - | - | - | - | 240
encouraging | - | - | - | - | - | 240 | - | - | - | - | - | 240
excited | - | - | - | - | 1041 | 240 | - | - | - | - | - | 1281
frustrated | - | - | - | - | 1849 | - | - | - | - | - | - | 1849
fear | 544 | 1271 | 72 | - | - | - | 358 | 400 | - | 60 | 192 | 2897
happy | 652 | 1271 | 70 | 3500 | 595 | 240 | 2307 | 400 | - | 60 | 192 | 9287
neutral | 714 | 1087 | - | 3500 | 1708 | 240 | 6432 | 400 | 1568 | 120 | 96 | 15865
pain | 558 | - | - | - | - | - | - | - | - | - | - | 558
sad | - | 1271 | 74 | 3500 | 1084 | 240 | 1002 | 400 | - | 60 | 192 | 7823
surprised | 695 | - | - | 3500 | 107 | - | 1403 | 400 | - | 60 | 192 | 6357
contempt | - | - | 50 | - | - | - | - | - | - | - | - | 50
amused | - | - | - | - | - | - | - | - | 1317 | - | - | 1317
sleepy | - | - | - | - | - | - | - | - | 1721 | - | - | 1721
calm | - | - | - | - | - | - | - | - | - | - | 192 | 192

In our investigation, we specifically extracted English speech and
corresponding labels from mixed-language datasets like ESD[27], ASVP-ESD[23],
and EmoFilm[25]. It's essential to highlight that the other datasets employed
in our study are exclusively in English, ensuring a consistent linguistic
foundation for our analysis. Regarding the IEMOCAP dataset, our dataset
preparation followed the guidelines outlined in a previous study[36], where
emotions were categorized into anger, happiness, sadness, and neutral, with
excitement categorized under happiness. We retained the original emotion
classes for the remaining datasets. Additionally, we downsampled speech across
all speech utterances to 16 kHz. Speaker identification for each audio clip in
every dataset was accomplished using the information provided in the dataset
metadata.

###  2.3 Combining datasets for the SER task

A major challenge in speech emotion recognition lies in the considerable
variance among the available datasets. This variance stems from differences in
their setup (acted, natural, and elicit), the quality of recordings, and the
subjective perception of emotions by different speakers and annotators.
Consequently, most speech emotion recognition systems do not generalize well
across different datasets (training and testing on different datasets) [16]
and, by extension, struggle to perform well in real-world applications.

In an effort to address this challenge, [16] showed promising results by
combining four of the most famous SER datasets (RAVDESS[14], TESS[19],
CREMA-D[17], and IEMOCAP[11]). In this work, in addition to these four
datasets, we added seven more (MELD[35], ASVP-ESD[23], EmoV-DB[24],
EmoFilm[25], SAVEE[18], JL-Corpus[26], and ESD[27]) for a total of eleven SER
datasets. To the best of our knowledge, combining 11 datasets presents the
largest training dataset for the SER task. This study aims to significantly
boost performance (against individual datasets) on the SER task, attributable
to the considerably larger and more diverse array of training data points.

###  2.4 Evaluation protocol

Throughout this work, we opted for the leave-one-speaker-out (LOSO) method to
evaluate our models. We used accuracy as a way to measure the performance of
SER systems. When dealing with datasets where all speakers contributed the
same amount of speech, we randomly selected one speaker from each dataset as
our test case. Conversely, for datasets where speakers had varying amounts of
speech time, such as MELD, ASVP-ESD, and IEMOCAP, we excluded the speaker with
the most speech duration. At the end of this process, we ensured that a
speaker from every dataset was chosen, guaranteeing a wide-ranging and
inclusive evaluation.

##  3 Experimental Study

In our work, we consider three different sets of experiments for training and
evaluation: 1. We establish an initial benchmark by training each dataset
individually using all their original emotion classes. 2. We use only four
emotions (neutral, angry, happy, and sad) to train and test our SER models. We
do this for both individual and combined settings. 3. We use five emotions
adding surprise to the previous four to train and test our SER models. We do
this for both individual and combined settings.

We used a consistent training setup involving 5-fold cross-validation. This
means we divided the data so that each fold used a different speaker for
validation and the rest for training. For datasets with less than five
speakers, we adjusted the number of folds to match the number of speakers.
Additionally, with the TESS dataset, which has only two speakers, we used one
for testing and the other for training, applying 5-fold cross-validation on
the training speaker's data. In the case of the EmoFilm dataset, we used 85%
of the data for training, still following the 5-fold cross-validation method.
For combined datasets, however, we adapted 5-fold cross-validation based on
percentage splits across the dataset to ensure even distribution. It is
important to note that we leave one speaker from each dataset as a test set
right from the start, calculating a weighted average on the speaker-out test
set across the 5-folds to ensure uniform and comprehensive evaluation metrics
across all experiments.

###  3.1 Individual and combined 4-emotion dataset evaluation

For every dataset, we train a Whisper-based model using the four primary
emotions: neutral, angry, happy, and sad. If a dataset lacks one of these
emotions, we include from the dataset whichever available subset from these
four emotions. Table 1 shows that we considered the most common 4 emotions
across all datasets. For the combined dataset, we combined 11 SER datasets
into a single comprehensive dataset based on the same 4 emotions. We included
audio clips with duration ranging from 2 to 13 seconds, to ensure consistency
and comparability across the combined dataset. Speech clips within this
duration range are the most common duration of data across all datasets. After
combining emotions from various datasets, the data distribution exhibited
imbalance, as illustrated in Table 2. This imbalance in the distribution of
emotional categories across the combined dataset could potentially influence
both the training and evaluation phases of our model. Explored four techniques
to determine if any could improve model generalization: no sampling, randomly
downsampling to the lowest category, and oversampling performed using the
Synthetic Minority Over-sampling Technique (SMOTE) [37] and the Adaptive
Synthetic Sampling Method for Imbalanced Data (ADASYN)[38]. We applied the
SMOTE and ADASYN algorithms to the audio data. The oversampling process was
conducted to balance the distribution of emotion labels, ensuring that low-
frequency emotions were over-sampled to match the frequency of the highest
emotion category.

Table 2: Comparison of emotion distribution in combined 11 SER datasets with
different sampling techniques.

| 4-Emotions | 5-Emotions
---|---|---
Emotion/Method | Original | Undersamp | SMOTE | ADASYN | Original | Undersamp | SMOTE | ADASYN
Neutral (neu) | 10044 | 6157 | 10044 | 10044 | 10044 | 4056 | 10044 | 10044
Angry (ang) | 7427 | 6157 | 10044 | 10044 | 7427 | 4056 | 10044 | 10044
Happy (hap) | 6286 | 6157 | 10044 | 10044 | 6286 | 4056 | 10044 | 10044
Sad (sad) | 6157 | 6157 | 10044 | 10044 | 6157 | 4056 | 10044 | 10044
Surprised (sur) | - | - | - | - | 4056 | 4056 | 10044 | 10044

###  3.2 Individual and combined 5-emotions dataset evaluation

This step was taken to ascertain the model's robustness in handling an
expanded set of emotion labels. we trained a Whisper-based model on a set of 5
emotions neutral, angry, happy, surprised, and sad. We train the model on each
dataset using the same settings as in the individual training described in the
previous experiment in Section 3.1. We repeat training and test settings in
combined dataset training in the previous experiment, but with 5 emotions
neutral, angry, happy, surprised, and sad.

Figure 2: t-SNE visualization of Whisper model's embeddings showcasing clusters of emotional speech from various datasets. Table 3: Experiment Results for 4-Emotions, 5-Emotions, and N-Emotions, detailing performance across training criteria and data sampling techniques; Tr Co refers to training combination of datasets, Tr. sep. refers to training datasets separate, WA as weighted average, DS as downsampling, UN as original samples, SM refers to SMOTE sampling and AD as ADASYN oversampling Dataset Name |  4-Emotions WA Test (%) |  5-Emotions WA Test (%) |  N-Emotions WA Test (%)
---|---|---|---
Emo. No. | Tr. Co. | Tr. Sep. | Emo. No. | Tr. Co. | Tr. Sep. | Emo. No. | Tr. Sep.
DS | UN | SM | AD | DS | UN | SM | AD
ESD | 4 | 79.11 | 77.5 | 72.73 | 74.45 | 76.85 | 5 | 78.81 | 74.45 | 74.72 | 74.72 | 72.04 | 5 | 72.04
MELD | 4 | 51.20 | 54.72 | 56.11 | 56.65 | 63 | 5 | 47.66 | 54.72 | 47.68 | 47.68 | 56.87 | 7 | 50
IEMOCAP | 4 | 67.75 | 70.00 | 68.79 | 68.96 | 72.43 | 5 | 66.37 | 70.00 | 67.42 | 67.42 | 69.5 | 7 | 56.1
CREMA-D | 4 | 93.50 | 98.00 | 95 | 93.5 | 90.41 | 4 | 91 | 98.00 | 95.50 | 95.50 | 90.41 | 6 | 80
EMOV_DB | 2 | 99.45 | 99.64 | 99.7 | 99.68 | 94.4 | 2 | 99.6 | 99.64 | 99.66 | 99.66 | 94.4 | 5 | 80.9
ASVP-ESD | 3 | 79.11 | 77.75 | 81.25 | 78.75 | 84.65 | 4 | 73.37 | 77.75 | 76.30 | 76.30 | 82.35 | 7 | 73.33
TESS | 4 | 77.35 | 71.08 | 74.9 | 74.07 | 66.95 | 5 | 62.5 | 71.08 | 55.83 | 55.83 | 56.62 | 7 | 55.52
JL_corpus | 4 | 67.50 | 56.45 | 68.95 | 63.42 | 53.47 | 4 | 53.42 | 56.45 | 64.34 | 64.34 | 53.47 | 10 | 32.25
RAVDESS | 4 | 85.00 | 87.50 | 87.50 | 81.67 | 80.83 | 5 | 87.5 | 87.50 | 88.13 | 88.13 | 80.5 | 8 | 74.69
SAVEE | 4 | 71.67 | 72.50 | 77.78 | 77.78 | 68.05 | 5 | 67.5 | 72.50 | 70.45 | 69.77 | 73.11 | 7 | 67.71
EmoFilm | 3 | 67.50 | 64.17 | 62.5 | 58.53 | 82.2 | 3 | 68.33 | 64.17 | 57.5 | 60.83 | 82.2 | 5 | 53.37
Mean | - | 76 | 78.64 | 77.24 | 75.22 | 75.84 | - | 72.36 | 75.11 | 72.56 | 73.38 | 73.74 | - | 63.20

##  4 Results and Discussion

In this section, we provide a comprehensive overview of speech emotion
recognition performance across different datasets, emotion categories,
training criteria, and data sampling techniques, offering valuable insights
for the generalization of speech emotion recognition. Table 3. illustrates the
performance of the Whisper-based speech emotion recognition system developed
under various combinations of dataset, and oversampling techniques. In
general, the performance of SER varies across datasets and emotion categories.
For instance, datasets like emov_db consistently demonstrate high accuracy,
while others like MELD exhibit lower performance under all conditions.
Overall, using original data distribution or using SMOTE sampling tends to
yield better performance compared to downsampling and ADASYN sampling. For
instance, mean accuracy for training SER using a combination of 4 emotions
demonstrates that using original data distribution and SMOTE sampling provides
better results than other conditions. This suggests that maintaining the
original sample distribution or using synthetic samples for minority classes
can enhance model performance. Furthermore, training systems separately on
each dataset for 4 emotion and 5 emotion experiments under performed in
comparison to combining datasets. Thereafter, performance varies across
datasets and emotion categories. For instance, datasets like emov_db
consistently showcase high accuracy, while others like MELD display
comparatively lower performance. The choice of data sampling technique
significantly influences emotion recognition performance. Techniques like
SMOTE and ADASYN, which address the class imbalance, result in improved
accuracy compared to simple downsampling. This highlights the importance of
addressing class imbalance in training data to improve the robustness of
emotion recognition models.

Each dataset exhibits unique characteristics concerning the designing process
of the dataset such as emotion label annotation, recording conditions of
emotions, speaker demographics, and linguistic variations. etc, have an impact
on emotion recognition performance. Hence, the factors such as data quality,
diversity, and class distribution influence the effectiveness of trained
models. For example, datasets like CREMA-D and emov_db, which offer high-
quality and diverse speech samples, tend to achieve better performance across
different emotion categories. Therefore, it is vital to understand how
different training conditions affect model generalization is essential for
developing robust speech emotion recognition systems. In conducting our
research on 11 SER datasets with Whisper-based models, we encountered several
limitations.

Initially, the inherent diversity and imbalance in emotional representation
across datasets introduced challenges in standardizing the training process,
particularly when reducing the emotion categories to 4 and then to 5 specific
emotions. Additionally, the process of combining datasets to improve model
resilience encountered challenges in aligning emotional labels and
intensities, which could potentially influence the consistency of our
findings. The average accuracy findings for both the 4-emotion and 5-emotion
experiments consistently demonstrated enhanced performance when utilizing the
combined dataset, as opposed to separately training SER on each individual
dataset. The comparison between models trained on individual datasets versus a
combined dataset approach revealed variations in performance, suggesting that
not all datasets contribute equally to model accuracy and generalizability.
t-SNE in Figure 2 visualizes the feature embeddings extracted from the Whisper
model, representing various emotional states as discerned from multiple speech
datasets. It's noteworthy how the datasets SAVEE, ESD, and JL_corpus form
closely knit clusters, suggesting that their emotional representations in the
feature space are similar to each other. On the other hand, the MELD, IEMOCAP,
and CREMA-D datasets tend to group closer to one another, indicating a
different but consistent internal representation of emotions within these
datasets. The RAVDESS and TESS datasets stand out with their unique
positioning in the plot, reflecting distinct emotion representation patterns
extracted by the Whisper model. Moreover, for each emotion exclusive
representation on each dataset reflects the inherent variations in emotions
across datasets and its impact on generalization of SER.

##  5 Conclusion

This study proposes a novel approach to generalize SER model in leaving-one-
speaker-out settings. Employing a leave-one-speaker-out methodology
highlighted the model's robustness in real-world scenarios, facing speaker
variability. Extensive experiments were conducted by combining multiple SER
datasets to train a Whisper-based model, followed by testing the model on a
single speaker from each dataset, which yielded promising results, indicating
a successful generalization across diverse datasets. These findings suggest
that through careful dataset combination and targeted model training
strategies, we can overcome some of the prevalent challenges in speech emotion
recognition, paving the way for more universally applicable SER systems. In
conclusion, our work yielded a generalized SER model adept at identifying
emotions across a spectrum of datasets. This represents progress in the field,
enabling the development of advanced emotion recognition systems capable of
effectively handling diverse datasets. Future directions for emotion
recognition in low-resource languages involve leveraging transfer learning and
unsupervised learning approaches while considering cultural nuances and
linguistic expertise.

## References

  * [1] A. Kulkarni, ``Expressivity transfer in deep learning based text-to-speech synthesis. (synthèse vocale expressive basée sur un apprentissage profond),'' in _University of Lorraine, France_ , 2022. [Online]. Available: https://theses.fr/2022LORR0122
  * [2] P. Zhou, X. P. Li, J. Li, and X. X. Jing, ``Speech emotion recognition based on mixed MFCC,'' in _Applied Mechanics and Mechanical Engineering III_ , 2013.
  * [3] S. Lalitha, A. Mudupu, B. V. Nandyala, and R. Munagala, ``Speech emotion recognition using DWT,'' in _IEEE International Conference on Computational Intelligence and Computing Research (ICCIC)_ , 2015.
  * [4] K. S. Rao, S. G. Koolagudi, and R. R. Vempada, ``Emotion recognition from speech using global and local prosodic features,'' _International Journal of Speech Technology_ , 2013.
  * [5] K. Han, D. Yu, and I. Tashev, ``Speech emotion recognition using deep neural network and extreme learning machine,'' _Proc. Interspeech_ , 2014.
  * [6] J. Lee and I. Tashev, ``High-level feature representation using recurrent neural network for speech emotion recognition,'' in _Proc. Interspeech_ , 2015.
  * [7] A. Satt, S. Rozenberg, and R. Hoory, ``Efficient Emotion Recognition from Speech Using Deep Learning on Spectrograms,'' in _Proc. Interspeech_ , 2017.
  * [8] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2021.
  * [9] L.-W. Chen and A. Rudnicky, ``Exploring wav2vec 2.0 fine-tuning for improved speech emotion recognition,'' in _ICASSP IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023.
  * [10] J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, ``Dawn of the transformer era in speech emotion recognition: Closing the valence gap,'' _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [11] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, ``Iemocap: Interactive emotional dyadic motion capture database,'' _Language resources and evaluation_ , 2008.
  * [12] N. Kurpukdee, S. Kasuriya, V. Chunwijitra, C. Wutiwiwatchai, and P. Lamsrichan, ``A study of support vector machines for emotional speech recognition,'' in _8th international conference of information and communication technology for embedded systems (IC-ICTES)_ , 2017.
  * [13] J. Wang, M. Xue, R. Culhane, E. Diao, J. Ding, and V. Tarokh, ``Speech emotion recognition with dual-sequence lstm architecture,'' in _ICASSP IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020.
  * [14] S. R. Livingstone and F. A. Russo, ``The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,'' _PLoS ONE_ , 2018.
  * [15] M. Farooq, F. Hussain, N. K. Baloch, F. R. Raja, H. Yu, and Y. B. Zikria, ``Impact of feature selection algorithm on speech emotion recognition using deep convolutional neural network,'' _Sensors_ , 2020.
  * [16] F. Dal Rì, F. Cifariello Ciardi, and N. Conci, ``Speech emotion recognition and deep learning: An extensive validation using convolutional neural networks,'' _IEEE Access_ , 2023.
  * [17] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, ``Crema-d: Crowd-sourced emotional multimodal actors dataset,'' _IEEE Transactions on Affective Computing_ , 2014.
  * [18] P. Jackson and S. Haq, ``Surrey audio-visual expressed emotion (savee) database,'' _University of Surrey: Guildford, UK_ , 2014.
  * [19] K. Dupuis and M. K. Pichora-Fuller, ``Toronto emotional speech set (tess)-younger talker_happy.'' Toronto: University of Toronto, Psychology Department, 2010., 2010.
  * [20] M. Farooq, F. Hussain, N. K. Baloch, F. R. Raja, H. Yu, and Y. B. Zikria, ``Impact of feature selection algorithm on speech emotion recognition using deep convolutional neural network,'' _Sensors (Basel, Switzerland)_ , 2020.
  * [21] L. Yue, P. Hu, S.-C. Chu, and J.-S. Pan, ``Equilibrium optimizer for emotion classification from english speech signals,'' _IEEE Access_ , 2023.
  * [22] F. Haider, S. Pollak, P. Albert, and S. Luz, ``Emotion recognition in low-resource settings: An evaluation of automatic feature selection methods,'' _Computer Speech & Language_, 2021.
  * [23] T. T. L. Dejoli, Q. He, and W. Xie, ``Audio,Speech and Vision Processing Lab Emotional Sound database (ASVP-ESD).'' Zenodo, 2021.
  * [24] A. Adigwe, N. Tits, K. E. Haddad, S. Ostadabbas, and T. Dutoit, ``The emotional voices database: Towards controlling the emotion dimension in voice generation systems,'' _arXiv preprint arXiv:1806.09514_ , 2018.
  * [25] E. Parada-Cabaleiro, G. Costantini, A. Batliner, A. Baird, and B. Schuller, ``Emofilm-a multilingual emotional speech corpus,'' in _Zenodo_ , 2018.
  * [26] J. James, L. Tian, and C. Watson, ``An open source emotional speech corpus for human robot interaction applications,'' in _Proc. Interspeech_ , 2018.
  * [27] K. Zhou, B. Sisman, R. Liu, and H. Li, ``Emotional voice conversion: Theory, databases and esd,'' _Speech Communication_ , 2022.
  * [28] J. Liu, Z. Liu, L. Wang, Y. Gao, L. Guo, and J. Dang, ``Temporal attention convolutional network for speech emotion recognition with latent representation,'' in _Proc. Interspeech_ , 2020.
  * [29] S. E. Eskimez, Z. Duan, and W. Heinzelman, ``Unsupervised learning approach to feature analysis for automatic speech emotion recognition,'' in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018.
  * [30] F. Bao, M. Neumann, and N. T. Vu, ``CycleGAN-Based Emotion Style Transfer as Data Augmentation for Speech Emotion Recognition,'' in _Proc. Interspeech_ , 2019.
  * [31] B. Schuller, S. Reiter, R. Muller, M. Al-Hames, M. Lang, and G. Rigoll, ``Speaker independent speech emotion recognition by ensemble classification,'' 2005.
  * [32] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' in _ICML_ , 2022.
  * [33] H. Cao, R. Verma, and A. Nenkova, ``Speaker-sensitive emotion recognition via ranking: Studies on acted and spontaneous speech,'' _Computer speech & language_, 2015.
  * [34] S. Basu, J. Chakraborty, A. Bag, and M. Aftabuddin, ``A review on emotion recognition using speech,'' in _International conference on inventive communication and computational technologies (ICICCT)_ , 2017.
  * [35] S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, ``Meld: A multimodal multi-party dataset for emotion recognition in conversations,'' in _arXiv preprint arXiv:1810.02508_ , 2018.
  * [36] Z.-T. Liu, P. Xiao, D.-Y. Li, and M. Hao, ``Speaker-independent speech emotion recognition based on cnn-blstm and multiple svms,'' in _Proc of ICIRA 2019_ , H. Yu, J. Liu, L. Liu, Z. Ju, Y. Liu, and D. Zhou, Eds. Springer, 2019.
  * [37] A. Fernández, S. Garcia, F. Herrera, and N. V. Chawla, ``Smote for learning from imbalanced data: progress and challenges, marking the 15-year anniversary,'' _Journal of Artificial Intelligence Research_ , 2018.
  * [38] H. He, Y. Bai, E. A. Garcia, and S. Li, ``Adasyn: Adaptive synthetic sampling approach for imbalanced learning,'' in _IEEE International Joint Conference on Neural Networks_ , 2008.


# Refining Coded Image in Human Vision Layer Using CNN-Based Post-Processing

Takahiro Shindo, Yui Tatsumi, Taiju Watanabe, Hiroshi Watanabe  Waseda
University, Tokyo, Japan

###### Abstract

Scalable image coding for both humans and machines is a technique that has
gained a lot of attention recently. This technology enables the hierarchical
decoding of images for human vision and image recognition models. It is a
highly effective method when images need to serve both purposes. However, no
research has yet incorporated the post-processing commonly used in popular
image compression schemes into scalable image coding method for humans and
machines. In this paper, we propose a method to enhance the quality of decoded
images for humans by integrating post-processing into scalable coding scheme.
Experimental results show that the post-processing improves compression
performance. Furthermore, the effectiveness of the proposed method is
validated through comparisons with traditional methods.

###### Keywords:

scalable image coding, image coding for machines, post-processing, image
compression

##  I Introduction

Rapid advancements in image recognition models are making our lives easier.
These models can estimate the size, location, and type of objects and
backgrounds in an image. However, image analysis using these models is not
flawless and often needs human verification. For example, in traffic
monitoring and farm animal management, machines perform the initial image
analysis, but human review is still necessary. Scalable image coding for both
humans and machines is a promising technology for these applications.

In traditional image codecs like HEVC[1] and VVC[2], post-processing is an
effective way to enhance the quality of decoded images. Many studies have also
explored using neural networks for post-processing to eliminate coding noise
[3, 4]. However, no research has yet focused on applying post-processing to
scalable coding for humans and machines. Scalable coding methods decode images
for humans by adding information to those decoded for image recognition
models, leading to different coding noise than standard image compression
methods. In this paper, we introduce a post-processing to reduce this coding
noise. Experimental results show that post-processing improves the quality of
decoded images for human in scalable image coding methods.

Figure 1: Image compression process and structure of the post-processing
model.

##  II Related Works

###  II-A Scalable Image Coding for Humans and Machines

There is ongoing research in scalable image coding methods that cater to both
image recognition models and human vision[5, 6]. This field is known as
”scalable image coding for humans and machines,” and several methods have been
proposed. These methods decode images for humans by adding supplementary
information to the images decoded for machines. Learned Image Compression
(LIC)[7], a neural network-based image compression technique, is frequently
used in this area. By combining the LIC model for machines with the LIC model
for additional information, a method for decoding images suitable for both
machines and humans has been developed[5]. The inclusion of SA-ICM[8] in
scalable coding method enables image compression for various image recognition
models and humans. SA-ICM is one of the image coding methods for machines and
is not optimized for a specific image recognition model[9]. Hence, scalable
image coding methods incorporating with SA-ICM can compress images for various
image recognition models and humans.

###  II-B Post-processing for Decoded Images

Post-processing for decoded images is an effective method for enhancing both
image quality[3] and image recognition accuracy[4]. Numerous neural network-
based models for post-processing have been proposed to improve compression
performance. These models are trained to make the encoded image more closely
resemble the original image. The Mean Squared Error (MSE) between the original
and output images is often used as the loss function.

##  III Proposed Method

We propose a method to enhance the image quality of compressed images by
incorporating a post-processing model into the scalable coding method. As a
scalable coding method, we employ a method that progressively compresses
images for both humans and machines by linking two LIC models[5]. The decoded
image produced by this method exhibits specific coding noise compared to the
LIC model typically used for human decoding. To address this noise, we aim to
enhance image quality by training the post-processing model to learn and
mitigate this unique noise. We adopt the Residual in Residual Dense Block
(RRDB) structure proposed by X.Wang et al., which is widely utilized for tasks
like image super-resolution and denoising[10]. Fig. 1 illustrates the model
structure and its position in the image compression pipeline.

Figure 2: Compression performance of the proposed and comparative methods. ”sic” stands for scalable image coding. TABLE I: Image quality (PSNR) in the decoded image method | $\lambda$
---|---
0.005 | 0.010 | 0.020 | 0.030
w/o post-processing | 29.76 | 31.26 | 32.82 | 33.81
w/ post-processing ($l=1$) | 29.87 | 31.40 | 33.03 | 34.08
w/ post-processing ($l=2$) | 29.88 | 31.42 | 33.05 | 34.09

##  IV Experiment

We utilize the COCO dataset[11] to train and evaluate post-processing model.
Initially, we compress all images in the dataset for human vision, using a
scalable coding method[5]. Subsequently, using the COCO-Train dataset prepared
in this manner, we train the post-processing model using MSE-loss function. To
compare experimental results across post-processing model sizes, we train one
model with a single RRDB and another with two RRDBs. Once trained, the models
are applied to the compressed images in the COCO-Val dataset to mitigate
encoding noise.

The image quality of with and without post-processing are depicted in Table I
and Fig.2. In Table I, the value of $l$ represents the number of RRDBs in the
model structure and the $\lambda$ value is a variable to control the bitrate.
In Fig.2, a comparison of the image compression performance between the
proposed and conventional methods is presented. The performance of our
proposed method is represented by the blue curve, while the performance of the
original scalable image coding method is depicted by the light blue curve. The
dotted line illustrates the total amount of additional information and image
information for machines in scalable image coding. The solid line shows only
the amount of additional information. Additionally, Fig.3 presents examples of
the input and output images from the post-processing model, as well as the
noise present in each image. These findings show the effectiveness of the
post-processing in scalable image coding.

##  V Conclusion

In this paper, we propose a post-processinging method aimed at eliminating
coding noise in scalable image coding method. Experimental results demonstrate
that employing the post-processing model enhances the image quality of the
decoded image. Future research endeavors should involve the incorporation of
evaluation metrics like SSIM and LPIPS, as well as an exploration of methods
to enhance image quality.

## Acknowledgment

The results of this research were obtained from the commissioned research
(JPJ012368C05101) by National Institute of Information and Communications
Technology (NICT), Japan.

Figure 3: An example of the decoded images with and without post-processing
and the coding noise in those images. Areas with large coding noise are shown
brighter.

## References

  * [1] High Efficiency Video Coding, Standard ISO/IEC 23008-2, ISO/IEC JTC 1, Apr. 2013.
  * [2] Versatile Video Coding, Standard ISO/IEC 23090-3, ISO/IEC JTC 1, Jul. 2020.
  * [3] F. Zhang et al., ”Enhancing VVC Through Cnn-Based Post-Processing,” 2020 IEEE International Conference on Multimedia and Expo (ICME), 2020, pp. 1-6.
  * [4] T. Shindo et al., ”Accuracy Improvement of Object Detection in VVC Coded Video Using YOLO-v7 Features,” 2023 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), 2023, pp. 247-251.
  * [5] T. Shindo et al., ”Scalable Image Coding for Humans and Machines Using Feature Fusion Network,” arXiv e-prints, 2024, arXiv:2405.09152.
  * [6] H. Choi et al., “Scalable Image Coding for Humans and Machines,” IEEE Transaction on Image Processing, vol. 31, 2022, pp. 2739-2754.
  * [7] J. Liu et al., “Learned Image Compression with Mixed Transformer-CNN Architectures,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 14388-14397.
  * [8] T. Shindo et al., ”Image Coding for Machines with Edge Information Learning Using Segment Anything.” arXiv e-prints, 2024, arXiv:2403.04173.
  * [9] T. Shindo et al., ”Image Coding for Machines with Object Region Learning,” 2024 IEEE 21st Consumer Communications & Networking Conference (CCNC), 2024, pp. 1040-1041.
  * [10] X. Wang et al., ”Esrgan: Enhanced super-resolution generative adversarial networks,” in Proceedings of the European conference on computer vision (ECCV) workshops. 2018. p. 0-0.
  * [11] T. Y. Lin et al., ”Microsoft COCO: Common Objects in Context,” Computer Vision - ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8693. 2014, pp 740-755.