# The problem of calculating the $\beta$-Bogoliubov coefficient in Non-
Oscillating models

Jaume de Haro<EMAIL_ADDRESS>Departament de Matemàtiques, Universitat
Politècnica de Catalunya, Diagonal 647, 08028 Barcelona, Spain  Llibert Aresté
Saló<EMAIL_ADDRESS>School of Mathematical Sciences, Queen Mary
University of London, Mile End Road, London, E1 4NS, United Kingdom

###### Abstract

The calculation of the Bogoliubov coefficients is a key piece to obtain the
reheating temperature of the Universe. In all cases this calculation is
performed either in toy models where some derivative of the potential is
discontinuous at some points or by making some approximations to the model.
The result of these calculations is applied to more realistic models without
checking if they really apply to this realistic model because the exact
calculation of the Bogoliubov coefficients for a viable model, which is
usually depicted by a smooth potential, requires a very complicated numerical
calculation. Here we want to point out the difficulties that one encounters
when trying to compute numerically these coefficients for a smooth potential
without making approximations, which could lead to completely different
results.

Bogoliubov coefficients; Gravitational particle production; Instant
preheating; Quintessential Inflation; $\alpha$-attractors.

###### pacs:

04.20.-q, 98.80.Jk, 98.80.Bp

##  I Introduction

As was stated by Alan Guth in his seminal paper guth , a viable inflationary
model needs a reheating mechanism to match inflation with the hot Big Bang
scenario. In standard inflation, where the potential contains a deep well, the
reheating mechanism is quite simple: at the end of the inflationary period the
inflaton field falls in the well and starts oscillating, thus releasing its
energy and creating particles, which reheats the universe after thermalization
.

However, after the discovery of the current cosmic acceleration riess ;
perlmutter , some models without a deep well were proposed, the so-called Non-
Oscillating models, such as Quintessential Inflation pv , which tried to unify
inflation with the current cosmic acceleration, where the inflaton field
evolves monotonically. For these models, there are basically two mechanisms to
produce particles which will reheat the universe. The first one is the so-
called gravitational particle production parker ; ford  and the other one is
the instant preheating fkl0 ; fkl . In the former case, when the reheating is
due to the production of very light particles, the calculation of the
$\beta$-Bogoliubov coefficients -the main ingredient to calculate the energy
density of the produced particles- is performed only for toy models with some
discontinuities in some derivative of the scale factor (see for instance
damour ; giovannini ), obtaining that the energy density of these produced
particles is proportional to the scale of inflation to the fourth power. This
result is, of course, applied to more smooth models but without checking its
validity, for example in pv . When the reheating is due to the gravitational
production of superheavy particles, which have to decay into lighter ones, one
encounters the same situation: The $\beta$-Bogoliubov coefficients can only be
calculated analytically for potentials with a discontinuity in some of its
derivatives. The problem is that the more realistic models depend on a smooth
potential and then the calculation of these coefficients requires, as we will
see, very complicated numeric calculations. Finally, dealing with instant
preheating, to calculate analytically the $\beta$-Bogoliubov coefficients one
has to perform some approximation to obtain a parabolic problem which has a
well-known analytical solution, but it is not justified that the correct value
does resemble the analytical one, obtained with these approximations.

Therefore, in this work we try to explain systematically some different ways
to find numerically, for smooth potentials, the exact value -without
unjustified approximations- of the Bogoliubov coefficients and the problems we
have encountered. Basically, we have checked that two different ways could
lead to different results, meaning that our numerical calculations are not
reliable at all. Moreover, we are sure that this problem is related with the
background which, for smooth potentials, has to be as well calculated
numerically. More precisely, the problem comes from the scale factor, which
due to the inflationary period grows more than 25 orders, and then its
numerical calculation needs also an interpolation which does not guarantee at
all its real value.

The present work is organized as follows: In Section II we start presenting
the well-known diagonalization method, obtaining the equations that satisfy
the Bogoliubov coefficients. Once we have these equations we consider an
approximation that does not affect the final result of our calculations.
Section III is devoted to the gravitational particle production. Starting with
toy models, we calculate analytically the value of the $\beta$-Bogoliubov
coefficient and we show that this analytical value does not coincide with any
of the values obtained with our numerical methods. We also explain and perform
the numerical calculations for a smooth potential, obtaining results that do
not seem reliable as we explain in the text. In Section IV we deal with
instant preheating, explaining the approximations made to obtain the
analytical expression of the Bogoliubov coefficients, and pointing out that a
numerical calculation without these approximations is needed to validate the
analytic results. The importance of the Bogoliubov coefficients is stressed in
Section V, where we show the expression of the reheating temperature and its
bounds as a function of these coefficients. Finally we present our conclusions
in the last section. We have included as well an appendix where we explain
more thoroughly the algorithm that has been used for obtaining the numerical
results and the methods that have been employed.

##  II The diagonalization method

Given a massive quantum scalar field conformally coupled to gravity, namely
$\chi$, the Klein-Gordon (K-G) equation in the Fourier space, which is
satisfied by the $\chi$-modes in the flat Friedmann-Lemaître-Robertson-Walker
(FLRW) spacetime, is given by

$\displaystyle\chi_{k}^{\prime\prime}(\tau)+\omega^{2}_{k}(\tau)\chi_{k}(\tau)=0,$
(1)

where the prime denotes the derivative with respect to the conformal time
$\tau$ and $\omega_{k}(\tau)$ is the time-dependent frequency, which in the
case of gravitational particle production parker ; ford  is given by

$\displaystyle\omega_{k}^{2}(\tau)=k^{2}+a^{2}(\tau)m_{\chi}^{2},$ (2)

being $m_{\chi}$ the heavy mass of the produced particles and $a$ the scale
factor. When particles are produced via Instant Preheating fkl0 ; fkl , the
frequency is given by

$\displaystyle\omega_{k}^{2}(\tau)=k^{2}+a^{2}(\tau)[m_{X}^{2}+g^{2}(\varphi-\varphi_{kin})^{2}],$
(3)

where $m_{X}$ stands for the bare mass of the produced particles, which is not
related with $m_{\chi}$, $\varphi$ is the inflaton field and $\varphi_{kin}$
its value at the beginning of kination (the regime where practically all
energy is kinetic).

As usual, the modes that define the vacuum state at a given initial time
$\tau_{i}$ are the ones that minimize the energy density, so they must satisfy
the conditions

$\displaystyle\chi_{k}(\tau_{i})=\frac{1}{\sqrt{2\omega_{k}(\tau_{i})}}e^{-i\int^{\tau_{i}}\omega_{k}(\bar{\eta})d\bar{\eta}},\quad\chi_{k}^{\prime}(\tau_{i})=-i\omega_{k}(\tau_{i})\chi_{k}(\tau_{i}),$
(4)

and thus, the vacuum expectation value of the energy density will be given by
Bunch

$\displaystyle\langle\rho(\tau)\rangle\equiv\langle
0|\hat{\rho}(\tau)|0\rangle=\frac{1}{4\pi^{2}a^{4}(\tau)}\int_{0}^{\infty}k^{2}dk\left(|\chi_{k}^{\prime}(\tau)|^{2}+\omega^{2}_{k}(\tau)|\chi_{k}(\tau)|^{2}-\omega_{k}(\tau)\right),$
(5)

where, in order to obtain a finite energy density, one has to subtract the
energy density of the zero-point oscillations of the vacuum
$\frac{1}{(2\pi)^{3}a^{4}(\tau)}\int d^{3}k\frac{1}{2}\omega_{k}(\tau)$.

Following the method developed in Zeldovich  (see also Section $9.2$ of
gmmbook ), we will write the modes as follows,

$\displaystyle\chi_{k}(\tau)=\alpha_{k}(\tau)\frac{e^{-i\int^{\tau}\omega_{k}(\bar{\tau})d\bar{\tau}}}{\sqrt{2\omega_{k}(\tau)}}+\beta_{k}(\tau)\frac{e^{i\int^{\tau}\omega_{k}(\bar{\tau})d\bar{\tau}}}{\sqrt{2\omega_{k}(\tau)}},$
(6)

where $\alpha_{k}(\tau)$ and $\beta_{k}(\tau)$ are the time-dependent
Bogoliubov coefficients. Now, imposing that the modes satisfy the condition

$\displaystyle\chi_{k}^{\prime}(\tau)=-i\omega_{k}(\tau)\left(\alpha_{k}(\tau)\frac{e^{-i\int^{\tau}\omega_{k}(\bar{\tau})d\bar{\tau}}}{\sqrt{2\omega_{k}(\tau)}}-\beta_{k}(\tau)\frac{e^{i\int^{\tau}\omega_{k}(\bar{\tau})d\bar{\tau}}}{\sqrt{2\omega_{k}(\tau)}}\right),$
(7)

one can show that the Bogoliubov coefficients must satisfy the system

$\displaystyle\left\\{\begin{array}[]{ccc}\alpha_{k}^{\prime}(\tau)&=&\frac{\omega_{k}^{\prime}(\tau)}{2\omega_{k}(\tau)}e^{2i\int^{\tau}\omega_{k}(\bar{\tau})d\bar{\tau}}\beta_{k}(\tau)\\\
\beta_{k}^{\prime}(\tau)&=&\frac{\omega_{k}^{\prime}(\tau)}{2\omega_{k}(\tau)}e^{-2i\int^{\tau}\omega_{k}(\bar{\tau})d\bar{\tau}}\alpha_{k}(\tau),\end{array}\right.$
(10)

in order for the expression (6) to be a solution of the equation (1).

Finally, inserting (6) into the expression for the vacuum energy density (5),
and taking into account that the Bogoliubov coefficients satisfy the equation
$|\alpha_{k}(\tau)|^{2}-|\beta_{k}(\tau)|^{2}=1$, one finds the following
diagonalized form of the energy density Zeldovich ,

$\displaystyle\langle\rho(\tau)\rangle=\frac{1}{2\pi^{2}a^{4}(\tau)}\int_{0}^{\infty}k^{2}\omega_{k}(\tau)|\beta_{k}(\tau)|^{2}dk,$
(11)

where it is important to notice that $|\beta_{k}(\tau)|^{2}$ encodes the
vacuum polarization effects and also the production of real particles, which
are only produced when the adiabatic evolution breaks. In non-oscillating
models this happens during the phase transition from the end of inflation to
the beginning of kination (a period where all the energy density is kinetic,
i.e., the potential energy is negligible), and fortunately the polarization
effects disappear immediately after the beginning of kination, when the value
of $|\beta_{k}(\tau)|$ stabilizes to a value which we will denote by
$|\beta_{k}|$, and thus, only encodes the production of real particles.

A final remark is in order: Once one has obtained the scale factor -in Section
3 we explain how to obtain it-, instead of solving numerically (10), another
way to calculate the value of $\beta_{k}$ (the value of the $\beta$-Bogoliubov
coefficient when it stabilizes) is to solve numerically the equation (1), with
initial conditions (4), for example, at the horizon crossing, i.e.,

$\displaystyle\chi_{k}(\tau_{*})=\frac{1}{\sqrt{2\omega_{k}(\tau_{*})}},\quad\chi_{k}^{\prime}(\tau_{*})=-i\omega_{k}(\tau_{*})\chi_{k}(\tau_{*}),$
(12)

where the star denotes that the quantities are evaluated at the horizon
crossing. This means that at that moment the quantum field is at the vacuum.
In fact, as we will see later, for the relevant modes the quantum field is in
the adiabatic vacuum at the horizon crossing, and it does not matter if one
chooses an early time for the initial conditions, because the relevant modes
continue in the adiabatic vacuum.

Then, after the beginning of kination the Bogoliubov coefficients stabilize
and the $k$-mode has the simple form

$\displaystyle\chi_{k}(\tau)=\alpha_{k}\phi_{k}(\tau)+\beta_{k}\bar{\phi}_{k}(\tau),$
(13)

where we have used the notation

$\displaystyle\phi_{k}(\tau)=\frac{e^{-i\int^{\tau}\omega_{k}(\bar{\tau})d\bar{\tau}}}{\sqrt{2\omega_{k}(\tau)}},$
(14)

and $\bar{\phi}_{k}(\tau)$ is the conjugate of ${\phi}_{k}(\tau)$.

Then, using the Wronskian $W[f,g]\equiv fg^{\prime}-f^{\prime}g$, one has

$\displaystyle\beta_{k}=\frac{W[\chi_{k},\phi_{k}]}{W[\bar{\phi}_{k},\phi_{k}]}.$
(15)

This is the first method that we present to calculate the $\beta$-Bogoliubov
coefficient, and it is used in hashiba  to obtain its numerical results.
However, as we will explain later there are other forms to obtain $\beta_{k}$.

###  II.1 Approximation

Using once again the relation
$|\alpha_{k}(\tau)|^{2}-|\beta_{k}(\tau)|^{2}=1$, and since $\beta_{k}(\tau)$
has to be very small, one can take the well-justified approximation
$\alpha_{k}(\tau)\cong 1$, and (10) becomes the simpler equation

$\displaystyle\beta_{k}^{\prime}(\tau)=\frac{\omega_{k}^{\prime}(\tau)}{2\omega_{k}(\tau)}e^{-2i\int^{\tau}\omega_{k}(\bar{\tau})d\bar{\tau}},$
(16)

whose solution is given by

$\displaystyle\beta_{k}(\tau)=\int_{\tau^{*}}^{\tau}\frac{\omega_{k}^{\prime}(\tilde{\tau})}{2\omega_{k}(\tilde{\tau})}e^{-2i\int^{\tilde{\tau}}_{\tau_{*}}\omega_{k}(\bar{\tau})d\bar{\tau}}d\tilde{\tau},$
(17)

where we have assumed that $\beta_{k}(\tau)$ vanishes at the horizon crossing,
which also means that at the horizon crossing the quantum field is at the
vacuum. This is a second way to calculate the Bogoliubov coefficients.

The problem is that this oscillatory integral seems difficult to be computed.
For this reason, in order to calculate this coefficient we make another
approximation ema  to simplify its calculation:

  1. 1.

When $k\leq a_{*}m_{\chi}$, one can make the approximation
$\omega_{k}(\tau)\cong a(\tau)m_{\chi}$ obtaining

$\displaystyle\beta_{k}(\tau)\cong\int_{t_{*}}^{t}\frac{H(t)}{2}e^{-2im_{\chi}(t-t_{*})}dt,$
(18)

where $H$ is the Hubble rate and $t$ denotes the cosmic time.

  2. 2.

When $k\geq a_{*}m_{\chi}$, we define $\tau_{k}$ such that
$k=a(\tau_{k})m_{\chi}$, and we make the approximation $\omega_{k}(\tau)\cong
k$ for $\tau<\tau_{k}$ and $\omega_{k}(\tau)\cong a(\tau)m_{\chi}$ when
$\tau>\tau_{k}$, obtaining

$\displaystyle\beta_{k}(\tau)\cong\int_{\tau_{*}}^{\tau_{k}}\frac{a(\tau)a^{\prime}(\tau)m_{\chi}^{2}}{2k^{2}}e^{-2ik(\tau-\tau_{*})}d\tau+e^{-2ik(\tau_{k}-\tau_{*})}\int_{t_{k}}^{t}\frac{H(t)}{2}e^{-2im_{\chi}(t-t_{k})}dt.$
(19)

However, the problem of this approximation is that the error made doing it
could be greater than the value of the $\beta$-Bogoliubov coefficient. So, in
practice this approximation does not seem viable.

Finally, a third method comes from the important observation that the modulus
of the solution (17) is also the modulus of the solution of the simple linear
differential equation

$\displaystyle
y_{k}^{\prime}-2i\omega_{k}(\tau)y_{k}=\frac{\omega^{\prime}_{k}(\tau)}{2\omega_{k}(\tau)},$
(20)

with initial condition $y_{k}(\tau_{*})=0$, because one has
$|\beta_{k}(\tau)|=|y_{k}(\tau)|$ by taking into account the formula of
variation of parameters for first order differential equations.

###  II.2 The energy density of produced particles

Let $\tau_{s}$ be the moment when $|\beta_{k}(\tau)|$ stabilizes, and as we
have already stated we will denote this value by $|\beta_{k}|$. Then, after
the beginning of kination, the energy density of the real particles will be

$\displaystyle\langle\rho(\tau)\rangle=\frac{1}{2\pi^{2}a^{4}(\tau)}\int_{0}^{\infty}k^{2}\omega_{k}(\tau)|\beta_{k}|^{2}dk.$
(21)

Focusing our analysis on the case of gravitational particle production, i.e.,
when $\omega_{k}(\tau)=\sqrt{k^{2}+a^{2}(\tau)m^{2}_{\chi}}$, and considering
superheavy particles satisfying $m_{\chi}\gg H_{*}$ (here $H_{*}$ denotes the
scale of inflation, i.e., the value of the Hubble rate at the horizon
crossing), one expects that the modes that really contribute to the energy
density satisfy $k\leq a_{kin}m_{\chi}$ where the sub-index $kin$ denotes that
the quantities are evaluated at the beginning of kination. We expect that
these modes are the only relevant ones because this is what happens for the
toy models that we will study in the next section, and the physical reason is
that the real particles are produced during the phase transition from the end
of inflation to the beginning of kination, so the modes satisfying $k\gg
a_{kin}m_{\chi}$ do not feel the gravity because they satisfy the K-G equation
$\chi_{k}^{\prime\prime}+k^{2}\chi_{k}=0$ which corresponds to the Minkowski
modes and where particles are not produced.

For these modes one can make the approximation $\omega_{k}(\tau)\cong
a(\tau)m_{\chi}$, and thus, the energy density of produced particles becomes

$\displaystyle\langle\rho(\tau)\rangle\cong\frac{m_{\chi}}{2\pi^{2}a^{3}(\tau)}\int_{0}^{a_{kin}m_{\chi}}k^{2}|\beta_{k}|^{2}dk.$
(22)

Here it is important to remark that the modes satisfying $k\leq a_{*}m_{\chi}$
do not contribute to the energy density of produced particles. Effectively,
the contribution of those modes is

$\displaystyle\frac{m_{\chi}}{2\pi^{2}a^{3}(\tau)}\int_{0}^{a_{*}m_{\chi}}k^{2}|\beta_{k}|^{2}dk\leq\frac{m_{\chi}}{2\pi^{2}a^{3}(\tau)}\int_{0}^{a_{*}m_{\chi}}k^{2}dk=\frac{m_{\chi}^{4}}{6\pi^{2}}\left(\frac{a_{*}}{a(\tau)}\right)^{3},$
(23)

where we have used that $|\beta_{k}|\leq 1$.

Now, taking into account that $a(\tau)\geq a_{*}e^{N}$, where $N$ is the
number of efolds from the horizon exit to the end of inflation, being
satisfied that $N>60$ for models with a kination phase, we conclude that the
contribution of these modes is less than

$\displaystyle\frac{m_{\chi}^{4}}{6\pi^{2}}e^{-3N},$ (24)

which is in practice a negligible energy density.

Using the same reasoning one can see that the modes satisfying $k\geq
a_{kin}H_{*}$ are in the adiabatic vacuum at the horizon crossing.
Effectively, we have to evaluate
$\frac{\omega_{k}^{\prime}(\tau_{*})}{\omega_{k}^{2}(\tau_{*})}$ whose value
is given by

$\displaystyle\frac{\omega_{k}^{\prime}(\tau_{*})}{\omega_{k}^{2}(\tau_{*})}=\frac{a_{*}^{3}H_{*}m_{\chi}^{2}}{\omega_{k}^{3}(\tau_{*})}\leq\frac{m_{\chi}^{2}}{H_{*}^{2}}\left(\frac{a_{*}}{a_{kin}}\right)^{3}\leq\frac{m_{\chi}^{2}}{H_{*}^{2}}e^{-3N}\ll
1,$ (25)

where we have used that $\omega_{k}(\tau_{*})\cong k\leq a_{kin}H_{*}$.

A final remark is in order: When one deals with superheavy particles the
energy density (21) has to evolve as matter, meaning that the evolution of the
energy density is

$\displaystyle\langle\rho(\tau)\rangle=\langle\rho_{kin}\rangle\left(\frac{a_{kin}}{a(\tau)}\right)^{3},$
(26)

where $\langle\rho_{kin}\rangle$ is the value of (21) at the beginning of
kination. This important result, as we will see in the next section, could be
checked studying simple toy models.

##  III Gravitational particle production

###  III.1 Toy models

To warm up, here we study some simple models containing discontinuities. First
of all, we consider the following scale factor (see for instance hashiba ),

$\displaystyle
a(t)=\left\\{\begin{array}[]{ccc}a_{*}e^{H_{inf}(t-t_{*})}&\mbox{when}&t\leq
t_{kin}\\\ &&\\\ a_{kin}\left(\frac{t}{t_{kin}}\right)^{1/3}&\mbox{when}&t\geq
t_{kin},\end{array}\right.$ (30)

where $a_{*}=1$, $H_{inf}=10^{-6}M_{pl}$, $t_{kin}=\frac{1}{3H_{inf}}$,
$t_{*}=t_{kin}-\frac{65}{H_{inf}}$ and $a_{kin}=a(t_{kin})=e^{65}$.

For this scale factor, which depicts a sudden phase transition from de Sitter
to kination, one can calculate analytically the $\beta_{k}$ Bogoliubov
coefficient using the formula (17). Since the derivative of the Hubble rate is
discontinuous at the time $\tau_{kin}$, integrating two times by parts and
disregarding the terms in $\tau_{*}$ and $\tau_{s}$ (we consider only relevant
modes, that is, satisfying $k\gg a_{*}m_{\chi}$, then at the horizon crossing
the corresponding terms are negligible because the adiabatic condition
$\omega_{k}^{\prime}/\omega_{k}^{2}\ll 1$ is strongly satisfied) one can
deduce that the leading term is given by

$\displaystyle|\beta_{k}|\cong\frac{3a_{kin}^{4}H^{2}_{inf}m_{\chi}^{2}}{8\omega_{k}^{4}(\tau_{kin})},$
(31)

which for $k\ll a_{kin}m_{\chi}$ is of the order

$\displaystyle|\beta_{k}|\sim\frac{3H_{inf}^{2}}{8m_{\chi}^{2}}.$ (32)

Then, for example, choosing a heavy mass of the order of $m_{\chi}\sim
10^{15}$ GeV one has

$\displaystyle|\beta_{k}|^{2}\sim 5\times 10^{-12}.$ (33)

Therefore, for modes in the range $a_{*}m_{\chi}\ll k\ll a_{kin}m_{\chi}$ one
would have to obtain numerically the same result. On the contrary, if the
numerical results differ from the analytical ones, this means that the
numerical method used does not work.

Note also that one can calculate analytically the energy density inserting the
Bogoliubov coefficient (31) in (22), and using the formula

$\displaystyle\int_{0}^{\infty}\frac{dx}{(1+x^{2})^{n+1}}=\frac{2n-1}{2n}\int_{0}^{\infty}\frac{dx}{(1+x^{2})^{n}}$
(34)

one gets

$\displaystyle\langle\rho(\tau)\rangle=\frac{9H_{inf}^{4}}{16^{3}\pi}\left(\frac{a_{kin}}{a(\tau)}\right)^{3}\sim
10^{-3}H_{inf}^{4}\left(\frac{a_{kin}}{a(\tau)}\right)^{3},$ (35)

which coincides with the result obtained in hashiba  and shows that this
energy density evolves like matter.

This result shows that the relevant modes, the ones that contribute to the
energy density, are the ones that satisfy $k\leq a_{kin}m_{\chi}$ because the
contribution of these modes is equal to

$\displaystyle\frac{9H_{inf}^{4}}{128\pi^{2}}\left(\frac{\pi}{64}-\frac{1}{48}\right)\left(\frac{a_{kin}}{a(\tau)}\right)^{3},$
(36)

which is of the same order than the expression obtained above. This is
consistent, as we have already expalined, with the fact that for $k\gg
a_{kin}m_{\chi}$ at the beginning of kination $\omega_{k}(\tau_{kin})\cong k$,
and thus, the K-G equation is given at that moment by
$\chi_{k}^{\prime\prime}+k^{2}\chi_{k}=0$, i.e., they are in the Minkowski
vacuum and consequently they do not feel the gravity. Thus, they do not
produce particles.

The second toy model is given by the potential

$\displaystyle
V(\varphi)=\left\\{\begin{array}[]{ccc}\frac{1}{2}m^{2}\varphi^{2}&\mbox{for}&\varphi\leq
0\\\ 0&\mbox{for}&\varphi\geq 0,\end{array}\right.$ (39)

where $m\cong 5\times 10^{-6}M_{pl}$,
$\varphi_{*}\cong-\frac{2\sqrt{2}}{\sqrt{1-n_{s}}}M_{pl}$ with $n_{s}\cong
0.96$, and
$H_{*}=\sqrt{\frac{V(\varphi_{*})}{3M_{pl}^{2}}}\cong\sqrt{\frac{m^{2}\varphi_{*}^{2}}{6M_{pl}^{2}}}.$

Now, the second derivative of the potential is discontinuous, which means that
the third derivative of the Hubble rate is discontinuous.

In that case, for $a_{*}m_{\chi}\ll k\ll a_{kin}m_{\chi}$, integrating (17)
four times by parts, disregarding the boundary terms because for the relevant
modes the adiabatic condition $\omega_{k}^{\prime}/\omega_{k}^{2}\ll 1$ is
strongly satisfied at the boundary points, and using the Raychaudhuri equation
$\dot{H}=-\frac{1}{2M_{pl}^{2}}\dot{\varphi}^{2}$, after a cumbersome
calculation one gets

$\displaystyle|\beta_{k}|\cong\frac{3a_{kin}^{6}m^{4}m_{\chi}^{2}}{32\omega_{k}^{6}(\tau_{kin})}\cong\frac{3m^{4}}{32m_{\chi}^{4}},$
(40)

which for $m_{\chi}\sim 10^{15}$ GeV leads to

$\displaystyle|\beta_{k}|^{2}\sim 4\times 10^{-18}.$ (41)

Effectively, integrating four times by parts one gets

$\displaystyle|\beta_{k}|\cong\frac{|\omega_{k}^{\prime\prime\prime\prime}(\tau_{kin}^{-})-\omega_{k}^{\prime\prime\prime\prime}(\tau_{kin}^{+})|}{32\omega^{5}_{kin}},$
(42)

where the beginning of kination $\tau_{kin}$ corresponds to the value of the
scalar field $\varphi=0$.

And, since
$|\omega_{k}^{\prime\prime\prime\prime}(\tau_{kin^{-}})-\omega_{k}^{\prime\prime\prime\prime}(\tau_{kin^{+}})|=\frac{a^{6}_{kin}|\dddot{H}(\tau_{kin}^{-})-\dddot{H}(\tau_{kin}^{+})|m_{\chi}^{2}}{\omega_{kin}}$,
one gets

$\displaystyle|\beta_{k}|\cong\frac{a_{kin}^{6}|\dddot{H}(\tau_{kin}^{-})-\dddot{H}(\tau_{kin}^{+})|m_{\chi}^{2}}{32\omega^{6}_{kin}}.$
(43)

Now, taking the temporal derivative of the Raychaudhuri equation we have
$\ddot{H}=\dot{\varphi}\ddot{\varphi}/M_{pl}^{2}$, and thus

$\displaystyle|\dddot{H}(\tau_{kin}^{-})-\dddot{H}(\tau_{kin}^{+})|=\frac{\dot{\varphi}_{kin}}{M_{pl}^{2}}|\dddot{\varphi}(\tau_{kin}^{-})-\dddot{\varphi}(\tau_{kin}^{+})|.$
(44)

On the other hand, from the conservation equation we have

$\displaystyle|\dddot{\varphi}(\tau_{kin}^{-})-\dddot{\varphi}(\tau_{kin}^{+})|\cong
V_{\varphi\varphi}(0^{+})\dot{\varphi}_{kin}=m^{2}\dot{\varphi}_{kin},$ (45)

so

$\displaystyle|\dddot{H}(\tau_{kin}^{-})-\dddot{H}(\tau_{kin}^{+})|=\dot{\varphi}_{kin}^{2}m^{2}/M_{pl}^{2}.$
(46)

Finally, at the end of inflation we have $\varphi_{END}=\sqrt{2}M_{pl}$ and
$H_{END}^{2}=\frac{3V(\varphi_{END})/2}{3M_{pl}^{2}}=m^{2}/2$. Assuming that
there is no substantial drop of energy between the end of inflation and the
beginning of kination $m/\sqrt{2}=H_{END}\cong H_{kin}$ since during kination
all energy is kinetic, one gets $\dot{\varphi}_{kin}\cong\sqrt{3}mM_{pl}$,
which leads to

$\displaystyle|\dddot{H}(\tau_{kin}^{-})-\dddot{H}(\tau_{kin}^{-})|\cong
3m^{4}.$ (47)

Unfortunately for the model (39), as we can see in Figure 1, the numerical
results differ in $8$ orders of magnitude with respect to the analytical ones.

Figure 1: Plot of the analytical and numerical values of $|\beta_{k}|^{2}$ for
different values of $k$ obtained by solving equation (20) for
$m_{\chi}=10^{15}$ GeV. We can see that for values of $k$ greater than
$a_{kin}m_{\chi}$ (for this model we have checked numerically that choosing
$a_{*}=1$ one has $a_{kin}\sim 10^{28}$) the square modulus of the
$\beta$-Bogoliubov coefficient decreases very fast as happens with the
analytical solution. However, there is a difference of around $8$ orders
between both solutions.

To conclude this section, one can also calculate the energy density after the
beginning of kination for this model, obtaining

$\displaystyle\langle\rho(\tau)\rangle=\frac{315m^{8}}{8\pi\times
16^{4}m_{\chi}^{4}}\left(\frac{a_{kin}}{a(\tau)}\right)^{3}\cong 9\times
10^{-8}m^{4}\left(\frac{a_{kin}}{a(\tau)}\right)^{3},$ (48)

for masses of the order $10^{15}$ GeV.

###  III.2 Smooth models

Given a non-oscillating potential, for example those which appear in
Quintessential Inflation, in order to solve (10) first of all one has to
obtain the background, i.e., integrate numerically the conservation equation
for the inflaton field in terms of the cosmic time, namely

$\displaystyle\ddot{\varphi}+3H\dot{\varphi}+V_{\varphi}=0,$ (49)

where
$H=\frac{1}{\sqrt{3}M_{pl}}\sqrt{\frac{\dot{\varphi}^{2}}{2}+V(\varphi)}$,
with initial conditions at the horizon crossing, i.e., when the pivot scales
leaves the Hubble radius. Recall that in that moment the system is in the
slow-roll phase and, since this regime is an attractor, one only has to take
initial conditions in the basin of attraction of the slow-roll solution, for
example, $\varphi=\varphi_{*}$ and
$\dot{\varphi}=-\frac{V_{\varphi}(\varphi_{*})}{3H_{*}}$, where the “star”
denotes that the quantities are evaluated at the horizon crossing.

Once one has obtained the evolution of the background, and in particular the
evolution of the Hubble rate, one computes the evolution of the scale factor,
which is given by

$\displaystyle a(t)=a_{*}e^{\int_{t_{*}}^{t}H(s)ds},$ (50)

where the value of $a_{*}$ is arbitrary and can be chosen to be $a_{*}=1$.

More precisely, for the $\alpha$-attractor potential displayed in Figure 2,

$\displaystyle V(\varphi)=\lambda
M_{pl}^{4}e^{-n\tanh\left(\frac{\varphi}{\sqrt{6\alpha}M_{pl}}\right)},$ (51)

where $\lambda$, $\alpha$ and $n$ are dimensionless parameters which have to
satisfy the following relation in order to match with the current observation
data,

$\displaystyle\frac{\lambda}{\alpha}e^{n}\sim
10^{-10}\qquad\mbox{and}\qquad\lambda e^{-n}\sim 10^{-120},$ (52)

one has to choose the values of $\alpha$, for example $\alpha=10^{-2},10^{-1}$
and $1$, and for a given value of $\alpha$ from equation (52) one finds the
corresponding values of $\lambda$ and $n$.

Figure 2: Plot of the Exponential $\alpha$-attractor potential, for
$\alpha\sim 10^{-2}$, $n\sim 10^{2}$ and $\lambda\sim 10^{-66}$.

The value of $\varphi_{*}$ is obtained from the relations (see for details
haro )

$\displaystyle\epsilon_{*}=\frac{n^{2}}{12\alpha}\frac{1}{\cosh^{4}\left(\frac{\varphi_{*}/M_{pl}}{\sqrt{6\alpha}}\right)},\qquad\eta_{*}\cong-\frac{n}{3\alpha}\frac{1}{\cosh^{2}\left(\frac{\varphi_{*}/M_{pl}}{\sqrt{6\alpha}}\right)},$
(53)

with $\varphi_{*}<0$. And using the relation

$\displaystyle n_{s}\cong 1-6\epsilon_{*}+2\eta_{*}\cong 1-\frac{2}{N},\qquad
r\cong 16\epsilon_{*}\cong\frac{12\alpha}{N^{2}},$ (54)

one gets from the equation (54)

$\displaystyle\epsilon_{*}=\frac{3\alpha}{16}(1-n_{s})^{2},$ (55)

which, together with the expression of $\epsilon_{*}$ given in (53), leads to

$\displaystyle\cosh\left(\frac{\varphi_{*}}{\sqrt{6\alpha}M_{pl}}\right)=\sqrt{\frac{2n}{3\alpha(1-n_{s})}},$
(56)

whose solution is given by

$\displaystyle\varphi_{*}=\sqrt{6\alpha}M_{pl}\ln\left(\sqrt{\frac{2n}{3\alpha(1-n_{s})}}-\sqrt{\frac{2n}{3\alpha(1-n_{s})}-1}\right),\qquad
n_{s}\cong 0.96.$

In the same way, from the formula of the power spectrum of scalar
perturbations

$\displaystyle H_{*}^{2}=16\pi^{2}\epsilon_{*}10^{-9}M_{pl}^{2},$ (57)

and using $\epsilon_{*}=\frac{3\alpha}{16}(1-n_{s})^{2}$, one gets the value
of $H_{*}$,

$\displaystyle H_{*}\cong 6\sqrt{\alpha}\times 10^{-6}M_{pl}.$ (58)

Now, taking into account that

$\displaystyle V_{\varphi}(\varphi_{*})=-\frac{n\lambda
M_{pl}^{3}}{\sqrt{6\alpha}}\frac{e^{-n\tanh\left(\frac{\varphi_{*}}{\sqrt{6\alpha}M_{pl}}\right)}}{\cosh\left(\frac{\varphi_{*}}{\sqrt{6\alpha}M_{pl}}\right)}\cong-\frac{1}{2}\sqrt{\frac{3\alpha}{2}}(1-n_{s})e^{-n}M_{pl}^{3},$
(59)

where we have used the formula (56), we have

$\displaystyle\dot{\varphi}_{*}=-\frac{V_{\varphi}(\varphi_{*})}{3H_{*}}\cong\frac{1}{12}\sqrt{\frac{3}{2}}(1-n_{s})10^{6}e^{-n}M_{pl}^{2}.$
(60)

In Figure $3$ we have plotted the evolution of the scalar field $\varphi$. For
$\alpha=10^{-2}$ we have obtained $\varphi_{*}\cong-1.7M_{Pl}$,
$\varphi_{END}\cong-0.89M_{Pl}$, $\varphi_{kin}\cong-0.54M_{Pl}$, where, as we
have already used, the sub-index “END” denotes the end of inflation.

Figure 3: Evolution of the scalar field $\varphi$, when $\alpha\sim 10^{-2}$.
Figure 4: Adiabatic evolution for a heavy field with mass $m_{\chi}\sim
10^{15}$ GeV, when $\alpha\sim 10^{-2}$. Here we have used the value
$k=a_{kin}H_{kin}$.

Then, in haro1 ; haro2  the equation (10) is transformed into a second order
differential equation, namely

$\displaystyle\left\\{\begin{array}[]{cc}&\alpha_{k}^{\prime\prime}(\tau)=\alpha_{k}^{\prime}(\tau)\left(\frac{\omega_{k}^{\prime\prime}(\tau)}{\omega_{k}^{\prime}(\tau)}-\frac{\omega_{k}^{\prime}(\tau)}{\omega_{k}(\tau)}+2i\omega_{k}(\tau)\right)+\left(\frac{\omega_{k}^{\prime}(\tau)}{2\omega_{k}(\tau)}\right)^{2}\alpha_{k}(\tau)\\\
&\beta_{k}^{\prime\prime}(\tau)=\beta_{k}^{\prime}(\tau)\left(\frac{\omega_{k}^{\prime\prime}(\tau)}{\omega_{k}^{\prime}(\tau)}-\frac{\omega_{k}^{\prime}(\tau)}{\omega_{k}(\tau)}-2i\omega_{k}(\tau)\right)+\left(\frac{\omega_{k}^{\prime}(\tau)}{2\omega_{k}(\tau)}\right)^{2}\beta_{k}(\tau)\end{array}\right..$
(63)

Assuming that at the horizon crossing the quantum field is the vacuum, which
is equivalent to taking the initial conditions $\alpha_{k}(\tau_{*})=1$ and
$\beta_{k}(\tau_{*})=0$ leading to $\alpha_{k}^{\prime}(\tau_{*})=0$, one can
solve the equation for $\alpha_{k}(\tau)$, which can be split into the real
and imaginary parts in the following way,

$\displaystyle\left\\{\begin{array}[]{cc}&\alpha_{k,Re}^{\prime\prime}(\tau)=\alpha_{k,Re}^{\prime}(\tau)\left(\frac{\omega_{k}^{\prime\prime}(\tau)}{\omega_{k}^{\prime}(\tau)}-\frac{\omega_{k}^{\prime}(\tau)}{\omega_{k}(\tau)}\right)-2\omega_{k}(\tau)\alpha_{k,Im}^{\prime}(\tau)+\left(\frac{\omega_{k}^{\prime}(\tau)}{2\omega_{k}(\tau)}\right)^{2}\alpha_{k,Re}(\tau)\\\
&\alpha_{k,Im}^{\prime\prime}(\tau)=\alpha_{k,Im}^{\prime}(\tau)\left(\frac{\omega_{k}^{\prime\prime}(\tau)}{\omega_{k}^{\prime}(\tau)}-\frac{\omega_{k}^{\prime}(\tau)}{\omega_{k}(\tau)}\right)+2\omega_{k}(\tau)\alpha_{k,Re}^{\prime}(\tau)+\left(\frac{\omega_{k}^{\prime}(\tau)}{2\omega_{k}(\tau)}\right)^{2}\alpha_{k,Im}(\tau)\end{array}\right.,$
(66)

and then $|\beta_{k}(\tau)|^{2}=|\alpha_{k}(\tau)|^{2}-1$ because of the well-
known conservation property of the Wronskian. This is the last method that we
will present here.

Numerical calculations have been done in the case of the exponential
$\alpha$-attractor haro2

$\displaystyle V(\varphi)=\lambda
M_{pl}^{4}e^{-n\tanh\left(\frac{\varphi}{\sqrt{6\alpha}M_{pl}}\right)},$ (67)

where by solving numerically (66) we have obtained that, for masses
$m_{\chi}\cong 10^{15}-10^{17}$ GeV and for a large range of values of
$\alpha$, the modes which are in the range $a_{kin}H_{kin}\lesssim k\lesssim
a_{kin}m_{\chi}$ lead to values of $|\beta_{k}|^{2}$ of order $10^{-9}$ for
$m_{\chi}\sim 10^{15}$ GeV and values of $|\beta_{k}|^{2}$ of order $10^{-10}$
for $m_{\chi}\sim 10^{16}-10^{17}$ GeV.

However, the numerical results obtained solving the exact solution (66) (see
Appendix A for more details) do not seem very confident for many reasons:

  1. 1.

The method breaks for modes out of the range $a_{kin}H_{kin}\lesssim k\lesssim
a_{kin}m_{\chi}$.

  2. 2.

It has been impossible to obtain numerically $\beta_{k}$ using the approximate
formula (17). If the results obtained solving (66) are correct, it seems clear
that one has to obtain the same ones using the integral form (17).

  3. 3.

The same happens using the differential equation (20). We have tried to obtain
the same results solving (20), but we have not been able to solve numerically
this equation.

  4. 4.

The results solving numerically (66) for the smooth potential (67) lead to
values of $|\beta_{k}|^{2}$ greater than the analytical ones obtained from the
potential (39), whose second derivative has a discontinuity. This seems a
contradiction.

  5. 5.

Using this method we cannot reproduce the analytical results obtained from the
toy models, as one can see in Figure $5$.

Figure 5: Plot of the analytical and numerical values of $|\beta_{k}|^{2}$,
corresponding to the model (39), for different values of $k$ obtained by
solving equation (66) for $m_{\chi}=10^{15}$ GeV. We can see that there is no
relation between both ways to calculate the $\beta$-Bogoliubov coefficient.

##  IV Instant Preheating

When the reheating mechanism is due to the production of particles via Instant
Preheating, as we have already seen, the frequency is given by
$\omega_{k}(\tau)=\sqrt{k^{2}+a^{2}(\tau)[m_{X}^{2}+g^{2}(\varphi-\varphi_{kin})^{2}]}$,
and the standard way to obtain analytically the value of the
$\beta$-Bogoliubov coefficient goes as follows: First of all, we make the
approximation $\varphi-\varphi_{kin,max}\cong\varphi_{kin}^{\prime}\tau$,
where we have chosen $\tau_{kin}=0$, and we assume a static universe with
$a(\tau)=a_{kin}$. Then, the frequency becomes

$\displaystyle\omega_{k}^{2}(\tau)=k^{2}+a^{2}_{kin}[m_{X}^{2}+g^{2}(\varphi^{\prime}_{kin})^{2}\tau^{2}].$
(68)

Now we make the change of variable
$\tau=\frac{y}{\sqrt{a_{kin}g\varphi^{\prime}_{kin}}}$ to obtain the
differential equation

$\displaystyle\frac{d^{2}\varphi_{k}}{dy^{2}}+(\kappa^{2}+y^{2})\varphi_{k}=0,$
(69)

where we have introduced the notation
$\kappa^{2}=\frac{k^{2}+a_{kin}^{2}m_{X}^{2}}{a_{kin}g\varphi^{\prime}_{kin}}$.

The positive frequency modes in the WKB approximation are

$\displaystyle\phi_{k,+}(y)=\frac{1}{(\kappa^{2}+y^{2})^{1/4}}e^{-i\int\sqrt{\kappa^{2}+y^{2}}dy},$
(70)

and for large values of $y$ one can make the approximation
$(\kappa^{2}+y^{2})^{1/4}\cong|y|^{1/2}$ and
$\sqrt{\kappa^{2}+y^{2}}\cong|y|\left(1+\frac{\kappa^{2}}{2y^{2}}\right)$,
obtaining

$\displaystyle\phi_{k,+}(y\ll-\kappa)\cong|\tau|^{-1/2}\tau^{i\kappa^{2}/2}e^{i\tau^{2}/2},\qquad\phi_{k,+}(y\gg\kappa)\cong|\tau|^{-1/2}\tau^{-i\kappa^{2}/2}e^{-i\tau^{2}/2},$
(71)

and for the negative frequency modes one has

$\displaystyle\phi_{k,-}(y\gg\kappa)=\frac{1}{(\kappa^{2}+y^{2})^{1/4}}e^{i\int\sqrt{\kappa^{2}+y^{2}}dy}\cong|\tau|^{-1/2}\tau^{i\kappa^{2}/2}e^{i\tau^{2}/2}.$
(72)

So, the frequency positive modes evolve as

$\displaystyle\phi_{k,+}(y\ll-\kappa)\longrightarrow\alpha_{k}\phi_{k,+}(y\gg\kappa)+\beta_{k}\phi_{k,-}(y\gg\kappa).$
(73)

To obtain the Bogoliubov coefficients one can use the WKB method in the
complex plane integrating the frequency along the path
$\gamma=\\{z=|y|e^{i\alpha},\pi\leq\alpha\leq 0\\}$, obtaining that for
$y\gg\kappa$ the early time positive frequency modes evolve at late time as

$\displaystyle
e^{-\frac{\kappa^{2}}{2}\pi}|\tau|^{-1/2}\tau^{i\kappa^{2}/2}e^{i\tau^{2}/2},$
(74)

and comparing with (73) one gets

$\displaystyle|\beta_{k}|^{2}\cong
e^{-\kappa^{2}\pi}\qquad\mbox{and}\qquad|\alpha_{k}|^{2}=1+|\beta_{k}|^{2}=1+e^{-\kappa^{2}\pi}.$
(75)

Therefore, the energy density of the produced particles evolve as

$\displaystyle\langle\rho(t)\rangle=\frac{1}{2\pi^{2}a^{4}(t)}\int_{0}^{\infty}\omega_{k}(t)k^{2}|\beta_{k}|^{2}dk\cong\frac{m_{eff}(t)(g\dot{\varphi}_{kin})^{3/2}}{8\pi^{3}}e^{-\frac{\pi
m_{X}^{2}}{g\dot{\varphi}_{kin}}}\left(\frac{a_{kin}}{a(t)}\right)^{3},$ (76)

where $m_{eff}(t)=\sqrt{m_{X}^{2}+g^{2}(\varphi(t)-\varphi_{kin})^{2}}$ is the
effective mass of the $\chi$-particles.

It is clear that this result was obtained by making the approximations we have
depicted above, so it would be interesting to check if this result is also the
same without performing these approximations and solving directly the system
(10) using the methods described in the previous sections.

##  V The importance of $\beta_{k}$ in the calculation of the reheating
temperature

In this section we consider the gravitational particle production of
superheavy particles (in that case one can safely ensure that the vacuum
polarization effects do not affect the inflationary period) and also Instant
Preheating. In the second case, the bare mass of the produced particles
$m_{X}$ is assumed to be negligible, but soon after its creation they acquire
a very heavy effective mass which, as we have already seen in last section, is
given by $m_{eff}(t)=\sqrt{m_{X}^{2}+g^{2}(\varphi(t)-\varphi_{kin})^{2}}$.
So, in practice, in both mechanisms of reheating the masses are very heavy,
and thus, in order to have a plasma of relativistic particles, they have to
decay into lighter ones. Then, two different situations will arise, namely the
decay of these superheavy particles could be before or after the end of the
kination period, and the reheating temperature differs in both cases.

  1. 1.

Decay before the end of kination: In the case of Instant Preheating, in order
to evade an undesirable second inflationary period, it is mandatory that the
produced particles decay into lighter ones before the end of kination fkl .

First of all, it is important to realise that the energy density of the heavy
particles scales as $a^{-3}(t)$, and during kination the one of the inflaton
field scales as $a^{-6}(t)$. Since during kination the energy density of the
inflaton dominates, from the Friedmann equation one deduces that the Hubble
rate scales as $a^{-3}(t)$ during kination. We recall as well that the decay
finishes when the decay rate is of the same order as the Hubble rate, that is
when ${\Gamma}\sim H_{dec}=H_{kin}\left(\frac{{a}_{kin}}{a_{dec}}\right)^{3}$,
where $\Gamma$ denotes the decay rate and the sub-index $dec$ means that the
quantities are evaluated at the decay end.

Then, the corresponding energy densities at the decay end will be

$\displaystyle\rho_{\varphi,dec}=3{\Gamma}^{2}M_{pl}^{2}\qquad\mbox{and}\qquad\langle\rho_{dec}\rangle=\langle\rho_{kin}\rangle\left(\frac{{a}_{kin}}{a_{dec}}\right)^{3}=\langle\rho_{kin}\rangle\frac{\Gamma}{H_{kin}},$
(77)

where $\langle\rho_{kin}\rangle$ was already defined as the value of the
energy density (21) at the beginning of kination, and it is the quantity that
depends on the $\beta$-Bogoliubov coefficient.

Imposing that the end of the decay precedes the end of kination, which means
$\langle\rho_{dec}\rangle\leq\rho_{\varphi,dec}$, and taking into account that
it is after the beginning of the kination, i.e., $\Gamma\leq H_{kin}$, one
gets

$\displaystyle\frac{\langle\rho_{kin}\rangle}{3H_{kin}M_{pl}^{2}}\leq\Gamma\leq
H_{kin}.$ (78)

Finally, the reheating temperature, i.e., the temperature of the universe when
the relativistic plasma in thermal equilibrium starts to dominate, which
happens when $\rho_{\varphi,reh}\sim\langle\rho_{reh}\rangle$, can be
calculated as follows: Since after the decay the evolution of the respective
energy densities is given by

$\displaystyle\rho_{\varphi,reh}=\rho_{\varphi,dec}\left(\frac{a_{dec}}{a_{reh}}\right)^{6}\qquad\mbox{and}\qquad\langle\rho_{reh}\rangle=\langle\rho_{dec}\rangle\left(\frac{a_{dec}}{a_{reh}}\right)^{4},$
(79)

we will have
$\frac{\langle\rho_{dec}\rangle}{\rho_{\varphi,dec}}=\left(\frac{a_{dec}}{a_{reh}}\right)^{2},$
and thus, from the Stefan-Boltzmann law
$\rho_{reh}=\frac{\pi^{2}}{30}g_{reh}T_{reh}^{4}$, where $g_{reh}=106.75$ is
the effective number of degrees of freedom for the Standard Model, the
reheating temperature will be

$\displaystyle
T_{reh}=\left(\frac{30}{\pi^{2}g_{reh}}\right)^{1/4}\langle\rho_{reh}\rangle^{\frac{1}{4}}=\left(\frac{30}{\pi^{2}g_{reh}}\right)^{1/4}\rho_{\chi,dec}^{\frac{1}{4}}\sqrt{\frac{\langle\rho_{dec}\rangle}{\rho_{\varphi,dec}}}$
$\displaystyle\cong\left(\frac{10}{3\pi^{2}g_{reh}}\right)^{1/4}\frac{\langle\rho_{kin}\rangle^{3/4}}{M_{pl}^{2}H_{kin}^{3/4}\Gamma^{1/4}}M_{pl},$
(80)

which clearly depends on $\langle\rho_{kin}\rangle$, and thus on the
$\beta$-Bogoliubov coefficient.

  2. 2.

Decay after the end of kination.

When the decay of the $\chi$-field is after the end of kination (recall that
kination ends when $\rho_{\varphi}\sim\langle\rho\rangle$), one has to impose
${\Gamma}\leq H(\tau_{end})\equiv H_{end}$, where we have denoted by
$\tau_{end}$ the time at which kination ends. Taking this into account, one
has

$\displaystyle H^{2}_{end}=\frac{2\rho_{\varphi,end}}{3M_{pl}^{2}}$ (81)

and

$\displaystyle\rho_{\varphi,end}={\rho}_{\varphi,kin}\left(\frac{{a}_{kin}}{a_{end}}\right)^{6}=\frac{\langle{\rho}_{kin}\rangle^{2}}{{\rho}_{\varphi,kin}},$
(82)

where we have used that the kination ends when
${\langle{\rho}_{end}\rangle}={{\rho}_{\varphi,end}}$, meaning that
$\left({a}_{kin}/a_{end}\right)^{3}=\frac{\langle{\rho}_{kin}\rangle}{{\rho}_{\varphi,kin}}$.
So, the condition ${\Gamma}\leq H_{end}$ leads to the bound

$\displaystyle\Gamma\leq\sqrt{\frac{2}{3}}\frac{\langle\rho_{kin}\rangle}{M_{pl}\sqrt{\rho_{\varphi,kin}}}.$
(83)

On the other hand, assuming once again instantaneous thermalization, the
reheating temperature (i.e., the temperature of the universe when the
thermalized plasma starts to dominate) will be obtained when all the
superheavy particles decay, i.e. when $H\sim\Gamma$, obtaining

$\displaystyle
T_{reh}=\left(\frac{30}{\pi^{2}g_{reh}}\right)^{1/4}\langle\rho_{dec}\rangle^{1/4}=\left(\frac{90}{\pi^{2}g_{reh}}\right)^{1/4}\sqrt{{\Gamma}M_{pl}},$
(84)

where we have used that after the end of the kination regime the energy
density of the produced particles dominates the one of the inflaton field.

In that case the reheating temperature is only related with the decay rate
$\Gamma$, which has to satisfy the bound (83) which depends on
$\langle\rho_{kin}\rangle$, and thus, on the $\beta$-Bogoliubov coefficient.

##  VI Conclusions

The main goal of the present work is to point out the problems we have
encountered calculating numerically the $\beta$-Bogoliubov coefficient for
smooth models. In fact, we have presented several ways to calculate it
numerically, but none of them seems to work correctly in the sense that we
cannot ensure, for many reasons, that the obtained results are the correct
ones. For example, working with toy models where one can obtain an analytical
solution, the numerical methods do not lead to the same results and, even
worse, different numerical methods lead to different results.

Then, from our viewpoint, this is a problem that deserves complicated future
studies at numerical level, because, as we have already shown, the calculation
of the $\beta$-Bogoliubov coefficient in non-oscillatory models is essential
to obtain the reheating temperature of the universe. Maybe the problem is very
technical and deep, and so, numerical mathematics are needed, but our believe
is that it could be possible to get the right value of these coefficients with
great accuracy.

## Acknowledgments

We would like to thanks Prof. Jaume Amorós for his valuable comments. JdH is
supported by grant MTM2017-84214-C2-1-P funded by
MCIN/AEI/10.13039/501100011033 and by “ERDF A way of making Europe”, and also
in part by the Catalan Government 2017-SGR-247. L.A.S thanks the School of
Mathematical Sciences (Queen Mary University of London) for the support
provided.

##  Appendix A Numerical methods

In this appendix we will discuss the way in which we have performed the
numerical computations in order to calculate the Bogoliubov coefficient.

The procedure yields as follows:

  1. 1.

Solve numerically the continuity equation
$\ddot{\varphi}+3H\dot{\varphi}+V_{\varphi}=0$ in cosmic time with initial
conditions at the Hubble crossing.

This arises no problems, since it leads to the expected results given in Fig
$1$ in attractors .

  2. 2.

Compute the scale parameter $a(t)=e^{\int_{t_{0}}^{t}H(s)}$.

Given that the value of the Hubble constant has been calculated numerically
and $a(t)$ spans through more than $25$ orders of magnitude, this is something
that needs to be done more accurately. So far, we have tried to obtain $a(t)$
both by computing the above integral or as the solution of the simple ODE
$a^{\prime}(t)=H(t)a(t)$.

  3. 3.

Once having $a(t)$, solve the ODE (20) or (66) for the different range of
values of $k$ in order to obtain the value of the Bogoliubov coefficient.

We have used the ordinary NDSolve function in Mathematica trying to tun the
needed precision and accuracy as well as the Filon-type methods in Matlab, but
all have happened to lead to different results for the two sets of ODEs.

Our guess is that the step $2$ has not been successfully achieved and we are
using wrong values of $a(t)$ in step $3$.

## References

  * (1) A. Guth, Phys. Rev. D 23, 347 (1981).
  * (2) A. G. Riess et al, Astron. J. 116, 1009 (1998) [arXiv:astro-ph/9805201].
  * (3) S. Perlmutter et al, Astrophys. J. 517, 565(1999) [arXiv:astro-ph/9812133].
  * (4) P. J. E. Peebles and A. Vilenkin, Phys. Rev. D 59, 063505 (1999) [arXiv:astro-ph/9810509].
  * (5) L. Parker, Phys. Rev. Lett. 21, 562 (1968).
  * (6) L. H. Ford, Phys. Rev. D 35, 2955 (1987).
  * (7) G. Felder, L. Kofman and A. Linde, Phys. Rev. D 59, 123523 (1999) [arXiv:hep-ph/9812289]
  * (8) G. Felder, L. Kofman and A. Linde, Phys. Rev. D 60, 103505 (1999) [arXiv:hep-ph/9903350].
  * (9) T. Damour and A. Vilenkin, Phys. Rev. D 53, 2981 (1996) [arXiv:hep-th/9503149].
  * (10) M. Giovannini, Phys. Rev. D 58, 083504 (1998) [arXiv:hep-ph/9806329].
  * (11) T. S. Bunch, J. Phys. A 13, 1297 (1980).
  * (12) Ya. B. Zeldovich and A. A. Starobinsky, JETP 34, 1159 (1972).
  * (13) A. A. Grib, S.G. Mamayev and V. M. Mostepanenko, Friedmann Laboratory Publishing for Theoretical Physics, St. Petersburg (1994).
  * (14) Y. Ema, K. Nakayama and Y. Tang, JHEP 09, 135 (2018) [arXiv:1804.07471 [hep-ph]].
  * (15) S. Hashiba and J. Yokoyama, JCAP 01, 028 (2019) [arXiv:1809.05410 [gr-qc]].
  * (16) L. Aresté Saló, D. Benisty, E. I. Guendelman and J. de Haro, Phys. Rev. D 103, 123535 (2021) [arXiv:2103.07892 [astro-ph.CO]].
  * (17) L. Aresté Saló and J. de Haro, Phys. Rev. D 104, 083544 (2021) [arXiv:2108.10795 [gr-qc]].
  * (18) L. Aresté Saló and J. de Haro, [arXiv:2112.12992 [gr-qc]].
  * (19) L. Aresté Saló, D. Benisty, E.I. Guendelman and J. de Haro, Phys. Rev. D 103, 123535 (2021) [arXiv:2103.07892 [astro-ph.CO]]


# The AffectToolbox: Affect Analysis for Everyone

Silvan Mertes, Dominik Schiller, Michael Dietz, Elisabeth André and Florian
Lingenfelser  Human-Centered Artificial Intelligence, Augsburg University
Augsburg, Germany
<EMAIL_ADDRESS>
###### Abstract

In the field of affective computing, where research continually advances at a
rapid pace, the demand for user-friendly tools has become increasingly
apparent. In this paper, we present the _AffectToolbox_ , a novel software
system that aims to support researchers in developing affect-sensitive studies
and prototypes. The proposed system addresses the challenges posed by existing
frameworks, which often require profound programming knowledge and cater
primarily to power-users or skilled developers. Aiming to facilitate ease of
use, the _AffectToolbox_ requires no programming knowledge and offers its
functionality to reliably analyze the affective state of users through an
accessible graphical user interface. The architecture encompasses a variety of
models for emotion recognition on multiple affective channels and modalities,
as well as an elaborate fusion system to merge multi-modal assessments into a
unified result. The entire system is open-sourced and will be publicly
available to ensure easy integration into more complex applications through a
well-structured, Python-based code base - therefore marking a substantial
contribution toward advancing affective computing research and fostering a
more collaborative and inclusive environment within this interdisciplinary
field.

###### Index Terms:

Affective Computing, Social Signal Analysis, Open Source, Emotion Recognition,
Affect Recognition, Multi-modal Fusion, Pleasure, Valence, Arousal, Dominance

##  I Introduction

_Affective computing_ describes a field of research, which incorporates the
recognition, interpretation and simulation of human emotions in software
systems. In particular, the _AffectToolbox_ aims at the automated recognition
of affective states - a sub-discipline of _machine learning_ and _artificial
intelligence_ research. As part of this ever-evolving realm, affect
recognition is characterized by rapid pace of new developments and a
continuous pushing of boundaries in research. The exploration of affective
computing often involves addressing recurring tasks such as interpreting
facial expressions, analysing speech prosody and sentiment or observing non-
verbal behaviour in body movements and poses. However, despite the growing
interest and demand in this field, existing tools and frameworks (e.g.
Microsoft’s _Platform for Situated Intelligence_[32] or Google’s
_MediaPipe_[25]) present challenges for swift development, as they require in-
depth programming knowledge and are primarily aimed at power users or
experienced developers.

Even within more accessible systems, a discernible gap exists, as most of them
are focusing predominantly on providing the input modalities and signals,
while lacking essential built-in functionality for comprehensive affect
analysis and recognition. This limitation underscores the need for more
versatile, accessible, and comprehensive architectures that cater to a broader
audience, including researchers and practitioners without extensive
programming backgrounds but high interest and demand for most recent
solutions. Addressing these challenges is pivotal to advancing affective
computing research and fostering a more inclusive and collaborative
environment in this dynamic and interdisciplinary field.

To close this gap, in this paper, we present the _AffectToolbox_ , a
comprehensive affect recognition software system,
that is

  * •

Easy to use. No programming knowledge required - all necessary functionality
can easily be controlled by a graphical user interface.

  * •

Comprehensive. A variety of models for interpreting accessible affective
channels is included.

  * •

Easy to integrate. The _AffectToolbox_ can easily be integrated into more
complex applications, using its built-in networking functionality, allowing
for a seamless connection to proprietary software implementations.

  * •

Open source. The whole software system and source code will be made publicly
available.111https://github.com/hcmlab/AffectToolbox

Figure 1: The modular architecture of the _AffectToolbox_ (Section III). In
its current state, audiovisual sensory devices (e.g. webcams) provide easy
means to generate all considered data streams (i.e. audio, transcript, video
and skeleton data). So called activity checks trigger the machine learning
based analysis of respective modalities (Section III-D). The uni-modal results
of applied affect recognition models are represented by a subset of pleasure,
arousal and/or dominance scores. These unimodal emotional cues are the input
for an event-driven fusion algorithm (Section III-E), which deduces a coherent
affective state, represented in the continuous PAD emotional space (Section
III-C).

##  II Related Work

To give a foundation for the conceptualization and development of the
_AffectToolbox_ , in the following we give a brief overview on existing
software that addresses related goals.

###  II-A Multi-modal Sensor Integration and Synchronization

_Apache Flink_ [14] is an open-source stream processing and batch processing
framework for big data processing and analytics. It is designed to efficiently
process large volumes of data in real-time and batch processing modes. _Flink_
provides a programming model and runtime for the processing of data streams in
a distributed and fault-tolerant manner.

_SiAM-dp_ [15] serves as a platform dedicated to the development of multi-
modal dialogue systems, with a key emphasis on effortlessly integrating
dispersed input and output devices within the realm of cyber-physical
environments.

The _Social Signal Interpretation Framework_ (_SSI_) [12] is an efficient
framework that offers robust support for diverse sensor devices, filter and
feature algorithms, and provides an extensive C++ API to add further
functionality and train custom machine learning models. Meanwhile, for end
users, _SSI_ provides accessibility through an XML interface. To also be
accessible on mobile devices, Damian et al. ported the core functionality of
_SSI_ to a Java-based framework which they called _SSJ_ [20].

Barz et al. introduced the _Multisensor Pipeline_ (_MSP_) [31], a Python
framework that serves as lightweight tool for prototyping multi-modal sensor
pipelines. A similar goal was followed by Saffaryazdi et al. [34], who
introduced a Python framework called _Octopus Sensing_ \- however, in contrast
to _MSP_ , which focuses more on real-time applications, _Octopus Sensing_
specifically addresses multi-modal data collection.

To the best of the authors’ knowledge, all of the frameworks mentioned above,
although providing comprehensive functionality to track and synchronize a
multitude of different sensor hardware, do not provide pre-trained machine
learning models, which allow for an immediate in-depth affective analysis of
the modalities. They are limited to providing an efficient and extensible
infrastructure for further computations (which may include the definition and
training of custom classification models).

###  II-B Multi-modal Signal Analysis

Due to the ongoing growth of the field, some large companies have also
recognised the economic potential of according systems recently. As such,
efforts were taken to release software that also incorporates ready-to-use
machine learning models. For example, Google released their _MediaPipe_
framework in 2019. _MediaPipe_ is an open-source framework developed by Google
that provides a comprehensive solution for building machine learning-based
applications for various multimedia tasks. It is designed to facilitate the
development of applications that involve real-time processing of audio, video,
and other sensor data [25]. _MediaPipe_ also includes a set of pretrained ML
models, e.g., models for object detection, image segmentation or face
detection. Microsoft developed the _Platform for Situated Intelligence_
(_psi_) [32] \- a versatile and open framework designed for development of
integrative AI systems that leverage multi-modal capabilities while also
incorporating various ML models for tasks such as voice activity or mouth
shape detection.

Although those systems offer a plenitude of possibilities, they are, due to
their industry-driven origin, designed to target a broad range of developers.
As such, when building an application for a specific use case, still much
effort has to be invested. Besides that being a time-consuming factor,
proficient programming knowledge is required to build the respective
applications, which often can be a hindering factor - especially if the system
is to be used in research projects where no computer science background is
represented at all.

##  III System Architecture

###  III-A Overview

The _AffectToolbox_ consists of a variety of independent components and
modules that communicate to each other in a queue-based, multi-threaded
runtime (Figure 1). Depending on how the components interact with the queue
system, we categorize them into:

  * •

Input components: Receive data from outside the framework (e.g., camera,
microphones) and write it into at least one queue.

  * •

Processing components: Read data from at least one queue, process it in
various ways, and write new data in at least one queue (e.g., preprocessing
algorithms, machine learning models, etc.)

  * •

Output components: Read data from at least one queue, but as a data sink do
not write new data to internal queues (e.g., GUI components, network adapters,
etc.)

As such, components can be easily added by defining an input queue that the
new component should load its data from, and/or an output queue that the new
component should write its data to, as well as the functionality that the
component should serve. Note that each module can process data in its own
frequency - the respective frequency and other component-specific parameters
are interfaced in a way that they can be adjusted via code or directly in the
graphical user interface (GUI).

Figure 2: User interacting with a virtual agent, whose emotional behaviour is
taking real-time input from the _AffectToolbox_ into consideration.

###  III-B Interface Design

The GUI design of the _AffectToolbox_ is crucial for its accessibility across
fields. We have carefully designed it in a way that basic and important
functions are easily visible, avoiding confusion for new users by keeping
unnecessary details out of the main design. However, note that all necessary
configuration parameters can be adjusted in respective context menus in
greater detail - as such, addressing the needs for slightly more sophisticated
use cases as well. In the GUI, all modalities and potential data flows are
always on displays. Specific modalities can be activated with a simple click,
triggering all necessary connections and dependencies automatically. Visually
highlighting the active data flow ensures that users see ongoing interactions
in real-time. The GUI can be seen in Figure 2, where the _AffectToolbox_ is
used to steer the emotional behaviour of a virtual agent.

###  III-C Emotion Model

For the implementation of the _AffectToolbox_ , a common representation of the
emotional user state has to be chosen. All included classification models need
to recognize towards a respective affective golden standard. Also, the result
is communicated to the user and / or subsequent applications in the chosen
format.

A common choice for automatic affect recognition are pre-selected categories
or labels. Categorical emotion models subsume affective states under discrete
categories like happiness, sadness, surprise or anger. This bears the
advantage, that there is a wide and common understanding of these discrete
categories of basic emotions [2].

Usage of discrete labels is however restricting, as many blended feelings and
emotions cannot adequately be described by the chosen categories. A more
precise way to describe emotions is to describe the experienced stimuli with
the help of continuous scales within dimensional models: Lang et al. [4]
suggest to characterize emotions along two continuous axes, pleasure and
arousal. Mehrabian [3] proposes dominance as an additional measurements (_PAD_
model). The pleasure scale describes the pleasantness of a given emotion. A
high value indicates an enjoyable emotion such as joy or happiness, lower
values are associated with negative emotions like sadness and fear. The
arousal scale measures the agitation level of an emotion. Dominance further
refines the model by adding assessment of the feelings of being in control of
a situation as well as autonomy.

Figure 3: The three dimensional _PAD_ model of affect. Continuous pleasure,
arousal and dominance scores describe affective states, from which discrete
emotion labels can be derived.

These continuous representations are less intuitive but allow continuous
blending between affective states. They describe multiple aspects of an
emotion, the combination of stimuli’s alignments on these scales defines
single emotions (Figure 3), which makes a conversion from dimensional values
to emotional labels possible. These characteristics make the dimensional _PAD_
model our choice for handling the affective golden standard within the
_AffectToolbox_.

###  III-D Modalities

The _AffectToolbox_ applies signal processing and deep learning methods to
interpret recorded signals by learning a mapping between observed signals and
affective states. In uni-modal affect classification, information from one
social channel,e.g. vocal properties, are used to make assumptions about the
current emotional condition of a user. But as the cues that describe emotional
conditions are indeed encoded within multiple modalities, the classification
process should incorporate as much multi-modal information as possible [9].

####  III-D1 Face Analysis

Facial expressions are considered the most expressive transmitter of human
emotions. Concentration and training would be needed to mask the depiction of
affect in one’s face reliably. But even then it takes a certain amount of time
to control the muscle reactions that were triggered by the underlying
emotional state and the correct emotion is expressed during this short period
[7]. Analysis of facial expressions has greatly benefited from advances in
image processing and resulted in reliable affect recognition models.

##### Camera Component

Being able to connect to a variety of different camera hardware setups is
essential for the accessibility of the toolbox. As such, we rely on OpenCV’s
camera interface solutions.222https://opencv.org/ By doing so, we ensure
compatibility with the majority of existing hardware, as such supporting both
standard and specialized use cases. According to a specified sample rate, our
camera component takes an image frame and writes it to the respective input
queue.

##### Face Preprocessing

Before passing the camera data on to our machine learning models, we process
them with two preprocessing components. First, we use the
MediaPipe333https://github.com/google/mediapipe face detection solution to
extract the users’ face. If a face is found, a bounding box is placed around
it, which is used to crop the image such that only face-relevant information
remains. If no face is found, the image is passed on as-is. Subsequently, we
normalize the images with min-max normalization ($min=-1.0,max=1.0$) and
resize them to $224x224$ pixel.

##### Face Activity Detection

We apply MediaPipe’s face mesh detection solution to the preprocessed camera
data. The resulting face landmarks have two major benefits:

  * •

First, if no face mesh is detected at all, it is likely that no user is in the
tracking area of the camera. Having that information, we can ignore the face
modality for the respective timeframes.

  * •

Second, by analyzing the landmark coordinates, we can obtain information on
whether the user is actually facing the camera and/or screen. Depending on the
application context, we can use that information to appropriately downweigh
the face modality.

##### Deep Facial Analysis

For the analysis of facial expressions we use a custom model that is based on
the MobileNetV2 architecture [22]. Its modifications include the addition of
two model heads to enable the recognition of continuous valence/arousal values
and the detection of eight discrete emotion classes (_Neutral, Happy, Sad,
Surprise, Fear, Disgust, Anger_ and _Contempt_). The reasoning behind these
choices was to improve the recognition accuracy while maintaining a
lightweight architecture for faster processing [30]. During the training
procedure, both tasks were learned simultaneously from the AffectNet dataset
[26]. Afterwards, the secondary model head for discrete emotion detection was
discarded as it only served to stabilize the primary task. The resulting model
uses the preprocessed camera data as input and provides predictions for
valence/arousal values as output.

Figure 4: Based on empirical studies, a dominance level can be derived from
facial expressions described in the valence/arousal space or inferred
emotional labels.

While there exist public corpora with reliable valence and arousal
annotations, the dominance dimension is less covered by respective resources.
To also derive a dominance score from the facial modality, we chose a
literature-based approach: Studies such as [1] describe observed relations
between the three scales of the PAD model as well as emotional labels. Based
on these insights, one can infer a perceived dominance level from recognized
facial expressions described in the valence/arousal space (or inferred
emotional label).

####  III-D2 Pose Analysis

Although early work like Caridakis et al. [5] describes the possibilities of
expressivity features (mainly calculated on arm and head movements extracted
from video sequences), the main assumption was that body movement only shows
the intensity of emotions. However, in recent years studies have shown that
dynamic body movement and gestures as well as static postures convey affective
states of a monitored person. Affective concepts such as a person’s current
expressivity [10] or engagement in a conversation [17] can be reliably
described by suitable movement features, body orientation, and posture.
Spatial expansive poses and postures including e.g. a straightened spine are
used to convey dominance and self-confidence and (though a problem of
replicating respective studies persists) there are clear hints towards effects
on self perception and emotional experience [29].

##### Skeleton Detection

The actions and positions of body, head, and limbs - also referred to as
_kinesics_ \- can be derived from detected pose landmarks in images. As a
reliable detection model, we chose to integrate the lightweight convolutional
neural network BlazePose [28], with which we can define a skeleton overlay for
each video frame to be further processed into pose features.

##### Pose Features

A logical consequence of this rising interest in kinesics is the effort to
develop a reliable coding system for body movement in emotion expression, just
like the by-now-available standards for facial expressions. For this reason,
Dael et al. [11] describe the Body Action and Posture System: The system
distinguishes body posture units and body action units [6] with the first
representing the general alignments of trunk, head and limbs to a resting
configuration (e.g. arms crossed) and the latter one describing a local and
short termed movement of head or arms (e.g. pointing gesture). We went for a
comparable rule-based approach by deducing a perceived dominance score based
on pose features calculated from the detected pose landmarks (skeleton).
Features by now include head and body tilts as well as overall activation of a
user. Further features are continuously developed.

####  III-D3 Voice Analysis

Human language encodes emotional information in a semantic as well as in a
paralinguistic way [16]. Paralinguistics refers to phenomena that accompany
speech and do not consist of linguistic units such as sounds, words,
sentences, etc., but give it an additional communicative aspect. This includes
acoustic characteristics such as pitch, volume or speaking speed.Figure 5
shows the audio signal of a person’s normal speech leading into an affective
burst of laughter. The characteristics of the signal parts can be well
differentiated. Semantics on the other hand describes the content and the
grammatical format of utterances, including the arrangement and choices of
words, phrases, and clauses. Although modern language analysis models show
promising results in analyzing both aspects of language within a unified
architecture [37], it is still advantageous to analyze paralinguistics and
semantics separately for optimal performance and flexibility.

Figure 5: Audio signal of a person’s normal speech leading into an affective
burst of laughter. The characteristics of the signal parts can be well
differentiated. Further processing of the signal carves out detailed
differences and enables the categorization of more subtle affective states.

##### Microphone Component

To track the user’s speech, we use
PyAudio444https://github.com/CristiFati/pyaudio. We chose PyAudio because it
enables us to directly connect to a multitude of different hardware setups and
allows for a large variety of different technical configuration options. The
tracked audio data is written into the respective input queue in chunks of
predefined size. As such, although we can keep the communication frequency
with the microphone comparably low, we still have the whole audio signal in
the queue system without any quality losses. By tweeking the chunk size and
grabbing frequency, the application-dependent trade-off between low latency
and resource efficiency can be controlled if necessary.

##### Voice Activity Detection

For affective voice analysis, it is crucial to know when the tracked audio
signals contain actual speech and not only background noises. As such, we make
use of the WebRTC voice activity detector555https://github.com/wiseman/py-
webrtcvad. We feed the audio by using a sliding window, resulting in
continuous probability estimations for the presence of voice. We can use those
probabilities to downweigh or even completely ignore all voice-related
modalities.

##### Paralinguistic Analysis

Paralinguistic analysis refers to the analysis of voice properties unrelated
to speech’s linguistic content. As such, characteristics like intonation,
pitch or loudness play an important role here. However, in recent years,
performing paralinguistic analysis in an end-to-end manner, i.e., without
handcrafting specific features, is becoming more and more standard[23]. As
such, for the _AffectToolbox_ , we use a current state-of-the-art network
architecture for assessing pleasure, arousal, and dominance from voice[37].
The model is based on the popular wav2vec2 architecture[27] and has proven to
obtain high performance on standard emotion recognition benchmark tasks[37].

####  III-D4 Sentiment Analysis

When analyzing emotions from spoken language, the paralinguistic analysis of
speech is complemented by the semantic analysis of the spoken content. While
the employed voice analysis model (III-D3) is already incorporating linguistic
information to some extent [35], we argue that the integration of a dedicated
sentiment analysis module is still adding value to the overall pipeline.

##### Speech-To-Text

In the preceding step, we must first convert the spoken language into a
textual representation that can be processed by a sentiment analysis model. To
this end we rely on the whisper model suite by Radford et el. [36] The model
suite consists of an off-the-shelf encoder-decoder Transformer architecture as
proposed by [19] at different scales. The novelty of the proposed work lies in
the introduction of a weakly supervised multitask training procedure that
relies heavily on large-scale data crawled from the web. The authors found
that their proposed training procedure enables the model to perform many
speech processing pipeline tasks like multilingual speech recognition, speech
translation, spoken language identification, and voice activity detection at
the same time while also enabling the resulting models to be competitive with
other supervised models but in a zero-shot transfer setting, i.e. on data for
which the model has not seen any examples. All in all, these features make the
model an ideal fit for the _AffectToolbox_.

##### Sentiment

To analyze the sentiment of a spoken utterance we rely on the XLM roBERTa
model from Barbieri et al. [33] The roBERTa model is a pretrained variant of
the BERT [21] architecture introduced by Liu et al. [24] that relies upon an
optimized training procedure to improve results. The XLM roBERTa multilingual
model has been further trained on a vast amount of Twitter data and provides
support for sentiment analysis in eight different languages. The authors chose
the model architecture and datasets specifically to make it suitable for
agent-human-interaction scenarios, which is also an ideal scenario for the
_AffectToolbox_

###  III-E Fusion

Including multiple modalities in the affect recognition process is generally
meant to enhance performance. In addition to the increase in information, the
system as a whole becomes more robust: Modalities that temporarily - due to
tracking failures or general lack of activity - do not contribute meaningful
information can be substituted by other channels. The most impactful factor
for the quality of a multi-modal affect recognition system is the ability to
extract informative features and results from its single modalities (Section
III-D). The fusion strategy to integrate this information into a coherent
decision is however of equal importance for the design of the _AffectToolbox_.

####  III-E1 Requirements

A key distinction between considered fusion approaches relates to the modeling
of temporally shifted occurrences of emotional cues throughout multiple
affective channels. While simple fusion strategies mainly apply a synchronous
strategy, which considers all information within a fixed time segment, more
elaborate systems regard these asynchronous characteristics of emotional
manifestations and try to model them within the fusion process. Multi-modal
affective events, such as facial expressions or vocal bursts that manifest
certain emotional states, are expected to occur at shifted points in time and
we consequently need to asynchronously treat modalities. We can therefore
formulate key requirements for the _AffectToolbox_ fusion algorithm.

Temporal Flow: If we recognize an affective cue in a modality, it enters the
fusion process and influences the continuous result. The initial influence
will diminish over time until the cue is ´no longer relevant and gets
discarded. Current cues therefore have a stronger impact on the fusion process
than the ones that lie further down the time axis.

Reinforcement and Attenuation: If complementary cues are detected in
overlapping time-segments, they reinforce each other by amplifying their
impact on the continuous fusion output. On the other hand, contradictory cues
neutralize each other and therefore have a lesser negative and attenuating
effect on the fusion result. This way, additional information from multiple
modalities is more likely to enhance the overall classification performance.

Real-Time Fusion Result: The _AffectToolbox_ targets applications, that are
reactive to current emotional states of a user. We consequently aim for near
real-time recognition tasks on online input, and need the processing speed of
all all algorithms to be suitable for this task. The result of the fusion
scheme is calculated by temporal influences (expressed through diminishing
weights) of registered cues and a current fusion result has to be available at
any given point in time, guaranteeing access to the latest affective
estimation for all subsequent components of an application.

Figure 6: The fusion result is calculated from active events in fixed update
steps: At each time frame, the influence of active event vectors is reduced
based on the defined decay speed, expired lifetime, and the initial norm of
the vector and weight of the event. Within the three-dimensional PAD space,
the fusion result is drawn towards the center of mass specified by weighted
event vectors.

####  III-E2 Implementation

We based the implementation of our fusion mechanism on preceding work done by
Gilroy et al. [8], which represents emotions as a vector within a dimensional
emotion model. We generalized this approach by designing an event-driven
fusion scheme that operates in a user-defined vector space [13], [18].

Affective cues, recognized in observed modalities, are registered as events
for an event-driven algorithm, which continuously calculates its multi-modal
fusion result based on the currently active affective events (Figure 6). As
the _AffectToolbox_ is operating with the pleasure-arousal-dominance emotion
model, respective events are represented as three-dimensional PAD vectors and
provided with several parameters: A score, which defines the position of the
vector within the dimensional PAD model, is assigned for each axis in the
event space. Each vector is given a weight parameter, which serves as a
quantifier for its impact on the calculation of the fusion result. This way
reliability of a modality or cue type can regulated. Finally, a decay speed
parameter describes the average lifespan of cues extracted from the respective
signal. It determines the time it takes for the event’s influence to fully
diminish and get discarded. Events with strong indications can be given longer
decay times to prolong their influence on the result.

Given these parameters, the fusion result can be calculated from active events
at each update step: At each time frame, active event vectors $e=1\ldots E$
are decayed by multiplying each vector element with a decay factor that is
calculated based on the defined decay speed, expired lifetime and the initial
norm of the vector:

$decay_{e}=norm_{e}-(lifetime_{e}*speed_{e})$ (1)

If the resulting norm of the decayed vector stays above zero, it remains
active - otherwise the vector is discarded. Afterwards, the fusion point
within the vector space is calculated from all active event vectors: For each
dimension $d=1\ldots D$ of the vector space respective scores of active event
vectors $e=1\ldots E$ (modified by their weight factor) are summed up.

$fusion\>point(d_{1\ldots
D})=\sum_{e=1}^{E}{(event\>vector_{d,e}*weight_{e})}$ (2)

The result is normalized by the sum of the weights of all contributing event
vectors.

$fusion\>point(d_{1\ldots D})=mass(d_{1\ldots D})/\sum_{e=1}^{E}{weight_{e}}$
(3)

The final result itself is a vector that approaches the calculated fusion
point within the PAD space with a predefined speed parameter (Figure 6). If no
events remain active in the vector space, the fusion vector approaches a
neutral state. The fusion vector serves as an additional means of smoothing:
If we can assume that the thought affective state is unlikely to undergo quick
changes, the vector can be defined to move slowly, making the algorithm more
robust to occasional misclassifications.

##  IV Discussion

We have by now seen how the _AffectToolbox_ can analyze multi-modal behaviour
and subsequently deduce the affective states of users. This capability offers
a wide range of possible applications but also features some technical
requirements and conceptual limitations that have to be taken into account.

###  IV-A Potential Use-Cases

Monitoring of affective user states is of great interest in theoretical
research scenarios as well as practical software design. Psychological studies
often include automated analysis and recording of proband behaviour. Several
instances of the _AffectToolbox_ can even be used to track emotional states of
several study participants (e.g. dyadic or multi-person conversations,
therapist and patient, etc.).

In most applications for which the _AffectToolbox_ was specifically designed,
the emotional analysis is used in a human-computer interaction (HCI) scenario,
i.e. the interaction between human users and a virtual agent. User emotion can
hereby serve as contextual information for the dialogue system to also include
the affective intent of the user in steering the conversational flow. Also,
the non-verbal behaviour of a virtual character can be designed in a very
natural and reactive manner if insights into the current emotional condition
of the human counterpart are available.

A further use-case in the field of HCI is the adaptation of systems to user
states. Techniques such as reinforcement learning can use e.g. PAD values as
additional input to the knowledge space or in the calculation of the reward
function. This way implicit feedback mechanisms greatly enhance the user
experience.

In all described fields of application, study and system designers can benefit
greatly from a comprehensive analysis tool, that is easy to set up and
integrate into the broader infrastructure without the need to allocate
resources to the implementation of the affect recognition pipeline. To this
end, the _AffectToolbox_ includes networking capabilities to broadcast its
results in several formats (JSON, XML, etc.) and protocols (UDP, Kafka, etc.).

###  IV-B Technical Requirements

To hold on to the ease-of-use paradigm, we try to keep the technical
requirements of the _AffectToolbox_ within reasonable limits: Audiovisual
sensory devices (e.g. webcams) provide easy means to generate all so far
considered data streams (i.e. audio, transcript, video, and skeleton data)
without reliance on expensive or complicated sensor hardware.

The same approach is followed for our reliance on computing power. The
_AffectToolbox_ features an extensive list of tuning parameters (e.g. analysis
frame rates, variable model sizes, etc.) to customize the system to the
capabilities of available hardware. Of course, the best performance with a
high frequency and accuracy of classification results is achieved with an up-
to-date hardware setup, we can nevertheless achieve acceptable results with
less costly and more available configurations. This is partly due to the
fusion process, which is inherently able to interpolate within a sparse result
space (Section III-E).

###  IV-C Limitations

Generally we consider the _AffectToolbox_ to be able to recognize so-called
_displayed_ emotions. This means, that - at least with the audiovisual
information used - there is no reliable way to differentiate between a genuine
feeling or socially altered affective state. Whether the displayed and
respectively measured emotions are felt or expressed for other reasons, like
using them as conversation techniques, resulting PAD values will be the same
from the system’s point of view.

Ways to investigate deeper into the actually felt emotion mostly include
context information and interpretation or additional physiological sensory
equipment. Physiological responses (e.g. heart rate, skin conductivity, and
respiration as well as eye gaze and pupil dilation) are more unconscious
reactions to affect that are hard to control by untrained subjects, as they
are regulated by the autonomous nervous system. Inclusion of this kind of
analysis would on the one hand contradict the accessability paradigm of the
_AffectToolbox_ , on the other hand might be worth the effort to optionally
offer further valuable user states such as stress, cognitive workload or user
engagement (Section V).

##  V Conclusion & Future Work

The _AffectToolbox_ is able to offer multi-modal recognition of displayed user
emotions in real-time. Results are given in fine-grained continuous pleasure-
arousal-dominance levels. In contrast to related solutions, it is easy to
apply, as no programming knowledge is required: A concise graphical user
interface provides accessible configuration and customization possibilities.
High-quality and ready-to-use recognition models are integrated into the
toolbox for each modality, a sophisticated fusion algorithm has been
implemented to handle the multi-modal affective cues and generate a coherent
result. Integration into application setups is easy to accomplish as fusion
results are continuously broadcasted and accessible at any point in time.

Several research projects are currently using the _AffectToolbox_ for the
development of affect-sensitive studies and prototypes. Given the multi-
disciplinary nature of these groups, we see numerous non-technical users
applying the toolbox without major complications. Acceptance of the system
underlines and reinforces the goal to provide an easy-to-handle solution to
integrate affective computing into state-of-the-art HCI systems.

The toolbox is in active development, which first means keeping the system up-
to-date with novel signal processing methods and machine learning models.
Second, we are investigating potential the demand for additional affective
states beyond PAD values. Offering insights into more medical and well-being-
related user states such as stress levels together with emotional states seems
to raise interest in potential user groups. Automatic measurement of
engagement in interactions as well as estimations of cognitive workload are
expected to be of great benefit for the adaptation of HCI systems to current
user needs, e.g. in training or educational scenarios.

## Acknowledgments

This research is funded by the Federal Ministry of Education and Research,
Project MITHOS (grand agreement 16SV8687) and Project UBIDENZ (grand agreement
13GW0568F).

## References

  * [1] James A Russell and Albert Mehrabian  “Evidence for a three-factor theory of emotions”  In _Journal of research in Personality_ 11.3 Elsevier, 1977, pp. 273–294
  * [2] Paul Ekman  “An argument for basic emotions”  In _Cognition & Emotion_ 6.3 Psychology Press, 1992, pp. 169–200
  * [3] A. Mehrabian  “Framework for a comprehensive description and measurement of emotional states.”  In _Genetic, social, and general psychology monographs_ 121.3, 1995, pp. 339–361
  * [4] Peter Lang, Margaret Bradley and B. Cuthbert  “Motivated attention: Affect, activation, and action”  In _Attention and orienting: Sensory and motivational processes_ Psychology Press, 1997, pp. 97–135
  * [5] G. Caridakis, A. Raouzaiou, K. Karpouzis and S. Kollias  “Synthesizing gesture expressivity based on real sequences”  In _LREC 2006 Conference_ , 2006
  * [6] Jinni Harrigan, Robert Rosenthal and Klaus Scherer  “New handbook of methods in nonverbal behavior research”  Oxford University Press, 2008
  * [7] P. Ekman  “The Philosophy of Deception: Lie Catching and Micro Expressions”  Ed. Clancy Martin, Oxford University Press, 2009
  * [8] Stephen Gilroy et al.  “PAD-based multimodal affective fusion”  In _International Conference on Affective Computing and Intelligent Interaction (ACII)_ , 2009  DOI: 10.1109/ACII.2009.5349552
  * [9] Zhihong Zeng, Maja Pantic, Glenn Roisman and Thomas Huang  “A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions”  In _IEEE Trans. Pattern Anal. Mach. Intell._ 31.1, 2009, pp. 39–58
  * [10] George Caridakis et al.  “A multimodal corpus for gesture expressivity analysis”  In _International Conference on Language Resources and Evaluation (LREC), Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality_ , 2010
  * [11] Nele Dael, Marcello Mortillaro and KlausR. Scherer  “The Body Action and Posture Coding System (BAP): Development and Reliability”  In _Nonverbal Behavior_ 36.2 Springer US, 2012, pp. 97–121  DOI: 10.1007/s10919-012-0130-0
  * [12] Johannes Wagner et al.  “The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time”  In _Proceedings of the 21st ACM international conference on Multimedia_ , 2013, pp. 831–834
  * [13] Florian Lingenfelser et al.  “An Event Driven Fusion Approach for Enjoyment Recognition in Real-time”  In _Proceedings of the ACM International Conference on Multimedia, MM’14, Orlando, FL, USA, November 03 - 07, 2014_ , 2014, pp. 377–386
  * [14] Paris Carbone et al.  “Apache flink: Stream and batch processing in a single engine”  In _The Bulletin of the Technical Committee on Data Engineering_ 38.4 Institute of ElectricalElectronics Engineers (IEEE), 2015
  * [15] Robert Neßelrath  “SiAM-dp: An open development platform for massively multimodal dialogue systems in cyber-physical environments”, 2015
  * [16] Johannes Wagner, Florian Lingenfelser and Elisabeth André  “Building a Robust System for Multimodal Emotion Recognition”  Wiley, 2015, pp. 379–410
  * [17] Tobias Baur, Dominik Schiller and Elisabeth André  “Modeling Users Social Attitude in a Conversational System”  In _Emotions and Personality in Personalized Services_ Springer, 2016, pp. 181–199
  * [18] Florian Lingenfelser et al.  “Asynchronous and Event-based Fusion Systems for Affect Recognition on Naturalistic Data in Comparison to Conventional Approaches”  In _IEEE Transactions on Affective Computing_ IEEE, 2016
  * [19] Ashish Vaswani et al.  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [20] Ionut Damian, Michael Dietz and Elisabeth André  “The SSJ framework: Augmenting social interactions using mobile signal processing and live feedback”  In _Frontiers in ICT_ 5 Frontiers Media SA, 2018, pp. 13
  * [21] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “Bert: Pre-training of deep bidirectional transformers for language understanding”  In _arXiv preprint arXiv:1810.04805_ , 2018
  * [22] M. Sandler et al.  “MobileNetV2: Inverted Residuals and Linear Bottlenecks”  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4510–4520  DOI: 10.1109/CVPR.2018.00474
  * [23] Johannes Wagner, Dominik Schiller, Andreas Seiderer and Elisabeth André  “Deep learning in paralinguistic recognition tasks: Are hand-crafted features still relevant?”, 2018
  * [24] Yinhan Liu et al.  “Roberta: A robustly optimized bert pretraining approach”  In _arXiv preprint arXiv:1907.11692_ , 2019
  * [25] Camillo Lugaresi et al.  “Mediapipe: A framework for building perception pipelines”  In _arXiv preprint arXiv:1906.08172_ , 2019
  * [26] A. Mollahosseini, B. Hasani and M.. Mahoor  “AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild”  In _IEEE Transactions on Affective Computing_ 10.1, 2019, pp. 18–31  DOI: 10.1109/TAFFC.2017.2740923
  * [27] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed and Michael Auli  “wav2vec 2.0: A framework for self-supervised learning of speech representations”  In _Advances in neural information processing systems_ 33, 2020, pp. 12449–12460
  * [28] Valentin Bazarevsky et al.  “BlazePose: On-device Real-time Body Pose tracking”, 2020  arXiv:2006.10204 [cs.CV]
  * [29] Robert Körner and Astrid Schuetz  “Dominance or prestige: A review of the effects of power poses and other body postures”  In _Social and Personality Psychology Compass_ 14, 2020  DOI: 10.1111/spc3.12559
  * [30] Dominik Schiller, Tobias Huber, Michael Dietz and Elisabeth André  “Relevance-based data masking: a model-agnostic transfer learning approach for facial expression recognition”  In _Frontiers in Computer Science_ 2 Frontiers Media SA, 2020, pp. 6
  * [31] Michael Barz et al.  “Multisensor-pipeline: a lightweight, flexible, and extensible framework for building multimodal-multisensor interfaces”  In _Companion Publication of the 2021 International Conference on Multimodal Interaction_ , 2021, pp. 13–18
  * [32] Dan Bohus et al.  “Platform for situated intelligence”  In _arXiv preprint arXiv:2103.15975_ , 2021
  * [33] Francesco Barbieri, Luis Espinosa Anke and Jose Camacho-Collados  “XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond”  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ Marseille, France: European Language Resources Association, 2022, pp. 258–266  URL: https://aclanthology.org/2022.lrec-1.27
  * [34] Nastaran Saffaryazdi, Aidin Gharibnavaz and Mark Billinghurst  “Octopus Sensing: A Python library for human behavior studies”  In _Journal of Open Source Software_ 7.71, 2022, pp. 4045
  * [35] Andreas Triantafyllopoulos et al.  “Probing Speech Emotion Recognition Transformers for Linguistic Knowledge”, 2022
  * [36] Alec Radford et al.  “Robust speech recognition via large-scale weak supervision”  In _International Conference on Machine Learning_ , 2023, pp. 28492–28518  PMLR
  * [37] Johannes Wagner et al.  “Dawn of the transformer era in speech emotion recognition: closing the valence gap”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ IEEE, 2023