#  Coherent states in microwave-induced resistance oscillations and zero
resistance states.

J. Iñarrea1,3 and G. Platero2,3 1Escuela Politécnica Superior,Universidad
Carlos III,Leganes, Madrid, 28911, Spain
2Instituto de Ciencia de Materiales, CSIC, Cantoblanco, Madrid, 28049, Spain.
3Unidad Asociada al Instituto de Ciencia de Materiales, CSIC, Cantoblanco,
Madrid, 28049, Spain.

###### Abstract

We investigate irradiated high-mobility two-dimensional electron systems
(2DES) under low or moderated magnetic fields. These systems present
microwave-induced magnetoresistance oscillations (MIRO) which, as we
demonstrate, reveal the presence of coherent states of the quantum harmonic
oscillator. We also show that the principle of minimum uncertainty of coherent
states is at the heart of MIRO and zero resistance states (ZRS). Accordingly,
we are able to explain, based on coherent states, important experimental
evidence of these photo-oscillations. Such as their physical origin, their
periodicity with the inverse of the magnetic field and their peculiar
oscillations minima and maxima positions in regards of the magnetic field.
Thus, remarkably enough, we come to the conclusion that 2DES, under low
magnetic fields, become a system of quasiclassical states or coherent states
and MIRO would be the smoking gun of the existence of these peculiar states is
these systems.

Introduction. The first idea of coherent states or quasiclassical states was
introduced by Schrödingersro  describing minimum uncertainty constant-shape
Gaussian wave packets of the quantum harmonic oscillator. They were
constructed by the quantum superposition of the stationary states of the
harmonic oscillator. These wave packets

Figure 1: Schematic diagrams of coherent states: The probability density of
the coherent state is a constant-shaped Gaussian distribution, whose center
oscillates in a harmonic potential similarly as its classical counterpart. The
lower part exhibits the 2D approach.

displaced harmonically oscillating similarly as their classical counterpartsro
. Later on, Glauberglau  applied the concept of coherent states to the
electromagnetic field being described by a sum of quantum field oscillators
for each field frequency or mode. These coherent states of electromagnetic
radiation introduced by Glauber are extensively used nowadays in quantum
optics. Coherent statesdodonov ; yurke ; noel ; dodonov2  are also an
essential and powerful tool in condensed matter when describing the dynamics
of quantum systems that are very close to a classical behaviour. One
remarkable example of this consists of one electron under the influence of a
moderate and constant magnetic field ($B$). The quantum mechanical solution of
this problem leads us to Landau states which are mere stationary states of the
quantum harmonic oscillator. Under low or moderate values of $B$, this system
can be described by an infinite superposition of Landau states, i.e., a
coherent state. The resulting wave packet oscillates classically at the
cyclotron frequency ($w_{c}$) inside the quadratic potential keeping constant
the Gaussian shape (see Fig. 1) and complying with the minimum uncertainty
condition.

The discovery of MIRO two decades ago led to a great deal of theoretical works
back then as the displacement model girvin , the inelastic modeldimi  and the
microwave-driven electron orbits modelina1 ; ina2 ; ina3 ; ina4 . According to
the latter, Landau states, under radiation, spatially and harmonically
oscillate with the guiding center at the radiation frequency ($w$) performing
classical trajectories. In this swinging motion electrons are scattered by
charged impurities giving rise to oscillations in the irradiated
magnetoresistance, i.e., MIRO.

In this letter we demonstrate that the electron dynamics and magnetotransport
in high-mobility 2DES is governed by the coherent states of the quantum
harmonic oscillator. In fact, we conclude that 2DES under low or moderate $B$
become a systems of coherent states and when irradiated, MIROmani ; zudov ;
ryzhii  bring to light the peculiar nature of these states. In other words,
irradiated coherent states of the quantum harmonic oscillator are at the heart
of MIRO. Accordingly, we incorporate the concept of coherent states to the
microwave-driven electron orbit modelina1 ; ina2 ; ina3 ; ina4 . Thus, a
remarkable obtained result is that the time, $\tau$ (evolution timecohen ), it
takes a scattered electron to jump between coherent states to give significant
contributions to the current has to be equal to the cyclotron period,
$T_{c}=2\pi/w_{c}$. For different values of $\tau$, the contribution turns out
negligible. This result holds in the dark and under radiation where $\tau$
will play an essential role. Thus, MIRO is mainly dependent on $\tau$ along
with $w$. $\tau$ also determines the peculiar $B$-dependent MIRO extrema
position and explain the periodicity of MIRO with the inverse of $B$. Thus,
MIRO finally reveals that coherent states of the quantum harmonic oscillator
are present in high-mobility 2DES when under low $B$ playing a lead role in
magnetotransport both in the dark and under radiation. On the other hand,
coherent states minimize the Heisenberg uncertainty principle and then, in our
model this would establish which states can be reached by scattering.

Figure 2: Schematic diagram of scattering process between coherent states
$\Psi_{\alpha}$ and $\Psi_{\alpha^{\prime}}$. The scattering is quasi-elastic.
The probability density for both coherent states is a constant-shaped Gaussian
wave packet. The process evolution time, $\tau$ is the cyclotron period. i.e.,
$\tau=2\pi/w_{c}=T_{c}$. $\Delta E$ is the energy difference between the
coherent states.

Theoretical model. We first obtain an expression for the coherent states of a
radiation-driven quantum harmonic oscillator. The starting point is the exact
solution of the time-dependent Schrödinger equation of a quantum harmonic
oscillator under a time-dependent force. This corresponds to the electronic
wave function for a 2DES in a perpendicular $B$, a DC electric field,
$E_{DC}$, and MW radiation which is considered semi-classically. The total
hamiltonian $H$ can be written as:

$\displaystyle H$ $\displaystyle=$
$\displaystyle\frac{P_{x}^{2}}{2m^{*}}+\frac{1}{2}m^{*}w_{c}^{2}(x-X(0))^{2}-eE_{dc}X(0)+$
(1) $\displaystyle+\frac{1}{2}m^{*}\frac{E_{dc}^{2}}{B^{2}}-eE_{0}\cos
wt(x-X(0))-$ $\displaystyle-eE_{0}\cos wtX(0)$ $\displaystyle=$ $\displaystyle
H_{1}-eE_{0}\cos wtX(0)$

where the corresponding wave function solution is given byina1 ; kerner ; park
:

$\Psi_{n}(x,t)=\phi_{n}(x-X(0)-x_{o}(t))e^{-iw_{c}(n+1/2)t}e^{\frac{i}{\hbar}\Theta(t)}$
(2)

where,

$\displaystyle\Theta(t)$ $\displaystyle=$
$\displaystyle\left[m^{*}\frac{dx_{o}(t)}{dt}x-\int_{0}^{t}{\it
L}dt^{\prime}\right]$ $\displaystyle+$ $\displaystyle
X(0)\left[-m^{*}\frac{dx_{o}(t)}{dt}x+\int_{0}^{t}{\it E_{0}\cos
wt^{\prime}}dt^{\prime}\right]$ (3)

$X(0)$ is the guiding center of the driven-Landau state, $E_{0}$ the MW
electric field intensity, $\phi_{n}$ is the solution for the Schrödinger
equation of the unforced quantum harmonic oscillator and $x_{0}(t)$ is the
classical solution of a forced harmonic oscillator:

$x_{0}(t)=\frac{eE_{o}}{m^{*}\sqrt{(w_{c}^{2}-w^{2})^{2}+w^{2}\gamma^{2}}}\sin
wt=A\sin wt$ (4)

where $\gamma$ is a phenomenologically-introduced damping factor for the
electronic interaction with acoustic phonons and ${L}$ is the classical
Lagrangian. Apart from phase factors, the wave function turns out to be the
same as a quantum harmonic oscillator (Landau state) where the center is
driven by $x_{0}(t)$. Thus, all driven-Landau states harmonically oscillates
in phase at the radiation frequency.

A coherent state denoted by $|\alpha\rangle$ is defined as the eigenvector of
the annihilation operator $\hat{a}$ with eigenvalue $\alpha$ and can be
expressed as a superposition of quantum harmonic oscillator statescohen ,

$|\alpha\rangle=\sum_{n}c_{n}(\alpha)|\phi_{n}\rangle=e^{-|\alpha|^{2}/2}\sum_{n}\frac{\alpha^{n}}{\sqrt{n!}}|\phi_{n}\rangle$
(5)

The coherent state $|\alpha\rangle$ can be also obtained with the displacement
operator $D(\alpha)$cohen  acting on the quantum harmonic oscillator ground
state $|\phi_{0}\rangle$, $|\alpha\rangle=D(\alpha)|\phi_{0}\rangle$, where
the unitary operator $D(\alpha)$ is defined by: $D(\alpha)=e^{\alpha
a^{\dagger}-\alpha^{\ast}a}$. The coherent state in the position
representation or wave function then reads, $\psi_{\alpha}(x)=\langle
x|D(\alpha)|\phi_{0}\rangle$. We observe, according to the obtained MW-driven
wave function (Eq. 2), that the irradiated Landau level structure remains
unchanged with respect to the undriven situation; same Landau level index and
energy. Then, we conclude that the system is quantized, in the same way as the
unforced quantum harmonic oscillatorpark . Thus, we can construct the driven-
coherent states based on driven-Landau states similarly as if they were
undrivencohen :

$\displaystyle|\psi_{\alpha}(x,t)\rangle$ $\displaystyle=$ $\displaystyle
e^{\frac{i}{\hbar}\Theta(t)}e^{-iw_{c}t/2}e^{-|\alpha|^{2}/2}$
$\displaystyle\times$ $\displaystyle\sum_{n}\frac{(\alpha
e^{-iw_{c}t})^{n}}{\sqrt{n!}}|\phi_{n}(x-X(0)-x_{o}(t))\rangle$

Now applying the displacement operator, we can calculate the wave function
corresponding to the coherent state of the MW-driven quantum oscillator:

$\displaystyle\psi_{\alpha}(x,t)=e^{\frac{i}{\hbar}\Theta(t)}e^{-iw_{c}t/2}\langle
x|D(\alpha)|\phi_{0}(x-X(0)-x_{o}(t))\rangle$
$\displaystyle=e^{\frac{i}{\hbar}\Theta(t)}e^{i\vartheta_{\alpha}}e^{-iw_{c}t/2}e^{\frac{i}{\hbar}\langle
p\rangle(t)x}\phi_{0}[x-X(0)-x_{o}(t)-\langle x\rangle(t)]$

where,

$\phi_{0}[x-X(0)-x_{o}(t)-\langle
x\rangle(t)]=\left(\frac{mw_{c}}{\pi\hbar}\right)^{1/4}e^{-\left[\frac{x-X(0)-x_{o}(t)-\langle
x\rangle(t)}{2\Delta x}\right]^{2}}$ (8)

$\langle x\rangle(t)$ and $\langle p\rangle(t)$ are the position and momentum
mean values respectivelycohen , $\langle
x\rangle(t)=\sqrt{\frac{2\hbar}{m^{\ast}w_{c}}}|\alpha_{0}|\cos(w_{c}t-\varphi)$
and $\langle p\rangle(t)=-\sqrt{2m^{\ast}\hbar
w_{c}}|\alpha_{0}|\sin(w_{c}t-\varphi)$ where we have used that
$\alpha=|\alpha_{0}|e^{-(iw_{c}t-\varphi)}$. $\Delta x$ is the position
uncertainty and the global phase factor,
$e^{i\vartheta_{\alpha}}=e^{\alpha^{\ast 2}-\alpha^{2}}$. Then, the wave
packet associated with $\Psi_{\alpha}(x,t)$ is therefore given by:

$|\Psi_{\alpha}(x,t)|^{2}=|\phi_{0}[x-X(0)-x_{o}(t)-\langle x\rangle(t)]|^{2}$
(9)

Thus, according to the above, the microscopic physical description of a high-
mobility 2DES under low or moderate $B$ would consist of constant-shaped
Gaussian wave packets harmonically displacing with $w_{c}$ in the undriven
case and with $w_{c}$ and $w$ under radiation.

To calculate the longitudinal magnetoresistance, $R_{xx}$, we first obtain the
longitudinal conductivity ${\sigma_{xx}}$ following a semiclassical Boltzmann
modelridley ; ando ; askerov ,

$\sigma_{xx}=2e^{2}\int_{0}^{\infty}dE\rho_{i}(E)(\Delta
X_{0})^{2}W_{I}\left(-\frac{df(E)}{dE}\right)$ (10)

being $E$ the energy, $\rho_{i}(E)$ the Landau states density of the initial
coherent state and $W_{I}$ is the electron-charged impurities scattering rate.
We consider now that the scattering takes place between coherent states of
quantum harmonic oscillators. Thus, $\Delta X_{0}$ is the distance between the
guiding centers of the scattering-involved coherent states.

We first study the dark case and according to the Fermi’s golden rule $W_{I}$
is given by,

$W_{I}=N_{i}\frac{2\pi}{\hbar}|<\psi_{\alpha^{{}^{\prime}}}|V_{s}|\psi_{\alpha}>|^{2}\delta(E_{\alpha^{{}^{\prime}}}-E_{\alpha})$
(11)

where $N_{i}$ is the number of charged impurities, $\psi_{\alpha}$ and
$\psi_{\alpha^{{}^{\prime}}}$ are the wave functions corresponding to the
initial and final coherent states respectively, $V_{s}$ is the scattering
potential for charged impuritiesando :
$V_{s}=\sum_{q}V_{q}e^{iq_{x}x}=\sum_{q}\frac{e^{2}}{2S\epsilon(q+q_{TF})}e^{iq_{x}x}$,
$S$ being the sample surface, $\epsilon$ the dielectric constant, $q_{TF}$ is
the Thomas-Fermi screening constantando  and $q_{x}$ the $x$-component of
$\overrightarrow{q}$, the electron momentum change after the scattering event.
$E_{\alpha}$ and $E_{\alpha^{{}^{\prime}}}$ stand for the coherent states
initial and final energies respectively.

The averaging on the impurities distribution has been considered in a very
simple approach following Askerovaskerov , Andoando  et. al., and J.
H.Daviesdavies . Thus, if the concentration of impurities is not too high, and
they are randomly distributed in the sample, the interferences caused by the
impurity centers can be neglected. Then, we have ignored those interreferences
and assume that the scattering due to each impurity is independent of the
others. As a result the total scattering is equal to the scattering rate for
one impurity center multiplied by the total number of impurities $N_{i}$.

The $V_{s}$ matrix element is given byridley ; ando ; askerov :

$|<\psi_{\alpha^{{}^{\prime}}}|V_{s}|\psi_{\alpha}>|^{2}=\sum_{q}|V_{q}|^{2}|I_{\alpha,\alpha^{{}^{\prime}}}|^{2}$
(12)

and the term $I_{\alpha,\alpha^{{}^{\prime}}}$ridley ; ando ; askerov ,

$I_{\alpha,\alpha^{{}^{\prime}}}=\int^{\infty}_{-\infty}\psi_{\alpha^{{}^{\prime}}}(x-X^{{}^{\prime}}(0)-\langle
x^{\prime}\rangle(t^{\prime}))e^{iq_{x}x}\psi_{\alpha}(x-X(0)-\langle
x\rangle(t))dx$ (13)

After lengthy algebra we obtain an expression for
$I_{\alpha,\alpha^{{}^{\prime}}}$,

$|I_{\alpha,\alpha^{{}^{\prime}}}|=e^{-\frac{[X^{{}^{\prime}}(0)-X(0)+\langle
x^{{}^{\prime}}\rangle(t^{\prime})-\langle x\rangle(t)]^{2}}{8(\Delta
x)^{2}}}e^{-\frac{q_{x}^{2}(t)2(\Delta x)^{2}}{4}}$ (14)

where $q_{x}(t)$is given by,

$\displaystyle q_{x}(t)$ $\displaystyle=$ $\displaystyle q_{x}+\sqrt{2m\hbar
w_{c}}/\hbar\left[|\alpha_{0}^{{}^{\prime}}|\sin(w_{c}t^{\prime})-|\alpha_{0}|\sin(w_{c}t)\right]$
$\displaystyle=$ $\displaystyle q_{x}+2\sqrt{2m\hbar
w_{c}}/\hbar|\alpha_{0}|\cos(w_{c}(t+\tau/2))\sin(w_{c}\tau/2))$

On the other hand,

$\langle x^{{}^{\prime}}\rangle(t^{{}^{\prime}})-\langle
x\rangle(t)\simeq\sqrt{\frac{2\hbar}{mw_{c}}}|\alpha_{0}|2\sin(w_{c}(t+\frac{\tau}{2})-\varphi)\sin(-w_{c}\frac{\tau}{2})$
(16)

where $t$ and $t^{{}^{\prime}}$ are the initial and final times for the
scattering event and $\tau$ is the evolution time between coherent states.
Thus, $t^{{}^{\prime}}=t+\tau$. We have considered also that for low values of
$B$, $|\alpha^{{}^{\prime}}_{0}|\simeq|\alpha_{0}|$. Developing the above
exponential we can finally get to,

$|I_{\alpha,\alpha^{{}^{\prime}}}|\propto
e^{-2|\alpha_{0}|^{2}\sin^{2}(w_{c}(t+\frac{\tau}{2})-\varphi)\sin^{2}(w_{c}\frac{\tau}{2})}$
(17)

For typical experimental values of $B$, $|\alpha_{0}|^{2}>50$ and thus,
$I_{\alpha,\alpha^{{}^{\prime}}}\rightarrow 0$. Accordingly, the scattering
rate and conductivity would be negligible too. Nonetheless, there is an
important exception when $\tau$ equals the cyclotron period $T_{c}$:
$\tau=\frac{2\pi}{w_{c}}$.

Figure 3: Schematic diagrams for electron scattering between coherent states
in the dark (undriven) and with radiation (mw-driven). a) Undriven scattering.
The average distance (advanced distance) between initial
$|\psi_{\alpha}\rangle$ and final coherent state,
$|\psi_{\alpha^{{}^{\prime}}}\rangle$, is $\Delta X(0)$. This distance mainly
determines $R_{xx}$. b) MW-driven scattering giving rise to peaks. Now the
average advanced distance is larger because the final state, minimizing the
Heisenberg uncertainty principle, is farther than the dark position due to the
swinging motion of the driven-coherent states. c) MW-driven scattering giving
rise to valleys. When the final coherent state is closer we obtain MIRO
valleys. b) Situation when MW power is high enough and the states go
backwards. In this scenario the final state ends up behind the initial state
dark position and the scattering jump can not take place.

In other words, the scattered electron begins and ends in the same position in
the Landau orbit. Only in this case $I_{\alpha,\alpha^{{}^{\prime}}}\neq 0$.
Thus, only scattering processes fulfilling the previous condition of $\tau$
will efficiently contribute to the current. The rest contributions can be
neglected. Finally the expression of $I_{\alpha,\alpha^{{}^{\prime}}}$
readsridley ,

$\displaystyle|I_{\alpha,\alpha^{{}^{\prime}}}|$ $\displaystyle=$
$\displaystyle
e^{-\left(\frac{\left(X^{{}^{\prime}}(0)-X(0)\right)^{2}}{8(\Delta
x)^{2}}+\frac{q_{x}^{2}(\Delta x)^{2}}{2}\right)}$ (18) $\displaystyle=$
$\displaystyle e^{-\left(\frac{q^{2}2(\Delta x)^{2}}{4}\right)}$

where $X^{{}^{\prime}}(0)-X(0)=[-q_{y}2(\Delta x)^{2}]$,ridley . This, in
turn, leads us to a final expression for $W_{I}$,

$W_{I}=\frac{n_{i}e^{4}}{2\pi\hbar\epsilon^{2}}\int\frac{e^{-q^{2}(\Delta
x)^{2}}}{(q+q_{TF})^{2}}(1-\cos\theta)\delta(E_{\alpha^{{}^{\prime}}}-E_{\alpha})d^{2}q$
(19)

where $n_{i}$ is the charged impurity density and $\theta$ is the scattering
angle. The density of initial Landau states $\rho_{i}(E)$ can be obtained by
using the Poisson sum rules to get toihn ,
$\rho_{i}(E)=\frac{m^{\ast}}{\pi\hbar^{2}}\left[1-2\cos\left(\frac{2\pi
E}{\hbar w_{c}}e^{-\pi\Gamma/\hbar w_{c}}\right)\right]$. Finally, gathering
all terms and solving the energy integral, we obtain an expression for
$\sigma_{xx}$ that reads,

$\displaystyle\sigma_{xx}=\frac{n_{i}e^{6}m^{\ast}}{2\pi^{3}\hbar^{3}\epsilon^{2}}(\Delta
X_{0})^{2}\frac{1}{\hbar w_{c}}\left(\frac{1+e^{-\pi\Gamma/\hbar
w_{c}}}{1-e^{-\pi\Gamma/\hbar w_{c}}}\right)\times$
$\displaystyle\left(1-\frac{2\chi_{s}}{\sinh(\chi_{s})}\cos\left(\frac{2\pi
E_{F}}{\hbar w_{c}}\right)e^{-\pi\Gamma/\hbar w_{c}}\right)\times$
$\displaystyle\int\frac{e^{-q^{2}(\Delta
x)^{2}}}{(q+q_{TF})^{2}}(1-\cos\theta)d^{2}q$ (20)

where $\chi_{s}=2\pi^{2}k_{B}T/\hbar w_{c}$, $k_{B}$ being the Boltzmann
constant, $E_{F}$ the Fermi energy and $\Gamma$ the Landau level width. To
obtain $R_{xx}$ we use the relation
$R_{xx}=\frac{\sigma_{xx}}{\sigma_{xx}^{2}+\sigma_{xy}^{2}}\simeq\frac{\sigma_{xx}}{\sigma_{xy}^{2}}$,
where $\sigma_{xy}\simeq\frac{n_{e}e}{B}$ and $\sigma_{xx}\ll\sigma_{xy}$,
$n_{e}$ being the 2D electron density.

One important condition that features coherent states is that they minimize
the Heisenberg uncertainty principle. Thus, for the time-energy uncertainty
relationcohen , $\Delta t\Delta E=h$. For our specific problem, $\Delta
t=\tau$ that implies $\Delta E=\hbar w_{c}$, $\Delta E$ being the energy
difference between scattering-involved coherent states. Thus, we obtain two
conditions for the scattering between coherent states to take place, first
$\tau=\frac{2\pi}{w_{c}}$ and second, the energy difference equals $\hbar
w_{c}$. There are also physical reasons that endorse the latter specially in
high-mobility samples where the levels are very narrow in terms of states
density. In these systems the only efficient contributions to scattering are
the ones corresponding to aligned Landau levels (see Fig. 2), i.e., when
$\Delta E=n\times\hbar w_{c}$. The most intense of them is when $n=1$ that
corresponds to the closest in distance coherent states or smallest value of
$\Delta X_{0}$, (see Eq. 18). This agrees with the condition that when $n=1$,
the Heisenberg uncertainty principle is minimized. The two conditions
discussed above hold in the dark and under radiation. For the latter case,
MIRO reveals, the important role played by $\tau$ in the based-on-coherent
states magnetotransport processes.

When we turn on the light, the term that is going to be mainly affected in the
$\sigma_{xx}$ expression is the distance between the coherent states guiding
centers, i.e., $\Delta X_{0}$. This average distance now turns into $\Delta
X_{MW}$inarashba ; inahole

$\displaystyle\Delta X_{MW}$ $\displaystyle=$ $\displaystyle
X^{{}^{\prime}}_{MW}-X_{MW}$ $\displaystyle=$ $\displaystyle\Delta
X_{0}-A\left(\sin w(t+\tau)-\sin wt\right)+$
$\displaystyle\sqrt{\frac{2\hbar}{m^{\ast}w_{c}}}|\alpha_{0}|\left(\cos
w_{c}(t+\tau)-\cos w_{c}t\right)$

If we consider, on average, that the scattering jump begins when the MW-driven
oscillations is at its mid-point, ($wt=2\pi n$, $n$ being a positive integer),
and being $\tau=2\pi/w_{c}$, we end up having,

$\Delta X_{MW}=\Delta X_{0}-A\sin 2\pi\frac{w}{w_{c}}$ (22)

This result affects dramatically $\sigma_{xx}$ and in turn $R_{xx}$. Now
photo-oscillations rise according to $\Delta X_{MW}$ and its built-in sine
function. In Fig. 3 we present schematic diagrams for the different situations
regarding MIRO peaks and valleys and ZRS. In the undriven scenario an electron
in the initial coherent state, scatters with charged impurities and jumps to
the final coherent state minimizing the Heisenberg uncertainty principle. The
latter condition determines what coherent states can be connected via
scattering. On average the advanced distance is $\Delta
X_{0}=X^{{}^{\prime}}_{0}-X_{0}$, (see Fig. 3a). When the light is on,
depending on the term $A\sin 2\pi\frac{w}{w_{c}}$, some times the minimum
uncertainty final state will be further away than in the dark regarding the
initial state position. Thus, on average, $\Delta X_{MW}>\Delta X_{0}$ and
$R_{xx}$ will be larger, giving rise to peaks (see Fig. 3b). On the other
hand, other times the final coherent state will be closer and $\Delta
X_{MW}<\Delta X_{0}$ and $R_{xx}$ will be smaller, giving rise to valleys,
(see Fig. 3c). Finally, when the driven coherent states are going backward and
the radiation power is large enough, the final state, minimizing the
uncertainty principle, will be behind the initial state in the dark (see Fig.
3d). However, the scattered electron can only effectively jump forward due to
the DC electric field direction and the final coherent state can never be
reached; in the forward direction there is no final coherent state fulfilling
the minimum uncertainty condition and the scattering can not be completed.
Thus, the system reaches the ZRS scenario where the electron remains in the
initial coherent state.

Figure 4: Calculated magnetoresistance as a function of $B$, for a radiation
frequency of $103$ GHz and $T=1$ K. The dark case is also exhibited. Minima
positions are indicated with arrows corresponding to,
$\frac{w}{w_{c}}=j+\frac{1}{4}$, $j$ being a positive integer. Zero resistance
states are obtained around $B\simeq 0.2T$. Inset: irradiated magnetoresistance
showing periodicity vs $1/B$.

Results. In Fig. 4 we present calculated results of the irradiated $R_{xx}$ vs
$B$ for a radiation frequency of $103$ GHz and $T=1$ K. The dark case is also
exhibited. In our simulations all results have been based on experimental
parameters corresponding to the experiments by Mani et al. mani . We obtain
clear MIRO where the minima positions are indicated with arrows and, as in
experimentsmani , correspond to, $\frac{w}{w_{c}}=j+\frac{1}{4}$, $j$ being a
positive integer. Minima positions show a clear $1/4$-cycle shift, which is a
universal property that features MIRO and shows up in any experiment about
MIRO irrespective of the sort of carrierzudovhole  and platforminamno . In the
minima corresponding to $j=1$, ZRS are found. Now with the help of our present
model based on coherent states we can explain such a peculiar value for the
minima position. Thus, it is straightforward to check out that if we
substitute equation $\frac{w}{w_{c}}=j+\frac{1}{4}$ in $\Delta X_{MW}$, (Eq.
22), we would obtain minima values of the latter and in turn of $R_{xx}$.
Therefore, from the minima positions relation we can obtain the value
$2\pi/w_{c}$ which would be the ”smoking gun” that would reveal the presence
of coherent states of quantum harmonic oscillators sustaining the
magnetorresistance of high-quality 2DES. Another evidence of the latter would
be the MIRO periodicity with the inverse of $B$ (see inset of Fig. 4) that
would be explained by the presence of $\tau$ in the argument of the sine
function.

Summary. Summing up, we have demonstrated that magnetoresistance in a high
mobility 2DES under MW radiation can be explained in terms of the coherent
states of the quantum harmonic oscillator. When irradiated these systems give
rise to MIRO that reveals the presence of these quasi-classical states in
high-quality samples when under low $B$. These MW-driven coherent states have
been used to calculate irradiated magnetoresistance finding that the principle
of minimum uncertainty of coherent states is crucial to understand MIRO and
their properties and zero resistance states. We conclude that any experiment
on irradiated magnetoresistance of 2D systems, regardless of carrier and
platform zudovhole ; bandu , showing MIRO reveals the existence of coherent
states of the quantum harmonic oscillator. We expect that dealing with even
higher mobility samples, ($\mu>10^{7}$), it would be possible to achieve the
quantum superposition of coherent states yielding, for instance in the case of
two, even and odd coherent states of the quantum harmonic oscillator. Then,
when irradiated, we expect that MIRO would evolve showing striking results
revealing the presence of coherent states superpositionzudov2 ; rui .

This work was supported by the MCYT (Spain) grant PID2020-117787GB-I00.

## References

  * (1) E. Srödinger, Der stetige Übergang von der Mikro- zur Makromechanik, Die Naturwissenschften, 14, 664, (1926).
  * (2) R.J. Glauber, Coherent and Incoherent States of the Radiation Field, Phys. Rev. 131, 2766, (1963)
  * (3) V.V. dodonov, ‘Nonclassical’ states in quantum optics: a ‘squeezed’ review of the first 75 years, J. Opt. B:Quantum Semiclass. Opt. 4, R1, (2002).
  * (4) B. Yurke and D. Stoler, Generating quantum mechanical superpositions of macroscopically distinguishable states via amplitude dispersion, Phys. Rev. Lett. 57 13 (1986).
  * (5) Michael W. Noel and C.R. Stroud, Excitation of an Atomic Electron to a Coherent Superposition of Macroscopically Distinct States, Phys. Rev. Lett. 77 1913 (1996).
  * (6) V.I. Man’ko Theory of nonclassical states of light. Chap. 4. Taylor and Francis. London and NewYork.
  * (7) R.G. Mani, J.H. Smet, K. von Klitzing, V. Narayanamurti, W.B. Johnson, V. Umansky, Zero-resistance states induced by electromagnetic wave excitation in GaAs/AlGaAs heterostructures, Nature 420 646 (2002).
  * (8) M.A. Zudov, R.R. Lu, N. Pfeiffer, K.W. West, Evidence for a New Dissipationless Effect in 2D Electronic Transport, Phys. Rev. Lett. 90 046807 (2003).
  * (9) V. I. Ryzhii, Photoconductivity characteristics in thin films subjected to crossed electric and magnetic fields, Sov. Phys. Solid State 11, 2078 (1970); V. I. Ryzhii, R. A. Suris, and B. S. Shchamkhalova, Photoconductivity of a two-dimensional electron gas in a strong magnetic field, Sov. Phys. Semicond. 20, 1299 (1986).
  * (10) A.C. Durst, S. Sachdev, N. Read, S.M. Girvin, Radiation-Induced Magnetoresistance Oscillations in a 2D Electron Gas, Phys. Rev. Lett.91 086803 (2003).
  * (11) I. A. Dmitriev, M. G. Vavilov, I. L. Aleiner, A. D. Mirlin, and D. G. Polyakov, Theory of microwave-induced oscillations in the magnetoconductivity of a two-dimensional electron gas, Phys. Rev. B 71, 115316 (2005)
  * (12) J. Iñarrea and G. Platero, Theoretical Approach to Microwave-Radiation-Induced Zero-Resistance States in 2D Electron Systems, Phys. Rev. Lett. 94 016806, (2005).
  * (13) J. Iñarrea, The two dimensional electron system as a nanoantenna in the microwave and terahertz bands, Appl. Phys. Lett., 99, 232115, (2011).
  * (14) J.Iñarrea, Linear polarization sensitivity of magnetotransport in irradiated two-dimensional electron systems , J. Appl. Phys. 113, 183717, (2013).
  * (15) J.Iñarrea and G. Platero, Microwave-induced resistance oscillations and zero-resistance states in two-dimensional electron systems with two occupied subbands, Phys. Rev. B. 84, 075313, (2011).
  * (16) Claude Cohen-Tannoudji, Bernard Diu, and Franck Laloë, Quantum Mechanics.. Wiley and sons. Paris.
  * (17) E.H. Kerner, Note on the forced and damped oscillator in quantum mechanics, J. Phys. 36, 371 (1958) .
  * (18) K. Park, Radiation-induced zero-resistance state at low magnetic fields and near half-filling of the lowest Landau level, Phys. Rev. B 69 201301(R) (2004).
  * (19) B.K. Ridley. Quantum Processes in Semiconductors, 4th ed. Oxford University Press, (1993)
  * (20) T. Ando, A. Fowler and F. Stern, Electronic properties of two-dimensional systems, Rev. Mod. Phys.54,(1982).
  * (21) B.M. Askerov, Electron Transport Phenomena in Semiconductors., World Scientific, (1994).
  * (22) J. H.Davies, The physics of low-dimensional semiconductors. Cambridge university press (1998).
  * (23) Thomas Ihn, Semiconductors nanostructures. Oxford University Press. New York.
  * (24) Jesus Iñarrea, Radiation-induced resistance oscillations in 2D electron systems with strong Rashba coupling, Sci. Rep. 7, 13573, (2017).
  * (25) Jesus Iñarrea and Gloria Platero, Radiation-induced resistance oscillations in a 2D hole gas: a demonstration of a universal effect, Journal of Physics: Condens. Matter, 27 415801 (2015).
  * (26) L. Bockhorn, P. Barthold, D. Schuh, W. Wegscheider, and R.J. Haug, Magnetoresistance in a high-mobility two-dimensional electron gas, Phys. Rev. B. 83 113301 (2011).
  * (27) R. G. Mani, A. Kriisa and W. Wegscheider, Size-dependent giant-magnetoresistance in millimeter scale GaAs/AlGaAs 2D electron devices, 3 2747 (2013).
  * (28) J. Iñarrea, Theoretical model for negative giant magnetoresistance in ultrahigh-mobility 2D electron systems. EPL, 106 47005 (2014).
  * (29) M. A. Zudov, O. A. Mironov, Q. A. Ebner, P. D. Martin, Q. Shi, and D. R. Leadley, Colossal negative magnetoresistance in a two-dimensional electron gas, Phys. Rev.B, 89, 12540, (2014).
  * (30) J. Iñarrea, Photo-oscillations in MgZnO/ZnO heterostructures, Scientific Reports 12, 22463, (2022).
  * (31) Erwin Mönch, Denis A. Bandurin, Ivan A. Dmitriev, Isabelle Y. Phinney, Ivan Yahniuk, Takashi Taniguchi, Kenji Watanabe, Pablo Jarillo-Herrero, and Sergey D. Ganichev, Observation of Terahertz-Induced Magnetooscillations in Graphene, Nano Lett. 20, 5943–5950, (2020).
  * (32) A. T. Hatke, M. A. Zudov, L. N. Pfeiffer and K. W. West, Giant microwave photoresistivity in high-mobility quantum Hall systems, Phys. Rev. B 83, 121301(R) (2011).
  * (33) Yanhua Dai, R. R. Du, L. N. Pfeiffer and K.W. West, Observation of a Cyclotron Harmonic Spike in Microwave-Induced Resistances in Ultraclean GaAs/AlGaAs Quantum Wells, Phys. Rev. Lett. 105, 246802 (2010).


# Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural
Network Performance

Youngmin Seo  Jinha Kim  Unsang Park
Sogang University
{ymin98 Corresponding Author<EMAIL_ADDRESS>jhkmo510
<EMAIL_ADDRESS>
###### Abstract

We propose the Swish-T family, an enhancement of the existing non-monotonic
activation function Swish. Swish-T is defined by adding a Tanh bias to the
original Swish function. This modification creates a family of Swish-T
variants, each designed to excel in different tasks, showcasing specific
advantages depending on the application context. The Tanh bias allows for
broader acceptance of negative values during initial training stages, offering
a smoother non-monotonic curve than the original Swish. We ultimately propose
the Swish-T${}_{\textbf{C}}$ function, while Swish-T and
Swish-T${}_{\textbf{B}}$, byproducts of Swish-T${}_{\textbf{C}}$, also
demonstrate satisfactory performance. Furthermore, our ablation study shows
that using Swish-T${}_{\textbf{C}}$ as a non-parametric function can still
achieve high performance. The superiority of the Swish-T family has been
empirically demonstrated across various models and benchmark datasets,
including MNIST, Fashion MNIST, SVHN, CIFAR-10, and CIFAR-100. The code is
publicly available at “https://github.com/ictseoyoungmin/Swish-T-pytorch”.

_K_ eywords Activation Function $\cdot$ Deep Learning $\cdot$ Neural Network

##  1 Introduction

Activation functions are crucial in deep learning for introducing non-
linearity, which is essential for modeling complex patterns. Early activation
functions like Sigmoid and Tanh enabled smooth transitions but were
susceptible to the vanishing gradient problem. The introduction of the
Rectified Linear Unit (ReLU)[1, 2, 3] marked a significant advancement due to
its simplicity and efficiency, enhancing gradient flow and accelerating
learning. However, despite its benefits, ReLU can suffer from the "dying ReLU"
problem, where neurons become inactive and cease to function across a range of
inputs.

To analyze the impact of various activation functions, we visualize the output
landscape of a 3-layer neural network with randomly initialized weights,
without training, as shown in Fig. 1. Fig. 1 highlights even before learning
begins. The "dying ReLU" issue can be considered as incurring from the flat
regions where ReLU neurons output zero, resulting in large inactive areas in
the landscape.

Figure 1: Comparison of Various Activation Functions including Sigmoid, ReLU,
Leaky ReLU, GELU, Swish, Mish, SMU, SMU-1, Swish-T${}_{\textbf{C}}$, and
Identity in the Output Landscape of a 3-Layer Neural Network. All the
networks’ weights are randomly initialized, and no training has been
performed, showcasing the initial output patterns induced by each activation
function.

To overcome the limitations of ReLU, variants such as Leaky ReLU [4] and
Parametric ReLU (PReLU) [5] have been developed. These modifications address
the "dying ReLU" issue by allowing a small, non-zero gradient when the unit is
inactive, thus maintaining learning capabilities during backpropagation.
Nonetheless, these functions introduce a new challenge as they are non-
differentiable at zero, leading to a discontinuity in the derivative. This
non-differentiability can complicate optimization, as the precise point where
this occurs may result in unpredictable behaviors in gradient-based learning
methods.

Further advancements have been made with functions like GELU [6], Swish [7],
ACONs [8], Pserf [9], ErfAct [9], and Mish [10], which are smooth
approximations of ReLU, thereby alleviating some of its issues. Notably, Mish,
inspired by Swish, and ACONs, introducing a dynamic choice between linear and
non-linear modes, enhances neural network’s adaptability and generalizes Swish
as a special case of ACON-A [8]. PSerf, as a non-monotonic, smooth and
parametric activation function, demonstrates significant performance
improvements over traditional activations like ReLU and Swish. SMU [11] has
shown significant performance improvements by providing smooth approximations
to the Maxout [12] family functions, including ReLU and Leaky ReLU.

Among the successors to ReLU, Swish has emerged as a particularly effective
function. As a result, various Swish-based functions such as E-Swish [13],
P-Swish [14], and Soft-Clipping Swish [15] have been proposed. These functions
improve performance by adding parameters to the original Swish or by combining
it with other functions, enhancing their capability to model complex patterns.

The contribution of this paper is summarized as follows:

  * •

Introduction of Swish-T Family: The Swish-T family enhances the existing Swish
activation function by adding a Tanh bias, allowing broader acceptance of
negative values and providing a smoother non-monotonic curve.

  * •

Variants of Swish-T: Swish-T${}_{\textbf{A}}$, Swish-T${}_{\textbf{B}}$, and
Swish-T${}_{\textbf{C}}$ are variants designed for improved computational
efficiency, adaptability, and stability, respectively.

  * •

Empirical Validation: Extensive experiments demonstrate the superior
performance of the Swish-T family across various datasets (MNIST, Fashion
MNIST, SVHN, CIFAR-10, CIFAR-100) and architectures (ResNet-18, ShuffleNetV2,
SENet-18, EfficientNetB0, MobileNetV2, DenseNet-121).

##  2 Related work and Motivation

Recent studies, including those by SMU, ACONs, and Pserf, have focused on
generalizing activation functions such as ReLU, Leaky ReLU, and PReLU through
smooth approximations while incorporating trainable parameters. These studies
suggest that Swish, despite its simple definition, demonstrates powerful
performance and holds the potential for further evolution into more advanced
forms.

The Swish function, defined as $y=x\cdot\text{Sigmoid}(\beta x)$, has emerged
as a powerful activation function, demonstrating superior performance in
various deep learning tasks such as image classification [16], object
detection [17, 18], and Natural Language Processing (NLP) [19]. Discovered
through Neural Architecture Search (NAS), Swish is characterized by its non-
monotonicity and smoothness, which allow it to maintain small negative
weights, enhancing model training and performance stability.

Swish offers several advantages:

  * •

Smooth Nonlinearity: Swish is a smooth, non-monotonic function, which improves
information propagation in deep networks.

  * •

Self-gating: Swish includes a self-gating mechanism that adjusts the output
scale based on the input value, allowing for a more flexible function form.

  * •

Training Efficiency and Performance: Empirical results indicate that Swish
outperforms ReLU in deep models and large datasets, handling gradient
vanishing and saturation problems more effectively.

Building on these strengths, we introduce a new concept by incorporating bias
terms directly within the activation function, inspired by the significant
role of bias in neural network layers [20]. We hypothesize that incorporating
bias terms into the activation function itself can provide more sophisticated
control of the activation threshold, enabling a more nuanced adaptation to the
input function. This hypothesis is based on the fact that bias in an existing
layer better adjusts the input signal for a desired output range, thereby
improving the learning ability of the network.

Our research proposes the Swish-T family, which incorporates a parameterized
bias term into the Swish function. This novel variant aims to optimize
performance further and enhance adaptability across various architectures and
applications. By integrating the bias term, we seek to improve the learning
dynamics and overall network performance.

##  3 Swish-T

Figure 2: Swish-T${}_{\textbf{C}}$, Swish activation function and first
derivatives. (a) Swish-T${}_{\textbf{C}}$ activation function with fixed alpha
and beta. (b) The first derivatives with fixed alpha$=$0.5 and different
betas. Beta controls how quickly the first derivative reaches the upper/lower
asymptotes. (c) Alpha determines the upper/lower bounds of the first
derivative.

We present the Swish with Tanh bias function (Swish-T) family. To complement
Swish, we conducted preliminary research to find an appropriate bias function.
The criteria for the bias function are as follows:

  * •

Zero-centered: This ensures that the function maintains the zero-centered
characteristic of the original Swish function.

  * •

Non-linearity: The bias function should be nonlinear to effectively adjust the
bias based on the value of $x$.

  * •

Bounded: It should be bounded by a scaling parameter $\alpha$ to ensure it
does not disrupt the properties of the original Swish function.

Among the combinations of $\alpha x^{2}$, $\alpha\tanh(x)$, and
$\alpha\tanh(\beta x)$, the configurations Swish + $\alpha\tanh(x)$ and
Swish-1 + $\alpha\tanh(\beta x)$ showed competitive performance. Specifically,
Swish + $\alpha\tanh(x)$ achieved a top-1 accuracy of 79.02 on the CIFAR-100
dataset [21] using the ResNet-18 model [22], leading to the derivation of the
Swish-T function family. In comparison, Swish + $\alpha x^{2}$ and Swish-1 +
$\alpha\tanh(\beta x)$ obtained top-1 accuracies of 77.31 and 78.22,
respectively.

###  3.1 Swish-T Family

The basic design of Swish-T is adding $\alpha\tanh(x)$ to the original Swish
function, expressed as:

$\displaystyle f(x;\beta,\alpha)$ $\displaystyle=x\sigma(\beta
x)+\alpha\tanh(x)$ (1) $\displaystyle=x\sigma(\beta x)+\alpha(2\sigma(2x)-1)$
$\displaystyle=x\sigma(\beta x)+2\alpha\sigma(2x)-\alpha$

Here, $\sigma(x)$ denotes the sigmoid function, $\beta$ is a trainable
parameter, and $\alpha$ is a hyper-parameter used to scale the tanh function
from $(-1,1)$ to $(-\left|\alpha\right|,\left|\alpha\right|)$. In our
experiments, the Swish-T family was initialized with $\beta=1.0$ and
$\alpha=0.1$, and $\beta$ was updated during training.

Swish-T converges to $\alpha$ in the negative region. The bias increases or
decreases until bounded by the tanh function and is determined by $x$. When
$x=0$, the tanh output is 0, maintaining the zero-centered property of the
original Swish function.

Eq. 1 cannot be simplified because the coefficients of $x$ inside the sigmoid
function differ. Simplifying the equation could improve the forward and
backward pass speeds in deep learning frameworks like TensorFlow [23] and
PyTorch [24]. To enhance speed while maintaining or improving performance, we
designed the Swish-T family, including Swish-T${}_{\textbf{A}}$,
Swish-T${}_{\textbf{B}}$, and Swish-T${}_{\textbf{C}}$.

Swish-TA simplifies Eq. 1 by unifying the coefficients of $x$ to 1, improving
speed:

$\displaystyle f_{A}(x;\alpha)$
$\displaystyle=x\sigma(x)+2\alpha\sigma(x)-\alpha$ (2)
$\displaystyle=\sigma(x)(x+2\alpha)-\alpha$

This function is equivalent to Swish-1($x$) + $\alpha\tanh(x/2)$ and has no
trainable parameters. This transformation is advantageous in scenarios where
computational efficiency is crucial.

Swish-T${}_{\textbf{B}}$ reintroduces a trainable parameter to Eq. 2:

$\displaystyle f_{B}(x;\beta,\alpha)$ $\displaystyle=\sigma(\beta
x)(x+2\alpha)-\alpha$ (3)

Rewriting Eq. 3 as $x\sigma(\beta x)+\alpha\tanh(\beta/2x)$ shows that the
bias depends on both $x$ and $\beta$.

Swish-T${}_{\textbf{C}}$ ensures symmetry when $\beta$ changes sign:

$\displaystyle f_{C}(x;\beta,\alpha)$ $\displaystyle=\sigma(\beta
x)(x+\frac{2\alpha}{\beta})-\frac{\alpha}{\beta},\quad\text{where \,}\beta\neq
0$ (4)

Simplifying Eq. 4 shows that the maximum bias coefficient is $\alpha/\beta$.
As $\beta$ approaches infinity, the bias converges to 0, making it act like a
smooth ReLU. When $\beta$ approaches 0, the function approaches 0. With
$\alpha=1.0$ and $\beta=1e^{-4}$, this function behaves as the Identity
function. Adjusting $\alpha$ changes the slope, requiring precise tuning for
exact Identity function behavior. This transformation is useful for
applications requiring stable performance across a wide range of inputs.

Each function variant exhibits unique convergence characteristics.
Swish-T${}_{\textbf{A}}$ simplifies to enhance computational efficiency.
Swish-T${}_{\textbf{B}}$ dynamically adapts the bias with $\beta$, suitable
for tasks with diverse input characteristics. Swish-T${}_{\textbf{C}}$ ensures
symmetry when $\beta$ changes sign, maintaining stable performance across
various input ranges. The bounded nature of the tanh function ensures that the
bias remains within a manageable range, preventing extreme changes that could
destabilize training. This is particularly evident in (Swish-T,
Swish-T${}_{\textbf{A}}$, Swish-T${}_{\textbf{B}}$) and
Swish-T${}_{\textbf{C}}$, where the bias converges to $\pm\alpha$ and
$\pm\alpha/\beta$, respectively.

###  3.2 Gradient Computation for Swish-T Family

In the context of training neural networks, the efficiency of
backpropagation[25] is crucial. This section derives the gradient expressions
for Swish-T and its variants, highlighting their impact on backpropagation.

The gradient of the Swish-T activation function can be expressed as:

$\displaystyle\frac{d}{dx}f=\sigma(\beta x)+\beta x\sigma(\beta
x)(1-\sigma(\beta x))+4\alpha\sigma(2x)(1-\sigma(2x)),$ (5)
$\displaystyle\text{where}\,\frac{d}{dx}\sigma(x)=\sigma(x)(1-\sigma(x))$

Among the Swish-T family, Swish-T${}_{\textbf{A}}$ shows the fastest learning
speed. Its derivative is given by:

$\displaystyle\frac{d}{dx}f_{A}$
$\displaystyle=\sigma(x)(1-\sigma(x)(x+2\alpha)+\sigma(x))$ (6)
$\displaystyle=\sigma(x)((1-\sigma(x)(x+2\alpha))+1)$
$\displaystyle=\sigma(x)(x+\alpha+1-(x\sigma(x)+2\alpha\sigma(x)-\alpha))$
$\displaystyle=\sigma(x)(x+\alpha+1-f_{A}(x))$

This expression, along with Eq. 6, shows that Swish-T${}_{\textbf{A}}$ can be
differentiated efficiently, making it suitable for backpropagation in terms of
both memory and speed. Similarly, the gradients for Swish-T${}_{\textbf{B}}$
and Swish-T${}_{\textbf{C}}$ include their own expressions and are derived as
follows:

For Swish-T${}_{\textbf{B}}$:

$\displaystyle\frac{d}{dx}f_{B}$ $\displaystyle=\beta\sigma(\beta
x)(1-\sigma(\beta x)(x+2\alpha)+\sigma(\beta x))$ (7)
$\displaystyle=\sigma(\beta x)(\beta(x+\alpha-f_{B}(x))+1)$

Eq. 7 describes the derivative of the Swish-T${}_{\textbf{B}}$ activation
function. The term $\beta\sigma(\beta x)$ represents the impact of the
activation function on the gradient, with the factor $\beta$ scaling the input
$x$. The expression $(1-\sigma(\beta x)(x+2\alpha)+\sigma(\beta x))$ captures
the adjustment made by the sigmoid function, $\sigma(\beta x)$, and the
constants $\alpha$ and $\beta$. This formulation ensures a controlled gradient
flow, balancing the input $x$ and the influence of the activation function.

For Swish-T${}_{\textbf{C}}$:

$\displaystyle\frac{d}{dx}f_{C}$ $\displaystyle=\beta\sigma(\beta
x)(1-\sigma(\beta x)(x+2\alpha/\beta)+\sigma(\beta x))$ (8)
$\displaystyle=\sigma(\beta x)(\beta(x-f_{C}(x))+\alpha+1)$

Eq. 8 describes the derivative of the Swish-T${}_{\textbf{C}}$ activation
function. Unlike Swish-T${}_{\textbf{B}}$, this formulation avoids the direct
multiplication of $\alpha$ and $\beta$, which helps prevent situations where
the maximum gradient could rapidly increase. This characteristic is evident in
Fig. 2, where the stability of the gradient flow is demonstrated. The term
$\beta\sigma(\beta x)$ remains, similar to Swish-T${}_{\textbf{B}}$, but the
expression $(1-\sigma(\beta x)(x+2\alpha/\beta)+\sigma(\beta x))$ shows a
different interaction between the input $x$ and the constants $\alpha$ and
$\beta$. This leads to a more stable gradient behavior, as highlighted by the
addition of $\alpha+1$ in the final term.

These derivatives show that the Swish-T family of activation functions can be
differentiated efficiently, making them suitable for backpropagation. The
variations in their formulations offer different advantages in terms of
learning speed and gradient stability, as illustrated in the respective
equations and supported by empirical evidence.

Below are the equations for the gradient of the Swish-T family activation
functions with respect to $\beta$:

$\displaystyle\frac{d}{d\beta}f$ $\displaystyle=x^{2}\cdot\sigma(\beta
x)(1-\sigma(\beta x))$ (9) $\displaystyle\frac{d}{d\beta}f_{B}$
$\displaystyle=x\cdot(x+2\alpha)\cdot\sigma(\beta x)(1-\sigma(\beta x))$ (10)
$\displaystyle\frac{d}{d\beta}f_{C}$
$\displaystyle=x\cdot\left(x+\frac{2\alpha}{\beta}\right)\sigma(\beta
x)(1-\sigma(\beta x))-\frac{2\alpha\sigma(\beta
x)}{\beta^{2}}+\frac{\alpha}{\beta^{2}}$ (11)

In the process of backpropagation, trainable parameters are updated through
the gradient of the activation function, specifically the value obtained by
differentiating with respect to the parameter in question.

For the Swish-T activation function, the gradient of the parameter $\beta$ is
given by Eq. 9. This expression shows that the gradient depends on the input
$x$, the sigmoid function $\sigma(\beta x)$, and its derivative. The $x^{2}$
term indicates that larger inputs have a greater influence on the gradient,
leading to faster updates of the $\beta$ parameter during training.

For the Swish-T${}_{\textbf{B}}$ variant, the gradient of $\beta$ is given by
Eq. 10. The additional term $(x+2\alpha)$ modifies the influence of the input
$x$, suggesting that the parameter $\alpha$ also plays a role in forming the
gradient. This interaction between $\alpha$ and $\beta$ allows for fine-tuning
the behavior of the activation function, providing more precise control over
the learning process.

For the Swish-T${}_{\textbf{C}}$ variant, the gradient of $\beta$ is given by
Eq. 11. The division by $\beta^{2}$ or $\beta$ indicates that the magnitude of
the gradient is inversely proportional to $\beta$, which helps prevent
excessively large updates during training.

Overall, the effect of $\beta$ for the Swish-T function emphasizes the complex
dependency between the input $x$, the trainable parameter $\beta$, and
constants such as $\alpha$. Carefully adjusting $\beta$ can enhance the
efficiency of backpropagation, leading to faster convergence and improved
learning outcomes for the neural network.

##  4 Experiments and Results

We apply both non-parametric and parametric functions uniformly across all
activation operations in the network. Consequently, the trainable parameters
of our functions and comparison functions are applied globally and updated
accordingly. This approach allows us to identify a single optimized function
for various architectures. The experiments primarily focus on image
classification tasks, with the potential to extend to other tasks such as
semantic segmentation and object detection to explore generalization
capabilities.

To evaluate the performance of our functions, we compare them against widely
used activation functions known for their high performance: ReLU [1, 2, 3],
GELU [6], Swish [7], SiLU (Swish-1) [26], and Mish [10], as well as the
recently introduced, high-performing SMU [11] and SMU-1. The $\beta$ parameter
in the Swish-T family is initialized to 1.0, the same as Swish, while the
hyperparameter $\alpha$ is set to 0.1 across all experiments. The trainable
parameter $\mu$ in SMU [11] and SMU-1 [11] is initialized to 1.0. For SMU, the
hyperparameter $\alpha$ is set to 0.0, while for SMU-1, $\alpha$ is set to
0.25. All trainable parameters of the activation functions are updated using
the backpropagation algorithm [27]. Experiments are conducted on an NVIDIA
2080Ti GPU, with PyTorch [24] version 2.0.1.

###  4.1 MNIST, Fashion MNIST and SVHN

In this section, we evaluate the performance of various activation functions
on three popular image classification datasets: MNIST [28], Fashion MNIST
[29], and SVHN [30]. The MNIST dataset contains 70,000 grayscale images of
handwritten digits (0-9), each of size $28\times 28$ pixels, widely used for
training and testing image processing systems. Fashion MNIST, a dataset
similar in structure to MNIST, consists of 70,000 grayscale images of
$28\times 28$ pixels depicting various types of clothing items, offering a
more complex alternative to digit classification. The SVHN (Street View House
Numbers) dataset includes 600,000 color images of house numbers captured from
Google Street View, with each image containing digits in a natural scene,
providing a challenging dataset due to its real-world variability. The
experiments are conducted using the LeNet [31] architecture, a well-known
convolutional neural network model designed for handwritten digit recognition.
The optimizer used was Stochastic Gradient Descent (SGD) [32, 33] with a
learning rate of 0.01, momentum of 0.9, and weight decay of 5e-4. The training
was carried out for 100 epochs, with the learning rate adjusted using the
cosine annealing [34] scheduler. Each experiment utilized a batch size of 128.
We used standard data augmentation techniques such as random affine transform.
Each training process was repeated 10 times. The results can be found in Table
1.

Activation Function MNIST Fashion MNIST SVHN ReLU 99.34 $\pm$ 0.028 89.89
$\pm$ 0.205 90.59 $\pm$ 0.203 GELU 99.36 $\pm$ 0.034 89.84 $\pm$ 0.230 90.79
$\pm$ 0.117 SiLU 99.31 $\pm$ 0.022 89.66 $\pm$ 0.084 90.61 $\pm$ 0.099 Swish
99.31 $\pm$ 0.010 89.80 $\pm$ 0.142 90.73 $\pm$ 0.101 Mish 99.32 $\pm$ 0.013
89.80 $\pm$ 0.151 90.65 $\pm$ 0.116 SMU 99.30 $\pm$ 0.022 89.91 $\pm$ 0.058
90.74 $\pm$ 0.230 SMU-1 99.35 $\pm$ 0.039 89.82 $\pm$ 0.235 90.72 $\pm$ 0.146
Swish-T 99.34 $\pm$ 0.033 89.85 $\pm$ 0.059 90.99 $\pm$ 0.199
Swish-T${}_{\textbf{B}}$ 99.37 $\pm$ 0.023 89.94 $\pm$ 0.189 90.80 $\pm$ 0.096
Swish-T${}_{\textbf{C}}$ 99.31 $\pm$ 0.029 90.03 $\pm$ 0.099 90.80 $\pm$ 0.206

Table 1: Comparison of various activation functions across the MNIST, Fashion
MNIST, and SVHN datasets using the LeNet architecture. The table shows the
mean Top-1 test accuracy and standard deviation of 10 runs.

###  4.2 CIFAR

In this section, we report the results on the popular image classification
benchmark datasets CIFAR-10 [21] and CIFAR-100 [21]. These datasets consist of
60,000 color images of 10 and 100 classes, respectively, with each class
containing 6,000 images of $32\times 32$ pixel size. CIFAR-10 contains 50,000
training images and 10,000 test images, while CIFAR-100 has the same number of
training and test images of 100 classes.

Activation Function RN-18 SF-V2 (1.x) SF-V2 (2.x) SENet-18 EN-B0 MN-V2 DN-121
#Params 11.1M 1.4M 5.5M 11.2M 3.6M 2.3M 1.0M ReLU 95.54 $\pm$ 0.08 91.90 $\pm$
0.24 91.93 $\pm$ 0.21 95.25 $\pm$ 0.32 90.73 $\pm$ 0.13 92.63 $\pm$ 0.19 94.82
$\pm$ 0.16 GELU 94.79 $\pm$ 0.16 93.06 $\pm$ 0.13 93.11 $\pm$ 0.26 94.68 $\pm$
0.10 92.14 $\pm$ 0.15 93.72 $\pm$ 0.17 93.89 $\pm$ 0.21 SiLU 94.22 $\pm$ 0.17
92.03 $\pm$ 0.11 92.34 $\pm$ 0.19 93.99 $\pm$ 0.04 91.19 $\pm$ 0.15 92.59 $\pm
0.16$ 92.75 $\pm$ 0.17 Swish 95.49 $\pm$ 0.07 94.06 $\pm$ 0.15 94.01 $\pm$
0.21 95.41 $\pm$ 0.14 93.46 $\pm$ 0.13 95.08 $\pm$ 0.12 94.89 $\pm$ 0.11 Mish
94.36 $\pm$ 0.12 92.34 $\pm$ 0.14 92.41 $\pm$ 0.08 94.09 $\pm$ 0.13 91.58
$\pm$ 0.14 92.81 $\pm$ 0.21 92.78 $\pm$ 0.19 SMU 95.58 $\pm$ 0.03 94.00 $\pm$
0.17 94.01 $\pm$ 0.12 95.48 $\pm$ 0.07 93.36 $\pm$ 0.10 95.12 $\pm$ 0.11 94.95
$\pm$ 0.20 SMU-1 95.12 $\pm$ 0.16 93.94 $\pm$ 0.19 93.93 $\pm$ 0.14 94.87
$\pm$ 0.15 92.98 $\pm$ 0.15 94.66 $\pm$ 0.13 94.42 $\pm$ 0.18 Swish-T 95.53
$\pm$ 0.14 94.15 $\pm$ 0.14 94.08 $\pm$ 0.21 95.40 $\pm$ 0.13 93.22 $\pm$ 0.13
94.89 $\pm$ 0.12 94.88 $\pm$ 0.18 Swish-T${}_{\textbf{B}}$ 95.17 $\pm$ 0.21
93.99 $\pm$ 0.18 93.91 $\pm$ 0.14 95.32 $\pm$ 0.04 93.04 $\pm$ 0.28 94.72
$\pm$ 0.20 94.71 $\pm$ 0.14 Swish-T${}_{\textbf{C}}$ 95.29 $\pm$ 0.03 94.26
$\pm$ 0.08 94.27 $\pm$ 0.14 95.50 $\pm$ 0.10 93.28 $\pm$ 0.19 94.97 $\pm$ 0.14
94.95 $\pm$ 0.09

Table 2: Comparison of the Swish-T family and various activation functions
across different architectures on the CIFAR-10 dataset. The table shows the
mean Top-1 accuracy and standard deviation of 5 runs.

The models used for evaluation include ResNet-18 (RN-18) [22], ShuffleNetV2
(SF-V2)(1.x, 2.x) [35], SENet-18 [36], EfficientNetB0 (EN-B0) [37],
MobileNetV2 (MN-V2) [38], and DenseNet-121 (DN-121) [39]. For all models, the
batch size is set to 128, and the learning rate is initialized to 0.1. The
learning rate is adjusted using a cosine annealing [34] scheduler, which
gradually decreases the learning rate from the initial value to zero over the
training period. We use the stochastic gradient descent (SGD) [32, 33]
optimizer with a momentum of 0.9 and a weight decay of 5e-4. The models are
trained for a total of 200 epochs. We used standard data augmentation
techniques such as random crop and horizontal flip.

Activation Function RN-18 SF-V2 (1.x) SF-V2 (2.x) SENet-18 EN-B0 MN-V2 DN-121
#Params 11.1M 1.4M 5.5M 11.2M 3.6M 2.3M 1.0M ReLU 78.46 $\pm$ 0.15 71.70 $\pm$
0.41 71.92 $\pm$ 0.45 77.85 $\pm$ 0.28 67.71 $\pm$ 0.49 72.65 $\pm$ 0.24 76.55
$\pm$ 0.39 GELU 78.08 $\pm$ 0.26 75.04 $\pm$ 0.21 74.83 $\pm$ 0.17 76.92 $\pm$
0.14 71.93 $\pm$ 0.34 75.83 $\pm$ 0.30 74.41 $\pm$ 0.32 SiLU 76.82 $\pm$ 0.24
73.19 $\pm$ 0.36 73.71 $\pm$ 0.39 75.59 $\pm$ 0.15 70.17 $\pm$ 0.35 73.49
$\pm$ 0.35 73.40 $\pm$ 0.21 Swish 78.60 $\pm$ 0.28 75.69 $\pm$ 0.17 75.73
$\pm$ 0.19 78.18 $\pm$ 0.20 72.92 $\pm$ 0.18 77.97 $\pm$ 0.16 77.00 $\pm$ 0.35
Mish 77.16 $\pm$ 0.24 73.58 $\pm$ 0.28 73.45 $\pm$ 0.20 75.74 $\pm$ 0.35 71.30
$\pm$ 0.09 73.39 $\pm$ 0.23 73.81 $\pm$ 0.27 SMU 78.85 $\pm$ 0.19 75.36 $\pm$
0.46 75.43 $\pm$ 0.20 78.28 $\pm$ 0.18 72.49 $\pm$ 0.19 77.84$\pm$ 0.16 76.89
$\pm$ 0.22 SMU-1 78.44 $\pm$ 0.26 76.01 $\pm$ 0.20 75.90 $\pm$ 0.23 76.71
$\pm$ 0.26 72.48 $\pm$ 0.11 77.46 $\pm$ 0.14 76.04 $\pm$ 0.10 Swish-T 79.02
$\pm$ 0.24 75.79 $\pm$ 0.23 76.04 $\pm$ 0.30 78.15 $\pm$ 0.29 72.76 $\pm$ 0.43
77.31 $\pm$ 0.14 77.00 $\pm$ 0.14 Swish-T${}_{\text{B}}$ 77.24 $\pm$ 0.99
75.84 $\pm$ 0.29 75.82 $\pm$ 0.29 77.80 $\pm$ 0.31 73.21 $\pm$ 0.19 77.40
$\pm$ 0.13 76.96 $\pm$ 0.12 Swish-T${}_{\text{C}}$ 78.72 $\pm$ 0.15 76.06
$\pm$ 0.36 76.04 $\pm$ 0.30 77.93 $\pm$ 0.20 72.75 $\pm$ 0.30 77.60 $\pm$ 0.26
77.15 $\pm$ 0.19

Table 3: Comparison of the Swish-T family and various activation functions
across different architectures on the CIFAR-100 dataset. The table shows the
mean Top-1 accuracy and standard deviation of 5 runs.

(a) Train and test accuracy

(b) Train and test loss

Figure 3: Train and test curves for ShuffleNetv2 (2.x) on the CIFAR100
dataset. This figure shows the comparison of the performance metrics (Top-1
accuracy and loss) between the Swish-T family and other activation functions.
The shaded areas represent the standard deviation.

(a) SENet-18

(b) DenseNet-121

Figure 4: Average training time for SENet-18 and DenseNet-121 on CIFAR-10
using a single GPU. (Performance metrics can be found in Table 2.)

Tables 2 and 3 demonstrate the performance of different activation functions
on CIFAR-10 and CIFAR-100 datasets across various deep learning architectures.
The Swish-T family of functions consistently outperforms traditional
activation functions like ReLU and GELU. Notably, Swish-T${}_{\textbf{C}}$
achieves the highest accuracy of 94.27% with the ShuffleNetV2 (2.x) model on
CIFAR-10, surpassing ReLU by 2.34%. On the CIFAR-100 dataset,
Swish-T${}_{\textbf{C}}$ also shows superior performance, with an improvement
of 4.12% over ReLU. Fig. 3 illustrates the train and test curves for
ShuffleNetV2 (2.x) on CIFAR-100, providing insights into the effect of various
activation functions, including the Swish-T family, on accuracy and loss over
training epochs.

In DenseNet-121, both SMU and Swish-T${}_{\textbf{C}}$ reach the top accuracy
of 94.95% on CIFAR-10. However, as Fig. 4 illustrates,
Swish-T${}_{\textbf{C}}$ exhibits faster training times compared to SMU. These
results suggest that Swish-T variants not only enhance accuracy across diverse
models and datasets but also offer computational efficiency. Fig 4 further
indicates that the Swish-T family maintains comparable or faster training
times compared to SMU and SMU-1, making them effective choices for practical
applications where both performance and speed are crucial.

##  5 Ablation Study

We run an ablation to analyze the proposed Swish-T${}_{\text{B}}$ and
Swish-T${}_{\text{C}}$ activations. In our study, these Swish-T variants are
utilized without trainable parameters, specifically by fixing the parameter
$\beta$ instead of updating it during training. This approach allows us to
achieve higher performance and faster training speed in some models. The fixed
$\beta$ simplifies the activation function, maintaining its effectiveness
while reducing the complexity involved in training.

###  5.1 Effect of Beta Fixation on ResNet-18 with Swish-T Family

Our experimental results, as shown in Tables 2 and 3, indicate that the
Swish-T family converges at similar beta values in some architectures. As
demonstrated in Table 4, Swish-T${}_{\textbf{C}}$ converges with beta values
ranging from 6.x to 7.x, and both Swish and the Swish-T family also converge
in a similar range. However, in ResNet-18, Swish-T${}_{\textbf{C}}$ and
Swish-T${}_{\textbf{B}}$ converge with beta values around 1.x and do not
exhibit the highest performance among the compared activation functions. We
hypothesized that the simple ResNet model, composed solely of skip-
connections, makes it difficult for beta to reach an optimal point during
training. Therefore, we arbitrarily set beta to 6.0, which we considered the
optimal point based on Table 4. In Swish-T${}_{\textbf{C}-6}$ and
Swish-T${}_{\textbf{B}-6}$, beta is fixed at 6.0 and not updated during
training. When we trained ResNet-18 with Swish-T${}_{\textbf{C}-6}$ and
Swish-T${}_{\textbf{B}-6}$, the results, as shown in Table 5, demonstrated
improved performance compared to previous experiments and higher performance
than those shown in Tables 2 and 3.

Activation Function RN-18 SF-V2 (1.x) SF-V2 (2.x) SENet-18 EF-B0 MN-V2 DN-121
Beta Value Swish 6.43 $\pm$ 0.02 6.57 $\pm$ 0.03 6.57 $\pm$ 0.02 5.96 $\pm$
0.01 6.97 $\pm$ 0.02 7.22 $\pm$ 0.10 6.86 $\pm$ 0.04 Swish-T 6.45 $\pm$ 0.02
6.77 $\pm$ 0.04 6.76 $\pm$ 0.04 5.94 $\pm$ 0.03 7.50 $\pm$ 0.02 7.56 $\pm$
0.03 7.11 $\pm$ 0.06 Swish-T${}_{\text{B}}$ 1.49 $\pm$ 0.02 6.74 $\pm$ 0.05
6.81 $\pm$ 0.03 6.07 $\pm$ 0.01 7.40 $\pm$ 0.20 7.33 $\pm$ 0.02 7.19 $\pm$
0.02 Swish-T${}_{\text{C}}$ 1.52 $\pm$ 0.10 6.77 $\pm$ 0.03 6.76 $\pm$ 0.04
6.16 $\pm$ 0.01 7.30 $\pm$ 0.03 7.38 $\pm$ 0.03 7.14 $\pm$ 0.02

Table 4: Comparison of beta values and standard deviations across different
models trained on CIFAR-100.

These results indicate that the Swish-T family can be used in a non-parametric
manner without training parameters, and by omitting the beta update process,
it can achieve faster and similar or higher performance compared to parametric
functions.

CIFAR-10 Activation Function Beta Value Top-1 Swish 5.81 $\pm$ 0.12 95.49
$\pm$ 0.07 Swish-T${}_{\textbf{B}}$ 1.49 $\pm$ 0.09 95.17 $\pm$ 0.21
Swish-T${}_{\textbf{C}}$ 1.43 $\pm$ 0.05 95.29 $\pm$ 0.06
Swish-T${}_{\textbf{B}-6}$ 6.0 (fiexd) 95.45 $\pm$ 0.14
Swish-T${}_{\textbf{C}-6}$ 6.0 (fiexd) 95.54 $\pm$ 0.22 CIFAR-100 Activation
Function Beta Value Top-1 Top-5 Swish 6.43 $\pm$ 0.02 78.60 $\pm$ 0.28 94.18
$\pm$ 0.14 Swish-T${}_{\textbf{B}}$ 1.49 $\pm$ 0.02 77.23 $\pm$ 0.99 93.18
$\pm$ 0.50 Swish-T${}_{\textbf{C}}$ 1.52 $\pm$ 0.10 78.72 $\pm$ 0.15 94.10
$\pm$ 0.25 Swish-T${}_{\textbf{B}-6}$ 6.0 (fiexd) 78.81 $\pm$ 0.27 94.10 $\pm$
0.06 Swish-T${}_{\textbf{C}-6}$ 6.0 (fiexd) 79.06 $\pm$ 0.04 94.29 $\pm$ 0.07

Table 5: Performance Comparison of ResNet-18 with Different Activation
Functions on CIFAR-10 and CIFAR-100.

##  6 Conclusion

In this work, we proposed the Swish-T family, which improves the Swish
activation function by introducing an adaptive bias based on the input value
$x$. Swish-T combines a Tanh bias, offering superior performance but slower
learning speeds. Swish-T${}_{\textbf{A}}$ simplifies the formula for faster
learning, while Swish-T${}_{\textbf{B}}$ reintroduces the $\beta$ parameter
for better performance metrics. Swish-T${}_{\textbf{C}}$ effectively controls
the bias using $\beta$, achieving stable performance.

Our experimental results demonstrate that the Swish-T family outperforms
traditional activation functions in various deep learning models, with
Swish-T${}_{\textbf{C}}$ showing the best overall performance. Through
ablation studies, we confirmed that Swish-T${}_{\textbf{B}}$ and
Swish-T${}_{\textbf{C}}$ exhibit high performance even when used as non-
parametric functions with magic numbers for $\beta$. Furthermore, they benefit
from fast learning speeds and low memory usage since they do not have
trainable parameters. The Swish-T family maintains zero-centering
characteristics and controls the receptive field in the negative domain,
making it a robust and versatile choice for diverse neural network
architectures. Future research could further optimize these functions for
specific applications.

## References

  * [1] Richard HR Hahnloser, Rahul Sarpeshkar, Misha A Mahowald, Rodney J Douglas, and H Sebastian Seung.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.  Nature, 405(6789):947–951, 2000.
  * [2] Kevin Jarrett, Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun.  What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference on computer vision, pages 2146–2153. IEEE, 2009.
  * [3] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010.
  * [4] Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng.  Rectifier nonlinearities improve neural network acoustic models.  In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
  * [5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, 2015.
  * [6] Dan Hendrycks and Kevin Gimpel.  Gaussian Error Linear Units (GELUs), 2023.
  * [7] Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Searching for Activation Functions, 2017.
  * [8] Ningning Ma, Xiangyu Zhang, Ming Liu, and Jian Sun.  Activate or Not: Learning Customized Activation.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2021.
  * [9] Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey.  ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functions.  Proceedings of the AAAI Conference on Artificial Intelligence, 36(6):6097–6105, June 2022.
  * [10] Diganta Misra.  Mish: A Self Regularized Non-Monotonic Activation Function, 2020.
  * [11] Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey.  Smooth Maximum Unit: Smooth Activation Function for Deep Networks using Smoothing Maximum Technique.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 784–793, 2022.
  * [12] Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.  Maxout Networks, 2013.
  * [13] Eric Alcaide.  E-swish: Adjusting Activations to Different Network Depths, 2018.
  * [14] Marina Adriana Mercioni and Stefan Holban.  P-Swish: Activation Function with Learnable Parameters Based on Swish Activation Function in Deep Learning.  In 2020 International Symposium on Electronics and Telecommunications (ISETC), pages 1–4, 2020.
  * [15] Marina Adriana Mercioni and Stefan Holban.  Soft-Clipping Swish: A Novel Activation Function for Deep Learning.  In 2021 IEEE 15th International Symposium on Applied Computational Intelligence and Informatics (SACI), pages 225–230, 2021.
  * [16] Natinai Jinsakul, Cheng-Fa Tsai, Chia-En Tsai, and Pensee Wu.  Enhancement of Deep Learning in Image Classification Performance Using Xception with the Swish Activation Function for Colorectal Polyp Preliminary Screening.  Mathematics, 7(12), 2019.
  * [17] Mingxing Tan, Ruoming Pang, and Quoc V. Le.  EfficientDet: Scalable and Efficient Object Detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020.
  * [18] Misba Farheen, M Manjushree, and Manish Kumar Pandit.  Skin Cancer Detection using CNN with Swish Activation Function.  2020\.
  * [19] Steffen Eger, Paul Youssef, and Iryna Gurevych.  Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
  * [20] Shengjie Wang, Tianyi Zhou, and Jeff Bilmes.  Bias Also Matters: Bias Attribution for Deep Neural Network Explanation.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6659–6667. PMLR, 09–15 Jun 2019.
  * [21] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition, 2015.
  * [23] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015.  Software available from tensorflow.org.
  * [24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library, 2019.
  * [25] Paul Werbos and Paul John.  Beyond regression : new tools for prediction and analysis in the behavioral sciences /.  01 1974.
  * [26] Stefan Elfwing, Eiji Uchibe, and Kenji Doya.  Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.  Neural Networks, 107:3–11, November 2018.
  * [27] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation Applied to Handwritten Zip Code Recognition.  Neural Computation, 1(4):541–551, 1989.
  * [28] Yann LeCun, Corinna Cortes, and CJ Burges.  MNIST handwritten digit database.  ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
  * [29] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:1708.07747, 2017.
  * [30] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading Digits in Natural Images with Unsupervised Feature Learning.  2011\.
  * [31] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel.  Handwritten digit recognition with a back-propagation network.  Advances in neural information processing systems, 2, 1989.
  * [32] H. Robbins and S. Monro.  A stochastic approximation method.  Annals of Mathematical Statistics, 22:400–407, 1951.
  * [33] J. Kiefer and J. Wolfowitz.  Stochastic Estimation of the Maximum of a Regression Function.  Annals of Mathematical Statistics, 23:462–466, 1952.
  * [34] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [35] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018.
  * [36] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [37] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [38] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [39] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.