ErfReLU: Adaptive Activation Function for Deep Neural Network
Recent research has found that the activation function (AF) selected for
adding non-linearity into the output can have a big impact on how effectively
deep learning networks perform. Developing activation functions that can adapt
simultaneously with learning is a need of time. Researchers recently started
developing activation functions that can be trained throughout the learning
process, known as trainable, or adaptive activation functions (AAF). Research
on AAF that enhance the outcomes is still in its early stages. In this paper, a
novel activation function 'ErfReLU' has been developed based on the erf
function and ReLU. This function exploits the ReLU and the error function (erf)
to its advantage. State of art activation functions like Sigmoid, ReLU, Tanh,
and their properties have been briefly explained. Adaptive activation functions
like Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and
Serf have also been described. Lastly, performance analysis of 9 trainable
activation functions along with the proposed one namely Tanhsoft1, Tanhsoft2,
Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf has been shown by
applying these activation functions in MobileNet, VGG16, and ResNet models on
CIFAR-10, MNIST, and FMNIST benchmark datasets.
Ashish Rajanand, Pradeep Singh
# ErfReLU: Adaptive Activation Function for Deep Neural Network

###### Abstract

Recent research has found that the activation function (AF) selected for adding non-linearity into the output can have a big impact on how effectively deep learning networks perform. Developing activation functions that can adapt simultaneously with learning is a need of time. Researchers recently started developing activation functions that can be trained throughout the learning process, known as trainable, or adaptive activation functions (AAF). Research on AAF that enhance the outcomes is still in its early stages. In this paper, a novel activation function 'ErfReLU' has been developed based on the erf function and ReLU. This function exploits the ReLU and the error function (erf) to its advantage. State of art activation functions like Sigmoid, ReLU, Tanh, and their properties have been briefly explained. Adaptive activation functions like Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf have also been described. Lastly, performance analysis of 9 trainable activation functions along with the proposed one namely Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf has been shown by applying these activation functions in MobileNet, VGG16, and ResNet models on CIFAR-10, MNIST, and FMNIST benchmark datasets.

Activation function. Deep learning. Image Classification. +
Footnote †: Corresponding author: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email Email: Email: Email: Email: Email: Email: Email: Email: Email: Email Email: Email: Email: Email: Email: Email: Email: Email: Email: Email Email: Email: Email: Email Email: Email: Email: Email: Email: Email: Email: Email: Email: Email Email: Email: Email: Email Email: Email: Email: Email: Email: Email: Email Email: Email: Email: Email: Email: Email: Email: Email Email: Email: Email Email: Email: Email: Emailmail: Email Email: Email: Email Email: Email: Email: Email: Email: Email: Email: Email: Email Email: Email: Email Email: Email: Email: Email Email: Email Email: Email: Email: Email Email: Email: Email: Email Email: Email Email: Email Email: Email: Email Email: Email: Email Email: Email Email: Email: Email Email: Email: Email: Email: Email Email: Email: Email Email: Email: Email: Email Email: Email: Email: Email: Email: Email Email: Email: Email: Email: Email: Email: Email: Email: Email Email: Email: Email: Email: Email: Email: Email: Email: Email: Email Email: Email: Email: Email: Email: Email: Email Email: Email: Email Email: Email: Email: Email: Email: Email Email: Email: Email Email: Email: Email: Email: Email Email: Email: Email Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email Email: Email: Email Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: Email: E: Email: Email: E
settings(Alkhouly et al., 2021). For example, see Eq. 1 and Fig. 1.

\[sigmoid=\frac{1}{(1+e^{-x})}\quad f^{\prime}(x)=\frac{1}{(1+e^{-x})^{2}} \tag{1}\]

\[tanh(x)=\frac{(e^{2x}-1)}{(e^{2x}+1)}\quad f^{\prime}(x)=1-tanh^{2}(x) \tag{2}\]

The hyperbolic tangent activation function is a scaled version of the sigmoid function. It also has the same S shape (Fig. 2). It is continuous between -1 and +1. This also suffers from a gradient vanishing problem. In contrast to tanh(x), a neural network with arctanh(x) has more severe gradient changes, allowing it to converge more quickly during network training(Sivri et al., 2022). Tanh is defined as Eq.2.

For high positive and small negative values, the Tanh activation function will reach saturation and stop responding to modest changes in the input data.

Therefore, the weight won't update and the gradient will simply fade, making DNN unable to finish its training (Lau and Lim, 2019).

### ReLU

The ReLU (Maniatopoulos and Mitianoudis, 2021), a quicker AF for learning the complex NN, has emerged as the most useful and popular feature. In comparison to the Sigmoid and Tanh AF, it offers greater deep learning performance. The ReLU retains the characteristics of linear models that make gradient-descent methods an easy way to optimize them because it is a nearly complete representation of the linear function. The ReLU AF executes a threshold operation for each input element, where the value is zero for negative arguments and the variable itself for positive arguments. The Equation for ReLU is given as Eq. 3.

\[ReLU=max(0,x)\quad\quad f^{\prime}(x)=max(0,1) \tag{3}\]

Due to its simplicity and reduced training time, the or non-differentiability at zero, which emerges when a significant negative bias is learned and causes the neuron's output to always be zero regardless of the input (Hao et al., 2020).

### Leaky ReLU

LReLU was one of the first rectified-based activation functions built on ReLU. The LReLU function was an attempt to solve the aforesaid ReLU's possible issues. Leaky rectifier activation function enables the small unit of negative value to produce a modest gradient. However, LReLU operates almost equally similar to normal rectifiers. It has a minor effect on network performance.

\[LReLU=\begin{cases}x&if~{}x>0\\ \alpha x&if~{}x\leq 0\end{cases}\quad f^{\prime}(x)=\begin{cases}1&if~{}x>0\\ \alpha&if~{}x\leq 0\end{cases} \tag{4}\]

It cannot be utilized for complicated Classification because of its linearity. For certain of the use cases, it performs worse than Sigmoid and Tanh(Maniatopoulos and Mitianoudis, 2021).

### Exponential linear unit (ELU)

The ELU, which was first described by the Clevert et al.(Clevert et al., 2015), is an activation function that preserves identity for inputs that are positive but takes non-zero values for arguments that are negative. It is given as Eq. 5. Where \(\alpha\) is a hyper-parameter, which defined the value for negative inputs(Kilicarslan and Celik, 2021).

\[ELU=\begin{cases}x&if~{}x\geq 0\\ \alpha(e^{x}-1)&if~{}x<0\end{cases}\quad f^{\prime}(x)=\begin{cases}1&if~{}x \geq 0\\ \alpha e^{x}&if~{}x<0\end{cases} \tag{5}\]

### Silu

The output of the sigmoid function is multiplied with its input in the sigmoid-weighted linear unit (SiLU)(Elfwing et al., 2018) AF as in the output range of (-0.5, \(\infty\)). It is defined as Eq.6. For large input, SiLU works as ReLU AF.

\[SiLU(x)=x\times Sigmoid(x)\quad f^{\prime}(x)=\frac{1+e^{-x}(x+1)}{(1+e^{-x})^ {2}} \tag{6}\]
Zheng et al(Zheng and Wang, 2020) present an AF using inverse hyperbolic tangent similar to Mish, called PATS. It is continuous non-monotonic. They used a parameter to fix the slope in the positive region by updating the parameter. It makes the deep networks more flexible and significantly lowers the risk of overfitting.

\[f(x)=xarctan\big{(}k\pi s(x)\big{)} \tag{7}\]

Where variables s(x) is the sigmoid activation function and k is a constant that brings out from the uniform distribution of the range.

### Swish

It is the combination of the input function with sigmoid AF. Some of the characteristics of the Swish function are smoothness, non-monotonicity, and bounding below and unbounded above (Dasgupta et al., 2021).

\[Swish=sigmoid(x)*x=\frac{X}{(1+e^{-(\beta x)})} \tag{8}\]

\[f^{\prime}(x)=\frac{1+e^{-(\beta x)}(x+1)}{(1+e^{-(\beta x)})^{2}} \tag{9}\]

It is given as Eq. 8 and also written like Swish(x) = x o(x). As the value of \(\beta\) increases toward infinity, it behaves like ReLU AF(Ramachandran et al., 2017).

### E-swish

A SiLU variant is presented by multiplying the SiLU function by a multiplicative coefficient \(\beta\). The \(\beta\) parameter needs to tune because E-Swish doesn't have any trainable parameters. The parameter is determined through a search strategy(Alcaide, 2018). E-swish is defined by Eq.10.

\[E-swish(x)=\alpha xsig(x)\quad f^{\prime}(x)=a\,\frac{1+e^{(-\beta x)}(x+1)}{ (1+e^{(-\beta x)})^{2}} \tag{10}\]

Where hyperparameter \(\beta\) (1\(<\)\(-\)\(\beta\)\(<\)\(-\)2) is pre-initialize or learnable. It is also unbounded like ReLU.

### Linearly Scaled Hyperbolic Tangent (LiSHT)

Linearly Scaled Hyperbolic Tangent (LiSHT) is combined by multiplying the attribute of ReLU and Tanh activation(Roy et al., 2018). Non-monotonicity and not being bound are properties of the LiSHT activation function. It is used to tackle the dying ReLU phenomena.

\[LiSHT(x)=xtanh(x)\quad f^{\prime(x)}=x(1-tanh^{2}(x)+tanh\,(x)) \tag{11}\]

### Mish

Similar to swish Mish AF is also non-monotonic. Identity function, Tanh, and SoftPlus are compounds to make Mish AF. It is based on self-gate property. It resolved the dying ReLU problem by preserving negative value. It doesn't saturate near non-zero values. It provides non-singularity in weight updating due to being continuously differentiable(Misra, 2019).

\[Mish=xtanh\big{(}softplus(x)\big{)} \tag{12}\]

\[f^{\prime}(x)=tanh\big{(}ln(1+e^{x})\big{)}+sec^{2}h\big{(}ln(1+e^{x})\big{)} \frac{x}{1+e^{-x}} \tag{13}\]

## 3 Adaptive activation functions (AAF)

An adaptive activation function integrated using all the AF, whose shapes are acquired during training based on dataset complexity. A classic AF is converted in the adaptive activation function by changing or by adding simple constraints on the network parameters, which update weights. KiseFak et al.(KiseFak et al., 2021) are described a parameterized version of a fixed standard function whose parameter values are learned from data. All adaptive activation functions include a set of trainable parameters that allows AF to adjust shape extremely close to a precisely known function such as ReLU, Sigmoid, and Tanh. In other words, a neural network architecture equipped with AAF behaves similarly to activation functions. For instance, partial derivatives of these new parameters are required to train all the network parameters, including the trainable activation functions. A gradient descent method based on back-propagation is used to update the parameters of AAF. The automatic discovery of parametric activation functions by the PANGAEA, is a genetic parameter tuning method (Bingham and Mikkulainen, 2022).

### TanhSoft

It is a zero-centric, non-monotonic, continuous, and trainable activation function. Biswas et al. have proposed three different tahnsoft AAF based on hyper-parameter (Biswas et al., 2021). These are defined as

\[tanhSoft1=tanh(ax)*Softplus(x) \tag{14}\]

\[f^{\prime}(x)=tanh(ax)\left[accept(ax)ln(1+e^{x})+\frac{1}{1+e^{-x}}\right] \tag{15}\]

\[tanhSoft2=xtanh(\beta e^{xx}) \tag{16}\]

\[f^{\prime}(x)=xsec^{2}h\big{(}ae^{(\beta x)}\big{)}ae^{(\beta x)}+tanh(\beta e ^{xx}) \tag{17}\]

\[tanhSoft3=ln\big{(}1+e^{x}tanh(\delta x)\big{)} \tag{18}\]

\[f^{\prime(x)}=\frac{1+\delta coth(\delta x)}{1+coth(\delta x)e^{-x}} \tag{19}\]

Here, \(a\), \(\beta\), \(\gamma\), and \(\delta\) is hyper-parameter. It is used for controlling the slope of the curve on both the positive and negative axis. TanhSoft with seven baseline activation functions Relu, Leaky ReLU, ELU, Swish, softplus, GeLU, and Mish. TanhSoft outperforms all these activations in different application areas such as object detection, image classification, Machine Translation, and Semantic segmentation.

### SiNN-Sigmoidal Linear Unit (SinLU)

Paul et al. proposed a SiLU-based activation function that removes the wiggle by multiplying a sinusoidal function is (x + sink). SinLU (Paul et al., 2022) AAF is given in Eq.20.

\[SinLU(x)=(x+axis\beta x).\alpha(x) \tag{20}\]

\[f^{\prime}(x)=\ \ \ \frac{e^{-x}(x+axis\beta x)}{(1+e^{-x})^{2}}+\frac{1+ \beta acos\beta x}{1+e^{-x}} \tag{21}\]

Here, \(\alpha\) and \(\beta\) are trainable parameters. The parameter \(\alpha\) is used to get different shape with amplitude from the rectifier linear unit family and \(\beta\) defined the frequency of the sin function. It is also monotonic and differentiable. It affects the self-gated property of SinLU, so AF works differently from other activation functions. Value of the hyper-parameter of this AAF obtained by the grid search method. They compared SinLU activation with basic activation functions like ReLU, GELU. SinLU outperforms other activation on the MNIST dataset using single-layer feed-forward networks. This AAF is computational expensive, and it can be trapped in the local minimum, making learning extremely difficult(Alkhouly et al., 2021).

### TanhLU

New alternatives of Tanh is proposed by integrating Tanh into a linear unit, called TanhLU(Shen et al., 2022), which is motivated by the scaled tanh characteristic and the positive unboundedness of the ReLU. TanhLU is defined as Eq.21.

\[f(x)=atanh(yx)+\beta x\quad f^{\prime}(x)=\alpha ysech^{2}(yx)+\beta \tag{22}\]
Here \(\alpha\), \(\beta\), and \(\gamma\) are the trainable parameters. Shen et al. showed that enlarging the value of the parameter will give a better-weighted gradient. However, if value of \(\beta\) is less than 0.5, it convergences quickly.

### SAAF (Shape Auto-tuning Activation Function)

It is non-monotonic and bounded AF. It can gather negative data to enhance generalization ability which prevents and corrects the output distribution from being scattered in the non-negative region. SAAF (Zhou et al., 2021) is given as

\[SAAF=\frac{x}{\frac{x}{\alpha}+e^{\frac{-x}{\beta}}0<\frac{\beta}{\alpha}<1 \tag{22}\]

\[f^{\prime}(x)=\frac{\alpha^{2}(x+\beta)\ e^{\frac{x}{\beta}}}{\beta\left(xe^{ \frac{-x}{\beta}}+\alpha\right)^{2}} \tag{23}\]

It is less saturated compared to the sigmoid and ReLU AF. A saturated region is one with a modest derivative that reduces variation and information propagation to the next layer. Activations are considered normalized if their mean and variance across samples are within prescribed intervals. SAAF provides zero-means property for convergence and decreases the gradient vanishing issue. It can adaptively adjust the output to a normalized distribution using two trainable parameters \(\alpha\) and \(\beta\).

Serf (Nag & Bhattacharyya, 2021) is inspired by the self-gating property of Swish and Mish, which multiply the output of a non-linear function of an input with the same non-modulated input. Self-gating has an advantage over traditional gating because it only needs a single two-scalar input as opposed to many two-scalar inputs. Serf has properties such as being boundless above, bound beneath, smooth, non-monotonic, and differentiable at every point. Additionally, a limited domain of negative weights are preserved. The serf being unbounded made an effort to get around the saturation issue. Serf mitigates the aforementioned issue in addition to enhancing expressivity and gradient flow by retaining a small portion of negative information. It is a continuous differential AAF.

\[Serf=xerf\left(ln(1+e^{x})\right) \tag{24}\]

\[f^{\prime}(x)=erf\left(ln(1+e^{x})\right)+\frac{2xe^{x-ln^{2}(e^{x}+1)}}{\sqrt {\pi}(e^{x}+1)} \tag{25}\]

### ErfAct and Pserf

ErfAct and Pserf (Biswas et al., 2022) are non-monotonic, continuous, and trainable AF, based on the Gaussian error function. Biswas et al. presented two AAF, by combining the erf function and parametric softplus. It is approximately smooth as ReLU, and outperforms the ReLU AF. Here, Pserf is a parametric version of the Serf Activation function. ErfAct and Pserf are represented by Eq. 26 and Eq. 28 respectively.

\[ErfAct=xerf\left(ae^{\beta x}\right) \tag{26}\]

\[f^{\prime}(x)=\ erf\left(ae^{\beta x}\right)+\frac{2\beta axe^{\beta x-a^{2}e^ {\beta x}}}{\sqrt{\pi}} \tag{27}\]

\[Pserf=xerf\left(\gamma ln\left(1+e^{\beta x}\right)\right) \tag{28}\]

\[f^{\prime}(x)=erf\left(\gamma ln\left(1+e^{\delta x}\right)\right)\ \ +\frac{2 yxe^{\delta-\gamma^{3}ln^{2}(1+e^{\delta x})}}{\sqrt{\pi}(1+e^{\delta x})} \tag{29}\]

### Smish

An adaptive activation function called Smish inherits the non-monotonic properties of the logisl function. Sigmoid(x) is used to reduce the range of values. The logarithm operation is applied to obtain a smooth curve and a flat trend. Smish then multiplies its Tanh operation by x. It has no upper bound and ranges from (-0.25 to \(\alpha\)). For the larger negative values, it works like the ReLU activation function(Wang et al., 2022).

\[f(x)=x\cdot P(x) \tag{30}\]

\[f^{\prime}(x)=\text{P}(x)+\frac{xe^{x}sech^{2}\left(ln(1+sigmoid(x))\right)}{2 e^{2x}+3e^{x}+1} \tag{31}\]

\[where\quad P(x)=tanh\left[ln(1+sigmoid(x))\right] \tag{32}\]

### IpLU

IpLU (Wu et al., 2021) is a parametric version of the softsign function. Unlike Tanh, softsign has polynomial convergence, whereas the Tanh function has exponential convergence. It is used in the negative region and a parameter \(\alpha\) is used to get smaller slop. The Linear function is used to avoid exploding gradient problems. It is zero centric, monotonic and its derivative is also zero centric.

\[f(x)=\begin{cases}x&ifx\geq 0\\ \frac{x}{1+|x|^{\alpha}}&ifx<0\end{cases} \tag{33}\]

\[f^{\prime}(x)=\begin{cases}1&ifx\geq 0\\ \frac{1+|x|^{\alpha}+x^{2}\alpha|x|^{(\alpha-2)}}{(1+|x|^{\alpha})^{2}}&ifx<0 \end{cases} \tag{34}\]

## 4 Proposed Activation function

After extensive literature survey, we obtained that most of the AAFs are reducing the effect of Dying ReLU phenomena at negative values(Zhou et al., 2021)(Shen et al., 2022)(Nag & Bhattacharyya, 2021)(Wang et al., 2022). ReLU activation, in which the negative part is not being considered. We came up with an activation function, which not only solved the dying ReLU by applying the function at the negative side (x<0) but also removes the negative saturation of sigmoid. It is achieved by piecewise combination of Erf and ReLU functions, called ErfReLU. The use of the erf function is inspired by the ErfAct activation function Eq.19. Proposed ErfReLU is a monotonic, zero-centered activation function. Erf is a trainable AF which is an approximation of ReLU with having negative size of ReLU. Biswas et al. (Biswas et al., 2022) have used of & Pserf with \(\alpha\),\(\beta\),\(\gamma\), and \(\delta\) as their parameter that controls the slop of the curve. The proposed AAF only uses one parameter. The Erf function is the gauss error function. It is bound below, which is represented in Eq.35.

\[erf(x)=\frac{2}{\sqrt{\pi}}{\int_{0}^{x}}e^{-t^{2}}\ dt \tag{35}\]

The derivative of the above function is given as

\[erf^{\prime}(x)=\frac{2}{\sqrt{\pi}}e^{-x^{2}} \tag{36}\]

ErfReLU is defined as

\[f(x)=\begin{cases}x&if\ x\geq 0\\ \alpha\operatorname{erf}(x)&if\ x<0\end{cases} \tag{37}\]
The derivative of ErfReLU is defined as Eq.23.

\[f^{\prime}(x)=\begin{cases}1&\text{if }x\geq 0\\ \frac{a2e^{-x^{2}}}{\sqrt{\pi}}&\text{if }x<0\end{cases} \tag{38}\]

Here \(\alpha\) is a parameter, used to control the slope of the function. The graph of ErfReLU and its derivative is shown in fig.10. It is continuous and above unbounded but bounded below AAF. The positive region of the ErfReLU function works as the ReLU activation function. It preserves a small amount of negative information to prevent the occurrence of the Dying ReLU phenomena and achieves the better accuracy with only one trainable parameter.

### Properties of the Proposed Function

* Smoothness: The Gaussian error function and ReLU are differentiable everywhere, so ErfReLU is also differentiable, which makes it a smooth function.
* Non-linearity: The Gaussian error function is a non-linear function, which introduces non-linearities into the network that helps to learn more complex functions and relationships between the input and output.
* Saturation: The Gaussian error function can become saturated for large input values, meaning that the output of the function approaches a constant value as the input becomes very large. However, ReLU is not saturated for large input(Nag & Bhattacharyya, 2021). This exploits the benefits of both Erf and ReLU to proposed ErfReLU to learn complex functions
* Boundedness: ErfReLU is lower bound as it is using the erf function. There is upper boundedness in the ErfReLU, due to owing the properties of the ReLU AF(Shen et al., 2022).

## 5 Experiments and Results

In this section, we describe the experiments performed on the above-defined adaptive activation functions on diverse datasets using different deep learning models. We compared our suggested activations to 9 prominent trainable and state of art activation functions on deep learning application areas such as image classification and machine translation. The experimental data shows that ErfReLU has significant improvement over other AAF and traditional activations in the deep learning problem. We examine the performance by simply switching the activation functions while maintaining all other network and training parameter constants.

In the adaptive activation function, the value of the parameter is depending upon input dataset, it used for updating the activation function and its shape. MobileNetV1, ResNet18, VGG16 and machine translation seq2seq model are used in performance analysis for AAFs. The experiments are performed using the PyTorch framework over a desktop system (Nvidia Quardo Rtx 5000 GPU had 128 GB RAM) consisting of 16 GB GPU.

### Image Classification

We first evaluate our proposed activation functions on diverse image classification dataset such as the MNIST, FMNIST, and CIFAR10. We conduct comparative studies on two types of representative architectures, a large model (like VGG and ResNet) and a lightweight model (MobileNet), maintaining all parameter constants. MobileNet V1 is a lightweight version of the traditional convolutional layers called "depthwise separable convolution" to reduces the number of parameters, memory consumption, and computation.

The ResNet 18 architecture consists of 18 layers, including convolutional layers, batch normalization layers, and rectified linear unit (ReLU) activation layers. The residual connections help to preserve the gradients flowing through the network, allowing the network to learn effectively even when it has many layers. The architecture of VGG16 is composed of a series of convolutional layers, followed by max-pooling layers, and several fully connected layers. We used the Cross Entropy Loss function for error calculation and classification of images. Adam optimizer is used with a learning rate of 0.001. We train the model for 100 epochs with batch size 128. Transformation is applied on the image with the size of 32 and 4-pixel zero-padding.

#### 5.1.1 Cifar-10

CIFAR-10 is a dataset of size 60,000 32x32 colour images in 10 classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images. The classes are airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Table 1 provides the classification accuracy of different models on the defined adaptive activation function. Our proposed activation function shows significant improvement over other adaptive activation based on several observations on MobileNet and ResNet. ErfReLU trounces all other AAFs. It achieves highest accuracy of 92.78% and 94.04% over Mobilenet and Resnet models, respectively. Only on VGG16 model, ErfAct achieved greater accuracy than ErfReLU. Table 2 provides the categorization outcomes for CIFAR10 as determined by the activation functions ReLU, Swish, Mish, LiSHT and ErfReLU. The ErfReLU outperforms the other defined the state-of-the-art activation function over Mobilenet and Resnet models. It also shows significant improvement over Swish, LiSHT, activation functions on VGG16 model, however ReLU has better performance.

\begin{table}
\begin{tabular}{c c c c c c} \hline  & \multicolumn{2}{c}{**Activati**} & \multicolumn{1}{c}{**Parameter**} & \multicolumn{1}{c}{**MobileN**} & \multicolumn{1}{c}{**ResN**} & \multicolumn{1}{c}{**VGG**} \\
**S/** & **on** & **er** & **et** & **et** & **16** \\
**N.** & **function** & & & & \\ \hline
**1** & \begin{tabular}{c} Tanhsoft \\ 1 \\ \end{tabular} & 
\begin{tabular}{c} \(\alpha\): 0.87 \\ \(\beta\): 0.75, \\ \(\gamma\): 0.75 \\ \end{tabular} & 92.24 & 93.67 & 92.58 \\
**2** & \begin{tabular}{c} Tanhsoft \\ 2 \\ \end{tabular} & 
\begin{tabular}{c} \(\beta\): 0.75, \\ \(\gamma\): 0.75 \\ \end{tabular} & 92.39 & 93.44 & 92.87 \\
**3** & \begin{tabular}{c} Tanhsoft \\ 3 \\ \end{tabular} & 
\begin{tabular}{c} \(\alpha\): 0.85 \\ \(\alpha\): 1, \(\beta\): \\ \end{tabular} & 92.45 & 93.43 & 92.57 \\
**4** & TanhLU & 0.5, & 90.86 & 90.35 & 89.84 \\  & & 
\begin{tabular}{c} \(\gamma\)=2 \\ \end{tabular} & 91.64 & 90.58 & 86.28 \\
**5** & SAAF & 
\begin{tabular}{c} \(\alpha\): 0.75, \\ \(\beta\): 0.75 \\ \end{tabular} & 92.52 & 93.82 & 93.24 \\
**7** & \begin{tabular}{c} Pserf \\ \end{tabular} & 
\begin{tabular}{c} \(\alpha\): 1.25, \\ \(\beta\): 0.85 \\ \end{tabular} & 92.67 & 93.84 & 92.76 \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of testing accuracy of different adaptive activation functions on the CIFAR10 dataset.

Figure 10: Proposed ErfReLU activation function and its derivative.
#### 5.1.2 Mnist

The MNIST (Biswas et al., 2022) dataset of handwritten digits contains 70K images (60k for training and 10k for testing purposes), and each image size is a 32*32-pixel grayscale image. Table 3 shows the comparative result of AAF with proposed. It can be seen in table 4, the ErfReLU shows significant improvement over the state-of-the-art activation function.

#### 5.1.3 Fmnist

The FMNIST(Biswas et al., 2022) dataset consists of 70,000 handwritten digit images, with 60,000 images allocated for training and 10,000 for testing. Each image is a 32*32-pixel grayscale image. As presented in Table 5, the ErfReLU activation function outperforms the other activation functions when used with Mobilenet and Resnet models. However, in VGG16, the Smish adaptive activation function displays better performance compared to other adaptive activation functions. Moreover, Table 6 demonstrates that the ErfReLU activation function demonstrates a significant improvement over the state-of-the-art activation function.

### _Machine Translation_

Machine translation is a type of technology that automatically translates text from one language to another. It is used in a variety of application areas such as website localization, document translation, and chatbots. The validation of AAF's effectiveness in machine translation has been carried out through English-to-German translation experiments, utilizing the standard Multi30k dataset for training and testing. This dataset comprises 29k sentences for training, as well as 1k each for validation and testing purposes. A Long Short-Term Memory (LSTM) auto-encoder network

\begin{table}
\begin{tabular}{l l l l l} \hline \hline \multicolumn{2}{c}{**FMNIS T**} & \multicolumn{2}{c}{**Activatio n function**} & \multicolumn{2}{c}{**MobileNe**} & \multirow{2}{*}{ResNet} & \multicolumn{1}{c}{**VGG16**} \\ \cline{1-1} \multicolumn{1}{c}{**T**} & \multicolumn{1}{c}{**n**} & & & \multicolumn{1}{c}{**n**} & \multicolumn{1}{c}{**n**} \\ \hline
1. & ReLU & 92.59 & 93.73 & 93.7 \\
2. & Swish & 92.74 & 93.42 & 92.8 \\
3. & Mish & 92.55 & 93.56 & 93.1 \\
4. & LiSHT & 90.59 & 92.38 & 93.2 \\
5. & **ErfReLU** & **92.78** & **93.72** & **93.3** \\ \hline \hline \end{tabular}
\end{table}
Table 6: The testing accuracy of various state-of-the-art activation functions are compared on the FMNIST dataset.

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline \multicolumn{2}{c}{**ACtivati**} & \multicolumn{2}{c}{**Activa**} & \multicolumn{2}{c}{**MobileNe**} & \multirow{2}{*}{**ResNet**} & \multicolumn{1}{c}{**VGG1**} \\ \multicolumn{2}{c}{**0**} & & \multicolumn{1}{c}{**n function**} & & & \multicolumn{1}{c}{**n**} \\ \hline
1. & \multicolumn{2}{c}{ReLU} & \multicolumn{1}{c}{90.13} & 93.63 & 93.9 \\
2. & \multicolumn{2}{c}{Swish} & \multicolumn{1}{c}{90.84} & 93.64 & 92.8 \\
3. & \multicolumn{2}{c}{Mish} & \multicolumn{1}{c}{90.78} & 93.43 & 93.1 \\
4. & \multicolumn{2}{c}{LiSHT} & \multicolumn{1}{c}{85.92} & 90.15 & 86.5 \\
5. & \multicolumn{2}{c}{**ErfReLU**} & \multicolumn{1}{c}{**92.78**} & \multicolumn{1}{c}{**94.04**} & 93 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of testing accuracy of our proposed AAF with different state of art activation functions over CIFAR10 dataset.

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline \multicolumn{2}{c}{**MNIST**} & \multicolumn{2}{c}{**Activatio n function**} & \multicolumn{2}{c}{**MobileNe**} & \multirow{2}{*}{**ResNet**} & \multicolumn{1}{c}{**VGG16**} \\ \multicolumn{2}{c}{1} & \multicolumn{2}{c}{ReLU} & \multicolumn{1}{c}{98.92} & 99.01 & 99.4 \\
2. & \multicolumn{2}{c}{Swish} & \multicolumn{1}{c}{99.05} & 99.09 & 99.6 \\
3. & \multicolumn{2}{c}{Mish} & \multicolumn{1}{c}{90.78} & 93.64 & 92.8 \\
3. & \multicolumn{2}{c}{Mish} & \multicolumn{1}{c}{90.78} & 93.43 & 93.1 \\
4. & \multicolumn{2}{c}{LiSHT} & \multicolumn{1}{c}{85.92} & 90.15 & 86.5 \\
5. & \multicolumn{2}{c}{**ErfReLU**} & \multicolumn{1}{c}{**92.78**} & \multicolumn{1}{c}{**94.04**} & 93 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison of testing accuracy of different adaptive activation functions on the FMNIST dataset.

\begin{table}
\begin{tabular}{l l l l l l} \hline \multicolumn{2}{c}{**CIFER1**} & \multicolumn{2}{c}{**Activa**} & \multicolumn{2}{c}{**MobileNe**} & \multirow{2}{*}{**ResNet**} & \multicolumn{1}{c}{**VGG1**} \\
**0** & & \multicolumn{1}{c}{**n function**} & & & \multicolumn{1}{c}{**6**} \\ \hline
1. & \multicolumn{2}{c}{ReLU} & \multicolumn{1}{c}{90.13} & 93.63 & 93.9 \\
2. & \multicolumn{2}{c}{Swish} & \multicolumn{1}{c}{90.84} & 93.64 & 92.8 \\
3. & \multicolumn{2}{c}{Mish} & \multicolumn{1}{c}{90.78} & 93.43 & 93.1 \\
4. & \multicolumn{2}{c}{LiSHT} & \multicolumn{1}{c}{85.92} & 90.15 & 86.5 \\
5. & \multicolumn{2}{c}{**ErfReLU**} & \multicolumn{1}{c}{**92.78**} & \multicolumn{1}{c}{**94.04**} & 93 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of testing accuracy of different AAF on the MNIST dataset.

\begin{table}
\begin{tabular}{l l l l l l} \hline \multicolumn{2}{c}{**CIFER10**} & \multicolumn{2}{c}{**Activa n function**} & \multicolumn{2}{c}{**MobileNe**} & \multirow{2}{*}{**ResNet**} & \multicolumn{1}{c}{**VGG1**} \\
**0** & & \multicolumn{1}{c}{**n function**} & & & \multicolumn{1}{c}{**6**} \\ \hline
1. & \multicolumn{2}{c}{ReLU} & \multicolumn{1}{c}{90.13} & 93.63 & 93.9 \\
2. & \multicolumn{2}{c}{Swish} & \multicolumn{1}{c}{90.84} & 93.64 & 92.8 \\
3. & \multicolumn{2}{c}{Mish} & \multicolumn{1}{c}{90.78} & 93.43 & 93.1 \\
4. & \multicolumn{2}{c}{LiSHT} & \multicolumn{1}{c}{85.92} & 90.15 & 86.5 \\
5. & \multicolumn{2}{c}{**ErfReLU**} & \multicolumn{1}{c}{**92.78**} & \multicolumn{1}{c}{**94.04**} & 93 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of testing accuracy of our proposed AAF with different state of art activation functions over CIFAR10 dataset.
is utilized as the benchmark Seq2Seq model for English-to-German translation. The encoder and decoder have an embedding size of 300, and both are equipped with a dropout factor of 0.5.The language translation model was trained for 25 epochs using a learning rate of 0.001 and a batch size of 256, while utilizing the Adam optimizer for cross-entropy loss training(Dubey et al., 2022).

The network's performance on the dataset was evaluated using the BiLingual Evaluation Understudy (BLEU) score measure. Table 7 illustrates the BLEU score achieved by the adaptive activation function. When compared to other adaptive activation functions, the ErfReLU activation function displayed a notable improvement in performance. Nevertheless, the SAAF activation function exhibited superior performance compared to other activation functions.

## 6 Conclusion

In this paper, adaptive activation functions that have been explored, the function that focuses on the ones that can be trained, are studied. These activation function attempts to avoid issues that arise during the training and optimization processes, such as learning saturation, exploding vanishing gradients, and dying problems. The control parameters are used in adaptive AFs to alter the efficiency of nonlinear transformations. We proposed an activation function, called ErfReLU, which combines the attribute of ReLU and Erf gives better convergence than state of art AF such as ReLU and other adaptive activation functions. An experimental analysis of nine adaptive AFs is performed, which can be used in many different areas of Deep learning. The CIFAR10, MNIST, and FMNIST dataset is used to evaluate the effectiveness of the adaptive AFs with MobileNet, ResNet, and Vgg16. Machine translation is performed on a multi3k dataset on an LSTM-based autoencoder. In general, the adaptive AFs show better convergence as they can adapt the data faster by learning the parameter from the data. ErfReLU has shown better convergence in all nine adaptive activation functions, although in some cases, Pserf showed better convergence.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
