Alternate Loss Functions for Classification and Robust Regression Can
  Improve the Accuracy of Artificial Neural Networks
All machine learning algorithms use a loss, cost, utility or reward function
to encode the learning objective and oversee the learning process. This
function that supervises learning is a frequently unrecognized hyperparameter
that determines how incorrect outputs are penalized and can be tuned to improve
performance. This paper shows that training speed and final accuracy of neural
networks can significantly depend on the loss function used to train neural
networks. In particular derivative values can be significantly different with
different loss functions leading to significantly different performance after
gradient descent based Backpropagation (BP) training. This paper explores the
effect on performance of using new loss functions that are also convex but
penalize errors differently compared to the popular Cross-entropy loss. Two new
classification loss functions that significantly improve performance on a wide
variety of benchmark tasks are proposed. A new loss function call smooth
absolute error that outperforms the Squared error, Huber and Log-Cosh losses on
datasets with significantly many outliers is proposed. This smooth absolute
error loss function is infinitely differentiable and more closely approximates
the absolute error loss compared to the Huber and Log-Cosh losses used for
robust regression.
Mathew Mithra Noel, Arindam Banerjee, Geraldine Bessie Amali D, Venkataraman Muthiah-Nakarajan
# Alternate Loss Functions Can Improve the Performance of Artificial Neural Networks

###### Abstract

All machine learning algorithms use a loss, cost, utility or reward function to encode the learning objective and oversee the learning process. This function that supervises learning is a frequently unrecognized hyperparameter that determines how incorrect outputs are penalized and can be tuned to improve performance. This paper shows that training speed and final accuracy of neural networks can significantly depend on the loss function used to train neural networks. In particular derivative values can be significantly different with different loss functions leading to significantly different performance after gradient descent based Backpropagation (BP) training. This paper explores the effect on performance of new loss functions that are more "liberal" or "strict" compared to the popular Cross-entropy loss in penalizing incorrect outputs. Eight new loss functions are proposed and a comparison of performance with different loss functions is presented. The new loss functions presented in this paper are shown to outperform Cross-entropy loss on computer vision and NLP benchmarks.

keywords: Loss Function, Artificial Neural Network, Convolutional Neural Network, Deep Learning +
Footnote †: journal: journal

## 1 Introduction

Artificial Neural Networks (ANNs) are a class of universal function approximators that learn a parametrized approximation to the target function through a Gradient Descent (GD) based optimization process [1]. Thus learning in ANNs is reduced to the problem of learning a finite set of real parameters namely the weights and
biases in the ANN model. Good parameters that result in a good approximation to the target function are computed by minimizing a loss (also known as cost) function that measures the difference average error between ANN outputs and targets. An important aspect of the loss function is that it distills the performance of a ANN model over the entire dataset into a single scalar value [2]. Thus the loss function is a single continuously differentiable real-valued function of all the parameters of the ANN model. The loss function must be continuously differentiable for GD to work. It is reasonable to choose a loss function inspired by statistical estimation theory to compute the most probable parameters given the dataset. Historically the Mean Square Error (MSE) and Cross-entropy from Maximum Likelihood Estimation theory (MLE) are almost universally used to train ANNs. MLE theory assigns to unknown parameters values that maximize the probability of observing the experimental data [3]. The logarithm of the probability of observing the data is often used for convenience and results in the Cross-entropy loss. Thus MLE estimation proceed by maximizing the logarithm of the probability of the observed data as a function of the unknown parameters.

MSE is used for regression problems where continuous real values have to be predicted and Cross-entropy is used for classification problems where the predicted outputs are discrete. Both MSE and Cross-entropy loss functions are derived from MLE theory as already described. MSE is a maximum-likelihood estimate when the model is linear and the noise is Gaussian. However it is frequently used in practice even when these assumptions cannot be justified. For classification problems both MSE and Cross-entropy losses can be used, but learning is generally faster with Cross-entropy as the gradient is larger due to the log function in Cross-entropy loss.

### Mathematical Formalism

Consider a dataset consisting of N training examples: \(D=\{x^{i},y^{i}:i=1..N\}\). Where \(x\in R^{m},y\in R^{n}\) in general for some natural numbers m and n. In the following we motivate and discuss possible loss functions for binary classification problems.

For binary classification problems \(y\in\{0,1\}\). For the binary classification problem, the final layer consists of a single sigmoidal neuron which outputs the probability \(P(y=1|x)\). This probability depends on all the weights and biases in the ANN and is hence denoted by \(h_{\theta}(x)\). To compute the unknown parameters using maximum likelihood estimation theory the probability of the data given the parameters should be first computed.

Assuming that each training pair in the dataset is independent:

\[P(D|\theta)=\prod_{i=1}^{N}P(x^{i},y^{i};\theta)\]
\(\prod_{i=1}^{N}P(x^{i},y^{i};\theta)=\prod_{i=1}^{N}P(y^{i}|x^{i};\theta)P(x^{i})\) by the definition of conditional probability.

Since \(P(x^{i})\) is independent of \(\theta\), maximizing \(P(D|\theta)\) is same as maximizing \(\log(\prod_{i=1}^{N}P(y^{i}|x^{i};\theta))\).

Thus the log-likelihood function to be maximized is

\[\log(\prod_{i=1}^{N}P(y^{i}|x^{i};\theta))=\sum_{i=1}^{N}P(y^{i}|x^{i};\theta)\]

But \(P(y^{i}|x^{i};\theta)=h_{\theta}(x^{i})^{y^{i}}(1-h_{\theta}(x^{i}))^{1-y^{i}}\).

Instead of maximizing the log-likelihood, the negative log-likelihood can be minimized.

This negative log-likelihood after simplification is the loss

\[L(\theta)=-\sum_{i=1}^{N}(y^{i}\log(h_{\theta}(x^{i}))+(1-y^{i})\log(1-h_{ \theta}(x^{i})))\]

Scaling with any positive number does not change the minimum, so for numerical convenience we can divide by N to obtain

\[L(\theta)=-\frac{1}{N}\sum_{i=1}^{N}(y^{i}\log(h_{\theta}(x^{i}))+(1-y^{i}) \log(1-h_{\theta}(x^{i}))))\]

\[L(\theta)=-\frac{1}{N}(\sum_{i=1}^{N}y^{i}\log(\hat{y}^{i})+(1-y^{i})\log(1- \hat{y}^{i})) \tag{1}\]

For notational convenience \(h_{\theta}(x)\) which is an approximation to the random variable y is denoted by \(\hat{y}\). The above binary Cross-entropy loss for a single training pair \((x,y)\) is

\[l(y,\hat{y})=-[y\log(\hat{y})+(1-y)\log(1-\hat{y})] \tag{2}\]
The target assumes exactly one of two possible values namely 0 or 1, so only one term in Eq. 2 is nonzero. When \(y=1\), \(l(y,\hat{y})=-\log(\hat{y})\) and when \(y=0\), \(l(y,\hat{y})=-\log(1-\hat{y})\). Figures 1 and 2 show the variation of BCE loss with final ANN output \(\hat{y}\) when the target y is 0 and 1 respectively. It is observed that the loss is exactly zero when target is equal to the output. Further the loss is differentiable, convex and tends to infinity as the output approaches the incorrect target value. Convexity is desirable

Figure 1: Plot of Binary Cross-entropy loss when the target y = 0.

Figure 2: Plot of Binary Cross-entropy loss when the target y = 1.
to avoid introducing unnecessary local minima and differentiability is required for gradient based BP learning. In the following we propose new loss functions with the following desirable attributes:

* \(l(y,\hat{y})=0\) and \(\hat{y}=y\)
* \(l(0,\hat{y})\) and \(l(1,\hat{y})\) are convex and differentiable functions of \(\hat{y}\)
* \(l(0,0)=l(1,1)=0\)
* \(\lim_{\hat{y}\to 1}l(0,\hat{y})=\infty\) and \(\lim_{\hat{y}\to 0}l(1,\hat{y})=\infty\)

In the above \(0<\hat{y}<1\) since \(\hat{y}\) is the probability \(P(y=1|x)\). For multiclass classification problems \(\hat{y}\) will be a vector with as many dimensions as classes. For C classes, the final layer will a softmax layer with C neurons and the loss function is the sum of the loss functions for each output: \(-\frac{1}{N}\sum_{j=1}^{C}\sum_{i=1}^{N}y_{j}^{i}\log(\hat{y_{j}}^{i})+(1-y_{j }^{i})\log(1-\hat{y_{j}}^{i})\). To compare different loss functions we introduce the following definition.

**Definition**: Consider two loss functions \(l_{1}\) and \(l_{2}\) defined for a single (output,target) pair \((\hat{y},y)\), where \(\hat{y}\) and \(y\) are vectors with C components. We define \(l_{1}\) to be stricter than \(l_{2}\) for some set of values of \(\hat{y}\) and \(y\) if \(\nabla l_{1}(\hat{y},y)\geq\nabla l_{2}(\hat{y},y)\). In the above definition the inequality between gradient vectors is to be interpreted componentwise.

We define \(l_{2}\) to be more lenient than \(l_{1}\), if \(l_{1}\) is stricter than \(l_{2}\). In the following we introduce alternate loss functions that penalize network errors differently than BCE and explore the effect on performance of using stricter and lenient loss functions. The paper attempts to address a fundamental question pertaining to ML algorithms on whether stricter penalties lead to improved performance for the same dataset or whether stricter penalties beyond a certain point lead to reduced performance.

The main contributions of this work are:

* Eight new loss functions that provide better performance than Cross-entropy on benchmarks are proposed
* The effects of using "stricter" and "lenient" loss functions compared to Cross-entropy is explored
* The distinguishing features and advantages of different loss functions are discussed
Hyperparameter tuning is a trick that can be used to improve the performance of ANNs and does not require extra data. However tuning hyperparameters require more computational resources. An alternative is to use state-of-the-art models that are known to provide better performance apriori. For example new activation functions that are known to perform better than popular activation functions like ReLu can be used [4], [5].

## 2 New loss functions

Next we explore alternate loss functions that satisfy properties listed in 1.1. Considering the binary classification problem and assuming that \(0<\hat{y}<1\) and \(y\in\{0,1\}\), the following new loss functions are proposed:

* **M Loss:**\(M(y,\hat{y})=y\left(\frac{1}{\hat{y}}-1\right)\)
* **L Loss:**\(L(y,\hat{y})=\frac{y}{\sqrt{1-(1-\hat{y})^{2}}}-1\)
* **Parametrized M Loss:**\(M_{\alpha}(y,\hat{y})=y\left(\frac{1}{\hat{y}^{\alpha}}-1\right)\)
* **Tan Loss:**\(T(y,\hat{y})=y\tan(\frac{\pi(1-\hat{y})}{2})\)
* **Parametrized L Loss:**\(L_{\alpha}(y,\hat{y})=\frac{y}{\sqrt{1-(1-\hat{y})^{\alpha}}}-1\)
* **Two parameter L Loss:**\(L_{\alpha,\beta}(y,\hat{y})=\frac{y}{\left(1-(1-\hat{y})^{\alpha}\right)^{ \beta}}-1\)
* **Parametrized Log Loss:**\(-y\log_{\alpha}(\hat{y})\)

In the above the loss is zero if y=0 and the ANN is penalized for errors only when y=1. Modified versions of the above losses that penalize the ANN for errors even when y = 0 are given below.
**Full loss functions:**

* **M Loss:**\(M(y,\hat{y})=\frac{y}{\hat{y}}+\frac{1-y}{1-\hat{y}}-1\)
* **L Loss:**\(L(y,\hat{y})=\frac{y}{\sqrt{1-(1-\hat{y})^{2}}}+\frac{1-y}{\sqrt{1-\hat{y}^{2}}}-1\)
* **Tan Loss:**\(T(y,\hat{y})=y\tan(\frac{\pi(1-\hat{y})}{2})+(1-y)\tan(\frac{\pi\hat{y}}{2})\)
* **Sec Loss:**\(S(y,\hat{y})=y\csc(\frac{\pi\hat{y}}{2})+(1-y)\sec(\frac{\pi\hat{y}}{2})-1\)
* **Parametrized M Loss:**\(M_{\alpha}(y,\hat{y})=\frac{y}{\hat{y}^{\alpha}}+\frac{1-y}{(1-\hat{y})^{ \alpha}}-1\)
* **Parametrized L Loss:**\(L_{\alpha}(y,\hat{y})=\frac{y}{\sqrt{1-(1-\hat{y})^{\alpha}}}+\frac{1-y}{ \sqrt{1-\hat{y}^{\alpha}}}-1\)
* **Two parameter L Loss:**\(L_{\alpha,\beta}(y,\hat{y})=\frac{y}{(1-(1-\hat{y})^{\alpha})^{\beta}}+ \frac{1-y}{(1-\hat{y}^{\alpha})^{\beta}}-1\)
* **Parametrized Cross-entropy Loss:**\(-(y\log_{\alpha}(\hat{y})+(1-y)\log_{\alpha}(1-\hat{y}))\)

In the above definitions of new loss functions the constant factor of -1 can be safely ignored since it does not change the derivative of the loss. This bias of -1 has been included only to fix the loss value to be zero when the output is equal to the target and is optional.
Figure 2 shows the difference between Cross-entropy and other loss functions proposed in this paper. To apply the above loss functions to multiclass classification problems, the losses for each output are added together. In particular if \(l(y_{j}^{i},\hat{y}_{j}^{i})\) is the loss for the jth output and ith training pair, the overall loss is \(\frac{1}{N}\sum_{j=1}^{C}\sum_{i=1}^{N}l(y_{j}^{i},\hat{y}_{j}^{i})\) where N is the size of the mini-batch for mini-batch gradient descent.

## 3 Comparison of performance on benchmark datasets

We tested both versions of the M-loss and the L-loss with deep neural networks on different benchmark datasets. We compared the results with the same experiments performed with cross-entropy loss too. We tested these losses on a VGG19 model with Imagenette and CIFAR10 datasets. The Adam optimizer with learning rate of 1e-5 was used for training the model. The model was trained on a Google Colab environment with 50 epochs and 100 steps per epoch. We compared the results among five loss functions - i) Cross Entropy loss, ii) L-loss, iii) M-loss, iv) Full L-loss, and v) Full M-loss. The training and test accuracy values for these five loss functions are shown in Table 1.

Figure 3: Plot of different loss functions when the target y = 1.
We tested the five losses on text datasets and gauged the performance of the models. We used an LSTM-based model to classify text from the Consumer Financial Protection Bureau (CFPB) dataset. The target variable has 10 possible classes. The model contains an embedding layer followed by a spatial dropout, an LSTM layer of 100 units, and a dense layer to output 10 possible classes. We also developed a sentiment analysis model using BERT to classify movie reviews as positive or negative, based on the text of the review. We used the Large Movie Review Dataset [6] that contains the text of 50,000 movie reviews from the Internet Movie Database (IMDB). We used the Smaller BERT model from TensorFlow Hub having 4 hidden layers (Transformer blocks), a hidden size of 512, and 8 attention heads. Adam optimizer was used with this BERT model. The test accuracy values for these five loss functions are shown in Table 2.

\begin{table}
\begin{tabular}{l l l l l}
**Dataset** & **Model** & **Loss** & **Train Acc.** & **Test Acc.** \\ \hline Imagenette & VGG19 & Cross Entropy loss & 0.73 & 0.74 \\ Imagenette & VGG19 & L-loss & 0.75 & **0.77** \\ Imagenette & VGG19 & M-loss & 0.74 & **0.78** \\ Imagenette & VGG19 & Full L-loss & 0.72 & **0.78** \\ Imagenette & VGG19 & Full M-loss & 0.74 & **0.76** \\ CIFAR10 & VGG19 & Cross Entropy loss & 0.5806 & 0.5668 \\ CIFAR10 & VGG19 & L-loss & 0.5791 & **0.5684** \\ CIFAR10 & VGG19 & M-loss & 0.5745 & 0.5666 \\ CIFAR10 & VGG19 & Full L-loss & 0.5827 & **0.571** \\ CIFAR10 & VGG19 & Full M-loss & 0.5762 & **0.5668** \\ \end{tabular}
\end{table}
Table 1: Results on image datasets
The experimental results show that both the variants of the proposed L-loss and M-loss can outperform the cross-entropy loss in most of the cases.

\begin{table}
\begin{tabular}{l l l l} \multicolumn{1}{c}{**Dataset**} & \multicolumn{1}{c}{**Model**} & \multicolumn{1}{c}{**Loss**} & \multicolumn{1}{c}{**Test Acc.**} \\ \hline \hline CFPB & LSTM & Cross Entropy loss & 0.822 \\ CFPB & LSTM & L-loss & **0.843** \\ CFPB & LSTM & M-loss & **0.827** \\ CFPB & LSTM & Full L-loss & **0.839** \\ CFPB & LSTM & Full M-loss & **0.825** \\ IMDB movie review & BERT & Cross Entropy loss & 0.8551 \\ IMDB movie review & BERT & L-loss & **0.8557** \\ IMDB movie review & BERT & M-loss & **0.8539** \\ IMDB movie review & BERT & Full L-loss & **0.8577** \\ IMDB movie review & BERT & Full M-loss & **0.8555** \\ \end{tabular}
\end{table}
Table 2: Results on text datasets

Figure 4: Variation of accuracy with epochs for the Imagenette dataset.
Figure 5: Variation of loss with epochs for the Imagenette dataset.

Figure 6: Variation of accuracy with epochs for the CIFAR10 dataset.
## 4 Conclusion

This paper explored the possible advantages of using alternate loss functions for training ANNs. Eight new loss functions were proposed. Preliminary results were obtained with two of the proposed loss functions. The M and L loss functions proposed in this paper are shown to outperform cross-entropy loss in most of the scenarios for both image and text classification. The fundamental question of whether more aggressive punishments and rewards can significantly improve the performance of ML algorithms remains unanswered and will be considered in future works.
