Empirical Loss Landscape Analysis of Neural Network Activation Functions
Activation functions play a significant role in neural network design by
enabling non-linearity. The choice of activation function was previously shown
to influence the properties of the resulting loss landscape. Understanding the
relationship between activation functions and loss landscape properties is
important for neural architecture and training algorithm design. This study
empirically investigates neural network loss landscapes associated with
hyperbolic tangent, rectified linear unit, and exponential linear unit
activation functions. Rectified linear unit is shown to yield the most convex
loss landscape, and exponential linear unit is shown to yield the least flat
loss landscape, and to exhibit superior generalisation performance. The
presence of wide and narrow valleys in the loss landscape is established for
all activation functions, and the narrow valleys are shown to correlate with
saturated neurons and implicitly regularised network configurations.
Anna Sergeevna Bosman, Andries Engelbrecht, Marde Helbig
# Empirical Loss Landscape Analysis of Neural Network Activation Functions

###### Abstract.

Activation functions play a significant role in neural network design by enabling non-linearity. The choice of activation function was previously shown to influence the properties of the resulting loss landscape. Understanding the relationship between activation functions and loss landscape properties is important for neural architecture and training algorithm design. This study empirically investigates neural network loss landscapes associated with hyperbolic tangent, rectified linear unit, and exponential linear unit activation functions. Rectified linear unit is shown to yield the most convex loss landscape, and exponential linear unit is shown to yield the least flat loss landscape, and to exhibit superior generalisation performance. The presence of wide and narrow valleys in the loss landscape is established for all activation functions, and the narrow valleys are shown to correlate with saturated neurons and implicitly regularised network configurations.

neural networks, activation functions, loss landscape, fitness landscape analysis +
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science

+
Footnote †: journal: Computer Science
Background

This study is concerned with the relationship between the activation functions employed in a NN architecture, and the resulting NN loss landscape properties. To that extent, the following topics are covered: Section 2.1 discusses the activation functions and their relevance. Section 2.2 provides a literature review of the existing loss landscape studies in the context of activation functions. Section 2.3 discusses the fitness landscape analysis techniques employed in this study.

### Activation functions

An activation function is applied to the _net_ input signal received by a neuron in order to decide whether the neuron should fire or not. Non-linear activation functions introduce non-linearity to NNs, and thus enable the universal function approximation properties (Kraus et al., 2017).

Activation functions fall into two categories: bounded and unbounded. One of the earliest smooth bounded activation functions is the sigmoid (S
is determined by using the negative gradient of the loss function. For a step \(\vec{x}_{l}\), the gradient vector \(\vec{g}_{l}\) is calculated. Then, \(\vec{g}_{l}\) is used to generate a binary mask \(\vec{b}_{l}\):

\[b_{lj}=\begin{cases}0&\text{if }g_{lj}>0,\\ 1&\text{otherwise},\end{cases}\]

where \(j\in\{1,\dots,m\}\) for \(\vec{g}_{l}\in\mathbb{R}^{m}\). The next step \(\vec{x}_{l+1}\) is obtained using the progressive random walk (PRW) (Kang et al., 2017). A single step of PRW constitutes randomly generating a step vector \(\Delta\vec{x}_{l}\in\mathbb{R}^{m}\), such that \(\Delta x_{lj}\in[0,\epsilon]\ \forall j\in\{1,\dots,m\}\), and determining the sign of each \(\Delta x_{lj}\) using the corresponding \(b_{lj}\) as

\[\Delta x_{lj}:=\begin{cases}-\Delta x_{lj}&\text{if }b_{lj}=0,\\ \Delta x_{lj}&\text{otherwise}.\end{cases}\]

To summarise, PGW randomises the magnitude of \(\Delta\vec{x}_{l}\) per dimension, and sets the direction according to \(\vec{g}_{l}\). Therefore, gradient information is combined with stochasticity, generating \(X^{\prime}\) that explores the areas of low error.

#### 2.3.2. Loss-gradient clouds

Loss-gradient clouds (LGCs) were first introduced in (Kang et al., 2017) as a visualisation tool for the purpose of empirically establishing the presence and characteristics of optima in NN error landscapes. To construct LGCs, the weight space of a NN is sampled. Once an appropriate number of samples has been obtained, LGC is generated by constructing a scatter plot with loss (i.e. error) values on the \(x\)-axis, and gradient magnitude (i.e. norm) on the \(y\)-axis. Points of zero error and zero gradient correspond to global minima. Points of non-zero error and zero gradient correspond to local minima or saddle points, i.e. suboptimal critical points. The local convexity of the points can be further identified using Hessian matrix analysis: positive eigenvalues of the Hessian indicate convex local minima, and a mixture of positive and negative eigenvalues indicate saddle points.

LGCs were successfully used to investigate the minima associated with different loss functions (Kang et al., 2017) as well as NN architectures (Kang et al., 2017). This study is a natural extension of (Kang et al., 2017; Kang et al., 2017), where the focus is shifted to the activation functions and their effect on the loss landscapes.

## 3. Experimental Procedure

This study aimed to empirically investigate the modality, i.e. local minima and their properties, associated with different activation functions in the hidden layer of a NN. Feed-forward NNs with a single hidden layer and the cross-entropy loss function were used in the experiments. As previously discussed in Section 2.1, TanH, ReLU, and ELU were considered for the hidden layer. For the output layer neurons, the sigmoid activation function was used. Section 3.1 below lists the benchmark problems used, and Section 3.2 outlines the sampling parameters.

### Benchmark problems

Four real world classification problems of varied dimensionality and complexity were used in the experiments. Table 1 summarises the NN architecture parameters and corresponding NN dimensionality per dataset. Sources of each dataset and/or NN architectures are also specified. The following datasets were considered in this study:

1. **XOR:** Exclusive-or (XOR) refers to the XOR logic gate, modelled by a NN with two hidden neurons. The dataset consists of 4 binary patterns only, but is not linearly separable.
2. **Iris:** The Iris data set (Kang et al., 2017) is a collection of 50 samples per three species of irises: _Setosa_, _Versicolor_, and _Virginica_, comprising 150 patterns.
3. **Heart:** This is a binary classification problem comprised of 920 samples, where each sample is a collection of various patient readings pertaining to heart disease prediction (Kang et al., 2017).
4. **MNIST:** The MNIST dataset (Kang et al., 2017) comprises 70000 examples of handwritten digits from 0 to 9. Each digit is stored as a \(28\times 28\) grey scale image. The 2D input is flattened into a 1D vector for the purpose of this study.

For all problems except XOR, the inputs were \(z\)-score standardised. Binary classification problems used binary output encoding, while multinomial problems were one-hot encoded. All code used is available at [https://github.com/annabosman/fla-in-tf](https://github.com/annabosman/fla-in-tf).

### Sampling parameters

PGW was used to sample the areas of low error. To ensure adequate solution space coverage, the number of independent PGWs was set to \(10\times m\), where \(m\) is the dimensionality of the problem. Since gradient-based methods are sensitive to the starting point (Kang et al., 2017), two initialisation ranges for PGW were considered: \([-1,1]\) and \([-10,10]\). Malan and Engelbrecht (Malan and Engelbrecht, 2018) observed that the step size of the walk influences the resulting FLA metrics. As such, two step size settings were adopted in this study: maximum step size = 1% of the initialisation range (micro), and maximum step size = 10% of the initialisation range (macro). Micro walks were executed for 1000 steps, and macro walks were executed for 100 steps.

For all problems with the exception of XOR, the 80/20 training/testing split was used. The gradient (\(G_{t}\)) and the error (\(E_{t}\)) of the current PGW point was calculated using the training set. The generalisation error (\(E_{g}\)) was calculated on the test set for each point on the walk. For MNIST, a batch size of 100 was used. For the remainder of the problems, full batch (entire train/test subset) was used. To identify minima discovered by PGWs, the \(G_{t}\) magnitude and \(E_{t}\) value were recorded per step. Eigenvalues of the Hessian were calculated for all problems, except MNIST (due to computational constraints), to determine if a PGW point is convex, concave, saddle, or singular.

## 4. Empirical Study of Modality

Empirical results of the study are presented in this section. Sections 4.1 to 4.4 provide per-problem analysis of the three hidden neuron activation functions.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Problem** & \# Input & \# Hidden & \# Output & Dimensionality & Source \\ \hline XOR & 2 & 2 & 1 & 9 & (Kang et al., 2017) \\ Iris & 4 & 4 & 3 & 35 & (Kang et al., 2017) \\ Heart & 32 & 10 & 1 & 341 & (Kang et al., 2017) \\ MNIST & 784 & 10 & 10 & 7960 & (Kang et al., 2017) \\ \hline \hline \end{tabular}
\end{table}
Table 1. Benchmark Problems and the NN Architectures
### Xor

Fig. 1 and 2 show the LGCs for the XOR problem under the micro and macro settings. The sampled points are separated into panes according to curvature. Fig. 1 shows that the activation functions have all yielded four stationary attractors (points of zero gradient), three of which constituted local minima. However, the loss landscape characteristics around the local minima varied for the three activation functions.

TanH exhibited a landscape with a clear transition from the saddle to the convex curvature, i.e. saddle and convex curvatures did not overlap. PGWs generally descended to a stationary point in the saddle space, and then made the transition to one of the convex minima. The three convex minima discovered were disconnected, i.e. the walks did not make transitions between the convex minima.

For ReLU (Fig. 1b) there was no clear separation between convex, saddle, and indefinite curvatures in the different areas sampled. Even though two convex local minima were discovered, these minima were evidently less attractive to PGWs than the global minimum. This observation indicates that a gradient-based algorithm is less likely to become trapped in a local minimum. The reason for this is probably the presence of saddle and indefinite (flat) points around the local minima, providing pathways through which a gradient-based algorithm can escape. Additionally, vertical clusters are observed around the local minima, indicating that diverse gradient information was available in the neighbourhood of the local minima. The predominance of indefinite curvature is explained by the fact that the ReLU function outputs zero for all negative inputs, which inevitably causes flatness.
landscape associated with ReLU was rugged and inconsistent when observed at a larger scale. Fewer walks have discovered convex global minima. Most points exhibited flatness. Thus, although ReLU offers a higher chance of escaping local minima, searchability of the loss landscape suffers, and becomes more dependent on the chosen step size.

ELU (see Fig. 2c) exhibited more evident structure than ReLU, and stronger gradients than TanH. Compared to TanH and ReLU, ELU did not exhibit any flatness, i.e. indefinite Hessians, which indicates that the loss landscape associated with ELU was more searchable.

Fig. 3 shows some of the LGGs obtained for the \([-10,10]\) PGWs. The \(x\)-axis is shown in square-root scale for readability. The same four stationary attractors were sampled for TanH, with two convex local minima. ReLU and ELU both yielded much higher errors than TanH, although convergence to the global minimum still took place. ReLU sampled the most flat curvature points, and ELU sampled the most saddle points. Both ReLU and ELU also exhibited two clusters: points of low error and high gradient, and points of low gradient and high error. These are attributed to the narrow and wide valleys present in the landscape, which could not be observed on the smaller scale. Out of the three activation functions, ELU exhibited the least number of indefinite curvature points. Thus, ELU yielded the most searchable loss landscape for the XOR problem.

### Iris

Fig. 4 shows the LGGs obtained for the Iris problem sampled with the \([-1,1]\) micro walks. The LGGs for the three activation functions exhibited a similar shape, but different curvature properties. Only one major attractor around the global minimum was discovered.

Fig. 4a shows that TanH was dominated by the saddle curvature. The sampled points were split into two clusters around the global minimum, namely points of higher error and lower gradient, and points of higher gradient and low error. The points of high gradient and low error overlapped with the points of indefinite curvature, indicating that this cluster exhibited flatness. Note that the flatness, i.e. lack of curvature in a particular dimension indicates that the corresponding weight did not contribute to the final prediction of the NN. For example, if a neuron is saturated, i.e. the neuron always outputs a value close to the asymptote, then the contribution of a single weight may become negligible. Therefore, indefinite curvature is likely associated with saturated neurons. However, neuron saturation is not the only explanation for non-contributing weights: if certain weights are unnecessary for the minimal solution to the problem at hand, then techniques such as regularisation can be employed to reduce the unnecessary weights to zero. Therefore, solutions with non-contributing weights can also be associated with regularised models. Smaller (regularised) architectures are

Figure 4. LGGs for the micro PGWs for Iris.
"embedded" in the weight space of larger NN architectures, and can therefore be discovered by optimisation algorithms by setting the unnecessary weights to zero [24]. A hypothesis is made that flat areas surrounding the global attraction basin correspond to the non-contributing weights, indicative of saturation. Some of the solutions with saturated neurons correspond to the minima that require fewer weights than available in the architecture. This hypothesis has the implication that the steep gradient attractor associated with indefinite curvature contains points of both poor and good performance, corresponding to unwanted saturation and implicit regularisation, respectively.

For ReLU, the prevalence of convex curvature is evident in Fig. 4b. Again, ReLU exhibited more flatness than the other two activation functions. Such behaviour is attributed to the hard saturation of ReLU, which can easily yield non-contributing weights. ELU (see Fig. 4c) was dominated by the saddle curvature. The least amount of flatness was discovered for ELU, making ELU a good choice for algorithms that rely on the gradient information.

Fig. 5 shows the LGGs obtained for the \([-1,1]\) macro walks. Larger step sizes revealed similar curvature tendencies for the three activation functions, with TanH exhibiting convexity at the global minimum, but being dominated by saddle points otherwise, ReLU exhibiting strong convexity, and ELU exhibiting no convexity and almost no flatness. All activation functions yielded a split into high error, low gradient, and high gradient, low error clusters. The high gradient, low error clusters were associated with indefinite Hessians for the three activation functions. This behaviour is again attributed to the embedded minima that require fewer weights, as well as hidden unit saturation. Larger steps are likely to arrive at larger weights, thus increasing the chances of saturation.

To test the saturation hypothesis, the degree of saturation was measured for TanH and ReLU. For TanH, the \(\zeta_{h}\) saturation measure proposed in [30] for bounded activation functions was used. The value of \(\zeta_{h}\) is in the \([0,1]\) continuous range, where \(0\) corresponds to a normal distribution of hidden neuron activations, \(0.5\) corresponds to a uniform distribution of hidden neuron activations, and \(1\) corresponds to a saturated distribution of hidden neuron activations, where most activations lie on the asymptotic ends. For ReLU, the proportion of zero activations for all hidden neurons was used as an estimate of saturation. Fig. 6 shows the box plots generated for TanH and ReLU for a few selected scenarios. Fig. 6 shows that singular curvature (indefinite Hessians) was indeed associated with higher saturation under various scenarios, especially under the \([-1,1]\) macro setting, which yielded a steep gradient cluster of indefinite points.

Fig. 7 shows the LGGs for the \([-10,10]\) walks under the macro settings. The \(x\)-axis is shown in square-root scaling for readability. According to Fig. 7a, TanH exhibited very few convex points. Large step sizes prevented the walks from converging to a convex basin, indicating that the width of the convex basin must have been smaller than \(2\) (maximum step size calculated as \(10\%\) of the \([-10,10]\) initialisation range). ReLU exhibited the most convexity out of the three activation functions, and the most flatness. Thus, ReLU yielded a wider convex attraction basin than TanH, and was also more likely to saturate. ELU exhibited little to no convexity, and flatness only along the steeper cluster, associated with embedded minima and/or saturation. The split into two clusters was still

Figure 5. LGGs for the macro PGWs for Iris.

Figure 6. Box plots illustrating the degree of saturation associated with the different curvatures for Iris.
evident for all activation functions. In general, with the increase in step size, the steep cluster became heavier than the shallow cluster. Thus, the steep cluster is confirmed to be associated with large weights, which cause saturation.

To further study the generalisation behaviour of the three activation functions, Fig. 8 shows the LGGs for the micro walks colourised according to their \(E_{g}\) values. Fig. 8 illustrates for the \([-1,1]\) setting that ELU exhibited the best generalisation behaviour. For the \([-10,10]\) setting, the low error, high gradient cluster generalised better than the high error, low gradient cluster for both ReLU and ELU. If the cluster of steep gradients corresponds to the embedded minima comprised of fewer contributing weights, then the steeper gradient solutions can be considered regularised, which explains better generalisation performance.

### Heart

For the heart problem, the split into two clusters was once again evident for all activation functions (figures not shown for brevity). Fig. 9 illustrates that the indefinite points corresponded to points of higher saturation for both TanH and ReLU. Further, Fig. 10 shows the LGGs for TanH and ReLU, colourised according to the estimated degree of saturation. For both activation functions, the steeper gradient cluster was clearly associated with the saturated neurons.

Fig. 11 shows the LGGs colourised according to \(E_{g}\) values. For the \([-1,1]\) walks, all activation functions yielded deteriorating \(E_{g}\) values as \(E_{t}\) approached zero. However, the band of good solutions was noticeably wider for ReLU and ELU as compared to TanH,

Figure 11. LGGs colourised according to \(E_{g}\) values for Heart, micro PGWs, \([-1,1]\) initialisation, \(E_{t}<0.05\).

Figure 8. LGGs colourised according to the \(E_{g}\) values for Iris, micro PGWs, \(E_{t}<0.05\).

Figure 10. LGGs for the macro PGWs initialised in the \([-10,10]\) range for Heart, colourised according to saturation.

Figure 7. LGGs for the macro PGWs for Iris.

Figure 9. Box plots illustrating the degree of saturation associated with the different curvatures for Heart, \([-10,10]\).
indicating that it was easier to find a good quality solution on the ReLU and ELU loss landscapes.

### Mnist

Due to the high dimensionality of MNIST, Hessians were not calculated for this experiment. Fig. 12 and 13 show the LGCs colourised according to \(E_{g}\) values. Convergence to a single global attractor is evident for all activation functions. Out of the activation functions considered, ELU once again yielded the most consistent generalisation performance.

Fig. 12 shows for the \([-1,1]\) interval that ReLU and ELU yielded much higher gradients than TanH. The two-cluster split was evident for all activation functions. For ReLU and ELU, the steep gradient cluster generally yielded poorer generalisation performance than the shallow gradient cluster. This behaviour is especially evident for ReLU under the macro setting (see Fig. 12e). The poor generalisation performance of high gradient points is attributed to neuron saturation.

Fig. 13 shows for the \([-10,10]\) range that TanH yielded an LGC without a well-formed structure. Low gradients and a noisy LGC indicate that the TanH error landscape was not very searchable under the \([-10,10]\) setting. ReLU and ELU also deteriorated with the increased step size and initialisation range, but not as drastically as TanH. The relative resilience of ReLU and ELU to the step size is attributed to the high gradients generated by these unbounded activation functions. ReLU and ELU evidently yielded a more searchable landscape for the high-dimensional MNIST problem, which correlates with the current deep learning insights (Bosman et al., 2017).

The split into two clusters became more pronounced for ReLU and ELU under the \([-10,10]\) setting, confirming the presence of narrow and wide valleys in the landscape. For ELU, points of good \(E_{g}\) values were found in the steep cluster, likely due to the embedded minima discovered. ReLU yielded higher \(E_{t}\) and \(E_{g}\) values than ELU, confirming that the ELU error landscape was more resilient to overfitting.

## 5. Conclusions

This paper empirically analysed the effect of using three different activation functions on the resulting NN error landscapes: TanH, ReLU, and ELU, used in the hidden layer. All experiments were conducted under four granularity settings, with different step sizes and initialisation ranges.

The choice of activation function did not have an effect on the total number of unique attractors (local minima) in the search space, but affected the properties of the discovered basins of attraction. ReLU and ELU yielded steeper attraction basins with stronger gradients than TanH. ReLU exhibited the most convexity, and ELU exhibited the least flatness. The stationary points exhibited by ReLU and ELU were generally more connected than the ones exhibited by TanH, indicating that ReLU and ELU yield more searchable landscapes. However, ReLU and ELU exhibited stronger sensitivity to the step size and the initialisation range than TanH.

All activation functions yielded a split into high error, low gradient, and high gradient, low error clusters. The high gradient cluster was associated with indefinite (i.e. flat) curvature, caused by inactive, or non-contributing weights. Sample points in this cluster were often associated with poor generalisation. Thus, high (or steep) gradients were attributed to narrow valleys, associated with saturated neurons. In individual cases, points of high generalisation performance were discovered in the steep gradient clusters. These were attributed to the embedded self-regularised minima.

Out of the three activation functions considered, ELU consistently exhibited superior generalisation performance. Thus, the loss landscape yielded by the ELU activation function was the most resilient to overfitting.

In future, a more in-depth study of the self-regularised solutions will be conducted. If their unique loss landscape properties can be determined, algorithms may be developed that converge to areas with good generalisation properties that contain smaller NN architectures.

###### Acknowledgements.

 This research was supported by the National Research Foundation (South Africa) Thuthuka Grant Number 13819413. The authors acknowledge the Centre for High Performance Computing (CHPC), South Africa, for providing computational resources to this research project.

Figure 12. LGCs for PGWs initialised in the \([-1,1]\) range for MNIST.

Figure 13. LGCs for PGWs initialised in the \([-10,10]\) range for MNIST.