Further author information: (Send correspondence to Lars Nieradzik)
Lars Nieradzik. E-mail<EMAIL_ADDRESS>
# Effect of the output activation function on the probabilities and errors in
medical image segmentation

Lars Nieradzik  Leipzig University, Augustusplatz 10, 04109 Leipzig, Germany
Gerik Scheuermann  Leipzig University, Augustusplatz 10, 04109 Leipzig,
Germany  Dorothee Saur  Leipzig University Medical Centre, Liebigstraße 20,
04103 Leipzig, Germany  Christina Gillmann  Leipzig University, Augustusplatz
10, 04109 Leipzig, Germany

###### Abstract

The sigmoid activation is the standard output activation function in binary
classification and segmentation with neural networks. Still, there exist a
variety of other potential output activation functions, which may lead to
improved results in medical image segmentation. In this work, we consider how
the asymptotic behavior of different output activation and loss functions
affects the prediction probabilities and the corresponding segmentation
errors. For cross entropy, we show that a faster rate of change of the
activation function correlates with better predictions, while a slower rate of
change can improve the calibration of probabilities. For dice loss, we found
that the arctangent activation function is superior to the sigmoid function.
Furthermore, we provide a test space for arbitrary output activation functions
in the area of medical image segmentation. We tested seven activation
functions in combination with three loss functions on four different medical
image segmentation tasks to provide a classification of which function is best
suited in this application scenario.

###### keywords:

Medical Imaging, Segmentation, Machine learning, Deep learning

##  1 INTRODUCTION

Image segmentation is an important subtask in medical image analysis [1]. In
recent years, machine learning became a popular tool to complete segmentation
tasks as it outperforms analytical methods in many medical disciplines such as
tumor detection [2], lesion prediction [3] or skin anomaly classification [4].
Here, the output activation function is usually chosen to be the sigmoid
function [5] in the case of two potential output classes or the softmax
function for more than two potential output classes. Unfortunately, there has
not been a testing environment or general test settings that aim to examine
the suitability of further potential output activation functions besides the
sigmoid function (see Section 2).

A major problem, when considering medical images, is the uncertainty inherent
in the medical images themselves and the medical imaging pipeline applied to
these images [6]. Gillmann et al. [7] provided a taxonomy of uncertainties
that can occur in the medical imaging pipeline and showed how they can affect
the decision-making process in clinical daily routine. Therefore, an output
activation function that is able to express the uncertainty inherent in a
prediction, while increasing the accuracy of the segmentation result, is
highly desired.

In this manuscript, we aim to test and evaluate a variety of output activation
functions in combination with loss functions (see Section 3). Here, we aim to
identify a proper combination of output activation (7 in total) and loss
functions (3 in total). We aim to test this setting against two different
evaluation metrics, namely negative log-likelihood, and dice coefficient while
training neural networks on 4 different medical datasets (see Section 4).

Therefore, this paper contributes:

  * •

A summary of potential output activation functions for medical image
segmentation.

  * •

A comprehensive evaluation of output activation functions in medical image
segmentation.

  * •

An analysis of the asymptotic behavior of the output activation functions in
relation to the errors and probabilities generated by the neural networks.

  * •

An open test space for output activation functions in medical image
segmentation.

Our results will be discussed in Section 5, where advantages, disadvantages,
and limitations of the test functions and the provided test space will be
shown. One of the conclusions (see Section 6) that we can draw is that the
sigmoid activation function is not the optimal choice of an output activation
function regarding medical imaging.

##  2 RELATED WORK

Medical image segmentation is an important sub-task of the medical image
processing pipeline [8]. There exist a variety of network architectures that
have been shown to deliver promising segmentation results that outperform
classic analytical approaches [9]. Depending on the chosen network
architecture, multiple parameters need to be fine-tuned during the training
process of a neural network. The output activation function is usually not one
of these parameters, and is in most cases set to be the sigmoid or softmax
function. In this section, we aim to shed light on related work that provides
knowledge on other output activation functions that may be used in medical
image segmentation.

The standard sigmoid activation is known in statistics as the logistic
function and corresponds to the logit model. There are a wide range of papers
that compare the results of this logistic function to two other models: the
probit and linear model [10, 11, 12, 13]. Horowitz et al.[10] reviewed the
linear, probit and logit model with examples from biometrics and econometrics.
Cakmakyapan et al.[11] showed in a simulation study that the logit model
performs better for large sample sizes (500 and 1000) and the probit model is
better for small sizes (40, 100, 200). Hoetker et al.[12] reviewed problems
when interpreting the coefficients of logit, probit and linear models.

However, the linear and probit model have never been analyzed in deep learning
as output activation functions. Gomes et al.[14] compared the complementary
log-log and probit model as an alternative to the sigmoid activation, but only
in intermediate layers. Another paper by Zeng[15] contrasted the logit/probit
model with neural networks for social science classification problems. They
found that neural network models perform significantly better than
logit/probit models. However, they did not try the activation function of the
probit model in neural networks.

We use these different statistical models as a starting point for our approach
to find alternative output activation functions. In this work, we will use
both the activation function of the probit and of the linear model for medical
image segmentation and contrast them with the regular sigmoid function.

Apart from the statistical literature, several activation functions were
proposed for non-output layers in neural networks. Ramachandran et al.[16]
used automated search techniques to find the Swish activation function as a
replacement to ReLU. This activation function was then extended and analyzed
by Ma et al. [17]. Since neuron inputs tend to follow a normal distribution,
Hendrycks et al.[18] suggested the use of the standard Gaussian cumulative
distribution function (CDF) as part of a new activation function. In
statistics, the normal CDF is also used as an activation function in the
probit model. This shows that the CDF is a viable option as an output
activation function.

A paper by Molina et al.[19] proposed the Padé Activation Unit, a trainable
activation function. By adapting automatically to the data at hand, they
obtained improvements relative to the ReLU activation. In this work, we will
also use this idea of adaptivity for an output activation function.

Finally, a review by Nwankpa et al.[5] summarized most of the common
activation functions used in neural networks. Another paper by Apicella et
al.[20] focused more on summarizing trainable activation functions. We also
take inspiration from some of these activation functions, which were
originally designed for intermediate layers.

Based on this review of the literature, we have seen that finding a better
activation function than ReLU is a common topic in the deep learning
literature. However, to the best of our knowledge, there are no other
publications that focus solely on the activation function in the output layer
of neural networks or examine the impact of this choice.

##  3 METHODS

Neural network-based segmentation models consist of an encoder that extracts
features and downsamples the image, and a decoder that restores the image to
its original size. This process is followed by an output activation function
that determines the probability of each class (e.g. foreground and background
pixel). During the training process, the loss function determines the
correctness of the prediction made and allows readjusting network parameters
accordingly.

This work proposes replacing the standard sigmoid activation function that is
applied to the decoder outputs with other functions with similar properties.
The goal is to clarify if there are more appropriate choices than the sigmoid
output activation function for medical image segmentation. Additionally, we
combine potential output activation functions with various loss functions when
training the network.

One of the results of this work is a test space that allows the user to train
segmentation models on various medical segmentation datasets. We can change
parameters such as the encoder, the decoder, the output activation, and the
loss function, as shown in Figure 1. The complete source code is available on
GitHub https://github.com/lars76/segmentation_activations.

Encoder Decoder Output Activation Loss Segmentation Model Figure 1: General
workflow of segmentation models. The neural network consists of an encoder,
decoder, and activation function. The activation function is applied to the
output of the decoder. The loss functions regulate the training process of the
neural network. Output activation and loss function are highlighted in green,
as they are the main focus of this work.

In the following, we introduce the output activation functions that are
evaluated in this paper (see Section 3.1), followed by the tested loss
functions (see Section 3.2).

###  3.1 Output Activation Function

The sigmoid or logistic function $f(x)=\frac{1}{1+\exp{\left(-x\right)}}$ is
the standard output activation function for binary segmentation or
classification tasks. It is a simplification of the softmax function that is
used for classification or segmentation tasks with more than two classes.

$-$6$-$4$-$202460.00.20.40.60.81.0Activation
functionsArctangentSoftsignInverse Square RootNormal CDFHardTanhLinearSigmoid

Figure 2:  Limiting behavior of the activation functions to their asymptotes.
The sigmoid function is shown as a dashed line. The remaining functions can
mainly be separated into functions with faster or slower rate of change than
sigmoid.

The sigmoid function shows an exponential growth around zero that quickly
saturates and slows to linear growth (see fig. 2, dashed line). At $x=-6$ and
$x=6$ the sigmoid function is already close to its asymptotes (zero and one).
However, this rate of change is not necessarily optimal. We shall consider two
cases regarding the asymptotes: a faster rate of change (less freedom of
action) and a slower rate of change (more freedom of action).

Both cases will be considered in our analysis using different activation
functions on the output layer. The choice of the functions will be explained
in the following.

It is possible to use any differentiable function in the output layer of
neural networks. There are, however, some requirements. Certain loss functions
need activation functions with range $[0,1]$ (e.g. dice loss). Furthermore,
the function should be symmetric such that the probabilities have the same
meaning no matter whether the input values are in the intervals $(-\infty,0)$
or $(0,\infty)$. Many of the activation functions listed by Nwankpa et al. [5]
are thus no option. Instead, we choose functions $f:\mathbb{R}\to[0,1]$ that
are for the most part monotonic increasing and symmetric with
$f(0)=\frac{1}{2}$.

There is naturally an infinite number of activation functions that satisfy the
above requirements. In order to restrict the space of possible functions, we
choose functions that are found in statistics (probit model, linear model)
and/or have a sigmoidal shape (S-shape). Then we rescale the activations to
the range $[0,1]$.

In order to better understand the limiting behavior of the functions to the
horizontal asymptotes at $0$ and $1$, we define the _effective domain_ as
$X=\\{x\mid f(x)\leq 1-\epsilon,f(x)\geq\epsilon\\}$. For most activation
functions, we do not need the complete real line $\mathbb{R}$, but can
restrict ourselves to a smaller domain $X$.

Function name | $f(x)$ | $f^{\prime}(x)$ | Effective domain
---|---|---|---
normal CDF | $\Phi(x)=\frac{1}{2}\text{erf}\left(\frac{x}{\sqrt{2}}\right)+\frac{1}{2}$ | $\phi(x)=\frac{\exp{\left(-x^{2}\right)}}{\sqrt{2\pi}}$ | $[-3,3]$
sigmoid | $\frac{1}{1+\exp{\left(-x\right)}}$ | $\frac{1}{1+\exp{\left(-x\right)}}\left(1-\frac{1}{1+\exp{\left(-x\right)}}\right)$ | $[-6,6]$
inverse square root | $\frac{1}{2}\frac{x}{\sqrt{1+x^{2}}}+\frac{1}{2}$ | $\frac{0.5}{\left(x^{2}+1\right)^{\frac{3}{2}}}$ | $[-10,10]$
arctangent | $\frac{\arctan(x)}{\pi}+\frac{1}{2}$ | $\frac{1}{\pi x^{2}+\pi}$ | $[-128,128]$
softsign | $\frac{1}{2}\frac{x}{1+|x|}+\frac{1}{2}$ | $\frac{0.5}{(|x|+1)^{2}}$ | $[-199,199]$
linear | $\frac{x-x_{\text{min}}}{x_{\text{max}}-x_{\text{min}}}$ | $\frac{1}{x_{\text{max}}-x_{\text{min}}}$ | $[x_{\text{min}},x_{\text{max}}]$
hardtanh | $\begin{cases}1&\text{if }x>1\\\ 0&\text{if }x<0\\\ x&\text{otherwise}\end{cases}$ | $\begin{cases}1&\text{if }0\leq x\leq 1\\\ 0&\text{otherwise}\end{cases}$ | $[0,1]$
Table 1: Output activation functions considered in this work. The table
includes the name, the derivative, and the effective domain of each function.
The effective domain was rounded by applying the ceiling and floor function.
We choose $\epsilon=0.0025$.

In Table 1, we present the activation functions that are evaluated in this
paper. We ordered the functions by their effective domain. Two exceptions are
the linear and hardtanh activation.

The linear function depends on the input values and automatically adapts the
domain to the image. This function comes originally from the linear
probability model in statistics [21, 10]. As the identity function $f(x)=x$
allows for values outside the interval $[0,1]$, we perform rescaling by
finding the minimum and maximum value in each image or batch of images.

The other exception is the hardtanh function, which has discontinuities and
does not output $f(0)=0$. Thus, it also has a different behavior than the
other functions.

#### Output activation functions with smaller effective domain than sigmoid

The first case is a faster rate of change than sigmoid. By increasing the rate
of change, we can decrease the input values to the activation function $f(x)$.
This forces the neural network to decrease the outputs $x$ of the previous
layer if probabilities like $f(x)=0.8$ are needed. A smaller domain than
$[-6,6]$ of the activation function leads to sharper decisions and less
uncertainty.

The normal cumulative distribution function (CDF) is an example of a function
with a fast rate of change. This activation function can be motivated by
considering the probit model from statistics, where the CDF is used as an
alternative to the sigmoid/logistic function. Together with the logit model
(sigmoid function) it is one of the most frequently used models for binary
outputs [21, 11].

#### Output activation functions with larger effective domain than sigmoid

The second case is a slower rate of change. This brings greater freedom of
movement for the output values $f(x)$. It can become easier to output
probabilities between $0$ and $1$. A function with a large effective domain
allows for more leeway than an exponential function that reaches the
asymptotes $0$ or $1$ too quickly. Thus, a slower rate of change can result in
better probabilities and counteract problems of probabilities being pushed
towards $0$ and $1$. The function would have effectively a larger domain than
sigmoid.

Three examples are the inverse square root function, arctangent, and softsign.
While the three functions have a sigmoidal shape, they are not commonly found
in statistics. However, the unscaled softsign function is found in the review
by Nwankpa et al. [5] for use in non-output layers of neural networks. From
the diagram, we can see that the inverse square root function is initially
faster than the sigmoid function. For $x=3$, we have $0.95$ (sigmoid) and
$0.97$ (inverse square root). This should also affect the results.

###  3.2 Loss function

The next step is the definition of the loss function. In the statistical
literature, a popular method for estimating the weights is maximum likelihood
estimation (MLE). The joint likelihood for binary variables is given by

${\mathcal{L}}(\beta;Y,X)=\prod_{i=1}^{n}f(x_{i}^{T}\beta)^{y_{i}}\left(1-f(x_{i}^{T}\beta)\right)^{1-y_{i}}\,,$

where $x_{i}$ are the inputs and $y_{i}\in\\{0,1\\}$ the outputs. $f(\cdot)$
defines the activation function. By applying the logarithm and changing the
sign, we obtain the cross-entropy loss function used in neural networks

${\mathcal{L}}(\beta;Y,X)=-\sum_{i=1}^{n}y_{i}\log
f(x_{i}^{T}\beta)+\left(1-y_{i}\right)\log\left(1-f(x_{i}^{T}\beta)\right)\,.$

For the binary case, the loss function is also called binary cross-entropy
(BCE).

Apart from BCE, a method for finding the coefficients in linear regression is
ordinary least squares. The corresponding loss function is the (mean) squared
error (MSE) in neural networks

${\mathcal{L}}(\beta;Y,X)=\sum_{i=1}^{n}\left(\hat{y}_{i}-y_{i}\right)^{2}\,,$

where $\hat{y}_{i}=f(x_{i}^{T}\beta)$. However, it is rarely used for
segmentation or classification [22]. In the Medical Segmentation Decathlon
(MSD) almost all participants used cross entropy [23].

A more common loss function than MSE in segmentation is dice loss. The
definition is as follows

${\mathcal{L}}(\beta;Y,X)=1-\frac{2\sum_{i}\hat{y}_{i}y_{i}}{\sum_{i}\hat{y}_{i}+y_{i}}\,.$

Dice loss optimizes the overlap between the predicted pixels and the ground
truth.

We visualize all loss functions for a single prediction with $y_{i}=1$ and
$\hat{y}_{i}=f(x)$ in fig. 3. As the activation functions differ mainly how
fast they approach $0$ or $1$, we refrain from showing all functions.

0.20.40.60.8Prediction0.00.51.01.52.02.5ErrorSoftsign$\displaystyle-\log(f(x))$$\displaystyle(1-f(x))^{2}$$\displaystyle
1-\frac{2f(x)}{1+f(x)}$

(a) The softsign function produces a lower error for bad predictions.

0.00.20.40.60.81.0Prediction0123456ErrorSigmoid$\displaystyle-\log(f(x))$$\displaystyle(1-f(x))^{2}$$\displaystyle
1-\frac{2f(x)}{1+f(x)}$

(b) The sigmoid activation has a much higher error.

Figure 3:  Error for a single prediction

We use the same domain $[-6,6]$ for softsign and sigmoid in fig. 3. Since
softsign has a wider domain, the function takes longer to reach a probability
of $0$ or $1$ than sigmoid. This affects also the error. At $-6$ we have a BCE
error of approx. $-\log(0.07)\approx 2.6$ for softsign against
$\log(1+e^{6})\approx 6$ for sigmoid.

The mean squared error moves more quickly to an error of $0$ than dice loss.
Since real pictures have tens of thousands of pixels, the plots do not show
the full behavior of the different loss functions.

We can expect mean squared error and negative log-likelihood to give better
probabilistic estimates than dice loss because both functions look at
individual pixels. Dice loss, in contrast, jointly optimizes all pixels.

##  4 EVALUATION

We evaluate 21 models that result from the 7 activation functions (see Section
3.1) and 3 loss functions (see Section 3.2). In the following subsections, we
first describe the evaluation metrics and the segmentation model that were
used. Then we present the results on the different medical segmentation
datasets.

###  4.1 Evaluation Metrics

All loss functions can also be used as metrics to determine the quality of the
trained network. In particular, cross-entropy and mean squared error are known
to be proper scoring rules that are suited for model comparisons [24, 25].
According to tests by Ashukha et al.[24], mean squared error and cross entropy
show high empirical correlation for a wide range of computer vision datasets.
Thus, we omit evaluations on the mean squared error, which is also known as
Brier score [26].

Cross entropy, often referred to as negative log-likelihood (NLL)[27], is used
for the evaluation of probabilistic errors. However, another approach that is
well known in the domain of uncertainty estimation is the reliability
diagram[28].

A reliability diagram has on the x-axis the confidence and on the y-axis the
fraction of positives. The first step is to put all predicted probabilities
into $n$ different intervals or bins. The _evenly spaced_ strategy uses the
intervals
$\left\\{\left[0,\frac{1}{n}\right),\left[\frac{1}{n},\frac{2}{n}\right),\dots,\left[\frac{n-1}{n},1\right)\right\\}$.
The _adaptive_ strategy chooses intervals based on quantiles to prevent empty
intervals or intervals with few probabilities. In segmentation, probabilities
tend to be close to $0$ because there are more background pixels than
foreground pixels. For this reason, Nixon et al.[29] proposed thresholding
predictions such that predictions below some $\epsilon$ are removed for
quantile binning. We choose $n=15$ bins and $\epsilon=10^{-2}$ and show both
strategies when evaluating the datasets. Additionally, we set another
threshold to $1-10^{-2}$ to prevent too many bins close to $1$.

The second step is to plot on the $x$-axis the confidence
$\frac{1}{|b|}\sum_{(y,\hat{y})\in b}\hat{y}$ and on the $y$-axis the relative
frequency $\frac{1}{|b|}\sum_{(y,\hat{y})\in b}y$ for all bins or intervals
$b$. The result is a reliability diagram.

Apart from probabilistic measures such as NLL or the reliability diagram, we
also evaluate the object overlap by considering the dice coefficient as a
metric.

The thresholded dice coefficient or F1 score can be defined as follows

$\text{Dice}_{t}=\frac{2\sum_{i}\mathbf{1}(\hat{y}_{i}>t)y_{i}}{\sum_{i}\mathbf{1}(\hat{y}_{i}>t)+y_{i}}\,,$

where $t$ is a given threshold. Here, the threshold is chosen that best
maximizes the dice coefficient. All thresholds from the set
$\\{0,0.05,0.1,\dots,0.95,1.0\\}$ were tried out for each cross validation
fold. We denote by $\mathbf{1}(\cdot)$ the indicator function that outputs $1$
when the given condition is true.

###  4.2 Datasets and Training

For evaluating the different loss and activation functions, we consider both
2D and 3D medical datasets. As it is computationally infeasible to train
neural networks on all possible medical datasets, we choose datasets that are
sufficiently different from one another. The four datasets cover different
parts of the body: heart (1), brain (2), colon (3), and prostate (4). The
objective is to generalize the results of this paper to other medical
datasets.

  1. 1.

The _Automated Cardiac Diagnosis Challenge_ (ACDC) [30] contains cardiac MRI
recordings.

  2. 2.

_Ischemic Stroke Lesion Segmentation_ (ISLES) 2018 [31, 32] deals with CT
perfusion data.

  3. 3.

_Kvasir-SEG_ [33] is a 2D polyp dataset.

  4. 4.

_Medical Segmentation Decathlon_[23] (MSD) consists of multiple 3D semantic
segmentation datasets. The neural networks here are trained on the fifth task,
involving T2 and ADC image sequences of prostates.

Since 2D convolutional neural networks (CNN) were used in the winning
solutions of competitions such as the _RSNA Intracranial Hemorrhage
Detection_[34] or ISLES 2018[35, 36], we use a standard 2D U-Net architecture
for all datasets [37]. Furthermore, it was shown that performing transfer
learning by employing an encoder pretrained on ImageNet, can lead to better
results [38, 39, 40]. We choose ResNet-34 [41] as encoder for its simplicity
and robustness.

ResNet-34 UpUpUpUp Activation Figure 4: ResNet-34 as the encoder and U-Net as
the decoder. A dotted line means concatenation with the last layer of the same
width and height in the encoder. An up layer consists of a nearest-neighbor
interpolation, two convolutions with ReLU activation, and batch
normalizations.

All models are trained with a batch size of 8 and the adaptive stochastic
optimizer Adam with a standard learning rate of $10^{-3}$ [42]. We reduce the
learning rate when $5$ successive epochs did not result in any improvement,
and stop the training after $10$ epochs without improvement. For validating
the results, 5-fold cross validation was used. We used PyTorch[43] as a deep
learning library. For certain activation functions, such as the normal CDF, we
had to rescale the outputs to $[10^{-7},1-10^{-7}]$ to ensure numerical
stability.

###  4.3 Evaluation of medical datasets

To evaluate the selected medical datasets, we provide the mean NLL and dice
coefficient of each cross-validated test run and their standard deviation. In
addition, we present examples of segmented images that offer high object
overlap, as well as examples that offer low overlap according to the selected
metric. A deep blue color on the images indicates a low probability, while a
deep red means a high probability. At last, we also show a reliability diagram
(with the evenly spaced and adaptive strategy) for each tested dataset,
comparing the winning activation function(s) to the well-known sigmoid output
activation function. With the arrows $\uparrow$ and $\downarrow$, we indicate
whether a higher or lower value is better. Except for linear and hardtanh, all
activation functions are ordered by their effective domain in ascending order.
The appendix 8 at the end of the paper contains all the predictions of a test
image for each dataset.

#### Automated Cardiac Diagnosis Challenge (ACDC)

The first dataset to be tested is the _Automated Cardiac Diagnosis Challenge_.
All trained networks and the quality of their outputs are shown in Table 2.
Since the predictions are quite similar, we only see to a certain extent the
presumed influence of the effective domain. If one considers the probabilistic
error NLL together with MSE / BCE, normal CDF is better than sigmoid and this
function is better than the inverse square root. With a larger effective
domain, the error increases. Without inverse square root, this increase in
error is even clearer: $0.0542\to 0.0556\to 0.0684\to 0.0698$. The reason that
inverse square root does not behave like the other functions is that it is
initially faster than sigmoid.

Dice loss shows the inverse effect: a larger effective domain leads to a
better NLL error. This can be explained by noting that dice loss does not
consider individual pixels like BCE / MSE. A wider domain gives dice loss more
flexibility to change individual pixels. We then have more uncertainty and
better predictions.

Activation | Loss |  | NLL $\downarrow$ | Dice $\uparrow$ |
---|---|---|---|---|---
normal CDF | BCE |  | $\mathbf{0.0542\pm 0.0048}$ | $0.9154\pm 0.0062$ |
Dice |  | $0.2033\pm 0.0146$ | $0.9156\pm 0.007$ |
MSE |  | $0.0567\pm 0.0044$ | $0.9161\pm 0.0063$ |
sigmoid | BCE |  | $0.0556\pm 0.0051$ | $0.9155\pm 0.0075$ |
Dice |  | $0.1409\pm 0.0185$ | $0.9152\pm 0.0073$ |
MSE |  | $0.059\pm 0.0043$ | $0.9147\pm 0.0074$ |
inverse square root | BCE |  | $0.1321\pm 0.0083$ | $0.9135\pm 0.007$ |
Dice |  | $0.1131\pm 0.0054$ | $0.9151\pm 0.0066$ |
MSE |  | $0.066\pm 0.0035$ | $\mathbf{0.9167\pm 0.0062}$ |
arctangent | BCE |  | $0.0684\pm 0.0071$ | $0.9157\pm 0.0064$ |
Dice |  | $0.0992\pm 0.0071$ | $0.9159\pm 0.0052$ |
MSE |  | $0.0887\pm 0.0033$ | $0.9153\pm 0.0068$ |
softsign | BCE |  | $0.0698\pm 0.0061$ | $\mathbf{0.9167\pm 0.0065}$ |
Dice |  | $0.0965\pm 0.0073$ | $0.9145\pm 0.0061$ |
MSE |  | $0.088\pm 0.007$ | $0.915\pm 0.0061$ |
linear | BCE |  | $0.0604\pm 0.0044$ | $0.9086\pm 0.0061$ |
Dice |  | $0.0701\pm 0.0063$ | $0.9043\pm 0.0078$ |
MSE |  | $0.0687\pm 0.0063$ | $0.9103\pm 0.0046$ |
hardtanh | BCE |  | $0.1178\pm 0.0178$ | $\underline{0.8191\pm 0.0159}$ |
Dice |  | $\underline{0.2825\pm 0.0217}$ | $0.9164\pm 0.0066$ |
MSE |  | $0.0994\pm 0.0094$ | $0.9162\pm 0.0077$ |
Table 2: Results of all tested models trained on the ACDC dataset, where the
best (bold) and worst (underline) results were highlighted.

The dice coefficient next to the NLL column in the table shows no major
differences between the individual activation functions for this dataset. If
we then look at the individual predictions of some images, we should not see
any big differences either.

(a) BCE with CDF activation

(b) MSE with inverse square root

(c) BCE with softsign

(d) Ground truth

Figure 5:  Best predictions for a single image in ACDC

Figure 5 shows an example of the segmentation result using the best models of
our test (highlighted in bold in the table). Figure 5 (a) comes visually close
to the provided ground truth in Figure 5 (d), while holding a slight
uncertainty at the border of the detected area. The mean squared error in
combination with inverse square root in Figure 5 (b) provides similar results,
whereas (c) predicted a smaller area relative to the ground truth (d). On the
other hand, the model predictions are very certain because the probabilities
are highlighted in deep red.

(a) Dice with hardtanh

(b) BCE with hardtanh

Figure 6:  Worst predictions for a single image in ACDC

When reviewing the worst models trained on the ACDC dataset, we can see from
Table 2 that the dice and hardtanh combination has the worst NLL error. The
reason is that dice loss optimizes the object overlap and not probabilities.
In Figure 6 (a) we see that there is no uncertainty on the border and the
probabilities are close to $1$. In contrast, the combination of hardtanh with
BCE optimizes the probabilities and shows more uncertainty. However, Figure 6
(b) shows massive mispredictions all over the considered image.

Figure 7 shows the reliability graph of the best activation and loss
combinations in comparison to the standard sigmoid function with BCE. We can
see that all configurations underestimate the true probabilities for values
less than $0.5$ and overestimate the true probabilities for values greater
than $0.5$. For BCE+softsign, the difference between confidence and the number
of occurrences is at most 20%. When the prediction states that a pixel is $1$
with a probability of x%, then this event will not occur exactly on x% of all
occasions. The reason is that the model both underestimates and overestimates
the true probabilities. The BCE+softsign combination has also the worst NLL
error of the four combinations (BCE+CDF, MSE+inverse square root,
BCE+softsign, BCE+sigmoid). Thus, a large effective domain like softsign does
not always result in better calibration or probabilistic errors.

When using adaptive binning, the big difference between the combinations
vanishes such that the reliability graphs are nearly equal. However, the
models are still not perfectly calibrated. Since the dice coefficient did not
vary considerably between the activation functions, we do not see a great
difference regarding the calibration.

0.00.20.40.60.81.0Confidence0.00.20.40.60.81.0Fraction of positivesReliability
diagramPerfectly calibratedBCE+sigmoidBCE+cdfMSE+inv_square_rootBCE+softsign

(a) Evenly spaced bins

0.00.20.40.60.81.0Confidence0.00.20.40.60.81.0Fraction of positivesReliability
diagramPerfectly calibratedBCE+sigmoidBCE+cdfMSE+inv_square_rootBCE+softsign

(b) Adaptive bins

Figure 7: Reliability diagrams on the ACDC dataset for the best performing
models compared to the sigmoid function as the output activation function.

#### Ischemic Stroke Lesion Segmentation Challenge (ISLES)

The second dataset is ISLES. Table 3 shows that BCE combined with normal CDF
achieves the best results for both evaluation metrics. Sigmoid+BCE and
linear+BCE have a similar NLL to CDF+BCE, but still a lower dice coefficient.

Activation | Loss |  | NLL $\downarrow$ | Dice $\uparrow$ |
---|---|---|---|---|---
normal CDF | BCE |  | $\mathbf{0.0319\pm 0.0021}$ | $\mathbf{0.661\pm 0.0309}$ |
Dice |  | $0.1084\pm 0.0327$ | $0.6267\pm 0.0533$ |
MSE |  | $0.0396\pm 0.002$ | $0.6217\pm 0.0419$ |
sigmoid | BCE |  | $0.0337\pm 0.0028$ | $0.6511\pm 0.0324$ |
Dice |  | $0.0905\pm 0.0269$ | $0.6385\pm 0.0481$ |
MSE |  | $0.0443\pm 0.0049$ | $0.6375\pm 0.0548$ |
inverse square root | BCE |  | $0.0518\pm 0.0042$ | $0.5973\pm 0.0633$ |
Dice |  | $0.0712\pm 0.0098$ | $0.6373\pm 0.0469$ |
MSE |  | $0.0548\pm 0.0031$ | $0.6108\pm 0.0495$ |
arctangent | BCE |  | $0.0556\pm 0.0083$ | $0.6338\pm 0.0616$ |
Dice |  | $0.0608\pm 0.0086$ | $0.6476\pm 0.0393$ |
MSE |  | $0.0787\pm 0.0079$ | $0.6348\pm 0.0469$ |
softsign | BCE |  | $0.0635\pm 0.0017$ | $0.6497\pm 0.0464$ |
Dice |  | $0.0644\pm 0.0076$ | $0.6415\pm 0.0547$ |
MSE |  | $0.0876\pm 0.0166$ | $0.6291\pm 0.0669$ |
linear | BCE |  | $0.0333\pm 0.0029$ | $0.6534\pm 0.0367$ |
Dice |  | $0.0471\pm 0.0069$ | $0.6516\pm 0.0465$ |
MSE |  | $0.0365\pm 0.0045$ | $0.6537\pm 0.027$ |
hardtanh | BCE |  | $0.0433\pm 0.0065$ | $\underline{0.5464\pm 0.0669}$ |
Dice |  | $\underline{0.1878\pm 0.0333}$ | $0.6244\pm 0.0562$ |
MSE |  | $0.0572\pm 0.0147$ | $0.602\pm 0.0391$ |
Table 3: Results of all tested models trained on the ISLES dataset where the
best (bold) and worst (underline) results were highlighted.

For this dataset, we clearly see the influence of the effective domain. As the
effective domain increases, the metric NLL gets worse and worse for the loss
function BCE. Similar to the last dataset, the loss function dice shows the
opposite effect, where a larger effective domain is better for NLL. When
considering the dice coefficient next to NLL in the table, there is also a
correlation between the domain and the metric, but not as strong as with NLL.

Since the dice coefficient is much lower in this dataset, we expect the
predicted images to have a higher uncertainty than the last dataset.

Figure 8 shows a selected slice of a lesion prediction in a patient, with (a)
showing the best prediction model and (b) showing the underlying ground truth.
The model is able to predict the lesion while maintaining a large interval
with a uncertain border around the prediction. Czolbe et al.[44] have shown
that uncertainty correlates with segmentation errors. Here, too, we find such
a correlation. We can therefore assume that the prediction at the borders was
difficult for the network.

(a) BCE and CDF

(b) Ground truth

Figure 8: Best predictions for a single image in ISLES

Figure 9 shows the two models that performed the worst on the ISLES dataset.
(a) shows the dice loss in combination with the hardtanh output activation
function. We see that this combination has the highest probabilistic error
(NLL) because there is no uncertainty. In contrast, the BCE loss in
combination with hardtanh (b) provides a very scattered lesion segmentation
over large areas in the image. This combination has the highest overlap error
and also shows a lot of uncertainty.

(a) Dice and hardtanh

(b) BCE and hardtanh

Figure 9: Worst predictions for a single image in ISLES

In Table 3, BCE in combination with normal CDF has the lowest NLL error.
However, in the reliability diagrams in Figure 10, we see that sigmoid comes
closer to the perfectly calibrated line.

This result is not surprising since CDF has a smaller effective domain than
sigmoid. It is more difficult for CDF to return probabilities between $0$ and
$1$ than it is for sigmoid. For the last dataset, however, we could not find
that big a difference between BCE+CDF and BCE+sigmoid. So it also depends on
the dataset.

Even if sigmoid is here better with respect to the calibration error, both
combinations overestimate the probabilities for values greater than $0.1$
(BCE+cdf) and for values greater than $0.25$ (BCE+sigmoid) in Figure 10 (a).
Only when the probabilities are close to $0$, the fraction of positives is
near the perfectly calibrated line. With the adaptive strategy in (b), there
are more bins close to $0$ because most ground truth pixels are $0$.

0.00.20.40.60.81.0Confidence0.00.20.40.60.81.0Fraction of positivesReliability
diagramPerfectly calibratedBCE+sigmoidBCE+cdf

(a) Evenly spaced bins

0.00.20.40.60.81.0Confidence0.00.20.40.60.81.0Fraction of positivesReliability
diagramPerfectly calibratedBCE+sigmoidBCE+cdf

(b) Adaptive bins

Figure 10: Reliability diagrams on the ISLES dataset for the best performing
models compared to the sigmoid function as the output activation function.

#### Kvasir-SEG

The next dataset is Kvasir-SEG. Hardtanh activation has a higher dice
coefficient than both sigmoid and CDF activation. In particular, simple
functions such as the hardtanh or the linear function show better results than
sigmoid-shaped activations. The winning model using NLL as the metric is MSE
with linear output activation function, while the winning model using the dice
metric is the combination BCE+hardtanh.

Table 4: Results of all tested models trained on the Kvasir-SEG dataset where the best (bold) and worst (underline) results were highlighted. Activation | Loss |  | NLL $\downarrow$ | Dice $\uparrow$ |
---|---|---|---|---|---
normal CDF | BCE |  | $0.2677\pm 0.0173$ | $0.6783\pm 0.0228$ |
Dice |  | $1.3776\pm 0.0863$ | $0.6475\pm 0.0312$ |
MSE |  | $0.2674\pm 0.0124$ | $0.667\pm 0.0252$ |
sigmoid | BCE |  | $0.255\pm 0.0114$ | $0.6749\pm 0.0212$ |
Dice |  | $1.1301\pm 0.0954$ | $0.6486\pm 0.0134$ |
MSE |  | $0.2635\pm 0.0183$ | $0.6686\pm 0.0252$ |
inverse square root | BCE |  | $0.3314\pm 0.0169$ | $0.694\pm 0.0104$ |
Dice |  | $0.7643\pm 0.0344$ | $0.6571\pm 0.0145$ |
MSE |  | $0.2595\pm 0.0134$ | $0.6731\pm 0.0267$ |
arctangent | BCE |  | $0.2651\pm 0.0194$ | $0.6753\pm 0.0229$ |
Dice |  | $0.5441\pm 0.0263$ | $0.6486\pm 0.0188$ |
MSE |  | $0.2614\pm 0.0137$ | $0.6759\pm 0.0176$ |
softsign | BCE |  | $0.2577\pm 0.0134$ | $0.6797\pm 0.0165$ |
Dice |  | $0.5072\pm 0.0094$ | $\underline{0.6427\pm 0.0171}$ |
MSE |  | $0.2602\pm 0.011$ | $0.6774\pm 0.0155$ |
linear | BCE |  | $0.2591\pm 0.0091$ | $0.6944\pm 0.01$ |
Dice |  | $0.7848\pm 0.0579$ | $0.6533\pm 0.0246$ |
MSE |  | $\mathbf{0.2493\pm 0.0145}$ | $0.6956\pm 0.0186$ |
hardtanh | BCE |  | $0.258\pm 0.0118$ | $\mathbf{0.7082\pm 0.0038}$ |
Dice |  | $\underline{1.5785\pm 0.1143}$ | $0.6516\pm 0.0242$ |
MSE |  | $0.3153\pm 0.0157$ | $0.6837\pm 0.009$ |

In this table, we again see the opposite effect, that a larger effective
domain leads to better probabilities for dice loss + NLL. This time, we cannot
find a correlation between BCE + NLL and the effective domain. The order is
more random.

Figure 11 shows the winning models using the Kvasir dataset with an example
patient. (b) shows the prediction of the NLL winning model using MSE and
linear activation. Here we can see that the ground truth area is partially
predicted while maintaining a large boundary of uncertainty. The model using
BCE and hardtanh gives similar results in (a) but has a smaller core of
certain predictions compared to the other prediction as shown in (b).

(a) BCE+hardtanh

(b) MSE+linear

(c) Ground truth

Figure 11: Best predictions for a single image in Kvasir.

Figure 12 shows the worst two models trained with this dataset. In (a) we can
see that the dice and hardtanh combination predicts the segmentation area at a
wrong position without any uncertainties. In (b) we see the same situation for
the dice and softsign combination. This again illustrates the problem with
dice loss that it does not output any uncertainties.

(a) Dice and hardtanh

(b) Dice and softsign

Figure 12: Worst predictions for a single image in Kvasir.

The reliability diagrams in Figure 13 are similar to those for ACDC. The
probabilities for values less than 0.35 are underestimated and overestimated
for values greater than 0.35. BCE+hardtanh is closer to the perfectly
calibrated line than MSE+linear, although the latter function has better NLL.
Therefore, a model’s calibration level does not always correlate with NLL. The
use of quantiles does not have a strong effect on the reliability diagram, as
can be seen in (b). Most bins are close to $0$.

0.00.20.40.60.81.0Confidence0.00.20.40.60.81.0Fraction of positivesReliability
diagramPerfectly calibratedBCE+hardtanhMSE+linearBCE+sigmoid

(a) Evenly spaced bins

0.00.20.40.60.81.0Confidence0.00.20.40.60.81.0Fraction of positivesReliability
diagramPerfectly calibratedBCE+hardtanhMSE+linearBCE+sigmoid

(b) Adaptive bins

Figure 13: Reliability diagrams on the Kvasir dataset for the best performing
models, compared to the sigmoid function as the output activation function.

#### Medical Segmentation Decathlon (MSD)

The final dataset is the _Medical Segmentation Decathlon_. As shown in Table
LABEL:tab:statsMSD, BCE with arctangent as the output activation function
provides the best prediction using Dice as the evaluation metric. In addition,
using NLL as the evaluation metric, the winning model is BCE with sigmoid.

Table 5: Results of all tested models trained on the MSD dataset where the best (bold) and worst (underline) results were highlighted. Activation | Loss |  | NLL $\downarrow$ | Dice $\uparrow$ |
---|---|---|---|---|---
normal CDF | BCE |  | $0.0411\pm 0.0125$ | $0.76\pm 0.0565$ |
Dice |  | $0.1727\pm 0.0407$ | $0.6694\pm 0.0716$ |
MSE |  | $0.0469\pm 0.017$ | $0.7642\pm 0.0546$ |
sigmoid | BCE |  | $\mathbf{0.0383\pm 0.0107}$ | $0.7682\pm 0.056$ |
Dice |  | $0.1123\pm 0.0286$ | $0.704\pm 0.067$ |
MSE |  | $0.0519\pm 0.0173$ | $0.7603\pm 0.0478$ |
inverse square root | BCE |  | $0.0593\pm 0.0118$ | $0.7404\pm 0.0494$ |
Dice |  | $0.077\pm 0.0211$ | $0.7293\pm 0.0812$ |
MSE |  | $0.0581\pm 0.0122$ | $0.7605\pm 0.0531$ |
arctangent | BCE |  | $0.0608\pm 0.0113$ | $\mathbf{0.7686\pm 0.0564}$ |
Dice |  | $0.0655\pm 0.0164$ | $0.7559\pm 0.0519$ |
MSE |  | $0.0897\pm 0.0216$ | $0.7568\pm 0.0451$ |
softsign | BCE |  | $0.0672\pm 0.0192$ | $0.764\pm 0.0519$ |
Dice |  | $0.0708\pm 0.0164$ | $0.7485\pm 0.055$ |
MSE |  | $0.0963\pm 0.0255$ | $0.7622\pm 0.0533$ |
linear | BCE |  | $0.04\pm 0.0083$ | $0.7335\pm 0.0501$ |
Dice |  | $0.059\pm 0.0116$ | $0.6834\pm 0.0793$ |
MSE |  | $0.0457\pm 0.0098$ | $0.7359\pm 0.0471$ |
hardtanh | BCE |  | $0.0512\pm 0.0098$ | $0.6517\pm 0.0664$ |
Dice |  | $\underline{0.2201\pm 0.0373}$ | $0.6914\pm 0.0707$ |
MSE |  | $0.2076\pm 0.2268$ | $\underline{0.605\pm 0.2861}$ |

The worst results are obtained with the hardtanh activation function.
hardtanh+MSE failed to converge on the last cross-validation fold and stopped
with a dice coefficient close to $0$. This explains the large standard
deviation of the MSE+hardtanh combination.

In this table, the effective domain has a partial influence on NLL+BCE. As we
move from sigmoid to softsign, the NLL increases. However, dice loss in
combination with NLL shows again the opposite effect.

If we look further at the winning models, as shown in Figure 14, we can see
that the ground truth area is well determined by the neural network. In (b)
there is higher uncertainty at the tips of the C-shaped prediction. In
contrast, (a) has more certainty than (b) for the top tip. For the non-border
area, the probability is close to one.

(a) BCE and arctan

(b) BCE and sigmoid

(c) Ground truth

Figure 14: Best predictions for a single image in MSD.

The worst predictions according to the table are shown in Figure 15. Since
MSE+hardtanh had convergence problems, we show BCE+hardtanh instead.

Hardtanh in combination with BCE loss results in poor segmentation with a high
level of uncertainty. In contrast, the combination with dice loss does not
produce any segmentation.

(a) BCE and hardtanh

(b) Dice and hardtanh

(c) Ground truth

Figure 15: Worst predictions for a single image in MSD.

Figure 16 shows the reliability diagrams of the best two models. In (a) the
BCE+sigmoid combination seems closer to the perfectly calibrated line, but in
(b) BCE+arctan is better. The adaptive strategy leads to more bins at $0$ and
$0.2$ for arctangent. The plot in (b) is more trustworthy, as adaptivity leads
to a better distributed plot.

Both combinations are not calibrated. The diagrams are similar to the previous
calibration diagrams and show overestimation and underestimation of the
probabilities. We see that arctan, which is an activation function with a
wider effective domain, produces better calibrated predictions than sigmoid.

0.00.20.40.60.81.0Confidence0.00.20.40.60.81.0Fraction of positivesReliability
diagramPerfectly calibratedBCE+arctanBCE+sigmoid

(a) Evenly spaced.

0.00.20.40.60.81.0Confidence0.00.20.40.60.81.0Fraction of positivesReliability
diagramPerfectly calibratedBCE+arctanBCE+sigmoid

(b) Adaptive.

Figure 16: Reliability diagrams on the MSD dataset for the best performing
models compared to the sigmoid function as the output activation function.

##  5 DISCUSSION

#### General Observations

The results of the conducted analysis have shown that the output activation
function in combination with the loss function has an important impact on the
segmentation result in medical imaging.

We have hypothesized in section 3.1 that the rate of change of the activation
function affects the range of the probabilities on the segmentation mask. From
the metrics, reliability diagrams, and examples, we see that the activation
function has a strong influence on the prediction results. However, the effect
on the range of the probabilities is not always obvious. For this reason, we
plot in Figure 17 the probability density function of two extreme examples:
CDF (small effective domain) and softsign (large effective domain).

For all datasets, the peaks at 0.0 and 1.0 when considering BCE+softsign are
slightly shifted to the right and left. This means that the network tends to
make fewer certain predictions. By choosing a different activation, we can
thus affect the distribution of the probabilities.

However, a less certain prediction is not always better, as we saw in the
tables from the last section. The softsign function did not do as well as CDF
or sigmoid in terms of segmentation errors. A smaller effective domain tends
to improve the segmentation error. One explanation is that we are restricting
the network’s freedom of action. When we reduce the domain from $[-6,6]$ to
$[-3,3]$, all the weights in the neural network must decrease. This then also
affects the backpropagation during training and can lead to better local
minima.

The advantage of having a larger effective domain is that it can improve the
reliability of the predictions. If we just look at the BCE loss function in
the graphs in the last section, we can see that a larger effective domain
tends to increase reliability. For the ISLES dataset, sigmoid is better than
CDF. For the MSD dataset, arctangent is better than sigmoid.

Two special cases are the linear activation and the hardtanh function. The
linear activation adapts the domain to the values in the image. For this
reason, it also gives good results, although it approaches the values 0 and 1
linearly. The hardtanh function has a small effective domain and should also
give good results. However, because the function has discontinuities and not a
good derivative, it has not always been able to converge or has had difficulty
doing so. This could also explain why hardtanh rarely performed well.

0.00.20.40.60.81.001234567DensityBCE+softsignBCE+CDF |  0.00.20.40.60.81.00.00.20.40.60.81.01.21.4DensityBCE+softsignBCE+CDF
---|---
(a) ACDC | (b) ISLES
0.00.20.40.60.81.00.00.51.01.52.02.5DensityBCE+softsignBCE+CDF |  0.00.20.40.60.81.00.00.51.01.52.02.53.03.5DensityBCE+softsignBCE+CDF
(c) Kvasir-SEG | (d) MSD
Figure 17: Kernel density estimation for $P(\hat{Y}\mid Y=1)$ of the first
validation fold

Another aspect related to the effective domain is the loss function. We have
seen that a greater effective domain can improve the probabilities for dice
loss. Table 6 shows the reverse effect of what we observed for MSE/BCE loss.
Furthermore, this decreasing effect is even clearer than the increasing effect
for MSE/BCE.

Dataset | normal CDF | sigmoid | inverse sqrt | arctangent | softsign
---|---|---|---|---|---
ACDC | $0.2033$ | $0.1409$ | $0.1131$ | $0.0992$ | $0.0965$
ISLES | $0.1084$ | $0.0905$ | $0.0712$ | $0.0608$ | $\underline{0.0644}$
Kvasir-SEG | $1.3776$ | $1.1301$ | $0.7643$ | $0.5441$ | $0.5072$
MSD | $0.1727$ | $0.1123$ | $0.077$ | $0.0655$ | $\underline{0.0708}$
Table 6: NLL error for models trained with dice loss. Activation functions
that do not show a decreasing behavior were underlined.

The effect is reversed because dice loss does not show any uncertainties and
optimizes the overlap. When we decrease the rate of change, it improves the
probabilistic estimates and the segmentation error. Since dice loss takes into
account the sum of many pixels at the same time, a change in the activation
function enables better adaptation of the error of individual pixels.

Apart from the tables and the reliability diagrams, we also analyzed the
results visually. We have seen that good predictions have a high probability
in the core area and low probabilities in the border area. If the uncertainty
for the whole area is high, the dice coefficient tends to be low. Thus, we
observed that the uncertainty also correlates with the dice coefficient.

Furthermore, we have seen that small differences in the dice coefficient are
not always obvious when looking at a single image. If the dice coefficient
differs only slightly, there are no major changes in the predictions. However,
if a function performs significantly worse, this is also reflected in the
predicted output.

#### Interpretation of Test Results.

We aggregate the results from Section 4 to compare them against sigmoid. The
best activation+loss function combination is contrasted with the best
sigmoid+loss function combination, as shown in Table 7.

Activation function | Won NLL | Won Dice | Sum
---|---|---|---
Normal CDF | 2 | 3 | 5
Inverse Square Root | 0 | 2 | 2
Arctangent | 0 | 3 | 3
Softsign | 0 | 2 | 2
HardTanh | 0 | 2 | 2
Linear | 2 | 2 | 4
Table 7: Number of models with output activation functions that perform better
than sigmoid output activation function across all datasets. Except for
hardtanh and linear, all functions were ordered by their effective domain.

Of a total of $4$ datasets, the normal CDF won 3 times against sigmoid with
respect to the optimal dice coefficient. Regarding the negative log-
likelihood, it is a tie. Hence, the normal CDF activation is at least as good
as sigmoid for the given datasets with respect to the segmentation error.

From Table 7, we also see that the linear activation fared well against
sigmoid. Here, we can determine that the sigmoid or CDF activation function
should not be used exclusively. Instead, we suggest testing multiple output
activation functions. In this paper, we have shown that there are several
functions that can be used as output activation functions that outperform the
sigmoid function. Table 8 shows the results of a few activation functions that
produced good results across all datasets.

Activation function | Loss function | Avg. dice coefficient
---|---|---
Sigmoid | dice loss | 0.726575
Arctangent | dice loss | 0.742
Linear | MSE | 0.748875
Sigmoid | BCE | 0.752425
Normal CDF | BCE | 0.753675
Table 8: Average dice coefficient across all datasets for a selected number of
output activation and loss functions.

Next, we count how many times each loss function won by dataset and activation
function.

Loss | Won NLL | Won Dice |  | Sum
---|---|---|---|---
BCE | 21 | 13 |  | 34
Dice | 0 | 6 |  | 6
MSE | 7 | 9 |  | 16
Table 9: The number of models with the respective loss functions that have won
the evaluation of the four datasets presented. The maximum of each of the
first two columns is 7 activation functions times 4 datasets.

The table shows that BCE was the best loss function for most activation
functions and datasets. By using BCE as the loss function we optimize NLL, but
in some cases, MSE was still the better loss function. Therefore, the metric
does not always have to be the same as the loss function.

Given the optimal threshold, the best single dice coefficient is more
frequently found by using BCE or MSE. For BCE/MSE, the threshold of the dice
coefficient is an additional parameter that increases or reduces the predicted
area. However, optimization of this parameter can also lead to overfitting to
the validation set. Dice loss has not the same issue, as it is largely
threshold-independent. One possible reason that BCE/MSE is better, then, is
the lack of an additional parameter that controls the predicted area.

Another reason is that dice loss is only a surrogate loss function to the dice
coefficient because it relaxes the constraint $\hat{y}\in\\{0,1\\}$ to
$\hat{y}\in[0,1]$ where $\hat{y}$ is the predicted pixel. While this makes the
loss function differentiable, it is no longer the dice coefficient. Then it is
not surprising that other loss functions can be better.

Our empirical result that dice loss is inferior to cross entropy contradicts
the findings by Bertels et al.[45]. One possible reason is that we stopped
training the network when the Dice coefficient stopped improving.
Comparatively, the authors stopped the training when the validation loss did
not reduce anymore. Another reason is that we used a pre-trained encoder and a
2D U-Net. We also obtained a 10-20% higher dice coefficient than the authors
on ISLES 2018 with 5-fold cross-validation. Our code is available on GitHub
for further testing.

#### Limitations and Future Directions.

A limitation of this paper is that we did not consider all possible loss
functions due to resource constraints. The Lovász-Softmax loss[46] is another
way to obtain a surrogate function of the dice coefficient. There are also
many extensions to dice loss[47, 48] that could be considered in further work.

Apart from that, we treated multi-label segmentation with the datasets ACDC
and MSD as multiple binary segmentation problems by applying the sigmoid
instead of the softmax function. The difference is that the softmax function
normalizes the sum of all classes to $1$. The softmax function corresponds to
a categorical distribution, while our sigmoid function corresponds to multiple
Bernoulli distributions. Hence, for proper multi-label classification, the
proposed activation functions have to be extended to produce a categorical
distribution.

We only considered the case of medical image segmentation. Further work could
also evaluate the effect of the output activation in regular neural networks
or other tasks like classification. Trainable activation functions could also
be considered. Molina et al. [19] have shown with their Padé Activation Unit
that trainable functions can surpass regular functions like ReLU.

Finally, another promising direction is to analyze more the effect of the
activation function on the reliability of the probabilities.

##  6 CONCLUSIONS

In this paper, we proposed alternative activation functions to the sigmoid
function. We motivated this by examining the rate of change of these
functions. Our results have shown that the activation function affects both
the segmentation error and the calibration of the neural network. Functions
with a high rate of change tend to produce better segmentation results, while
functions with a lower rate of change can produce better calibration. In
particular, we saw that the normal CDF is a viable alternative to sigmoid in
binary segmentation. Furthermore, we showed that the loss function has also an
impact on which activation function is best suited for segmentation. When
changing the loss function to dice loss, arctangent gave the best results. We
encourage further research to find better output activation functions.

In future research, we aim to extend our analysis to other application areas
of deep learning, such as mechanical engineering or natural language
processing. Further, we want to provide visual assistance tools that make it
possible to communicate the effectiveness of a chosen output activation
function.

##  7 CONFLICT OF INTEREST

The authors declare that there is no conflict of interest.

##  8 APPENDIX

###  8.1 Automated Cardiac Diagnosis Challenge (ACDC)

|  |  |
---|---|---|---
(a) BCE+Arctangent | (b) Dice+Arctangent | (c) MSE+Arctangent | (d) BCE+CDF
|  |  |
(e) Dice+CDF | (f) MSE+CDF | (g) BCE+HardTanh | (h) Dice+HardTanh
|  |  |
(i) MSE+HardTanh | (j) BCE+InvSquareRoot | (k) Dice+InvSquareRoot | (l) MSE+InvSquareRoot
|  |  |
(m) BCE+Linear | (n) Dice+Linear | (o) MSE+Linear | (p) BCE+Sigmoid
|  |  |
(q) Dice+Sigmoid | (r) MSE+Sigmoid | (s) BCE+Softsign | (t) Dice+Softsign
|  |  |
(u) MSE+Softsign | (v) Ground truth |  |

Figure 18: All $21$ predictions for a single image of the ACDC dataset, with
the last image being the ground truth.

###  8.2 Ischemic Stroke Lesion Segmentation Challenge 2018 (ISLES)

|  |  |
---|---|---|---
(a) BCE+Arctangent | (b) Dice+Arctangent | (c) MSE+Arctangent | (d) BCE+CDF
|  |  |
(e) Dice+CDF | (f) MSE+CDF | (g) BCE+HardTanh | (h) Dice+HardTanh
|  |  |
(i) MSE+HardTanh | (j) BCE+InvSquareRoot | (k) Dice+InvSquareRoot | (l) MSE+InvSquareRoot
|  |  |
(m) BCE+Linear | (n) Dice+Linear | (o) MSE+Linear | (p) BCE+Sigmoid
|  |  |
(q) Dice+Sigmoid | (r) MSE+Sigmoid | (s) BCE+Softsign | (t) Dice+Softsign
|  |  |
(u) MSE+Softsign | (v) Ground truth |  |

Figure 19: All $21$ predictions for a single image of the ISLES dataset, with
the last image being the ground truth.

###  8.3 Kvasir-SEG

|  |  |
---|---|---|---
(a) BCE+Arctangent | (b) Dice+Arctangent | (c) MSE+Arctangent | (d) BCE+CDF
|  |  |
(e) Dice+CDF | (f) MSE+CDF | (g) BCE+HardTanh | (h) Dice+HardTanh
|  |  |
(i) MSE+HardTanh | (j) BCE+InvSquareRoot | (k) Dice+InvSquareRoot | (l) MSE+InvSquareRoot
|  |  |
(m) BCE+Linear | (n) Dice+Linear | (o) MSE+Linear | (p) BCE+Sigmoid
|  |  |
(q) Dice+Sigmoid | (r) MSE+Sigmoid | (s) BCE+Softsign | (t) Dice+Softsign
|  |  |
(u) MSE+Softsign | (v) Ground truth |  |

Figure 20: All $21$ predictions for a single image of the Kvasir-SEG dataset,
with the last image being the ground truth.

###  8.4 Medical Segmentation Decathlon (MSD)

|  |  |
---|---|---|---
(a) BCE+Arctangent | (b) Dice+Arctangent | (c) MSE+Arctangent | (d) BCE+CDF
|  |  |
(e) Dice+CDF | (f) MSE+CDF | (g) BCE+HardTanh | (h) Dice+HardTanh
|  |  |
(i) MSE+HardTanh | (j) BCE+InvSquareRoot | (k) Dice+InvSquareRoot | (l) MSE+InvSquareRoot
|  |  |
(m) BCE+Linear | (n) Dice+Linear | (o) MSE+Linear | (p) BCE+Sigmoid
|  |  |
(q) Dice+Sigmoid | (r) MSE+Sigmoid | (s) BCE+Softsign | (t) Dice+Softsign
|  |  |
(u) MSE+Softsign | (v) Ground truth |  |

Figure 21: All $21$ predictions for a single image of the MSD dataset, with
the last image being the ground truth.

## References

  * [1] Ramesh, K., Kumar, G. K., Swapna, K., Datta, D., and Rajest, S. S., “A review of medical image segmentation algorithms,” EAI Endorsed Transactions on Pervasive Health and Technology 7 (4 2021).
  * [2] Febrianto, D. C., Soesanti, I., and Nugroho, H. A., “Convolutional neural network for brain tumor detection,” IOP Conference Series: Materials Science and Engineering 771, 012031 (mar 2020).
  * [3] Gillmann, C., Hempel, L., Schmidt, C., Saur, D., and Scheuermann, G., “Visualizing Multi-Modal Deep Learning for Lesion Prediction,” in [IEEE Computer Graphics and Applications ], (2021).
  * [4] Liu, L., Tsui, Y. Y., and Mandal, M., “Skin lesion segmentation using deep learning with auxiliary task,” Journal of Imaging 7(4) (2021).
  * [5] Nwankpa, C., Ijomah, W., Gachagan, A., and Marshall, S., “Activation functions: Comparison of trends in practice and research for deep learning,” CoRR abs/1811.03378 (2018).
  * [6] Gillmann, C., Smit, N. N., Gröller, E., Preim, B., Vilanova, A., and Wischgoll, T., “10 Challenges in Medical Visualization,” in [IEEE Computer Graphics and Applications ], (2021).
  * [7] Gillmann, C., Saur, D., Wischgoll, T., and Scheuermann, G., “Uncertainty-aware Visualization in Medical Imaging - A Survey,” Computer Graphics Forum (2021).
  * [8] Gillmann, C., Post, T., Wischgoll, T., Hagen, H., and Maciejewski, R., “Hierarchical image semantics using probabilistic path propagations for biomedical research,” IEEE Computer Graphics and Applications , 1–1 (2019).
  * [9] Lei, T., Wang, R., Wan, Y., Zhang, B., Meng, H., and Nandi, A. K., “Medical image segmentation using deep learning: A survey,” (2020).
  * [10] Horowitz, J. L. and Savin, N. E., “Binary response models: Logits, probits and semiparametrics,” Journal of Economic Perspectives 15, 43–56 (December 2001).
  * [11] Cakmakyapan, S. and Goktas, A., “A comparison of binary logit and probit models with a simulation study,” Journal of Social and Economic Statistics 2(1), 1–17 (2013).
  * [12] Hoetker, G., “The use of logit and probit models in strategic management research: Critical issues,” Strategic Management Journal 28(4), 331–343 (2007).
  * [13] de Faria, R. Q., dos Santos, A. R. P., Amorim, D. J., Cantão, R. F., da Silva, E. A. A., and Sartori, M. M. P., “Probit or logit? which is the better model to predict the longevity of seeds?,” Seed Science Research 30(1), 49–58 (2020).
  * [14] Gomes, G. S. d. S. and Ludermir, T. B., “Complementary log-log and probit: Activation functions implemented in artificial neural networks,” in [2008 Eighth International Conference on Hybrid Intelligent Systems ], 939–942 (2008).
  * [15] ZENG, L., “Prediction and classification with neural network models,” Sociological Methods & Research 27, 499–524 (May 1999).
  * [16] Ramachandran, P., Zoph, B., and Le, Q. V., “Searching for activation functions,” CoRR abs/1710.05941 (2017).
  * [17] Ma, N., Zhang, X., and Sun, J., “Activate or not: Learning customized activation,” CoRR abs/2009.04759 (2020).
  * [18] Hendrycks, D. and Gimpel, K., “Bridging nonlinearities and stochastic regularizers with gaussian error linear units,” CoRR abs/1606.08415 (2016).
  * [19] Molina, A., Schramowski, P., and Kersting, K., “Padé activation units: End-to-end learning of flexible activation functions in deep networks,” in [International Conference on Learning Representations ], (2020).
  * [20] Apicella, A., Donnarumma, F., Isgrò, F., and Prevete, R., “A survey on modern trainable activation functions,” Neural Networks 138, 14–32 (Jun 2021).
  * [21] Chatla, S. and Shmueli, G., “Linear probability models (LPM) and big data: The good, the bad, and the ugly,” SSRN Electronic Journal (2013).
  * [22] Goodfellow, I., Bengio, Y., and Courville, A., [Deep Learning ], MIT Press (2016).  http://www.deeplearningbook.org.
  * [23] Antonelli, M., Reinke, A., Bakas, S., Farahani, K., AnnetteKopp-Schneider, Landman, B. A., Litjens, G., Menze, B., Ronneberger, O., Summers, R. M., van Ginneken, B., Bilello, M., Bilic, P., Christ, P. F., Do, R. K. G., Gollub, M. J., Heckers, S. H., Huisman, H., Jarnagin, W. R., McHugo, M. K., Napel, S., Pernicka, J. S. G., Rhode, K., Tobon-Gomez, C., Vorontsov, E., Huisman, H., Meakin, J. A., Ourselin, S., Wiesenfarth, M., Arbelaez, P., Bae, B., Chen, S., Daza, L., Feng, J., He, B., Isensee, F., Ji, Y., Jia, F., Kim, N., Kim, I., Merhof, D., Pai, A., Park, B., Perslev, M., Rezaiifar, R., Rippel, O., Sarasua, I., Shen, W., Son, J., Wachinger, C., Wang, L., Wang, Y., Xia, Y., Xu, D., Xu, Z., Zheng, Y., Simpson, A. L., Maier-Hein, L., and Cardoso, M. J., “The medical segmentation decathlon,” (2021).
  * [24] Ashukha, A., Lyzhov, A., Molchanov, D., and Vetrov, D., “Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning,” in [International Conference on Learning Representations ], (2020).
  * [25] Gneiting, T. and Katzfuss, M., “Probabilistic Forecasting,” Annual Review of Statistics and Its Application 1(1), 125–151 (2014).
  * [26] Brier, G. W., “Verification of forecasts expressed in terms of probability,” Monthly Weather Review 78, 1–3 (Jan. 1950).
  * [27] Bishop, C. M., [Pattern Recognition and Machine Learning ], Springer (2006).
  * [28] Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q., “On Calibration of Modern Neural Networks,” in [Proceedings of the 34th International Conference on Machine Learning - Volume 70 ], ICML’17, 1321–1330, JMLR.org (2017).
  * [29] Nixon, J., Dusenberry, M., Zhang, L., Jerfel, G., and Tran, D., “Measuring calibration in deep learning,” CoRR abs/1904.01685 (2019).
  * [30] Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.-A., Cetin, I., Lekadir, K., Camara, O., Gonzalez Ballester, M. A., Sanroma, G., Napel, S., Petersen, S., Tziritas, G., Grinias, E., Khened, M., Kollerathu, V. A., Krishnamurthi, G., Rohé, M.-M., Pennec, X., Sermesant, M., Isensee, F., Jäger, P., Maier-Hein, K. H., Full, P. M., Wolf, I., Engelhardt, S., Baumgartner, C. F., Koch, L. M., Wolterink, J. M., Išgum, I., Jang, Y., Hong, Y., Patravali, J., Jain, S., Humbert, O., and Jodoin, P.-M., “Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: Is the problem solved?,” IEEE Transactions on Medical Imaging 37(11), 2514–2525 (2018).
  * [31] Maier, O., Menze, B. H., von der Gablentz, J., Häni, L., Heinrich, M. P., Liebrand, M., Winzeck, S., Basit, A., Bentley, P., Chen, L., Christiaens, D., Dutil, F., Egger, K., Feng, C., Glocker, B., Götz, M., Haeck, T., Halme, H.-L., Havaei, M., Iftekharuddin, K. M., Jodoin, P.-M., Kamnitsas, K., Kellner, E., Korvenoja, A., Larochelle, H., Ledig, C., Lee, J.-H., Maes, F., Mahmood, Q., Maier-Hein, K. H., McKinley, R., Muschelli, J., Pal, C., Pei, L., Rangarajan, J. R., Reza, S. M., Robben, D., Rueckert, D., Salli, E., Suetens, P., Wang, C.-W., Wilms, M., Kirschke, J. S., Krämer, U. M., Münte, T. F., Schramm, P., Wiest, R., Handels, H., and Reyes, M., “ISLES 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI,” Medical Image Analysis 35, 250–269 (Jan. 2017).
  * [32] Winzeck, S., Hakim, A., McKinley, R., Pinto, J. A. A. D. S. R., Alves, V., Silva, C., Pisov, M., Krivov, E., Belyaev, M., Monteiro, M., Oliveira, A., Choi, Y., Paik, M. C., Kwon, Y., Lee, H., Kim, B. J., Won, J.-H., Islam, M., Ren, H., Robben, D., Suetens, P., Gong, E., Niu, Y., Xu, J., Pauly, J. M., Lucas, C., Heinrich, M. P., Rivera, L. C., Castillo, L. S., Daza, L. A., Beers, A. L., Arbelaezs, P., Maier, O., Chang, K., Brown, J. M., Kalpathy-Cramer, J., Zaharchuk, G., Wiest, R., and Reyes, M., “ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI,” Frontiers in Neurology 9 (Sept. 2018).
  * [33] Jha, D., Smedsrud, P. H., Riegler, M. A., Halvorsen, P., de Lange, T., Johansen, D., and Johansen, H. D., “Kvasir-seg: A segmented polyp dataset,” in [International Conference on Multimedia Modeling ], 451–462, Springer (2020).
  * [34] Flanders, A. E., Prevedello, L. M., Shih, G., Halabi, S. S., Kalpathy-Cramer, J., Ball, R., Mongan, J. T., Stein, A., Kitamura, F. C., Lungren, M. P., Choudhary, G., Cala, L., Coelho, L., Mogensen, M., Morón, F., Miller, E., Ikuta, I., Zohrabian, V., McDonnell, O., Lincoln, C., Shah, L., Joyner, D., Agarwal, A., Lee, R. K., and and, J. N., “Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge,” Radiology: Artificial Intelligence 2, e190211 (5 2020).
  * [35] Clèrigues, A., Valverde, S., Bernal, J., Freixenet, J., Oliver, A., and Lladó, X., “Acute ischemic stroke lesion core segmentation in CT perfusion images using fully convolutional neural networks,” Computers in Biology and Medicine 115, 103487 (2019).
  * [36] Liu, P., “Stroke Lesion Segmentation with 2D Novel CNN Pipeline and Novel Loss Function,” in [Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries ], Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., and van Walsum, T., eds., 253–262, Springer International Publishing, Cham (2019).
  * [37] Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmentation,” CoRR abs/1505.04597 (2015).
  * [38] Huh, M.-Y., Agrawal, P., and Efros, A. A., “What makes imagenet good for transfer learning?,” CoRR abs/1608.08614 (2016).
  * [39] Iglovikov, V. and Shvets, A., “TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation,” CoRR abs/1801.05746 (2018).
  * [40] Iglovikov, V. I., Seferbekov, S. S., Buslaev, A. V., and Shvets, A., “Ternausnetv2: Fully convolutional network for instance segmentation,” CoRR abs/1806.00844 (2018).
  * [41] He, K., Zhang, X., Ren, S., and Sun, J., “Deep residual learning for image recognition,” CoRR abs/1512.03385 (2015).
  * [42] Kingma, D. P. and Ba, J., “Adam: A method for stochastic optimization.,” CoRR abs/1412.6980 (2014).
  * [43] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” (2019).
  * [44] Czolbe, S., Arnavaz, K., Krause, O., and Feragen, A., “Is segmentation uncertainty useful?,” CoRR abs/2103.16265 (2021).
  * [45] Bertels, J., Eelbode, T., Berman, M., Vandermeulen, D., Maes, F., Bisschops, R., and Blaschko, M. B., “Optimizing the dice score and jaccard index for medical image segmentation: Theory & practice,” CoRR abs/1911.01685 (2019).
  * [46] Berman, M. and Blaschko, M. B., “Optimization of the jaccard index for image segmentation with the lovász hinge,” CoRR abs/1705.08790 (2017).
  * [47] Yeung, M., Sala, E., Schönlieb, C.-B., and Rundo, L., “Unified focal loss: Generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation,” (2021).
  * [48] Salehi, S. S. M., Erdogmus, D., and Gholipour, A., “Tversky loss function for image segmentation using 3d fully convolutional deep networks,” CoRR abs/1706.05721 (2017).