13.2. DENSE FEEDFORWARD NETWORKS
521
3
2
1
1
2
3
1
1
2
3
3
2
1
1
2
3
1
1
2
3
(a)
(b)
Figure 13.5: The ReLU (a) and ELU (b), with α = 1 functions
node’s output get “stuck” at 0 through the rest of the training. This is called
the dying ReLU problem.
The Leaky ReLU attempts to ﬁx this problem by deﬁning the activation
function as follows:
f(x) =
(
x,
for x ≥0
αx,
for x < 0
where α is typically a small positive value such as 0.01. The Parametric ReLU
(PReLU) makes α a parameter to be optimized as part of the learning process.
An improvement on both the original and leaky ReLU functions is Expo-
nential Linear Unit, or ELU. This function is deﬁned as:
f(x) =
(
x,
for x ≥0
α(ex −1),
for x < 0
where α ≥0 is a hyperparameter. That is, α is held ﬁxed during the learning
process, but we can repeat the learning process with diﬀerent values of α to
ﬁnd the best value for our problem.
The node’s value saturates to −α for
large negative values of x, and a typical choice is α = 1.
ELU’s drive the
mean activation of nodes towards zero, which appears to speed up the learning
process compared to other ReLU variants.
13.2.7
Loss Functions
A loss function quantiﬁes the diﬀerence between a model’s predictions and the
output values observed in the real world (i.e., in the training set). Suppose
