520
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
Accuracy of Softmax Calculation
The denominator of the softmax function involves computing a sum of the
form P
j exj. When the xj’s take a wide range of values, their exponents
exj take on an even wider range of values – some tiny and some very large.
Adding very large and very small ﬂoating point numbers leads to numerical
inaccuracy issues in ﬁxed-width ﬂoating point representations (such as 32-
bit or 64-bit). Fortunately, there is a trick to avoid this problem. We
observe that
µ(xi) =
exi
P
j exj =
exi−c
P
j exj−c
for any constant c. We pick c = maxj xj, so that xj −c ≤0 for all j.
This ensures that exj−c is always between 0 and 1, and leads to a more
accurate calculation.
Most deep learning frameworks will take care to
compute softmax in this manner.
Section 13.2.9. We address the problem of diﬀerentiating the softmax function
in Section 13.3.3.
13.2.6
Reciﬁed Linear Unit
The rectiﬁed linear unit, or ReLU, is deﬁned as:
f(x) = max(0, x) =
(
x,
for x ≥0
0,
for x < 0
The name of this function derives from the analogy to half-wave rectiﬁcation in
electrical engineering. The function is not diﬀerentiable at 0 but is diﬀerentiable
everywhere else, including at points arbitrarily close to 0. In practice, we “set”
the derivative at 0 to be either 0 (the left derivative) or 1 (the right derivative).
In modern neural nets, a version of ReLU has replaced sigmoid as the de-
fault choice of activation function. The popularity of ReLU derives from two
properties:
1. The gradient of ReLU remains constant and never saturates for positive
x, speeding up training.
It has been found in practice that networks
that use ReLU oﬀer a signiﬁcat speedup in training compared to sigmoid
activation.
2. Both the function and its derivative can be computed using elementary
and eﬃcient mathematical operations (no exponentiation).
ReLU does suﬀer from a problem related to the saturation of its derivative
when x < 0. Once a node’s input values become negative, it is possible that the
