13.2. DENSE FEEDFORWARD NETWORKS
519
6
4
2
0
2
4
6
0.5
1.0
3
2
1
1
2
3
1.0
0.5
0.5
1.0
(a)
(b)
Figure 13.4: The logistic sigmoid (a) and hyperbolic tangent (b) functions
Figure 13.4 shows the logistic sigmoid and hyperbolic tangent functions.
Note the diﬀerence in scale along the x-axis between the two charts. It is easy
to see that the functions are identical after shifting and scaling.
13.2.5
Softmax
The softmax function diﬀers from sigmoid functions in that it does not operate
element-wise on a vector. Rather, the softmax function applies to an entire
vector. If x = [x1, x2, . . . , xn], then its softmax µ(x) = [µ(x1), µ(x2), . . . , µ(xn)]
where
µ(xi) =
exi
P
j exj
Softmax pushes the largest component of the vector towards 1 while pushing all
the other components towards zero. Also, all the outputs sum to 1, regardless
of the sum of the components of the input vector. Thus, the output of the
softmax function can be intepreted as a probability distribution.
A common application is to use softmax in the output layer for a classi-
ﬁcation problem. The output vector has a component corresponding to each
target class, and the softmax output is interpreted as the probability of the
input belonging to the corresponding class.
Softmax has the same saturation problem as the sigmoid function, since one
component gets larger than all the others. There is a simple workaround to this
problem, however, when softmax is used at the output layer. In this case, it is
usual to pick cross entropy as the loss function, which undoes the exponentiation
in the deﬁnition of softmax and avoids saturation. Cross entropy is explained in
