518
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
13.2.3
The Sigmoid
Given that we cannot use the step function, we look for alternatives in the
class of sigmoid functions – so called because of the S-shaped curve that these
functions exhibit. The most commonly used sigmoid function is the logistic
sigmoid:
σ(x) =
1
1 + e−x =
ex
1 + ex
Notice that the sigmoid has the value 1/2 at x = 0. For large x, the sigmoid
approaches 1, and for large, negtive x, the sigmoid approaches 0.
The logistic sigmoid, like all the functions we shall discuss, are applied to
vectors elementwise, so if x = [x1, x2, . . . , xn] then
σ(x) = [σ(x1), σ(x2), . . . , σ(xn)]
The logistic sigmoid has several advantages over the step function as a way
to deﬁne the output of a perceptron. The logistic sigmoid is continuous and dif-
ferentiable, so it enables us to use gradient descent to discover the best weights.
Since its value is in the range [0, 1], it is possible to interpret the outputs of the
network as a probability. However, the logistic sigmoid saturates very quickly
as we move away from the “critical region” around 0. So the derivative goes
towards zero and gradient-based learning can stall out. That is, weights almost
stop changing, once they get away from 0.
In Section 13.3.3, when we describe the backpropagation algorithm, we shall
see that we need the derivatives of activation functions and loss functions. As
an exercise, you can verify that if y = σ(x), then
dy
dx = y(1 −y)
13.2.4
The Hyperbolic Tangent
Closely related to sigmoid is the hyperbolic tangent function, deﬁned by:
tanh(x) = ex −e−x
ex + e−x
Simple algebraic manipulation yields:
tanh(x) = 2σ(2x) −1
So the hyperbolic tangent is just a scaled and shifted version of the sigmoid.
It has two desirable properties that make it attractive in some situations: its
output is in the range [−1, 1] and is symmetric around 0. It also shares the
good properties and the saturation problem of the sigmoid. You may show that
if y = tanh(x) then
dy
dx = 1 −y2
