13.2. DENSE FEEDFORWARD NETWORKS
517
class. This arrangement results in an output vector y = [y1, y2, . . . , yn], where
n is number of classes. The simple network from the prior section had a boolean
output, corresponding to two output classes (true and false), so we could have
modeled the output equally well as a 2-vector. In the case of a vector output,
we connect the hidden and output layers by a weight matrix of appropriate
dimensions in place of the weight vector we used for the example.
The perceptrons in our example used a nonlinear step function. More gen-
erally, we can use any other nonlinear function, called the activation function,
following the linear transformation.
We describe commonly used activation
functions starting in Section 13.2.2.
Our simple example used a single hidden layer of nodes between the input
and output layers. In general, we can have many hidden layers, as was sugested
in Fig. 13.3. Each hidden layer introduces an additional matrix of weights and
vector of biases, as well as its own activation function. This kind of network is
called a feedforward network, since all edges are oriented “forward,” from input
to output, without cycles.
Suppose there are ℓhidden layers and an additional output layer, numbered
ℓ+ 1. Let the weight matrix for the ith layer be Wi and let the bias vector
for that layer be bi. The weights W1, W2, . . . , Wl+1 and biases b1, b2, . . . , bl+1
constitute the parameters of the model. Our objective is to learn the best values
for these parameters to achieve the task at hand. We will soon describe how to
go about learning the model parameters.
13.2.2
Activation Functions
A node (perceptron) in a neural net is designed to give a 0 or 1 (yes or no)
output. Often, we want to modify that output is one of several ways, so we apply
an activation function to the output of a node. In some cases, the activation
function takes all the outputs of a layer and modiﬁes them as a group. the
reason we need an activation function is as follows. The approach we shall use
to learn good parameter values for the network is gradient descent. Thus, we
need activation functions that “play well” with gradient descent. In particular,
we look for activation functions with the following properties:
1. The function is continuous and diﬀerentiable everywhere (or almost ev-
erywhere).
2. The derivative of the function does not saturate (i.e., become very small,
tending towards zero) over its expected input range. Very small deriva-
tives tend to stall out the learning process.
3. The derivative does not explode (i.e., become very large, tending towards
inﬁnity), since this would lead to issues of numerical instability.
The step function does not satisfy conditions (2) and (3). Its derivative explodes
at 0 and is 0 everywhere else. Thus the step function does not play well with
gradient descent and is not a good choice for a deep neural network.
