512
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
important part of the design process for neural nets. Especially, note that the
output layer can have many nodes. For instance, the neural net could classify
inputs into many diﬀerent classes, with one output node for each class.
.
.
.
.
.
.
.
.
.
. . .
x
Hidden layers
Output
layer
x
1
2
.
.
.
x n
y
y
y 1
2
k
Input
layer
Figure 13.3: The general case of a neural network
Each layer, except for the input layer, consists of one or more nodes, which
we arrange in the column that represents that layer. We can think of each node
as a perceptron. The inputs to a node are outputs of some or all of the nodes in
the previous layer. So that we can assume the threshold for each node is zero,
we can also allow a node to have an input that is a constant, typically 1, as
we suggested in Fig. 13.2. Associated with each input to each node is a weight.
The output of the node depends on P xiwi, where the sum is over all the inputs
xi, and wi is the weight of that input. Sometimes, the output is either 0 or 1;
the output is 1 if that sum is positive and 0 otherwise. However, as we shall
see in Section 13.2, it is often convenient, when trying to learn the weights for
a neural net that solves some problem, to have outputs that are almost always
close to 0 or 1, but may be slightly diﬀerent. The reason, intuitively, is that
it is then possible for the output of a node to be a continuous function of its
inputs. We can then use gradient descent to converge to the ideal values of all
the weights in the net.
