516
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
Why Use Linear Algebra?
Notational brevity is one reason to use linear algebra notation for neu-
ral networks. Another is performance. It turns out that graphics pro-
cessing units (GPU’s) have circuitry that allows for highly parallelized
linear-algebra operations. Mutiplying a matrix and a vector using a sin-
gle linear algebra operator is much faster than coding the same operator
using nested loops. Modern deep-learning frameworks (e.g., TensorFlow,
PyTorch, Caﬀe) harness the power of GPU’s to dramatically speed up
neural network computations.
The threshold inputs to the hidden layer nodes (i.e., the negatives of the thresh-
olds) form a 2-vector b = [b1, b2], often called the bias vector. The perceptron
applies the nonlinear step function to produce its output, deﬁned as:
step(z) =
(
1
when z > 0
0
otherwise
Each hidden node hi can now be described using the expression:
hi = step(wT
i x + bi) for i = 1, 2
We could organize the weight vectors w1 and w2 into a 2 × 4 weight matrix
W, where the ith row of W is wT
i . The hidden nodes can thus be described
using the expression:
h = step(Wx + b)
In the case of a vector input, the step function just operates element-wise on
the vector to produce an output vector of the same length. We can use a similar
arrangement to describe the transformation that produces the ﬁnal output from
the hidden layer. In this case, the ﬁnal output is a scalar y, so instead of a weight
matrix W we need only a weight vector u = [u1, u2] and a single bias c. We
thus have:
y = step(uTh + c)
Linear-algebra notation works just as well when we have larger inputs and
many more nodes in one hidden layer. We just need to scale the weight matrix
and bias vector appropriately. That is, the matrix W has one row for each
node in the layer and one column for each output from the previous layer (or
for each input, if this is the ﬁrst layer); the bias vector has one component
for each node. It is also easy to handle the case where there is more than one
node in the output layer. For example, in a multiclass classiﬁcation problem, we
might have an output node yi corresponding to target class i. For a given input,
the outputs specify the probability that the input belongs to the corresponding
