532
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
lead to convergence. Usually picking the right learning rate is a matter of trial
and error. It is also possible and common to vary the learning rate. Start with
an initial learning rate η0. Then, at each iteration, multiply the learning rate
by a factor β (0 < β < 1) until the learning rate reaches a suﬃciently low value.
When we have a large training set, we may not want to use the entire
training set for each iteration, as it might be too time-consuming. So for each
iteration we randomly sample a “minibatch” of training examples. This variant
is called stochastic gradient descent,” as was discussed in Section 12.3.5, since
we estimate the gradients using a diﬀerent sample of the training set at each
iteration.
We have left open the question of how the parameter values are initialized
before we start gradient descent.
The usual approach is to choose them at
random. Popular approachaes include sampling each entry uniformly at ran-
dom in [−1, 1], or choosing randomly using a normal distribution. Notice that
initializing all the weights to the same value would cause all nodes in a layer
to behave the same way, and thus we would never reap the beneﬁt of having
diﬀerent nodes in a layer recognize diﬀerent features of the input.
13.3.5
Tensors
Previously, we have imagined that the inputs to a neural net are one-dimensional
vectors. But there is no reason why we cannot view the input as having a higher
dimension.
Example 13.6 : A grey-scale photo might be represented by a two-dimensional
array of real numbers, corresponding to the intensity of each pixel. Each pixel
of a color image typically requires 3 dimensions. That is, each pixel itself is
a vector with three components, say for the intensity of the pixel in the red,
green, and blue colors. One useful way to view a color image as input to a
neural net is to think of each training example as a two-dimensional array of
pixels, where the value of each pixel is not a real number, as we have heretofore
imagined, but a vector with three dimensions, one for each of the three colors.
✷
Similarly, we have viewed each layer of a neural net as a column of nodes.
But there is no reason we cannot imagine the nodes in a layer to be organized
as a two-dimensional array or even an array of dimension greater than two.
Finally, we have viewed the input values as real numbers and similarly viewed
the values produced by each node as a real number. But we could also think of
the values attached to each input or output of a node as a vector or a higher-
dimensional structure. The natural generalization of vectors and matrices is
the tensor, which is an n-dimensional array of scalars.
Unfortunately, the backpropagation algorithm we described works for vec-
tors, not higher-dimensional tensors. In such cases, we resort to the same trick
we used in Section 13.3.3, where we unrolled the matrix W into collection of
