13.3. BACKPROPAGATION AND GRADIENT DESCENT
533
vectors. Just as we regard an m × n matrix as a set of m n-vectors, we can re-
gard a 3-dimensional tensor of dimensionality l×m×n as a set of lm n-vectors,
and similarly for tensors of higher dimension.
Example 13.7 : This example is based on the MNIST dataset.6 This dataset
consists of 28 × 28 monochrome images, each represented by a two-dimensional
square bit array whose sides are of length 28. Our goal is to build a neural net
that determines whether an image corresponds to a handwritten digit (0-9) and
if so which one. Consider a single image X, which is a 28 × 28 matrix. Suppose
the ﬁrst layer of our network is a dense layer7 consisting of 49 hidden nodes,
which we shall imagine is arranged in a 7 × 7 array. We model the hidden layer
as a 7 × 7 matrix H, where the output of the node in row i and column j is hij.
We can model the weights for each of the 28 × 28 inputs and each of the
7×7 nodes as a weight tensor W with dimensions 7×7×28×28. That is, Wijkl
represents the weight for the input pixel in row i and column j of the image to
the node whose position in the array of nodes is row k and column l. Then:
hij =
28
X
k=1
28
X
l=1
wijklxkl for 1 ≤i, j ≤7
where we omit the bias term for simplicity (i.e., we assume all thresholds of all
nodes are 0.
An equivalent way to think about this structure to ﬂatten the input X into
a vector x of length 784 (since 28 × 28 = 784) and the hidden layer H into a
vector h of length 49. We ﬂatten the weight tensor W as follows: the last two
dimensions are ﬂattened into a single dimension to match x, and its ﬁrst two
dimensions are ﬂattened into a single dimension to match the hidden vector h,
resulting in a 49 × 784 weight matrix. Suppose as in Section 13.2.1, we have
wT
i denote the ith row of this new weight matrix. We can now write:
hi = wT
i x for 1 ≤i ≤49
It’s straightforward to see that there is a 1-to-1 mapping between the hidden
nodes in the old and new arrangements. Moreover, the output of each hidden
node is determined by a dot product, just as in Section 13.2.1. Thus the tensor
notation is just a convenient way to group vectors.
✷
Thus the tensors used in neural networks have little in common with the
tensors used in Physics and other mathematical sciences. A tensor in our con-
text is just a nested collection of vectors. The only tensor operation we shall
need is the ﬂattening of a tensor by merging dimensions as in Example 13.7. We
can use the backpropagation algorithm described in Section 13.3.3 for tensors
once we have ﬂattened them approriately.
6See yann.lecun.com/exdb/mnist/.
7In reality, the ﬁrst network layer for this problem is likely to be a convolutional layer.
See Section 13.4
