13.8. REFERENCES FOR CHAPTER 13
555
the same weights, so the training process therefore needs to deal with a
relatively small number of weights.
✦Long Short-Term Memory Networks: These improve on RNN’s by adding
a second state vector – the cell state – to enable some information about
the sequence to be retained, while most information is forgotten after a
while. In addition, we learn gate vectors that control what information is
retained from the input, the state, and the output.
✦Avoiding Overﬁtting: There are a number of specialized techniques de-
signed to avoid overﬁtting a deep network. These include penalizing large
weights, randomly dropping some nodes each time we apply a step of
gradient descent, and use of a validation set to enable us to stop training
when the loss on the validation set bottoms out.
13.8
References for Chapter 13
For information on TensorFlow, see [12]. You can learn about Pytorch at [10]
and about Caﬀe at [1]. The MNIST database is described in [9].
Backpropagation as the way to train deep neural nets is from [11].
The idea of convolutional neural networks begins with [2], which deﬁned
convolutional layers and pooling layers. However, it was [13] that introduced
the idea of requiring nodes in one convolutional layer to share weights. The
application of these networks to character recognition and other important tasks
is from [8] and [7].
Also see [6] for the application to ImageNet of CNN’s.
ImageNet is available from [5].
Recurrent neural networks ﬁrst appeared in [4]. Long short-term memory
is from [3].
1. http://caffe2.ai
2. Fukushima, K., “Neocognitron, a self-organizing neural network model
for a mechanism of pattern recognition unaﬀected by shift of position,”
Biological Cybernetics 36:1 (1980), pp. 193–202.
3. Hochreiter, S. and J. Schmidhuber, “Long Short-Term Memory,” Neural
Computation 9:8 (1997), pp. 1735-1780.
4. J. J. Hopﬁeld, “Neural networks and physical systems with emergent col-
lective computational abilities,” Proceedings of the National Academy of
Sciences 79:8 (1982), pp. 2554-2558.
5. http://www.image-net.org.
6. Krizhevsky, A, I. Sutskever, and G.E. Hinton, “Image classiﬁcation with
deep convolutional neural networks,” Advances in Neural Information
Processing Systems, pp. 1097–1105, 2012.
