538
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
13.4.2
Convolution and Cross-Correlation
This subsection is a short detour to explain why Convolutional Neural Networks
are so named. It is not a pre-requisite for any of the other material in this
chapter.
The convolutional layer is named because of the resemblance it bears to the
convolution operation from functional analysis, which is often used in signal
processing and probability theory. Given functions f and g, usually deﬁned
over the time domain, their convolution (f ∗g)(t) is deﬁned as:
(f ∗g)(t) =
Z ∞
−∞
f(τ)g(t −τ)dτ =
Z ∞
−∞
f(t −τ)g(τ)dτ
Here we are interested in the discrete version of convolution, where f and g are
deﬁned over the integers:
(f ∗g)(i) =
∞
X
k=−∞
f(k)g(i −k) =
∞
X
k=−∞
f(i −k)g(k)
Often convolution is viewed as using function g to transform function f. In
this context, the function g is sometimes called the kernel. When the kernel is
ﬁnite, so g(k) is only deﬁned for k = 0, 1, . . ., m−1, the deﬁnition simpliﬁes to:
(f ∗g)(i) =
m−1
X
k=0
f(i −k)g(k)
We can extend the deﬁnition to 2-dimensional functions:
(f ∗g)(i, j) =
m−1
X
k=0
m−1
X
l=0
f(i −k, j −l)g(k, l)
Let us deﬁne h to be the kernel obtained by ﬂipping g, i.e., h(i, j) = g(−i, −j)
for i, j ∈{0, . . . , m −1}. It can veriﬁed that the convolution f ∗h of f with the
ﬂipped kernel h is given by:
(f ∗h)(i, j) =
m−1
X
k=0
m−1
X
l=0
f(i + k, j + l)g(k, l)
(13.4)
Note the similarity of Equation 13.4 to Equation 13.1, ignoring the bias term b.
The operation of the convolutional layer can be thought of as the convolution of
the input with a ﬂipped kernel. This similarity is the reason why convolutional
layers are so named, and ﬁlters are sometimes called kernels.
The cross-correlation f ⋆g is deﬁned by (f ⋆g)(x, y) = (f ∗h)(x, y) where
h is the ﬂipped version of g. Thus the operation of the convolutional layer can
also be seen as the cross-correlation of the input with the ﬁlter.
