I was reading the paper by Kalchbrenner et al. titled A Convolutional Neural Network for Modelling Sentences and I am struggling to understand their definition of convolutional layer.

First, let's take a step back and describe what I'd expect the 1D convolution to look like, just as defined in Yoon Kim (2014).

sentence. A sentence of length n (padded where necessary) is represented as

$x_{1:n} = x_1 \oplus x_2 \oplus \dots ⊕ x_n,$ (1)

where $\oplus$ is the concatenation operator. In general, let $x_{i:i+j}$ refer to the concatenation of words $x_i, x_{i+1}, \dots, x_{i+j}$. A convolution operation involves a filter $w \in \mathbb{R}^{hk}$, which is applied to a window of h words to produce a new feature. For example, a feature ci is generated from a window of words $x_{i:i+h−1}$ by

$c_i = f(w \cdot x_{i:i+h−1} + b)$ (2).

Here $b \in \mathbb{R}$ is a bias term and $f$ is a non-linear function such as the hyperbolic tangent. This filter is applied to each possible window of words in the sentence $\{x_{1:h}, x_{2:h+1}, \dots, x_{n−h+1:n}\}$ to produce a feature map

$c = [c_1, c_2, \dots, c_{n−h+1}]$, (3)

with $c \in \mathbb{R}^{n−h+1}$.

Meaning a single feature detector transforms every window from the input sequence to a single number, resulting in $n-h+1$ activations.

Whereas in Kalchbrenner's paper, the convolution is described as follows:

If we temporarily ignore the pooling layer, we may state how one computes each d-dimensional column a in the matrix a resulting after the convolutional and non-linear layers. Define $M$ to be the matrix of diagonals:

$M = [diag(m:,1), \dots, diag(m:,m)]$ (5)

where $m$ are the weights of the d filters of the wide convolution. Then after the first pair of a convolutional and a non-linear layer, each column $a$ in the matrix a is obtained as follows, for some index $j$:

enter image description here

Here $a$ is a column of first order features. Second order features are similarly obtained by applying Eq. 6 to a sequence of first order features $a_j, \dots, a_{j+m'−1}$ with another weight matrix $M'$. Barring pooling, Eq. 6 represents a core aspect of the feature extraction function and has a rather general form that we return to below. Together with pooling, the feature function induces position invariance and makes the range of higher-order features variable.

As described in this question, the matrix $M$ has dimensionalty of $d$ by $d * m$ and the vector of concatenated $w$'s has dimensionality $d * m$. Thus the multiplication produces a vector of dimensionality d (for a single convolution of a single window!).

Architecture visualization from the paper seems to confirm this understanding:

enter image description here

The two matrices in the second layer represent two feature maps. Each feature map has dimensionality $(s + m - 1) \times d$, and not $(s + m - 1)$ as I would expect.

Authors refer to a "conventional" model where feature maps have only one dimension as Max-TDNN and differentiate it from their own.

As the authors point out, feature detectors in different rows are fully independent from each other until the top layer. Thus they introduce the Folding layer, which merges each pair of rows in the penultimate layer (by summation), reducing their number in half (from $d$ to $d/2$).


Sorry for the prolonged introduction, here are my two main questions:

  1. What is the possible motivation for this definition of convolution (as opposed to Max-TDNN or e.g. Yoon Kim's model)

  2. In the Folding layer, why is it satisfying to only have dependence between pairs of corresponding rows? I don't understand the gain over no dependence at all.


Your Answer


Browse other questions tagged or ask your own question.