<p>The max-over-time pooling is usually applied in NLP (unlike ordinary max-pool, which is common in CNNs for computer vision tasks), so the setup is a little bit different.</p>

<p>The input to the max-over-time pooling is a feature map <code>c = [c(1), ..., c(n-h+1)]</code>, which is computed over a sentence of length <code>n</code> with a filter of size <code>h</code>. The convolution operation is very similar to one with images, but in this case it's applied to 1-dimensional vector of words. This is the formula (3) in the <a href="https://arxiv.org/pdf/1408.5882.pdf" rel="noreferrer">paper</a>.</p>

<p>The max-over-time pooling operation is very simple: <code>max_c = max(c)</code>, i.e., it's a single number that gets a max over the whole feature map. The reason to do this, instead of "down-sampling" the sentence like in a CNN, is that in NLP the sentences naturally have different length in a corpus. This makes the feature maps different for different sentences, but we'd like to reduce the tensor to a fixed size to apply softmax or regression head in the end. As stated in the paper, it allows to capture the most important
feature, one with the highest value for each feature map.</p>

<p>Note that in computer vision, images are usually<sup>1</sup> of the same size, like <code>28x28</code> or <code>32x32</code>, that's why it is unnecessary to downsample the feature maps to <code>1x1</code> immediately.</p>

<p>Sum-pooling-over-time is the same.</p>

<hr>

<p><sup>1</sup> Modern CNN can be trained with images of different size, but this requires the network to be all-convolutional, so it doesn't have any pooling layers. See <a href="https://stackoverflow.com/q/48230031/712995">this question</a> for more details.</p>
Pooling vs Pooling-over-time
<p>I understand conceptually what is happening in a max/sum pool as a CNN layer operation, but I see this term "max pool over time", or "sum pool over time" thrown around (e.g., <a href="https://arxiv.org/pdf/1408.5882.pdf" rel="noreferrer">"Convolutional Neural Networks for Sentence Classification"</a> paper by Yoon Kim). What is the difference?</p>