Universal Pooling -- A New Pooling Method for Convolutional Neural Networks
Junhyuk Hyun Hongje Seong Euntai Kim
cs.CV







Universal Pooling - A New Pooling Method for Convolutional Neural Networks
    Junhyuk Hyun     Hongje Seong     Euntai Kim

 Yonsei University, Seoul, Korea

{jhhyun, hjseong, etkim}@yonsei.ac.kr






    December 30, 2023
===============================================================================================================================

empty



Pooling is one of the main elements in convolutional neural networks. The pooling reduces the size of the feature map, enabling training and testing with a limited amount of computation. This paper proposes a new pooling method named universal pooling. Unlike the existing pooling methods such as average pooling, max pooling, and stride pooling with fixed pooling function, universal pooling generates any pooling function, depending on a given problem and dataset. Universal pooling was inspired by attention methods and can be considered as a channel-wise form of local spatial attention. Universal pooling is trained jointly with the main network and it is shown that it includes the existing pooling methods. Finally, when applied to two benchmark problems, the proposed method outperformed the existing pooling methods and performed with the expected diversity, adapting to the given problem. 





§ INTRODUCTION


Since gaining popularity in the ImageNet Large Scale Visual Recognition Competition (ILSVRC) <cit.> of 2012, convolutional neural networks (CNNs) have received much attention from various research fields. A CNN consists of several elements, including convolutional layers, pooling layers, and a fully connected layer. Since CNN first became famous in ILSVRC 2012, pooling has been included in all popular networks: AlexNet <cit.>, GoogleNet <cit.>, Visual Geometry Group (VGG) <cit.>, Residual Neural Network (ResNet) <cit.>, and DenseNet <cit.>.

Pooling plays several important roles in CNNs. First, it reduces the size of the feature map, enabling training and testing with a limited amount of computation. It also reduces the computation of the network. Second, pooling increases the size of receptive field size in each neuron, enabling each neuron to "see" a larger portion of the input image. This functionality improves the recognition performance. Third, pooling reduces the negative effects of noise and slight distortion. Pooling admits only the representative values and removes the small noises. In essence, pooling simultaneously improves the recognition performance and reduces the computational demands.

The deep learning performance depends on the choice of the pooling method. The most popular pooling methods are average pooling, max pooling, and stride pooling. Overall, these simple methods are efficient because they easily compute the representative values. However, they do not consider the various input patterns and there is still a room to exploit for improving the pooling. Recently, several new pooling approaches have been reported, including a linear combination of simple existing poolings <cit.>, and an adaptive pooling method that magnifies the spatial changes and preserves the important structural details <cit.>. Most of the new pooling methods assume fixed weights for feature training, which do not account for the characteristics of individual images. Therefore, the fixed-weight constraint on the pooling might degrade the performance of the CNN.

 On the other hands, researches on attention module <cit.> have been actively conducted to improve the performance of the network. The attention concept weights a major feature in the feature map. Several studies have shown performance improvements after weighting appropriate parts of the feature maps. Attenuation modules  have been investigated in the spatial domain, time domain, and channel direction.
 
This paper proposes a new pooling method named universal pooling. The proposed pooling is the most general pooling and includes the previous simple poolings such as max pooling, average pooling and stride pooling as special cases. The basic idea of universal pooling is to interpret pooling as attention and extend it to the general channel-wise local spatial attention. That is, the universal pooling selects pooling weights for each channel and they are trained together with other feature extraction parts. Because it embraces the existing simple poolings, universal pooling is expected to outperform the independent simple poolings.
The remainder of this paper is organized as follows. Section <ref> reviews some related works, and Section <ref> introduces the preliminary fundamentals of pooling. Section <ref> proposes the universal pooling concept and discusses its relationship with the simple pooling approaches. Section <ref> applies the proposed pooling to two benchmark datasets and presents the experimental results. Conclusions are drawn in Section <ref>.




§ RELATED WORK


The commonest pooling methods are max and average pooling, but stride pooling in which the feature map is reduced using the stride in the convolution layer has also gained traction in recent years.

Max pooling divides the feature map into blocks and collects the maximum feature value in each block into a smaller output matrix. Max pooling is commonly used between convolution layers and is employed in AlexNet <cit.> and VGG <cit.>. Average pooling operates similarly to max pooling, but outputs the average of each block in the feature map. Global average pooling (GAP), which applies average pooling over the entire feature map, is commonly used in convolutional networks such as ResNet <cit.> and DenseNet <cit.>. In DenseNet, average pooling is applied between the convolution layers. Meanwhile, stride pooling is equivalent to importing values from a fixed position after convoluting across the entire area. This approach is adopted in ResNet.

All of these pooling methods are efficient but simple, and it seems that there is a room to improve the performance. S3pool <cit.> and stochastic pooling <cit.> adopt a probability-based pooling approach. L_p  pooling of various coefficients' norms was proposed in <cit.> and <cit.>, and a fractional version of max pooling was proposed in <cit.>. The spectral space was down-sampled through a filter in <cit.> and <cit.>. In <cit.>, the existing simple pooling methods were combined to improve the pooling performance. Detail-preserving pooling, which preserves the feature details by applying existing down-sampling techniques in the image processing area and by learning the parameters of the function over the network, was proposed in <cit.>.




§ PRELIMINARY FUNDAMENTALS


The pooling operation, which down-samples the feature map, can be considered as a multiple input function

    ρ : ^S^2→

which returns only a scalar value. S^2 is the area of the pooling block ( S × S ). For example, let us consider a pooling of size 2 with a stride 2, as shown in Figure <ref>.








In this case, the pooling can be represented by a multiple input function 

    o_0,0 = ρ (f_0,0,f_0,1,f_1,0,f_1,1) .

As mentioned above, there are three popular pooling methods: max pooling, average pooling, and stride pooling. In the following, these methods will be explained on a pooling block of size S=2 and stride 2.




 §.§ Max pooling


Max pooling computes the output using the maximum function. The output from the max pooling is defined as 

    o_0,0 = max (f_0,0,f_0,1,f_1,0,f_1,1)

and its example is given in Figure <ref>.








 §.§ Average pooling


Average pooling returns the average of each pooling block in the feature map. The averaging function is given by

    o_0,0 = 1/4(f_0,0 + f_0,1 + f_1,0 + f_1,1).


In average pooling, all inputs of the corresponding block in the feature map equally contribute to the output. A simple example of average pooling is given in Figure <ref>.








 §.§ Stride pooling (stride convolution)


Stride convolution reduces the size of the feature map by applying the stride in the convolution layer, without invoking an extra function. The convolution layer with a stride performs the same function as the convolution layer without a stride, followed by fixed-position pooling. The stride pooling is represented by 

    o_0,0 = f_0,0 ,

and is demonstrated by a simple example in Figure <ref>.









§ UNIVERSAL POOLING


This section proposes our new pooling method named universal pooling, which automatically determines the pooling mechanism by training the pooling weights. Universal pooling can be considered as a channel-wise local spatial attention module. Here we show that the proposed method encompasses max pooling, average pooling and stride pooling. For simplicity, we assume a pooling size of 2 and a stride of 2, but the concept is easily extended to more general cases.



 §.§ Motivation


The popular standard pooling methods, demonstrated by simple examples in Section III, are represented by linear combinations of the weighted features within a pooling block in the feature map. More specifically, the simple poolings are expressed as 


    o_0,0    = universal_pooling( f_0,0,f_0,1,f_1,0,f_1,1) 
        = π _0,0f_0,0 + π _0,1f_0,1 + π _1,0f_1,0 + π _1,1f_1,1
       Δ = π⊗f

where π denote the pooling weights, f is the input feature map, and the operator ⊗ performs linear combinations within the pooling block. Equation (<ref>) interprets the pooling operation as an element-wise multiplication and summation within each pooling-size block in the feature map, as shown in Figure <ref>. In universal pooling, the pooling weight π is not manually selected, but is automatically determined from the input image or feature map. The pooling weight is also trained by back-propagation, by a mechanism similar to an attention map. 








 §.§ Structure


The pooling operation computes a scalar value representing the pooling block in the feature map. In implementations, the universal pooling operation should be divided into two smaller operations. The first sub-operation determines the contribution of each element in each pooling block of the feature map, which is equivalent to determining the pooling weight π. The second sub-operation applies the ⊗ operator to the pooling weights and feature map, and returns the representative value of an element in the next-layer feature map.

The characteristics of the proposed universal pooling are similar to those of the existing pooling methods, and are subjected to the following constraints: (C1) Pooling is conducted in a channel-wise manner and excludes the information from all other channels. The pooling weight π is thus trained separately from channel to channel. (C2) Only the feature map elements in a given pooling block are pooled; the elements outside of the block are excluded. When training the pooling weights, the receptive fields of the weights are thus restricted to their corresponding pooling block. (C3) The scale of the features is unchanged by the pooling. For this purpose, the pooling weights are normalized such that π _0,0 + π _0,1 + π _1,0 + π _1,1 = 1. Our universal pooling is designed to satisfy constraints C1, C2, and C3. As shown in Figure 6, the universal pooling module has two main blocks: Block B1, in which the pooling weights are computed, and Block B2, in which the pooling is performed using the pooling weights.








The pooling-weight block consists of several layers-the convolution, fully connected, batch normalization, and activation layers-as in other CNN modules. Using the softmax( ·) function, the output of this block is then normalized over the corresponding pooling block to obtain π _0,0 + π _0,1 + π _1,0 + π _1,1 = 1. The second block applies the ⊗ operator to the pooling weights and feature map over the pooling block to obtain the pooling result. To simplify the description, we assume identical size and stride number in the pooling method of the proposed approach, but unequal size and stride number are equally valid.  

To clarify the relationship between the input and output of universal pooling, we give the equations of the proposed pooling. Before normalization, the output of the first block (B1) is represented as

    [                                           f^ℓ ,c = ϕ (f^ℓ ,c;W); f^ℓ ,c = ( f_i,j^ℓ ,c) ∈ ^H × W,f^ℓ ,c = ( f_i,j^ℓ ,c) ∈ ^H × W ]

where ϕ ( · ) denotes the CNN module receiving f^ℓ,c, the feature map of channel c in the l-th layer, and returning the temporary feature map f^ℓ ,c = ϕ (f^ℓ ,c;W) of channel c (which is the same size as f^ℓ,c). W is a trainable parameter inside the network ϕ ( · ). To satisfy constraint (C3), the corresponding pooling block is processed by the softmax function, which guarantees that the pooling weights within the pooling block sum to one, as shown in Figure <ref>. This operation is represented by

    π^ℓ ,c = softmax(f^ℓ ,c) 
    π _pS + m,qS + n^ℓ ,c
       = 1/∑_i = 0^S - 1∑_j = 0^S - 1exp (f_pS + i,qS + j^ℓ ,c)exp (f_pS + m,qS + n^ℓ ,c) 
    π^ℓ ,c = ( π _i,j^ℓ ,c) ∈ ^H × W
    
      p = 0,1, ⋯ ,⌊0.7exH/
     .-0.7exS⌋  - 1,q = 0,1, ⋯ ,⌊0.7exW/
     .-0.7exS⌋  - 1

where S denotes the stride number and size of the pooling, softmax( ·) is applied to an S × S pooling block, and  denotes the learnable pooling weight. Here, π^ℓ ,c and f_i,j are the same size. 








Then, the output of the first block (B1) is related to its input as follows:

    π^ℓ ,c = softmax(ϕ (f^ℓ ,c;W)).


In (<ref>), the learnable parameter W inside the module determines the pooling function. To satisfy constraint (C1), ϕ ( · ) is applied only to the corresponding channel. The universal pooling then learns the different pooling functions in the different channels.
In the second block (B2), pooling weights are assigned to the feature map, and pooling is performed through the ⊗ operation. The output from (B2) is represented by 

    o^ℓ ,c = π^ℓ ,c⊗f^ℓ ,c
    
      o_p,q^ℓ ,c = ∑_m = 0^S - 1∑_n = 0^S - 1π _pS + m,qS + n^ℓ ,c· f_pS + m,qS + n^ℓ ,c
    o^ℓ ,c = ( o_p,q^ℓ ,c) ∈ ^⌊0.7exH/
     .-0.7exS⌋×⌊0.7exW/
     .-0.7exS⌋
    
      p = 0,1, ⋯ ,⌊0.7exH/
     .-0.7exS⌋  - 1,q = 0,1, ⋯ ,⌊0.7exW/
     .-0.7exS⌋  - 1

where the output o^ℓ ,c from (B2) defines a new feature map in the (ℓ+1)-th layer. Obviously, the scale of the features is unchanged and constraint (C2) is satisfied because the pooling weights are already normalized by (<ref>). In (<ref>), the pooling weights   can be rewritten as

    o^ℓ ,c = π^ℓ ,c⊗f^ℓ ,c = softmax(ϕ (f^ℓ ,c;W)) ⊗f^ℓ ,c.

Because all functions in this equation are simple multiplications and additions, all the learnable parameters are easily updated by back-propagation.




 §.§ Relationship between Universal Pooling and the Existing Pooling Methods


As mentioned above, the proposed pooling module can be considered as a channel-wise local spatial attention module. This subsection demonstrates that the popular existing pooling methods are special cases of universal pooling.



  §.§.§ Average Pooling


Average pooling averages the elements inside the pooling block of the feature map. The averaging is performed as follows:

    o_p,q^ℓ ,c = 1/S^2∑_m = 0^S - 1∑_n = 0^S - 1f_pS + m,qS + n^ℓ ,c

where the pooling block is an S × S matrix. Using the pooling weights π^ℓ ,c, we can rewrite (<ref>) as

    o_p,q^ℓ ,c = ∑_m = 0^S - 1∑_n = 0^S - 1π _pS + m,qS + n^ℓ ,c· f_pS + m,qS + n^ℓ ,c , 
    π _pS + m,qS + n^ℓ ,c = 1/S^2 .

The proposed universal pooling behaves like average pooling if all pooling weights are equal and sum to unity. The pooling weights are constant for any input image. This implies that before softmax( · ) the function, the output f^ℓ ,c of (B1.1) is the same over the whole pooling block. Thus, when universal pooling performs like average pooling, its form reduces to 

    o_p,q^ℓ ,c = ∑_m = 0^S - 1∑_n = 0^S - 1π _pS + m,qS + n^ℓ ,c· f_pS + m,qS + n^ℓ ,c , 
    π _pS + m,qS + n^ℓ ,c
       = 1/∑_i = 0^S - 1∑_j = 0^S - 1exp (f_pS + i,qS + j^ℓ ,c)exp (f_pS + m,qS + n^ℓ ,c) 
       = 1/S^2 , 
    f_pS + i,qS + j^ℓ ,c = f_pS + x,qS + y^ℓ ,c(x  i,y  j) .

In the simplest example of average pooling by the proposed universal pooling, the learnable parameter W is set to 0. In this case, all outputs are zero and the softmax( · ) function generates equal pooling weights that sum to one.





  §.§.§ Max Pooling


Max pooling outputs the maximum value of the elements inside a given pooling block. The max pooling function can be written as follows.

    o_p,q^ℓ ,c = max_0 ⩽ m,n ⩽ S - 1 f_pS + m,qS + n^ℓ ,c .


In terms of the pooling weights π^ℓ ,c, (<ref>) can be rewritten as

    o_p,q^ℓ ,c = ∑_m = 0^S - 1∑_n = 0^S - 1π _pS + m,qS + n^ℓ ,c· f_pS + m,qS + n^ℓ ,c , 
    π _pS + m,qS + n^ℓ ,c = {[                                             1 m,n = max_0 ⩽ i,j ⩽ S - 1 f_pS + i,qS + j^l,c;                                             0                                     otherwise ] . .


The pooling weight is set to one at the site containing the maximum value in the block, and zero at all other sites. If the output f^ℓ ,c of (B1.1) equals the input feature map f^ℓ ,c before invoking the softmax( · ) function, the universal pooling becomes

    o_p,q^ℓ ,c = ∑_m = 0^S - 1∑_n = 0^S - 1π _pS + m,qS + n^ℓ ,c· f_pS + m,qS + n^ℓ ,c , 
    π _pS + m,qS + n^ℓ ,c = softmax(f^ℓ ,c) = softmax(f^ℓ ,c) 
             ≈{[                                             1 m,n = max_0 ⩽ i,j ⩽ S - 1 f_pS + i,qS + j^ℓ,c;                                             0                                     otherwise ] ..


In this case, the proposed universal pooling becomes very similar to max pooling. In summary, the condition under which the proposed method behaves like max pooling is

    f^ℓ ,c = f^ℓ ,c = ϕ (f^ℓ ,c) .

That is, universal pooling reduces to max pooling when (B1.1) is trained as an identity mapping.





  §.§.§ Stride Pooling


Stride pooling obtains an output from a specific location inside an area. When the specific location is the top-left site of the pooling block, the stride pooling is described by

    o_p,q^ℓ ,c = f_pS,qS^ℓ ,c.


In terms of the pooling weights, (<ref>) is rewritten as

    o_p,q^ℓ ,c = ∑_m = 0^S - 1∑_n = 0^S - 1π _pS + m,qS + n^ℓ ,c· f_pS + m,qS + n^ℓ ,c , 
    π _pS + m,qS + n^ℓ ,c = {[         1 m,n = 0,0;         0 otherwise ] . .


The proposed universal pooling behaves like stride pooling when the first block (B1) outputs unity at the top-left location of the pooling block, and zero at the remaining positions. 

Above, we showed that the proposed universal pooling includes the popular existing pooling methods as special cases. This implies that the performance of the existing pooling methods can be improved by replacing them with universal pooling. In particular, by appropriately training W in universal pooling, we can expect to improve the pooling to an unprecedented level, with concomitant improvement in CNN performance.





§ EXPERIMENTATION


In this section, the proposed method is experimentally tested on two benchmark classification datasets: CIFAR10 <cit.> and Places2 <cit.>. The VGG19 and ResNet18 datasets are employed as backbone networks.



 §.§ Implementation


Pooling can be divided into local pooling and global pooling. In local pooling, the pooling block is smaller than the feature map, whereas in global pooling, it covers the entire feature map. In the present experiment, local pooling was implemented by one or two fully connected layers, as shown in Figure <ref>. Global pooling also can be implemented either by fully connected layers (Figure <ref>), or by several convolution layers (Figure <ref>).







 §.§ CIFAR10


The Canadian Institute for Advanced Research 10 (CIFAR10) dataset is a classification dataset containing 60,000 images <cit.>, each of size 32 × 32 on the RGB channels. The dataset contains ten object classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Typical images contained in the dataset are shown in Figure <ref>. The CNNs were trained by the stochastic gradient descent (SGD) method. The initial learning rate, momentum, and weight decay in the SGD were set to 0.1, 0.9, and 0.0001, respectively. The whole training procedure was implemented over 450 epochs, reducing the learning rate by one tenth at 150-epoch intervals.







The backbone networks were two pre-trained models: VGG19 and ResNet18. The original VGG19 network consists of 16 convolution layers, 3 fully connected layers, and 5 max pooling layers. Instead of global pooling, VGG vectorizes the last feature map and passes it to the fully connected layer. The existing VGG structure, which is designed for large-sized images, was here modified for processing small images by employing one fully connected layer. As the last feature map is already vectorized by the final pooling process, it requires no further vectorization, so the final pooling can be considered as a global pooling. This structure of the network is given in Table <ref>.

The original VGG network (here termed V1) performs five max poolings. When the input image is a 32 × 32 CIFAR 10 image, the last-pooled map and the input feature map are the same size. Thus, the final pooling is considered as a global pooling while the preceding poolings are local poolings. As competing methods for comparison, we also employed five variants of VGG19. In variant V2, all five max poolings were replaced with their average pooling procedures. In variant V3, the local max pooling was replaced with local stride pooling in the first four pooling layers, and GAP was applied in the last pooling layer. The structure of V3 matched that of ResNet. Variants V4-V6 were constructed by various combinations of max and average pooling <cit.>, as shown in Table <ref>. In this table, "Mixed" refers to a linear combination of max and average pooling with real-numbered weights. Meanwhile, "Gated-channel" and "Gated-layer" refer to gated max-average pooling in which the gating mask is a linear combination of the weighted channels and weighted layers, respectively. Four variants of the proposed method, labeled P1-P5, were also constructed for a comparison evaluation. In variants P1-P3, both local and global poolings were implemented by fully connected layers, while in variants P4 and P5, the first four local poolings were implemented by fully connected layers, and the final (global) pooling was implemented by a convolution layer (see Table <ref>). Interestingly, the average pooling V2 (rather than the original VGG V1) delivered the best performance among the standard pooling methods. The combined max and average pooling methods (V4-V6) outperformed the standard pooling methods (V1-V3). Meanwhile, among variants P1-P5 of the proposed universal pooling method, constructed as competitors against the standard methods, P4 and P3 delivered the best and second-best performances, respectively. Overall, increasing the number of layers improved the network performance. However, the performance was degraded in the deepest network (P5), possibly because the network overfitted the training data.

Second, let us consider ResNet. The original ResNet18 consists of 17 convolution layers, 1 fully connected layer, 5 local pooling layers, and 1 global pooling layer. In the original ResNet, the local and global poolings are performed by striding and averaging (V3), respectively. The existing ResNet structure, which is suitable for large-sized images, was here modified for small-image processing by employing three (rather than five) local pooling layers. The network structure is given in Table <ref>. As in VGG, two standard pooling methods (V1 with max pooling and V2 with average pooling) and three combinations of max and average poolings (V4-V6) were constructed as competing methods. All experiments were conducted ten times, and the performances are summarized in Table <ref> and Figure <ref>. As confirmed in the table, average pooling delivered the best performance among the standard pooling methods (V1-V3, where V3 is the original ResNet18). The performance of average pooling (V2) was significantly higher than that of max pooling (V1). Interestingly, the average pooling outperformed even the combination methods (V4-V6). This suggests that training all poolings to behave like average pooling is a difficult task. However, the proposed universal pooling outperformed the average pooling, as shown in Table <ref>. The overall best performance was achieved by a network with one fully connected layer for local pooling, and two fully connected layers for global pooling. As demonstrated in this experiment, the optimal performance is dictated by the structure of the entire network, and the proposed universal pooling outperformed all standard pooling methods and their combinations proposed in <cit.>.



























 §.§ Places2









The Places2 dataset is a classification dataset containing 365 place classes. The dataset includes 1,803,460 training images and 50 validation images per class. Example images from the Places2 dataset are given in Figure 11. The network was trained by the SGD optimizer and the entire learning period was 120 epochs with a batch size of 256. The initial learning rate, momentum and weight decay of the SGD were set to 0.1, 0.9, and 0.0001, respectively. The learning rate was reduced by one tenth at 30-epoch intervals. The Places2 experiment was conducted in the pre-trained ResNet18 model, and the results are given in Table 2. Clearly, the classification accuracy was improved by the universal pooling.






The universal pooling function was qualitatively evaluated by varying the global pooling weights in the universal pooling method. The results of training with the different weights are visualized in Figures <ref>-<ref>. As implied in the three cases below, the universal pooling can train various pooling functions, and appropriately selects the pooling function to enhance the classification performance.







  §.§.§ Training by Average Pooling


In almost one-third of the channels (129 out of 512 channels), the pooling weights are the same over the whole feature map, and the pooling reduces to average pooling. The first of these 129 channels is depicted in Figure <ref>. The trained pooling weights exhibited the same values over the input feature map.









  §.§.§ Training by Flexible Pooling


In almost a fifth of the channels (95 out of 512 channels), the pooling weights depended on the input (see Figure <ref>). This pooling, called flexible pooling, includes max pooling as a special case. Figure <ref> shows how the pooling weights change with the feature maps. The trained universal pooling operates similarly, but not identically, to max pooling. In certain data, such as data #1 and #4 in the figure, the trained pooling exhibited similar characteristics to fixed pooling (discussed next).









  §.§.§ Training by Fixed Pooling


In more than one-half of the 512 channels (288 out of 512 channels), the trained pooling weights were almost fixed, and were nonresponsive to changes in the input feature map. This pooling, called fixed pooling, includes both average and stride pooling. However, average pooling is classified separately from other fixed poolings because the conditions of average and stride pooling differ in the universal pooling. The results of two channels with fixed training weights are visualized in Figures <ref> and <ref>. In both cases, the pooling weights depended only on the pooling position in the image, and were independent of the input feature map.













§ CONCLUSION


This paper proposed a new pooling method named universal pooling, which dynamically learns the pooling weights from the given dataset, and which includes the existing popular pooling methods as special cases. Thus, replacing the existing pooling with universal pooling will likely improve the network performance. The proposed pooling was applied to two benchmark datasets and two pretrained models and was validated in comparison experiments.








































































































































































































































































































































































































ieee_fullname



