 Global feature pooling is a modern variant of feature pooling providing better interpretatability and regularization. Although alternative pooling methods exist (eg. max, lp norm, stochastic), the averaging operation is still the dominating global pooling scheme in popular models. As fine-grained recognition requires learning subtle, discriminative features, we consider the question: is average pooling the optimal strategy? We first ask: ``is there a difference between features learned by global average and max pooling?'' Visualization and quantitative analysis show that max pooling encourages learning features of different spatial scales. We then ask ``is there a single global feature pooling variant that's most suitable for fine-grained recognition?'' A thorough evaluation of nine representative pooling algorithms finds that: max pooling outperforms average pooling consistently across models, datasets, and image resolutions; it does so by reducing the generalization gap; and generalized pooling's performance increases almost monotonically as it changes from average to max. We finally ask: ``what's the best way to combine two heterogeneous pooling schemes?'' Common strategies struggle because of potential gradient conflict but the ``freeze-and-train'' trick works best. We also find that post-global batch normalization helps with faster convergence and improves model performance consistently. Deeply rooted in the works of complex cells in the visual cortex (Hubel & Wiesel, 1962) and locally orderless images (Koenderink & Van Doorn, 1999) , feature pooling has been an indispensable component of visual recognition in both traditional bag-of-words (BOW) frameworks (Csurka et al., 2004; Lazebnik et al., 2006) using hand-crafted features (e.g. SIFT (Lowe, 2004) , HOG (Dalal & Triggs, 2005) ), and modern convolutional neural networks (CNNs) (LeCun et al., 1998; Krizhevsky et al., 2012) . A recent variant of this technique, called "global feature pooling" (Lin et al., 2013) , distinguishes itself by defining its pooling kernel the same size as input feature map. The pooling output is a scalar value indicating the existence of certain features (or patterns). Benefits of global pooling are two-fold: allowing better interpretation of the underlying filters as feature detectors, and serving as a strong network regularizer to reduce overfitting. Global pooling is thus used in most, if not all, recent state-of-the-art deep models He et al., 2016; Szegedy et al., 2017; Huang et al., 2017; Hu et al., 2018) in visual recognition. Unless otherwise noted, all the pooling methods discussed in this paper are used as the global pooling layer. Feature pooling is also of special interests to Fine-grained Visual Categorization (FGVC) (Rosch et al., 1976; Nilsback & Zisserman, 2010; Farrell et al., 2011) , where objects are classified into subcategories rather than basic categories. Carefully designed pooling schemes can learn helpful discriminative features and yield better performance without requiring more conv-layers in the network. Wang et al. (2018) provided a good example that combines three pooling operations: average, max and cross-channel pooling to learn to capture class-specific discriminative patches. Another major research direction is higher-order pooling: Lin et al. (2015) proposed to apply bilinear pooling (also know as second-order pooling) to capture pairwise correlations between the feature channels and model part-feature interactions; Gao et al. (2016) proposed compact bilinear pooling that applies random maclaurin projection and tensor sketch projection to approximate the outer product operation, greatly reducing parameters without sacrificing accuracy; Works along this line of research include low-rank bilinear pooling (Kim et al., 2016) , grassmann pooling (Wei et al., 2018) , kernel pooling (Cui et al., 2017) , and Alpha-pooling Simon et al. (2017) , etc. Although higher-order pooling methods output a vector rather than a scalar, they're still relevant as they reside in the same location as the global pooling layer. The most common pooling operations are average, max and striding. Striding always takes the activation at a fixed location, thus is never applied as global pooling. An abundant set of pooling flavors exist for both traditional and modern feature extractors. Stochastic pooling randomly chooses an activation according to a multinomial distribution decided by activation strength in the pooling region. Fractional max pooling (Graham, 2014) can be adapted to fractional sized pooling regions. Spatial pyramid pooling (He et al., 2015) outputs the combination of multiple max pooling with different sized pooling kernels. S3Pool (Zhai et al., 2017) , or stochastic spatial sampling Pooling, randomly picks a sub-region to apply max pooling to. Detail-preserving pooling (Saeedan et al., 2018) computes the output as the linear combination of input feature pixels whose weight is proportional to differences of the input intensities. Translation invariant pooling (Zhang, 2019) borrowed the idea of anti-alias by low-pass filtering from signal processing. A major pooling family, generalized pooling, aims to find a smooth transition between average and max pooling: k-max pooling (Kalchbrenner et al., 2014) outputs the average of the k highest activations of the feature map; l p norm pooling generalizes pooling to the p-norm of the input feature map (Boureau et al., 2010) ; soft pooling (Boureau et al., 2010) , or softmax pooling, outputs the sum of feature map weighted by softmax output; mixed pooling (Lee et al., 2016 ) computes a weighted sum of the max and average pooling; gated pooling (Lee et al., 2016 ) is similar to mixed pooling but the weight is learned instead. To the best of our knowledge, these pooling operations remain largely unexplored in the global pooling scenario. An interesting observation is that all highly-ranked classification models "happen" to choose the same averaging operation in their global pooling layer. Is this an arbitrary choice or actually the optimal strategy? How does average pooling compare against the other pooling schemes (e.g. max) in general image classification and also fine-grained visual recognition? Research (Boureau et al., 2010; Murray & Perronnin, 2014; Scherer et al., 2010; Hu et al., 2018; has shown that the selection of feature pooling affects the algorithm's performance, whether using hand-crafted features or deep features. Specially, Murray & Perronnin (2014) showed max pooling has superior performance in the traditional recognition framework because of its better pattern discriminability, and the same conclusion was made by an experimental evaluation of Scherer et al. (2010) using LeNet-5 (LeCun et al., 1998) on the Caltech 101 (Fei-Fei et al., 2007) and NORB (Jarrett et al., 2009 ) dataset. Boureau et al. (2010) provided a theoretical proof that "max pooling is particularly well suited to the separation of features that are very sparse." However, in squeeze and excitation networks (Hu et al., 2018) , global max pooling is reported to achieve 0.29% higher top-1 error and 0.05% higher top-5 error than average pooling. Similar results were reported by using VGG (Simonyan & Zisserman, 2015) and GoogleNet (Szegedy et al., 2016) . It seems max pooling is less preferred as a global pooling scheme than before. These intriguing contrasts call for a careful examination of both pooling schemes. Our investigation begins with the two most common global average and max pooling. Specially, we're interested to know what features have both pooling methods helped learned. Feature map visualization indicates that max pooling produces sparser final conv-layer feature maps. This is further verified quantitatively by two perceptually-consistent sparsity metrics: discrete entropy and thresholded l 0 norm. Visualization of final conv-layer filters further helps us conclude empirically that: global average pooling encourages object-level features while global max pooling focuses more on part-level features. As class-specific features often reside in localized object parts in finegrained datasets, it's equal to say global max pooling find more discriminative features, well aligned with previous findings (Murray & Perronnin, 2014; . The second question to answer is that "is there a single optimal pooling operation on different finegrained datasets across different models?" We evaluate nine representative pooling schemes, which are: average, max, k-max, l p norm, soft, logavgexp, mixed, gated, and stochastic pooling, in the experiment section. We make several observations: max pooling outperforms average pooling across datasets, input resolutions, and models. The reason behind this phenomenon, besides their feature differences, is relevant to the fact that max pooling generalizes better. Most pooling methods we evaluated performs better than average pooling, with k-max (k = 2) and mixed pooling (α = 0.5) being the top two. Our k-max pooling model, when trained properly, beats all previous higher-order pooling methods using the same backbone. The fact that no single pooling works best for all models leads to the need for learnable pooling, where the pooling function is not chosen by heuristic, but optimized via gradient descent. However, our finding that model performance decrease and generalization gap increases in an almost monotonic way when generalized pooling changes from max to average casts a shadow upon the learnable generalized pooling. A pooling is better not because it minimizes training loss, but because it better regularizes the model. Throughout our experiment, post-global batch normalization is applied as another key ingredient achieving consistent performance improvement and faster convergence. Finally, we explore the integration of heterogeneous pooling. Since different features can be learned by average or max pooling, our assumption is that learning a model with heterogeneous poolings will lead to better performance, but what's the best way to integrate them? We review and evaluate three common strategies, but found their improvement upon single pooling is limied. Our hypothesis is that different pooling methods interfere and cancel each other out when learned together. We instead propose to apply the "freeze-and-train" trick. The intuition is that the frozen branch won't degrade during training and the gradients will be well separated. The resulting architecture only adds a tiny amount of parameters to a backbone network, but consistently outperforms single pooling models. In this paper, we focus on the global pooling layer in popular classification models as applied to the task of fine-grained recognition. By visualizing the final conv-layer filters and feature maps, we discover that max pooling produces much sparser feature maps and helps the network learn part-level features. Average pooling, on the other hand, encourages object-level features to be learned. We evaluated nine representative global pooling schemes for fine-grained recognition. K-max (k = 2) pooling outperformed all other global pooling schemes and is actually better than all higher-order pooling models. We made several observations from pooling benchmark experiments: (1) max pooling performs better than average pooling across datasets, models, and input resolution; (2) max pooling generalizes better than average pooling; and (3) model performance displays an approximately monotonically increasing characteristic when generalized pooling changes from average to max. Based on these observations, we discussed the potential risk of learning a generalized pooling: namely that minimizing training loss may lead to average pooling and thus be prone to overfitting. We highlight the importance of post-global batch normalization -which is absent from most, if not all, popular state-of-the-art models -in helping to attain faster convergence and in consistently improving model performance. We evaluated several strategies for heterogeneous pooling integration. The freeze-and-train trick performs best among all end-to-end learnable models. For future work, we suggest consideration of models learned from scratch alongside those fine-tuned from pretrained weights. In addition, experiments should be explored on a broader set of data, not just on fine-grained datasets, in order to affirm whether the findings presented here generalize to more general-purpose datasets such as ImageNet and/or MS-COCO.
A benchmark of nine representative global pooling schemes reveals some interesting findings. For fine-grained classification tasks, this paper validated that maxpooling would encourage sparser feature maps than and outperform avgpooling. 