To be consistent with the Resnet50, we replace the two FC layers with a global average pooling layer which calculates the average activation of each feature map for the output of the last VGG block. For example, if the output of the last VGG block has a size of $7 \times 7 \times 512$ (height $\times$ width $\times$ channel), after the global average pooling layer the output becomes $512$. This output is then connected to the classification output with a FC layer.

A straightforward approach to construct a whole image classifier from a patch classifier involves flattening the heatmap and connect it to the image’s classification output using FC layers. To increase the model’s translational invariance to the patch classifier’s output, a max pooling layer can be used after the heatmap. Further, a shortcut can be made between the heatmap and the output to make the training easier. The heatmap is directly from the patch classifier’s output which uses the softmax activation: $$f(\bm{z})_{j} = \frac{e^{z_{j}}}{\Sigma_{i=1}^{c}e^{z_{i}}}\textrm{ for }j = 1, ..., c$$ However, the softmax activation diminishes gradients for large inputs, which may impede gradient flow when it is used in an intermediate layer. Therefore, the rectified linear units (ReLU) can 