be used instead: $$f(\bm{z})_{j} = max(0, z_{j})\textrm{ for }j = 1, ..., c$$ In the following, when we refer to the heatmap in a whole image classifier, the activation is always assumed to be ReLU unless otherwise stated.

We further propose to use convolutional layers as top layers, which preserve spatial information. Two blocks of convolutional layers (VGG or Resnet) can be added on top of the patch classifier layers, followed by a global average pooling layer and then the image’s classification output (Fig. \[fig:patch2image\]). Therefore, this design creates an “all convolutional” network for whole image classification. As Fig. \[fig:patch2image\] shows, the heatmap creates a bottleneck-like structure between the patch classifier layers and the top layers, which may cause information loss in the whole image classification. We remove the heatmap entirely from the whole image classifier to allow the top layers to fully utilize the features extracted from the patch classifier and compare the two choices in the following.

Results
=======

Developing patch and whole image classifiers on DDSM
----------------------------------------------------

### Setup and processing of the dataset

The DDSM [@heath_digital_2001] contains digitized film mammograms in a lossless-JPEG format that is obsolete. We used a later version of the database called 