heatmap to transform the outputs and connect with the final classification output of the image. Adding a convolutional layer on top of the patch classifier’s outputs turns the entire patch classifier into a filter and enlarges its receptive field. For example, if the patch classifier has a receptive field of $224 \times 224$ with a stride$=32$, adding a $3 \times 3$ convolutional layer on top of it increases each side of the receptive field to $224+(3-1) \times 32=288$. Thus, the top layers effectively use the patch classifier to “scan” the whole image, looking for cues of cancerous lesions and extracting higher level features that can finally be used for whole image classification. Using function $g$ to represent the top layers, the whole image classification function can be written as $h(M) = g(f(M)) \in \rm I\!R^{d}$, where $d$ is the number of classes of the whole image. Typically, $d = 2$ represents the two classes we want to predict: benign (or normal) and malignant.

The function $h$ accepts whole images as input and produces labels at the whole image level. Therefore, it is end-to-end trainable, providing two advantages against the two-step approach: First, the entire network can be jointly trained, avoiding 