found that pre-training improved the training speed as well as the accuracy of handwritten digit recognition. Another popular training method is to first train a deep learning model on a large database such as the ImageNet [@russakovsky_imagenet_2015] and then fine-tune the model for another task. Although the specific task may not be related to the initial training dataset, the model’s weight parameters are already initialized for recognizing primitive features, such as edges, corners and textures, which can be readily used for a different task. This often saves training time and improves the model’s performance.

In this study, we propose an “end-to-end” approach in which a model to classify local image patches is pre-trained using a fully annotated dataset with ROI information. The patch classifier’s weight parameters are then used to initialize the weight parameters of the whole image classifier, which can be further fine-tuned using datasets without ROI annotations (see discussion in [@shen_breast_2017]). We used a large public film-based mammography database—with thousands of images—to develop the patch and whole image classifiers, and then transferred the whole image classifiers to a public FFDM database—with hundreds of images. We evaluated various network designs for constructing the patch and whole image classifiers to 