the selection of the optimized number of features for classification and the training of the final classifier are all performed on the training data set, while the testing of this trained classifier is performed on a completely separate test set without calibrating the classifier on the test data. See Figure \[fig:cv\] and Algorithm \[algo:cv\] for details. In other words, in contrast to previous studies, we strictly distinguish between training and test data.

To prevent biases associated with a specific secondary data source, we tested the algorithms on different types of secondary data sources. (See the Materials and Methods section for detailed descriptions of all these data sources.) We also used two different classifier types, the nearest mean classifier (NMC) and logistic regression (LOG) to evaluate the influence of the classifier on prediction performance. We chose these classifiers since Popovici [*et al.*]{}[@Popovici2010] confirmed earlier findings that these classifiers performed best on various breast cancer related classification tasks. Similarly, different feature extraction strategies were employed. While the included set of feature extraction approaches is by no means exhaustive, we employed approaches that were shown to perform well on gene expression based diagnostic problems [@Wessels2005]. All evaluations were performed on a curated collection 