1 Medical imaging are often obtained in 3D volumes, not only limited to 2D images. So experiments should include the 3D volume data as well for the general community, rather than all on 2D images. And the lesion detection is another important task for the medical community, which has not been studied in this work.
2 More analysis and comments are recommended on the performance trending of increasing the number of parameters for ViT (DeiT) in the Figure 3. I disagree with authors' viewpoint that "Both CNNs and ViTs seem to benefit similarly from increased model capacity". In the Figure 3, the DeiT-B models does not outperform DeiT-T in APTOS2019, and it does not outperform DeiT-S on APTOS2019, ISIC2019 and CheXpert (0.1% won't be significant). However, CNNs can give more almost consistent model improvements as the capacity goes up except on the ISIC2019.
3 On the segmentation mask involved with cancer on CSAW-S, the segmentation results of DEEPLAB3-DEIT-S cannot be concluded as better than DEEPLAB3-RESNET50. The implication that ViTs outperform CNNs in this segmentation task cannot be validly drawn from an 0.2% difference with larger variance.
Questions: 1 For the grid search of learning rate, is it done on the validation set?
Minor problems: 1 The n number for Camelyon dataset in Table 1 is not consistent with the descriptions in the text in Page 4.
2 More analysis and comments are recommended on the performance trending of increasing the number of parameters for ViT (DeiT) in the Figure 3. I disagree with authors' viewpoint that "Both CNNs and ViTs seem to benefit similarly from increased model capacity". In the Figure 3, the DeiT-B models does not outperform DeiT-T in APTOS2019, and it does not outperform DeiT-S on APTOS2019, ISIC2019 and CheXpert (0.1% won't be significant). However, CNNs can give more almost consistent model improvements as the capacity goes up except on the ISIC2019.