The experiment part is kind of weak. Only experiments of CIFAR-100 are conducted and only DeiT-small/-base are compared. 1. For comparison with DeiT, can you add DeiT variants with the same number of layers, heads, hidden dimension as CMHSA to do a fair comparison? DeiT with 12 layers performs worse than CMHSA with 6 layers on CIFAR-100 is as expected, thus not a convincing comparison.
2. If possible, can you add CNN models to show if CMHSA would actually make ViT performs better/on-par with CNN models, which should be the ultimate goal of training ViT in low-data regime, otherwise one would pretrain ViT on large scale dataset or just use CNN models.
3. If possible, results on ImageNet can be more convincing of the proposed method
3. If possible, results on ImageNet can be more convincing of the proposed method