1. Performance gains on downstream tasks of detection and instance segmentation are much lower -- how would the authors propose to improve these? 2. If the primary goal is to improve SSL performance on small models, I would have liked to see more analysis on how different design choices of setting up contrastive learning affect model performance and if these could aid performance improvement, in addition to knowledge distillation.
Questions and suggestions: 1. Adding fully-supervised baselines for small models in table 1 will be useful in understanding the gap between full supervision and SSL for these models. 2. In figure 3, does 100% (green line) represent the student network trained with 100% of labeled imagenet supervised data? It is hard to interpret what these numbers represent. 3. Minor point: Some citations, which should not be in parentheses, are in parentheses (e.g., Romero et al. page 8). Please fix this in the revision.
1. Adding fully-supervised baselines for small models in table 1 will be useful in understanding the gap between full supervision and SSL for these models.