* There are numerous approaches to reduce ConvNet's memory footprint and computational resources at inference time, including but not limited to channel pruning, dynamic computational graph, and model distillation. Why is removing shortcut connection the best way to achieve the same goal? The baselines considered in Table 3 and 4 are rather lacking. For example, how does the proposed method compare to: 1. Pruning method that reduces ResNet-50 channel counts to match the memory footprint and FLOPs of plain-CNN 50. What will be the drop in accuracy? 2. Distill a ResNet-50 model to a smaller ResNet with similar memory footprint as plain-CNN 50. How does this compare to the proposed training scheme? * L107-111 states that "At the early stages of the training process, the gradients from ResNets play a bigger role with a larger weight, and at the later stages, the gradients contributed by ResNets fade out ...". There doesn't seem to be any empirical results or citations to back up this claim. * L133-L143, are there any ablation studies with path 1 - path 3 removed? What is the result of a naive KD method (without those skip connections)? EDIT AFTER REBUTTAL ================== I have read the authors' rebuttal and other reviews. I think the authors have done an excellent job in addressing my concerns. The updated table of results and efficiency metrics are very convincing. Please incorporate this into your final paper! I have thus updated my rating to accept.
2. Distill a ResNet-50 model to a smaller ResNet with similar memory footprint as plain-CNN 50. How does this compare to the proposed training scheme?