---:
1: This paper ensembles some existing compression/NAS approaches to improve the performance of BNNs, which is not significant enough.
The dynamic routing strategy (conditional on input) has been widely explored. For example, the proposed dynamic formulation in this paper has been used in several studies [2, 3].
Varying width and depth has been extensively explored in the quantization literature, especially in AutoML based approaches [Shen et al. 2019, Bulat et al. 2020], to design high capacity quantized networks.
The effectiveness of the group convolution in BNNs was initially studied in [1]. Later works also incorporate the group convolution into the search space in NAS+BNNs methods [e.g., Bulat et al. 2020a] to reduce the complexity.
2: In each layer, the paper introduces a full-precision fully-connected layer to decide which expert to use. However, for deeper networks, such as ResNet-101, it will include ~100 full-precision layers, which can be very expensive especially in BNNs. As a result, it deteriorates the benefits and practicability of the dynamic routing mechanism.
3: The actual speedup, memory usage and energy consumption on edge devices (e.g., CPU/GPU/FPGA) or IoT devices must be reported. Even though the full-precision operations only account for a small amount of computations in statistics, it can have a big influence on the efficiency on platforms like FPGA.
4: This paper proposes to learn the binary gates via gradient-based optimization while exploring the network structure via EfficientNet manner. Then the problem comes. This paper can formulate the <width, depth, groups and layer arrangement> as configuration vectors and optimize them using policy gradients and so on, with the binary gates learning unified in a gradient-based framework. So what is the advantage of the "semi-automated" method of EfficientNet over the gradient-based optimization? In addition, how about learning a policy agent via RL to predict the gates? I encourage the authors can add comparsions and discussions with these alternatives.
5: More experiments on deeper networks (e.g., ResNet-50) and other network structures (e.g., MobileNet) are needed to further strengthen the paper. References:
[1] MoBiNet: A Mobile Binary Network for Image Classification, in WACV 2020.
[2] Dynamic Channel Pruning: Feature Boosting and Suppression, in ICLR2019.
[3] Learning Dynamic Routing for Semantic Segmentation, in CVPR2020.
5: More experiments on deeper networks (e.g., ResNet-50) and other network structures (e.g., MobileNet) are needed to further strengthen the paper. References: [1] MoBiNet: A Mobile Binary Network for Image Classification, in WACV 2020. [2] Dynamic Channel Pruning: Feature Boosting and Suppression, in ICLR2019. [3] Learning Dynamic Routing for Semantic Segmentation, in CVPR2020.