1) One of the key components is the matching metric, namely, the Pearson correlation coefficient (PCC). However, the assumption that PCC is a more relaxed constraint compared with KL divergence because of its invariance to scale and shift is not convincing enough. The constraint strength of a loss function is defined via its gradient distribution. For example, KL divergence and MSE loss have the same optimal solution while MSE loss is stricter than KL because of stricter punishment according to its gradient distribution. From this perspective, it is necessary to provide the gradient comparison between KL and PCC. 2) The experiments are not sufficient enough. 2-1) There are limited types of teacher architectures. 2-2) Most compared methods are proposed before 2019 (see Tab. 5). 2-3) The compared methods are not sufficient in Tab. 3 and 4. 2-4) The overall performance comparisons are only conducted on the small-scale dataset (i.e., CIFAR100). Large datasets (e.g., ImageNet) should also be evaluated. 2-5) The performance improvement compared with SOTAs is marginal (see Tab. 5). Some students only have a 0.06% gain compared with CRD. 3) There are some typos and some improper presentations. The texts of the figure are too small, especially the texts in Fig.2. Some typos, such as “on each classes” in the caption of Fig. 3, should be corrected.
The authors have discussed the limitations and societal impacts of their works. The proposed method cannot fully address the binary classification tasks.
1) One of the key components is the matching metric, namely, the Pearson correlation coefficient (PCC). However, the assumption that PCC is a more relaxed constraint compared with KL divergence because of its invariance to scale and shift is not convincing enough. The constraint strength of a loss function is defined via its gradient distribution. For example, KL divergence and MSE loss have the same optimal solution while MSE loss is stricter than KL because of stricter punishment according to its gradient distribution. From this perspective, it is necessary to provide the gradient comparison between KL and PCC.