.
The method.
Although I like the experimental findings in Section 4. I have a lot of concerns: 1) experimental settings for Figure 1 to Figure 9 are totally missing, which makes them hard to be convincing; 2) how about the observations when using different teacher-student pairs (with either similar or different network architectures? 3) the metrics (Eq. 4 and its reduced version Eq. 7) are not explicitly used in the designs of FT-KD and PESF-KD. This breaks the flow of the method section. Why not to add a related loss in the optimization?
The proposed FT-KD and PESF-KD are heavily based on existing works deep mutual learning (DML) [1] and parameter-efficient transfer learning methods [2-3], leading to limited technical novelty. What's more, they actually need huge training costs instead of more efficient claimed by the authors, in my understanding (see below comments for details).
To me, claiming FT-KD as a new contribution (the authors call it an adaptive knowledge transfer learning method) is misleading to some degree. As FT-KD merely removes the online knowledge transfer path from the student to the teacher, in DML. Besides, in FT-KD, is the teacher model pre-trained beforehand or trained from scratch jointly the student model? I think, it should be pre-trained beforehand as the authors state that it will be fine-tuned during the training of the student model, is it true?
Note that PESF-KD (may also include FT-KD) uses the pre-trained teacher model. That is, PESF-KD is not a real online KD method. It not only needs the pre-trained stage, but also the online joint training stage. As a result, the claimed 'more efficient' advantage is totally misleading.
[1] Deep Mutual Learning, CVPR 2018.
[2] TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning, NeurIPS 2020.
[3] Towards a Unified View of Parameter-Efficient Transfer Learning, ICLR 2022.
The experiments.
The improvement of the proposed methods is mostly marginal, compared to DML, or other competitive methods.
Note that PESF-KD (may also include FT-KD) uses the pre-trained teacher model, and thus the experimental comparisons of training cost in Table 2, Table 3, etc. are wrong. That is, PESF-KD is not a real online KD method. It not only needs the pre-trained stage, but also the online joint training stage. As a result, the claimed 'more efficient' advantage is totally misleading.
In performance comparison, the authors only consider relatively old KD methods, many recently proposed high-performance KD methods are ignored. Some representative methods are SSKD, SRRL, SemCKD, ReviewKD and SimKD. Others.
The writing can be improved significantly: 1) there are so many bolded descriptions; 2) the organization of figures is bad; 3) there are many typos and grammar errors. ----Update----
I keep my original score as my concerns are not well addressed or ignored in the rebuttal.
1) experimental settings for Figure 1 to Figure 9 are totally missing, which makes them hard to be convincing;