# StableKD: Breaking Inter-block Optimization Entanglement
for Stable Knowledge Distillation

Shiu-hong Kao111These authors contributed equally. Jierun Chen††footnotemark:
S.H. Gary Chan
The Hong Kong University of Science and Technology
<EMAIL_ADDRESS>{jcheneh<EMAIL_ADDRESS>
###### Abstract

Knowledge distillation (KD) has been recognized as an effective tool to
compress and accelerate models. However, current KD approaches generally
suffer from an accuracy drop and/or an excruciatingly long distillation
process. In this paper, we tackle the issue by first providing a new insight
into a phenomenon that we call the _inter-block optimization entanglement_
(IBOE), which makes the conventional end-to-end KD approaches unstable with
noisy gradients. We then propose StableKD, a novel KD framework that breaks
the IBOE and achieves more stable optimization. StableKD distinguishes itself
through two operations: _Decomposition_ and _Recomposition_ , where the former
divides a pair of teacher and student networks into several blocks for
separate distillation, and the latter progressively merges them back, evolving
towards end-to-end distillation. We conduct extensive experiments on CIFAR100,
Imagewoof, and ImageNet datasets with various teacher-student pairs. Compared
to other KD approaches, our simple yet effective StableKD greatly boosts the
model accuracy by 1% – 18%, speeds up the convergence up to $10\times$, and
outperforms them with only $40\%$ of the training data.

##  1 Introduction

While the great potential of deep learning has been showcased by many large-
scale models running on the cloud (Devlin et al., 2018; Holcomb et al., 2018),
a burgeoning direction is to bring the intelligence to the edge, by building
compact and computation-efficient models. Knowledge Distillation (KD), among
various branches of methods, is well-known for its conceptual simplicity and
ease of implementation. KD aims to distill knowledge from a large-scale
teacher model to a smaller and faster student model. A typical vanilla KD
approach provides the informative “soft targets” from the last layer of the
teacher to the student (Hinton et al., 2015). The feature-based KD approaches
further enforce the alignment of intermediate feature maps (Heo et al., 2019a;
Jung et al., 2021). Though effective in downsizing the model and reducing the
computational complexity, they generally lead to deterioration in accuracy,
leaving an accuracy gap between the teacher and the student. Beyer et al.
(2022) narrow down the gap by distilling patiently for a large number of
epochs. However, such an excruciatingly long distillation process is costly in
terms of computational resources and energy. The situation can worsen when the
task becomes more complex, involving a huge training dataset. Collectively,
these factors diminish the appeal of KD for intensive and practical
deployment.

_How can we improve a student’s accuracy and simultaneously accelerate the KD
process?_ We probe into the problem from a closely related optimization
perspective. Given a student network composed of a series of blocks111A block
refers to a layer or a stack of consecutive layers in a neural network.,
current KD approaches mainly train it in an end-to-end manner. Therefore, the
optimization of a block is conditioned on or entangled with other blocks by
noisy gradients. We call this phenomenon _inter-block optimization
entanglement_ (IBOE). We hypothesize that such entanglement can incur
optimization instability especially when the randomly initialized parameters
are far from convergence at the early training phase. The optimization
instability will then lead to accuracy fluctuation and slow down the
convergence (Shalev-Shwartz et al., 2017).

The above observation motivates us to propose a novel KD framework that has a
more stable optimization – StableKD. It consists of two operations,
_Decomposition_ and _Recomposition_. _Decomposition_ breaks the IBOE by
dividing a pair of teacher and student networks into separate blocks. Notably
the input of a student block is identical to the input of its corresponding
teacher block, rather than the output of its preceding student block. This
simple design essentially differentiates StableKD from prior works, as shown
in Figure LABEL:fig:comparison. It enables the student blocks to learn
independently, stably, and rapidly from their teacher. However, it incurs an
architectural discrepancy between training and inference, i.e., the student
blocks are blind to each other during training but have to collaborate for
inference. Such discrepancy, as verified in the paper, can adversely affect
the student’s final accuracy. To overcome this side effect, _Recomposition_ is
imposed to progressively merge the divided blocks back along the training
process. This is reasonable as the student blocks get more mature after
certain epochs, and the IBOE becomes less of a concern.

As such, we keep our StableKD framework simple yet effective in increasing the
optimization stability, and thereby boosting the model accuracy and
accelerating the KD process. We conduct experiments on the widely adopted
CIFAR100 (Krizhevsky, 2009), Imagewoof (Howard, 2019), and ImageNet
(Russakovsky et al., 2015) datasets, using diverse combination of teacher and
student pairs. The results show that StableKD consistently outperforms the
state of the art by 1% – 18% in absolute accuracy, especially when constrained
by a limited amount of training budget. Equipped with StableKD, the Swin-T
network (Liu et al., 2021) achieves impressive 82.6% accuracy on ImageNet,
much better than its 81.2% counterpart with vanilla KD. When having similar
accuracy, StableKD can also speed up the KD process up to 10$\times$. For
example, StableKD reaches 78.45% accuracy on CIFAR100 with 20 epochs while
vanilla KD requires 200 epochs to reach 78.07%. Moreover, our StableKD
framework enjoys further benefits of being data-efficient. For example, with
only 40% of the CIFAR100 or 60% of the ImageNet training data, StableKD
attains comparable or even higher accuracy than vanilla KD with the whole
dataset.

To summarize, our contributions are as follows:

  * •

We reveal and verify, for the first time, the IBOE issue commonly occurred in
end-to-end KD approaches.

  * •

We present StableKD, a simple yet effective KD framework, to mitigate the IBOE
issue.

  * •

We conduct extensive experiments and verify that our StableKD framework is
highly accurate, fast, and data-efficient. It is a generic framework and can
be applied to various tasks and neural networks.

##  2 Related Work

Knowledge distillation (KD), introduced by Hinton et al. (2015), is a
prominent technique for model compression that leverages a large teacher
network to guide a smaller student network (Hinton et al., 2015; Li et al.,
2017, 2021). Existing works can be broadly classified into two types based on
the knowledge transferred: logit-based and feature-based. The former imposes
various regularization on the student’s output (Hinton et al., 2015; Zhao et
al., 2022; Furlanello et al., 2018; Mirzadeh et al., 2020), while the latter
further transfers intermediate representations (Zagoruyko and Komodakis,
2016a; Heo et al., 2019b; Huang and Wang, 2017; Kim et al., 2018; Shen et al.,
2019; Guan et al., 2020; Chen et al., 2021a; Romero et al., 2014; Peng et al.,
2019; Chen et al., 2021b).

Though promising in transferring diverse knowledge from a teacher to its
student, those KD approaches requires intensive computation and time cost to
maintain the teacher’s performance (Beyer et al., 2022). Some recent works
have attempted to accelerate the KD process (Yun et al., 2021; Shen and Xing,
2022), for instance, by caching the teacher’s output for later use in the KD
process (Shen and Xing, 2022). However, these approaches incur storage
overhead, performance degradation, and limited acceleration rate (usually less
than 2$\times$). By contrast, our StableKD overcomes these drawbacks and
achieves a more substantial improvement in reducing the distillation time.

Another line of research is to enhance the data efficiency of KD, by either
updating a subset of parameters alternately (Wang et al., 2018; Kulkarni et
al., 2019), adding cross data flow as regularization (Bai et al., 2020; Shen
et al., 2021), or expanding the training set (Wang et al., 2020; Nguyen et
al., 2022; Laskar and Kannala, 2020). Similar to those works, our StableKD
benefits from high data efficiency by enabling blockwise KD. However, our
StableKD is much simpler to implement, without complex data or architectural
manipulations. Our StableKD is also faster to train, as it performs the
blockwise KD almost parallelly.

##  3 Hypothesis and Experimental Validation

(a) Validation accuracy increases with fluctuations for most of the epochs and
for all runs with different learning rates.

(b) The randomly initialized R-Conv17 incurs higher Euclidean distance of the
FC parameters, than the pre-trained counterparts.

(c) Further supporting the conclusion drawn in (1(b)), yet from the angle of
cumulative Euclidean distance.

Figure 1: Results of the preliminary experiments. All point to the soundness
of our IBOE hypothesis.

It is commonly believed that the success of deep learning largely lies in the
nature of being deep, _i.e._ , the more layers stacked, the better performance
a neural network could achieve. However, optimizing very deep neural networks
is challenging for various reasons, such as the problems of gradient explosion
(Iskoldskii et al., 1977) and gradient vanishing (Hochreiter, 1998).
Specifically, Shalev-Shwartz et al. (2017) identify the issue of noisy
gradient in end-to-end training, which slows down the convergence
significantly. Inspired by this, we investigate the cause of noisy or non-
informative gradients and propose the following hypothesis:

##### The IBOE Hypothesis.

For each block in a randomly-initialized neural network, the randomness of its
back-propagated gradients is influenced by the random noises in the parameters
of its adjacent blocks. This causes optimization instability, reduces the
network accuracy, and delays the convergence, especially when the network is
far from mature. We refer to this phenomenon as Inter-block Optimization
Entanglement (IBOE).

To empirically show that IBOE exists, we conduct two simple exploratory
experiments: 1) Monitor a student network’s validation accuracy during the
vanilla KD process, and check if there’s any fluctuations occurred, as a proxy
to qualitatively reflect the optimization stability. 2) Monitor the Euclidean
distance between the parameters of two consecutive steps, as another proxy to
quantitatively measure the optimization stability (Hardt et al., 2016).

For the first toy experiment, we use the widely-adopted CIFAR100 (Krizhevsky,
2009) dataset, and use the golden ResNet34 and ResNet18 (He et al., 2016) as
the teacher and student networks, respectively. We then conduct the vanilla KD
(refer to Section 4.1 for more details) for 60 epochs using the SGD optimizer
(Paszke et al., 2017) and the OneCycle scheduler (Smith and Topin, 2019). The
base learning rate is set as 0.005, 0.01, and 0.02 for three different runs.
From Figure 1(1(a)), we can clearly see that their validation accuracy
increases with fluctuations for most of the epochs. One may argue that it is
because the step size or the learning rate is too large and overshoots in the
optimal direction. However, such fluctuations still exist for relatively
smaller learning rate. Indeed, this phenomenon can be interpreted as the
consequence of the IBOE issue, where different blocks collaborate but also
interfere with each other.

To further validate our hypothesis quantitatively, we conduct the second
experiment using three feature extractors: R-Conv17, P-Conv17, and P-Conv33.
The prefixes “R-” and “P-” indicate whether a network is randomly initialized
or pre-trained, while “Conv17” and “Conv33” denote the convolutional layers of
a ResNet18 and a ResNet34, respectively, both excluding the original fully
connected (FC) layer. We then attach a new FC layer as the classifier to each
of them. Note that the parameters of the three FC layers have the same initial
random values. We train them following the similar setting to our first
experiment, but now we monitor the Euclidean distance between the FC
parameters of two consecutive steps, as well as its cumulative version. As
shown in Figure 1 (1(b)) and (1(c)), we highlight three key points:

  1. 1)

With R-Conv17, the classifier FC suffers from larger distances and thereby
higher instability than those with P-Conv17 and P-Conv33.

  2. 2)

This gap is much larger in the early training steps.

  3. 3)

P-Conv33 has a similar distance level to P-Conv17, showing that the
optimization stability is less relevant to the network capacity but more
restricted by the mature state or noise level of the parameters.

In short, all these results are consistent to our IBOE hypothesis, and
motivates our next step to address the issue.

##  4 StableKD: Stable Knowledge Distillation

This section begins with a brief overview of the vanilla KD method, which is a
standard baseline in the field. Next, we present our StableKD framework in
detail, focusing on the loss functions and the algorithm design.

###  4.1 Preliminary: Vanilla KD

The vanilla KD approach proposed by Hinton et al. (2015) is well-known for its
simplicity and effectiveness. One typical task is the image classification,
where a student network learns to imitate a teacher by predicting soft
targets, _e.g._ , assigning an image a 0.8 probability of being a cat and a
0.2 probability of being a grassland. We present the mathematical formulation
of this process below.

An input image $x$ is fed into a teacher network
$f^{\mathcal{T}}(\cdot|\theta^{\mathcal{T}})$ with parameters
$\theta^{\mathcal{T}}$ as well as a student network
$f^{\mathcal{S}}(\cdot|\theta^{\mathcal{S}})$ with parameters
$\theta^{\mathcal{S}}$. Their outputs are denoted as
$\hat{y}^{\mathcal{T}}=\text{Softmax}\left(f^{\mathcal{T}}(x|\theta^{\mathcal{T}})\right)$
and
$\hat{y}^{\mathcal{S}}=\text{Softmax}\left(f^{\mathcal{S}}(x|\theta^{\mathcal{S}})\right)$
correspondingly, which are then compared with the ground truth label $y$ to
compute the training loss. Note that the notation of Softmax temperature is
ignored for simplicity, where value $1$ is used for all experiments in this
paper. There are two loss terms involved in the vanilla KD, the cross-entropy
(CE) loss and the Kullback–Leibler (KL) divergence. Both serve as a measure of
statistical distance, but the former is calculated between
$\hat{y}^{\mathcal{S}}$ and $y$ while the latter is between
$\hat{y}^{\mathcal{S}}$ and $\hat{y}^{\mathcal{T}}$, _i.e.,_

$\ell_{CE}=\mathcal{L}_{CE}\left(\hat{y}^{\mathcal{S}},y\right),$ (1)
$\ell_{KL}=\mathcal{L}_{KL}\left(\hat{y}^{\mathcal{S}},\hat{y}^{\mathcal{T}}\right).$
(2)

Now the overall vanilla KD loss can be calculated by

$\mathcal{L}_{KD}=(1-\alpha)\ell_{CE}+\alpha\ell_{KL},$ (3)

where $\alpha\in(0,1)$ is a hyper-parameter balancing the above two loss
terms.

###  4.2 Details of StableKD

Figure 2: Overview of our StableKD. It has two key operations, _Decomposition_
and _Recomposition_. The former divides the teacher and student networks into
several pairs of blocks, which then perform the KD process separately for
certain epochs. After that, the latter merges the adjacent blocks two by two,
followed by a new KD stage.

The above formulation shows that the vanilla KD naively treats the student
network as a whole and a black box for end-to-end training, which inherently
suffers from the IBOE issue as we have examined in Section 3. Building upon
this finding, we introduce StableKD, a simple yet effective KD framework that
enables more fine-grained block-wise training to mitigate the IBOE problem.
Specifically, our StableKD involves two novel operations _Decomposition_ and
_Recomposition_. The _Decomposition_ operation, as shown in Figure 2 (a),
divides a pair of teacher and student networks into several blocks for
separate distillation. Note that the intermediate feature maps from the
teacher are directed to the subsequent block of the student. By contrast, the
_Recomposition_ operation, as shown in Figure 2 (b), progressively merges the
decomposed blocks back along the training process, to preserve the
architectural consistency between training and testing, avoiding potential
accuracy drop. To elaborate our workflow more clearly, we next provide a
rigorous mathematical description.

####  4.2.1 Decomposition.

A neural network $f(\cdot|\theta)$ with parameters $\theta$ can be decomposed
into multiple blocks $f_{1},f_{2},...,f_{k}$ satisfying
$f(\cdot|\theta)=f_{k}\circ...\circ f_{2}\circ f_{1}(\cdot|\theta)$ for some
$k\in\mathbb{N}$, where $f\circ g(\cdot)\coloneqq f(g(\cdot))$. For
simplicity, we denote
$\overset{k}{\underset{i=1}{\operatorname{\bigcirc}}}f_{i}\coloneqq f_{k}\circ
f_{k-1}\circ...\circ f_{1}.$ As there are different ways of decomposition, we
denote $\mathcal{D}$ as one specific decomposition strategy, which specifies
the number of divided blocks $k$ and how these blocks are partitioned from the
student and teacher models. $\mathcal{D}$ can be empirically determined and
preset at the beginning of StableKD process. In this paper, we consider the FC
layer at the end as a single block and divide the remaining layers, _e.g._ ,
convolutional layers and multi-head attention layers into 2 – 4 blocks, based
on their feature dimensions and receptive fields (refer to appendix for more
details).

After the decomposition, we get multiple pairs of teacher and student blocks.
Training is then conducted separately for each pair. For the last pair, the
training loss is similar to Equation 3, comprising a CE loss and a KL
divergence:

$\ell_{CE}=\mathcal{L}_{CE}(\hat{y}^{\mathcal{T},\mathcal{S}},y),$ (4)
$\ell_{KL}=\mathcal{L}_{KL}(\hat{y}^{\mathcal{T},\mathcal{S}},\hat{y}^{\mathcal{T}}),$
(5)

where $\hat{y}^{\mathcal{T},\mathcal{S}}$ denotes the prediction of the last
student block, which relies on its teacher’s preceding blocks, _i.e._ ,
$\hat{y}^{\mathcal{T},\mathcal{S}}=\text{Softmax}\left(f^{\mathcal{S}}_{k}\circ\overset{k-1}{\underset{i=1}{\operatorname{\bigcirc}}}f^{\mathcal{T}}_{i}(x|\theta^{\mathcal{S}},\theta^{\mathcal{T}},\mathcal{D})\right)$.
For the remaining pairs of blocks, we adopt the mean squared error (MSE) loss
to densely transfer the knowledge. Specifically, the MSE loss
$\ell_{MSE}^{\hskip 1.39998pti}$ for block $f_{i}$ is written as

$\ell_{MSE}^{\hskip
1.39998pt1}=\mathcal{L}_{MSE}\left(f^{\mathcal{T}}_{1}(x|\theta^{\mathcal{T}},\mathcal{D}),f^{\mathcal{S}}_{1}(x|\theta^{\mathcal{S}},\mathcal{D})\right),$
(6)

for $i=1$, and

$\displaystyle\ell_{MSE}^{\hskip 1.39998pti}=$
$\displaystyle\mathcal{L}_{MSE}\Bigl{(}\operatorname*{\operatorname{\bigcirc}}_{j=1}^{i}f^{\mathcal{T}}_{j}(x|\theta^{\mathcal{T}},\mathcal{D}),$
(7) $\displaystyle
f^{\mathcal{S}}_{i}\circ\operatorname*{\operatorname{\bigcirc}}_{j=1}^{i-1}f^{\mathcal{T}}_{j}(x|\theta^{\mathcal{S}},\theta^{\mathcal{T}},\mathcal{D})\Bigl{)},$

for $i=2,3,...,k-1$. Although different pairs of blocks are trained
separately, we can also put all these loss terms together as a summary:

$\mathcal{L}_{StableKD}=\ell_{CE}+\lambda\ell_{KL}+\sum_{i=1}^{k-1}\ell_{MSE}^{\hskip
1.39998pti}.$ (8)

Overall, Equation (8) enables various kinds of knowledge transfer from the
teacher blocks to their corresponding student blocks concurrently yet without
interfering with each other. With only a few epochs, the loss shrinks quickly,
and the students blocks start to converge.

####  4.2.2 Recomposition.

While the _Decomposition_ operation is designed to alleviate IBOE issue, the
divided blocks are blind to each other. This leaves the student an
architectural discrepancy between training and testing, and makes the student
less accurate during testing, as we verified in Section 5.3. Therefore, we
propose to progressively merge the divided blocks back along the training
process. To be more concrete, we have the following definition:

######  Definition 1 (Recomposing Function $\mathcal{R}$).

Given $\mathcal{D}$ that encodes the information of $k$ blocks decomposed from
a teacher or a student network, $\mathcal{R}(\mathcal{D})$ merges the original
$i$-th and $\left(i+1\right)$-th blocks together as a new block for every
positive odd number $i\leq k-1$.

In other word, $\mathcal{R}(\cdot)$ is a function taking $\mathcal{D}$ as the
input and returning an updated $\mathcal{D}$, which has
$\lceil\frac{k}{2}\rceil$ blocks after recomposition. Figure 2 (b)
demonstrates this operation in which $i^{\prime}=(i+1)/2$ for $i=1,3,...,k-1$,
supposing $k-1$ is an odd number. Every time the recomposing function is
performed, our StableKD enters a new stage of training, and finally evolves
towards the vanilla KD.

####  4.2.3 Complete pipeline.

Algorithm 1 StableKD algorithm

Parameter: $\theta^{\mathcal{S}},\theta^{\mathcal{T}}$
Input: decomposition variable $\mathcal{D}$, number of recompositions $n$,
number of training epochs for each stage $e_{i}$, balancing factor of losses
$\lambda$

1: Initialize $\theta^{\mathcal{S}}$ and freeze $\theta^{\mathcal{T}}$.

2: for training stage $c=0,1,...,n$ do

3: for each of $e_{c}$ epochs do

4: Update $\theta^{\mathcal{S}}$ with Equation (8). {Decomposition.}

5: end for

6: $\mathcal{D}\leftarrow\mathcal{R}(\mathcal{D})$ {Recomposition.}

7: end for

8: return $\theta^{\mathcal{S}}$

Given the initial decomposition strategy $\mathcal{D}$ and the number of
recompositions to perform $n$, we present the complete pipeline of StableKD in
Algorithm 1. The complete pipeline is simple to understand and easy to
implement. It’s expected to achieve a higher accuracy and converge faster than
vanilla KD. Moreover, our StableKD is as computation-efficient as the vanilla
KD for each iteration, by avoiding complex and heavy data and architectural
manipulations. Note that we denote StableKD-k/n as the full StableKD pipeline
with $k$ divided blocks initially and with $n$ recompositions.

##  5 Experimental Evaluation

In this section, we begin by validating the efficacy of StableKD in mitigating
the IBOE issue. We then present empirical evidence showcasing the advantages
of StableKD from three practical aspects: convergence speed, accuracy, and
data efficiency. Our experimental analyses encompass convolutional neural
networks and transformer-based models trained on diverse datasets, such as
CIFAR100 (Krizhevsky, 2009), Imagewoof (Howard, 2019), and ImageNet
(Russakovsky et al., 2015). The results illustrate that StableKD achieves
superior accuracy compared to previous approaches, while requiring fewer
training epochs and reduced training data. Furthermore, we conduct ablation
studies to assess the indispensability of the decomposition and recomposition
operations in StableKD, and also explore potential avenues for future
enhancements.

###  5.1 StableKD in Alleviating IBOE

Figure 3:  StableKD effectively alleviates the inter-block optimization
entanglement.

This experiment aims to validate the efficacy of StableKD in mitigating the
IBOE issue. We adopt the ResNet18 (He et al., 2016) as the student and the
ResNet34 with pre-trained accuracy $78.39\%$ as the teacher on CIFAR100. We
explore three decomposition strategies with the values of k equal to 1, 3, and
5, while no recompositions are used. Note that setting $k=1$ is equivalent to
the vanilla KD. For all the three runs, we conduct the training for 60 epochs,
with a learning rate of 0.05 and a OneCycleLR scheduler (Smith and Topin,
2019). We monitor their validation accuracies along the training process.
Figure 3 shows that the accuracy of vanilla KD or StableKD-1/0 suffer from
severe fluctuations, whereas the accuracy of StableKD-5/0 and StableKD-3/0
increase much more stably, effectively circumventing the IBOE issue.

###  5.2 Illustrative Results

(a) CIFAR100

(b) Imagewoof

Figure 4:  StableKD outperforms others by achieving higher test accuracy with
fewer training epochs on CIFAR100 and Imagewoof datasets.

(c) WRN16-10 & WRN28-10

(d) ResNeXt20($32\times 4d$) & ResNeXt38($32\times 4d$)

Figure 5:  For different teacher-student pairs, StableKD consistently achieves
higher accuracy on CIFAR100 with fewer epochs than the vanilla KD.

####  5.2.1 Comparison of convergence speed and accuracy.

In deep learning, the convergence speed generally refers to the number of
epochs, iterations, or training time spent in achieving a high level of
accuracy. By contrast, the accuracy is often reported given a fixed number of
epochs. For comprehensive evaluation, here we mainly compare the trade-offs
between the test accuracy and the total number of epochs. We first conduct
experiments on CIFAR100 and Imagewoof datasets. We train a ResNet18 with a
teacher ResNet34 for 20, 40, 60, 100, 150, and 200 epochs. For StableKD, we
set the following hyper-parameters: $k=5,n=2,e_{0}=\lfloor
e/3\rfloor,e_{1}=\lfloor e/3\rfloor,e_{2}=e-2\lfloor e/3\rfloor$, where $e$ is
the total number of epochs. We compare StableKD against six representative
works, including the vanilla KD (Hinton et al., 2015), Attention Transfer (AT;
Zagoruyko and Komodakis 2016a), Sobolev Training (ST; Czarnecki et al. 2017),
Neuron Selectivity Transfer (NST; Huang and Wang 2017), Knowledge Review
(ReviewKD; Chen et al. 2021b), and Decoupled Knowledge Distillation (DKD; Zhao
et al. 2022). Results in Figure 5 show that StableKD renews the state of the
art, consistently achieving the highest accuracy given varied number of
epochs, where the gap is more evident with limited training budget. From
another perspective, StableKD converges faster than others in achieving a high
accuracy.

These conclusions are consistent when further applied to other teacher-student
pairs, _e.g._ , the WRN28-10-WRN16-10 and the ResNeXt38 ($32\times
4d$)-ResNeXt20 ($32\times 4d$). Results in Figure 5 show that StableKD speeds
up the KD process by 5$\times$, from 200 epochs to 40 epochs when trained on
CIFAR100. With only 20 epochs, StableKD also obtains a significant 3.8% and
4.0% accuracy gain for the two pairs, respectively.

We also conduct experiments on ImageNet, a much larger scale of classification
dataset with 1.2 million images. We use transformer-based backbones including
ViT (Dosovitskiy et al., 2020) and Swin (Liu et al., 2021). Results are
summarized in Table 1. Compared to training w/o KD, vanilla KD, and DeiT
(Touvron et al., 2021), StableKD improves the accuracy by 2.0% - 3.7% for both
90 and 300 epochs. StableKD-5/3 trained with 90 epochs even beats the vanilla
KD method with 300 epochs (81.51% _v.s._ 81.15%).

Student $\mathcal{S}$ | Teacher $\mathcal{T}$ | Epochs | KD Scheme |  Acc. (%)
---|---|---|---|---
ViT-S/8 |  ViT-S/16 (74.03%) | 90 | w/o KD | 74.00
KD | 75.05
DeiT | 75.45
StableKD-5/3 | 77.72
Swin-T |  Swin-S (83.33%) | 90 | w/o KD | 79.53
KD | 79.95
StableKD-5/3 | 81.51
Swin-T |  Swin-S (83.33%) | 300 | w/o KD | 80.37
KD | 81.15
StableKD-5/3 | 82.61

Table 1: Performance comparison on the large scale ImageNet dataset. ViT-S/8
denotes a network with depth 8 derived from ViT-S/16. DeiT is designed for ViT
and not directly applicable to Swin.

####  5.2.2 Comparison of data efficiency.

Dataset | Student $\mathcal{S}$ | Teacher $\mathcal{T}$ | # epochs |  KD Scheme | Acc. (%) for various subset sizes
---|---|---|---|---|---
$20\%$ | $40\%$ | $60\%$ | $80\%$ | 100%
CIFAR100 | ResNet18 |  ResNet34 (78.39%) | 300 | KD | 55.65 | 67.64 | 72.88 | 75.91 | 78.12
AT | 56.53 | 67.22 | 71.99 | 75.31 | 77.99
ST | 55.47 | 67.54 | 71.87 | 75.26 | 77.75
NST | 60.84 | 69.87 | 74.36 | 76.16 | 78.14
ReviewKD | 62.20 | 71.25 | 75.20 | 77.19 | 78.97
DKD | 69.76 | 75.60 | 77.39 | 78.77 | 79.39
StableKD-5/2 | 76.10 | 78.50 | 79.18 | 79.63 | 79.41
ImageNet | Swin-T |  Swin-S (83.33%) | 90 | w/o KD | 66.01 | 74.98 | 77.76 | 78.67 | 79.53
KD | 69.64 | 76.10 | 78.33 | 79.52 | 79.95
StableKD-5/3 | 76.03 | 78.87 | 80.34 | 80.89 | 81.51

Table 2: Comparison of data efficiency with varying subsets of the training
data.

We compare the data efficiency by examining how well different KD methods
perform with reduced training dataset, from 100% to only 20% of the whole
dataset. Other training settings and baselines are the same to previous
experiments. We present the results in Table 2. Compared to prior works,
StableKD demonstrates significantly higher data efficiency by maintaining
robustly good accuracy with varying subset sizes. With 20% of the training
data, StableKD outperforms others by a large margin ranging from 6% to 20% in
accuracy. Furthermore, StableKD trained with 40% - 60% of the training data
performs similarly or even better than others trained with the whole dataset.

It is common practice to pre-train transformer-based models on external data
before fine-tuning them on the target dataset to prevent overfitting and
improve performance. To further validate the strengths of StableKD in
achieving higher data efficiency and thereby reducing the training cost, we
trained transformer-based student networks from scratch on CIFAR100 without
any pre-training on external data. Results in Table 3 show that all baselines
perform poorly while StableKD stands out with 10% - 18% significantly higher
accuracy, indicating its superior data efficiency.

###  5.3 Ablation Studies

##### Effect of decomposition.

To validate the effectiveness of decomposition, we degrade StableKD to
conventional feature-based KD methods by imposing the feature alignment but
removing the independence of different blocks. We adopt the ResNet18-ResNet34
pair and train the student for 60 epochs. Results in Table 4 show that the
feature alignment through MSE loss does enhance the vanilla KD by 1.0% in
accuracy. However, a more substantial improvement of 3.2% is achieved with the
implementation of StableKD. This, therefore, substantiates the necessity of
our decomposition operation, particularly the importance of enabling
_independent_ block-wise KD.

Student $\mathcal{S}$ |  Teacher $\mathcal{T}$ | Scheme |  Acc. (%)
---|---|---|---
ViT-T/8 |  ViT-T/16 (88.98%) | w/o KD | 67.99
KD | 66.46
DeiT | 71.01
StableKD-5/2 | 84.18
Swin-T |  Swin-S (90.39%) | w/o KD | 73.78
KD | 73.84
StableKD-5/2 | 83.92

Table 3: Comparison when training transformer-based networks on CIFAR100 from
scratch without pre-training on external data. We use 600 training epochs
here.

Scheme | KD | KD + MSE | StableKD-5/2
---|---|---|---
Acc. (%) | 76.19 | 77.15 | 79.39

Table 4: With feature alignment, KD + MSE is still inferior to StableKD.

##### Effect of recomposition.

We next evaluate the effectiveness of progressively merging the decomposed
blocks by training w/ and w/o recomposition. Other training settings remain
the same with the previous ablation experiment. Results in Table 5 show that,
without recomposition, StableKD-5/0 surpasses the vanilla KD for 20, 40, and
60 epochs due the use of decomposition in accelerating the convergence.
However, StableKD-5/0 gets lower accuracy than the vanilla KD for more
training epochs. This is because individual blocks are trained independently
but lack mutual awareness, leading to sub-optimal accuracy. To address it, we
introduce the recomposition procedure, which consistently yields higher
accuracy as is evident in the results.

KD | Acc. (%) for various numbers of epochs
---|---
schemes |  20 |  40 |  60 |  100 |  150 |  200
KD |  72.49 |  75.00 |  76.19 |  77.08 |  77.50 |  78.07
StableKD-5/0 |  76.59 |  76.90 |  77.26 |  76.99 |  77.39 |  77.23
StableKD-5/2 |  78.45 |  79.02 |  79.39 |  79.21 |  79.61 |  79.46

Table 5: StableKD-5/2 with recomposition consistently obtains higher accuracy
than StableKD-5/0 without recomposition and the vanilla KD.

Student | Teacher | Projector | Test accuracy (%)
---|---|---|---
(Acc., # params) | (# params) | w/o KD | KD | StableKD-2/1
WRN40-2 | WRN40-4 | ✗(2.2M) | 74.56 | 75.35 | n/a
($77.11\%$, 8.9M) | ✔(2.3M) | 74.96 | 75.40 | 75.61
WRN16-8 | WRN16-10 | ✗(11.0M) | 78.53 | 78.67 | n/a
($79.69\%$, 17.1M) | ✔(11.6M) | 78.75 | 78.73 | 79.57

Table 6: More results for heterogeneous architectures. The reported accuracies
of WRN40-2 and WRN16-8 in their original paper are $73.96\%$ and $77.93\%$,
respectively.

##### Dealing with heterogeneous feature dimensions.

Our main experiments are conducted when the student and teacher networks have
the same feature dimension in each block. What if the dimension differs, _e.g_
, having different network width? To address this and extend the application
scope of StableKD, we can simply insert additional projectors, _e.g._ , the
conv $1\times 1$ layer, within the student network, and thus enable the use of
our StableKD. To validate this, we conduct experiments on Wide ResNet
(Zagoruyko and Komodakis, 2016b) with varying width factors. We train them on
CIFAR100 for 200 epochs. Results in Table 6 demonstrate the promising
applicability of StableKD to heterogeneous feature dimensions.

##  6 Conclusion

In this paper, we have studied and uncovered a common issue of existing KD
approaches – the inter-block optimization entanglement that causes their slow
convergence and lower accuracy. To overcome it, we have proposed a novel
StableKD framework, which divides a pair of teacher and student networks into
several blocks, and distills the block-wise knowledge separately before
combining them together for inference. We have conducted extensive experiments
and validated that StableKD is highly accurate, fast-converging, and data-
efficient. We hope that our StableKD can make KD more practical as a tool for
model compression and acceleration, particularly in the era of building
foundation models towards artificial general intelligence.

Appendix

##  Appendix A Limitations and Future Work

We have verified the application of StableKD in image classification across
several different neural networks. However, the proposed decomposition can
become complex when the model architecture includes longer shortcuts, _i.e._ ,
operations that pass features from shallow layers to deep ones. For example,
networks for semantic segmentation, such as FCN (Long et al., 2015) and U-Net
(Ronneberger et al., 2015), often concatenate features from shallow and deep
layers to enhance performance. Since our proposed StableKD primarily focuses
on dividing blocks in a sequential order, the exploration of non-sequential
neural networks remains an open area for future work.

##  Appendix B Training Details

##### Learning rate scheduler.

We propose a novel scheduler to enable per-training-stage warm-up and
annealing. Intuitively, our learning rate extends the concepts from the
cyclical learning rate (Smith, 2017) in PyTorch CyclicLR with triangular2
strategy; that is, instead of using cycles with same number of epochs, each
cycle corresponds to the number of epochs in each StableKD stage. Illustration
can be found in Figure 6. For other comparison schemes, we use a OneCycle
scheduler (Smith and Topin, 2019) in the CIFAR100 and Imagewoof experiments
(Figure 5, 5, Table 2, 3, 4, 5, 6), and a cosine-annealing scheduler with
linear warm-up in the ImageNet experiments (Table 1,2). Note that all
experiments in Figure 3 are conducted with a OneCycle scheduler.

##### Optimizer.

We adopt SGD as the optimizer in the CIFAR100 and Imagewoof experiments
(Figure 3, 5, 5, Table 2, 3, 4, 5, 6), and use AdamW for the ImageNet
experiments (Table 1, 2).

##### Hyperparameters.

Our empirical experience recommends a higher learning rate for StableKD. In
the CIFAR100 and Imagewoof experiments, all the experiments are conducted with
a batch size of $128$ and weight decay of $5e-4$. The maximal learning rate of
StableKD is set to $0.5$. For other comparison schemes, we use a maximal
learning rate of $0.1$, where they demonstrate better performance and thus
serve as stronger baselines. In the ImageNet experiments, we set the maximal
learning rate, batch size, and weight decay to $2e-2,256,2e-1$ for StableKD,
and to $2e-3,512,4e-2$ for other schemes, respectively. We found that StableKD
generally performs better when a larger learning rate, a smaller batch size,
and a smaller weight decay are used. This differs from previous works, which a
smaller batch size usually accompanies a larger weight decay and a smaller
learning rate. We attribute this difference to the benefits of higher
stability that make the model easier to train.

Figure 6: Learning rate scheduler used in StableKD. Figure 7: Comparison of accuracy on CIFAR100 with the ResNet34-ResNet18 pair. KD | Average time per epoch (s)
---|---
schemes | CIFAR100 | Imagewoof
KD | 36.3 | 14.9
AT | 37.5 | 15.6
ST | 120.6 | 44.7
NST | 60.0 | 30.0
ReviewKD | 72.0 | 28.4
DKD | 37.3 | 15.3
DnD | 35.4 | 14.7
Table 7: Comparison of time cost per epoch with the ResNet34-ResNet18 pair.

##  Appendix C Time Cost

StableKD has low computational complexity and is easy to implement, as no
overlapping backward process takes place in the student network. Table 7 shows
the average training time per epoch on the CIFAR100 and Imagewoof datasets,
with a teacher ResNet34 and a student ResNet18 on a single RTX-2080Ti GPU.
With these results, we can also adjust the horizontal axis of Figure 5 to
display time cost for clearer visualization. For example, Figure 7 shows the
results on CIFAR100 with the same teacher-student pair.

##  Appendix D Decomposition Details

This paragraph describes the decomposition recipes of StabelKD in the main
paper. In our experiment, we regard the last MLP layer as a single block. For
ResNet (He et al., 2016), Wide ResNet (Zagoruyko and Komodakis, 2016b), and
ResNeXt (Xie et al., 2017) models, we divide the convolutional layers based on
the dimension of each residual block. For example, the first decomposed block
of ResNet18 in StableKD-5/n consists of the layers from first convolution
layer to the last one with an output channel of 64. The second, third, and
fourth blocks include the layers with output channelS of 128, 256, and 512,
respectively. The fifth block represents the fully-connected layer. A similar
recipe is used in ResNet34 as well. For Swin models, we follow the division
method in its original paper (Liu et al., 2021), where the attention layers
are decomposed into blocks with depths of 2, 2, 6, 2 in Swin-T, and 2, 2, 18,
2 in Swin-S. For ViT models (Dosovitskiy et al., 2020), the attention layers
are equally distributed to the first four blocks of StableKD-5/n, and the last
block contains the fully-connected layer.

## References

  * Bai et al. [2020] Haoli Bai, Jiaxiang Wu, Irwin King, and Michael Lyu.  Few shot network compression via cross distillation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34(04), pages 3203–3210, 2020.
  * Beyer et al. [2022] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10925–10934, 2022.
  * Chen et al. [2021a] Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen.  Cross-layer distillation with semantic calibration.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35(8), pages 7028–7036, 2021.
  * Chen et al. [2021b] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia.  Distilling knowledge via knowledge review.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008–5017, 2021.
  * Czarnecki et al. [2017] Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu.  Sobolev training for neural networks.  Advances in Neural Information Processing Systems, 30, 2017.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * Furlanello et al. [2018] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.  Born again neural networks.  In International Conference on Machine Learning, pages 1607–1616. PMLR, 2018.
  * Guan et al. [2020] Yushuo Guan, Pengyu Zhao, Bingxuan Wang, Yuanxing Zhang, Cong Yao, Kaigui Bian, and Jian Tang.  Differentiable feature aggregation search for knowledge distillation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16, pages 469–484. Springer, 2020.
  * Hardt et al. [2016] Moritz Hardt, Ben Recht, and Yoram Singer.  Train faster, generalize better: Stability of stochastic gradient descent.  In International conference on machine learning, pages 1225–1234. PMLR, 2016.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * Heo et al. [2019a] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi.  A comprehensive overhaul of feature distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921–1930, 2019.
  * Heo et al. [2019b] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi.  Knowledge transfer via distillation of activation boundaries formed by hidden neurons.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33(01), pages 3779–3787, 2019.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7), 2015.
  * Hochreiter [1998] Sepp Hochreiter.  The vanishing gradient problem during learning recurrent neural nets and problem solutions.  International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):107–116, 1998.
  * Holcomb et al. [2018] Sean D Holcomb, William K Porter, Shaun V Ault, Guifen Mao, and Jin Wang.  Overview on deepmind and its alphago zero ai.  In Proceedings of the 2018 international conference on big data and education, pages 67–71, 2018.
  * Howard [2019] Jeremy Howard.  Imagenette/imagewoof, 2019.
  * Huang and Wang [2017] Zehao Huang and Naiyan Wang.  Like what you like: Knowledge distill via neuron selectivity transfer.  arXiv preprint arXiv:1707.01219, 2017.
  * Iskoldskii et al. [1977] AM Iskoldskii, Iu E Nesterikhin, Aleksandr Zaharovich Patashinskii, Vladimir Kirillovich Pinus, and Ia G Epelbaum.  On the instability of a gradient explosion.  In Akademiia Nauk SSSR Doklady, volume 236, pages 1346–1349, 1977.
  * Jung et al. [2021] Sangwon Jung, Donggyu Lee, Taeeon Park, and Taesup Moon.  Fair feature distillation for visual recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12115–12124, 2021.
  * Kim et al. [2018] Jangho Kim, SeongUk Park, and Nojun Kwak.  Paraphrasing complex network: Network compression via factor transfer.  Advances in neural information processing systems, 31, 2018.
  * Krizhevsky [2009] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Tech Report, 2009.
  * Kulkarni et al. [2019] Akshay Kulkarni, Navid Panchi, Sharath Chandra Raparthy, and Shital Chiddarwar.  Data efficient stagewise knowledge distillation.  arXiv preprint arXiv:1911.06786, 2019.
  * Laskar and Kannala [2020] Zakaria Laskar and Juho Kannala.  Data-efficient ranking distillation for image retrieval.  In Proceedings of the Asian Conference on Computer Vision, 2020.
  * Li et al. [2017] Quanquan Li, Shengying Jin, and Junjie Yan.  Mimicking very efficient network for object detection.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 6356–6364, 2017.
  * Li et al. [2021] Zheng Li, Jingwen Ye, Mingli Song, Ying Huang, and Zhigeng Pan.  Online knowledge distillation for efficient pose estimation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 11740–11750, 2021.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * Long et al. [2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * Mirzadeh et al. [2020] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh.  Improved knowledge distillation via teacher assistant.  In Proceedings of the AAAI conference on artificial intelligence, volume 34(04), pages 5191–5198, 2020.
  * Nguyen et al. [2022] Dang Nguyen, Sunil Gupta, Kien Do, and Svetha Venkatesh.  Black-box few-shot knowledge distillation.  In European Conference on Computer Vision, pages 196–211. Springer, 2022.
  * Paszke et al. [2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * Peng et al. [2019] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang.  Correlation congruence for knowledge distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007–5016, 2019.
  * Romero et al. [2014] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550, 2014.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * Shalev-Shwartz et al. [2017] Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah.  Failures of gradient-based deep learning.  In International Conference on Machine Learning, pages 3067–3075. PMLR, 2017.
  * Shen and Xing [2022] Zhiqiang Shen and Eric Xing.  A fast knowledge distillation framework for visual recognition.  In European Conference on Computer Vision, pages 673–690. Springer, 2022.
  * Shen et al. [2019] Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song.  Amalgamating knowledge towards comprehensive classification.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33(01), pages 3068–3075, 2019.
  * Shen et al. [2021] Chengchao Shen, Xinchao Wang, Youtan Yin, Jie Song, Sihui Luo, and Mingli Song.  Progressive network grafting for few-shot knowledge distillation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35 (3), pages 2541–2549, 2021.
  * Smith and Topin [2019] Leslie N Smith and Nicholay Topin.  Super-convergence: Very fast training of neural networks using large learning rates.  In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pages 369–386. SPIE, 2019.
  * Smith [2017] Leslie N Smith.  Cyclical learning rates for training neural networks.  In 2017 IEEE winter conference on applications of computer vision (WACV), pages 464–472. IEEE, 2017.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International conference on machine learning, pages 10347–10357. PMLR, 2021.
  * Wang et al. [2018] Hui Wang, Hanbin Zhao, Xi Li, and Xu Tan.  Progressive blockwise knowledge distillation for neural network acceleration.  In IJCAI, pages 2769–2775, 2018.
  * Wang et al. [2020] Dongdong Wang, Yandong Li, Liqiang Wang, and Boqing Gong.  Neural networks are more productive teachers than human raters: Active mixup for data-efficient knowledge distillation from a blackbox model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1498–1507, 2020.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * Yun et al. [2021] Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun.  Re-labeling imagenet: from single to multi-labels, from global to localized labels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2340–2350, 2021.
  * Zagoruyko and Komodakis [2016a] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  arXiv preprint arXiv:1612.03928, 2016.
  * Zagoruyko and Komodakis [2016b] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.
  * Zhao et al. [2022] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang.  Decoupled knowledge distillation.  In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962, 2022.