# ISD: Self-Supervised Learning by Iterative Similarity Distillation

Ajinkya Tejankar${}^{1,}\quad$11footnotemark: 1 Soroush Abbasi
Koohpayegani${}^{1,}\thanks{Equal contribution}\quad$ Vipin
Pillai${}^{1}\quad$ Paolo Favaro${}^{2}\quad$ Hamed Pirsiavash${}^{1}\quad$

1University of Maryland Equal contribution Baltimore County 2University of
Bern

###### Abstract

Recently, contrastive learning has achieved great results in self-supervised
learning, where the main idea is to push two augmentations of an image
(positive pairs) closer compared to other random images (negative pairs). We
argue that not all random images are equal. Hence, we introduce a self
supervised learning algorithm where we use a soft similarity for the negative
images rather than a binary distinction between positive and negative pairs.
We iteratively distill a slowly evolving teacher model to the student model by
capturing the similarity of a query image to some random images and
transferring that knowledge to the student. We argue that our method is less
constrained compared to recent contrastive learning methods, so it can learn
better features. Specifically, our method should handle unbalanced and
unlabeled data better than existing contrastive learning methods, because the
randomly chosen negative set might include many samples that are semantically
similar to the query image. In this case, our method labels them as highly
similar while standard contrastive methods label them as negative pairs. Our
method achieves better results compared to state-of-the-art models like BYOL
and MoCo on transfer learning settings. We also show that our method performs
better in the settings where the unlabeled data is unbalanced. Our code is
available here: https://github.com/UMBCvision/ISD

Figure 1: Our method: We initialize both teacher and student networks from
scratch and update the teacher as running average of the student. We feed some
random images to the teacher, and feed two different augmentations of a query
image to both teacher and student. We capture the similarity of the query to
the anchor points in the teacher’s embedding space and transfer that knowledge
to the student. We update the student based on KL divergence loss and update
the teacher to be a slow moving average of the student. This can be seen as a
soft version of MoCo [14] which can handle negative images that are similar to
the query image. Note that unlike contrastive learning and BYOL, we never
compare two augmentations of the query images directly (positive pair).

##  1 Introduction

We can view the recent crop of SSL methods as iterative self-distillation
where there is a teacher and a student. Both teacher and student improve
simultaneously while the teacher is evolving more slowly (running average)
compared to the student: (1) In the case of contrastive methods e.g. MoCo
[14], we classify images to positive and negative pairs in the binary form.
(2) In the case of clustering methods (DC-v2, SwAV [7], SeLA [37]), the
student predicts the quantized representations from the teacher. (3) In the
case of BYOL [12], the student simply regresses the teacher’s embeddings
vector. Here, we introduce a novel method using similarity based distillation
to transfer the knowledge from the teacher to the student. We argue that our
method is more regularized compared to prior work and improves the quality of
the features in transfer learning.

In the standard contrastive setting, , MoCo [14], there is a binary
distinction between positive and negative pairs, but in practice, many
negative pairs may be from the same category as the positive one. Thus,
forcing the model to classify them as negative is misleading. This can be more
important when the unlabeled training data is unbalanced, for example, when a
large portion of images are from a small number of categories. Such scenario
can happen in applications like self-driving cars, where most of the data is
just repetitive data captured from a high-way scene with a couple of cars in
it. In such cases, the standard contrastive learning methods will try to learn
features to distinguish two instances of the most frequent category that are
in a negative pair, which may not be helpful for the down-stream task of
understanding rare cases.

We are interested in relaxing the binary classification of contrastive
learning with soft labeling, where the teacher network calculates the
similarity of the query image with respect to a set of anchor points in the
memory bank, convert that into a probability distribution over neighboring
examples, and then transfer that knowledge to the student, so that the student
also mimics the same neighborhood similarity. In the experiments, we show that
our method performs better than SOTA self-supervised methods on ImageNet and
also we show an improved accuracy on the rare cases when trained on
unbalanced, unlabeled data (for which we use a subset of ImageNet).

Note that we are using a teacher and student model similar to MoCo, but it is
for a different purpose: in MoCo the goal of having two models is to be able
to maintain a large memory bank for the negatives, but in our case, the
teacher acts as a role model for the student to follow. In fact, SimCLR [8]
shows that there is no need for two models in contrastive learning if we can
handle the large memory bank with large mini-batches.

Our method is different from BYOL in that we are comparing the query image
with other random images rather than only with a different augmentation of the
same query image. We believe our method can be seen as a more relaxed version
of BYOL. Instead of imposing that the embedding of the query image should not
change at all due to an augmentation (as done in BYOL), we are allowing the
embedding to vary as long as its neighborhood similarity does not change. In
other words, the augmentation should not change the similarity of the image
compared to its neighboring images. This relaxation lets self-supervised
learning focus on what matters most in learning rich features rather than
forcing an unnecessary constraint of no change at all, which is difficult to
achieve.

Our distillation method is inspired by the CompRess method [1], which
introduces an analogous similarity-based distillation method to compress a
deeper self-supervised model to a smaller one and get better results compared
to training the small model from scratch. Our method is different from [1] in
that in our case, both teacher and student share the same architecture, we do
self-supervised learning from scratch rather than compressing from another
deeper model, and also the teacher evolves over time as a running average of
the student rather than being frozen as in [1].

Most recent self-supervised learning methods including ours use an aggressive
augmentation to learn good features. However, those augmented images do not
look natural to human eyes. This means they add a bias to the data that does
not exist in real world applications. We show that we can use a model trained
with aggressive augmentation as a frozen teacher and then adapt it to another
model that has not seen those aggressive augmentations. Interestingly the new
model performs better than the teacher in down-stream transfer learning tasks.
We believe this happens since the new model has adopted priors of a more
natural dataset through our similarity-based distillation.

Figure 2: We sample some query images randomly (left column), calculate their
teacher probability distribution over all anchor points in the memory bank
(size=128K) and rank them in descending order (right columns). The second left
column is another augmented version of the query image that contrastive
learning methods use for the positive pair. Our students learns to mimic the
probability number written below each anchor image while contrastive learning
method (e.g., MoCo) learn to predict the one-hot encoding written below the
images. Note that there are lots of images in the top anchor points that are
semantically similar to the query point that MoCo tries to discriminate them
from the query while our method does not.

##  2 Related Work

Self-supervised learning: The task of learning representations by solving a
pretext task without using any supervised annotations is called self-
supervised learning. Various pretext tasks like solving jigsaw puzzles [23],
predicting rotations [11], counting the visual primitives [24], filling up a
missing patch [28], predicting missing channels of input [39, 40], and
contrastive learning [13, 14] are explored in the literature. We are proposing
a novel pretext task based on iterative similarity based distillation.

Contrastive learning: The task of learning unsupervised representations by
contrasting the representations of an image with other images is called
contrastive learning [13]. Contrastive learning is essentially
positive/negative classification where the positive and negative pairs of
embeddings can be defined in various ways. In [34, 8, 14], the positive pairs
are augmented views of the same image while the negative pairs are those of
different images. In [15], the positive pairs are a patch and context
embeddings from the same image while the negatives pairs are a patch and
context embedding from different images. In [17, 4], the positives pairs are
global and local features from the same image while the negative pairs are
global and local features from different images. In [37, 7, 6], the positive
pairs are members of the same cluster while the negative pairs are member of
different clusters. We are different from these contrastive methods in that we
do not consider all negatives equally: we calculate a soft labeling for the
negatives using similarity of the data points. Also, we do not consider
positive pairs directly: the comparison for the positive pairs is done through
the similarity distillation.

Knowledge Distillation: The task of transferring the knowledge from one model
to the other is called knowledge distillation [16, 3]. The knowledge from the
teacher can be extracted and transferred in various ways. The knowledge in the
activations of intermediate layers can be transferred through regression [30,
36, 38]. While in most works the teacher is a deeper model and the student is
a shallower model, in [5, 10] both teacher and the student use the same
architecture. Techniques from knowledge distillation can also be used in an
unsupervised way to improve self-supervised learning [1, 25, 35]. Instead of
using knowledge distillation for model compression or reducing the
generalization gap, we use it iteratively to evolve the teacher and student
together to learn rich representations from scratch.

Similarity based knowledge distillation: While the above methods [16, 3, 30,
36, 38] only extract the information about a single data point from the
teacher, similarity based distillation methods [27, 29, 26, 33, 2, 32, 1]
represent the knowledge from the teacher in terms of similarities between data
points. Probably CompRess [1] is the closest related work to our method that
uses similarity based distillation to compresses a large self supervised model
to a smaller one. Our method is different since we use the similarity based
distillation to transfer the similarity knowledge from evolving teacher to the
student iteratively to learn rich representations.

Consistency regularization: Consistency regularization is a method of
regularization that seeks to make the output of a model consistent across
small perturbations in either the input [21] or the model parameters [31].
Recently, BYOL [12] applied a variant of Mean Teachers [31] for self-
supervised learning. Our method uses a variation of this idea through
similarity based loss rather than regression loss defined on data points
individually.

##  3 Method

We are interested in learning rich representations from unlabeled data. We
have a teacher network and a student network. We initialize both models from
scratch and update the teacher to be a slower version of the student: we use
the momentum idea from MoCo in updating the teacher so that it is running
average of the student. The method is described in Figure 1. Following the
notation in [1], at each iteration, we pick a random query image and a bunch
of random other images that we call anchor points. We augment those images and
feed them to the teacher model to get their embeddings. Then, we augment the
query again independent of the earlier augmentation and feed it to the student
model only. We calculate the similarity of the query point compared to the
anchor points in the teacher’s embedding space and then optimize the student
to mimic the same similarity for the anchor point at the student’s embedding
space. Finally, we update the teacher with a momentum to be the running
average of the student similar to MoCo and BYOL. Note that our method is
closely related to ComPress method [1] which uses the similarity distillation
for compressing a frozen larger model to a smaller one.

More formally, we assume a teacher model $t$ and a student model $s$. Given a
query image $q$, we augment it twice independently to get $q_{t}$ and $q_{s}$.
We also assume a set of $n$ augmented random images $\\{x_{i}\\}_{i=1}^{n}$.
We feed $\\{x_{i}\\}_{i=1}^{n}$ to the teacher model to get their embeddings
$\\{t(x_{i})\\}_{i=1}^{n}$ and call them anchor points. We also feed $q_{t}$
to the teacher and $q_{s}$ to the student to get $t(q_{t})$ and $s(q_{s})$
respectively. Then, we calculate the similarity of the query embedding
$t(q_{t})$ compared to all anchor points, divide by a temperature, and convert
to a probability distribution using a SoftMax operator to get:

$p_{t}(i)=-\text{log}\frac{\text{exp}\big{(}\text{sim}\big{(}t(q_{t}),t(x_{i})\big{)}/\tau\big{)}}{\sum_{j=1}^{n}\text{exp}\big{(}\text{sim}\big{(}t(q_{t}),t(x_{j})\big{)}/\tau\big{)}}$

where $\tau$ is the temperature and $\text{sim}(.,.)$ refers to the similarity
between two vectors. In our experiments, we use cosine similarity which is
standard is most recent contrastive learning methods.

Then, we calculate a similar probability distribution for the student’s query
embedding to get:

$p_{s}(i)=-\text{log}\frac{\text{exp}\big{(}\text{sim}\big{(}s(q_{s}),t(x_{i})\big{)}/\tau\big{)}}{\sum_{j=1}^{n}\text{exp}\big{(}\text{sim}\big{(}s(q_{s}),t(x_{j})\big{)}/\tau\big{)}}$

Finally, we optimize the student only by minimizing the following loss:

$L=KL(p_{t}||p_{s})$

and the teacher is updated using the following rule:

$\theta_{t}\leftarrow m\theta_{t}+(1-m)\theta_{s}$

where $\theta$ refers to the parameters of a model and $m$ is a the momentum
hyperparameter that is set be close to one ($0.999$ in our experiments) as in
MoCo. Since the teacher is not optimized by the loss directly, the loss can be
simplified as cross entropy loss instead of KL divergence.

Note that unlike the positive pair in contrastive learning methods and BYOL,
the query image is never compared to its own augmentation as it is not
included in the anchor points. We do this since when the features are mature,
the similarity of the query to itself will be very large and will dominate the
whole probability distribution. Note that one can convert our method to MoCo
by including the query in anchor points and replacing the probability of the
teacher $p_{t}$ with a one-hot encoding vector in which the positive pair
(query) corresponds to one and all other anchor points correspond to zero.
Figure 2 shows some example teacher probabilities for both ISD and MoCo.

Our method can benefit from a large number of anchor points to cover the
neighborhood of any query image, and also the anchor points are fed to the
teacher only that evolves slowly. Hence, we use MoCo’s trick of a large memory
bank (queue) for the anchor points. The queue size is 128K in our experiments.

Method | ImageNet | CUB200 | CARS196 | Stanford Online Products
---|---|---|---|---
NN | Linear | R@1 | R@2 | R@4 | R@8 | R@1 | R@2 | R@4 | R@8 | R@1 | R@2 | R@4 | R@8
ResNet-18
Sup-IN | 63.0 | 69.8 | 49.3 | 62.0 | 74.4 | 84.7 | 39.4 | 51.4 | 63.7 | 76.0 | 51.9 | 57.4 | 62.2 | 66.6
CompRess* [1] | 53.5 | 62.6 | 25.1 | 35.4 | 46.4 | 58.5 | 25.8 | 34.9 | 45.9 | 58.5 | 47.1 | 52.2 | 57.1 | 61.6
MoCo [14] | 37.7 | 51.0 | 15.8 | 23.2 | 32.8 | 45.3 | 20.0 | 27.6 | 37.2 | 48.8 | 48.0 | 52.7 | 57.3 | 61.6
BYOL [12] | 40.0 | 52.6 | 17.2 | 25.0 | 35.0 | 47.2 | 21.4 | 29.3 | 39.9 | 52.0 | 46.7 | 51.5 | 56.1 | 60.6
ISD | 41.5 | 53.8 | 19.5 | 28.1 | 39.3 | 52.5 | 27.2 | 36.5 | 46.6 | 58.5 | 49.8 | 54.6 | 59.4 | 63.6
ResNet-50
Sup-IN | 71.4 | 76.2 | 50.3 | 64.3 | 76.0 | 86.0 | 43.5 | 55.7 | 67.2 | 78.2 | 53.8 | 59.2 | 64.0 | 68.4
CompRess* [1] | 63.3 | 71.9 | 29.9 | 40.6 | 52.7 | 65.6 | 31.3 | 41.9 | 53.0 | 65.0 | 50.2 | 55.1 | 59.7 | 64.4
MoCo [14] | 57.3 | 71.1 | 22.9 | 32.3 | 43.9 | 56.5 | 26.7 | 35.8 | 46.3 | 57.7 | 52.0 | 57.0 | 61.6 | 66.1
SwAV [7] | 60.7 | 75.3 | 27.8 | 37.8 | 49.0 | 61.0 | 37.5 | 47.4 | 58.3 | 69.8 | 61.3 | 66.5 | 70.9 | 74.8
DCv2 [7] | 62.2 | 75.2 | 30.8 | 41.7 | 53.0 | 64.9 | 41.0 | 51.4 | 62.4 | 73.0 | 61.4 | 66.5 | 70.9 | 74.7
BYOL [12] | 62.8 | 74.3 | 39.8 | 52.7 | 64.6 | 75.4 | 40.4 | 51.2 | 62.4 | 72.7 | 57.3 | 62.6 | 67.3 | 71.3
ISD init BYOL | 63.4 | 74.3 | 45.7 | 58.7 | 70.9 | 81.1 | 42.4 | 53.6 | 63.9 | 74.2 | 57.6 | 62.8 | 67.6 | 71.7

Table 1: We compare our method with other SSL methods on Linear and Nearest
Neighbor (NN) evaluation on ImageNet as well as recall on transfer learning
settings. Our method (ISD) outperforms MoCo and BYOL in most evaluations. *
denotes that CompRess is not directly comparable as it uses a larger SSL model
as the teacher: MoCo-V2(ResNet50) for ResNet18 and SimCLR(ResNet50x4) for
ResNet50.

##  4 Experiments

We describe various experiments and their results in this section. First, we
compare our proposed self-supervised method with other state-of-the-art
methods on ImageNet and transfer learning. Next, we show how the method can be
modified to remove unwanted augmentation biases. Finally, we demonstrate the
advantage of our method compared to MoCo on unbalanced, unlabeled dataset.

Implementation details:  For all experiments, we use PyTorch with SGD
optimizer (momentum = $0.9$, weight decay = $1e-4$, batch size = 256) except
when stated otherwise. Details about the specific architecture, epochs for
training, and learning rate are described for each experiment in its
respective section. We follow the evaluation protocols in [1] for nearest
neighbor (NN) and linear layer (Linear) evaluation. We use the ImageNet labels
only in the setting of evaluating the learned features. To evaluate how SSL
features transfer to new tasks, we perform image retrieval on different
datasets like CUB200, CARS196, and Stanford Online Product. Unlike linear
evaluation, the task of image retrieval does not have any hyper-parameters
which allows faster and more consistent evaluation of the learned features.

Practical issues in recent SSL benchmarks: Although it is common to use
ResNet-50 model for self-supervised learning, it is an expensive model to
train and experiment with. For instance, training ResNet-50 for 200 epochs of
MoCo takes at least 6 days on four RTX 2080-Ti GPUs. Most recent papers [9, 8,
12, 20, 7] report results for 800 epochs or more which is very expensive
without specialized hardware. Further, some methods [12, 7] are even slower as
they forward the mini-batch through the model more than once. For instance,
BYOL method forwards the images twice to calculate the symmetric loss, so
$800$ epochs of BYOL is equivalent to almost $1,600$ epochs of MoCo in terms
of running time. We believe such demanding benchmarks makes this research area
less accessible to smaller research labs and is not aligned with the goals of
democratizing AI.

Moreover, linear evaluation of the SSL features is also somehow an established
benchmark in the community. We believe linear evaluation has even more sever
issues as each linear evaluation takes several hours and has some
hyperparameters. It is possible to spend lots of computation to tune the
hyperparameters which makes the benchmark inconsistent and maybe misleading.
We believe recently emerging evaluation of SSL methods with nearest neighbor
resolves most of these issues as it does not have any hyperparameters and is
very fast.

Method | CUB | CARS | Aircraft | Flowers | iNat
---|---|---|---|---|---
| 200 | 196 | [19] | [22] | [18]
ResNet-18
Sup-IN | 47.2 | 24.0 | 33.0 | 75.4 | 26.3
CompRess* [1] | 22.0 | 14.0 | 25.1 | 67.4 | 18.9
MoCo [14] | 12.0 | 10.8 | 19.3 | 61.2 | 13.0
BYOL [12] | 14.7 | 11.5 | 20.2 | 62.4 | 13.8
ISD | 15.7 | 15.3 | 26.3 | 72.5 | 16.2
ResNet-50
Sup-IN | 49.2 | 26.7 | 31.2 | 72.2 | 28.6
CompRess* [1] | 27.9 | 16.9 | 30.1 | 78.9 | 25.2
MoCo [14] | 20.6 | 13.9 | 22.3 | 72.5 | 21.1
BYOL [12] | 36.6 | 24.0 | 36.1 | 84.4 | 29.9
ISD init BYOL | 42.9 | 25.8 | 38.2 | 86.0 | 31.3

Table 2: We evaluate our models on transfer tasks using nearest neighbor
evaluation. * denotes that CompRess is not directly comparable as it uses a
larger SSL model as the teacher: MoCo-V2(ResNet50) for ResNet18 and
SimCLR(ResNet50x4) for ResNet50.

###  4.1 Self-supervised learning

ResNet-18 experiments: We train a ResNet-18 model from scratch for 200 epochs
using our ISD method. This experiment takes at least 4 days on two RTX 2080-Ti
GPUs. For baselines, we re-implement BYOL in PyTorch from the details provided
in [12] and use the official implementation of MoCo-v2 [14, 9]. For a fair
comparison with our method and MoCo, for BYOL, we use a non-symmetric loss, a
small batch size (256), and train for 200 epochs. We use following details for
our re-implementation of BYOL: momentum for the teacher ($0.99$), hidden units
in the MLP for projection and prediction layers ($1024$), and output embedding
dimension ($128$).

Following CompRess [1], we train our self-supervised ResNet-18 model with the
initial learning rate set to $0.01$ and multiplied by $0.2$ at epochs 140 and
180. We follow [12] and add a prediction layer for the student. The hidden and
output dimensions of the prediction MLP layer are set to $512$. Unlike in
[12], we do not have any projection layer since it increases the complexity of
the model. We use the same set of augmentations used in [9, 8, 12]. The
results are shown in Tables 1 and 2. Our model outperforms both baselines on
ImageNet nearest neighbor (NN) evaluation, ImageNet linear evaluation, and
also transfer learning settings on CUB200, CARS196, and Stanford products
datasets.

It is important to note that our method can be seen as a soft version of MoCo,
so ISD outperforming MoCo, empirically supports our main motivation of
improving representations by smoothing the contrastive learning approaches:
not considering all negatives equally negative.

ResNet-50 experiments: As mentioned above, ResNet-50 is recently used as a
benchmark in the community. Unfortunately, we cannot run the model for 800
epochs to do a fair comparison, so we use a different approach to evaluate our
model on ResNet-50 architecture. We downloaded the official pretrained BYOL
model that is already trained for 800 epochs with symmetric loss. Then, we
initialized our model with this BYOL model (both teacher and student) and
finetuned it using our method on unlabeled data for 10 epochs only. Note that
finetuning usually means using labeled data, but in this case we are just
optimizing our self-supervised method after 800 epochs of BYOL. We train for
10 epochs with initial learning rate set to $0.005$ and multiplied by $0.2$ at
epochs 7 and 9. We realize that a small memory bank of size $1.2K$ works
better in this setting. We suspect since the representations are already good,
the query points have close neighbors from the same category, so a small
memory bank is enough and is even probably better since it is less
constrained. This needs further investigation as future work.

Tables 1 and 2 show the results on both ImageNet and transfer learning
settings. Interestingly, our method outperforms all baselines including BYOL
(even though by a small margin in some cases) in all settings except Stanford
Online Products. As mentioned earlier, we believe our method is more relaxed
compared to BYOL as our method lets the embeddings of augmented images move as
long as their similarity relationship with neighbors has not changed. Hence,
fine-tuning BYOL model using this more relaxed method (ISD) may result in a
model that can generalize better to other tasks. Moreover, Table 3 shows our
results when only limited labels are available in ImageNet dataset. Figure 4
shows random images samples from random clusters where each row corresponds to
a cluster. Note that each row contains semantically similar images.

Note that we have used the pretrained model for DCv2 and SwAV baselines
provided by the authors, so they are using different set of augmentations.

Evolution of teacher and student models: In Figure 3, for every 10 epoch of
ResNet-18, we evaluate both teacher and student models for BYOL, MoCo, and ISD
methods using nearest neighbor. For all methods, the teacher performs usually
better than the student in the initial epochs when the learning rate is small
and then is very close to the student when learning rate shrinks. This is
interesting as we have not seen previous papers comparing the teacher with the
student. This might happen since the teacher is a running average of the
student so can be seen as an ensemble over many student networks similar to
[31]. We believe this deserves more investigation as future work.

Figure 3: Evolution of teacher and student models: Comparing the teacher and
student ResNet18 models using Nearest Neighbor while training for MoCo, BYOL,
and ISD methods. Interestingly, the teacher performs better than the student
before shrinking the learning rate. Most previous works use the student as the
final model which seems to be sub-optimal. We believe this is due to
ensembling effect similar to [31] and needs more investigation.

Ablation study: We varied the temperature for our method on ResNet-18 with 130
epochs and reported the results in Table 4. Here, $LR=0.01$ and it is
multiplied by $0.2$ at 90 and 120 epochs. Also, for more fair comparison with
BYOL on ResNet18, we varied the learning rate and chose the best one for BYOL.
Table 5 shows the results of this experiment.

Method | Top-1 | Top-5
---|---|---
1% | 10% | 1% | 10%
Supervised | 25.4 | 56.4 | 48.4 | 80.4
InstDisc [34] | - | - | 39.2 | 77.4
PIRL [20] | - | - | 57.2 | 83.8
SimCLR [8] | 48.3 | 65.6 | 75.5 | 87.8
BYOL [12] | 53.2 | 68.8 | 78.4 | 89.0
SwAV [7] | 53.9 | 70.2 | 78.5 | 89.9
Only the linear layer is trained.
BYOL | 55.7 | 68.6 | 80.0 | 88.6
ISD init BYOL | 56.0 | 68.3 | 79.9 | 88.5
CompRess* [1] | 59.7 | 67.0 | 82.3 | 87.5
Table 3: Our ResNet-50 model when evaluated on smaller subset of ImageNet using a linear. Our linear evaluation of BYOL is better than the finetuning numbers in the original paper, so we report both.  $\tau$ | 0.003 | 0.007 | 0.01 | 0.02 | 0.04 | 0.06
---|---|---|---|---|---|---
NN | 37.2 | 37.8 | 37.7 | 39.7 | 35.3 | 32.5
Table 4: Effect of changing temperature for our method ISD on ResNet-18 model. LR | 0.01 | 0.05 | 0.10 | 0.20
---|---|---|---|---
NN | 37.3 | 40.0 | 38.6 | 37.3
Table 5: Comparison of learning rates for BYOL for ResNet-18. The model is
trained for 200 epochs with cosine learning rate decay.

###  4.2 Removing the bias of aggressive augmentations

Most recent self-supervised learning methods work by pulling embeddings of two
different augmentations of an image close to each other. It is shown that
using aggressive augmentations improves the performance probably by
introducing more variation between the pair of augmented images. Hence, using
aggressive augmentation is standard in the new SSL methods. However, the
augmented images do not look natural anymore, so the SSL model is trained on
non-natural images. We argue that this bias may reduce the quality of the
features in the final tasks where the images will be natural. Figure 2 shows
some random augmentations that MoCo-v1 is trained on. The later works (e.g.,
SimCLR, MoCo-v2, etc) use even more aggressive augmentations. We realize that
one can use a variation of our method to remove this unnatural bias from a SSL
model that is already trained with aggressive augmentations.

Our ISD method performs similarity-based distillation iteratively where the
student is optimized by distilling the teacher and the teacher is updated as a
moving average of the student. One can initialize the teacher from an already
trained model and set the momentum to $1$, which will ignore updating the
teacher. Such a method can distill the teacher into a student model that is
initialized from scratch. Note that this method is closely related to CompRess
[1] except that we are using it for self-distillation instead of compressing
from a deep model to a shallow one.

We use a SSL model already trained with aggressive augmentations and then
distill it using this variation of our method on a dataset that includes only
standard, mild augmentations. We believe the final model should perform better
in natural images as it has never seen unnaturally augmented images, so has
not learned the unwanted bias.

We do this experiment using ResNet50 trained with MoCo-v2 method and distill
it to another ResNet50 that is initialized from scratch. We use ImageNet
dataset with mild augmentations that are standard in training vanilla
supervised learning models with cross entropy loss. Table 6 shows the results.
Interestingly, our method is on par with the MoCo model on Linear evaluation,
but outperforms the teacher on ImageNet NN evaluation and also transfer
learning settings by a large margin. We believe this empirically demonstrates
that removing the augmentation bias helps SSL methods like MoCo.

Figure 4: Random Clusters: We cluster ImageNet dataset into 1000 clusters
using k-means and show random samples from random clusters. Each row
corresponds to a cluster. Note that semantically similar images are clustered
together.

| ImageNet | iNat | CUB-200 | CARS-196 | Aircraft | Flowers
---|---|---|---|---|---|---
Method | NN | Linear | NN | Linear | NN | Linear | NN | Linear | NN | Linear | NN | Linear
MoCo-R50 | 57.3 | 71.1 | 21.1 | 49.5 | 20.6 | 51.2 | 13.9 | 48.1 | 22.3 | 49.4 | 72.2 | 90.3
Ours (from MoCo-R50) | 60.1 | 71.1 | 27.4 | 52.7 | 30.3 | 61.5 | 18.9 | 55.2 | 31.6 | 54.5 | 83.6 | 93.3

Table 6: We use a variation of our method to remove the bias of aggressive augmentations that most recent SSL methods are trained with. We believe the student model performs better than the teacher on transfer learning settings since it has learned less unnatural biases. We use Linear evaluation and NN for evaluation. Method | $D_{1}$ | $D_{2}$ | $D_{3}$ | $D_{4}$ | $D_{5}$ | $D_{6}$ | $D_{7}$ | $D_{8}$ | $D_{9}$ | $D_{10}$
---|---|---|---|---|---|---|---|---|---|---
Evaluation On All 38 Categories
MoCo | 48.9 | 57.8 | 59.4 | 56.7 | 62.0 | 54.6 | 57.8 | 57.2 | 62.4 | 54.4
ISD | 49.6 | 57.9 | 61.9 | 58.6 | 62.3 | 56.3 | 57.8 | 58.1 | 62.5 | 55.3
Diff | +0.7 | +0.1 | +2.5 | +1.9 | +0.3 | +1.7 | 0 | +0.9 | +0.1 | +0.9
Evaluation Only on 30 Rare Categories
MoCo | 44.7 | 52.3 | 57.3 | 53.1 | 57.7 | 50.7 | 51.1 | 51.9 | 58.9 | 59.8
ISD | 46.5 | 53.9 | 60.8 | 56.8 | 60.5 | 54.5 | 53.1 | 55.0 | 60.7 | 51.5
Diff | +1.7 | +1.6 | +3.5 | +3.7 | +2.8 | +3.8 | +2.0 | +3.1 | +1.8 | +1.7
Table 7: Nearest Neighbor (NN) results for the unbalanced data when we
consider all 38 categories and 30 small categories separately. We repeat the
experiment 10 times with different random sets of 38 categories. NN is done on
the validation set of ImageNet (which has uniform distribution) by searching
the nearest neighbors among all ImageNet training data of those 38 categories
(so the training data of NN also has uniform distribution). Hence, the whole
evaluation is on balanced data to make sure we observe the effect of the
unbalanced, “unlabeled” data only. “Diff” shows the improvement of our method
over MoCo. Interestingly the improvement is bigger in the rare categories.
This is aligned with out hypothesis that our method can handle unbalanced,
unlabeled data better since it does not consider all negative images equally
negative.

###  4.3 Self-Supervised Learning on Unbalanced Dataset

Most recent self-supervised learning methods are benchmarked by training on
unlabeled ImageNet. However, we know that ImageNet has a particular bias of
having an almost uniform distribution over the number of samples per category.
We believe this bias does not exist in many real-world applications where the
data is unbalanced: a few categories have a large number of samples while the
rest of the data have a small number of samples. For instance, in self driving
car applications, it is really important to learn features for understanding
rare scenes while most of the data is captured from repetitive safe highway
scenes. Hence, we believe it is important to design and evaluate self-
supervised learning methods for such unbalanced data.

As mentioned earlier, since standard contrastive learning methods e.g., MoCo,
consider all negative examples equally negative, when the query image is from
a large category, it is possible to have multiple samples from the same
category in the memory bank. Then, the contrastive loss in MoCo pushes their
embeddings to be far apart as negative pairs. However, our method can handle
such cases since our teacher assigns a soft label to the negative samples, so
if an anchor example is very similar to the query, it will have high
similarity and the student is optimized to reproduce such similarity.

To study our method on unbalanced data, we design a controlled setting to
introduce the unbalanced data in the SSL training only and factor out its
effect in the feature evaluation step. Hence, we sub-sample ImageNet data with
38 random categories where 8 categories are large (use all of almost 1300
images per category) and 30 categories are small (use only 100 images per
category.) We train our SSL method and then evaluate by nearest neighbor (NN)
classifier on the balanced validation data. To make sure that the feature
evaluation is not affected by the unbalanced data, we keep both evaluation and
the training data of NN search balanced, so for NN search, we use all ImageNet
training images (almost $1300\times 38$ images) for those 38 categories.

We repeat the sampling of 38 categories 10 times to come up with 10 datasets
and report the results for our method and also MoCo in Table 7. To measure the
effect of the unbalanced data. we report the accuracy on all 38 categories and
also on those 30 small categories only separately. Our method performs
consistently better than MoCo, but more interestingly, the gap the improvement
is larger when we evaluate on the 30 small categories only. We believe this
empirically proves our hypothesis that our method may be able to handle
unbalanced data more effectively. For a fair comparison, we train both our
model and MoCo for 400 epochs with a memory bank size of $8192$ and cosine
learning schedule.

##  5 Conclusion

We introduce ISD, a novel self-supervised learning method. It is a variation
of contrastive learning (e.g., MoCo) in which negative samples are not all
treated equally. Our extensive experiments show that our method performs
better than state-of-the-art SSL methods on ImageNet, transfer learning tasks,
and when the unlabeled data is unbalanced. We also show that our method
improves when initialized by an already trained BYOL model.

Acknowledgment: This material is based upon work partially supported by the
United States Air Force under Contract No. FA8750‐19‐C‐0098, funding from SAP
SE, and also NSF grant number 1845216. Any opinions, findings, and conclusions
or recommendations expressed in this material are those of the authors and do
not necessarily reflect the views of the United States Air Force, DARPA, or
other funding agencies.

## References

  * [1] Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash.  Compress: Self-supervised learning by compressing representations.  Advances in Neural Information Processing Systems, 33, 2020.
  * [2] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai.  Variational information distillation for knowledge transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9163–9171, 2019.
  * [3] Jimmy Ba and Rich Caruana.  Do deep nets really need to be deep?  In Advances in neural information processing systems, pages 2654–2662, 2014.
  * [4] Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  In Advances in Neural Information Processing Systems, pages 15535–15545, 2019.
  * [5] Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi.  Label refinery: Improving imagenet classification through label progression.  arXiv preprint arXiv:1805.02641, 2018.
  * [6] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149, 2018.
  * [7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [9] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [10] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.  Born-again neural networks.  In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1602–1611. PMLR, 2018\.
  * [11] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In International Conference on Learning Representations, 2018.
  * [12] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  arXiv preprint arXiv:2006.07733, 2020.
  * [13] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE, 2006.
  * [14] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [15] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord.  Data-efficient image recognition with contrastive predictive coding.  arXiv preprint arXiv:1905.09272, 2019.
  * [16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [17] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization.  In International Conference on Learning Representations, 2019.
  * [18] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset, 2018.
  * [19] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.  Fine-grained visual classification of aircraft.  Technical report, 2013.
  * [20] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  arXiv preprint arXiv:1912.01991, 2019.
  * [21] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  IEEE transactions on pattern analysis and machine intelligence, 41(8):1979–1993, 2018.
  * [22] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
  * [23] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European Conference on Computer Vision, pages 69–84. Springer, 2016.
  * [24] Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro.  Representation learning by learning to count.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5898–5906, 2017.
  * [25] Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash.  Boosting self-supervised learning via knowledge transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9359–9367, 2018.
  * [26] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019.
  * [27] Nikolaos Passalis and Anastasios Tefas.  Learning deep representations with probabilistic knowledge transfer.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 268–284, 2018.
  * [28] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [29] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang.  Correlation congruence for knowledge distillation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5007–5016, 2019.
  * [30] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [31] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In Advances in neural information processing systems, pages 1195–1204, 2017.
  * [32] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation.  In International Conference on Learning Representations, 2020.
  * [33] Frederick Tung and Greg Mori.  Similarity-preserving knowledge distillation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1365–1374, 2019.
  * [34] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
  * [35] Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan.  Clusterfit: Improving generalization of visual representations.  In CVPR, 2020.
  * [36] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133–4141, 2017.
  * [37] Asano YM., Rupprecht C., and Vedaldi A.  Self-labelling via simultaneous clustering and representation learning.  In International Conference on Learning Representations, 2020.
  * [38] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  In ICLR, 2017.
  * [39] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer, 2016.
  * [40] Richard Zhang, Phillip Isola, and Alexei A Efros.  Split-brain autoencoders: Unsupervised learning by cross-channel prediction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058–1067, 2017.