2021

[1]Aniket Anand Deshmukh

Work was done while author was at Microsoft

[1] Microsoft, Mountain View, 94043, CA, USA

2] The Ohio State University, Columbus, 43210, OH, USA

# Representation Learning for Clustering via Building Consensus

<EMAIL_ADDRESS>Jayanth Reddy Regatti<EMAIL_ADDRESS>Eren Manavoglu
<EMAIL_ADDRESS>Urun Dogan<EMAIL_ADDRESS>*  [

###### Abstract

In this paper, we focus on unsupervised representation learning for clustering
of images. Recent advances in deep clustering and unsupervised representation
learning are based on the idea that different views of an input image
(generated through data augmentation techniques) must be close in the
representation space (exemplar consistency), and/or similar images must have
similar cluster assignments (population consistency). We define an additional
notion of consistency, consensus consistency, which ensures that
representations are learned to induce similar partitions for variations in the
representation space, different clustering algorithms or different
initializations of a single clustering algorithm. We define a clustering loss
by executing variations in the representation space and seamlessly integrate
all three consistencies (consensus, exemplar and population) into an end-to-
end learning framework. The proposed algorithm, consensus clustering using
unsupervised representation learning (ConCURL), improves upon the clustering
performance of state-of-the-art methods on four out of five image datasets.
Furthermore, we extend the evaluation procedure for clustering to reflect the
challenges encountered in real-world clustering tasks, such as maintaining
clustering performance in cases with distribution shifts. We also perform a
detailed ablation study for a deeper understanding of the proposed algorithm.
The code and the trained models are available at
https://github.com/JayanthRR/ConCURL_NCE.

###### keywords:

Unsupervised Learning, Clustering, Representation Learning, Consensus
Clustering

##  1 Introduction

The field of artificial intelligence (AI) advanced significantly in the
previous decade due to developments in deep learning (LeCun ., 2015). In the
early years of this field, deep learning methods exhibited stellar supervised
learning performance, where each data sample was coupled with a ground truth
(labeled data), e.g., each image was associated with a category.
Unfortunately, generating labeled datasets is time consuming and expensive,
and there may not be enough experts to label the data at hand (e.g., medical
images). The straightforward solution is to use clustering for large-scale
problems.

In this work, we focus on representation learning for the unsupervised
learning task of clustering images. Clustering is a ubiquitous task and has
been actively used in many different scientific and practical pursuits (Frey
Dueck, 2007; Masulli Schenone, 1999; Jain ., 1999; Xu Wunsch, 2005).
Clustering algorithms do not learn representations and are hence limited to
data for which we have a good representation available.

Advancements in deep learning techniques have enabled the end-to-end learning
of rich image representations for supervised learning. For the purposes of
clustering, however, such features learned via supervised learning cannot be
obtained due to lack of available labels. Therefore, supervised learning
approaches fall short of providing a solution. Self-supervised learning
addresses the issue of learning representations without labeled data. Self-
supervised learning is a subfield of unsupervised learning in which the main
goal is to learn general-purpose representations by exploiting user-defined
tasks (pretext tasks) (Z. Wu ., 2018; Zhuang ., 2019; He ., 2020; Zhuang .,
2019; Chen ., 2020; Grill ., 2020; Caron ., 2020). Representation learning
algorithms have been shown to achieve good results when evaluated using a
linear evaluation protocol, semisupervised training on ImageNet, or transfer
to downstream tasks. A straightforward solution to the clustering problem is
to use the features obtained via self-supervised learning and apply an out-of-
the-box clustering algorithm (such as k-means) to compute data clusters.
However, the performance of these features for clustering (using an out-of-
the-box clustering algorithm) is not known, and as seen in our results, these
features may be improved for clustering purposes.

On the other hand, deep clustering involves simultaneously learning cluster
assignments and features using deep neural networks. Simultaneously learning
the feature spaces with a clustering objective in deep clustering may lead to
degenerate solutions, which until recently limited end-to-end implementations
of clustering with representation learning approaches (Caron ., 20181).
Subsequently, several works have been developed (Xie ., 20161; Caron ., 20181;
Shah Koltun, 2018; Ji ., 20191; Niu ., 20201; J. Wu ., 20191; Huang ., 20201;
Tao ., 2021). We provide details regarding some of these works in Section 1.1.
Our previous work shows some encouraging results but we extend the work
substantially (Regatti ., 2021). We categorize the current clustering and
representation learning works based on the consistency constraints that are
used to define their objective functions. We define an additional notion of
consistency, consensus consistency, which ensures that representations are
learned to induce similar partitions for variations in the representation
space, different clustering algorithms or different initializations of a
clustering algorithm. We use consensus consistency and propose an end-to-end
learning approach that outperforms other end-to-end learning methods for image
clustering. We summarize our contributions as follows:

  1. 1.

We introduce different notions of consistency (exemplar, population and
consensus) that are used in unsupervised representation learning.

  2. 2.

We propose a novel clustering algorithm that incorporates the above three
consistency constraints and can be trained in an end-to-end way. An ensemble
is generated in the consensus clustering objective by performing random
transformations on the underlying embeddings. We combine several methods,
which is not trivial, and this combination, along with our new consensus loss,
is novel.

  3. 3.

We show that the proposed algorithm ConCURL (consensus clustering with
unsupervised representation learning) outperforms baselines on popularly used
computer vision datasets when evaluated with clustering metrics.

  4. 4.

We demonstrate the clustering abilities of trained models under a data shift
and argue for the need for different evaluation metrics for deep clustering
algorithms.

  5. 5.

We study the impacts of various hyperparameters, data augmentation methods,
and image resolutions on the clustering ability of the proposed algorithm.

###  1.1 Related Work

Self-Supervised Learning: Self-supervised learning is used to learn
representations in an unsupervised way by defining some pretext tasks. There
are many different flavors of self-supervised learning, such as instance
discrimination (ID) tasks (Z. Wu ., 2018; Zhuang ., 2019) and contrastive
techniques (He ., 2020; Chen ., 2020). In ID tasks, each image is considered
its own category so that the learned embeddings are well separated (Z. Wu .,
2018). Building on the ID task, Zhuang . (2019) proposed a local aggregation
(LA) method based on a robust clustering objective (using multiple runs of
k-means) to move statistically similar data points closer in the
representation space and dissimilar data points further away. In contrastive
techniques such as simple contrastive learning of visual representations
(SimCLR) (Chen ., 2020) and momentum contrast (MoCo) (He ., 2019),
representations are learned by maximizing the agreement between different the
augmented views of the same data example (known as positive pairs) and
minimizing the agreement between the augmented views of different examples
(known as negative pairs). Recent works, including Bootstrap Your Own Latent
(BYOL) (Grill ., 2020) and swapping assignments between multiple views (SwAV)
(Caron ., 2020), have achieved state-of-the-art results without requiring
negative pairs. Although self-supervised learning methods exhibit impressive
performance on a variety of problems, it is not clear whether learned
representations are good for clustering.

Clustering with Representation Learning:

DEC (Xie ., 20161) is one of the first algorithms to show that deep learning
can be used to effectively cluster images in an unsupervised manner; this
approach uses features learned from an autoencoder to fine-tune the cluster
assignments. DeepCluster (Caron ., 20181) shows that it is possible to train
deep convolutional neural networks (DeCNNs) in an end-to-end manner with
pseudolabels that are generated by a clustering algorithm. Subsequently,
several works (Shah Koltun, 2018; Ji ., 20191; Niu ., 20201; J. Wu ., 20191;
Huang ., 20201) have introduced end-to-end clustering-based objectives and
achieved state-of-the-art clustering results. For example, in the Gaussian
attention network for image clustering (GATCluster) (Niu ., 20201), training
is performed in two distinct steps (similar to Caron . (20181)), where the
first step is to compute pseudotargets for a large batch of data and the
second step is to train the model in a supervised way using these
pseudotargets. Both DeepCluster and GATCluster use k-means to generate
pseudolabels that may not scale well. J. Wu . (20191) proposed deep
comprehensive correlation mining (DCCM), where discriminative features are
learned by taking advantage of the correlations among the data using
pseudolabel supervision and the triplet mutual information among the features.
However, DCCM may be susceptible to trivial solutions (Niu ., 20201).
Invariant information clustering (IIC) (Ji ., 20191) maximizes the mutual
information between the class assignments of two different views of the same
image (paired samples) to learn representations that preserve the
commonalities between the views while discarding instance-specific details. It
has been argued that the presence of an entropy term in mutual information
plays an important role in avoiding degenerate solutions. However, a large
batch size is needed for the computation of mutual information in IIC; this
process may not be scalable for larger image sizes, which are common in
popular datasets (Ji ., 20191; Niu ., 20201). Huang . (20201) extended the
celebrated maximal margin clustering idea to the deep learning paradigm by
learning the most semantically plausible clusters through the minimization of
a proposed partition uncertainty index. Their pixel intensity clustering
(PICA) algorithm uses a stochastic version of this index, thereby facilitating
minibatch training. PICA fails to assign a sample-correct cluster when that
sample has either high foreground or background similarity to samples in other
clusters. In a more recent approach, contrastive clustering (Li ., 2021), a
contrastive learning loss (as in SimCLR (Chen ., 2020)) was adopted along with
an entropy term to avoid degenerate solutions. Similarly, Tao . (2021)
combined ID (Z. Wu ., 2018) with novel softmax-formulated decorrelation
constraints for representation learning and clustering. Their approach
outperforms state-of-the-art methods and improves upon the instance
discrimination method. Our method also improves upon ID and outperforms the
method of Tao . (2021) on all datasets considered. There are other non-end-to-
end approaches, such as SCAN (Van Gansbeke ., 2020), which use the learned
representations from a pretext task to find the images that are semantically
closest to the given image using the nearest neighbors algorithm. Similarly,
one more state of the art non-end-to-end approach, SPICE (Niu ., 2021) divides
the clustering network in two parts - one to measure instance level similarity
and one to identify cluster level discrepancy.

##  2 Consensus Clustering

One of the distinguishing factors between supervised learning and unsupervised
learning is the existence of ground truth labels that construct a global
constraint based on examples. In most self-supervised learning methods, the
ground truth is replaced with some consistency constraint (Chen ., 2020).
Without a doubt, the performance of any self-supervised method is a function
of the power of the consistency constraint used. We define two types of
consistency constraints: exemplar consistency and population consistency.

######  Definition 1.

Exemplar consistency: Representation learning algorithms that learn closer
representations (in terms of some distance metric) for different augmentations
of the same data point are said to follow exemplar consistency.

Examples of the usage of exemplar consistency include contrastive learning
methods such as MoCo (He ., 2019) and SimCLR (Chen ., 2020). In these methods,
a positive pair of images is defined as any two image augmentations of the
same image, and a negative pair consists of any two different images.

######  Definition 2.

Population consistency: Representation learning algorithms that ensure that
learned representations satisfy the consistency constraint, where two similar
data points or any augmentations of the same data points should belong to the
same cluster (or population), are said to follow population consistency.

Deep Cluster (Caron ., 20181) is a prominent self-supervised method that
utilizes population consistency, i.e., Definition 2, by enforcing a clustering
constraint on the input dataset. Please note that each cluster assignment
contains data points that are similar to each other. Similarly, SwAV (Caron .,
2020) is an example of the population consistency method.

######  Definition 3.

Consensus consistency: Representation learning algorithms that are able to
learn representations that induce similar partitions for variations in the
given representation space (subsets of features, random projections, etc. ),
different clustering algorithms (k-means, Gaussian mixture models (GMMs),
etc.) or different initializations of clustering algorithms are said to follow
consensus consistency.

Earlier works on consensus consistency did not consider representation
learning and used the knowledge reuse framework (see Strehl Ghosh (2002),Ghosh
Acharya (2011)), where the cluster partitions were available (the features
were irrelevant) or the features of the data were fixed. For example, Fern
Brodley (2003) successfully applied random projections to consensus clustering
by performing k-means clustering on multiple random projections of the fixed
features of input data. In contrast, the notion of consensus consistency here
deals with learning representations that achieve a consensus regarding the
cluster assignments of multiple clustering algorithms. One example of a method
that enforces consensus consistency is LA (Zhuang ., 2019). LA builds on the
ID task (Z. Wu ., 2018) and was proposed as a method based on a robust
clustering objective (using multiple runs of k-means) to move statistically
similar data points closer in the representation space and dissimilar data
points further away. However, Zhuang . (2019) did not evaluate the method with
clustering metrics and only focused on linear evaluation using the learned
features. Subsequently, we conducted a study to evaluate the clustering
performance of these features (see Appendix) and observed that LA performed
poorly when evaluated for clustering accuracy. In Definition 3, we inherently
assume that the clustering algorithms under consideration have been tuned
properly. Unfortunately, the definition of consensus consistency is ill posed,
and there can be arbitrarily many different partitions that can satisfy the
given condition111a) Degenerate solution where all cluster assignments are the
same. b) Random assignment can satisfy this condition given that all
clustering process produces the same but random assignments. We show that when
exemplar consistency is used as an inductive bias, the resulting objective
function achieves impressive performance on challenging datasets. Combining
the exemplar and population constraints with consensus consistency seamlessly
and effectively for clustering is the basis of our proposed method.

###  2.1 Loss for Consensus and Population Consistency

We focus on learning generic representations that satisfy Definition 3 for
clustering. By using different clustering algorithms or different
representation variations (such as projections), one can easily generate
multiple different partitions of the same data. In unsupervised learning, it
is not known which partitioning is correct. To tackle this problem, some
additional assumptions are needed.

We assume that there is an underlying latent space $\mathcal{Z}^{*}$ (possibly
not unique) such that all clusterings (based on latent space, algorithm or
initialization variations) that take input data from this latent space produce
similar data partitions. Furthermore, every clustering algorithm that also
takes the true number of clusters as input produces the partition that is
closest to the hypothetical ground truth. Moreover, we assume that there
exists a function $h:X\rightarrow\mathcal{Z}^{*}$, where X represents the
input space and $\mathcal{Z}^{*}$ represents the underlying latent space. We
call this assumption the principle of consensus. The open question is how one
constructs an efficient loss that reflects the principle of consensus. We
define one such way below.

Given an input batch of images $\mathcal{X}_{b}\subset\mathcal{X}$, the goal
is to partition these images into $K$ clusters. We obtain $p$ views of these
images (by different image augmentation approaches) and define a loss such
that cluster assignment of any of the $p$ views matches the target estimated
from any other view. Without loss of generality, we define a loss for $p=2$
views. The two views $\mathcal{X}_{b}^{1},\mathcal{X}_{b}^{2}$ are generated
using two randomly chosen image augmentations.

We learn a representation space $\mathcal{Z}_{0}$ at the end of every training
iteration and obtain $M$ variations of $\mathcal{Z}_{0}$ as
$\\{\mathcal{Z}_{1},\mathcal{Z}_{2},...,\mathcal{Z}_{M}\\}$ (e.g., random
projections). The goal is to build an efficient loss according to the
principle of consensus among $\mathcal{Z}_{0}$ and its $M$ variations
$\\{\mathcal{Z}_{1},\mathcal{Z}_{2},...,\mathcal{Z}_{M}\\}$ such that we learn
the latent space $\mathcal{Z}^{*}$ at the end of training (i.e., the learned
features lie in the latent space described above). For a given batch of images
$\mathcal{X}_{b}$ and a representation space $\mathcal{Z}_{m},\forall
m\in[1,...,M]$, we denote the cluster assignment probability of image $i$ and
cluster $j$ for view $1$ as $\mathbf{p}_{i,j}^{1}(\mathcal{Z}_{m})$ and that
for view 2 as $\mathbf{p}_{i,j}^{2}(\mathcal{Z}_{m})$. We concisely use
$\tilde{\mathbf{p}}^{(1,m)},\tilde{\mathbf{p}}^{(2,m)}$ when we talk about all
the images and all the clusters. Here, we define a loss that incorporates
“population consistency” and “consensus consistency”. We assume that the
target cluster assignment probabilities for the representation
$\mathcal{Z}_{0}$ are given (as in DeepCluster (Caron ., 20181)), and they are
denoted as $\mathbf{q}_{i,j}^{1}$ for view 1 and $\mathbf{q}_{i,j}^{2}$ for
view 2.

We define the loss for any representation space $\mathcal{Z}$ and batch of
images $\mathcal{X}_{b}$ as

$\displaystyle\begin{split}L_{\mathcal{Z}_{m}}^{1}&=-\frac{1}{2B}\sum_{i=1}^{B}\sum_{j=1}^{K}\mathbf{q}^{2}_{ij}\log\mathbf{p}^{1}_{ij}(\mathcal{Z}_{m})\\\
L_{\mathcal{Z}_{m}}^{2}&=-\frac{1}{2B}\sum_{i=1}^{B}\sum_{j=1}^{K}\mathbf{q}^{1}_{ij}\log\mathbf{p}^{2}_{ij}(\mathcal{Z}_{m}),\\\
L_{\mathcal{Z}}&=\sum_{m=1}^{M}\Big{(}L_{\mathcal{Z}_{m}}^{1}+L_{\mathcal{Z}_{m}}^{2}\Big{)}.\end{split}$
(1)

Note that here, consensus among the clustering results is defined via the
number of common targets $\mathbf{q}$. An overview of the procedure is shown
in Figure 1. The exact details regarding how to obtain variations of
$\mathcal{Z}_{0}$ and calculate the cluster assignment probabilities
$\mathbf{p}$ and targets $\mathbf{q}$ are described in the next section.

Figure 1: An illustration of the consensus loss part of ConCURL

###  2.2 End-to-End Stochastic Gradient Descent (SGD)-Based Trainable
Consensus Loss

In this section, we propose an end-to-end trainable algorithm and define a way
to compute $\mathbf{p}$ and $\mathbf{q}$. When the cluster assignment
probabilities $\mathbf{p}$ can take any values in the set $[0,1]$, we refer to
the process as soft clustering, and when $\mathbf{p}$ is restricted to the set
$\\{0,1\\}$, we refer to the process as hard clustering.

Without loss of generality, in this paper, we focus on soft clustering, which
makes it easier to define a loss function using the probabilities and update
the parameters using the gradients to enable end-to-end learning. We follow
the soft clustering framework presented in SwAV (Caron ., 2020), which is a
centroid-based technique that aims to maintain consistency between the
clusterings of the augmented views $\mathcal{X}_{b}^{1}$ and
$\mathcal{X}_{b}^{2}$. We store a set of randomly initialized prototypes
$C_{0}=\\{\mathbf{c}_{0}^{1},\cdots,\mathbf{c}_{0}^{K}\\}\in\mathbb{R}^{d\times
K}$, where $K$ is the number of clusters and $d$ is the dimensionality of the
prototypes. These prototypes are used to represent clusters and define a
“consensus consistency” loss. We compute $M$ variations of $C_{0}$ as
$C_{1},...,C_{M}$ exactly as we compute the $M$ variations of
$\mathcal{Z}_{0}$.

####  2.2.1 Cluster assignment probability $\mathbf{p}$

We use a two-layer multilayer perceptron (MLP) $g$ to project the features
$\mathbf{f}^{1}=f_{\theta}(\mathcal{X}_{b}^{1})$ and
$\mathbf{f}^{2}=f_{\theta}(\mathcal{X}_{b}^{2})$ to a lower-dimensional space
$\mathcal{Z}_{0}$ (of size $d$). The outputs of this MLP (referred to as
cluster embeddings) are denoted as
${Z}_{0}^{1}=\\{\mathbf{z}_{0}^{1,1},\ldots,\mathbf{z}_{0}^{1,B}\\}$ and
${Z}_{0}^{2}=\\{\mathbf{z}_{0}^{2,1},\ldots,\mathbf{z}_{0}^{2,B}\\}$ for view
$1$ and view $2$, respectively. Note that
$h:\mathcal{X}\rightarrow\mathcal{Z}$ defined in 2.1 is equivalent to the
composite function of $f:\mathcal{X}\rightarrow\Phi$ and
$g:\Phi\rightarrow\mathcal{Z}$.

For a latent space $\mathcal{Z}$, we compute the probability of assigning a
cluster $j$ to image $i$ using the normalized vectors
$\bar{\mathbf{z}}^{1,i}=\frac{\mathbf{z}^{1,i}}{\|\mathbf{z}^{1,i}\|}$,
$\bar{\mathbf{z}}^{2,i}=\frac{\mathbf{z}^{2,i}}{\|\mathbf{z}^{2,i}\|}$ and
$\bar{\mathbf{c}}_{j}=\frac{\mathbf{c^{j}}}{\|\mathbf{c^{j}}\|}$ as

$\displaystyle\begin{split}\mathbf{p}^{1}_{i,j}(\mathcal{Z},C)&=\frac{\exp(\frac{1}{\tau}\langle\mathbf{\bar{z}}_{i}^{1},\mathbf{\bar{c}}_{j}\rangle)}{\sum_{j^{{}^{\prime}}}\exp(\frac{1}{\tau}\langle\mathbf{\bar{z}}^{1}_{i},\mathbf{\bar{c}}_{j^{{}^{\prime}}}\rangle)},\quad\\\
\mathbf{p}^{2}_{i,j}(\mathcal{Z},C)&=\frac{\exp(\frac{1}{\tau}\langle\mathbf{\bar{z}}_{i}^{2},\mathbf{\bar{c}}_{j}\rangle)}{\sum_{j^{{}^{\prime}}}\exp(\frac{1}{\tau}\langle\mathbf{\bar{z}}^{2}_{i},\mathbf{\bar{c}}_{j^{{}^{\prime}}}\rangle)}.\end{split}$
(2)

We concisely write
$\mathbf{p}^{1}_{i}(\mathcal{Z})=\\{\mathbf{p}^{1}_{i,j}(\mathcal{Z},C)\\}_{j=1}^{K}$
and $\mathbf{p}^{2}_{i}=\\{\mathbf{p}^{2}_{i,j}(\mathcal{Z},C)\\}_{j=1}^{K}$.
Here, $\tau$ is a temperature parameter, and we set its value to $0.1$,
similar to Caron . (2020). Note that we use $\mathbf{p}_{i}$ to denote the
predicted cluster assignment probabilities for image $i$ (when not referring
to a particular view), and the shorthand notation $\mathbf{p}$ is used when
$i$ is clear from context.

####  2.2.2 Targets $\mathbf{q}$

The idea of predicting the assignments $\mathbf{p}$ and then comparing them
with the high-confidence estimates $\mathbf{q}$ (referred to as codes
henceforth) of the predictions was proposed by Xie . (20161). While Xie .
(20161) used pretrained features (from autoencoders) to compute the predicted
assignments and the codes, the use of their approach in an end-to-end
unsupervised manner might lead to degenerate solutions. Asano . (2019) avoided
such degenerate solutions by enforcing an equipartition constraint (the
prototypes equally partitioned the data) during code computation using the
Sinkhorn-Knopp algorithm (Cuturi, 2013). Caron . (2020) followed a similar
formulation but computed the codes for the two views separately in an online
manner for each minibatch. The assignment codes are computed by solving the
following optimization problem:

$\displaystyle\begin{split}Q^{1}&=\operatorname*{arg\,max}_{Q\in\mathcal{Q}}\text{Tr}(Q^{T}C_{0}^{T}Z_{0}^{1})+\epsilon
H(Q)\\\
Q^{2}&=\operatorname*{arg\,max}_{Q\in\mathcal{Q}}\text{Tr}(Q^{T}C_{0}^{T}Z_{0}^{2})+\epsilon
H(Q),\end{split}$ (3)

where $Q=\\{\mathbf{q}_{1},\ldots,\mathbf{q}_{B}\\}\in\mathbb{R}_{+}^{K\times
B}$, $\mathcal{Q}$ is the transportation polytope defined by

$\mathcal{Q}=\\{\mathbf{Q}\in\mathbb{R}^{K\times
B}_{+}~{}\text{s.t}~{}\mathbf{Q}\mathbf{1}_{B}=\frac{1}{K}\mathbf{1}_{K},\mathbf{Q}^{T}\mathbf{1}_{K}=\frac{1}{B}\mathbf{1}_{B}\\}$

$\mathbf{1}_{K}$ is a vector of ones of dimension $K$ and
$H(Q)=-\sum_{i,j}Q_{i,j}\log Q_{i,j}$. The above optimization is computed
using a fast version of the Sinkhorn-Knopp algorithm (Cuturi, 2013), as
described by Caron . (2020).

After computing the codes $Q^{1}$ and $Q^{2}$, to maintain the consistency
between the clustering results of the augmented views, the loss is computed
using the probabilities $\mathbf{p}_{ij}$ and the assigned codes
$\mathbf{q}_{ij}$ by comparing the probabilities of view $1$ with the assigned
codes of view $2$ and vice versa, as in equation 1.

####  2.2.3 Defining variations of $Z_{0}$ and $C_{0}$

To compute $\\{Z_{1},...,Z_{M}\\}$, we project the $d$-dimensional space
$Z_{0}$ to a $D$-dimensional space using a random projection matrix. We follow
the same procedure to compute $\\{C_{1},...,C_{M}\\}$ from $C_{0}$. At the
beginning of the algorithm, we randomly initialize $M$ such transformations
and fix them throughout training. Suppose that by using a particular random
transformation (a randomly generated matrix $A$), we obtain
$\tilde{\mathbf{z}}=A\mathbf{z},\;\tilde{\mathbf{c}}=A\mathbf{c}$. We then
compute the softmax probabilities using the normalized vectors
$\tilde{\mathbf{z}}/\|\tilde{\mathbf{z}}\|$ and
$\tilde{\mathbf{c}}/\|\tilde{\mathbf{c}}\|$. This step is repeated with the
$M$ transformation results in the $M$ predicted cluster assignment
probabilities for each view. When the network is untrained, the embeddings
$\mathbf{z}$ are random, and applying the random transformations, followed by
computing the predicted cluster assignments, leads to a diverse set of soft
cluster assignments. The parameter weights are trained by using the stochastic
gradients of the loss for updates.

####  2.2.4 Backbone loss

To better capture exemplar consistency, based on previous evidence of
successful clustering with the ID approach (Tao ., 2021), we use ID (Z. Wu .,
2018) as one of the losses, as in Tao . (2021). The exemplar objective of ID
is to classify each image as its own class.

Given $n$ images and a neural network $f_{\theta}$ for calculating features,
we first normalize the features
$\bar{f}_{\theta}(x)=\frac{f_{\theta}(x)}{\|f_{\theta}(x)\|}$. Then, ID
defines the probability of an example $x$ being recognized as the $i$-th
example as

$P(i|f_{\theta}(x))=\frac{\exp\left(\langle\bar{f}_{\theta}(x_{i}),\bar{f}_{\theta}(x)\rangle/\tau\right)}{\sum_{j=1}^{n}\exp\left(\langle\bar{f}_{\theta}(x_{j}),\bar{f}_{\theta}(x)\rangle/\tau\right)}.$
(4)

ID then uses the uniform distribution as a noise distribution
$P_{n}=\frac{1}{n}$ to compute the probability that data example $x$ comes
from a data distribution $P_{d}$ as opposed to the noise distribution $P_{n}$
as
$h(i,f_{\theta}(x)):=\frac{P(i|f_{\theta}(x))}{P(i|f_{\theta}(x))+mP_{n}(i)}$.
Assuming that the noise samples are $m$ times more frequent than actual data
samples, the ID loss is defined as

$\displaystyle\begin{split}L_{b}&=-E_{P_{d}}\left[\log
h(i,x)\right]-mE_{P_{n}}\left[\log(1-h(i,x^{\prime}))\right],\end{split}$ (5)

where $x^{\prime}$ is the feature from a randomly drawn image other than image
$x$ in a given dataset. We exactly follow the framework developed in Z. Wu .
(2018) to implement the ID loss.

The final loss that we seek to minimize is the combination of the losses
$L_{\mathcal{Z}}$ (equation 1) and $L_{b}$ (equation 5),

$L_{\text{total}}=\alpha L_{\mathcal{Z}}+\beta L_{b}.$ (6)

where $\alpha,\beta$ are nonnegative constants. Details of the algorithm are
given Algorithm 1 and we also provide a PyTorch-style psuedocode in Algorithm
2 in the Appendix.

Algorithm 1 Consensus Clustering algorithm (ConCURL)

1:Dataset $\mathcal{X}=\\{x_{i}\\}_{i=1}^{N}$, number clusters $K$, batch size
$B$, weights $(\alpha,\beta)$, number of random transformations $M$,
projection dimension $d$

2:Randomly initialize network the networks $f_{\theta},g$; $K$ prototypes
($\mathbf{c}_{1:K}$), and $M$ random projection matrices to dimension $d$
($R_{1:M}$) and epoch $e=0$

3:while $e<$ total epoch number do

4: for $b\in\\{1,2,\dots,\lfloor\frac{N}{B}\rfloor\\}$ do

5: Select $B$ samples as $\mathcal{X}_{b}$ from $\mathcal{X}$

6: Make a forward pass on the two views: $f_{\theta}(\mathcal{X}_{b}^{1})$ and
$f_{\theta}(\mathcal{X}_{b}^{2})$

7: $\mathbf{z}^{1}_{1:B}\longleftarrow
g(f_{\theta}(\mathcal{X}_{b}^{1})),\;\mathbf{z}^{2}_{1:B}\longleftarrow
g(f_{\theta}(\mathcal{X}_{b}^{2}))$

8: Compute loss $L_{b}$ using (4, 5)

9: Compute $\mathbf{p}^{1}_{i,j},\mathbf{p}^{2}_{i,j}$ for both views using
(2)

10: Compute codes of the current batch $\mathbf{q}$ using (3)

11: for $m\in\\{1,2,\dots,M$ } do

12: $\tilde{\mathbf{z}}\longleftarrow
R_{m}\mathbf{z},\;\tilde{\mathbf{c}}\longleftarrow R_{m}\mathbf{c}$

13: Compute
($\tilde{\mathbf{p}}^{(1,m)}_{i,j},\tilde{\mathbf{p}}^{(2,m)}_{i,j}$) for
normalized $\tilde{\mathbf{z}},\tilde{\mathbf{c}}$ using (2)

14: end for

15: Compute loss $L_{\mathcal{Z}}$ using (1)

16: Compute total loss (6). Update the network $f_{\theta},g$ parameters, and
prototypes using gradients

17: end for

18: $e:=e+1$

19:end while

20:Make forward pass on all the data and store the features

21:Compute k-means clustering of the features

####  2.2.5 Computing the cluster metrics

In this section, we describe the approach used to compute the cluster
assignments and the metrics chosen to evaluate their quality. Note that we
assume that the number of true clusters ($K$) in the data is known.

There are two ways to compute the cluster assignments. The first way is to use
the embeddings generated by the backbone; here, the embeddings are the outputs
of the ID block $f_{\theta}(x)$. The embeddings of all the images are
computed, and then we perform k-means clustering.

The second method is to use the soft clustering block to compute the cluster
assignments. It is sufficient to use the computed probability assignments
$\\{\mathbf{p}_{i}\\}_{i=1}^{N}$ or the computed codes
$\\{\mathbf{q}_{i}\\}_{i=1}^{N}$ and assign the cluster index as
$c_{i}=\arg\max_{k}\mathbf{q}_{ik}$ for the $i^{\text{th}}$ data point. Once
the model is trained, in this second approach, cluster assignment can be
performed online without requiring the computation of the embeddings of all
the input data.

We evaluate the quality of the clusterings using metrics such as the cluster
accuracy, normalized mutual information (NMI), and adjusted Rand index (ARI).
To compute the clustering accuracy, we are required to solve an assignment
problem (computed using a Hungarian match (Kuhn, 1955, 1956)) between the true
class labels and the cluster assignments. In our analysis, we observe that
using k-means with the embeddings produced by the ID block achieves better
clustering accuracy, and we use this method throughout the paper while
evaluating our proposed algorithm.

###  2.3 Generating Multiple Clustering Results

Table 1: Different ways to generate ensembles Data Representation |  Clustering algorithms
---|---
Different data preprocessing techniques |  Multiple clustering algorithms (k-means, GMMs, etc)
Subsets of features |  Same algorithm with different parameters or initializations
Different transformations of the features |  Combination of multiple clustering results and different parameters or initializations

Fred Jain (2005) discussed different ways to generate cluster ensembles; these
methods are tabulated in Table 1. In our proposed algorithm, we focus on
choosing of the appropriate data representation to generate cluster ensembles.

By fixing a stable clustering algorithm, we can generate arbitrarily large
ensembles by applying different transformations on the embeddings. Random
projections were previously successfully used in consensus clustering (Fern
Brodley, 2003). By generating ensembles using random projections, we have
control over the amount of diversity we can induce into the framework by
varying the dimensionality of the random projection. In addition to random
projections, we also use diagonal transformations (Hsu ., 2018) where
different components of the representation vector are scaled differently. Hsu
. (2018) illustrated that such scaling enables a diverse set of clusterings,
which is helpful for the meta learning task. We study ablations over the
number of transformations needed and the dimensions of these transformations
in Section 5.

##  3 Understanding the Consensus Objective

We investigate a potential hypothesis regarding “training driven by noisy
cluster assignments” that can shed light on the success of ConCURL222 A
theoretically grounded explanation of ConCURL is considered future work due to
the nonconvexity of deep learning methods and the nonconvexity of the proposed
loss.. The hypothesis stems from the following intuition. Using different
clustering algorithms, the generated cluster assignments are noisy versions of
the hypothetical ground truth; as the training process progresses, the noise
in the cluster assignments is reduced, and eventually all the different
clustering algorithms considered generate similar cluster assignments.

Figure 2: a: Three-cluster dataset = z, b: Normalized data Table 2: Predicted
cluster assignment probabilities and target probabilities obtained from the
Sinkhorn algorithm for four data points

Cluster assignment probability $\mathbf{p}$ | Cluster assignment probability $\tilde{\mathbf{p}}$ | Targets $\mathbf{q}$ from Sinkhorn
---|---|---
|  | Cluster 1 | Cluster 2 | Cluster 3
---|---|---|---
1 | 0.6473 | 0.2587 | 0.0940
2 | 0.7180 | 0.1812 | 0.1008
3 | 0.0832 | 0.3160 | 0.6008
4 | 0.1543 | 0.5917 | 0.2541
|  | Cluster 1 | Cluster 2 | Cluster 3
---|---|---|---
1 | 0.6764 | 0.2271 | 0.0965
2 | 0.7305 | 0.1680 | 0.1015
3 | 0.0802 | 0.3371 | 0.5827
4 | 0.1357 | 0.5784 | 0.2860
|  | cluster 1 | cluster 2 | cluster 3
---|---|---|---
1 | 1.0 | 8.9e-09 | 1.7e-17
2 | 1.0 | 9.1e-13 | 8.7e-18
3 | 6.8e-18 | 2.2e-6 | 1.0
4 | 2.5 e-12 | 1.0 | 5.4e-8

We verify this hypothesis empirically with the help of the following
experiments on the STL-10 dataset: (i) we observe the noisy clusterings
generated by using random projections and (ii) verify that the noise in the
cluster assignments is reduced as training progresses.

For the purpose of demonstrating noisy cluster assignments, we use synthetic
data as follows. We generate three clusters in $\mathbb{R}^{2}$, as shown in
Figure 2, and compute the centroids of each cluster. Here, the centroids act
as the prototypes. We then generate a Gaussian random projection matrix $A$
with a dimensionality of $\mathbb{R}^{2\times 2}$. We first normalize the
embeddings (2-dimensional features) and the centroids (see Figure 2). Using
the matrix $A$, we transform both the embeddings and prototypes for the new
space and normalize the resultant vectors.

We follow the soft clustering framework discussed earlier and compute the soft
cluster assignments for the original and transformed data. We observe that the
cluster assignment probabilities in the new space are noisy versions of the
cluster assignment probabilities in the original space (see Table 2).

Figure 3: Pairwise NMI values as a way to measure the diversity in the
ensemble; the results are obtained for the STL-10 dataset, and the pairwise
NMI values for different random projection dimensions are shown (the original
dimensionality of $\mathbf{z}$ is 256)

To verify that the noise in the cluster assignment probabilities is reduced as
training progresses, we perform the following experiment. We measure the
similarity among the cluster assignments at every epoch to observe the effect
of consensus as training progresses. For each random projection used, we use
the cluster assignment probabilities $\tilde{\mathbf{p}}$ and compute cluster
assignments by taking an $\operatorname*{arg\,max}$ on $\tilde{\mathbf{p}}$
for each image. We obtain M such cluster assignments due to the M random
projections. We then compute a pairwise NMI (similar to the analysis of Fern
Brodley (2003)) between every two cluster assignments and compute the average
and standard deviation of the pairwise NMI values across the
$\frac{M(M-1)}{2}$ pairs. An NMI score of $1.0$ signifies that the two
clusters perfectly correlate with each other, and a score of $0.0$ implies
that the two clusters are uncorrelated. We observe from Figure 2 that the
pairwise NMI increases as training progresses and becomes closer to 1. At the
beginning of training, the cluster assignments are very diverse (small NMI
scores with a large standard deviation), and as training progresses, the
diversity is reduced (large NMI scores with a smaller standard deviation).
This observation leads us to conclude that for the applied clustering
algorithms (defined using different random projections), we have learned an
embedding space where the different cluster assignments concur. In other
words, “consensus consistency” is achieved. Additionally, it is evident from
our empirical results in Section 4 that we achieve an improved overall
clustering accuracy.

If noisy cluster assignments are the reason behind the improved performance,
one might wonder if it is sufficient to simply add noise to the original
cluster assignments rather than computing multiple cluster assignments.
However, this may not be fruitful because if noise is added externally, one
must define a scheduler to reduce the noise as training progresses. However,
in the case of ConCURL, the end-to-end learning algorithm determines the rate
of consensus or agreement between $\mathbf{p}$ and $\tilde{\mathbf{p}}$
itself. In the next section, we provide empirical evidence of the
effectiveness of our method.

##  4 Empirical Evaluation

Evaluating clustering algorithms is a notoriously hard problem. The reference
text Jain Dubes (1988) states the following: The validation of clustering
structures is the most difficult and frustrating part of cluster analysis.
Without a strong effort in this direction, cluster analysis will remain a
black art accessible only to those true believers who have experience and
great courage.

In the literature on representation learning for clustering, e.g., Li .
(2021); Huang . (20202); Tao . (2021), to evaluate the performance of
different algorithms, the following methodology has been used: as a set of
models with some hyperparameters are trained, these models are sorted by the
observed clustering performance. Finally the best model’s results are
reported. This methodology is called max performance in the remainder of the
paper. We assess the quality of the learned embeddings by using five
challenging image datasets for clustering and report their performance with
the max-performance strategy. Although the max-performance procedure provides
some insights into the performance of the method under consideration, we
provide additional insights by significantly extending the evaluations. In
practice, it is desirable that the learned models can be utilized for
different datasets other than the training dataset. However, the max-
performance method may not be suitable for this purpose. To address this, we
design two additional experiments that focus on the performance of cross-model
features under a distribution shift. Furthermore, we also assess the quality
of the learned embeddings in image retrieval tasks. Finally, we present a
detailed ablation study to assess the impact of the loss terms, data
augmentation methods, hyperparameters and architecture choices utilized to
obtain a more complete picture.

###  4.1 Image Clustering with the Max-Performance Strategy

We evaluate our algorithm and compare it with existing methods on some popular
image datasets, namely, ImageNet-10, ImageNet-Dogs, STL-10, CIFAR-10, and
CIFAR100-20.

For CIFAR100-20, we use the 20 meta classes as the class labels while
evaluating the clustering results. For STL-10, similar to the earlier PICA
(Huang ., 20201) and GATCluster (Niu ., 20201) approaches, we use both
training and testing splits for training and evaluation. Note that PICA also
uses an unlabeled data split with 100k points in STL-10, which we do not use.
ImageNet-10 and ImageNet-Dogs are subsets of ImageNet, and we use only the
training splits for these two datasets (Deng ., 2009). We use the same classes
as Chang . (2017) for evaluation on the ImageNet-10 and ImageNet-Dogs
datasets. The dataset summary is given in Table 3. We evaluate the cluster
accuracy, NMI, and ARI of each computed cluster assignment (see the Appendix
for details).

Table 3: Dataset summary Dataset | Classes | Split | Samples | Resolution
---|---|---|---|---
ImageNet-10 | 10 | train | 13000 | 160 $\times$ 160
ImageNet-Dogs | 15 | train | 19500 | 160 $\times$ 160
STL-10 | 10 | train+test | 13000 | 96$\times$ 96
CIFAR-10 | 10 | train | 50000 | 32$\times$ 32
CIFAR100-20 | 20 | train | 50000 | 32$\times$ 32

####  4.1.1 Comparison with state-of-the-art baselines

Table 4: Methods Compared  Method | Reference
---|---
k-means | Macqueen (1967)
SC | Ng . (2002)
AC | Franti . (2006)
NMF | Cai . (2009)
AE | Bengio . (2007)
SDAE | Vincent . (2010)
DeCNN | Zeiler . (2010)
JULE | Yang . (2016)
DEC | Xie . (20162)
DAC | Chang . (2017)
Deep Cluster | Caron . (20182)
DDC | Chang . (2019)
IIC | Ji . (20192)
DCCM | J. Wu . (20192)
GATCluster | Niu . (20202)
PICA | Huang . (20202)
CC | Li . (2021)
ADC | Haeusser . (2019)
ID | Tao . (2021)
IDFD | Tao . (2021)
Table 5: Clustering with the max-performance strategy

Method | STL-10 | ImageNet-10 | ImageNet-Dogs | CIFAR-10 | CIFAR100-20
---|---|---|---|---|---
ACC | NMI | ARI | ACC | NMI | ARI | ACC | NMI | ARI | ACC | NMI | ARI | ACC | NMI | ARI
k-means | 0.192 | 0.125 | 0.061 | 0.241 | 0.119 | 0.057 | 0.105 | 0.055 | 0.020 | 0.229 | 0.087 | 0.049 | 0.130 | 0.084 | 0.028
SC | 0.159 | 0.098 | 0.048 | 0.274 | 0.151 | 0.076 | 0.111 | 0.038 | 0.013 | 0.247 | 0.103 | 0.085 | 0.136 | 0.090 | 0.022
AC | 0.332 | 0.239 | 0.140 | 0.242 | 0.138 | 0.067 | 0.139 | 0.037 | 0.021 | 0.228 | 0.105 | 0.065 | 0.138 | 0.098 | 0.034
NMF | 0.180 | 0.096 | 0.046 | 0.230 | 0.132 | 0.065 | 0.118 | 0.044 | 0.016 | 0.190 | 0.081 | 0.034 | 0.118 | 0.079 | 0.026
AE | 0.303 | 0.250 | 0.161 | 0.317 | 0.210 | 0.152 | 0.185 | 0.104 | 0.073 | 0.314 | 0.239 | 0.169 | 0.165 | 0.100 | 0.048
SDAE | 0.302 | 0.224 | 0.152 | 0.304 | 0.206 | 0.138 | 0.190 | 0.104 | 0.078 | 0.297 | 0.251 | 0.163 | 0.151 | 0.111 | 0.046
DeCNN | 0.299 | 0.227 | 0.162 | 0.313 | 0.186 | 0.142 | 0.175 | 0.098 | 0.073 | 0.282 | 0.240 | 0.174 | 0.133 | 0.092 | 0.038
JULE | 0.277 | 0.182 | 0.164 | 0.300 | 0.175 | 0.138 | 0.138 | 0.054 | 0.028 | 0.272 | 0.192 | 0.138 | 0.137 | 0.103 | 0.033
DEC | 0.359 | 0.276 | 0.186 | 0.381 | 0.282 | 0.203 | 0.195 | 0.122 | 0.079 | 0.301 | 0.257 | 0.161 | 0.185 | 0.136 | 0.050
DAC | 0.470 | 0.366 | 0.257 | 0.527 | 0.394 | 0.302 | 0.275 | 0.219 | 0.111 | 0.522 | 0.396 | 0.306 | 0.238 | 0.185 | 0.088
Deep Cluster | 0.334 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.374 | N/A | N/A | 0.189 | N/A | N/A
DDC | 0.489 | 0.371 | 0.267 | 0.577 | 0.433 | 0.345 | N/A | N/A | N/A | 0.524 | 0.424 | 0.329 | N/A | N/A | N/A
IIC | 0.610 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.617 | N/A | N/A | 0.257 | N/A | N/A
DCCM | 0.482 | 0.376 | 0.262 | 0.710 | 0.608 | 0.555 | 0.383 | 0.321 | 0.182 | 0.623 | 0.496 | 0.408 | 0.327 | 0.285 | 0.173
GATCluster | 0.583 | 0.446 | 0.363 | 0.762 | 0.609 | 0.572 | 0.333 | 0.322 | 0.200 | 0.610 | 0.475 | 0.402 | 0.281 | 0.215 | 0.116
PICA | 0.713 | 0.611 | 0.531 | 0.870 | 0.802 | 0.761 | 0.352 | 0.352 | 0.201 | 0.696 | 0.591 | 0.512 | 0.337 | 0.310 | 0.171
CC | 0.850 | 0.746 | 0.726 | 0.893 | 0.859 | 0.822 | 0.429 | 0.445 | 0.274 | 0.790 | 0.705 | 0.637 | 0.429 | 0.431 | 0.266
ADC | 0.530 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.325 | N/A | N/A | 0.160 | N/A | N/A
ID | 0.726 | 0.64 | 0.526 | 0.937 | 0.867 | 0.865 | 0.476 | 0.47 | 0.335 | 0.776 | 0.682 | 0.616 | 0.409 | 0.392 | 0.243
IDFD | 0.756 | 0.643 | 0.575 | 0.954 | 0.898 | 0.901 | 0.591 | 0.546 | 0.413 | 0.815 | 0.711 | 0.663 | 0.425 | 0.426 | 0.264
ConCURL | 0.749 | 0.636 | 0.566 | 0.958 | 0.907 | 0.909 | 0.695 | 0.63 | 0.531 | 0.846 | 0.762 | 0.715 | 0.479 | 0.468 | 0.3034

In our comparison, we consider some state-of-the-art methods that were
developed for image clustering problems and targeted for end-to-end training
scenarios with random initialization. We should note that we do not consider
baselines that use prior information, e.g., the nearest neighbors algorithm
derived by using pretrained models. The implementation details of ConCURL are
provided in the Appendix, and the results are presented in Table 5.

We observe that ConCURL outperforms the baseline algorithms considered in
terms of all three metrics for all the datasets except STL-10. ConCURL
improves the state-of-the art clustering accuracy by approximately $17.5\%$ on
ImageNet-Dogs, by $12.7\%$ on CIFAR100-20 and by $3.8\%$ on CIFAR-10. Although
ConCURL improves upon the results of ID (Tao ., 2021), please note that ID is
the backbone used in this paper and is slightly worse than IDFD, as shown in
Tao . (2021).

The proposed method achieves good clustering performance on popular computer
vision datasets. Similar to all the algorithms considered, we assume that $K$,
the number of clusters, is known. However, this may not hold true in practice
in real-world applications. In such a case, we may assume an estimate for the
upper bound on the number of clusters to use as the number of prototypes.
Additionally, we also assume that the dataset is equally distributed among the
$K$ clusters. If this assumption (also common in the literature (Huang .,
20201; Niu ., 20201)) does not hold, the fast Sinkhorn-Knopp algorithm used to
solve Eq. 3 may not be optimal.

(a) ImageNet-10 (b) ImageNet-Dogs (c) CIFAR-10 (d) STL-10 (e) CIFAR100-20

Figure 4:  Confusion matrices

####  4.1.2 Performance on the Test split

Table 6: Clustering on the test dataset Dataset | Train Dataset | Test Dataset
---|---|---
ACC | NMI | ARI | ACC | NMI | ARI
CIFAR-10 | 0.846 | 0.762 | 0.715 | 0.838 | 0.744 | 0.700
CIFAR100-20 | 0.479 | 0.468 | 0.303 | 0.427 | 0.430 | 0.254
ImageNet-10 | 0.958 | 0.908 | 0.910 | 0.914 | 0.876 | 0.838
ImageNet-Dogs | 0.695 | 0.630 | 0.532 | 0.660 | 0.675 | 0.515

In the previous section, we studied the clustering performance on the training
split used to train the algorithm. Here, we shall evaluate the clustering
performance on a held out test set. We shall use the standard test split for
the datasets CIFAR-10, CIFAR100-20, ImageNet-10, and ImageNet-Dogs. We used
the trained models to extract the features for each test dataset and compute
the clustering as above. We observe from Table 6 that the performance doesn’t
get affected much on the test set. This shows that the algorithm is able to
extract good feature representations for clustering on data not used for
training which shows good generalization ability of the algorithm when the
data is drawn from the same distribution.

####  4.1.3 Class-specific accuracy

We present the class-specific accuracies (percentage) and confusion matrices
in Figure 4. In each row $i$, $j^{th}$ entry in the matrix represents the
percentage of samples from category $i$ belonging to the cluster of category
$j$. For better visualization, we round each percentage to the nearest
integer.Row sum may not be equal to 100 because we are rounding to the nearest
integer. For perfect clustering, all elements along the diagonal should be
equal to 100. Here, we note some interesting observations. For ImageNet-10,
the airliner category shows the worst performance, with 7% of airliner samples
being confused with the airship category. Additionally, 4% of the samples from
the orange category are categorized as soccer balls. In ImageNet-Dogs, there
are three types of spaniels- Blenheim spaniels, Brittany spaniels and Welsh
springer spaniel, which look very similar to each other. Nineteen percent of
the samples from the Blenheim spaniel category are categorized as Brittany
spaniels, and 5% are categorized as Welsh springer spaniels. Similarly, 16% of
the samples from the Brittany spaniel category are categorized as Blenheim
spaniels, and 16% more are categorized as Welsh springer spaniels. Forty-four
percent of the samples from the Welsh springer spaniel category are
categorized as Brittany spaniels, and 15% are categorized as Blenheim
spaniels. Kelpies and Dobermans are also confused with each other, where 39%
of the kelpie samples are categorized as Dobermans, and 29% of the Doberman
samples are categorized as kelpies. For CIFAR-10, 33% of the samples from the
dog category are categorized as cats, and 16% of the samples from the cat
category are categorized as dogs.

For STL-10, none of the samples from the cat category are categorized
correctly. Even though STL-10 and CIFAR-10 have the same list of categories,
STL-10 seems harder to cluster than CIFAR-10. Note that STL-10 has only 13000
images to learn representations, while in CIFAR-10, 60000 images are used. For
CIFAR100-20, none of the samples from aquatic mammals are categorized
correctly. Thirty-seven percent of the samples from aquatic mammals are
categorized as fish, and 19% are categorized as large omnivores and
herbivores. In the case of reptiles, only 15% of the examples are categorized
correctly, but 23% are categorized as fish, 16% as large carnivores and 13% as
insects. Surprisingly, 28% of the samples from trees are categorized as
aquatic mammals.

###  4.2 Out-of-Distribution Results

In this section, we evaluate ConCURL by performing clustering on dataset that
is not used during training. We focus mainly on studying the clustering
performance on datasets that maybe similar to the training dataset and
datasets that may have a different number of clusters than the training
dataset.

####  4.2.1 Cross-Model Accuracy

Here, we calculate the clustering performance achieved when the model is
trained on one dataset but evaluated on a different dataset that may have a
different number of clusters. For example, the first row in Table 7 gives the
performance of the model trained on ImageNet-10 and evaluated on both
ImageNet-10 and ImageNet-Dogs. Similarly, the second row shows the performance
of the model trained on ImageNet-Dogs. We find that performance on ImageNet-10
is decreased to 35.6% when the model trained on ImageNet-Dogs is used instead
of the model trained on ImageNet-10. Similarly, the performance on ImageNet-
Dogs is decreased to 17.7 % when the model trained on ImageNet-10 is used
instead of the model trained on ImageNet-Dogs. Table 8 provides the same
performance metrics for CIFAR-10 and CIFAR100-20.

For cross-model performance to be high, the embedding function must be
generalizable to the out of distribution dataset. It is important to observe
that for each pair of datasets considered, the distributions of the datasets
are very different due to the classes being completely different in both cases
(ImageNet-10 vs ImageNet-Dogs, and CIFAR-10 vs CIFAR100-20). However, since we
are considering datasets with a small number of datapoints and small number of
classes (see Table 3), the representation power of the learnt embeddings is
limited and this affects the cross-model accuracy. Moreover, the consensus
loss $L_{\mathcal{Z}}$ here assumes knowledge of the number of clusters in the
dataset. Therefore, the embeddings learnt by optimizing the $L_{total}$ loss
on one dataset may be sub-optimal for evaluating clustering on a dataset with
different number of clusters.

It is clear that these performance drops are significant, and the
generalization performance of the learned embeddings needs to be assessed by
taking out-of-distribution datasets into account. However, since there are
only 2 groups of different datasets, it is difficult to reach a definitive
conclusion. Hence, in the following section, we propose a new evaluation
methodology that sheds light on the out-of-distribution performance of the
learned embeddings.

Table 7: ImageNet-10 vs. ImageNet-Dogs: cross-model performance Model Training Dataset | ImageNet-10 | ImageNet-Dogs
---|---|---
ACC | NMI | ARI | ACC | NMI | ARI
ImageNet-10 | 0.958 | 0.908 | 0.910 | 0.177 | 0.127 | 0.068
ImageNet-Dogs | 0.356 | 0.298 | 0.184 | 0.695 | 0.630 | 0.532
Table 8: CIFAR-10 vs. CIFAR100-20: cross-model performance Model Training Dataset | CIFAR-10 | CIFAR100-20
---|---|---
ACC | NMI | ARI | ACC | NMI | ARI
CIFAR-10 | 0.846 | 0.762 | 0.715 | 0.178 | 0.158 | 0.061
CIFAR100-20 | 0.464 | 0.359 | 0.250 | 0.480 | 0.468 | 0.304

####  4.2.2 ImageNet random-10 and random-15 accuracies

Here, we compare the baseline model trained with ID (Z. Wu ., 2018) with our
proposed method ConCURL. We randomly sample 10 and 15 classes from the
1000-class ImageNet data and evaluate the clustering accuracy obtained on the
training split of the data using the model trained on the original ImageNet-10
and ImageNet-Dogs sets. We repeat the process 100 times for both the 10-class
and 15-class datasets and call them the random-10 and random-15 datasets,
respectively. Note that we do not retrain the model on the randomly sampled
dataset; we only evaluate the model on this set. We show the histogram of the
obtained accuracies for these 100 random datasets. In Figure 5, we compare the
accuracy of the ConCURL model and the baseline ID model trained on ImageNet-10
on both random-10 and random-15. Along with the histogram, we show a Gaussian
distribution (along the red dotted line) with first and second moments equal
to the average and standard deviation of all accuracies, respectively.
Similarly, in Figure 6, we show the accuracies obtained based on models
trained on ImageNet-Dogs. Among the models trained on ImageNet-10, the
baseline ID model performs slightly better than the proposed ConCURL model.
The trend is reversed for the evaluation based on the model trained on
ImageNet-Dogs, where ConCURL performs better than the baseline model. Even
though the proposed method performs best with the max-performance strategy, it
performs slightly worse on random-10. This result strengthens our argument
regarding the need to go beyond the traditional reporting of maximum
performance based on the ACC, NMI and ARI metrics.

Figure 5: Histogram of clustering accuracies for models trained on
ImageNet-10: (a) ConCURL model evaluated on random-10, (b) ConCURL model
evaluated on random-15, (c) Baseline (ID) model evaluated on random-10, (d)
Baseline (ID) model evaluated on random-15

Figure 6: Histogram of clustering accuracies for models trained on ImageNet-
Dogs: (a) ConCURL model evaluated on random-10, (b) ConCURL model evaluated on
random-15, (c) Baseline (ID) model evaluated on random-10, (d) Baseline (ID)
model evaluated on random-15 Figure 7: Images from the same cluster: In
ImageNet-10, we randomly sample six images from all ten clusters and show them
above. Each row presents one cluster and images that have red-and-yellow
borders are categorized incorrectly and should belong to different clusters.
Figure 8: Image retrieval: The first image in each row was used as a query
(random samples from the dataset ImageNet-10), and the five images nearest to
the query image were retrieved using their representations. Images with red-
and-yellow borders are retrieved incorrectly

###  4.3 Cluster Visualizations

In this section, we provide two visualizations of the ImageNet-10 dataset. In
Figure 7, each row presents randomly drawn images from each cluster. Images
that have red-and-yellow borders are categorized incorrectly and should belong
to different clusters. For example, the first image in the fourth row should
be in the truck category, but it is categorized as an airliner. In the soccer
ball category, there are two mistakes: the first image should be categorized
as a truck, and the fourth image should be categorized as a dog. In the ninth
row, the last image should be categorized as an airship, but it is categorized
as a truck.

To check whether the learned representations that are closest to each other
belong to the same category, we use a retrieval task. In Figure 8, we show the
results. The first image in each row was used as a query (random samples from
the dataset), and the five images nearest to the query image were retrieved
using their representations. Ideally, one would expect all retrieved images to
belong to the same category as the query image. For the example from the
soccer ball category, the first image retrieved does not belong to this
category; however, both images have water as their main feature. In the last
row, the second image retrieved is a penguin and ideally should not be one of
the closest matches to an image in the orange category.

##  5 Ablation Studies

Although the proposed method is trained in an end-to-end manner, each
component of the method may have a different impact on the results. We conduct
various controlled experiments to quantify the impact of the losses (Section
5.1), the data augmentation methods (Section 5.2), the image resolution
(Section 5.3), the number of transformations (Section 5.4), the dimensionality
of each transformation (Section 5.4) and the architecture choice (Section
5.5).

###  5.1 Effects of the Loss Terms

We consider the following scenarios: train with only loss $L_{\mathcal{Z}}$
(Consensus Loss), train with only loss $L_{b}$ (ID), train with both losses
$L_{b}+L_{\mathcal{Z}}$ (ConCURL). We do this for CIFAR-10, CIFAR100-20
datasets. We then compare the clustering performance of all three scenarios
and observe that training with both losses improves the performance.
Additionally, we also compare the loss trajectories during training. We
compare the $L_{b}$ loss for the case when we train only $L_{b}$ and for the
case when we train only $L_{\mathcal{Z}}$. We repeat this for loss
$L_{\mathcal{Z}}$. The results are summarized in Table 9, where we observe
that training with both the losses provides a much better clustering
performance as compared to training with the losses individually.

Table 9: Ablation on the losses during training Loss used | CIFAR100-20 | CIFAR-10
---|---|---
ACC | NMI | ARI | ACC | NMI | ARI
$L_{b}$ only 333For CIFAR-10, CIFAR100-20 we provide results of rerun that is better than reported in (Tao ., 2021) | 0.437 | 0.429 | 0.259 | 0.826 | 0.732 | 0.684
$L_{\mathcal{Z}}$ only | 0.302 | 0.333 | 0.145 | 0.739 | 0.667 | 0.539
$L_{b}+L_{\mathcal{Z}}$ (ConCURL) | 0.479 | 0.468 | 0.303 | 0.846 | 0.762 | 0.715

Figure 9:  We perform an ablation over the losses used and compare the loss
values for each of the cases for CIFAR100-20.

Figure 10:  We compare the clustering metrics for ablation over the losses
used for CIFAR100-20 and observe that Train with $L_{b}+L_{\mathcal{Z}}$
(ConCURL) outperforms training with only $L_{b}$ or $L_{\mathcal{Z}}$.

The exemplar consistency trains with the objective of classifying each data
point into its own class. The population and consensus consistencies train
without regarding for discrimination among the individual data points. An
algorithm that is trained with only the consensus loss therefore is not
effective in discriminating individual data points. From Figure 9, we can
observe that when we train with only $L_{\mathcal{Z}}$, we do not observe any
improvement in the loss $L_{b}$; when we train with $L_{b}+L_{\mathcal{Z}}$,
we observe a similar trajectory as training only with $L_{b}$. This shows that
training with $L_{\mathcal{Z}}$ does not conflict with $L_{b}$ loss.

On the other hand, it is possible that an algorithm that is trained to
discriminate individual data points can show some improvement on the consensus
loss $L_{\mathcal{Z}}$. From Figure 9, we can observe that $L_{\mathcal{Z}}$
is lesser when trained with $L_{b}+L_{\mathcal{Z}}$ than when training only
with $L_{\mathcal{Z}}$. We observe a small decrease in $L_{\mathcal{Z}}$ value
when trained only with $L_{b}$. This shows that optimizing $L_{b}$ helps to
some extent in achieving a better $L_{\mathcal{Z}}$.

From this discussion, we observe that both losses contribute differently to
the training without much interference or conflicts. They indeed complement
each other as we observe improved clustering performance for ConCURL from
Table 9 and Figure 10.

###  5.2 Effects of Data Augmentation Methods

Table 10: Data augmentation details  Data Augmentation Type | Abbreviation | CIFAR-10 | CIFAR100-20
---|---|---|---
All data augmentations | All DA | 0.8459 | 0.4798
No random horizontal flip | No RHF | 0.7689 | 0.4551
No random gray scale | No RGS | 0.7750 | 0.4357
No randomly applied blur | No RAB | 0.7907 | 0.4295
No color jitter | No CJ | 0.6463 | 0.2536
No random resized cropping | No RRC | 0.2988 | 0.1322
Figure 11: Effect of data augmentation on CIFAR-10 Figure 12: Effect of data
augmentation on CIFAR100-20

Augmenting the training data is a standard technique for training deep
learning methods (Shorten Khoshgoftaar, 2019). The backbones used in this
study rely on the different views that are generated by applying different
augmentations to the input image. Recently, Tian . (2020) investigated the
impact of data augmentation on contrastive learning methods and shed some
light on this topic. In our setting, we would like to quantify the impacts of
several data augmentations on the consensus loss.

In Table 10, we show the maximum accuracy achieved when all data augmentation
approaches are used and when we skip one data augmentation technique at a
time. When all data augmentation methods are used, the maximum accuracies
achieved are 0.8459 and 0.4798 on CIFAR-10 and CIFAR100-20, respectively. When
random resized cropping data augmentation is dropped, we obtain the maximum
accuracy drops for both datasets, followed by color jitter. Other data
augmentation techniques are important for obtaining the best possible accuracy
but do not have as much of an effect as color jitter and random resized
cropping. In Figures 11 and 12, we show how the running mean of accuracy
progresses during training for each of the experiments in Table 10.

###  5.3 Effect of Image Resolution

Table 11: Effects of different resolutions for STL-10, ImageNet-10 and ImageNet-Dogs | $32$ | $64$ | $96$ | $160$ | $224$
---|---|---|---|---|---
STL-10 | 0.531 | 0.724 | 0.749 | NA | NA
ImageNet-10 | NA | NA | 0.887 | 0.958 | 0.946
ImageNet-Dogs | NA | NA | 0.629 | 0.695 | 0.679

Image resolution is often considered a free parameter (Niu ., 20201), and
however, its effect on clustering performance is not evaluated rigorously in
most works. We try to quantify the effects of different resolutions to the
greatest extent possible, given that some datasets are available only at
specific resolutions. For STL-10, we use $32\times 32$, $64\times 64$ and
$96\times 96$ resolutions. For ImageNet-$10$ and ImageNet-Dog-$15$, we use
$96\times 96$, $160\times 160$ and $224\times 224$ resolutions. The results
are given in Table 11.

The best performance for ImageNet-10 and ImageNet-Dogs is obtained at a
resolution of 160, and for STL-10, the best performance is obtained at a
resolution of 96. It is not clear why ImageNet-10 and ImageNet-Dogs do not
yield the best performance at high resolutions, and further investigation is
needed; we keep this as an open problem.

###  5.4 Distribution of Accuracies Across the Set of Hyperparameters

Table 12: Hyperparameters and the range values used for the experiments | min | max | step size
---|---|---|---
$\tau$ = temperature parameter | 0.3 | 1 | 0.1
$l$ = learning rate | 0.015 | 0.09 | 0.015
$\eta$ = natural log of the number of transformations | 0 | 7 | 1
$d$ = dimensionality of the projection space | 80 | 168 | 8
Table 13: Hyperparameters for obtaining maximum performance | STL-10 | ImageNet-10 | ImageNet-Dogs | CIFAR-10 | CIFAR100-20
---|---|---|---|---|---
$\tau$ | 0.8 | 1.0 | 0.5 | 0.85 | 0.35
$l$ | 0.03 | 0.03 | 0.06 | 0.015 | 0.06
$\exp(\eta)$ | 4 | 1 | 64 | 64 | 16
$d$ | 80 | 104 | 136 | 152 | 32

(a) STL-10: log of the number of transformations  (b) STL-10: dimensionality
of the projection space (c) CIFAR100-20: log of the number of transformations
(d) CIFAR100-20: dimensionality of the projection space

Figure 13: Components of the consensus loss; ablation of STL-10 and
CIFAR100-20

The proposed consensus loss has two parameters. The first is the number of
transformations used, and the second is the dimensionality of the projection
space. To understand the proposed loss, we conduct a detailed experimental
study on STL-10 and CIFAR100-20444We conduct a similar but smaller study on
the remaining dataset, and we observe similar trends.. The hyperparameters
used are given in Table 12.

Due to the sheer number of conducted experiments, we supply the summary
statistics obtained on a random set. We report the empirical mean and standard
deviation of the marginal distribution of the quantity under investigation.
Let $P_{\tau,\eta,d,l}$ be the joint distribution over the hyperparameters
$\tau$ (temperature parameter), $l$ (learning rate), $\eta$ (natural log of
the number of transformations) and $d$ (dimensionality of the projection
space). We consider $n_{h}$ as the number of distinct values used in the
experiment for each hyperparameter $h\in\\{\tau,\eta,d,l\\}$ based on Table
12. We the denote accuracy from each experiment based on the hyperparameters
used as $a_{\tau,\eta,d,l}$ . Let $P_{h_{i}|h_{j}}$ be the conditional
marginal distribution of hyperparameter $h_{i}$ given $h_{j}$ and the
conditional empirical mean of $P_{h_{i}|h_{j}}$ be $m(P_{h_{i}|h_{j}})$. In
this case, the conditional empirical mean $m(P_{h_{i}|h_{j}})$ when $h_{i}=d$
and $h_{j}=\tau$ can be calculated using
$m(P_{d|\tau})=\frac{1}{n_{\eta}\times
n_{l}}\sum_{\eta}\sum_{l}a_{\tau,\eta,d,l}$. The conditional empirical means
and standard deviations of other hyperparameters are calculated in the same
way. In Figure 13, we show each conditional empirical mean with a blue dot,
and each red line around a dot represents a standard deviation. For both
STL-10 and CIFAR100-20, we see a trend regarding the number of projections.
For STL-10, the smaller the number of random projections, the better the
results are, and for CIFAR100-20, increasing the number of random projections
is helpful for improving the clustering accuracy up to some point. Note that
when the number of random projections is equal to zero, our setting is
equivalent to the baseline ID model, and our approach always performs better
than ID. This means that the optimal number of random projections is greater
than or equal to one. There is no such clear trend in the number of dimensions
of the random projections.

Figure 14: The dotted red lines show the accuracy of the baseline, i.e., ID
Tao . (2021), on the corresponding dataset, and DE is the density estimate of
the empirical distribution: (a) Empirical accuracy distribution for STL-10,
(b) Empirical accuracy distribution for CIFAR100-20

The max-performance procedure provides some insights into the performance of
the algorithms at hand, although it does not provide the whole picture because
it does not consider the robustness of the performance differences. In Table
13, we give the hyperparameters that yield the max performance. In other
words, finding a hyperparameter set that yields better performance than the
baseline is the core idea behind the max-performance procedure. We ask the
following question: given a hyperparameter grid, how likely is our method to
achieve better accuracy than the baseline? In Figure 14, we report the
empirical accuracy distributions on STL-10 and CIFAR100-20 for all
hyperparameters given in Table 12. The red dotted lines show the corresponding
baseline accuracy for each dataset. For STL-10, only approximately $12.5\%$ of
the hyperparameter sets yield better results than the baseline. On the other
hand, for CIFAR100-20, approximately $90\%$ of the hyperparameter sets yield
better results than the baseline. In other words, it does not require a
significant amount of computational power to find a better model than the
state-of-the art models for CIFAR100-20; however, the situation is the
opposite for STL-10. The results given in Figure 14 suggest that when
comparing models, multiple metrics need to be considered, not only the max-
performance procedure.

###  5.5 Effect of Architecture Choice

Figure 15: Empirical distribution of the performance difference between
Residual Network (ResNet)-50 and ResNet-18. DE is the density estimate of the
empirical distribution.

In this work, we use ResNet-18 and ResNet-50 as network architectures. For
both ResNet-18 and ResNet-50, we sweep over the same set of hyperparameter
choices, i.e., the temperature, number of projections and projection
dimensionality, and report the results for ImageNet-10 dataset with image
resolution of 160$\times 160$. Figure 15 shows the distribution of
$\Delta_{acc}$, which is defined as the accuracy difference between ResNet-50
and ResNet-18. Figure 15 indicates that ResNet-50 slightly outperforms
ResNet-18, i.e., the mean difference is approximately $0.5\%$.

###  5.6 Runtime Comparison

Figure 16: Comparison of the runtimes per epoch for ID and ConCURL on CIFAR-10
dataset. We vary the number of random transformations in the computation of
the consensus loss and is mentioned in paranthesis.

To study the runtime of the proposed method, we compare the time taken per
epoch for the baseline ID algorithm and the proposed algorithm. Due to the
additional loss computation, the time taken to run the proposed algorithm is
higher which can be observed from Figure 16. The additional time taken is
mainly due to computing the consensus loss for the different number of
transformations. The current implementation computes the forward pass for the
different transformations sequentially thus increasing the runtime. However, a
more time efficient implementation where the forward passes for all the
different random transformations are computed in parallel can make the runtime
more comparable to the baseline ID algorithm.

##  6 Conclusion

In this work, we introduce different notions of the consistency constraints
that are enforced in different unsupervised/self-supervised learning
algorithms. We propose a novel clustering algorithm that seamlessly
incorporates all three consistency constraints (exemplar, population and
consensus) and achieves state-of-the-art clustering results for four out of
five popular and challenging computer vision datasets. Our work on consensus
clustering is significantly different from earlier consensus clustering works
that do not learn representations. Moreover, we initiate a discussion on the
adequacy of the currently used methods for evaluating clustering algorithms.
We significantly extend the evaluation procedure for clustering algorithms,
thereby reflecting the challenges of applying clustering to real-world tasks.
We provide evaluation results for ConCURL and other state-of-the-art
clustering algorithms based on max-performance criteria, according to which
ConCURL outperforms other algorithms on most datasets. However, its average
performance according to out-of-distribution criteria highlights the need to
use the proposed evaluation methods for deep clustering algorithms.

##  Appendix A Pseudo-code

Algorithm 2 PyTorch-style pseudocode for ConCURL

⬇

# f = feature network (e.g., ResNet18)

# g = MLP network that outputs embeddings

# c = prototypes - initialized randomly

# $\tau$ = temperature parameter

# list_A = random matrices for transformations

def ConsensusLoss(list_A, c, z1, z2, q1, q2):

# Transform the embeddings and prototypes

for counter, A in enumerate(list_A):

$\mathbf{\tilde{z}1}$= normalize(torch.mm(z1,A), p=2)

$\mathbf{\tilde{z}2}$ = normalize(torch.mm(z2,A), p=2)

with torch.no_grad():

$\mathbf{\tilde{c}}$ = normalize(torch.mm(c,A), p=2)

$\mathbf{\tilde{p}1}$= Softmax($\mathbf{\tilde{z}1}$, $\mathbf{\tilde{c}}$),
$\mathbf{\tilde{p}2}$= Softmax($\mathbf{\tilde{z}2}$, $\mathbf{\tilde{c}}$)

# CE = cross-entropy loss

consensus_loss = CE(q2,$\mathbf{\tilde{p}1}$) + CE(q1,$\mathbf{\tilde{p}2}$)

# training loop

for x, index in dataloader:

# Return embeddings from the model

x1 = random_transformation(x), x2 = random_transformation(x)

# embeddings (ID loss)

fx1 = f.forward(x1), fx2 = f.forward(x2)

# embeddings (clustering)

z1 = g.forward(fx1), z2 = g.forward(fx2)

# Compute ID loss

ID_loss = ID(normalized(fx1),index) # Using equation 5

# Compute p and q vectors

with torch.no_grad():

c = normalize(c,p=2)

out1 = torch.mm(c,normalize(z1,p=2)

out2 = torch.mm(c,normalize(z2,p=2)

q1 = Sinkhorn(out1) # Using equation 3

q2 = Sinkhorn(out2) # Using equation 3

consensus_loss = ConsensusLoss(list_A, c, z1, z2, q1, q2)

loss = $\alpha$*ID_loss + $\beta$*consensus_loss

optimizer.zero_grad(); loss.backward(); optimizer.step()

##  Appendix B Evaluation Metrics

We evaluate our algorithm by computing traditional clustering metrics (the
ACC, NMI, and ARI), which we discuss below in detail.

###  B.1 ACC

The ACC is computed by first computing a cluster partition of the input data.
Once the partitions are computed and cluster indices are assigned to each
input data point, a linear assignment map is computed using the Kuhn-Munkres
(Hungarian) algorithm, which reassigns the cluster indices to the true labels
of the data. The ACC is then given by

$ACC=\frac{\sum_{i=1}^{N}\mathbb{I}\\{y_{true}(x_{i})=c(x_{i})\\}}{N},$

where $y_{true}(x_{i})$ is a true label of $x_{i}$ and $c(x_{i})$ is the
cluster assignment produced by an algorithm (after Hungarian mapping).

###  B.2 NMI

For two clusters $U,V$, where each contains U,
V$clustersand$U_i$representsthenumberofsamplesincluster$U_i$ofclusteringresult$U$(similarfor$V$),theMIisgivenby$$MI(U,V)=\sum_{i=1}^{|U|}\sum_{j=1}^{|V|}\frac{|U_{i}\cap
V_{i}|}{N}\log\frac{N|U_{i}\cap
V_{j}|}{|U_{i}||V_{j}|}$$where$N$isthenumberofdatapointsunderconsideration.TheNMIisdefinedas$$NMI(U,V)=\frac{MI(U,V)}{\sqrt{MI(U,U)MI(V,V)}}$$\par$

###  B.3 ARI

555https://scikit-learn.org/stable/modules/clustering.html#adjusted-rand-score
Suppose that R is the ground truth clustering result and that S is a
partition. The RI of S is given as follows. Let $a$ be the number of pairs of
elements that are in the same set in R and in S; let $b$ be the number of
pairs of elements that are in different sets in R and in S. Then,
$RI=\frac{a+b}{{n\choose 2}}$
$ARI=\frac{RI-\mathbb{E}[RI]}{\max(RI)-\mathbb{E}[RI]}$

##  Appendix C Implementation Details

In this section, we discuss the implementation of the proposed algorithm. We
use PyTorch version 1.7.1 for the implementation. The ID block of the
algorithm uses the code from the implementation of ID (Z. Wu ., 2018)
available at https://github.com/zhirongw/lemniscate.pytorch. The repository
uses PyTorch version 0.3, and appropriate changes are made to use it with the
latest version of PyTorch. We use ResNet-18 and ResNet-50 blocks during our
experiments. In the ID block, the ResNet architecture is modified as follows.
The final fully connected layer consists of 128 dimensions instead of the
usual 1000 dimensions. The output of the final fully connected layer proceeds
to compute the noise contrastive estimation (NCE) loss, and the feature
representations (the layer before the fully connected layer) are fed to the
clustering part. For the clustering part, the MLP projection head $g$ consists
of a hidden layer of size 2048, followed by batch normalization and rectified
linear unit (ReLU) layers, and an output layer of size 256. The prototypes are
thus chosen to have 256 dimensions. Note that we fix the number of prototypes
to be equal to the number of ground truth classes in the dataset. It has been
shown, however, that overclustering leads to better representations (Caron .,
2020; Ji ., 20191; Asano ., 2019), and we can extend our model to include an
overclustering block with a larger set of prototypes (Ji ., 20191) and
alternate the training procedure between the blocks. We train the algorithm
for 2000 epochs on all datasets. We use the SGD optimizer with a learning rate
decay of 0.1 at prespecified epochs (600, 950, 1300, 1650, 2000) to perform
the updates for all datasets. We perform a coarse learning rate search and
find that 0.03 is the best-performing setting. We use a batch size of 128 for
all the datasets. To evaluate the cluster accuracy, we compute the cluster
assignments using MiniBatchKMeans666https://scikit-
learn.org/stable/modules/clustering.html#mini-batch-kmeans with a batch size
of 6000 and 20 random initializations.

###  C.1 Image augmentations

The different views $\mathcal{X}_{b}^{1},\mathcal{X}_{b}^{2}$ are not the same
as the views in multiview datasets (Schops ., 2017). The views referred to in
this paper correspond to different augmented views that are generated by image
augmentation techniques, such as RandomHorizontalFlip and RandomCrop. We
explain the generation process of multiple augmented views, which have been
shown to be very effective in unsupervised learning (Chen ., 2020). Indeed, it
is possible to use more than two augmented views, but we limit to the number
to two for the sake of simplicity. Caron . (2020) proposed an augmentation
technique (Multi-Crop) to use more than two views. In this work, we use the
augmentation methods used in Chen . (2020); Grill . (2020). We first crop a
random patch of the input image with a scale ranging from 0.08 to 1.0 and
resize the cropped patch to 224$\times$224 (96$\times$96 for smaller-
resolution datasets such as STL10). The resulting image is then flipped
horizontally with a probability of 0.5. We then apply color transformations,
starting by applying grayscale with a probability of 0.2 followed by randomly
changing the brightness, contrast, saturation and hue with a probability of
0.8. Then, we apply a Gaussian blur with a kernel size of 23$\times$23 and a
sigma chosen uniformly and randomly between 0.1 and 2.0. The probabilities of
applying Gaussian blur are 1.0 for view 1 and 0.5 for view 2. During the
evaluation, we resize the image such that the smaller edge of the image is of
size 256 (not required for STL-10, CIFAR-10, and CIFAR100-20), and a center
crop operation is performed with the resolution mentioned in the main paper.
We finally normalize the image channels with the mean and standard deviation
computed on ImageNet. Additionally, during training, we experiment with
applying a Sobel filter after all the image augmentation steps are performed
but before the forward pass. Applying a Sobel filter reduces the number of
channels in the input images to 2. We also experiment by augmenting the RGB
images with the output of the Sobel transform, resulting in 5-channel input
images. In both of these cases, the input channels in the first convolution
layer are modified accordingly. All image augmentations are computed using
PyTorch’s torchvision module (available in version 1.7.1).

###  C.2 Random transformations

To compute the random transformations on the embeddings $\mathbf{z}$, we
follow two techniques. We use Gaussian random projections with an output
dimensionality of $d$ and transform the embeddings $\mathbf{z}$ to the new
space with a dimensionality of $d$. In Gaussian random projections, the
projection matrix is generated by picking rows from a Gaussian distribution
such that they are orthogonal. We also use diagonal transformation (Hsu .,
2018), where we multiply $\mathbf{z}$ with a randomly generated diagonal
matrix with the same dimensions as $\mathbf{z}$. We initialize $M$ random
transformations at the beginning, and they are kept fixed throughout the
training process.

##  Appendix D Comparison with LA

LA (Zhuang ., 2019) builds on nonparametric ID (Z. Wu ., 2018) and uses a
robust clustering objective (it uses a closest neighbors set generated using
multiple runs of k-means) similar to consensus clustering to move
statistically similar points closer in the representation space and dissimilar
points farther away. By using the linear evaluation protocol on ImageNet, the
authors demonstrate that the representations learned with LA are better than
those obtained without LA. However, the performance of these features with
respect to clustering was not discussed. Since LA is similar to our work in
spirit, we perform a study on the clustering performance of the features
learned using LA777using the official PyTorch implementation available at
https://github.com/neuroailab/LocalAggregation-PyTorch/tree/master/config on
the ImageNet-10 and ImageNet-Dogs datasets. The results are presented in Table
14. Table 14: Comparison with LA on ImageNet-10  Datasets ImageNet-10
ImageNet-Dogs Method\Metrics Acc NMI ARI Acc NMI ARI LA 0.393 0.346 0.213
0.201 0.133 0.06 ConCURL 0.958 0.907 0.909 0.695 0.63 0.531 We observe that
the clustering performance of our proposed ConCURL algorithm is much better
than the clustering performance of the LA method. Note that the clustering
performance of the ID features is better than that of LA, and our algorithm
further improves upon the clustering performance of ID. One major difference
between our work and LA is the way in which we generate the ensemble. Our
approach allows us to control and measure the diversity of the ensemble, which
can be useful in making algorithm design choices. Although LA controls the
ensemble by varying the number of clustering results and the number of
clusters in each clustering result, which aptly suits the objective that LA is
solving, the resultant ensembles are limited to utilizing k-means clustering
(the authors showed that other clustering approaches were either not scalable
or not optimal). In our case, by applying feature space transformations, we
have much more freedom in generating the ensemble. We use random projections
and diagonal transformations, but there could be other transformations on the
feature space that we have not yet explored.

###  D.1 Implementation Details of LA

We train the model for 500 epochs. Since the original implementation of LA was
designed for ImageNet, we perform a hyperparameter search as follows. Using
the config file from the official PyTorch implementation, we create 36
configuration files by varying the learning rate and k-means-k. In the
original config file provided by the authors, k-means-k = 30000 (for 1.28
million images). We scale the k-means-k for ImageNet-10 (13000 images) and
ImageNet-Dogs (19500 images) accordingly and try six different values. In
particular, we try learning rates = [0.003, 0.005, 0.01, 0.03, 0.05, 0.1], and
k-means-k = [10, 15, 100, 310, 452, 500]. The number of background neighbors
is 4096, as used in the original paper. We run ResNet-18 experiments for the
full hyperparameter search (36 experiments) and evaluate the cluster metrics.
Additionally, we choose the top 5 choices of hyperparameters from above and
run ResNet-34 experiments with those parameters for both datasets. In Table
14, we present the best results obtained for each dataset. The code repository
uses a different version of ResNet (PreActResNet). Therefore, for evaluating
the clustering performance, we take the output of the layer before the final
dense layer. For ResNet-18, the output dimensions of this layer are (512,7,7),
and we take the mean along the (1,2) dimensions and use the resulting
512-dimensional vector. We compute the k-means clustering results on these
embeddings using faiss (https://github.com/facebookresearch/faiss) for the
training split of the data and compute the cluster metrics as mentioned in our
paper.

## Declarations

• Funding - Work was done at Microsoft with Microsoft’s support. • Conflict of
interests - Authors are from Microsoft and The Ohio State University. There
are no conflict of interests to disclose. • Ethics approval - Not applicable •
Consent to participate - Yes • Consent for publication - Yes • Availability of
data and materials - The datasets used are available at
https://www.cs.toronto.edu/ kriz/cifar.html, https://cs.stanford.edu/
acoates/stl10/ or can be requested at https://image-net.org/. All the trained
models, their usage is available https://github.com/JayanthRR/ConCURL_NCE. •
Code availability - Code is available
https://github.com/JayanthRR/ConCURL_NCE. • Authors’ contributions - All
authors - Aniket Anand Deshmukh, Jayanth Reddy Regatti, Eren Manavoglu and
Urun Dogan contributed to this work and were essential to complete this
submission.

## References

  *   * Asano . (2019) asano2019selfAsano, Y.M., Rupprecht, C. Vedaldi, A.  2019\.  Self-labelling via simultaneous clustering and representation learning Self-labelling via simultaneous clustering and representation learning.  arXiv preprint arXiv:1911.05371.
  * Bengio . (2007) Bengio2007GreedyBengio, Y., Lamblin, P., Popovici, D., Larochelle, H. Montreal, U.  2007\.  Greedy layer-wise training of deep networks Greedy layer-wise training of deep networks.  NeurIPS19153-160.
  * Cai . (2009) NMFCai, D., He, X., Wang, X., Bao, H. Han, J.  2009\.  Locality Preserving Nonnegative Matrix Factorization Locality preserving nonnegative matrix factorization.  IJCAI Ijcai ( 1010–1015).  http://ijcai.org/Proceedings/09/Papers/171.pdf
  * Caron . (20181) caron2018deepCaron, M., Bojanowski, P., Joulin, A. Douze, M.  20181\.  Deep clustering for unsupervised learning of visual features Deep clustering for unsupervised learning of visual features.  Proceedings of the European Conference on Computer Vision (ECCV) Proceedings of the european conference on computer vision (eccv) ( 132–149).
  * Caron . (20182) vf2018Caron, M., Bojanowski, P., Joulin, A. Douze, M.  20182\.  Deep Clustering for Unsupervised Learning of Visual Features. Deep clustering for unsupervised learning of visual features.  ECCV Eccv ( 11218, 139-156).
  * Caron . (2020) caron2020unsupervisedCaron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P. Joulin, A.  2020\.  Unsupervised learning of visual features by contrasting cluster assignments Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882.
  * Chang . (2019) chang2019deepChang, J., Guo, Y., Wang, L., Meng, G., Xiang, S. Pan, C.  2019\.  Deep Discriminative Clustering Analysis. Deep discriminative clustering analysis.
  * Chang . (2017) Chang_2017_ICCVChang, J., Wang, L., Meng, G., Xiang, S. Pan, C.  2017Oct.  Deep Adaptive Image Clustering Deep adaptive image clustering.  The IEEE International Conference on Computer Vision (ICCV). The ieee international conference on computer vision (iccv).
  * Chang . (2017) DAIC2017Chang, J., Wang, L., Meng, G., Xiang, S. Pan, C.  2017Oct.  Deep Adaptive Image Clustering Deep adaptive image clustering.  ICCV Iccv ( 5880-5888).  10.1109/ICCV.2017.626
  * Chen . (2020) chen2020simpleChen, T., Kornblith, S., Norouzi, M. Hinton, G.  2020\.  A simple framework for contrastive learning of visual representations A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709.
  * Cuturi (2013) cuturi2013sinkhornCuturi, M.  2013\.  Sinkhorn distances: Lightspeed computation of optimal transport Sinkhorn distances: Lightspeed computation of optimal transport.  Advances in neural information processing systems Advances in neural information processing systems ( 2292–2300).
  * Deng . (2009) deng2009imagenetDeng, J., Dong, W., Socher, R., Li, LJ., Li, K. Fei-Fei, L.  2009\.  Imagenet: A large-scale hierarchical image database Imagenet: A large-scale hierarchical image database.  2009 IEEE conference on computer vision and pattern recognition 2009 ieee conference on computer vision and pattern recognition ( 248–255).
  * Fern Brodley (2003) fern2003randomFern, X.Z. Brodley, C.E.  2003\.  Random projection for high dimensional data clustering: A cluster ensemble approach Random projection for high dimensional data clustering: A cluster ensemble approach.  Proceedings of the 20th international conference on machine learning (ICML-03) Proceedings of the 20th international conference on machine learning (icml-03) ( 186–193).
  * Franti . (2006) Pasi2006FastFranti, P., Virmajoki, O. Hautamaki, V.  2006Nov.  Fast Agglomerative Clustering Using a k-Nearest Neighbor Graph Fast agglomerative clustering using a k-nearest neighbor graph.  IEEE Transactions on Pattern Analysis and Machine Intelligence28111875-1881.
  * Fred Jain (2005) fred2005combiningFred, A.L. Jain, A.K.  2005\.  Combining multiple clusterings using evidence accumulation Combining multiple clusterings using evidence accumulation.  IEEE transactions on pattern analysis and machine intelligence276835–850.
  * Frey Dueck (2007) frey2007clusteringFrey, B.J. Dueck, D.  2007\.  Clustering by passing messages between data points Clustering by passing messages between data points.  science3155814972–976.
  * Ghosh Acharya (2011) ghosh2011clusterGhosh, J. Acharya, A.  2011\.  Cluster ensembles Cluster ensembles.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery14305–315.
  * Grill . (2020) grill2020bootstrapGrill, JB., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E.others  2020\.  Bootstrap your own latent: A new approach to self-supervised learning Bootstrap your own latent: A new approach to self-supervised learning.  arXiv preprint arXiv:2006.07733.
  * Haeusser . (2019) ADC2019Haeusser, P., Plapp, J., Golkov, V., Aljalbout, E. Cremers, D.  2019\.  Associative Deep Clustering: Training a Classification Network with No Labels Associative deep clustering: Training a classification network with no labels.  T. Brox, A. Bruhn M. Fritz (), Pattern Recognition Pattern recognition ( 18–32).  ChamSpringer International Publishing.
  * He . (2019) he2019mocoHe, K., Fan, H., Wu, Y., Xie, S. Girshick, R.  2019\.  Momentum Contrast for Unsupervised Visual Representation Learning Momentum contrast for unsupervised visual representation learning.  arXiv preprint arXiv:1911.05722.
  * He . (2020) he2020momentumHe, K., Fan, H., Wu, Y., Xie, S. Girshick, R.  2020\.  Momentum contrast for unsupervised visual representation learning Momentum contrast for unsupervised visual representation learning.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Proceedings of the ieee/cvf conference on computer vision and pattern recognition ( 9729–9738).
  * Hsu . (2018) hsu2018unsupervisedHsu, K., Levine, S. Finn, C.  2018\.  Unsupervised learning via meta-learning Unsupervised learning via meta-learning.  arXiv preprint arXiv:1810.02334.
  * Huang . (20201) huang2020deepHuang, J., Gong, S. Zhu, X.  20201\.  Deep Semantic Clustering by Partition Confidence Maximisation Deep semantic clustering by partition confidence maximisation.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Proceedings of the ieee/cvf conference on computer vision and pattern recognition ( 8849–8858).
  * Huang . (20202) Huang_2020_CVPRHuang, J., Gong, S. Zhu, X.  20202June.  Deep Semantic Clustering by Partition Confidence Maximisation Deep semantic clustering by partition confidence maximisation.  CVPR. Cvpr.
  * Jain Dubes (1988) jain1988algorithmsJain, A.K. Dubes, R.C.  1988\.  Algorithms for clustering data Algorithms for clustering data.  Prentice-Hall, Inc.
  * Jain . (1999) jain1999dataJain, A.K., Murty, M.N. Flynn, P.J.  1999\.  Data clustering: a review Data clustering: a review.  ACM computing surveys (CSUR)313264–323.
  * Ji . (20191) ji2019invariantJi, X., Henriques, J.F. Vedaldi, A.  20191\.  Invariant information clustering for unsupervised image classification and segmentation Invariant information clustering for unsupervised image classification and segmentation.  Proceedings of the IEEE International Conference on Computer Vision Proceedings of the ieee international conference on computer vision ( 9865–9874).
  * Ji . (20192) IIC2019Ji, X., Henriques, J.F. Vedaldi, A.  20192October.  Invariant Information Clustering for Unsupervised Image Classification and Segmentation Invariant information clustering for unsupervised image classification and segmentation.  ICCV. Iccv.
  * Kuhn (1955) kuhn1955hungarianKuhn, H.W.  1955\.  The Hungarian method for the assignment problem The hungarian method for the assignment problem.  Naval research logistics quarterly21-283–97.
  * Kuhn (1956) kuhn1956variantsKuhn, H.W.  1956\.  Variants of the Hungarian method for assignment problems Variants of the hungarian method for assignment problems.  Naval Research Logistics Quarterly34253–258.
  * LeCun . (2015) lecun2015deepLeCun, Y., Bengio, Y. Hinton, G.  2015\.  Deep learning Deep learning.  nature5217553436–444.
  * Li . (2021) cons_clusLi, Y., Hu, P., Liu, Z., Peng, D., Zhou, J.T. Peng, X.  2021\.  Contrastive Clustering Contrastive clustering.  AAAI. Aaai.
  * Macqueen (1967) kmeans1967Macqueen, J.  1967\.  Some methods for classification and analysis of multivariate observations Some methods for classification and analysis of multivariate observations.  In 5-th Berkeley Symposium on Mathematical Statistics and Probability In 5-th berkeley symposium on mathematical statistics and probability ( 281–297).
  * Masulli Schenone (1999) masulli1999fuzzyMasulli, F. Schenone, A.  1999\.  A fuzzy clustering based segmentation system as support to diagnosis in medical imaging A fuzzy clustering based segmentation system as support to diagnosis in medical imaging.  Artificial intelligence in medicine162129–147.
  * Ng . (2002) Spectral2002Ng, A.Y., Jordan, M.I. Weiss, Y.  2002\.  On Spectral Clustering: Analysis and an algorithm On spectral clustering: Analysis and an algorithm.  T.G. Dietterich, S. Becker Z. Ghahramani (), NeurIPS Neurips ( 849–856).  MIT Press.  http://papers.nips.cc/paper/2092-on-spectral-clustering-analysis-and-an-algorithm.pdf
  * Niu . (2021) niu2021spiceNiu, C., Shan, H. Wang, G.  2021\.  Spice: Semantic pseudo-labeling for image clustering Spice: Semantic pseudo-labeling for image clustering.  arXiv preprint arXiv:2103.09382.
  * Niu . (20201) niu2020gatclusterNiu, C., Zhang, J., Wang, G. Liang, J.  20201\.  GATCluster: Self-Supervised Gaussian-Attention Network for Image Clustering Gatcluster: Self-supervised gaussian-attention network for image clustering.  735–751.
  * Niu . (20202) gatclusterNiu, C., Zhang, J., Wang, G. Liang, J.  20202\.  GATCluster: Self-supervised Gaussian-Attention Network for Image Clustering Gatcluster: Self-supervised gaussian-attention network for image clustering.  ECCV Eccv ( 735–751).
  * Regatti . (2021) regatti2020consensusRegatti, J.R., Deshmukh, A.A., Manavoglu, E. Dogan, U.  2021\.  Consensus Clustering with Unsupervised Representation Learning Consensus clustering with unsupervised representation learning.  International Joint Conference on Neural Networks (IJCNN) arXiv preprint arXiv:2010.01245.
  * Schops . (2017) schops2017multiSchops, T., Schonberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M. Geiger, A.  2017\.  A multi-view stereo benchmark with high-resolution images and multi-camera videos A multi-view stereo benchmark with high-resolution images and multi-camera videos.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 3260–3269).
  * Shah Koltun (2018) shah2018deepShah, S.A. Koltun, V.  2018\.  Deep continuous clustering Deep continuous clustering.  arXiv preprint arXiv:1803.01449.
  * Shorten Khoshgoftaar (2019) shorten2019surveyShorten, C. Khoshgoftaar, T.M.  2019\.  A survey on image data augmentation for deep learning A survey on image data augmentation for deep learning.  Journal of Big Data611–48.
  * Strehl Ghosh (2002) strehl2002clusterStrehl, A. Ghosh, J.  2002\.  Cluster ensembles—a knowledge reuse framework for combining multiple partitions Cluster ensembles—a knowledge reuse framework for combining multiple partitions.  Journal of machine learning research3Dec583–617.
  * Tao . (2021) taoclusteringTao, Y., Takagi, K. Nakata, K.  2021\.  CLUSTERING-FRIENDLY REPRESENTATION LEARN-ING VIA INSTANCE DISCRIMINATION AND FEATURE DECORRELATION Clustering-friendly representation learn-ing via instance discrimination and feature decorrelation.  arXiv preprint arXiv:2106.00131.
  * Tian . (2020) tian2020makesTian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C. Isola, P.  2020\.  What makes for good views for contrastive learning What makes for good views for contrastive learning.  arXiv preprint arXiv:2005.10243.
  * Van Gansbeke . (2020) van2020scanVan Gansbeke, W., Vandenhende, S., Georgoulis, S., Proesmans, M. Van Gool, L.  2020\.  Scan: Learning to classify images without labels Scan: Learning to classify images without labels.
  * Vincent . (2010) SDAE2010Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y. Manzagol, P.A.  2010\.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  Journal of Machine Learning Research11123371-3408.
  * J. Wu . (20191) wu2019deepWu, J., Long, K., Wang, F., Qian, C., Li, C., Lin, Z. Zha, H.  20191\.  Deep comprehensive correlation mining for image clustering Deep comprehensive correlation mining for image clustering.  Proceedings of the IEEE International Conference on Computer Vision Proceedings of the ieee international conference on computer vision ( 8150–8159).
  * J. Wu . (20192) Wu_2019_ICCVWu, J., Long, K., Wang, F., Qian, C., Li, C., Lin, Z. Zha, H.  20192October.  Deep Comprehensive Correlation Mining for Image Clustering Deep comprehensive correlation mining for image clustering.  ICCV. Iccv.
  * Z. Wu . (2018) wu2018unsupervisedWu, Z., Xiong, Y., Yu, S.X. Lin, D.  2018\.  Unsupervised feature learning via non-parametric instance discrimination Unsupervised feature learning via non-parametric instance discrimination.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 3733–3742).
  * Xie . (20161) xie2016unsupervisedXie, J., Girshick, R. Farhadi, A.  20161\.  Unsupervised deep embedding for clustering analysis Unsupervised deep embedding for clustering analysis.  International conference on machine learning International conference on machine learning ( 478–487).
  * Xie . (20162) Xie2016Xie, J., Girshick, R. Farhadi, A.  20162\.  Unsupervised Deep Embedding for Clustering Analysis Unsupervised deep embedding for clustering analysis.  ICML Icml ( 478–487).  JMLR.org.  http://dl.acm.org/citation.cfm?id=3045390.3045442
  * Xu Wunsch (2005) xu2005surveyXu, R. Wunsch, D.  2005\.  Survey of clustering algorithms Survey of clustering algorithms.  IEEE Transactions on neural networks163645–678.
  * Yang . (2016) Yang2016JointYang, J., Parikh, D. Batra, D.  2016\.  Joint Unsupervised Learning of Deep Representations and Image Clusters Joint unsupervised learning of deep representations and image clusters.  CVPR. Cvpr.
  * Zeiler . (2010) Zeiler2010DeconvolutionalZeiler, M.D., Krishnan, D., Taylor, G.W. Fergus, R.  2010\.  Deconvolutional networks Deconvolutional networks.  Computer Vision and Pattern Recognition. Computer vision and pattern recognition.
  * Zhuang . (2019) zhuang2019localZhuang, C., Zhai, A.L. Yamins, D.  2019\.  Local aggregation for unsupervised learning of visual embeddings Local aggregation for unsupervised learning of visual embeddings.  Proceedings of the IEEE International Conference on Computer Vision Proceedings of the ieee international conference on computer vision ( 6002–6012).