#  UCRDNet: Unsupervised Collaborative Representation Deep Network for
Clustering

Jielei Chu, , Hongjun Wang, Jing Liu,
Zeng Yu, Zhiguo Gong, , Tianrui Li Jielei Chu, Hongjun Wang (the corresponding
author), Zeng Yu, Tianrui Li are with the Institute of Artificial
Intelligence, School of Information Science and Technology, Southwest Jiaotong
University, Chengdu 611756, China. Tianrui Li is also with National
Engineering Laboratory of Integrated Transportation Big Data Application
Technology, Southwest Jiaotong University, Chengdu 611756, China. e-mails:
{jieleichu, wanghongjun, zyu<EMAIL_ADDRESS>Liu is with the School
of Business, Sichuan University, Sichuan, 610065, Chengdu, China. e-mail:
<EMAIL_ADDRESS>Gong is with the State Key Laboratory of Internet of
Things for Smart City, Department of Computer and Information Science,
University of Macau, Macau, China. Email<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose an unsupervised collaborative representation deep
network (UCRDNet) which consists of novel collaborative representation RBM
(crRBM) and collaborative representation GRBM (crGRBM). The UCRDNet is a novel
deep collaborative feature extractor for exploring more sophisticated
probabilistic models of real-valued and binary data. Unlike traditional
representation methods, one similarity relation between the input instances
and another similarity relation between the features of the input instances
are collaboratively fused together in the representation process of the crGRBM
and crRBM models. Here, we use the Locality Sensitive Hashing (LSH) method to
divide the input instance matrix into many mini blocks which contain similar
instance and local features. Then, we expect the hidden layer feature units of
each block gather to block center as much as possible in the training
processes of the crRBM and crGRBM. Hence, the correlations between the
instances and features as collaborative relations are fused in the hidden
layer features. In the experiments, we use K-means and Spectral Clustering
(SC) algorithms based on four contrast deep networks to verify the deep
collaborative representation capability of the UCRDNet architecture. One
architecture of the UCRDNet is composed with a crGRBM and two crRBMs for
modeling real-valued data and another architecture of it is composed with
three crRBMs for modeling binary data. The experimental results show that the
proposed UCRDNet has more outstanding performance than the Autoencoder and
DeepFS deep networks (without collaborative representation strategy) for
unsupervised clustering on the MSRA-MM2.0 and UCI datasets. Furthermore, the
proposed UCRDNet shows more excellent collaborative representation
capabilities than the CDL deep collaborative networks for unsupervised
clustering. The hidden collaborative features of the proposed UCRDNet also
show powerful generalization ability for different algorithms.

###### Index Terms:

restricted Boltzmann machine; deep collaborative representation; feature
learning; unsupervised clustering.

##  I Introduction

Recently collaborative representation, in particular, deep collaborative
representation as an effective feature learning methods for classification [1,
2], target detection [3], face recognition [4, 5, 6] and deep collaborative
filtering [7, 8] etc. has attracted much attentions. It originates the
influential sparse representation-based classification (SRC) [9]. Akhtar et
al. [10] explored the source of collaborative representation superior
performance. Up to now, there is no research on deep collaborative
representation for unsupervised clustering. In this paper, we focus on
exploring the capability of deep collaborative representation for unsupervised
clustering by establishing novel unsupervised collaborative representation
deep architectures.
There are various deep collaborative models which have been successfully
applied in practice [11, 12, 13, 14, 15]. For multimodal brain development
study, the traditional methods can not obtain complex nonlinear relationship
between multiple data. So, Hu et al. [11] proposed a novel neural network
framework to extract complex crossdata relationships. They first use a deep
neural network to represent original data, then design a collaborative
learning layer using collaborative regression to seek their correlations and
link the data representation with phenotypical information. But, the deep
learning process and collaborative learning of it belong to two separate
processes. For video person re-identification, Zhang et al. [12] developed a
novel deep architecture termed Self-and-Collaborative Attention Network (SCAN)
based on the convolutional neural networks (CNN). The SCAN contains two kinds
of subnetworks: a self attention subnetwork (SAN) to enhance feature
representation and a collaborative attention subnetwork (CAN) to select frames
from probe based on the representation of the other one. In [13], Liu et al.
presented a collaborative deconvolutional neural network (C-DCNN) to jointly
model two problems of single-view depth estimation and semantic segmentation
in computer vision. Two deconvolutional neural networks (DCNNs) compose a
C-DCNN and the depth features and semantic are combined in a unified deep
network. In the training process, the depth features and semantic are
integrated together to benefit each other. For face photo-sketch synthesis,
Zhu et al. [14] proposed a deep collaborative framework with two opposite
networks constrained by a collaborative loss which can utilize the mutual
interaction of two opposite mappings. For social image understanding task, Li
et al. [15] developed a Deep Collaborative Embedding (DCE) model which
incorporates the collaborative factor analysis and end-to-end learning in one
unified framework for the optimal compatibility of latent space discovery and
representation learning. The DCE model integrates the tag correlation, image
correlation and weakly-supervised image-tag correlation simultaneously and
seamlessly to collaboratively explore the rich information of social images.
For deep collaborative filtering, Fu et al. [7] developed a novel deep
learning method to promote the effectiveness of intelligent recommendation
system by understanding the items beforehand and users. The low-dimensional
vectors of items and users are learned separately in the initial stage. Then,
in the prediction stage, the interaction between item and user is simulated by
a feed-forward neural networks. Yue et al. [8] proposed a novel denoising
method via deep fusion of convolutional and collaborative filtering for high
ISO JPEG images. It fuses the strengths of collaborative and convolutional
filtering using a deep CNN.
Classic RBM [16, 17], GRBM [18] and their variants [19, 20, 21, 22, 23, 24,
25, 26] have been proven to be some excellent feature representation models.
In our previous work [27], we also proposed a powerful variant of GRBM called
pcGRBM for semi-supervised feature learning. To enhance And further more, many
deep networks [28, 29, 30, 31, 32, 33] based on classical RBM and variants are
developed in practical applications (e.g. Autoencoder [34] and DeepFS [35]).
The Autoencoder as a classic deep model consists of a stack of RBMs for
modeling binary data. It also is used to model real-valued data by replacing
binary visible layer units with Gaussian linear visible units. The DeepFS is a
feature selection model based on a Deep Boltzmann Machine which has powerful
capability of removing irrelevant features from input data. But none of them
have collaborative representation capabilities. By adding a collaborative
strategy, Wang et al. [36] proposed a popular hierarchical Bayesian model
called collaborative deep learning (CDL) which jointly performs collaborative
filtering and deep representation learning.
Our motivation is that how to explore novel variants of RBM and GRBM by means
of unsupervised collaborative strategy to make them have collaborative
representation capabilities. On this basis, we look forward to establish novel
deep collaborative network architectures with superior performance of deep
collaborative representation for unsupervised clustering.
In this paper, we propose an unsupervised collaborative representation deep
network (UCRDNet) which consists of novel variant of RBM called collaborative
representation RBM (crRBM) and novel variant of GRBM called collaborative
representation GRBM (crGRBM). The UCRDNet as a novel deep collaborative
feature extractor for exploring more sophisticated probabilistic models of
real-valued and binary data. One novel architecture of it is composed with a
crGRBM and two crRBMs which has Gaussian linear visible layer units for
modeling real-valued data and another novel architecture of it is composed
with three crRBMs which has with binary visible layer units for modeling
binary data. In the representation process of the crGRBM and crRBM models, one
similarity relation between the input instances and another similarity
relation between the features of the input instances are collaboratively fused
together. Here, we use the Locality Sensitive Hashing (LSH) [37] method to
divide the input instance matrix into many mini blocks which contain similar
instance and local features. Then, we expect the hidden layer feature units of
each block gather to block center as much as possible in the training
processes of crRBM and crGRBM. Hence, the correlations between the instances
and features as collaborative relations are fused in the hidden layer
features.
To best of our knowledge, our work is the first one to study unsupervised deep
collaborative representation for unsupervised clustering. Our major
contributions of this paper are summarized as follows:

  * •

One novel variant of RBM called collaborative representation RBM (CRBM) and
another novel variant of GRBM called collaborative representation GRBM
(crGRBM) are proposed by fusing the collaborative relations between the
instances and local features as in its encoding process.

  * •

A novel unsupervised collaborative representation deep network (UCRDNet) is
proposed with three hidden layers. For modeling real-valued data, one
architecture of the UCRDNet is composed of a crGRBM and two crRBMs which has
Gaussian linear visible units. For modeling binary data, another architecture
of the UCRDNet is composed of with three crRBMs which has binary visible
units.

  * •

The experiments demonstrate that the proposed UCRDNet has more powerful deep
collaborative representation capability than the Autoencoder, DeepFS deep
networks and CDL deep collaborative networks on real-valued and binary
datasets. Furthermore, the hidden collaborative features of the UCRDNet show
powerful generalization ability for different algorithms.

The remaining of the paper is organized as follows. The theoretical background
is described in Section II. The unsupervised collaborative representation deep
network (UCRDNet) is developed in Section III. The experimental results are
shown in Section IV. Finally, our contributions are summarized in Section V.

##  II Theoretical Background

###  II-A Restricted Boltzmann Machine

For classic RBMs [16], its architecture is a shallow two-layer structure,
which consists of a binary visible and hidden layer. The energy function of
RBMs is defined by:

$E(\textbf{{v}},\textbf{{h}})=-\sum\limits_{i\in
visibles}a_{i}v_{i}-\sum\limits_{j\in
hiddens}b_{j}h_{j}-\sum\limits_{i,j}v_{i}h_{j}w_{ij},$ (1)

where v and h are the visible and hidden layer vectors, respectively, $v_{i}$
and $h_{j}$ are the binary visible and hidden units, respectively, $w_{ij}$ is
the symmetric connection weight between them, $a_{i}$ and $b_{j}$ are the
biases of visible and hidden units, respectively.
Given a visible vector v, the binary state $h_{j}$ is equal to 1 with
probability

$p(h_{j}=1|\texttt{{v}})=\sigma(b_{j}+\sum\limits_{i}v_{i}w_{ij}),$ (2)

where $\sigma(x)=\frac{1}{1+\texttt{exp}(-x))}$, which is a logistic sigmoid
function.
Similarly, given a hidden vector h, an unbiased sample of the binary state
$v_{i}$ is equal to 1 with probability

$p(v_{i}=1|\texttt{{h}})=\sigma(a_{i}+\sum\limits_{j}h_{j}w_{ij}).$ (3)

Because it is difficult to get an unbiased sample of $<v_{i}h_{j}>_{model}$,
Hinton proposed a faster learning algorithm by Contrastive Divergence (CD)
[17], [19] method. Then the update rules of parameters is given by:

$\Delta w_{i}{j}=\varepsilon(<v_{i}h_{j}>_{data}-<v_{i}h_{j}>_{recon}),$ (4)
$\Delta a_{i}=\varepsilon(<v_{i}>_{data}-<v_{i}>_{recon}),$ (5) $\Delta
b_{j}=\varepsilon(<h_{j}>_{data}-<h_{j}>_{recon}),$ (6)

where $\varepsilon$ is a learning rate.

###  II-B Gaussian Linear Visible Units

For modeling real-valued data, the binary visible units are replaced by
Gaussian linear visible units. The energy function becomes:

$\displaystyle E(\textbf{{v}},\textbf{{h}})=$ $\displaystyle-\sum\limits_{i\in
visibles}\frac{(v_{i}-a_{i})^{2}}{2\sigma_{i}^{2}}-\sum\limits_{j\in
hiddens}b_{j}h_{j}$ (7)
$\displaystyle-\sum\limits_{i,j}\frac{v_{i}}{\sigma_{i}}h_{j}w_{ij},$

where $\sigma_{i}$ is the standard deviation of the Gaussian noise for visible
unit $i$. It is difficult to use CD method to learn the variance of the noise.
In practice, we normalise the original data to have unit variance and zero
mean. So, the reconstructed result of a Gaussian linear visible unit is equal
to the input from hidden binary units plus the bias.

###  II-C Locality Sensitive Hashing

The Locality Sensitive Hashing (LSH) [37] exploits the probability that two
similar samples likely collide by mapping with a weak hash function. In fact,
the probability of the collision is proportional to their similarity. One
classic hash function is the Minwise Independent Permutation (Minhash) [38]
which defines the probability of collision is proportional to the Jaccard
similarity of two hashed objects. The Jaccard similarity varies from 0 to 1.
The value of it is 1 means that the two hashed objects are equal.

##  III The Unsupervised Collaborative Representation Deep Network

In this section, we firstly present the key basis of unsupervised
collaborative representation deep network (UCRDnet) that is the crRBM and
crGRBM models (novel variants of RBM and GRBM). Then, we present the
inference, learning algorithm and complexity analysis of the crRBM model.
Finally, we propose two UCRDnet deep architectures based on the crRBM and
crGRBM models for modeling real-valued and binary data, respectively.

###  III-A The crRBM and crGRBM Models

In this section, we present the key basics of the UCRDNet deep architecture
that is the crRBM and crGRBM models. Here, we use the LSH [37] method to
divide the input instance matrix into many mini blocks which contain similar
instance and local features. Then, we expect the hidden layer feature units of
each block gather to block center as much as possible in the training
processes of crRBM and crGRBM models. By this way, the correlations between
the instances and features (collaborative relations) are fused in the hidden
layer features. Due to the same mapping relations from the visible lay units
to hidden layer units (sigmoid transformation) between the crRBM and crGRBM
models, we only present the crRBM model and its inference and learning
algorithm.
Let
$\mathcal{D}=\\{\textbf{{{v}}}_{1},\textbf{{{v}}}_{2},\cdots,\textbf{{{v}}}_{N}\\}$
be an original data set with $N$ vectors and $M$ features of each vector. The
visible layer vector
$\textbf{{{v}}}_{s}=(v_{s1},v_{s2},\cdots,v_{si},\cdots,v_{sM})$,
$(i=1,2,\cdots,M$, and $s=1,2,\cdots,N)$. The hidden layer vector
$\textbf{{h}}_{s}=(h_{s1},h_{s2},\cdots,h_{sj},\cdots,h_{sM^{\prime}})$,
$(j=1,2,\cdots,M^{\prime}$, and $s=1,2,\cdots,N)$. The reconstructed visible
layer vector
$\textbf{{v}}_{s}^{(r)}=(v_{s1}^{(r)},v_{s2}^{(r)},\cdots,v_{si}^{(r)},\cdots,v_{sM}^{(r)})$,
$(i=1,2,\cdots,M$, and $s=1,2,\cdots,N)$. The reconstructed hidden layer
vector
$\textbf{{h}}_{s}^{(r)}=(h_{s1}^{(r)},h_{s2}^{(r)},\cdots,h_{sj}^{(r)},\cdots,h_{sM}^{(r)})$,
$(j=1,2,\cdots,M^{\prime}$, and $s=1,2,\cdots,N)$. The matrix
$(\textbf{{v}}_{1}^{T}\textbf{{v}}_{2}^{T}\cdots\textbf{{v}}_{N}^{T})^{T}$ is
partitioned into $K$ row clusters by LSH and each cluster has a serial number
set of vectors $\Re_{k},(k=1,2,\cdots,K$ and
$\Re_{1}\cup\Re_{2}\cdots\cup\Re_{K}=\\{1,2,\cdots,N\\})$. Simultaneously, the
matrix is partitioned into $L$ column clusters by LSH and each cluster has a
serial number set of vectors $\ell_{l},(l=1,2,\cdots,L$ and
$\ell_{1}\cup\ell_{2}\cdots\cup\ell_{K}=\\{1,2,\cdots,M\\})$. So, the matrix
is divided into $K\times L$ blocks.
Based on the expectations of our collaborative representation method and the
training objective of classic RBM, our novel objective function takes the
form:

$\displaystyle G(\texttt{V},\theta)=$ (8)
$\displaystyle-\frac{\eta}{N}\sum\limits_{\textbf{{v}}_{i}}\mathbf{log}^{p(\textbf{{v}}_{i};\theta)}+\frac{(1-\eta)}{K\times
L}\Bigg{[}\sum\limits_{k=1}^{K}\sum\limits_{l=1}^{L}\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}d(h_{st},u_{kl})$
$\displaystyle+\sum\limits_{k=1}^{K}\sum\limits_{l=1}^{L}\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}d(h_{st}^{(r)},u_{kl}^{(r)})\Bigg{]},$

where
$\theta=\\{\mathbf{\textbf{a}},\mathbf{\textbf{b}},\mathbf{\textbf{{W}}}\\}$
are the model parameters, $\eta$ is an adjusting parameter, $d(h_{st},u_{kl})$
and $d(h_{st}^{(r)},u_{kl}^{(r)})$ are the Bregman divergences [39] [40]
distances, which are defined as: $d(h_{st},u_{kl})=(h_{st}-u_{kl})^{2}$ and
$d(h_{st}^{(r)},u_{kl}^{(r)})=(h_{st}^{(r)}-u_{kl}^{(r)})^{2}$, respectively.
$u_{kl}$ and $u_{kl}^{(r)}$ are the centers of block ($\Re_{k},\ell_{l}$) in
hidden layer and reconstructed hidden layer, respectively. They take the form:

$\displaystyle
u_{kl}=\frac{\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}h_{st}}{|\Re_{k}||\ell_{l}|},u_{kl}^{(r)}=\frac{\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}h_{st}^{(r)}}{|\Re_{k}||\ell_{l}|}.$
(9)

###  III-B The Inference

In this subsection, we use the gradient descent algorithm to obtain the update
rules of the parameters of the crRBM model. The detailed inference is shown as
follows.
Suppose that

$\displaystyle
C_{data}=\sum\limits_{k=1}^{K}\sum\limits_{l=1}^{L}\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}(h_{st}-u_{kl})^{2},$
(10) $\displaystyle
C_{recon}=\sum\limits_{k=1}^{K}\sum\limits_{l=1}^{L}\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}(h_{st}^{(r)}-u_{kl}^{(r)})^{2}.$
(11)

Using the introduced variables $C_{data}$ and $C_{recon}$, the objective
function have another concise equivalent form:

$\displaystyle G(\texttt{V},\theta)=$ (12)
$\displaystyle-\frac{\eta}{N}\sum\limits_{\textbf{v}_{i}}\mathbf{log}^{p(\textbf{v}_{i};\theta)}+\frac{(1-\eta)}{K\times
L}(C_{data}+C_{recon}).$

The following crucial problem is that how to solve this multi-objective
optimization problem. For the average log-likelihood
$\frac{\eta}{N}\sum\limits_{\textbf{v}_{i}}\mathbf{log}^{p(\textbf{v}_{i};\theta)}$,
the CD method was presented to approximately follow the gradient of two
divergences
$\texttt{CD}_{n}=\texttt{KL}(p_{0}||p_{\infty})-\texttt{KL}(p_{n}||p_{\infty})$
to avoid enormous difficulties of the log-likelihood gradient computing.
Normally, we run the Markov chain from the data distribution $p_{0}$ to
$p_{1}$ (one step) in CD learning. So, the following key task is how to obtain
the approximative gradient of $C_{data}+C_{recon}$.
Suppose that $J_{data}=h_{st}-u_{kl}$, $J_{recon}=h_{st}^{(r)}-u_{kl}^{(r)}$,
then

$\displaystyle J_{data}$
$\displaystyle=h_{st}-\frac{\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}h_{st}}{|\Re_{k}||\ell_{l}|}=\sigma\big{(}\sum\limits_{m=1}^{M}v_{sm}w_{mt}+b_{st}\big{)}$
(13)
$\displaystyle-\frac{\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}\sigma\big{(}\sum\limits_{m=1}^{M}v_{sm}w_{mt}+b_{st}\big{)}}{|\Re_{k}||\ell_{l}|}$

and

$\displaystyle J_{recon}$
$\displaystyle=h_{st}^{(r)}-\frac{\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}h_{st}^{(r)}}{|\Re_{k}||\ell_{l}|}=\sigma\big{(}\sum\limits_{m=1}^{M}v_{sm}^{(r)}w_{mt}+b_{st}^{(r)}\big{)}$
(14)
$\displaystyle-\frac{\sum\limits_{s\in\Re_{k}}\sum\limits_{t\in\ell_{l}}\sigma\big{(}\sum\limits_{m=1}^{M}v_{sm}^{(r)}w_{mt}+b_{st}^{(r)}\big{)}}{|\Re_{k}||\ell_{l}|},$

where $\sigma$ is a sigmoid function.
When $t=j\in\ell_{l}$, the partial derivative of $J_{data}$ is given by:

$\displaystyle\frac{\partial J_{data}}{\partial w_{ij}}=$
$\displaystyle\frac{e^{-\big{(}\sum\limits_{m=1}^{M}v_{sm}w_{mj}+b_{sj}\big{)}}v_{si}}{\bigg{[}1+e^{-(\sum\limits_{m=1}^{M}v_{sm}w_{mj}+b_{sj})}\bigg{]}^{2}}$
(15)
$\displaystyle-\frac{\sum\limits_{s\in\Re_{k}}\frac{e^{-\big{(}\sum\limits_{m=1}^{M}v_{sm}w_{mj}+b_{sj}\big{)}}v_{si}}{\bigg{[}1+e^{-(\sum\limits_{m=1}^{M}v_{sm}w_{mj}+b_{sj})}\bigg{]}^{2}}}{|\Re_{k}|}$
$\displaystyle=$
$\displaystyle(1-h_{sj})h_{sj}v_{si}-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj})h_{sj}v_{si}}{|\Re_{k}|}$

Obviously, if $t\neq j$, then $\frac{\partial J_{data}}{\partial w_{ij}}=0$.
Similarly, if $t=j\in\ell_{l}$, the partial derivative of $J_{recon}$ is given
by:

$\displaystyle\frac{\partial J_{recon}}{\partial
w_{ij}}=(1-h_{sj}^{(r)})h_{sj}^{(r)}v_{si}^{(r)}-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj}^{(r)})h_{sj}^{(r)}v_{si}^{(r)}}{|\Re_{k}|}$
(16)

As for model parameter b, if $t=j$, the partial derivative takes the forms:

$\displaystyle\frac{\partial J_{data}}{\partial
b_{j}}=(1-h_{sj})h_{sj}-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj})h_{sj}}{|\Re_{k}|},$
(17) $\displaystyle\frac{\partial J_{recon}}{\partial
b_{j}}=(1-h_{sj}^{(r)})h_{sj}^{(r)}-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj}^{(r)})h_{sj}^{(r)}}{|\Re_{k}|}.$

It is obvious that model parameter a is independent of $J_{data}$ and
$J_{recon}$. So, we can obtain that $\frac{\partial J_{data}}{\partial
a_{i}}=0$ and $\frac{\partial J_{recon}}{\partial a_{i}}=0$. Then, the partial
derivative of the $C_{data}$ in terms of $w_{ij}$ takes the form:

$\displaystyle\frac{\partial C_{data}}{\partial w_{ij}}=$ $\displaystyle
2\sum\limits_{k=1}^{K}\sum\limits_{s\in\Re_{k}}\big{(}h_{sj}-\frac{\sum\limits_{s\in\Re_{k}}h_{sj}}{|\Re_{k}|}\big{)}\Bigg{[}(1-h_{sj})h_{sj}v_{si}$
(18)
$\displaystyle-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj})h_{sj}v_{si}}{|\Re_{k}|}\Bigg{]}.$

And the partial derivative of the $C_{recon}$ in terms of $w_{ij}$ takes the
form:

$\displaystyle\frac{\partial C_{recon}}{\partial w_{ij}}=$ $\displaystyle
2\sum\limits_{k=1}^{K}\sum\limits_{s\in\Re_{k}}\big{(}h_{sj}^{(r)}-\frac{\sum\limits_{s\in\Re_{k}}h_{sj}^{(r)}}{|\Re_{k}|}\big{)}\Bigg{[}(1-h_{sj}^{(r)})h_{sj}^{(r)}v_{si}^{(r)}$
(19)
$\displaystyle-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj}^{(r)})h_{sj}^{(r)}v_{si}^{(r)}}{|\Re_{k}|}\Bigg{]}.$

Similarly, the partial derivative of the $C_{data}$ in terms of $b_{j}$ is
given by:

$\displaystyle\frac{\partial C_{data}}{\partial b_{j}}=$ $\displaystyle
2\sum\limits_{k=1}^{K}\sum\limits_{s\in\Re_{k}}\big{(}h_{sj}-\frac{\sum\limits_{s\in\Re_{k}}h_{sj}}{|\Re_{k}|}\big{)}\Bigg{[}(1-h_{sj})h_{sj}$
(20)
$\displaystyle-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj})h_{sj}}{|\Re_{k}|}\Bigg{]}.$

And the partial derivative of the $C_{recon}$ in terms of $b_{j}$ is given by:

$\displaystyle\frac{\partial C_{recon}}{\partial b_{j}}=$ $\displaystyle
2\sum\limits_{k=1}^{K}\sum\limits_{s\in\Re_{k}}\big{(}h_{sj}^{(r)}-\frac{\sum\limits_{s\in\Re_{k}}h_{sj}^{(r)}}{|\Re_{k}|}\big{)}\Bigg{[}(1-h_{sj}^{(r)})h_{sj}^{(r)}$
(21)
$\displaystyle-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj}^{(r)})h_{sj}^{(r)}}{|\Re_{k}|}\Bigg{]}.$

Combined with the CD learning with 1 step Gibbs sampling, the update rule of
the proposed model parameter W takes the forms:

$\displaystyle
w_{ij}^{(\tau+1)}=w_{ij}^{(\tau)}+\eta\varepsilon(<v_{i}h_{j}>_{data}-<v_{i}h_{j}>_{recon})$
(22) $\displaystyle+\frac{2(1-\eta)}{K\times
L}\Bigg{\\{}\sum\limits_{k=1}^{K}\sum\limits_{s\in\Re_{k}}\big{(}h_{sj}-\frac{\sum\limits_{s\in\Re_{k}}h_{sj}}{|\Re_{k}|}\big{)}\Bigg{[}(1-h_{sj})h_{sj}v_{si}$
$\displaystyle-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj})h_{sj}v_{si}}{|\Re_{k}|}\Bigg{]}+\sum\limits_{k=1}^{K}\sum\limits_{s\in\Re_{k}}\big{(}h_{sj}^{(r)}-\frac{\sum\limits_{s\in\Re_{k}}h_{sj}^{(r)}}{|\Re_{k}|}\big{)}$
$\displaystyle\Bigg{[}(1-h_{sj}^{(r)})h_{sj}^{(r)}v_{si}^{(r)}-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj}^{(r)})h_{sj}^{(r)}v_{si}^{(r)}}{|\Re_{k}|}\Bigg{]}\Bigg{\\}},$

where $\varepsilon$ is learning rate, the average $<v_{i}h_{j}>_{data}$ and
$<v_{i}h_{j}>_{recon}$ are computed using the sample data and reconstructed
data, respectively.
For the parameters of the biases a and b, the update rules of them take the
forms:

$\displaystyle
a_{i}^{(\tau+1)}=a_{i}^{(\tau)}+\eta\varepsilon(<v_{i}>_{data}-<v_{i}>_{recon}),$
(23)

and

$\displaystyle
b_{j}^{(\tau+1)}=b_{j}^{(\tau)}+\eta\varepsilon(<h_{j}>_{data}-<h_{j}>_{recon})$
(24) $\displaystyle+\frac{2(1-\eta)}{K\times
L}\Bigg{\\{}\sum\limits_{k=1}^{K}\sum\limits_{s\in\Re_{k}}\big{(}h_{sj}-\frac{\sum\limits_{s\in\Re_{k}}h_{sj}}{|\Re_{k}|}\big{)}\Bigg{[}(1-h_{sj})h_{sj}$
$\displaystyle-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj})h_{sj}}{|\Re_{k}|}\Bigg{]}+\sum\limits_{k=1}^{K}\sum\limits_{s\in\Re_{k}}\big{(}h_{sj}^{(r)}-\frac{\sum\limits_{s\in\Re_{k}}h_{sj}^{(r)}}{|\Re_{k}|}\big{)}$
$\displaystyle\Bigg{[}(1-h_{sj}^{(r)})h_{sj}^{(r)}-\frac{\sum\limits_{s\in\Re_{k}}(1-h_{sj}^{(r)})h_{sj}^{(r)}}{|\Re_{k}|}\Bigg{]}\Bigg{\\}}.$

###  III-C The Algorithm

Algorithm 1 Learning algorithm of crRBM with 1 step Gibbs sampling

Input:
$\mathcal{D}$: input data sets;
$\mathcal{B}$: training batch sets;
$\varepsilon$: learning rate;
($\Re_{k},\ell_{l}$): matrix blocks of $\mathcal{D}$, $k\in[1,K]$ and
$l\in[1,L]$;

Output:
$\theta$: model parameters of crRBM.

Initialize: $\mathbf{a}$, $\mathbf{b}$ and $\mathbf{W}$.
while$\tau$ not exceeding maximum iteration do
for all training batch $\mathcal{B}$ do
Encoder: sample the states of hidden units by Eq. (2).
Decoder: sample the reconstructed states of visible
units using Eq. (3).
for all $\Re_{k}$ do
for all $\ell_{l}$ do
Compute $\frac{\partial C_{data}}{\partial w_{ij}}$ using Eq. (18).
Compute $\frac{\partial C_{recon}}{\partial w_{ij}}$ using Eq. (19).
Compute $\frac{\partial C_{data}}{\partial b_{j}}$ using Eq. (20).
Compute $\frac{\partial C_{recon}}{\partial b_{j}}$ using Eq. (21).
end for
end for
Update parameter $\mathbf{W}$ using Eq. (22).
Update parameter $\mathbf{a}$ using Eq. (23).
Update parameter $\mathbf{b}$ using Eq. (24).
end for
$\tau=\tau+1$.
end while
return$\mathbf{a}$, $\mathbf{b}$ and $\mathbf{W}$.

In the reconstruction process of crGRBM model, a linear reconstruction method
replaces the nolinear reconstruction method of crRBM model. The steps of the
learning algorithms of our crRBM and crGRBM models are almost the same, except
the reconstruction process. So, we omit the learning algorithm of crGRBM
model.

###  III-D Complexity Analysis

In this subsection, we analyze the time complexity of above learning
algorithm. Supposing that the input data sets $\mathcal{D}$ is divided into
$TB$ training batch. Then the time complexities of the encoder and decoder
steps are $O(TB)$ in each iteration. When partial derivatives $\frac{\partial
C_{data}}{\partial w_{ij}}$, $\frac{\partial C_{recon}}{\partial w_{ij}}$,
$\frac{\partial C_{data}}{\partial b_{j}}$ and $\frac{\partial
C_{recon}}{\partial b_{j}}$ are calculated, they take $O(TB\times(K\times L))$
in each iteration. The complexities of update parameters $\mathbf{W}$ ,
$\mathbf{a}$ and $\mathbf{b}$ are $O(TB)$ in each iteration. Supposing that
the maximum iteration is $IT$. Then, the time complexity of the crRBM learning
algorithm with 1 Step Gibbs sampling is $O(IT\times TB\times(K\times L))$.

###  III-E The UCRDNet Deep Architecture

To obtain superior performance of deep collaborative representation for
unsupervised clustering, we present an unsupervised collaborative
representation deep network (UCRDNet) which has three hidden layers (See Fig.
1) in this subsection. The Gaussian linear visible units of the UCRDNet are
used to model real-valued input data. In other words, the architecture of the
UCRDNet for modeling real-valued is composed of a crGRBM and two crRBMs (the
left one in Fig. 1). To explore the applicability of the UCRDNet, the binary
visible units of it are used to model binary input data. So, the architecture
of the UCRDNet for modeling binary is composed of only three crRBMs (the right
one in Fig. 1).

Figure 1: The unsupervised collaborative representation deep network (UCRDNet)
architecture. Left: The UCRDNet with Gaussian linear visible units for
modeling real-valued data. Right: The UCRDNet with binary visible units for
modeling binary data.

##  IV Experiment

This section elaborates the experimental setting and datasets. Thereafter, the
proposed UCRDNet is compared with the Autoencoder and DeepFS deep networks. To
show the superior performance of our UCRDNet for deep collaborative
representation, we further compare it with the CDL deep collaborative
networks.

###  IV-A Experimental Setting

To validate the capability of deep collaborative representation of the
proposed UCRDNet, we compare it with Autoencoder [34] and DeepFS [35]) deep
networks, which have not collaborative representation strategy. Furthermore,
we compare our UCRDNet with the collaborative deep learning (CDL) [36] model,
which has collaborative representation strategy. For modeling real-valued data
(MSRA-MM 2.0 datasets), the visible layer of the proposed UCRDNet uses
Gaussian linear units. But, the visible layer of it uses binary units for
modeling binary data. In each hidden layer of Autoencoder, DeepFS, CDL and our
UCRDNet, the feature dimensions remains the same as the visible layer.
We use unsupervised K-means [41] and Spectral Clustering (SC) [42] algorithms
to compare the representation capabilities of the hidden layer features of all
contrastive deep network, particularly the collaborative representation
capabilities of our UCRDNet. Three popular external evaluations, e.g.,
accuracy [43], Jaccard index [44], Fowlkes and Mallows Index (FMI) [45] are
used to evaluate the performance. Furthermore, we use advanced nonparametric
tests (e.g., Friedman Aligned Ranks test statistic [46]) to provide a fair
comparison of the different algorithms based on Autoencoder, DeepFS, CDL and
our UCRDNet. The accuracy [43] takes the form:

$\displaystyle accuracy=\frac{\sum\limits_{i=1}\delta(s_{i},map(r_{i}))}{n},$
(25)

where $map(r_{i})$ maps label $r_{i}$ of each cluster to the equivalent label
and $n$ is the total number of instances. If $x=y$ , then $\delta(x,y)$ equals
to 1 . Otherwise, it is zero. The Jaccard index [44] metric is defined by:

$\displaystyle Jac=\frac{|A\cap B|}{|A\cup B|},$ (26)

where $A$ and $B$ are finite sample sets and $0\leq J(A,B)\leq 1$. The FMI
[45] is given by:

$\displaystyle FMI=\sqrt{\frac{TP}{TP+FP}\times\frac{TP}{TP+FN}},$ (27)

where $TP$ is the number of true positives, $FP$ is the number of false
positives and and $FN$ is the number of false negatives.
The Friedman Aligned Ranks test statistic [46] takes the form:

$\displaystyle
T=\frac{(n_{a}-1)(\sum\limits_{j=1}^{n_{a}}\widehat{r}_{.j}^{2}-n_{a}n_{d}^{2}(n_{a}n_{d}+1)^{2}/4)}{n_{a}n_{d}(n_{a}n_{d}+1)(2n_{a}n_{d}+1)/6-\sum\limits_{i=1}^{n_{d}}\widehat{r}_{i.}^{2}/n_{a}},$
(28)

where $\widehat{r}_{i.}$ and $\widehat{r}_{.j}$ are the ranks total of the
$j$th algorithm and $i$th data set, respectively, $n_{a}$ and $n_{d}$ are the
numbers of algorithm and data set, respectively. For $n_{a}-1$ degrees of
freedom, the test statistic $T$ is compared for significance with a chi-square
distribution.

###  IV-B Datasets

In the following experiments, we choose two kinds of datasets to test the
capability of deep collaborative representation of the proposed UCRDNet. One
kind is twelve real-valued data sets (banner, beret, bugat, bugatti, building,
vista, vistawallpaper, voituretuning, water, weddingring, wing and worldmap)
from MSRA-MM 2.0 [47]. Every of them are listed in the Table I. Another kind
is twelve UCI data sets (balance, biodegradation, car, Climate Model, cred,
dermatology, Haberman Survival, ILPD, Kdd, OLD, parkinsons and secom), which
are listed in the Table II.

TABLE I: Summary of the experiment datasets I.

No. | Dataset | classes | Instances | features
---|---|---|---|---
1 | banner | 3 | 860 | 892
2 | beret | 3 | 876 | 892
3 | bugat | 3 | 882 | 892
4 | bugatti | 3 | 882 | 892
5 | building | 3 | 911 | 892
6 | vista | 3 | 799 | 899
7 | vistawallpaper | 3 | 799 | 899
8 | voituretuning | 3 | 879 | 899
9 | water | 3 | 922 | 899
10 | weddingring | 3 | 897 | 899
11 | wing | 3 | 856 | 899
12 | worldmap | 3 | 935 | 899

TABLE II: Summary of the experiment datasets II.

No. | Dataset | classes | Instances | features
---|---|---|---|---
1 | balance | 3 | 625 | 4
2 | biodegradation | 2 | 1055 | 41
3 | car | 4 | 1728 | 6
4 | Climate Model | 2 | 540 | 18
5 | cred (Credit Approval) | 2 | 690 | 15
6 | dermatology | 6 | 366 | 34
7 | Haberman Survival | 2 | 306 | 3
8 | ILPD(Indian Liver Patient) | 2 | 583 | 8
9 | Kdd (1999 partial data) | 3 | 1280 | 41
10 | OLD (Ozone Level Detection) | 2 | 2534 | 72
11 | parkinsons | 2 | 195 | 22
12 | secom | 2 | 1567 | 590

###  IV-C Exp I: UCRDNet on Real-valued Datasets

In this subsection, we demonstrate the collaborative representation capability
of the UCRDNet on the MSRA-MM 2.0 [47] datasets by unsupervised clustering
with three external evaluation metrics and the Friedman aligned ranks test
statistic.

####  IV-C1 Accuracy

The accuracy is one of the most popular and important external metrics to
evaluate the clustering performance. Table III shows the accuracy results
(mean$\pm$std) of K-means and SC algorithms based on Autoencoder, DeepFS, CDL
and our UCRDNet, respectively. As we can see, the proposed UCRDNet shows more
outstanding performance both for K-means and SC algorithms than other deep
networks. The average accuracies of K-means and SC algorithms based on our
UCRDNet are 0.5987 and 0.6330, respectively. In contrast to Autoencoder, the
performance of the proposed UCRDNet is improved by 19% and 27.88% for K-means
and SC algorithms, respectively. To compare with DeepFS, the performance of
the proposed UCRDNet is improved by 16.53% and 14.32% for K-means and SC
algorithms, respectively. In contrast to CDL, which has deep collaborative
representation strategy, the performance of the proposed UCRDNet is improved
by 16.16% and 10.91% for K-means and SC algorithms, respectively. For all
datasets in Table I, the intuitive performance comparison about the accuracy
is shown in Fig. 3 among all four deep networks. Clearly, we can observe that
our UCRDNet not only has more powerful deep learning capability than
Autoencoder and DeepFS, but also has more excellent deep collaborative
representation capability than CDL deep network.
A more intuitive contrast of the performance among all four deep networks is
shown in Fig. 2 for each data set. Clearly, the most outstanding performance
distributes in the section of K-means or SC algorithms based on our UCRDNet.

####  IV-C2 Jaccard Index

The Jaccard index (Jac) is another popular external evaluation metric for
clustering. Table IV shows the Jac results of K-means and SC algorithms based
on Autoencoder, DeepFS, CDL and our UCRDNet, respectively. The average Jac
values of K-means and SC algorithms based on our UCRDNet are 0.4424 and
0.5032, respectively. The average Jac values of K-means and SC algorithms
based on Autoencoder and DeepFS are 0.2793, 0.2563, 0.2889 and 0.3467,
respectively. In contrast to Autoencoder and DeepFS, the performance of the
proposed UCRDNet is improved by 0.1632, 0.2469, 0.1535 and 0.1565 for K-means
and SC algorithms, respectively. To compare with CDL, the performance of the
proposed UCRDNet is improved by 0.1424 and 0.1275 for K-means and SC
algorithms, respectively. Clearly, the most excellent performance for each
dataset in Table IV distributes the section of K-means or SC algorithms based
on our UCRDNet.
The visual contrast of average Jac of all datasets in Table I is shown in Fig.
3. With regard to deep feature representation capability, the proposed UCRDNet
shows better performance than Autoencoder and DeepFS. Furthermore, our UCRDNet
shows better deep collaborative representation capability than CDL deep
network.

####  IV-C3 Fowlkes and Mallows Index

The Fowlkes and Mallows Index (FMI) is also an frequently used external metric
to evaluate the clustering performance. Table V shows the FMI results of
K-means and SC algorithms based on Autoencoder, DeepFS, CDL and our UCRDNet,
respectively. It’s obvious that the most outstanding performance for each
dataset distributes in the section of K-means or SC algorithm based on the
proposed UCRDNet. The average FMI values of K-means and SC algorithms based on
our UCRDNet are 0.6168 and 0.6864, respectively. In contrast to Autoencoder,
the performance of the proposed UCRDNet is improved by 0.1724 and 0.2658 for
K-means and SC algorithms, respectively. To compare with DeepFS, the
performance of the proposed UCRDNet is improved by 0.1592 and 0.1695 for
K-means and SC algorithms, respectively. In contrast to CDL, which has deep
collaborative representation strategy, the performance of the proposed UCRDNet
is improved by 0.1500 and 0.1285 for K-means and SC algorithms, respectively.
The visual contrast of average FMI of all datasets in Table I is shown in Fig.
3. From the comparison results, the proposed UCRDNet has better deep
collaborative representation capability than Autoencoder, DeepFS and CDL deep
network.

####  IV-C4 The Friedman Aligned Ranks Test Statistic

Table VI shows the ranks (in the parentheses) and average ranks of K-means and
SC algorithms based on Autoencoder, DeepFS, CDL and our UCRDNet. The average
rank of SC algorithm based on our UCRDNet is 13.3333, it is the least value of
all algorithms based on the contrastive deep networks. The average rank of
K-means algorithm based on the proposed UCRDNet is 19.9167, it the second
smallest value in Table VI. For K-means algorithm based on Autoencoder, DeepFS
and CDL, the average ranks are 73.4167, 60.7500 and 59.9167, respectively. For
SC algorithm based on Autoencoder, DeepFS and CDL, the average ranks are
88.0000, 41.5833 and 31.0833, respectively. Clearly, our UCRDNet shows better
performance than other contrastive deep networks. By means of the Friedman
Aligned test statistic, _T=68.262_ is the chi-square distribution with 7
degrees of freedom. The _p_ -value is $3.314e-12$ which is computed by
$\chi^{2}(7)$ distribution for one tailed test. Then, the null hypothesis is
rejected at a high level significance. The _p_ -values are far less than 0.05,
so the experimental results of algorithms are significantly different.

Figure 2: The accuracies of Autoencoder, DeepFS, CDL and our UCRDNet deep
networks on the MSRA-MM2.0 datasets.

Figure 3: The clustering performance comparison (average accuracy, Jac and
FMI) of Autoencoder, DeepFS, CDL and our UCRDNet deep networks on the MSRA-
MM2.0 datasets.

Figure 4: The accuracies of Autoencoder, DeepFS, CDL and our UCRDNet deep
networks on UCI datasets.

Figure 5: The clustering performance comparison (average accuracy, Jac and FMI) of Autoencoder, DeepFS, CDL and our UCRDNet deep networks on the UCI datasets.  TABLE III: The accuracy of the contrastive deep networks on the MSRA-MM2.0 datasets. The best performance on each data set is bolded.  Dataset | K-means with hidden layer features | SC with hidden layer features
---|---|---
Autoencoder | DeepFS | CDL | UCRDNet | Autoencoder | DeepFS | CDL | UCRDNet
banner |  0.5105$\pm$ 0.0004 |  0.4364$\pm$ 0.0003 |  0.5267$\pm$0.0000 | 0.9372$\pm$0.0006 |  0.3570$\pm$0.0000 |  0.5093$\pm$0.0001 |  0.6031$\pm$0.0000 |  0.9349$\pm$0.0089
beret |  0.3893$\pm$ 0.0018 |  0.4254$\pm$ 0.0000 |  0.4304$\pm$0.0001 | 0.6895$\pm$0.0002 |  0.3642$\pm$0.0001 |  0.4001$\pm$0.0000 |  0.4063$\pm$0.0000 |  0.4224$\pm$0.0112
bugat |  0.3889$\pm$ 0.0038 |  0.4005$\pm$ 0.0005 |  0.4380$\pm$0.0000 |  0.4297$\pm$0.0004 |  0.3515$\pm$0.0000 |  0.4947$\pm$0.0002 |  0.4769$\pm$0.0051 | 0.5748$\pm$0.0117
bugatti |  0.4161$\pm$ 0.0010 |  0.4305$\pm$ 0.0005 |  0.4390$\pm$0.0000 |  0.4989$\pm$0.0003 |  0.3639$\pm$0.0002 |  0.4947$\pm$0.0001 |  0.4769$\pm$0.0051 | 0.7007$\pm$0.0089
building |  0.4501$\pm$ 0.0013 |  0.5159$\pm$ 0.0001 |  0.4918$\pm$0.0014 | 0.7146$\pm$0.0000 |  0.3469$\pm$0.0001 |  0.4812$\pm$0.0030 |  0.4877$\pm$0.0002 |  0.7058$\pm$0.0026
vista |  0.3842$\pm$ 0.0006 |  0.4251$\pm$ 0.0003 |  0.4030$\pm$0.0000 | 0.6308$\pm$0.0005 |  0.3567$\pm$0.0001 |  0.4927$\pm$0.0000 |  0.5424$\pm$0.0053 |  0.5720$\pm$0.0068
vistawallpaper |  0.3780$\pm$ 0.0008 |  0.4251$\pm$ 0.0003 |  0.4030$\pm$0.0000 |  0.4906$\pm$0.0002 |  0.3717$\pm$0.0002 |  0.4927$\pm$0.0002 |  0.5924$\pm$0.0053 | 0.6320$\pm$0.0054
voituretuning |  0.3788$\pm$ 0.0013 |  0.4331$\pm$ 0.0002 |  0.4020$\pm$0.0001 | 0.6394$\pm$0.0009 |  0.3481$\pm$0.0001 |  0.4391$\pm$0.0000 |  0.5631$\pm$0.0074 |  0.6359$\pm$0.0025
water |  0.3482$\pm$ 0.0009 |  0.4140$\pm$ 0.0000 |  0.3955$\pm$0.0002 |  0.5108$\pm$0.0004 |  0.3449$\pm$0.0001 |  0.4638$\pm$0.0001 |  0.5201$\pm$0.0000 | 0.5705$\pm$0.0044
weddingring |  0.4091$\pm$0.0010 |  0.4064$\pm$ 0.0000 |  0.3987$\pm$0.0000 |  0.4192$\pm$0.0061 |  0.3456$\pm$0.0001 |  0.4966$\pm$0.0000 |  0.5054$\pm$0.0000 | 0.5217$\pm$0.0024
wing |  0.3680$\pm$0.0010 |  0.4151$\pm$ 0.0003 |  0.4065$\pm$0.0000 | 0.6192$\pm$0.0053 |  0.3540$\pm$0.0001 |  0.5456$\pm$0.0001 |  0.5668$\pm$0.0002 |  0.6121$\pm$0.0046
worldmap |  0.4834$\pm$ 0.0020 |  0.4731$\pm$ 0.0002 |  0.5077$\pm$0.0001 |  0.6043$\pm$0.0030 |  0.3455$\pm$0.0001 |  0.5672$\pm$0.0000 |  0.5455$\pm$0.0001 | 0.7134$\pm$0.0081
Average | 0.4087 | 0.4334 | 0.4371 | 0.5987 | 0.3542 | 0.4898 | 0.5239 | 0.6330
TABLE IV: The Jac of the contrastive deep networks on the MSRA-MM2.0 datasets. The best performance on each data set is bolded. Dataset | K-means with hidden layer features | SC with hidden layer features
---|---|---
Autoencoder | DeepFS | CDL | UCRDNet | Autoencoder | DeepFS | CDL | UCRDNet
banner | 0.3720 | 0.3363 | 0.3955 | 0.8820 | 0.3192 | 0.4672 | 0.4906 | 0.8779
beret | 0.2825 | 0.2755 | 0.3082 | 0.5348 | 0.2633 | 0.3496 | 0.3679 | 0.5328
bugat | 0.2668 | 0.2792 | 0.3090 | 0.2801 | 0.2600 | 0.3530 | 0.2694 | 0.4000
bugatti | 0.2774 | 0.2792 | 0.3090 | 0.3194 | 0.2606 | 0.3530 | 0.2694 | 0.5420
building | 0.3070 | 0.3480 | 0.3406 | 0.5574 | 0.2629 | 0.2717 | 0.3657 | 0.5470
vista | 0.2582 | 0.2780 | 0.2751 | 0.4738 | 0.2435 | 0.3314 | 0.3318 | 0.4120
vistawallpaper | 0.2796 | 0.2780 | 0.2751 | 0.2714 | 0.2432 | 0.3314 | 0.3318 | 0.4736
voituretuning | 0.2515 | 0.2765 | 0.2584 | 0.4760 | 0.2435 | 0.3236 | 0.3271 | 0.4709
water | 0.2321 | 0.2495 | 0.2473 | 0.3561 | 0.2321 | 0.3085 | 0.4351 | 0.4351
weddingring | 0.2615 | 0.2774 | 0.2597 | 0.2574 | 0.2393 | 0.3391 | 0.4620 | 0.3274
wing | 0.2491 | 0.2727 | 0.2663 | 0.4714 | 0.2430 | 0.3433 | 0.4694 | 0.4598
worldmap | 0.3145 | 0.3181 | 0.3555 | 0.4290 | 0.2647 | 0.3885 | 0.3887 | 0.5602
Average | 0.2793 | 0.2889 | 0.3000 | 0.4424 | 0.2563 | 0.3467 | 0.3757 | 0.5032
TABLE V: The FMI of the contrastive deep networks on the MSRA-MM2.0 datasets. The best performance on each data set is bolded. Dataset | K-means with hidden layer features | SC with hidden layer features
---|---|---
Autoencoder | DeepFS | CDL | UCRDNet | Autoencoder | DeepFS | CDL | UCRDNet
banner | 0.5876 | 0.5577 | 0.6070 | 0.9392 | 0.5424 | 0.6629 | 0.6805 | 0.9367
beret | 0.4488 | 0.4410 | 0.4771 | 0.7313 | 0.4285 | 0.5184 | 0.5380 | 0.7288
bugat | 0.4322 | 0.4467 | 0.4778 | 0.4469 | 0.4294 | 0.5223 | 0.4345 | 0.5715
bugatti | 0.4430 | 0.4467 | 0.4778 | 0.4902 | 0.4259 | 0.5223 | 0.4345 | 0.7279
building | 0.4765 | 0.5223 | 0.5123 | 0.7466 | 0.4303 | 0.4416 | 0.5360 | 0.7353
vista | 0.4143 | 0.4386 | 0.4338 | 0.6870 | 0.3979 | 0.4980 | 0.4984 | 0.6028
vistawallpaper | 0.4380 | 0.4386 | 0.4338 | 0.4321 | 0.3975 | 0.4980 | 0.4984 | 0.6865
voituretuning | 0.4065 | 0.4361 | 0.4160 | 0.6899 | 0.3979 | 0.4890 | 0.4931 | 0.6808
water | 0.3802 | 0.4016 | 0.3986 | 0.5344 | 0.3803 | 0.4276 | 0.4585 | 0.6585
weddingring | 0.4167 | 0.4377 | 0.4156 | 0.4133 | 0.3914 | 0.4567 | 0.4791 | 0.4933
wing | 0.4038 | 0.4330 | 0.4256 | 0.6866 | 0.3969 | 0.5113 | 0.6840 | 0.6692
worldmap | 0.4849 | 0.4915 | 0.5266 | 0.6043 | 0.4333 | 0.5603 | 0.5600 | 0.7460
Average | 0.4444 | 0.4576 | 0.4668 | 0.6168 | 0.4206 | 0.5169 | 0.5579 | 0.6864
TABLE VI: The Rank of the contrastive deep networks on the MSRA-MM2.0 datasets. The smaller relative rank, the better performance. Dataset | K-means with hidden layer features | SC with hidden layer features | Total
---|---|---|---
Autoencoder | DeepFS | CDL | UCRDNet | Autoencoder | DeepFS | CDL | UCRDNet
banner | -0.0914(78) | -0.1655(93) | -0.0751(74) | 0.3353(1) | -0.2449(96) | -0.0926(81) | 0.0012(38) | 0.3330(2) | 463
beret | -0.0517(66) | -0.0155(44) | -0.0106(42) | 0.2486(3) | -0.0768(75) | -0.0408(57) | -0.0346(52) | -0.0186(45) | 384
bugat | -0.0555(67) | -0.0439(59) | -0.0064(40) | -0.0147(43) | -0.0929(82) | 0.0503(28) | 0.0326(30) | 0.1305(13) | 362
bugatti | -0.0614(69) | -0.0470(63) | -0.0394(55) | 0.0214(31) | -0.1135(88) | 0.0172(35) | -0.0005(39) | 0.2232(4) | 384
building | -0.0724(73) | -0.0083(41) | -0.0325(50) | 0.1904(5) | -0.1774(94) | -0.0431(58) | -0.0365(53) | 0.1816(7) | 381
vista | -0.0916(79) | -0.0507(65) | -0.0729(72) | 0.1549(11) | -0.1192(90) | 0.0168(36) | 0.0665(24) | 0.0961(17) | 394
vistawallpaper | -0.0952(83) | -0.0481(64) | -0.0702(70) | 0.0174(33) | -0.1015(86) | 0.0195(32) | 0.1192(16) | 0.1588(9) | 393
voituretuning | -0.1011(85) | -0.0469(61) | -0.0780(76) | 0.1594(8) | -0.1318(91) | -0.0408(56) | 0.0832(19) | 0.1560(10) | 406
water | -0.0983(84) | -0.0325(51) | -0.0470(62) | 0.0644(25) | -0.1016(87) | 0.0174(34) | 0.0737(22) | 0.1240(15) | 380
weddingring | -0.0287(48) | -0.0314(49) | -0.0391(54) | -0.0187(46) | -0.0923(80) | 0.0588(27) | 0.0675(23) | 0.0839(18) | 345
wing | -0.1179(89) | -0.0708(71) | -0.0794(77) | 0.1332(12) | -0.1319(92) | 0.0596(26) | 0.0809(20) | 0.1262(14) | 401
worldmap | -0.0466(60) | -0.0569(68) | -0.0223(47) | 0.0743(21) | -0.1845(95) | 0.0372(29) | 0.0155(37) | 0.1834(6) | 363
Total | 881 | 729 | 719 | 239 | 1056 | 499 | 373 | 160 | 4656
Average rank | 73.4167 | 60.7500 | 59.9167 | 19.9167 | 88.0000 | 41.5833 | 31.0833 | 13.3333 |

###  IV-D Exp II: UCRDNet on UCI Datasets

As in the previous experiments, we demonstrate the collaborative
representation capability of the proposed UCRDNet with binary visible units on
the UCI datasets in this subsection.

####  IV-D1 Accuracy

Table VII shows the accuracy results (mean$\pm$std) of K-means and SC
algorithms based on Autoencoder, DeepFS, CDL and our UCRDNet deep networks.
For the average accuracies of twelve UCI datasets, the results of K-means and
SC algorithms based on our UCRDNet are 0.7122 and 0.7014, respectively. To
compare with other algorithms, they show the best performance. Specifically,
in contrast to Autoencoder, the performance of the proposed UCRDNet is
improved by 11.71% and 7.8% for K-means and SC algorithms, respectively. To
compare with DeepFS, the performance of the proposed UCRDNet is improved by
15.74% and 6.71% for K-means and SC algorithms, respectively. In contrast to
CDL, which has deep collaborative representation strategy, the performance of
the proposed UCRDNet is improved by 14.92% and 9.75% for K-means and SC
algorithms, respectively. For each UCI datasets in Table II, the best result
distributes in the section of K-means or SC algorithms based on our UCRDNet
except for the HabermanSurvival dataset. The SC algorithm based on Autoencoder
and our UCRDNet show best results (accuracies=0.7255) at the same time for the
HabermanSurvival dataset. Clearly, the proposed UCRDNet shows more outstanding
performance both for K-means and SC algorithms than other deep networks. The
intuitive comparisons are shown in Fig. 4 for each dataset. Similarly, our
UCRDNet not only has more powerful deep learning capability than Autoencoder
and DeepFS, but also has more excellent deep collaborative representation
capability than CDL deep network on the UCI datasets. The intuitive comparison
of average accuracy are shown in Fig. 5.

####  IV-D2 Jaccard Index

Table VIII shows the Jac results of K-means and SC algorithms based on
Autoencoder, DeepFS, CDL and our UCRDNet on the UCI datasets, respectively.
For our UCRDNet, the average Jac values of K-means and SC algorithms are
0.5132 and 0.5404, respectively. For Autoencoder, the average Jac values of
K-means and SC algorithms reduce to 0.4344 and 0.4932, respectively. For
DeepFS, the average Jac values of K-means and SC algorithms reduce to 0.4284
and 0.5300, respectively. For CDL, the average Jac values of K-means and SC
algorithms reduce to 0.4425 and 0.5298, respectively. To compare with
Autoencoder, DeepFS and CDL deep networks, our UCRDNet show the best
performance. It shows more excellent feature representation capability than
Autoencoder and DeepFS deep networks and outstanding deep collaborative
representation capability than CDL deep networks. The intuitive comparison of
average Jac are shown in Fig. 5.
For cred dataset, the K-means algorithm based on DeepFS shows the best
performance. The Jac value of it reaches 0.5039. For HabermanSurvival dataset,
the K-means algorithm based on Autoencoder shows the best performance. The Jac
value of it reaches 0.6081. For parkinsons dataset, the SC algorithm based on
Autoencoder shows the best performance. The Jac value of it reaches 0.6319.
For other datasets, the best performance distributes in the section of K-means
or SC algorithms based on the proposed UCRDNet.

####  IV-D3 Fowlkes and Mallows Index

Table IX shows the FMI results of K-means and SC algorithms based on
Autoencoder, DeepFS, CDL and our UCRDNet, respectively. For our UCRDNet, the
average FMI values of K-means and SC algorithms are 0.6567 and 0.7128,
respectively. For Autoencoder and DeepFS, the average FMI values of K-means
and SC algorithms reduce to 0.5962,6784, 5933 and 0.6891, respectively.
Clearly, the proposed UCRDNet shows more outstanding deep feature learning
capabilities than Autoencoder and DeepFS deep networks. For CDL, the average
FMI values of K-means and SC algorithms reduce to 0.6012 and 0.6805,
respectively. It is obvious that our UCRDNet shows more excellent deep
collaborative feature learning capabilities than CDL deep network. The
intuitive comparison of average FMI is shown in Fig. 5.
For balance, biodegradation and parkinsons datasets, the SC algorithm based on
Autoencoder shows the best performance. The FMI values of them reach 0.6533,
0.7431 and 0.7938, respectively. For HabermanSurvival dataset, the K-means
algorithm based on Autoencoder shows the best performance. The FMI value of it
reaches 0.7778. For ILPD dataset, the SC algorithm based on CDL shows the best
performance. The FMI value of it reaches 0.7223. For all other datasets, the
best performance distributes in the section of K-means or SC algorithms based
on our UCRDNet.

####  IV-D4 The Friedman Aligned Ranks Test Statistic

Table VI shows the ranks (in the parentheses) and average rank of K-means and
SC algorithms based on Autoencoder, DeepFS, CDL and our UCRDNet. The average
rank of K-means algorithm based on our UCRDNet is 26.1667, which is the least
value of all algorithms based on the contrastive deep networks. The average
rank of SC algorithm based on the proposed UCRDNet is 26.3333, which is the
second smallest value in Table VI. For K-means algorithm based on Autoencoder,
DeepFS and CDL, the average ranks are 55.4167, 67.6667 and 69.0000,
respectively. For SC algorithm based on Autoencoder, DeepFS and CDL, the
average ranks are 46.3333, 43.9167 and 53.1667, respectively. It is obvious
that our UCRDNet shows better performance than other contrastive deep
networks. By means of the Friedman Aligned test statistic, _T=26.273_ is the
chi-square distribution with 7 degrees of freedom. For one tailed test, the
_p_ -value is $0.0004501$ which is computed by $\chi^{2}(7)$ distribution.
Then, the null hypothesis is rejected at a high level significance. The
experimental results of algorithms are significantly different because the _p_
-values are far less than 0.05.

TABLE VII: The accuracy of the contrastive deep networks on the UCI datasets. The best performance on each data set is bolded. Dataset | K-means with hidden layer features | SC with hidden layer features
---|---|---
Autoencoder | DeepFS | CDL | UCRDNet | Autoencoder | DeepFS | CDL | UCRDNet
balance |  0.3904$\pm$ 0.0000 |  0.4768$\pm$ 0.0017 |  0.4208$\pm$ 0.0000 | 0.6224$\pm$0.0001 |  0.4640$\pm$0.0001 |  0.4584$\pm$0.0000 |  0.4524$\pm$ 0.0000 |  0.4640$\pm$0.0003
biodegradation |  0.6199$\pm$ 0.0002 |  0.6398$\pm$ 0.0000 |  0.5223$\pm$ 0.0000 | 0.6673$\pm$0.0004 |  0.6645$\pm$0.0000 |  0.6190$\pm$0.0000 |  0.6445$\pm$ 0.0000 |  0.6645$\pm$0.0002
car |  0.3414$\pm$ 0.0000 |  0.4199$\pm$ 0.0000 |  0.4132$\pm$ 0.0000 |  0.4426$\pm$0.0001 |  0.5434$\pm$0.0000 |  0.6325$\pm$0.0022 |  0.6485$\pm$ 0.0000 | 0.6985$\pm$0.0002
ClimateModel |  0.5130$\pm$ 0.0001 |  0.5407$\pm$ 0.0001 |  0.5037$\pm$ 0.0000 |  0.5500$\pm$0.0004 |  0.5926$\pm$0.0003 |  0.8093$\pm$0.0000 |  0.5185$\pm$ 0.0509 | 0.9111$\pm$0.0011
cred |  0.6652$\pm$ 0.0014 |  0.5551$\pm$ 0.0000 |  0.5623$\pm$ 0.0000 | 0.6841$\pm$0.0026 |  0.6087$\pm$0.0016 |  0.6029$\pm$0.0014 |  0.5565$\pm$ 0.0004 |  0.6391$\pm$0.0021
dermatology |  0.4372$\pm$ 0.0010 |  0.2377$\pm$ 0.0000 |  0.2705$\pm$ 0.0002 | 0.4754$\pm$0.0005 |  0.3552$\pm$0.0007 |  0.2814$\pm$0.0006 |  0.3087$\pm$ 0.0000 |  0.4508$\pm$0.0004
HabermanSurvival |  0.6242$\pm$ 0.0009 |  0.6157$\pm$ 0.0000 |  0.5196$\pm$ 0.0000 |  0.6275$\pm$0.0012 | 0.7255$\pm$0.0000 |  0.7055$\pm$0.0005 |  0.6755$\pm$ 0.0000 | 0.7255$\pm$0.0007
ILPD |  0.6844$\pm$ 0.0001 |  0.6552$\pm$ 0.0000 |  0.6329$\pm$ 0.0008 | 0.8136$\pm$0.0001 |  0.5111$\pm$0.0000 |  0.6515$\pm$0.0079 |  0.6533$\pm$ 0.0000 |  0.6690$\pm$0.0000
Kdd |  0.5523$\pm$ 0.0000 |  0.3875$\pm$ 0.0000 |  0.6164$\pm$ 0.0204 | 0.9877$\pm$0.0007 |  0.6414$\pm$0.0002 |  0.7211$\pm$0.0009 |  0.6586$\pm$ 0.0000 |  0.8156$\pm$0.0005
OLD |  0.9065$\pm$ 0.0002 |  0.7616$\pm$ 0.0000 |  0.9017$\pm$ 0.0000 | 0.9369$\pm$0.0002 |  0.7932$\pm$0.0001 |  0.8481$\pm$0.0260 |  0.8833$\pm$ 0.0000 |  0.9013$\pm$0.0001
parkinsons |  0.6359$\pm$0.0002 |  0.6051$\pm$ 0.0000 |  0.5641$\pm$ 0.0040 | 0.8051$\pm$0.0003 |  0.7641$\pm$0.0000 |  0.7487$\pm$0.0000 |  0.7341$\pm$ 0.0000 |  0.7641$\pm$0.0000
secom |  0.7709$\pm$ 0.0006 |  0.7620$\pm$ 0.0072 |  0.8283$\pm$ 0.0000 | 0.9336$\pm$0.0003 |  0.8175$\pm$0.0002 |  0.5333$\pm$0.0168 |  0.5124$\pm$ 0.0000 |  0.7128$\pm$0.0004
Average | 0.5951 | 0.5548 | 0.5630 | 0.7122 | 0.6234 | 0.6343 | 0.6039 | 0.7014
TABLE VIII: The Jac of the contrastive deep networks on the UCI datasets. The best performance on each data set is bolded. Dataset | K-means with hidden layer features | SC with hidden layer features
---|---|---
Autoencoder | DeepFS | CDL | UCRDNet | Autoencoder | DeepFS | CDL | UCRDNet
balance | 0.2683 | 0.3036 | 0.2585 | 0.2488 | 0.4285 | 0.4135 | 0.4255 | 0.4285
biodegradation | 0.3893 | 0.5126 | 0.3916 | 0.3898 | 0.5506 | 0.3715 | 0.5495 | 0.5525
car | 0.3050 | 0.2957 | 0.2444 | 0.2143 | 0.4204 | 0.4657 | 0.5376 | 0.5408
ClimateModel | 0.4573 | 0.4582 | 0.4573 | 0.4601 | 0.4719 | 0.8307 | 0.4543 | 0.8370
cred | 0.3934 | 0.5039 | 0.5030 | 0.4397 | 0.4499 | 0.4181 | 0.5019 | 0.4666
dermatology | 0.2239 | 0.1099 | 0.1192 | 0.3555 | 0.2082 | 0.1748 | 0.1942 | 0.1987
HabermanSurvival | 0.6081 | 0.5316 | 0.3817 | 0.4324 | 0.5983 | 0.5748 | 0.5953 | 0.5983
ILPD | 0.4305 | 0.5143 | 0.5090 | 0.5817 | 0.4751 | 0.5366 | 0.5764 | 0.4164
Kdd | 0.4266 | 0.3385 | 0.5040 | 0.9806 | 0.5492 | 0.5104 | 0.5576 | 0.5707
OLD | 0.8215 | 0.6213 | 0.8215 | 0.8816 | 0.4961 | 0.7387 | 0.8724 | 0.7560
parkinsons | 0.4103 | 0.4583 | 0.4134 | 0.4103 | 0.6319 | 0.6108 | 0.6289 | 0.6319
secom | 0.5766 | 0.4926 | 0.7061 | 0.7634 | 0.6383 | 0.7149 | 0.4669 | 0.4875
Average | 0.4344 | 0.4284 | 0.4425 | 0.5132 | 0.4932 | 0.5300 | 0.5298 | 0.5404
TABLE IX: The FMI of the contrastive deep networks on the UCI datasets. The best performance on each data set is bolded. Dataset | K-means with hidden layer features | SC with hidden layer features
---|---|---
Autoencoder | DeepFS | CDL | UCRDNet | Autoencoder | DeepFS | CDL | UCRDNet
balance | 0.4263 | 0.4570 | 0.4138 | 0.4010 | 0.6533 | 0.6323 | 0.6183 | 0.6533
biodegradation | 0.5551 | 0.6862 | 0.5632 | 0.5615 | 0.7431 | 0.5429 | 0.7079 | 0.7429
car | 0.3743 | 0.4587 | 0.4217 | 0.3798 | 0.5977 | 0.6366 | 0.6995 | 0.7346
ClimateModel | 0.6491 | 0.6400 | 0.6490 | 0.6527 | 0.6626 | 0.9066 | 0.6143 | 0.9140
cred | 0.5647 | 0.6989 | 0.7071 | 0.6137 | 0.6273 | 0.5991 | 0.6751 | 0.6494
dermatology | 0.3659 | 0.1882 | 0.2132 | 0.5249 | 0.4452 | 0.3675 | 0.3981 | 0.4398
HabermanSurvival | 0.7778 | 0.6868 | 0.5547 | 0.6038 | 0.7699 | 0.7462 | 0.7349 | 0.7699
ILPD | 0.6026 | 0.6785 | 0.6851 | 0.4263 | 0.6487 | 0.7068 | 0.7223 | 0.5898
Kdd | 0.6097 | 0.5718 | 0.6893 | 0.9853 | 0.7308 | 0.6769 | 0.7101 | 0.7282
OLD | 0.9023 | 0.7634 | 0.9023 | 0.9390 | 0.6864 | 0.8467 | 0.9002 | 0.8612
parkinsons | 0.5843 | 0.6188 | 0.5858 | 0.5839 | 0.7938 | 0.7761 | 0.7588 | 0.7938
secom | 0.7419 | 0.6711 | 0.8293 | 0.8658 | 0.7844 | 0.8312 | 0.6268 | 0.6771
Average | 0.5962 | 0.5933 | 0.6012 | 0.6567 | 0.6784 | 0.6891 | 0.6805 | 0.7128
TABLE X: The Rank of the contrastive deep networks on the UCI datasets. The smaller relative rank, the better performance. Dataset | K-means with hidden layer features | SC with hidden layer features | Total
---|---|---|---
Autoencoder | DeepFS | CDL | UCRDNet | Autoencoder | DeepFS | CDL | UCRDNet
balance | -0.0782(78) | 0.0082(44) | -0.0478(68) | 0.1538(7) | (-0.0046(48.5) | -0.0103(53) | -0.0163(57) | -0.0046(48.5) | 404
biodegradation | -0.0103(54) | 0.0096(43) | -0.1079(86) | 0.0371(28) | 0.0343(32.5) | -0.0112(55) | 0.0143(41) | 0.0343(32.5) | 372
car | -0.1761(93) | -0.0976(81) | -0.1043(83) | -0.0749(76) | 0.0259(37) | 0.1150(11) | 0.1310(9) | 0.1810(5) | 395
ClimateModel | -0.1044(84) | -0.0766(77) | -0.1137(87) | -0.0674(73) | -0.0248(60) | 0.1919(4) | -0.0988(82) | 0.2937(2) | 469
cred | 0.0560(23) | -0.0542(70) | -0.0469(67) | 0.0748(17) | -0.0005(46) | -0.0063(51) | -0.0527(69) | 0.0299(35) | 378
dermatology | 0.0850(15) | -0.1144(88) | -0.0816(79) | 0.1233(10) | 0.0031(45) | -0.0707(74) | -0.0434(66) | 0.0987(13) | 390
HabermanSurvival | -0.0282(63) | -0.0367(65) | -0.1328(90) | -0.0249(61) | 0.0731(18.5) | 0.0531(24) | 0.0231(39) | 0.0731(18.5) | 379
ILPD | 0.0255(38) | -0.0036(47) | -0.0259(62) | 0.1547(6) | -0.1477(92) | -0.0073(52) | -0.0056(50) | 0.0101(42) | 389
Kdd | -0.1202(89) | -0.2851(96) | -0.0562(71) | 0.3151(1) | -0.0312(64) | 0.0485(25) | -0.0140(56) | 0.1430(8) | 410
OLD | 0.0399(27) | -0.1049(85) | 0.0352(30) | 0.0703(20) | -0.0734(75) | -0.0185(58) | 0.0167(40) | 0.0348(31) | 366
parkinsons | -0.0668(72) | -0.0975(80) | -0.1386(91) | 0.1025(12) | 0.0614(21.5) | 0.0461(26) | 0.0314(34) | 0.0614(21.5) | 358
secom | 0.0370(29) | 0.0281(36) | 0.0945(14) | 0.1998(3) | 0.0836(16) | -0.2006(94) | -0.2214(95) | -0.0210(59) | 346
Total | 665 | 812 | 828 | 314 | 556 | 527 | 638 | 316 | 4656
Average rank | 55.4167 | 67.6667 | 69.0000 | 26.1667 | 46.3333 | 43.9167 | 53.1667 | 26.3333 |

##  V Conclusions

In this paper, we proposed an novel unsupervised collaborative representation
deep network architectures to explore the capability of deep collaborative
representation for real-valued and binary data. We used the proposed UCRDNet
with Gaussian linear visible units and binary visible units to show how the
deep collaborative representation architecture can yield superior performance
as a collaborative feature extractor and compared it to the Autoencoder,
DeepFS deep networks and CDL deep collaborative networks for unsupervised
clustering on the MSRA-MM2.0 datasets and UCI datasets, respectively. From the
experimental results, we can conclude that our UCRDNet not only has more
powerful deep learning capability than Autoencoder and DeepFS, but also has
more excellent deep collaborative representation capability than CDL deep
collaborative network. Furthermore, the hidden collaborative features of the
UCRDNet show powerful generalization ability for different algorithms. In the
future work, we will explore the effect of the hidden layer dimensionality on
the collaborative representation performance.

##  VI Acknowledgement

This work were partially supported by the National Science Foundation of China
(Nos. 61773324, 61573292, 61876158) and Sichuan Science and Technology Program
(2019YFS0432).

## References

  * [1] C. Zheng and N. Wang, “Collaborative representation with k-nearest classes for classification,” _Pattern Recognition Letters_ , vol. 117, pp. 30 – 36, 2019\.
  * [2] P. Du, L. Gan, J. Xia, and D. Wang, “Multikernel adaptive collaborative representation for hyperspectral image classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 56, no. 8, pp. 4664–4677, Aug 2018.
  * [3] W. Li, Q. Du, and B. Zhang, “Combined sparse and collaborative representation for hyperspectral target detection,” _Pattern Recognition_ , vol. 48, no. 12, pp. 3904–3916, 2015.
  * [4] J. Feng, X. Ma, and W. Zhuang, “Collaborative representation bayesian face recognition,” _Science China Information Sciences_ , vol. 60, no. 4, p. 048101, Dec 2016.
  * [5] J. Lai and X. Jiang, “Class-wise sparse and collaborative patch representation for face recognition,” _IEEE Trans Image Process_ , vol. 25, no. 7, pp. 3261–3272, 2016.
  * [6] W. Deng, J. Hu, and J. Guo, “Face recognition via collaborative representation: Its discriminant nature and superposed representation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 10, pp. 2513–2521, Oct 2018.
  * [7] M. Fu, H. Qu, Z. Yi, L. Lu, and Y. Liu, “A novel deep learning-based collaborative filtering model for recommendation system,” _IEEE Transactions on Cybernetics_ , vol. 49, no. 3, pp. 1084–1096, March 2019.
  * [8] H. Yue, J. Liu, J. Yang, T. Nguyen, and F. Wu, “High iso jpeg image denoising by deep fusion of collaborative and convolutional filtering,” _IEEE Transactions on Image Processing_ , pp. 1–1, 2019.
  * [9] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, “Robust face recognition via sparse representation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 31, no. 2, pp. 210–227, Feb 2009.
  * [10] N. Akhtar, F. Shafait, and A. Mian, “Efficient classification with sparsity augmented collaborative representation,” _Pattern Recognition_ , vol. 65, no. Complete, pp. 136–145, 2017.
  * [11] W. Hu, B. Cai, A. Zhang, V. D. Calhoun, and Y. Wang, “Deep collaborative learning with application to multimodal brain development study,” _IEEE Transactions on Biomedical Engineering_ , pp. 1–1, 2019.
  * [12] R. Zhang, J. Li, H. Sun, Y. Ge, P. Luo, X. Wang, and L. Lin, “Scan: Self-and-collaborative attention network for video person re-identification,” _IEEE Transactions on Image Processing_ , pp. 1–1, 2019\.
  * [13] J. Liu, Y. Wang, Y. Li, J. Fu, J. Li, and H. Lu, “Collaborative deconvolutional neural networks for joint depth estimation and semantic segmentation,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 29, no. 11, pp. 5655–5666, Nov 2018.
  * [14] M. Zhu, J. Li, N. Wang, and X. Gao, “A deep collaborative framework for face photo-sketch synthesis,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–13, 2019.
  * [15] Z. Li, J. Tang, and T. Mei, “Deep collaborative embedding for social image understanding,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2018.
  * [16] G. Hinton and T. Sejnowski, “Learning and releaming in boltzmann machines,” _Parallel distributed processing: Explorations in the microstructure of cognition_ , vol. 1, pp. 282–317, 1986.
  * [17] G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” _Neural Computation_ , vol. 14, no. 8, pp. 1771–1800, 2002\.
  * [18] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” 2009.
  * [19] M. A. Carreira-Perpinan and G. E. Hinton, “On contrastive divergence learning,” in _Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics_. Citeseer, 2005, pp. 33–40.
  * [20] S. Osindero and G. E. Hinton, “Modeling image patches with a directed hierarchy of markov random fields,” in _Advances in Neural Information Processing Systems_ , 2008, pp. 1121–1128.
  * [21] A. Courville, G. Desjardins, J. Bergstra, and Y. Bengio, “The spike-and-slab rbm and extensions to discrete and sparse data distributions,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 36, no. 9, pp. 1874–1887, 2014.
  * [22] C. Chen, C. Y. Zhang, L. Chen, and M. Gan, “Fuzzy restricted boltzmann machine for the enhancement of deep learning,” _IEEE Transactions on Fuzzy Systems_ , vol. 23, no. 6, pp. 2163–2173, 2015.
  * [23] Z. Han, Z. Liu, J. Han, C.-M. Vong, S. Bu, and X. Li, “Unsupervised 3d local feature learning by circle convolutional restricted boltzmann machine,” _IEEE Transactions on Image Processing_ , vol. 25, no. 11, pp. 5331–5344, 2016.
  * [24] C. L. P. Chen and S. Feng, “Generative and discriminative fuzzy restricted boltzmann machine learning for text and image classification,” _IEEE Transactions on Cybernetics_ , pp. 1–12, 2018.
  * [25] D. Chen, J. Lv, and Z. Yi, “Graph regularized restricted boltzmann machine,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 29, no. 6, pp. 2651–2659, June 2018.
  * [26] S. Wang, Z. Zheng, S. Yin, J. Yang, and Q. Ji, “A novel dynamic model capturing spatial and temporal patterns for facial expression analysis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2019.
  * [27] J. Chu, H. Wang, H. Meng, P. Jin, and T. Li, “Restricted boltzmann machines with gaussian visible units guided by pairwise constraints,” _IEEE Transactions on Cybernetics_ , vol. 49, pp. 4321–4334, December 2019.
  * [28] R. Salakhutdinov and G. Hinton, “An efficient learning procedure for deep boltzmann machines,” _Neural Computation_ , vol. 24, no. 8, pp. 1967–2006, 2012.
  * [29] K. H. Cho, T. Raiko, and A. Ilin, “Gaussian-bernoulli deep boltzmann machine,” in _the 2013 International Joint Conference on Neural Networks_. IEEE, 2013, pp. 1–7.
  * [30] J. Zhang, G. Tian, Y. Mu, and W. Fan, “Supervised deep learning with auxiliary networks,” in _the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, 2014, pp. 353–361.
  * [31] Y. Sun, X. Wang, and X. Tang, “Hybrid deep learning for face verification,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 10, pp. 1997–2009, Oct 2016.
  * [32] L. Kim, “Deepx: Deep learning accelerator for restricted boltzmann machine artificial neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 29, no. 5, pp. 1441–1453, May 2018.
  * [33] L. Gu, J. Huang, and L. Yang, “On the representational power of restricted boltzmann machines for symmetric functions and boolean functions,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 30, no. 5, pp. 1335–1347, May 2019.
  * [34] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” _Science_ , vol. 313, no. 5786, pp. 504–507, 2006\.
  * [35] A. Taherkhani, G. Cosma, and T. M. McGinnity, “Deep-fs: A feature selection algorithm for deep boltzmann machines,” _Neurocomputing_ , vol. 322, pp. 22–37, 2018.
  * [36] H. Wang, N. Wang, and D. Yeung, “Collaborative deep learning for recommender systems,” in _SIGKDD_ , 2015, pp. 1235–1244.
  * [37] F. O. D. Fran?a, “A hash-based co-clustering algorithm for categorical data,” _Expert Systems with Applications_ , vol. 64, pp. 24–35, 2016.
  * [38] S. Har-Peled, P. Indyk, and R. Motwani, “Approximate nearest neighbor: Towards removing the curse of dimensionality,” _Theory of Computing_ , vol. 8, no. 14, pp. 321–350, 2012.
  * [39] A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh, “Clustering with bregman divergences,” _Journal of Machine Learning Research_ , vol. 6, no. 4, pp. 1705–1749, 2005.
  * [40] “Twcc: Automated two-way subspace weighting partitional co-clustering,” _Pattern Recognition_ , vol. 76, pp. 404 – 415, 2018.
  * [41] S. Lloyd, “Least squares quantization in pcm,” _IEEE Transactions on Information Theory_ , vol. 28, no. 2, pp. 129–137, 1982.
  * [42] A. Y. Ng, M. I. Jordan, Y. Weiss _et al._ , “On spectral clustering: Analysis and an algorithm,” _Advances in Neural Information Processing Systems_ , vol. 2, pp. 849–856, 2002.
  * [43] D. Cai, X. He, and J. Han, “Document clustering using locality preserving indexing,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 17, no. 12, pp. 1624–1637, 2005.
  * [44] F. O. de Fran?a, “A hash-based co-clustering algorithm for categorical data,” _Expert Systems with Applications_ , vol. 64, pp. 24 – 35, 2016.
  * [45] R. Liu, H. Wang, and X. Yu, “Shared-nearest-neighbor-based clustering by fast search and find of density peaks,” _Information Sciences_ , vol. 450, pp. 200 – 226, 2018.
  * [46] S. García, A. Fernández, J. Luengo, and F. Herrera, “Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power,” _Information Sciences_ , vol. 180, no. 10, pp. 2044–2064, 2010.
  * [47] H. Li, M. Wang, and X.-S. Hua, “Msra-mm 2.0: A large-scale web multimedia dataset,” in _IEEE International Conference on Data Mining Workshops_. IEEE, 2009, pp. 164–169.

|  Jielei Chu received the B.S. and Ph.D. degrees in computer application and
computer science and technology from Southwest Jiaotong University, Chengdu,
China in 2008 and 2020. His research interests include deep learning, big
data, semi-supervised learning and ensemble learning. He is a member of the
IEEE.
---|---
|  Hongjun Wang received his Ph.D. degree in computer science from Sichuan
University of China in 2009. He is currently Associate Professor of the Key
Lab of Cloud Computing and Intelligent Techniques in Southwest Jiaotong
University. His research interests include machine learning, data mining and
ensemble learning. He published over 50 research papers in journals and
conferences and he is a member of ACM and CCF.
---|---
|  Jing Liu received his Ph.D. degree in management from Southwest Jiaotong
University.She is currently an Assistant Professor of Business School in
Sichuan University. Her research interests are machine learning, financial
technology and modelling and forecasting high-frequency data.
---|---
|  Zeng Yu received the BS and MS degrees from the School of Mathematics,
China University of Mining and Technology in 2008 and 2011, and the PhD degree
from the School of Information Science and Technology, Southwest Jiaotong
University in 2018, respectively. He was a visiting PhD student with Georgia
State University, USA, from 2014 to 2016. He is currently an Assistant
Professor in the School of Information Science and Technology, Southwest
Jiaotong University. His research interests include data mining and deep
learning.
---|---
|  Zhiguo Gong received the Ph.D. degree in computer science from the
Institute of Mathematics, Chinese Academy of Science, Beijing,China. He is
currently a Professor with the Faculty of Science and Technology, University
of Macau, Macau, China. His current research interests include machine
learning, data mining, database, and information retrieval.
---|---
|  Tianrui Li (SM’11) received the B.S., M.S., and Ph.D. degrees from
Southwest Jiaotong University, Chengdu, China, in 1992, 1995, and 2002,
respectively. He was a Post-Doctoral Researcher with Belgian Nuclear Research
Centre, Mol, Belgium, from 2005 to 2006, and a Visiting Professor with Hasselt
University, Hasselt, Belgium, in 2008; University of Technology, Sydney,
Australia, in 2009; and University of Regina, Regina, Canada, in 2014. He is
currently a Professor and the Director of the Key Laboratory of Cloud
Computing and Intelligent Techniques, Southwest Jiaotong University. He has
authored or co-authored over 300 research papers in refereed journals and
conferences. His research interests include big data, machine learning, data
mining, granular computing, and rough sets.
---|---