2021

[1]Xuesong Yin

1]Department of Digital Media Technology, Hangzhou Dianzi University,
Hangzhou, 310018, China

# Self-Supervised Deep Subspace Clustering with Entropy-norm

Guangyi Zhao<EMAIL_ADDRESS>Simin Kou  yinxs@hdu.edu.cn(X. Yin) Guodao
Zhang  Jieyue Yu  Yigang Wang  [

###### Abstract

Auto-Encoder based deep subspace clustering (DSC) is widely used in computer
vision, motion segmentation and image processing. However, it suffers from the
following three issues in the self-expressive matrix learning process: the
first one is less useful information for learning self-expressive weights due
to the simple reconstruction loss; the second one is that the construction of
the self-expression layer associated with the sample size requires high-
computational cost; and the last one is the limited connectivity of the
existing regularization terms. In order to address these issues, in this paper
we propose a novel model named Self-Supervised deep Subspace Clustering with
Entropy-norm (S3CE). Specifically, S3CE exploits a self-supervised contrastive
network to gain a more effetive feature vector. The local structure and dense
connectivity of the original data benefit from the self-expressive layer and
additional entropy-norm constraint. Moreover, a new module with data
enhancement is designed to help S3CE focus on the key information of data, and
improve the clustering performance of positive and negative instances through
spectral clustering. Extensive experimental results demonstrate the superior
performance of S3CE in comparison to the state-of-the-art approaches.

###### keywords:

Deep subspace clustering, Self-Supervise, Entropy-norm

##  1 Introduction

As an important technique of unsupervised learning, clustering is widely used
in computer vision, pattern recognition and other fields. It divides the data
into different clusters by calculating the similarities among data points and
maximizes the inter-cluster variability and intra-cluster similarity. The
impressive performance of subspace clustering bib1 ; bib3  in grouping high-
dimensional data has made it a hot topic in the field in recent years.

Most subspace clustering focus on spectral clustering-based methods, which are
usually divided by two subproblems. One is to construct the affinity matrix
based on the self-expression property bib2 , i.e., each data point can be
expressed as a linear combination of other points. The other is to segment
data into different clusters via normalized cuts or spectral clustering. In
general, constructing a proper affinity matrix is a challenging task. Thus,
recent researchers use different types of regularization, such as the nuclear
norm bib4 , $\ell_{0}$-norm bib7 , $\ell_{2}$-norm bib5 , traceLasso norm bib8
and various variants bib9 ; bib10 ; bib11 ; bib12 ; bib13 ; bib14  to get the
high-quality affinity matrix. However, it is unsatisfying to simply fit the
data into a linear relationship, and it is also difficult to choose a suitable
kernel in kernel methods bib50 .

The enormous success of deep neural networks (DNNs) in many fields gives a new
direction to the development of clustering. bib15 ; bib16 ; bib17 ; bib18 ;
bib19 ; bib20 ; bib21 ; bib23 . Compared with traditional subspace clustering,
deep subspace clustering methods utilize deep neural networks to learn a
nonlinear mapping from the original data samples to a latent feature space.
The introduction of DNNS compensates for the lack of attention to a local
structure by the self-expression property. Various deep models such as
Autoencoders (AE) bib25 , Generative Adversarial Network (GAN) bib26  and
Variational Autoencoders (VAE) bib27  are used in deep subspace clustering to
obtain a good representation of the data. An impressive model is deep subspace
clustering (DSC) bib16 , which introduces a self-expression layer between the
encoder and decoder in the autoencoder. Although the emergence of the AE
brings a qualitative leap in the effectiveness of subspace clustering, the
quality of the representations obtained by it is beyond satisfactory. In other
words, the reconstruction loss of AE focuses too much on pixel-level
differences and ignores important local information between neighboring
points. Deep subspace clustering expects to obtain a representation through
the model that better reflects the important features of the data, but
reconstruction loss imposes a resistance to this goal.

Self-supervised learning is a kind of the unsupervised learning, which learns
a general feature representation for downstream tasks by supervising itself.
Traditional autoencoders mainly obtain a representation of the original data
sample in the implicit space by reconstructing the paradigm of the input data
bib25 . This representation, although being effective from the point of view
of data reduction, is more redundant for discriminative tasks, e.g.,
clustering and classification. The reason is that it gives the same level of
attention to various implicit properties in the representation rather than
focusing on the most representative aspects of the data, which is inefficient
for discriminating data. In contrast, contrastive learning bib28 , based on
the idea of maximizing mutual information, is dedicated to using fewer symbols
to express more critical information. That is, it concentrates more on
patterns that occur frequently or repeatedly in the data samples. This reduces
the redundancy and improves the efficiency and friendliness of representation
learning for discriminative tasks. The core idea of contrastive learning is to
acquire a representation learning model by constructing similar and dissimilar
instances. Then a representation is obtained that correctly selects positive
examples from a dataset of mixed positive and negative example samples. The
representation is able to uncover the essence of the data and then
discriminate it, which facilitates the efficient completion of downstream
tasks.

Figure 1: The architecture of Self-Supervised Deep Subspace Clustering with
Entropy-norm (S3CE ). S3CE consists of two stages: pre-training stage and
subspace clustering stage. The data augmented is used to train an encoder by
self-supervised contrastive learning in the pre-training stage. In particular,
S3CE feeds the data into a pre-trained encoder and fine-tunes it. The self-
expression property and the entropy-norm are used to obtain the clustering
results

Motivated by recent progress in deep subspace clustering and self-supervised
learning, in this paper we propose a self-supervised deep subspace clustering
with entropy-norm(S3CE). S3CE can obtain a good encoder by contrastive
discriminative methods during the pre-training stage, and then the encoder
will be used to get representation with frozen parameters. Finally, our method
will use the representation after the projection layer mapping to obtain the
similarity matrix with entropy-norm. The result on seven benchmark datasets
show the effectiveness of our method.

As shown in Fig. 1, the structure of our method consists of a pre-training
phase and a deep subspace clustering phase. The highlights of our approach are
the following four-fold:

(1) Rather than focusing on self-reconstruction which tends to be detrimental
to the relationships between data points, we utilize one encoder to reduce the
time cost in the learning process. In addition, the local structure
information between data points is effectively preserved after removing the
encoder.

(2) We introduce the entropy-norm for better constraining the affinity to
unify and tension the distribution of data samples in each subspace, which is
more useful to improve the performance with the next step of the spectral
clustering.

(3) We exploit data augmentation to allow the encoder to learn the key
information in the data and fine-tune the frozen model with two linear layers.
Moreover, we present a general framework by combining contrastive learning
into DSC, which is more suitable for solving real-world problems.

(4) We analyze the consistency of contrast learning and the entropy-norm in
constructing and optimizing the coefficient matrix from an entropy
perspective. It is important to form a clear, uniform and well-spaced block-
diagonal matrix.

##  2 Related Works

###  2.1 Traditional Subspace Clustering

Subspace clustering (SC) refers to the method of dividing the data into
different classes, and each class corresponds to a different subspace under
ideal conditions. Most traditional SC algorithms are based on linear
representation, namely self-expression theory. Specifically, the whole sample
data matrix is used as a dictionary to obtain the linear representation of
each data point (e.g. Eq. 2), and then the similarity matrix $W$ is
constructed by using the obtained coefficient matrix C. Mathematically, this
idea can be expressed as:

$\min_{C\in\mathbb{R}^{n\times
n}}\frac{1}{2}\|X-XC\|_{F}^{2}+\|C\|_{p}\quad\text{ s.t.
}\operatorname{diag}(C)=0$ (1)

where $\|\cdot\|_{F}^{2}$ represents Frobenius norm, $X$ is the data matrix
consists of a set of data points $\left\\{x_{i}\right\\}_{i=1,\ldots,N}$ which
let the data points belong to the union of $K$ linear subspace
$\left\\{S_{i}\right\\}_{i=1}^{K}$. Here $\|\cdot\|_{p}$ represents any one
norm, and uses the diagonal constraint on $C$ avoids trivial solutions for
sparsity inducing norms, such as the $\ell_{1}$-norm in SSC bib2 , the
$\ell_{2}$-norm in LSR bib11 , the $\ell_{2,1}$-norm in LRR bib4 , the nuclear
norm in LRSC bib29 , the Forbenius norm in EDSC bib30 , etc. It is noteworthy
that the properties of data similarity matrix $W$ are sparse, symmetrical,
nonnegative and nonlinearly representational. In fact, the SSC algorithms
hardly guarantee the nonnegative and symmetry attribute of matrix $W$ because
SSC explicitly converts the coefficient matrix $C$ by Eq. 3 into a symmetrical
and nonnegative matrix $W$. This conversation may bring some misleading
information. Although some works add a few mandatory constraints such as the
symmetric positive semi-definite (PSD) bib31 , the computation of sparse
optimization program is very complicated, and it may even be a NP-Hard
problem.

$x_{j}=\sum_{i\neq j}c_{ij}x_{i}+e_{j}$ (2)

where $e_{j}$ denotes as a disturbance term for noise or corruption, $x_{j}$
is an arbitrary data point, $c_{ij}$ represents to an entry of column $j^{th}$
of row $i^{th}$ in self-expression coefficient matrix $C$. Moreover,
$c_{ij}\neq 0$ and $c_{ii}=0$ are guaranteed to maintain the block-diagonal
structure under the assumption of subspace independence bib30 .

$W=\frac{(\lvert C\rvert+\lvert C^{T}\rvert)}{2}$ (3)

Specifically, SC relies on the self-expressive to obtain coefficient matrix
$C$, then generally constructs the data similarity graph $W$ by Eq. 3, and
finally leverages the spectral clustering to obtain the clustering results.

###  2.2 Deep Subspace Clustering

Compared with traditional subspace clustering, deep subspace clustering (DSC)
aims to seek the deep representation of data samples. Deep representation of
the original data is used to optimize the affinity matrix by a typical sparse
or low-rank regularization strategy. This substantially improves the
performance of subspace clustering for high-dimensional complex data samples
at the nonlinear structure level. The well-known DSC-Net bib16  with the
addition of an autoencoder describes the generic DSC model as:

$\mathcal{L}(\Theta,C)=\frac{1}{2}\|X-\hat{X}\|_{F}^{2}+\lambda_{1}\|C\|_{q}+\lambda_{2}\|Z-ZC\|_{F}^{2},\quad\text{
s.t. }\operatorname{diag}(C)=0$ (4)

where $\hat{X},Z$ represents the original data $X$ reconstructed by the
decoder and the output of the encoder. $\lambda_{1}>0$ and $\lambda_{2}>0$ are
the tradeoff parameters. DSC-Nets, as a general framework for DSC, introduces
a self-representation layer constructed by a single fully connected network
layer without bias and activation to implement the self-representation
property in traditional subspace clustering. It is ingenious that embeds the
layer between the encoder and the decoder to guide the encoder to learn a
valid discriminative representation.

Following the DSC framework, a number of works have been proposed. DASC bib18
gets the real data and the faked data after a linear combination by the
generator and feeds them into the discriminator. After adversarial learning, a
better similarity matrix $C$ and feature expression $Z$ would be obtained.
Using a dual self-supervision mechanism, S2ConvSCN bib20  makes the
convolution, self-representation module and spectral clustering into an
organic whole. Also, for self-supervised learning, PSSC bib23  introduces
pseudo-graphs and pseudo-labels to supervise similarity learning during the
training process. In addition, DSC is also used in multi-view bib33  and
multi-modal data bib32 . However, almost all existing methods use the self-
reconstruction loss which is less friendly to relationships between data
points.

###  2.3 Self-Supervised Learning

According to Liu et al. bib48 , self-supervised learning is divided into three
main categories: Generative, Contrastive, and Adversarial (Generative-
Contrastive). Generative methods mainly map the original data to an explicit
vector and then revert to the original data through losses such as Mean Square
Error (MSE). The representative of this type of approach is Auto-Encoding (AE)
model. The adversarial approach improves the realism of the generated images
using the generator against the discriminator. The GAN model is a concrete
manifestation of this method.

Contrastive learning focuses on salient feature areas of an image by pulling
in positive examples and pushing out negative examples. Unlike mapping
features to classes singularly, contrastive learning increases inter-class
dispersion and intra-class compactness. With the information entropy loss
constraint, the model can learn valid class-related information, which guides
more robust feature learning.

Proposed by Oord et al. bib28 , the objective function of the earliest
contrastive self-supervised models Contrastive Predictive Coding (CPC) can be
described by Eq. 5 called InfoNCE. To learn better representations, self-
supervised research work based on the comparison of positive and negative
examples has developed rapidly in recent years. Tian et al. bib35  proposed
Contrastive Multiview Coding (CMC), which increases the number of positive
examples with multimodality of data and uses a cached form of Memory Bank to
quickly acquire more negative example representations. However, the
representation is not a real-time negative example representation acquired by
the current model and the information lacks consistency. Thereafter, He et al.
bib36 ; bib37 ; bib38  proposed the MoCo model using a dictionary-based
momentum update method, which solved the non-consistency problem of CMC to
some extent. Focusing more on visual content, Chen et al. bib39  proposed a
comparative representation learning framework SimCLR. SimCLR treats a pair of
augmented data of a certain data sample as a positive example and augmented
data of other samples as a negative example. It is inspiring to design a
general framework and achieve acceptable results at the same time.

$\mathcal{L}_{N}^{\text{InfoNCE
}}=-\mathbb{E}_{X}\left[\log\frac{\exp\left(f(x)^{T}f\left(x^{+}\right)\right)}{\exp\left(f(x)^{T}f\left(x^{+}\right)\right)+\sum_{j=1}^{N}\exp\left(f(x)^{T}f\left(x^{-}\right)\right)}\right]$
(5)

where $f(\cdot)$ is the model to be learned. $x^{+}$ and $x^{-}$ denote
positive and negative examples respectively, i.e. , similar samples and
dissimilar samples.

##  3 Proposed Method

In this section, we will present our S3CE in detail. Our model contains two
main parts: a contrastive pre-training module and a subspace clustering module
with entropy-norm. In addition, we will discuss the entropy-norm regularized
DSC.

###  3.1 Pre-training stage with Contrastive Learning

In general, we assume that the more important information of the original data
is retained, the better the feature extraction is. One conventional practice
is that reducing the encoded vector back to the original image through the
decoder. The final loss is the MSE of the original image and the reconstructed
image. Later, the distribution of the encoded vector is limited to be as close
to a Gaussian distribution as possible, which leads to the Variational
Autoencoders. However, one can find that the reconstruction of the original
image by low-dimensional encoding is usually blurred, which can be explained
by the fact that the loss function MSE requires too demanding pixel-by-pixel
reconstruction. Alternatively, it can be explained by the absence of a
suitable loss for image reconstruction. The ideal approach would be to train a
discriminator using an adversarial network, but this would further increase
the difficulty of the task.

But the truth is that when dealing with tasks like image classification, image
clustering, models don’t need to draw these images all over again but just
distinguish the differences in these images. That is, reasonable and
sufficient features for the dataset and task do not necessarily accomplish
image reconstruction. The basic principle of a good feature should be the
ability to identify that sample from the entire data set. In other words,
extracting the (most) unique information about the sample. How to measure that
the extracted information is unique to the sample? We use Eq. 5 proposed by
Oord et al. bib28  instead of MSE calculated by the encoder and decoder to
measure.

In the pre-training stage, S3CE first obtains $2N$ enhanced images after a
combination of several data enhancement operations such as random cropping,
random rotation, random color jitter, Gaussian Blur, random grayscale and
flipping on a batch of $N$ input images. These data enhancements methods are
randomly combined, which can introduce more uncertainty to the data to be fed
into the model. Next, the $2N$ enhanced images are put into the encoder to
obtain $N$ pairs of high-dimensional vector $h_{a}$ and $h_{b}$. Then, $h_{i}$
and $h_{j}$ will pass through a projection layer consisting of MLPs into $N$
pairs of $z_{i}$ and $z_{j}$. Finally, we compute the cosine similarity of
each pair $z_{i}$ and $z_{j}$ as $s_{i,j}$.

$s_{i,j}=\frac{z_{i}z_{j}}{\left\|z_{i}\right\|\left\|z_{j}\right\|}$ (6)

A positive pair of the infoNCE loss $l(i,j)$ is defined as:

$l(i,j)=-\log\frac{\exp\left(\frac{s_{i,j}}{T}\right)}{\sum_{k=1}^{2N}\exp\left(\frac{s_{i,k}}{T}\right)}\quad(k\neq
i)$ (7)

where $T$ denotes a temperature parameter. The final loss of the pre-training
stage is calculated for all positive pairs, which can be defined as:

$\mathcal{L}_{1}=\frac{1}{2N}\sum_{k=1}^{N}[l(2k-1,2k)+l(2k,2k-1)]$ (8)

S3CE chooses Resnet-50 bib42  as encoder to handle the images processed by
data enhancement. Logically, there is no requirement for the size of the
dataset at this stage. We map the feature vector of the encoder output to
$B\times N$ dimensions, where $B$ is the batch size and $N$ is related to the
chosen encoder. The similarity is calculated using a two-layer MLPs to project
the features into the lower dimension. Pre-trained encoder will freeze the
weight for the next stage as a feature extractor.

###  3.2 Entropy-norm regularized DSC

In reinforcement learning, the probability of the output sometimes is expected
not to be focused on one action when training a neural network . At least some
non-zero probability should be given to other actions that can be explored.
The entropy component can be added to the final loss to make the probability
distribution more chaotic, that is, to make the probability distribution more
uniform and not simply over-converge to one action. For example, the purpose
of SAC bib49  is to encourage exploration and achieve a multi-modal effect.

High-quality subspace clustering requires high similarity within subspaces and
low similarity between subspaces, i.e., tight aggregation within subspaces and
mutual independence of each subspace. Among them, there may be multiple
connected components in each subspace under the sparse constraint. That is,
the weak connection between certain data within that subspace increases the
possibility of these samples being incorrectly partitioned into adjacent
subspaces. While there is only one connected component in each subspace under
the low-rank constraint, some data points are only associated with a small
number of samples in that subspace, which leading to a larger possibility that
these data samples are incorrectly assigned to other subspaces.

Entropy-norm was proposed by Bai et al. bib34  as a regularization term to
replace the $\ell_{1}$-norm for SSC, namely SSC+E, and the objective function
is defined as:

$\mathrm{Q}=\sum_{i=1}^{n}\sum_{j=1}^{n}\left\|x_{i}-x_{j}\right\|w_{ij}+\gamma\sum_{i=1}^{n}\sum_{j=1}^{n}w_{ij}\ln
w_{ij}\quad\text{ s.t., }\sum_{i=1}^{n}w_{ij}=1,\operatorname{diag}(W)=0$ (9)

where $w_{ij}$ represents the $i^{th}$ row and $j^{th}$ column of the affinity
matrix $W$. Under the regularization of entropy-norm, the similarity between
data in the subspace is uniformly and densely distributed, which is more
suitable for the requirement of tight aggregation in the subspace. This
distribution is easier to achieve concatenation between any two data samples.
The schematic plots of the affinity matrices with different norm constraint is
shown in Fig. 2 bib51 . One can observe that the affinity matrix used for
spectral clustering has better connectivity under the entropy-norm constraint.
Its uniformly distributed block-diagonal structure is closer to the affinity
matrix in the ideal state.

Figure 2: Schematic plot of various affinity matrices under different norm
constraints. (a)Ideal (b)Sparse (c)Low-rank (d)Ours

###  3.3 Subspace clustering stage with Entropy-norm

Inspired by SSC+E and the power of DSC-Net, we impose the entropy-norm
constraint on the coefficient matrix $C$ learned from the encoder. Meanwhile,
considering that reconstruction loss not only ignores the rich relational
information between adjacent points, but also reduces the quality of potential
representations. Our model only use the self-expression loss as well as the
entropy-norm constraint to train the coefficient matrix. Considering that
$c_{ij}$ is a value between 0 and 1, the value after the entropy-norm
constraint is negative. Therefore, we use a function to constrain the value of
this item to between 0 and 1. The object function of the subspace clustering
module can be defined as:

$\mathcal{L}_{2}=\lambda_{1}\|Z-ZC\|_{F}^{2}+\lambda_{2}\exp\left(\sum_{i=1}^{n}\sum_{j=1}^{n}c_{ij}\ln
c_{ij}\right),\quad\text{ s.t. }\sum_{j=1}^{n}c_{ij}=1$ (10)

where $Z$ represents the output of the encoder obtained from the pre-training
stage. $\lambda_{1}>0$ and $\lambda_{2}>0$ are the tradeoff parameters.
$c_{ij}$ denotes the $i^{th}$ row and $j^{th}$ column of the coefficient
matrix $C$. Unlike most existing deep subspace clustering algorithms that use
a typical method to enhance the block-diagonal structure bib30 . After the
coefficient matrix $C$ obtained, S3CE uses Eq. 4 to calculate the affinity
matrix W directly. Compared to previous methods, the proposed method can
eliminate the need to explore multiple trade-off parameter. It makes our
method more general and can be applied to more practical problems.

Therefore, the objective functions 8 and 10 consist the entire objective
functions of our S3CE. Mutual information is applied to contrastive learning
from the perspective of entropy. Mutual information is a measure of the
correlation between two sets of events. In other words, it is the amount of
information obtained by observing another random variable after obtaining
information about one random variable. The potential variables shared by the
input are extracted in CPC bib28  by maximizing the mutual information between
the coded representations.

The use of entropy-norm in subspace clustering defines the maximum information
entropy as the regular term driving the learning affinity matrix W. The
problem of poor symmetry of the sparse self-representation matrix can be
solved.

Contrastive learning based on mutual information in the pre-training stage
reduces the similarity between each potential vector except itself. It makes
the representations encoded by the encoder enhanced independence between each
potential vector when constructing the self-expression matrix. Moreover, the
weights between different categories in the coefficient matrix are reduced. At
the same time, thanks to the entropy-norm, each sample occupies neither too
large nor too small a weight in the coefficient matrix when the same
categories are represented to each other in the coefficient matrix. The
distribution of the data samples within each subspace are more uniform and
dense. This benefits to the next step of spectral clustering to perform graph
partitioning algorithm in accuracy, which improves the subspace clustering
performance.

##  4 Experiments

To evaluate the effectiveness of our S3CE, we conduct experiments on seven
benchmark datasets and compare with eight subspace clustering methods in this
section.

###  4.1 Datasets

Figure 3: Sample images from seven datasets used in the experiments. (a)
CIFAR10 (b) COIL40 and COIL100 (c) OxFlowers17 (d) COIL20 (e) Fashion-MNIST
(f) MNIST-1000

Datasets include three gray-scale datasets (COIL20 bib41 , Fashion-MNIST bib44
and MNIST-1000 bib43 ) and five RGB datasets (COIL40 bib41 , COIL100 bib41 ,
CIFAR-10 bib45 , OxFlowers17 bib46 ).

  * •

COIL20: This dataset contains 1,440 gray-scale images of 20 objects with 72
different views. The original image is 128 × 128 pixels. In order to construct
positive examples, we resize it to 64 × 64 pixels.

  * •

Fashion-MNIST-test: We use the test part of the dataset Fashion-MNIST.
Fashion-MNIST-test covers 10,000 different front images of products from 10
categories (e.g., coat, trouser, shirt, dress, bag, etc.), where each gray-
scale image is of size 28 × 28.

  * •

MNIST-1000: MNIST consists of 70,000 handwritten digits (0-9) in the gray-
scale level. We randomly selected 100 images from each of these 10 categories
to form Mnist-1000. And each image has 28 × 28 pixels.

  * •

COIL40, COIL100: Like coil20, coil40 and coil100 are datasets consisting of
different objects imaged at different angles in a 360° rotation, containing
128 × 128 color images of 100 objects (each with 72 poses). Different from
COIL100, the first 40 classes of COIl100 are taken as COIl40. We also resize
them to 64×64 pixels.

  * •

CIFAR-10: CIFAR-10 is a dataset for identifying pervasive objects. A total of
10 categories of RGB color images are included (e.g. automobile, bird, cat,
deer, etc.). Each image in this dataset has 32×32 pixels. CIFAR-10 is divided
into CIFAR-10-train and CIFAR-10-test, consisting of 50,000 and 10,000 images,
respectively. We use the CIFAR-10-test for training.

  * •

OxFlowers17: This dataset contains 17 species of flowers selected by the
University of Oxford that are common in the UK. The whole dataset has 1360
images and each specie has 80 images. We resize each image to the pixel size
of 64 × 64.

The detailed information for the datasets are summed up in Table 1 and the
examples from seven datasets are shown in Fig. 3.

Table 1: Statistics Of The Datasets Dataset | Samples | Classes | Dimensions
---|---|---|---
COIL20 | 1440 | 20 | 64×64
COIL40 | 2880 | 40 | 64×64
COIL100 | 7200 | 100 | 64×64
OxFlowers17 | 1360 | 17 | 64×64
MNIST-1000 | 1000 | 10 | 28×28
CIFAR-10 | 10000 | 10 | 32×32
Fashion-MNIST | 10000 | 10 | 28×28
|  |  |

###  4.2 Comparison Methods

For fair evaluation and comparison, we compare S3CE with both traditional and
deep subspace clustering methods, including Sparse Subspace Clustering (SSC)
bib2 , elastic net subspace clustering (ENSC) bib47 , SSC by orthogonal
matching pursuit (SSC-OMP) bib22 , Low-rank representation (LRR) bib4 ,
efficient dense subspace clustering (EDSC) bib30 , DSC-Nets bib16 , Deep
Adversarial Subspace Clustering (DASC) bib18 , Self-Supervised Convolutional
Subspace Clustering Network (S2ConvSCN) bib20 .

###  4.3 Evaluation Metrics

To evaluate the clustering performance of our approach, we use the standard
unsupervised evaluation metrics for comparing the results with the above
methods, which are unsupervised clustering Accuracy rate (ACC) and Normalized
Mutual Information (NMI). The higher value of ACC and NMI indicates the better
performance they achieved. ACC can be defined as:

$ACC=\max_{m}\frac{\sum_{i-1}^{n}\mathbb{I}\left\\{l_{i}=m\left(c_{i}\right)\right\\}}{n}$
(11)

where $l_{i}$ is the ground-truth label, $c_{i}$ denotes the subspace
clustering assignment generated by the algorithm, i.e., the number of clusters
K, and m is the range of all possible one-to-one mappings between subspace
clustering and labels. Furthermore, these mappings are usually computed by
Hungarian algorithm in most of existing implementation. NMI represents the
correlation between predict labels and ground-truth labels, where NMI ranges
from 0 to 1. The discrete calculation of NMI is according to:

$NMI=\frac{\sum_{i=1}^{K}\sum_{j=1}^{K}n_{ij}\log\left(\frac{n\cdot
n_{ij}}{n_{j}\cdot\hat{n}_{j}}\right)}{\sqrt{\left(\sum_{i=1}^{K}n_{i}\log\frac{n_{i}}{n}\right)\left(\sum_{j=1}^{K}\hat{n}_{j}\log\frac{\hat{n}_{j}}{n}\right)}}$
(12)

where $n_{i}$ denotes the sample number of the $c_{i}(1\leq i\leq K)$ cluster
provided by the clustering algorithm, $\hat{n}_{j}$ is the number of data
belonging to the $j^{th}$ ground-truth class $K$, and $n_{ij}$ is the number
of data that are in the intersection between the cluster $c_{i}$ and the
$j^{th}$ class.

###  4.4 Implementation Details

We use the Adam optimizer with momentum by the gradient descent algorithm to
minimize the loss function, where the learning rate in pre-training stage is
set as $3.0\times 10^{-4}$ and the learning rate in subspace clustering stage
is set as $1.0\times 10^{-5}$. About hyperparameters, we set $\lambda_{1}=1.0$
and $\lambda_{2}=75.0$ for COIL20 and COIL40, $\lambda_{1}=1.0$ and
$\lambda_{2}=15.0$ for COIL100, $\lambda_{1}=0.1$ and $\lambda_{2}=10.0$ for
OxFlowers17, $\lambda_{1}=1.0$ and $\lambda_{2}=6.0$ for OxFlowers17, It is
noteworthy that in the pre-training stage with contrastive learning, the
larger batch size the higher quality of encoder feature extraction proved by
Chen et al. bib39 . To be fair, in the pre-training stage, the batch size of
all datasets are set to 256. Furthermore, the deep auto-encoder based on
backbone network Resnet-50 bib42  is designed in this paper, but our method is
non-limited to this convolutional network. It can also be a fully-connected or
another convolutional auto-encoder. We also implemented the baseline methods,
in which the details strictly follow the source codes released by the authors
or the suggestions from the corresponding paper. Finally, it should be
stressed here that all the experiments are implemented in Python using
PyTorch1.7.1 on an NVIDIA GeForce RTX 3060 GPU.

###  4.5 Results

Table 2 shows the clustering results on the seven publicly available datasets.
The top-ranked method in each measure is highlighted in bold. In most cases,
S3CE performs significantly better than other methods. Particularly, we have
the following observations.

Figure 4: Visualization of the affinity matrices for all subjects on COIL20
and MNIST-1000 dataset calculated from Eq. 4

  * •

S3CE performs outstandingly better than traditional linear subspace clustering
methods. This is mainly attributed to the excellent fitting ability of the DNN
model.

  * •

S3CE outperforms the other deep clustering methods including DSC. Specially,
S3CE extracts more representative features through comparative learning and
focuses more on local information on object datasets. We have also tried to
test on the face dataset but found the results to be poor. The reason for this
result is speculated to be that the feature differences between faces are not
obvious. It is difficult for contrastive learning to learn effective
information from them.

  * •

Compared with DASC bib18  and S2ConvSCN bib20  ,our S3CE produce superior
performance . Concerning S2ConvSCN, the performance of DASC is more stable.

Table 2: Clustering Results On Benchmark Datasets Datasets | COIL20 | COIL40 | COIL100 | OxFlowers17 | MNIST-1000 | CIFAR-10 | Fashion-MNIST
---|---|---|---|---|---|---|---
Methods | ACC | NMI | ACC | NMI | ACC | NMI | ACC | NMI | ACC | NMI | ACC | NMI | ACC | NMI
SSC | 0.8631 | 0.8892 | 0.7191 | 0.8212 | 0.5510 | 0.5841 | 0.2441 | 0.2741 | 0.4530 | 0.4709 | 0.2189 | 0.1108 | 0.5741 | 0.6286
ENSC | 0.8760 | 0.8952 | 0.7426 | 0.8380 | 0.5732 | 0.5924 | 0.2264 | 0.1689 | 0.4983 | 0.5495 | 0.1718 | 0.0503 | 0.6194 | 0.6578
SSC-OMP | 0.6410 | 0.7412 | 0.4431 | 0.6545 | 0.3271 | 0.6756 | 0.0764 | 0.0941 | 0.3400 | 0.3272 | 0.1115 | 0.0096 | 0.6148 | 0.6216
LRR | 0.8118 | 0.8747 | 0.6493 | 0.7828 | 0.4682 | 0.4721 | 0.0676 | 0.0132 | 0.5386 | 0.5632 | 0.1188 | 0.0037 | 0.2463 | 0.1847
EDSC | 0.8371 | 0.8828 | 0.6870 | 0.8139 | 0.6187 | 0.7369 | —— | —— | 0.5722 | 0.5823 | —— | —— | 0.4667 | 0.5794
DSC | 0.9368 | 0.9408 | 0.8075 | 0.8941 | 0.6771 | 0.8908 | 0.1882 | 0.2496 | 0.7456 | 0.7420 | 0.1978 | 0.0718 | 0.5789 | 0.6215
DASC | 0.9639 | 0.9686 | 0.8354 | 0.9196 | 0.7215 | 0.7286 | 0.1632 | 0.1475 | 0.8040 | 0.7800 | 0.2154 | 0.0768 | 0.5919 | 0.6147
S2ConvSCN | 0.9786 | 0.9633 | 0.6333 | 0.7516 | 0.7333 | 0.8522 | 0.0860 | 0.0700 | 0.8245 | 0.8134 | 0.1052 | 0.0080 | 0.4477 | 0.3865
Ours | 0.9812 | 0.9826 | 0.9490 | 0.9894 | 0.7404 | 0.9440 | 0.5699 | 0.5391 | 0.9160 | 0.8981 | 0.4345 | 0.3341 | 0.6903 | 0.6337

In addition, to visually demonstrate the powerful performance of our models,
we visualize the affinity matrix $W$ of S3CE and DSC according to Eq. 4 in
Fig. 4, where $W_{ij}$ represents the similarity between $Z_{i}$ and $Z_{j}$.
Brighter pixel means higher similarity between two data points. As shown in
the figure, most of the bright pixels are located at the block matrix on the
diagonal. Compared with our method, DSC appears more bright pixels in non-
diagonal regions, which means that the similarity between sample points
calculated by the model is inaccurate. In addition, our method computes a
clearer block-diagonal matrix, which means that the data are more evenly
distributed. This partly explains the outstanding performance of our approach.

Figure 5: 2D visualization of the embedding spaces learned on the MNIST-1000
dataset

Taking the MNIST-1000 data as an example, we visualize the output
representations after a self-expression layer using the t-SNE bib40  method.
As shown in Fig. 5, the data point distribution has a serious overlap between
clusters in DSC. This method can only separate the categories with more
different features. For S2ConvSCN, this method introduces a classification
module to supervise the training of feature learning, the inter-class distance
of the method is enlarged. But the intra-class distance cannot be controlled
to be a small value, so the effect is not too satisfactory. Because of the
introduction of contrastive learning to extract features, it can be seen that
the features extracted by the S3CE method are not only discrete between
classes but also compact within classes.

Figure 6: The influence of parameters on ACC(a) and NMI(b) of COIL20 dataset
Figure 7: The influence of parameters on ACC(a) and NMI(b) of MNIST-1000
dataset

###  4.6 Parameter Analysis

There are two hyperparameters $\lambda_{1}$ and $\lambda_{2}$ in S3CE. These
two hyper-parameters affect the proportion of the self-expression loss and the
entropy-norm in the total loss. In general, they are open and data-dependent,
so we perform the sensitivity experiments on COIL20 and MNIST-1000. We can
gain similar insight on the remaining datasets and thus leave out them. The
ACC and NMI variation with the changes of $\lambda_{1}$ and $\lambda_{2}$
values are shown in Fig. 6-7 with three-dimensional histograms. Although the
parameters affect the performance of our method, S3CE outperforms within a
wide range of values

##  5 Conclusion

In this paper, we present a novel deep subspace clustering algorithm called
S3CE. S3CE firstly combines the idea of contrastive leaning to obtain features
with richer information while keeping the local structure information, and it
introduces an extra regulation term entropy-norm that not only respects the
intrinsic geometric but nonlocal structure of the latent representation.
Experimental results on seven available datasets show that S3CE is superior to
some start-of-the-art subspace clustering algorithms in clustering
performance.

Acknowledgments

This work was supported by Public-welfare Technology Application Research of
Zhejiang Province in China under Grant LGG22F020032, and Key Research and
Development Project of Zhejiang Province in China under Grant 2021C03137,
Zhejiang Provincial Natural Science Foundation of China under Grant
LY21F020001, Science and Technology Plan Project of Wenzhou in China under
Grant ZG2020026.

## Declarations

The authors have no financial or proprietary interests in any material
discussed in this article.

## References

  *   * (1) Vidal, R.: Subspace clustering. IEEE Signal Processing Magazine 28(2), 52–68 (2011)
  * (2) Vidal, R., Ma, Y., Sastry, S.: Generalized principal component analysis (gpca). IEEE transactions on pattern analysis and machine intelligence 27(12), 1945–1959 (2005)
  * (3) Elhamifar, E., Vidal, R.: Sparse subspace clustering: Algorithm, theory, and applications. IEEE transactions on pattern analysis and machine intelligence 35(11), 2765–2781 (2013)
  * (4) Liu, G., Lin, Z., Yu, Y., et al.: Robust subspace segmentation by low-rank representation. In: Icml, vol. 1, p. 8 (2010). Citeseer
  * (5) Yang, Y., Feng, J., Jojic, N., Yang, J., Huang, T.S.: $\ell^{0}$-Sparse Subspace Clustering. In: Computer Vision – ECCV 2016, pp. 731–747 (2016)
  * (6) Favaro, P., Vidal, R., Ravichandran, A.: A closed form solution to robust subspace estimation and clustering. In: CVPR 2011, pp. 1801–1807 (2011). IEEE
  * (7) Lu, C., Feng, J., Lin, Z., Yan, S.: Correlation adaptive subspace segmentation by trace lasso. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1345–1352 (2013)
  * (8) Wang, S., Yuan, X., Yao, T., Yan, S., Shen, J.: Efficient subspace segmentation via quadratic programming. In: Twenty-Fifth AAAI Conference on Artificial Intelligence (2011)
  * (9) Zhuang, L., Gao, H., Lin, Z., Ma, Y., Zhang, X., Yu, N.: Non-negative low rank and sparse graph for semi-supervised learning. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2328–2335 (2012). IEEE
  * (10) Lu, C.-Y., Min, H., Zhao, Z.-Q., Zhu, L., Huang, D.-S., Yan, S.: Robust and efficient subspace segmentation via least squares regression. In: European Conference on Computer Vision, pp. 347–360 (2012). Springer
  * (11) Zheng, Y., Zhang, X., Yang, S., Jiao, L.: Low-rank representation with local constraint for graph construction. Neurocomputing 122, 398–405 (2013)
  * (12) Chen, J., Yi, Z.: Subspace clustering by exploiting a low-rank representation with a symmetric constraint. arXiv preprint arXiv:1403.2330 (2014)
  * (13) Feng, J., Lin, Z., Xu, H., Yan, S.: Robust subspace segmentation with block-diagonal prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3818–3825 (2014)
  * (14) Xiao, S., Tan, M., Xu, D., Dong, Z.Y.: Robust kernel low-rank representation. IEEE transactions on neural networks and learning systems 27(11), 2268–2281 (2015)
  * (15) Peng, X., Xiao, S., Feng, J., Yau, W.-Y., Yi, Z.: Deep subspace clustering with sparsity prior. In: IJCAI, pp. 1925–1931 (2016)
  * (16) Ji, P., Zhang, T., Li, H., Salzmann, M., Reid, I.: Deep subspace clustering networks. Advances in neural information processing systems 30 (2017)
  * (17) Peng, X., Feng, J., Xiao, S., Lu, J., Yi, Z., Yan, S.: Deep sparse subspace clustering. arXiv preprint arXiv:1709.08374 (2017)
  * (18) Zhou, P., Hou, Y., Feng, J.: Deep adversarial subspace clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1596–1604 (2018)
  * (19) Yang, S., Zhu, W., Zhu, Y.: Residual encoder-decoder network for deep subspace clustering. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 2895–2899 (2020). IEEE
  * (20) Zhang, J., Li, C.-G., You, C., Qi, X., Zhang, H., Guo, J., Lin, Z.: Self-supervised convolutional subspace clustering network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5473–5482 (2019)
  * (21) Zhang, T., Ji, P., Harandi, M., Huang, W., Li, H.: Neural collaborative subspace clustering. In: International Conference on Machine Learning, pp. 7384–7393 (2019). PMLR
  * (22) Lv, J., Kang, Z., Lu, X., Xu, Z.: Pseudo-supervised deep subspace clustering. IEEE Transactions on Image Processing 30, 5252–5263 (2021)
  * (23) Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. science 313(5786), 504–507 (2006)
  * (24) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural information processing systems 27 (2014)
  * (25) Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  * (26) Van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv e-prints, 1807 (2018)
  * (27) Vidal, R., Favaro, P.: Low rank subspace clustering (lrsc). Pattern Recognition Letters 43, 47–61 (2014)
  * (28) Ji, P., Salzmann, M., Li, H.: Efficient dense subspace clustering. In: IEEE Winter Conference on Applications of Computer Vision, pp. 461–468 (2014). IEEE
  * (29) Ni, Y., Sun, J., Yuan, X., Yan, S., Cheong, L.-F.: Robust low-rank subspace segmentation with semidefinite guarantees. In: 2010 IEEE International Conference on Data Mining Workshops, pp. 1179–1188 (2010). IEEE
  * (30) Tang, X., Tang, X., Wang, W., Fang, L., Wei, X.: Deep multi-view sparse subspace clustering. In: Proceedings of the 2018 VII International Conference on Network, Communication and Computing, pp. 115–119 (2018)
  * (31) Abavisani, M., Patel, V.M.: Deep multimodal subspace clustering networks. IEEE Journal of Selected Topics in Signal Processing 12(6), 1601–1614 (2018)
  * (32) Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., Tang, J.: Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering (2021)
  * (33) Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding. In: European Conference on Computer Vision, pp. 776–794 (2020). Springer
  * (34) He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)
  * (35) Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020)
  * (36) Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640–9649 (2021)
  * (37) Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607 (2020). PMLR
  * (38) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
  * (39) Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International Conference on Machine Learning, pp. 1861–1870 (2018). PMLR
  * (40) Bai, L., Liang, J.: Sparse subspace clustering with entropy-norm. In: International Conference on Machine Learning, pp. 561–568 (2020). PMLR
  * (41) Peng, Z., Jia, Y., Liu, H., Hou, J., Zhang, Q.: Maximum entropy subspace clustering network. IEEE Transactions on Circuits and Systems for Video Technology (2021)
  * (42) Nene, S.A., Nayar, S.K., Murase, H., et al.: Columbia object image library (coil-20 and coil-100) (1996)
  * (43) Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
  * (44) LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)
  * (45) Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * (46) Nilsback, M.-E., Zisserman, A.: A visual vocabulary for flower classification. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 2, pp. 1447–1454 (2006). IEEE
  * (47) You, C., Li, C.-G., Robinson, D.P., Vidal, R.: Oracle based active set algorithm for scalable elastic net subspace clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3928–3937 (2016)
  * (48) You, C., Robinson, D., Vidal, R.: Scalable sparse subspace clustering by orthogonal matching pursuit. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3918–3927 (2016)
  * (49) Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008)


# Learning Spatial Similarity Distribution for Few-shot Object Counting

Yuanwu Xu  Feifan Song  Haofeng Zhang∗ School of Artificial Intelligence,
Nanjing University of Science and Technology {xuyuanwu, sff,
<EMAIL_ADDRESS>
###### Abstract

Few-shot object counting aims to count the number of objects in a query image
that belong to the same class as the given exemplar images. Existing methods
compute the similarity between the query image and exemplars in the 2D spatial
domain and perform regression to obtain the counting number. However, these
methods overlook the rich information about the spatial distribution of
similarity on the exemplar images, leading to significant impact on matching
accuracy. To address this issue, we propose a network learning Spatial
Similarity Distribution (SSD) for few-shot object counting, which preserves
the spatial structure of exemplar features and calculates a 4D similarity
pyramid point-to-point between the query features and exemplar features,
capturing the complete distribution information for each point in the 4D
similarity space. We propose a Similarity Learning Module (SLM) which applies
the efficient center-pivot 4D convolutions on the similarity pyramid to map
different similarity distributions to distinct predicted density values,
thereby obtaining accurate count. Furthermore, we also introduce a Feature
Cross Enhancement (FCE) module that enhances query and exemplar features
mutually to improve the accuracy of feature matching. Our approach outperforms
state-of-the-art methods on multiple datasets, including FSC-147 and CARPK.
Code is available at https://github.com/CBalance/SSD.

$*$$*$footnotetext: Corresponding author.

##  1 Introduction

Visual object counting aims at counting how many times a certain object occurs
in the query image, which has received growing attention in the past years.
Existing methods often focus on specific domains, such as crowd counting Shu
et al. (2022); Wang et al. (2020); Abousamra et al. (2021), animal counting
Arteta et al. (2016), or car counting Hsieh et al. (2017). These methods
typically rely on large amounts of data to train accurate counting models.
Furthermore, they are limited to counting objects of specific categories and
cannot generalize well to novel categories.

To overcome these limitations, a recent approach called Few-shot Object
Counting (FSC) has been introduced and gained great attention with the
emergence of a dataset Ranjan et al. (2021). FSC addresses the challenge of
counting objects from arbitrary categories using only a few exemplars. This
enables the model to generalize to unseen categories, offering potential for
application in various scene categories beyond those encountered during
training. By leveraging few exemplars, FSC provides a more flexible and
adaptable solution for object counting tasks.

Figure 1: Comparison between existing methods and our method. Compared to the
feature similarity computation process in previous methods, our approach
preserves the spatial structure of exemplars. Each position is computed with
query features, and in the subsequent convolutional regression process, we
fully utilize the spatial similarity distribution information between query
and exemplar features at a point-to-point level. Figure 2: Heatmap depicting
the similarity distribution of objects at different positions on the exemplar.

As shown in Fig.1, existing few-shot object counting methods typically follow
a general workflow. They first calculate the similarity between query and
exemplar features, and then directly regress the similarity matrix or enhance
the query features using the similarity matrix and exemplar features before
regression. In terms of similarity computation, some methods, as demonstrated
in Ranjan et al. (2021); Yang et al. (2021); You et al. (2023); Đukić et al.
(2023), employ exemplar features as fixed kernels to perform convolution with
query feature. However, in this approach, the distribution of kernel features
remains fixed during convolution matching, limiting its adaptability to
different sizes and shapes of object features in the query. Another approach,
such as the one used in Shi et al. (2022); Lin et al. (2022); Liu et al.
(2022), involves pooling exemplar features to obtain $1\times 1$ feature
prototypes, followed by cosine similarity computation between feature vector
of each position in the query and these prototypes. This method disregards the
distribution information of query and exemplar features, and counting
performance becomes dependent on the performance of previous feature
extraction and self-attention mechanisms.

To accurately locate the center of an object and generate an appropriate
density distribution, we leverage distinct similarity distribution
characteristics between each parts, such as object centers, edges and
background, when compared to exemplars. Explicitly, as shown in Fig.2, the
similarity distribution of the object center in exemplars gradually diminishes
from the central position towards the surrounding regions, while the
similarity distribution at the edges exhibits variations across different
locations. On the other hand, the background demonstrates generally lower
similarity values across all positions except the background area. Taking
advantage of these patterns, we propose a novel method that tries to preserve
the spatial structure of exemplars during similarity computation, and name it
as learning Spatial Similarity Distribution (SSD). Concretely, this method
yields a 4D similarity tensor, which allows for flexible extraction of point-
to-point similarity distribution information between query and exemplar
features using convolution operations in the 4D space. The features obtained
through convolution enable precise calculation of density values for each
position in the query during regression. In addition, we introduce a Feature
Cross Enhancement (FCE) module for query and exemplar features. This employs
the similarity matrices as weights to mutually enhance the features, aiming to
achieve higher matching accuracy for objects belonging to the given category.

We conduct comprehensive experiments on two renowned public benchmark
datasets, i.e., FSC-147 Ranjan et al. (2021) and CARPK Hsieh et al. (2017).
The results clearly illustrate that our approach surpasses the performance of
current state-of-the-art methods. Our contributions can be summarized as
follows:

  * $\bullet$

We design a model based on learning the 4D spatial similarity distribution
between query and exemplar features in Similarity Learning Module (SLM). This
model is capable of obtaining accurate counting results after comprehensive
integration of similarity distribution information among point pairs and their
surroundings.

  * $\bullet$

Before calculating the similarity between query and exemplar features, we
introduce a Feature Cross Enhancement (FCE) module, which enhances the
interaction between them, reducing the distance between the target objects and
exemplar features to achieve better matching performance.

  * $\bullet$

Extensive experiments on large-scale counting benchmarks, such as FSC-147 and
CARPK, are conducted and the results demonstrate that our method outperforms
the state-of-the-art approaches.

##  2 Related Work

###  2.1 Class-Specific Object Counting

Class-specific object counting focuses on counting a specific class of
objects, such as crowd Stewart et al. (2016); Liang et al. (2023); Lin and
Chan (2023); Du et al. (2023), animals Arteta et al. (2016), or cars Hsieh et
al. (2017). In related methods, the class information can be incorporated into
the feature extraction process without additional classification steps.
Existing methods can be broadly categorized into detection-based and
regression-based approaches.

Detection-based methods detect the positions of objects in an image to perform
counting. However, counting accuracy in these methods relies heavily on the
performance of the detection process, which introduces errors. This limits the
effectiveness of counting tasks in scenarios with densely packed objects. To
address this issue, regression-based methods have been proposed to generate a
density map, where the sum of the density values represents the predicted
object count.

Classic detection-based methods, for example, Stewart et al. (2016) propose a
model that decodes an image into a set of people detections, generating
distinct detection hypotheses directly from the input image. On the other
hand, recent research in regression-based methods, such as Cheng et al.
(2022), utilizes locally connected multivariate Gaussian kernels as
replacements for convolution filters. Moreover, a recent work Liang et al.
(2023) proposes knowledge transfer from a vision-language pre-trained model
(CLIP) to unsupervised crowd counting tasks, eliminating the need for density
map annotation.

Figure 3: The whole architecture of the proposed SSD framework.

###  2.2 Few-shot Object Counting

In recent years, few-shot object counting (FSC) has gained significant
attention and witnessed a surge of interest. FSC aims to accurately count
objects in an image by leveraging only a few exemplars as references. This
ability to adapt to unseen categories during the testing phase is a key
advantage of FSC.

Several noteworthy methods have been proposed for FSC. GMN Lu et al. (2019)
concatenates support features and query features together, and regresses a
predicted density map based on this concatenation. In contrast, FamNet Ranjan
et al. (2021) convolves the query image with exemplars used as convolutional
kernels, generating multiple similarity maps that provide insights into the
comparison results between the query and exemplars. A predicted density map is
then regressed from these similarity maps. Another approach, BMNet Shi et al.
(2022), employs global pooling to transform exemplars into prototypes, and
replaces fixed inner product operations with a learnable bilinear similarity
metric for comparing exemplar prototypes with query image features. CounTR Liu
et al. (2022) introduced a transformer-based architecture for extracting image
features and utilized cross-attention modules for effective feature matching.
Recently, LOCAĐukić et al. (2023) is proposed and considers the exemplar shape
and appearance properties separately and iteratively adapts these into object
prototypes by a new object prototype extraction (OPE) module considering the
image-wide features.

###  2.3 Generalized Loss

Generalized loss function Wan et al. (2021) is proposed for learning density
maps for crowd counting and localization, which is based on unbalanced optimal
transport. And Wan et al. (2021) prove that both L2 loss and Bayesian loss Ma
et al. (2019) are special cases of the generalized loss. The approach proposed
in Lin et al. (2022) also utilizes this loss function and introduces a scale-
sensitive generalized loss that applies different loss computation methods to
object categories of different scales.

##  3 Methodology

###  3.1 Problem Setting

In few-shot object counting, the dataset is split into base classes $C_{base}$
and novel classes $C_{novel}$, where $C_{base}$ and $C_{novel}$ do not
overlap. The remarkable generalization capability of Few-shot Object Counting
(FSC) lies in its ability to achieve high performance on the val set and test
set, even for categories $C_{novel}$ that have not been encountered during
training on $C_{base}$. FSC addresses the task of counting the number of
objects of interest present in a query image $\mathbf{X}\in\mathbb{R}^{3\times
H\times W}$, with the assistance of $K$ exemplars $\mathbf{Z}$.

###  3.2 Overall Architecture

Figure 4: Concatenation of multi-level similarity matrices.

As shown in Fig.3, our entire framework follows the following steps: (1)
Feature extraction, (2) Feature Cross Enhancement (FCE), (3) Similarity
pyramid calculation, (4) Similarity learning and (5) Regression decoder.
Initially, the ResNet-50 He et al. (2016) feature extractor is used to extract
features from the image $\mathbf{X}\in\mathbb{R}^{3\times H\times W}$, with
the option of weight freezing (no fine-tuning is performed), and generated
pyramid features $\left\\{\mathbf{F}_{l}^{q}\right\\}_{l=1}^{L}$, where each
level feature $\mathbf{F}_{l}^{q}\in\mathbb{R}^{C_{L_{p}}\times
H_{L_{p}}\times W_{L_{p}}}\left(l\in L_{p}\right)$. Among all levels, several
adjacent levels have features with the same spatial dimensions. All these
levels together form a large layer
$L_{p}\left(p=1,2,\cdot\cdot\cdot,P\right)$. For each level in the feature
pyramid, $K$ exemplar features $\mathbf{F}_{l}^{s}\in\mathbb{R}^{K\times
C_{L_{p}}\times h\times w}$ are extracted using the RoIAlign method He et al.
(2017). $C_{L_{p}}$ represents the feature channel dimension of the respective
layer, while $H_{L_{p}}\times W_{L_{p}}$ and $h\times w$ denote the spatial
dimensions of the query feature and exemplar features at that layer. Here we
maintain all exemplar features at a uniform size $h\times w$.

The $K$ exemplar features are input into the FCE module, along with the query
feature at the same level, resulting in enhanced features $\mathbf{E}_{l}^{q}$
and $\mathbf{E}_{l}^{s}$ with the same dimensions as $\mathbf{F}_{l}^{q}$ and
$\mathbf{F}_{l}^{s}$. We partition the $K$ exemplars, with each exemplar
corresponding to a set of feature pyramid combinations
$\left\\{\mathbf{E}_{l}^{q},\mathbf{E}_{l,k}^{s}\right\\}_{l=1}^{L}$, where
$\mathbf{E}_{l,k}^{s}\in\mathbb{R}^{C_{L_{p}}\times h\times w}$. For pyramid
feature set of each exemplar, we perform cosine multiplication on each feature
pair to generate the similarity matrix:

$\mathbf{S}_{l,k}\left(x^{q},x^{s}\right)={\rm
ReLU}\left(\frac{\mathbf{E}_{l}^{q}\left(x^{q}\right)\cdot\mathbf{E}_{l,k}^{s}\left(x^{s}\right)}{\left\|\mathbf{E}_{l}^{q}\left(x^{q}\right)\right\|\left\|\mathbf{E}_{l,k}^{s}\left(x^{s}\right)\right\|}\right).$
(1)

Here, $x^{q}$ and $x^{s}$ denote 2-dimensional spatial positions of query
feature map $\mathbf{E}_{l}^{q}$ and exemplar feature map
$\mathbf{E}_{l,k}^{s}$, respectively. ‘$\cdot$’ denotes vector dot product.
For each similarity tensor, $\mathbf{S}_{l,k}\in\mathbb{R}^{H_{L_{p}}\times
W_{L_{p}}\times h\times w}$.

As shown in Fig. 4, we concatenate the similarity matrices of the same large
layer $L_{p}$ and partition $\left\\{\mathbf{S}_{l,k}\right\\}_{l=1}^{L}$
accordingly, transforming it into
$\left\\{\left\\{\mathbf{S}_{l,k}\right\\}_{l\in L_{p}}\right\\}_{p=1}^{P}$
and set it as $\left\\{\mathbf{S}_{L_{p},k}\right\\}_{p=1}^{P}$. And each
tensor in it is $\mathbf{S}_{L_{p},k}\in\mathbb{R}^{\left|L_{p}\right|\times
H_{L_{p}}\times W_{L_{p}}\times h\times w}$, where $\left|L_{p}\right|$ is the
number of pyramid levels. They are then fed into the Similarity Learning
Module (SLM) to produce a learned and fused feature map $\mathbf{M}_{k}$.
Finally, $\mathbf{M}_{k}$ is input into the regression decoder module to
obtain the density map $\mathbf{P}_{k}\in\mathbb{R}^{1\times H\times W}$. The
$K$ sets of feature pyramids correspond to the generation of $K$ density maps.
The final predicted density map, denoted as $\mathbf{P}$, is obtained by
taking the mean of these $K$ maps:

$\mathbf{P}=\frac{\sum_{k=1}^{K}\mathbf{P}_{k}}{K}.$ (2)

###  3.3 Feature Cross Enhancement

The distribution of object features within the query features of the same
category is often uneven. Directly matching and counting using the original
features can result in varying density values for each object. To address this
issue, we propose a Feature Cross Enhancement (FCE) module that aims to bring
the object features within the query closer to the exemplar features while
also facilitate the exemplar features to be closer to the center position of
all object features. By enhancing the proximity of the object features
specific to a certain category, the model is able to generate more accurate
density values.

In the FCE module, the input features $\mathbf{F}_{l}^{q}$ and
$\mathbf{F}_{l}^{s}$ are jointly transformed into
$\mathbf{V}^{q}\in\mathbb{R}^{C_{p}^{e}\times H_{L_{p}}\times W_{L_{p}}}$ and
$\mathbf{V}^{s}\in\mathbb{R}^{C_{p}^{e}\times K\times h\times w}$ through a
convolutional layer. They are then individually passed through other two
convolutional layers, with $\mathbf{F}_{l}^{q}$ being transformed into
$\mathbf{Q}$ and $\mathbf{F}_{l}^{s}$ into $\mathbf{K}$, which are the same
dimensions as $\mathbf{V}^{q}$ and $\mathbf{V}^{s}$. Multiplying the transpose
of $\mathbf{Q}$ and $\mathbf{K}$ matrices results in the attention matrix
$\mathbf{A}_{l}$:

$\mathbf{A}_{l}={\rm SoftMax}\left(\mathbf{Q}^{T}\mathbf{K}\right).$ (3)

Then we utilize $\mathbf{A}_{l}$ to separately enhance $\mathbf{F}_{l}^{q}$
and $\mathbf{F}_{l}^{s}$:

$\displaystyle\mathbf{E}_{l}^{q}$ $\displaystyle=\mathbf{F}_{l}^{q}+{\rm
MLP}\left(\mathbf{F}_{l}^{q}\right)\odot{\rm
Trans}\left(\mathbf{V}^{s}\mathbf{A}_{l}^{T}\right)$ (4)
$\displaystyle\mathbf{E}_{l}^{s}$ $\displaystyle=\mathbf{F}_{l}^{s}+{\rm
MLP}\left(\mathbf{F}_{l}^{s}\right)\odot{\rm
Trans}\left(\mathbf{V}^{q}\mathbf{A}_{l}\right).$

Here, ${\rm MLP\left(\cdot\right)}$ is a multi-layer perceptron consisting of
fully connected layers and activation functions, and used to map the channel
vector into a channel-wise feature space of similarity relation. ${\rm
Trans\left(\cdot\right)}$ represents the convolutional layer that transforms
channel $C_{p}^{e}$ into the original channel $C_{L_{p}}$, and $\odot$ denotes
element-wise multiplication.

###  3.4 Similarity Learning Module

4D convolution. Several existing works Rocco et al. (2018); Yang and Ramanan
(2019); Min et al. (2021) have proposed various implementations of 4D
convolutions. In our framework, we employ the center-pivot 4D convolution from
Min et al. (2021) which sparsifies a significant portion of unimportant
weights and computations. This method focuses solely on the information
associated with the convolution center, reducing computational overhead while
maintaining effectiveness. With 4D convolutions, tensors are fused for each 4D
position based on convolution kernel weights, integrating information from the
vicinity in 4D space and transforming the vector at that position into the
corresponding output dimension.

For the input set of similarity tensors
$\left\\{\mathbf{S}_{L_{p}}\right\\}_{p=1}^{P}$ (here we omit the exemplar
subscript $k$), each tensor is fed into its corresponding 4D convolutional
module:

${\mathbf{S}}^{\prime}_{L_{p}}=f_{L_{p}}^{e}\left(\mathbf{S}_{L_{p}}\right)\in\mathbb{R}^{C_{out}\times
H_{L_{p}}\times W_{L_{p}}\times{h}^{\prime}\times{w}^{\prime}},$ (5)

where $f_{L_{p}}^{e}\left(\cdot\right)$ is an encoding module composed of
multiple 4D convolutional layers, group normalization Wu and He (2018), and
ReLU activation function. The large strides of the 4D convolution compresses
the spatial dimensions $h\times w$ of the exemplar spatial structure to
${h}^{\prime}\times{w}^{\prime}$, while embedding the dimensions of all
similarity tensors from $\left|L_{p}\right|$ into $C_{out}$.

Next, starting from the apex of pyramid
$\left\\{{\mathbf{S}}^{\prime}_{L_{p}}\right\\}_{p=1}^{P}$, we proceed to fuse
each subsequent layer downwards. For instance, the tensor
${\mathbf{S}}^{\prime}_{L_{P}}\in\mathbb{R}^{C_{out}\times H_{L_{P}}\times
W_{L_{P}}\times{h}^{\prime}\times{w}^{\prime}}$ is upsampled on its dimensions
$H_{L_{P}}\times W_{L_{P}}$ to match the corresponding dimensions
$H_{L_{P-1}}\times W_{L_{P-1}}$ of the layer below. It is then added to the
respective tensor ${\mathbf{S}}^{\prime}_{L_{P-1}}$ of the layer below and
passed through a fusion module based on 4D convolution:

$\mathbf{S}_{L_{p-1}}^{Mix}=f_{L_{p-1}}^{Mix}\left({\rm
upsample}\left({\mathbf{S}}^{\prime}_{L_{p}}\right)+{\mathbf{S}}^{\prime}_{L_{p-1}}\right).$
(6)

The structure of function $f_{L_{p-1}}^{Mix}$ is identical to that of function
$f_{L_{p}}^{e}$, with the difference being that the stride of
$f_{L_{p-1}}^{Mix}$ is set to 1, which does not alter the spatial dimensions
of the tensor. And the input and output dimensions in $f_{L_{p-1}}^{Mix}$ are
all set to $C_{out}$.

$\mathbf{S}_{L_{P-1}}^{Mix}$ is fused with the tensor
${\mathbf{S}}^{\prime}_{L_{P-2}}$ in a similar manner, iteratively continuing
the fusion process with each subsequent layer until reaching the bottom layer
of the pyramid ${\mathbf{S}}^{\prime}_{L_{1}}$. Consequently, we obtain the
final fused tensor $\mathbf{S}_{L_{1}}^{Mix}\in\mathbb{R}^{C_{out}\times
H_{L_{1}}\times W_{L_{1}}\times{h}^{\prime}\times{w}^{\prime}}$. By
calculating the mean along the last two dimensions, we derive the fused
feature $\mathbf{M}\in\mathbb{R}^{C_{out}\times H_{L_{1}}\times W_{L_{1}}}$.

###  3.5 Regression Decoder

The decoder module used for regression consists of multiple component modules
composed of convolutional layers, ReLU activation layers, and upsampling
layers. With each component module, the size of feature $\mathbf{M}$ is
increased to twice until reaching the size of the input image $H\times W$.
Subsequently, it passes through a $1\times 1$ convolutional layer and a ReLU
activation layer. The output is the predicted density map.

###  3.6 Generalized Loss

In previous object counting tasks, ground truth density maps are generated by
convolving dot labels with fixed Gaussian kernels. The MSE loss function is
then employed for supervised training of the predicted density map. In a
recent study Wan et al. (2021), a generalized loss function was proposed that
directly measures the distance between the predicted density map and the dot
labels. This loss function is based on entropic-regularized unbalanced optimal
transport cost.

We represent the predicted results as
$\mathbf{A}=\left\\{\left(a_{i},\mathbf{x}_{i}\right)\right\\}_{i=1}^{n}$,
where $a_{i}$ denotes the predicted density value at pixel
$\mathbf{x}_{i}\in\mathbb{R}^{2}$. Here, n represents the total number of
pixels. Then we denote the predicted density map as
$\mathbf{a}=\left[a_{i}\right]_{i}$. On the other hand, the ground truth dot
label is denoted as
$\mathbf{B}=\left\\{\left(b_{j},\mathbf{y}_{j}\right)\right\\}_{j=1}^{m}$,
where $\mathbf{y}_{j}$ indicates the location of the j-th annotation and
$b_{j}$ represents the number of objects represented by that annotation. In
general, it is assumed that $\mathbf{b}=\left[b_{j}\right]_{j}=1_{m}$. The
whole loss function can be defined as:

$\displaystyle L_{\mathbf{C}}\left(\mathbf{A},\mathbf{B}\right)=$
$\displaystyle\min_{\mathbf{D}}\left\langle\mathbf{C},\mathbf{D}\right\rangle-\varepsilon
H\left(\mathbf{D}\right)+\tau\left\|\mathbf{D}1_{m}-\mathbf{a}\right\|_{2}^{2}$
(7) $\displaystyle+\tau\left\|\mathbf{D}^{T}1_{n}-\mathbf{b}\right\|_{1},$

where $\mathbf{C}$ is the transport cost matrix, with $C_{ij}$ representing
the cost of moving the predicted density at $\mathbf{x}_{i}$ to the ground
truth dot annotation $\mathbf{y}_{j}$. $\mathbf{D}$ is the transport matrix
that assigns fractional weights to associate each location $\mathbf{x}_{i}$ in
$\mathbf{A}$ with its corresponding $\mathbf{y}_{j}$ in $\mathbf{B}$ for cost
calculation. The optimal transport cost is obtained by minimizing the loss
over $\mathbf{D}$. $H\left(\mathbf{D}\right)=-\sum_{ij}D_{ij}{\rm log}D_{ij}$
is the entropic regularization. The intermediate density map representation
$\hat{\mathbf{a}}=\mathbf{D}1_{m}$ is constructed from the ground truth
annotations, while $\hat{\mathbf{b}}=\mathbf{D}^{T}1_{n}$ is the
reconstruction of the ground truth dot annotations.

###  3.7 Dynamic Image Scale

Certain sample images may contain objects that are small sizes or densely
distributed, leading to challenges in effectively distinguishing between
individual objects. This results in overlapping density within the predicted
density map, thereby impacting model performance. To address this issue, we
dynamically resize the input images based on the size of exemplar boxes before
entering the backbone. This resizing is performed proportionally to the
dimensions of the exemplar boxes, allowing the model to better recognize
samples containing smaller objects. For an input image $\mathbf{X}$, we
compute the average length and width of K exemplar boxes
$\mathbf{B}\in\mathbb{R}^{K\times 2}$:

$\bar{\mathbf{B}}=\frac{\sum_{k=1}^{K}\mathbf{B}_{k}}{K}.$ (8)

If ${\rm min}\left(\bar{\mathbf{B}}\right)$ is below a threshold $\gamma$ , we
calculate the scale of image expansion:

$scale=\frac{\gamma-{\rm min}\left(\bar{\mathbf{B}}\right)}{\eta}+1,$ (9)

where both $\gamma$ and $\eta$ are hyperparameters to be tuned. Finally, the
image size and exemplar boxes $\mathbf{B}$ are simultaneously expanded by the
determined scale value before being input into the model.

Methods | Backbone | 3shot | 1shot
---|---|---|---
Val | Test | Val | Test
MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE
GMN Lu et al. (2019) | ResNet-50 | 29.66 | 89.81 | 26.52 | 124.57 | — | — | — | —
MAML Finn et al. (2017) | ConvNet | 25.54 | 79.44 | 24.90 | 112.68 | — | — | — | —
FamNet Ranjan et al. (2021) | ResNet-50 | 23.75 | 69.07 | 22.08 | 99.54 | 26.55 | 77.01 | 26.76 | 110.95
CFOCNet Yang et al. (2021) | ResNet-50 | 21.19 | 61.41 | 22.10 | 112.71 | 27.82 | 71.99 | 28.60 | 123.96
LaoNet Lin et al. (2021) | VGG-19 | — | — | — | — | 17.11 | 56.81 | 15.78 | 97.15
BMNet+ Shi et al. (2022) | ResNet-50 | 15.74 | 58.53 | 14.62 | 91.83 | 17.89 | 61.12 | 16.89 | 96.65
SAFECount You et al. (2023) | ResNet-18 | 15.28 | 47.20 | 14.32 | 85.54 | — | — | — | —
SPDCN Lin et al. (2022) | VGG-19 | 14.59 | 49.97 | 13.51 | 96.80 | — | — | — | —
CounTR Liu et al. (2022) | ViT/ConvNet | 13.13 | 49.83 | 11.95 | 91.23 | 13.15 | 49.72 | 12.06 | 90.01
LOCA Đukić et al. (2023) | ResNet-50 | 10.24 | 32.56 | 10.79 | 56.97 | 11.36 | 38.04 | 12.53 | 75.32
SSD(ours) | ResNet-50 | 9.73 | 29.72 | 9.58 | 64.13 | 11.03 | 34.83 | 11.61 | 71.55
Table 1: Comparison with state-of-the-art approaches on the FSC-147 dataset.
‘—’ means the result is not reported.

##  4 Experiments

###  4.1 Datasets and Metrics

Datasets. FSC-147 is a comprehensive multi-class few-shot object counting
dataset. It comprises a total of 6,135 images covering 89 distinct object
categories. The images in the dataset exhibit significant variations in terms
of object counts, ranging from as low as 7 to as high as 3,731 objects, with
an average count of 56 per image. Notably, each image in the dataset is
accompanied by three or four exemplar images that are annotated with bounding
boxes. To facilitate experimentation, the dataset is further divided into
training, validation, and testing subsets, with each subset containing 29 non-
overlapping object categories.

CARPK is a class-specific car counting dataset, which consists of 1448 images
of parking lots from a bird’s view. These images are captured from four
different parking lots, encompassing various scenes. The training set
comprises three scenes, while a separate scene is designated for test.

Metrics. We employ Mean Average Error (MAE) and Root Mean Squared Error (RMSE)
as performance metrics for evaluating the SSD method, as these metrics are
widely utilized in counting tasks.

$\displaystyle MAE$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\left|C_{pred}^{i}-C^{i}\right|,$ (10)
$\displaystyle RMSE$
$\displaystyle=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(C_{pred}^{i}-C^{i})^{2}},$

where $N$ is the number of all the query images, $C^{i}$ and $C_{pred}^{i}$
are the ground truth and the predicted number of objects for $i$-th image
respectively.

Figure 5: Qualitative results on the FSC-147 dataset.

###  4.2 Implementation Details

Architecture Details. Our approach involves resizing the input image initially
to $384\times 576$. Then the image is dynamically resized to the suitable
scale, followed by the application of the pre-trained ResNet-50 backbone,
utilizing features extracted from the final three layers. The number of
features at each layer, denoted as $\left|L_{p}\right|$, is 4, 6, and 3, with
corresponding feature channel dimensions of 512, 1024, and 2048, respectively.
The size of exemplar features extracted from each layer is uniformly resized
to $16\times 16$. In the FCE module, the embedded channel dimension is set to
half of dimension of the input feature. In the Similarity Learning module, the
4D convolution module consists of three component modules with output
dimensions of 32, 128, and 256, respectively. To fuse the three layers of
similarity tensors, two fusion modules are required, each containing three
component modules, and all the output dimensions are 256. $\gamma$ and $\eta$
in DIS method are set to 32 and 12.

Training Details. We apply AdamW Loshchilov and Hutter (2017) as the optimizer
with a learning rate of $1\times 10^{-4}$ and the learning rate decays with a
rate of 0.95 after each epoch. The batch size is 4 and the model is trained
for 100 epochs.

###  4.3 Comparison with State of the Art

We evaluate the proposed SSD on the FSC-147 dataset with several state-of-the-
art methods. the results are summarized in Tab.1. We conduct both 3-shot and
1-shot experiments on the dataset. SSD consistently outperforms existing
methods in terms of performance.

In the 3-shot scenario, even compared to the recent state-of-the-art method
LOCA Đukić et al. (2023), SSD demonstrates superior performance on the val set
with a 5.0% improvement in MAE and an 8.7% improvement in RMSE. Notably, SSD
also exhibits better performance on the test set with a 11.2% improvement in
MAE.

In the 1-shot scenario, SSD surpasses all previous state-of-the-art methods.
This underscores the minimal dependence of SSD on reference samples,
showcasing its robust adaptability to scenarios with limited available data.

Qualitative Results. In Fig.5, we visualize and compare the predicted density
maps of BMNet, LOCA, and SSD. The results demonstrate that SSD has higher
accuracy compared to the other two methods.

Methods | BackBone | Test
---|---|---
MAE | RMSE
FamNet Ranjan et al. (2021) | ResNet-50 | 28.84 | 44.47
BMNet Shi et al. (2022) | ResNet-50 | 10.44 | 13.77
LOCA Đukić et al. (2023) | ResNet-50 | 9.97 | 12.51
SSD(ours) | ResNet-50 | 9.58 | 12.15
Table 2: Comparison with the state-of-the-art approaches on the CARPK dataset.

###  4.4 Cross-dataset Generalization

Following Ranjan et al. (2021), we validate the generalizability of SSD on the
CARPK dataset. The model is trained on the FSC-147 dataset and then tested on
the CARPK dataset, with the car category samples excluded during training.
During testing, we randomly select twelve annotations from the CARPK dataset
as exemplars to count cars in images. The experimental results is presented in
Tab.2. SSD outperforms three other methods, achieving an improvement of 3.9%
in MAE and 2.9% in RMSE compared to the most recent state-of-the-art method
LOCA.

FCE | G-Loss | DIS | Val | Test
---|---|---|---|---
MAE | RMSE | MAE | RMSE
✗ | ✗ | ✗ | 18.96 | 64.44 | 16.75 | 108.12
✓ | ✗ | ✗ | 18.56 | 61.69 | 16.39 | 108.64
✗ | ✓ | ✗ | 15.14 | 54.75 | 14.99 | 107.40
✗ | ✗ | ✓ | 13.68 | 45.53 | 14.25 | 91.17
✓ | ✓ | ✗ | 13.92 | 51.08 | 14.43 | 106.84
✗ | ✓ | ✓ | 10.50 | 31.86 | 11.38 | 74.45
✓ | ✗ | ✓ | 13.37 | 39.26 | 12.90 | 82.16
✓ | ✓ | ✓ | 9.73 | 29.72 | 9.58 | 64.13

Table 3: Ablation studies on the FSC-147 dataset. ‘G-Loss’ means Generalized
Loss. ‘DIS’ denotes Dynamic Image Scale.

###  4.5 Ablation Study

We design a series of experiments to validate the individual contributions of
the FCE module, generalized loss, and dynamic image scale on the performance
improvement. In the absence of the FCE module, the model directly computes the
similatity between $\mathbf{F}_{l}^{q}$ and $\mathbf{F}_{l}^{s}$. When
excluding the generalized loss, we replace it with the more commonly used MSE
loss. Each component undergoes four sets of comparative experiments with and
without that component.

FCE module. Analysis of the four sets of experiments involving the FCE module
reveals a consistent improvement in model performance. The addition of the FCE
module results in a performance boost ranging from 2% to 19% in MAE and up to
14% in RMSE. This indicates that FCE module significantly enhances the ability
of model to recognize objects within a given category by minimizing the
distance between individual object features and exemplar features, leading to
improved accuracy and uniformity in similarity and final density predictions
across object positions.

Generalization loss. The contribution of generalization loss is pronounced to
performance improvement. The four sets of comparative experiments show
performance gains ranging from 10% to 27% in MAE and 1% to 30% in RMSE. The
substantial improvement attributed to the generalization loss is due to the
more precise recognition capabilities compared to MSELoss. By measuring point-
to-point distance loss between predicted values and ground true labels, the
generalized loss effectively guides the model to accurately locate object
center positions.

Dynamic image scale. The utilization of dynamic image scale also significantly
improves model performance, particularly for dense samples. Expanding image
scales proves effective in distinguishing between individual objects and
counting them separately. Application of this method results in performance
improvements ranging from 15% to 35% in MAE and 14% to 43% in RMSE.

Channel ratio in the FCE module. The query features and example features are
embedded into another channel before they enhance each other. We conduct a
series of experiments to determine the optimal ratio of the embedded channel
length to the original channel length, setting various ratios at
$\frac{1}{8}$, $\frac{1}{4}$, $\frac{1}{2}$, and 1. We then train the model to
achieve the best performance and present the experimental results in Fig.6
(a). It is observed that as the ratio increases, the model performance peaks
at a ratio of $\frac{1}{2}$ and then begins to deteriorate.

Number of component modules in fusion module. The fusion module used to
integrate tensors from different levels of the similarity pyramid is composed
of several component modules. We set the number of component modules to range
from 1 to 5 and conduct experiments on the FSC-147 dataset, with the results
displayed in Fig.6 (b). The line graph in the figure indicates that as the
number of component modules increases from 1 to 3, the performance of the
model gradually improves, peaking at 3, and then begins to decline. This
decline could be attributed to an increase in the number of model parameters
due to more component modules, leading to overfitting and negatively affecting
model performance.

(a) Channel ratio in FCE module (b) Number of component modules in fusion
module

Figure 6: Ablation study of channel ratio in FCE module and number of
component modules in fusion module. The vertical coordinates are the values of
MAE on the val set.

##  5 Conclusion

We propose a novel few-shot object counting method, SSD, which leverages a
point-to-point 4D space to learn the spatial similarity distribution between
query and exemplar features. In contrast to existing methods, we exploit the
distribution information of similarity, enabling accurate identification of
the position and precise prediction of the count for objects of arbitrary
classes. Additionally, we introduce a Feature Cross Enhancement (FCE) module
that enhances the interaction between query and exemplar features, reducing
the feature distance within the same class for improved matching. Experimental
results on datasets such as FSC-147 and CARPK demonstrate that SSD outperforms
state-of-the-art methods.

## Acknowledgments

This work was supported in part by National Natural Science Foundation of
China under the Grants 62371235 and 62072246, and in part by Key Research and
Development Plan of Jiangsu Province (Industry Foresight and Key Core
Technology Project) under the Grant BE2023008-2.

## References

  * Abousamra et al. [2021] Shahira Abousamra, Minh Hoai, Dimitris Samaras, and Chao Chen.  Localization in the crowd with topological constraints.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 872–881, 2021.
  * Arteta et al. [2016] Carlos Arteta, Victor Lempitsky, and Andrew Zisserman.  Counting in the wild.  In Proceedings of the European Conference on Computer Vision, pages 483–498, 2016.
  * Cheng et al. [2022] Zhi-Qi Cheng, Qi Dai, Hong Li, Jingkuan Song, Xiao Wu, and Alexander G Hauptmann.  Rethinking spatial invariance of convolutional networks for object counting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19638–19648, 2022.
  * Du et al. [2023] Zhipeng Du, Jiankang Deng, and Miaojing Shi.  Domain-general crowd counting in unseen scenarios.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 561–570, 2023.
  * Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In Proceedings of the International Conference on Machine Learning, pages 1126–1135, 2017.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980–2988, 2017.
  * Hsieh et al. [2017] Meng-Ru Hsieh, Yen-Liang Lin, and Winston H Hsu.  Drone-based object counting by spatially regularized regional proposal network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4145–4153, 2017.
  * Liang et al. [2023] Dingkang Liang, Jiahao Xie, Zhikang Zou, Xiaoqing Ye, Wei Xu, and Xiang Bai.  Crowdclip: Unsupervised crowd counting via vision-language model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2893–2903, 2023.
  * Lin and Chan [2023] Wei Lin and Antoni B Chan.  Optimal transport minimization: Crowd localization on density maps for semi-supervised counting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21663–21673, 2023.
  * Lin et al. [2021] Hui Lin, Xiaopeng Hong, and Yabin Wang.  Object counting: You only need to look at one.  arXiv preprint arXiv:2112.05993, 2021.
  * Lin et al. [2022] Wei Lin, Kunlin Yang, Xinzhu Ma, Junyu Gao, Lingbo Liu, Shinan Liu, Jun Hou, Shuai Yi, and Antoni Chan.  Scale-prior deformable convolution for exemplar-guided class-agnostic counting.  In Proceedings of the British Machine Vision Conference, 2022.
  * Liu et al. [2022] Chang Liu, Yujie Zhong, Andrew Zisserman, and Weidi Xie.  Countr: Transformer-based generalised visual counting.  In Proceedings of the British Machine Vision Conference, 2022.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * Lu et al. [2019] Erika Lu, Weidi Xie, and Andrew Zisserman.  Class-agnostic counting.  In Proceedings of the Asian Conference on Computer Vision, pages 669–684, 2019.
  * Ma et al. [2019] Zhiheng Ma, Xing Wei, Xiaopeng Hong, and Yihong Gong.  Bayesian loss for crowd count estimation with point supervision.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6142–6151, 2019.
  * Min et al. [2021] Juhong Min, Dahyun Kang, and Minsu Cho.  Hypercorrelation squeeze for few-shot segmentation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6941–6952, 2021.
  * Ranjan et al. [2021] Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai.  Learning to count everything.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3394–3403, 2021.
  * Rocco et al. [2018] Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Neighbourhood consensus networks.  Advances in neural information processing systems, 31, 2018.
  * Shi et al. [2022] Min Shi, Hao Lu, Chen Feng, Chengxin Liu, and Zhiguo Cao.  Represent, compare, and learn: A similarity-aware framework for class-agnostic counting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9529–9538, 2022.
  * Shu et al. [2022] Weibo Shu, Jia Wan, Kay Chen Tan, Sam Kwong, and Antoni B Chan.  Crowd counting in the frequency domain.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19618–19627, 2022.
  * Stewart et al. [2016] Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng.  End-to-end people detection in crowded scenes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2325–2333, 2016.
  * Đukić et al. [2023] Nikola Đukić, Alan Lukežič, Vitjan Zavrtanik, and Matej Kristan.  A low-shot object counting network with iterative prototype adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18872–18881, 2023.
  * Wan et al. [2021] Jia Wan, Ziquan Liu, and Antoni B Chan.  A generalized loss function for crowd counting and localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1974–1983, 2021.
  * Wang et al. [2020] Qi Wang, Junyu Gao, Wei Lin, and Xuelong Li.  Nwpu-crowd: A large-scale benchmark for crowd counting and localization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6):2141–2149, 2020.
  * Wu and He [2018] Yuxin Wu and Kaiming He.  Group normalization.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * Yang and Ramanan [2019] Gengshan Yang and Deva Ramanan.  Volumetric correspondence networks for optical flow.  Advances in neural information processing systems, 32, 2019.
  * Yang et al. [2021] Shuo-Diao Yang, Hung-Ting Su, Winston H Hsu, and Wen-Chin Chen.  Class-agnostic few-shot object counting.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 870–878, 2021.
  * You et al. [2023] Zhiyuan You, Kai Yang, Wenhan Luo, Xin Lu, Lei Cui, and Xinyi Le.  Few-shot object counting with similarity-aware feature enhancement.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6315–6324, 2023.