# Dynamic Correlation Learning and Regularization for Multi-Label Confidence
Calibration

Tianshui Chen, Weihang Wang, Tao Pu, Jinghui Qin, Zhijing Yang, Jie Liu, Liang
Lin _Fellow, IEEE_ Tianshui Chen, Weihang Wang, Jinghui Qin, and Zhijing Yang
are with the Guangdong University of Technology (Email:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>yzhj@gdut.edu.cn). Tao Pu and Liang Lin are with the Sun Yat-Sen University
(Email<EMAIL_ADDRESS>linliang@ieee.org). Jie Liu is with North China
University of Technology (Email: liujxxxy@126.com). (_Corresponding author:
Jinghui Qin_). This work was supported in part by National Natural Science
Foundation of China (NSFC) under Grant No. 62206060 and No. 62206314, in part
by Guangzhou Basic and Applied Basic Research Foundation under Grant No.
SL2022A04J01626 and No. 2022A1515011835, Science and Technology Projects in
Guangzhou under Grant No. 2024A04J4388.

###### Abstract

Modern visual recognition models often display overconfidence due to their
reliance on complex deep neural networks and one-hot target supervision,
resulting in unreliable confidence scores that necessitate calibration. While
current confidence calibration techniques primarily address single-label
scenarios, there is a lack of focus on more practical and generalizable multi-
label contexts. This paper introduces the Multi-Label Confidence Calibration
(MLCC) task, aiming to provide well-calibrated confidence scores in multi-
label scenarios. Unlike single-label images, multi-label images contain
multiple objects, leading to semantic confusion and further unreliability in
confidence scores. Existing single-label calibration methods, based on label
smoothing, fail to account for category correlations, which are crucial for
addressing semantic confusion, thereby yielding sub-optimal performance. To
overcome these limitations, we propose the Dynamic Correlation Learning and
Regularization (DCLR) algorithm, which leverages multi-grained semantic
correlations to better model semantic confusion for adaptive regularization.
DCLR learns dynamic instance-level and prototype-level similarities specific
to each category, using these to measure semantic correlations across
different categories. With this understanding, we construct adaptive label
vectors that assign higher values to categories with strong correlations,
thereby facilitating more effective regularization. We establish an evaluation
benchmark, re-implementing several advanced confidence calibration algorithms
and applying them to leading multi-label recognition (MLR) models for fair
comparison. Through extensive experiments, we demonstrate the superior
performance of DCLR over existing methods in providing reliable confidence
scores in multi-label scenarios.

###### Index Terms:

Multi-Label Image Recognition, Confidence Calibration, Over-Confidence,
Trusted Artificial Intelligence

##  I Introduction

Modern visual recognition models, built on complex deep neural networks [1,
2], often suffer from overfitting to training data, inevitably leading to
overly confident and unreliable predicted score dilemma [3, 4, 5]. This
dilemma severely prevents their applications to high-risk scenarios, such as
self-driving [6, 7] and medical diagnosis [8, 9]. To deal with this issue,
numerous works [10, 3, 11] are intensively proposed for confidence calibration
that can provide more accurate and reliable predicted confidence scores to
indicate an accurate probability of correctness. Despite achieving impressive
progress, these efforts predominantly concentrate on single-label settings,
where each image is associated with a single category. However, these works
can hardly be applied to multi-label scenarios, which are more reflective of
real-world scenarios where images often contain objects from multiple
categories [12, 13, 14]. Our work targets the multi-label confidence
calibration (MLCC) task, seeking to extend and enhance calibration techniques
for these more complex and practical scenarios.

Figure 1: Two examples of predicted scores by current advanced MLR models with
and without DCLR calibration. The categories existing in the image are
highlighted in bold.

Current multi-label recognition (MLR) models [15, 16] mainly use the one-hot
target labels for each class as supervision, overlooking the information about
other categories. Consequently, these models are prone to indiscriminately
assigning overconfident scores to their predictions, culminating in the
critical issue of overconfidence. Moreover, these models either learn holistic
features [16, 17] or category-specific features [15] for prediction. However,
given the presence of multiple semantic objects scattered throughout an image,
these features often capture information from various semantic objects,
leading to significant semantic confusion and further exacerbating the
overconfidence issue. In the first example shown in Figure 1, existing models
exhibit confusion between “vase” and “potted plant” due to their similar
appearances, leading to an overly confident score being assigned to the non-
existent “potted plant”. Similarly, objects resembling “cell phone” and “book”
are also non-existent, and the models mistakenly allocate high-confidence
scores to these two categories. Traditional confidence calibration algorithms
[10, 3] typically employ the label smoothing mechanism [10], which smooths
each category by equally and independently reallocating a small value to all
non-target categories from the target category. However, these algorithms do
not consider category correlations that can capture semantic confusion to
obtain adaptive regularization, leading to sub-optimal performance in the MLCC
scenario.

In this work, we introduce the Dynamic Correlation Learning and Regularization
(DCLR) algorithm, specifically designed to learn and integrate category
correlations for modeling semantic confusion in multi-label images, thereby
facilitating more effective adaptive regularization. We employ a category-
specific contrastive learning module to learn distinct feature representations
for each category of a given image and their feature similarities. This
approach enables us to model category correlations using instance-level
feature similarities, allowing for the allocation of higher values to closely
correlated categories and lower values to those less correlated, thus forming
a soft label vector for adaptive regularization. Further enhancing our model,
we learn prototype representations for each category and calculate the
similarities between these prototypes and the features to establish more
robust category correlations. These correlations are then used to construct an
additional soft label vector in an identical fashion. The integration of both
soft label vectors, encompassing diverse and robust category correlations,
allows them to be seamlessly incorporated into any existing multi-label
recognition (MLR) methods [16, 15, 18] for effective calibration in a play-
and-plug manner. As shown in Figure 1, adding the DCLR algorithms to current
models can lead to notably well-calibrated scores.

Currently, there exists no benchmark for MLCC evaluation, which severely
prevents the development of this task. In this work, a unified evaluation
benchmark is further built for fair MLCC comparisons and facilitates research
in this field. Specifically, we first re-implement the traditional label
smoothing [10] and several leading confidence calibration algorithms [11, 19,
20, 21], and adapt them to the MLCC task. For fair and comprehensive
comparisons, we apply these algorithms and the proposed DCLR algorithms to
three representative MLR models, i.e., SSGRL [15] and ML-GCN [16] that learns
category-specific and holistic features for classification, and C-Tran [18]
that uses more advanced transformer networks. We use the traditional metrics
like adaptive calibration error (ACE) [22], expected calibrated error (ECE)
[23], maximum calibration error (MCE) [20] and reliability diagram [24] for
performance evaluation.

The contributions can be summarized into four folds. First, we extend
confidence calibration from single-label recognition to the more practical and
essential multi-label scenarios and present an in-depth analysis of the MLCC
task. Second, we introduce a novel dynamic correlation learning and
regularization algorithm that learns category correlations to model semantic
confusion and integrates these correlations to regularize MLR model training
to address the over-confidence dilemma in a play-and-plug manner. Third, we
construct a fair and comprehensive MLCC evaluation benchmark, which can well
evaluate the actual effects of each well-performing algorithm and facilitate
the research in this field. Finally, we follow the unified benchmark to
conduct extensive experiments to verify the effectiveness of the proposed
algorithm. The unified benchmark, including the re-implemented codes and
trained models of all re-implemented and our proposed DCLR algorithms, is
available at https://github.com/wkml/MLCC-DCLR.

##  II Related Works

We review the related literature in terms of two main streams, i.e., multi-
label image recognition and confidence calibration.

###  II-A Multi-Label Image Recognition

In recent years, multi-label image recognition (MLR) has garnered substantial
academic interest, as evidenced by seminal works [13, 15, 25, 26, 27, 18, 28,
29][30]. This heightened attention to MLR underscores its pragmatic relevance
and superiority over its single-label analog. Early MLR works propose to
identify the local regions that contain more discriminative objects and their
parts to learn more powerful feature representation. These works either pivot
towards object proposals [12, 31] or resort to visual attention mechanisms
[32, 33, 34]. As a pioneer work, Wei et al., [12] uses off-the-shelf
algorithms to extract thousands of object proposes and aggregate their
predicted scores to obtain the final MLR results. Despite achieving impressive
progress, these algorithms merely consider visual features and do not consider
label correlations. Indeed, there inherently exist strong label correlations
in multi-label images, and their correlation can serve as additional knowledge
to regularize MLR model training. Previous works, like [35, 34] introduce
recurrent neural networks (RNNs) or long short-term memory networks (LSTM) to
learn contextualized feature representation that implicitly captures these
correlations. Inspired by the graph propagation networks [36, 37], recent
works [16, 15] postulate the use of structured graph representations to model
label correlations in an explicit manner, which obviously facilitate MLR
performance. For example, Chen et al. [15] first learn category-specific
features, exploit label co-occurrence correlation to correlate these features
and introduce graph neural networks to propagate information through the graph
to learn contextualized features. Benefiting from the more powerful
transformer networks [38, 2][39], more recent works further introduce
transformer networks [40, 18] to learn more discriminative features or capture
label correlations better and thus obtain better MLR performance.

###  II-B Confidence Calibration

Confidence calibration of multi-class classification has been extensively
studied for a long time. Confidence Calibration aims to make a classifier
correctly quantify uncertainty or confidence associated with its instance-wise
predictions. Many methods have been proposed [10, 41, 3, 11, 42, 20, 21,
19][43, 44] to correct the model confidence. Label smoothing (LS) [10] was
proposed as a foundation technology of model calibration to prevent the
network from becoming over-confident. Focal Loss (FL) [41] focused training on
a sparse set of hard examples and prevented many easy negatives from
overwhelming the detector during training initially. In a follow-up study [3],
it was found that it can act as a model calibration method. Then, Focal Loss
with sample-dependent schedule (FLSD) [3] was proposed as an improved focal
loss to select the hyperparameter for model calibration automatically. The
difference between predicted confidence and accuracy (DCA) [11], a calibration
method based on expected calibration error, was proposed to correct confidence
by adding the difference between predicted confidence and accuracy as an
auxiliary loss. Maximum Mean Calibration Error (MMCE) [42], an RKHS kernel-
based measure of calibration, is sound for perfect calibration that is
minimized and whose finite sample estimates are consistent and enjoy fast
convergence rates. Multi-class Difference in Confidence and Accuracy (MDCA)
[20] acts as a novel auxiliary loss function to help a model achieve the same
MDCA. Margin-based Label Smoothing (MbLS) [21] provides a unifying constrained
optimization perspective of current state-of-the-art calibration losses and
then achieves a simple yet flexible generalization based on inequality
constraints by imposing a controllable margin on logit distances. Dynamically
Weighted Balance Loss (DWBL) [19] adjusted the model confidence with a class
rebalancing strategy based on a class-balanced dynamically weighted loss. It
can mitigate the class distribution imbalance issue in deep learning by
assigning weights to different classes based on the class frequency and
predicted probability of the ground-truth class. Adapting its weights
automatically depending on the prediction scores allows a model to adjust for
instances with varying difficulty levels, resulting in model calibration.

Although the above methods have made significant progress in confidence
calibration of the multi-class classification task, they still have not been
explored under the setting of multi-label recognition. In this work, we extend
them into the multi-label recognition task to validate their effectiveness and
construct an evaluation benchmark by including them as baselines for a fair
comparison. Besides, these methods do not consider category correlation that
can well capture semantic confusion, leading to sub-optimal performance in the
MLCC scenarios. To address this issue, we propose a novel confidence
calibration method, DCLR, to learn category correlations to model semantic
confusion in multi-label images and integrate the correlations to build
adaptive and more effective calibration.

##  III Dynamic Correlation Learning and Regularization

Figure 2: An overall illustration of the proposed DCLR algorithm and its
integration into MLR models. Initially, an input image is processed through a
backbone network followed by the SARL module to extract category-specific
features. Subsequently, we compute instance-level and prototype-level
correlation matrices by calculating the similarities between the extracted
features and those of selected instance samples, as well as between the
extracted features and prototype representations, to effectively model
semantic confusion. Finally, we calculate the instance-level and prototype-
level softened label vectors based on the respective correlation matrices and
the ground truth labels. The softened label vectors are utilized for MLR model
training.

MLR focuses on predicting labels of samples in a multi-class classification
problem where each sample may belong to more than one class. Given a training
set $\mathcal{D}=\\{(\mathbf{x}^{n},\mathbf{y}^{n})\\}^{N}_{n=1}$ and $C$
object categories, an MLR model aims to predict one or more object categories
that truly belong to a given image. Here, $N$ is the total number of samples,
$\mathbf{x}^{n}\in\mathbb{R}^{H\times W\times 3}$ denotes $n$-th image in
$\mathcal{D}$, and
$\mathbf{y}^{n}=\\{y_{1}^{n},\cdots,y_{C}^{n}\\}\in\\{1,0\\}^{C}$ is the
multi-hot encoded label vector. Most existing methods optimized a model to
achieve this goal by applying the ground-truth multi-hot label vectors as
supervisory signals with a binary cross-entropy loss function. The ground-
truth multi-hot label vector comprises multiple hard labels of different
categories where each hot is 0 or 1. Although these hard labels can define the
categories to which an image belongs, optimizing a model with these hard
labels often makes the model become over-confident, causing its output
distribution to be untrustworthy and hard to associate with the accurate
probability of correctness for the predictions [4, 21].

To mitigate the over-confident issue of neural networks, label smoothing (LS)
[10, 21], which is a form of output distribution regularization and a well-
established single-label confidence calibration method, is often deployed to
increase robustness and improve multi-class recognition problems. Label
smoothing converts the hard labels into soft labels by applying a weighted
average between the uniform distribution and the hard labels. It can be
formulated as follows:

$\tilde{y}^{i}_{c}=\left\\{\begin{aligned} 1-\alpha,\quad&y^{i}_{c}=1\\\
\frac{\alpha}{C-1},\quad&y^{i}_{c}=0\end{aligned}\right.$ (1)

It has been shown to calibrate the learned models implicitly by preventing the
model from assigning the full probability mass to a single class and
maintaining a reasonable distance between the logits of the ground-truth class
and the other classes [45, 21]. Intuitively, the label smoothing
regularization can similarly be extended to multi-label recognition problems.
It can be formulated as follows:

$\tilde{y}^{i}_{c}=\left\\{\begin{aligned} 1-\alpha,\quad&y^{i}_{c}=1\\\
\frac{\alpha M}{C-M},\quad&y^{i}_{c}=0\end{aligned}\right.$ (2)

where $M\leq C$ is the class number of the sample $\mathbf{x}^{i}$ and
$\alpha\geq 0$ is a hyperparameter for output distribution regularization. If
$M$ equals 1, Eq. (2) is equal to Eq. (1). If $\alpha$ is set to 0, Eq. (2)
equals the original labels. Besides, if $\alpha$ is too large, the model will
fail to predict the ground-truth label.

The existing label smoothing algorithms distribute the penalty term, $\alpha$,
uniformly across negative categories. However, these algorithms overlook the
semantic confusion and correlation between similar object categories, a
phenomenon naturally occurring in multi-label images. As a result, they fail
to effectively regulate negative categories that bear a strong correlation
with the positive categories. Thus, these highly-correlated categories remain
confused with each other, leading to sub-optimal calibration performance in
MLR scenarios.

In this work, we posit a straightforward yet cogent assumption: negative
categories with high similarity to positive ones should incur higher penalty
values, while others should receive lesser penalties. To this end, we
introduce the Dynamic Correlation Learning and Regularization (DCLR)
algorithm, which is adept at learning category correlations to effectively
model semantic confusion in multi-label images and incorporates these
correlations to construct an adaptive and efficient regularization on MLR
models. Figure 2 illustrates the DCLR algorithm and its integration with
existing MLR models. DCLR builds on a Category-Specific Contrastive Learning
module to learn semantic-aware feature representation for each category. Then,
the algorithm explores both instance-to-instance and instance-to-prototype
similarities to model category correlations and computes softened label
vectors based on these correlations. These vectors are subsequently employed
to supervise existing MLR models. By implementing this adaptive
regularization, DCLR effectively penalizes highly correlated categories,
thereby more accurately resolving semantic confusion.

###  III-A Category-Specific Contrastive Learning

To accurately gauge category correlation, it is essential to learn
semantically-aware features for each category that satisfy two key criteria.
Firstly, the feature vector for category $c$ should predominantly encapsulate
the semantic information specific to that category, effectively minimizing the
influence of information from other categories. Secondly, it is anticipated
that features of correlated categories will exhibit a high degree of
similarity, whereas those of uncorrelated categories will demonstrate
significantly less similarity. To achieve this goal, we introduce a category-
specific contrastive learning (CSFL) module. This module integrates category
semantics to guide capturing the specific semantic information and is trained
using a combination of classification and contrastive losses. It not only
enhances focus on relevant semantic content but also guarantees the effective
differentiation and alignment of features as per category correlations,
ensuring that it adheres to the two aforementioned criteria.

Given an input image $I^{m}$, we first extract category-specific feature
vectors via a standard backbone (e.g., ResNet101 [1]) followed by semantic-
embedded attention mechanism as previous work [15], shorted as CNN&SARL .
Formally, it can be represented as

$\\{\mathbf{f}^{m}_{1},\mathbf{f}^{m}_{2},\cdots,\mathbf{f}^{m}_{C}\\}=\phi(I^{m}).$
(3)

To ensure the learned features satisfy two key criteria, we introduce a
combination of classification and contrastive losses.

Auxiliary classification loss.  To ensure that the feature vector
$\mathbf{f}^{m}_{c}$ effectively encapsulates the semantic information
pertinent to its category, we employ a classifier tasked with predicting the
specific category $c$ solely based on the corresponding feature vector
$\mathbf{f}^{m}_{c}$. This process is applied across all learned feature
vectors, resulting in a probability distribution
$\hat{\mathbf{p}}^{m}=\\{\hat{p}^{m}_{1},\hat{p}^{m}_{2},\ldots,\hat{p}^{m}_{C}\\}$.
Subsequently, we leverage softened label vectors—specifically,
$\tilde{\mathbf{y}}^{m}=\\{\tilde{y}^{m}_{1},\tilde{y}^{m}_{2},\cdots,\tilde{y}^{m}_{C}\\}$
and
$\hat{\mathbf{y}}^{m}=\\{\hat{y}^{m}_{1},\hat{y}^{m}_{2},\cdots,\hat{y}^{m}_{C}\\}$
(the elaboration of these vectors can be found in Sections III-B and III-C) to
define the classification loss.

$\mathcal{L}_{acl}=\eta\sum_{m=1}^{N}(\ell(\hat{\mathbf{p}}^{m},\hat{\mathbf{y}}^{m})+\ell(\hat{\mathbf{p}}^{m},\tilde{\mathbf{y}}^{m}))$
(4)

where

$\ell(\mathbf{p},\mathbf{y})=\sum_{c=1}^{C}(y_{c}\log(p_{c})+(1-y_{c})\log(1-p_{c}))$
(5)

In this context, we set $\eta$ to 0.5 to align the magnitude of the
traditional cross-entropy loss which is a common-used classification loss.

Category-specific contrastive loss. Another criterion we consider is the
expectation that features within correlated categories will show a high degree
of similarity while those in uncorrelated categories will display less
similarity. However, due to the lack of precise annotations detailing these
category similarities, accurate supervision signals to learn these
correlations are unavailable. In this work, we introduce a category-specific
contrastive loss to leverage existing label annotations for learning these
correlations. For any two images and a given category, our approach seeks to
draw the corresponding features closer when the given category is present in
both images and to distance them otherwise. Since objects from the same
category in different images typically share similar visual characteristics
[46], it can effectively learn visual correlations across different
categories.

Formally, given two images $m$, $n$, and category $c$, we compute their
similarity via the cosine distance

$s_{c}^{m,n}=\mathrm{cosine}(\mathbf{f}_{c}^{m},\mathbf{f}_{c}^{n})=\dfrac{\mathbf{f}_{c}^{m}\cdot\mathbf{f}_{c}^{n}}{||\mathbf{f}_{c}^{m}||\cdot||\mathbf{f}_{c}^{n}||}.$
(6)

It is expected that $s_{c}^{m,n}$ to be high if both images $m$ and $n$
contain category $c$ and to be low otherwise. To this end, we formulate the
category-specific contrastive loss as

$\mathcal{L}_{c}=\sum_{m=1}^{N}\sum_{c=1}^{C}\sum_{n}\ell_{c}^{m,n},$ (7)

where

$\ell^{m,n}_{c}=\begin{cases}1-s_{c}^{m,n}&y^{i}_{c}=1,y^{j}_{c}=1\\\
1+s_{c}^{m,n}&otherwise.\end{cases}$ (8)

Therefore, the final loss of the category-specific contrastive learning (CSCL)
module is defined as follows:

$\mathcal{L}_{cscl}=\mathcal{L}_{acl}+\mathcal{L}_{c}$ (9)

Traning Algorithm. The overall training algorithm of our CSCL is presented in
Algorithm 1. As introduced before, at each training step, Lines 4-6 take
charge of producing soft labels at the instance level and prototype level.
Then, Lines 7-9 calculate the auxiliary classification loss
$\mathcal{L}_{acl}$, category-specific contrastive loss $\mathcal{L}_{c}$, and
the final loss $\mathcal{L}_{cscl}$. The training procedure will continue over
the data epochs till convergence. After the CSCL is trained, it can be used to
generate soft labels to guide the training of various MLR models.

Algorithm 1 Training Procedure of Category-Specific Contrastive Learning
(CSCL)

Input: Training image dataset $\mathcal{D}$;

Output: well-trained CNN&SARL module $\phi$;

1: initialize CNN&SARL module $\phi$;

2: repeat

3: for all $I^{m}\in\mathcal{D}$ do

4:
$\\{\mathbf{f}^{m}_{1},\mathbf{f}^{m}_{2},\cdots,\mathbf{f}^{m}_{C}\\}=\phi(I^{m})$

5: obtain instance-level soft label vector $\hat{\mathbf{y}}^{m}$ according to
Equation (13)

6: obtain prototype-level soft label vector $\tilde{\mathbf{y}}^{m}$ according
to Equation (15)

7: compute $\mathcal{L}_{acl}$ according to Equation (4)

8: compute $\mathcal{L}_{c}$ according to Equation (7)

9: $\mathcal{L}_{cscl}=\mathcal{L}_{acl}+\mathcal{L}_{c}$

10: update the model parameters of $\phi$ by minimizing $\mathcal{L}_{cscl}$

11: end for

12: until $\phi$ convergence

13: return the CNN&SARL module $\phi$

###  III-B Instance-Level Correlation-Aware Regularization

After acquiring the category-specific feature vector, we can calculate
correlations between various categories through category-specific feature
similarities, resulting in the formation of $\mathbf{R}^{ins}$. In this
matrix, the element $r^{ins}_{ij}$ positioned in the $i$-th row and $j$-th
column signifies the correlation between categories $i$ and $j$. Subsequently,
the generated matrix facilitates the derivation of a soft label vector, which
is instrumental in training the MLR models.

Given an image $m$ and category $c$, to compute the correlation of $c$ with
$c^{\prime}$, we first retrieve a subset of images that have positive label
$c^{\prime}$. The correlation value can be computed by

$r^{ins}_{cc^{\prime}}=\sum_{t=1}^{T}\mathrm{cosine}(\mathbf{f}_{c}^{m},\mathbf{f}_{c^{\prime}}^{n_{t}})$
(10)

Here, $T$ is the number of retrieved images. For each image $m$, we compute
the correlation values across all categories $c$ and $c^{\prime}$ to obtain
the correlation matrix $\mathbf{R}^{ins}_{m}$.

To eliminate the influence of the label itself in the penalty term, we need to
mask the diagonal of the correlation matrix, which can be implemented by

$r^{ins}_{cc^{\prime}}=\begin{cases}-\infty,&c=c^{\prime}\\\
r^{ins}_{cc^{\prime}},&otherwise.\end{cases}$ (11)

Subsequently, we normalize the similarity scores among all categories using a
$\mathrm{softmax}$ function as follows:

$r^{ins}_{cc^{\prime}}=\dfrac{\exp(r^{ins}_{cc^{\prime}})}{\sum_{k}\exp(r^{ins}_{ck})}.$
(12)

Finally, we can obtain a soft label vector according to the instance-level
correlation matrices $\mathbf{R}^{m}_{ins}$ as follows:

$\hat{y}^{m}_{c}=\left\\{\begin{aligned} 1-\alpha,\quad&y^{m}_{c}=1\\\
\sum_{k=1}^{C}\alpha
r^{ins}_{kc}y^{m}_{k},\quad&y^{m}_{c}=0\end{aligned}\right.$ (13)

We repeat the process for all categories and obtain instance-level soft label
vector
$\hat{\mathbf{y}}^{m}=\\{\hat{y}^{m}_{1},\hat{y}^{m}_{2},\cdots,\hat{y}^{m}_{C}\\}$.

###  III-C Prototype-Level Correlation-Aware Regularization

Although the above instance-level correlation-aware regularization can be
deployed as an improved label smoothing, it only captures relatively local
inter-class correlations because only a subset of images is sampled to compute
the category-specific correlations for each input image. To incorporate global
inter-class correlations into our framework to build more robust smoothed
labels for the model calibration, we further model the category-specific
correlations globally by introducing category prototypes that can represent
the global features of categories.

To achieve prototype-level correlation-aware regularization, for each category
$c$, we first extract all instance-level features from all the images that
have been labeled the category $c$ as one of their labels. Then, we apply
$K$-means algorithm [47] to cluster instance-level features into $K$
prototype-level features
$\\{\mathbf{p}^{1}_{c},\mathbf{p}^{2}_{c},...,\mathbf{p}^{K}_{c}\\}$. Here,
the $K$ is set to 10 since the images belonging to any category $c$ have
visual variations. For example, objects belonging to the same category but
from different images may have different appearances. Therefore, we cluster
multiple prototype-level features to account for visual variations for any
category $c$.

Given an image $m$ and category $c$, we can compute the correlation value in
an identical manner, formulated as

$r^{pro}_{cc^{\prime}}=\sum_{k=1}^{K}\mathrm{cosine}(\mathbf{f}_{c}^{m},\mathbf{p}_{c}^{k}).$
(14)

Similarly, we compute the correlation values for all categories $c$ and
$c^{\prime}$ according to equation 14, followed by assigning values of the
diagonal as $-\infty$ and normalizing them with the $\mathrm{softmax}$
function to obtain the prototype-level correlation matrix
$\mathbf{R}^{p}ro_{m}$. Afterwards, we apply it to obtain the soft label
vector as

$\tilde{y}^{m}_{c}=\left\\{\begin{aligned} 1-\alpha,\quad&y^{m}_{c}=1\\\
\sum_{k=1}^{C}\alpha
r^{pro}_{kc}y^{m}_{k},\quad&y^{m}_{c}=0\end{aligned}\right.$ (15)

Finally, we can obtain prototype-level soft label vector
$\tilde{\mathbf{y}}^{m}=\\{\tilde{y}^{m}_{1},\tilde{y}^{m}_{2},\cdots,\tilde{y}^{m}_{C}\\}$.

###  III-D Regularization on Existing MLR Models

When the DCLR algorithm is trained, it can be applied to generate softened
label vectors instead of traditional one-hot label vectors. These softened
vectors are then used to supervise the training of MLR models. Given an input
image $n$, it is fed into the MLR model to predict the probability
distribution $\mathbf{p}^{n}=\\{p^{n}_{1},p^{n}_{2},\ldots,p^{n}_{C}\\}$ and
fed into the DCLR model to obtain the soften label vectors, i.e.,
$\mathbf{p}_{ins}^{n}=\\{y^{n}_{ins,1},y^{n}_{ins,2},\ldots,y^{n}_{ins,C}\\}$
and
$\mathbf{p}_{pro}^{n}=\\{y^{n}_{pro,1},y^{n}_{pro,2},\ldots,y^{n}_{pro,C}\\}$.
The classification loss function can be defined as

$\mathcal{L}_{cls}=\eta(\mathcal{L}^{ins}+\mathcal{L}^{pro})$ (16)

where

$\displaystyle\mathcal{L}^{ins}$
$\displaystyle=\sum_{m=1}^{N}(\ell(\mathbf{p}^{m},\hat{\mathbf{y}}_{m})$ (17)
$\displaystyle\mathcal{L}^{pro}$
$\displaystyle=\sum_{m=1}^{N}(\ell(\mathbf{p}^{m},\tilde{\mathbf{y}}_{m})$

In this formulation, $\ell(\mathbf{p},\mathbf{y})$ is defined according to
Equation 5, with $\eta$ consistently set to 0.5.

To address the issue of overconfidence in existing MLR models [15, 16, 18], we
substitute the traditional cross-entropy loss, which is based on one-hot label
vectors, with the classification loss as defined in Equation 16. Apart from
replacing the classification loss, all other aspects, such as network details
and training processes, remain identical to those in the original works.

##  IV Unified Evaluation Benchmark

In this section, we present the selected multi-label image recognition
methods, competing algorithms, and datasets involved in constructing the
unified multi-label confidence calibration (MLCC) evaluation benchmark. Then,
we introduce the unified evaluation protocols of the benchmark for fair
comparison.

###  IV-A Selected Multi-Label Image Recognition Methods

Most calibration algorithms are implemented on different baselines or targeted
in different scenarios, preventing fair evaluation. To evaluate calibration
algorithms fairly on the multi-label image recognition task, we choose three
multi-label recognition methods as the backbones, including SSGRL [15], ML-GCN
[16], and C-Tran [18]. A brief introduction of these methods is as follows:

  * •

SSGRL [15]: It is a semantic-specific graph representation learning framework
with two key modules. The first key module is a semantic decoupling module
that guides semantic-specific representation learning by incorporating
category semantics. Another module is a semantic interaction module that
correlates these semantic-specific representations with a graph built on the
statistical label co-occurrence and explores their interactions via a graph
propagation mechanism.

  * •

ML-GCN [16]: It is a multi-label classification model based on Graph
Convolutional Network (GCN) to capture and explore label dependencies where
objects normally co-occur in an image. It builds a directed graph over the
object labels and applies learnable GCN to map the label graph into a set of
inter-dependent object classifiers that are applied to the image descriptors
extracted by another sub-net. Besides, it also applies a new re-weighted
scheme to construct an effective label correlation matrix to guide the
optimization of the learnable GCN.

  * •

C-Tran [18]: C-Tran is a general multi-label image classification framework to
leverage transformers [38] to exploit the complex dependencies among visual
features and labels. It consists of a Transformer encoder trained to predict a
set of target labels given an input set of masked labels and visual features
from a CNN. Besides, a novel label masking training objective is also proposed
to use a ternary encoding scheme to represent the label states as positive,
negative, or unknown during training.

The reason that we chose these three multi-label image recognition methods as
the backbones is their different representative characteristics for different
mainstream multi-label image recognition methods. SSGRL is a multi-label image
recognition method with category-specific features, while ML-GCN models
holistic features for classification. C-Tran [18] is a multi-label image
recognition model driven by the transformer which is a new foundation model.

###  IV-B Competing Algorithms

For fair evaluation, we choose some classical algorithms and reimplement them
on our selected backbones. A simple overview of different competing algorithms
is introduced as follows:

  * •

NLL. The negative log-likelihood loss (NLL) is useful for training a
classification problem with multiple classes. Since multi-label visual
recognition is a special multiclass classification task, it is adopted for one
of our baselines.

  * •

LS[10]. Label smoothing (LS) is a foundation technology of model calibration
since it can prevent the network from becoming over-confident. Besides, our
work is also to improve upon it.

  * •

FL[41]. Focal Loss (FL) focuses training on a sparse set of hard examples and
prevents many easy negatives from overwhelming the detector during training
initially. In a follow-up study [3], researchers found it also can be applied
to model calibration. Therefore, we treat it as one of the baselines for
comparison.

  * •

FLSD[3]. Focal Loss with sample-dependent schedule (FLSD) is an improved focal
loss by automatically selecting the hyperparameter for model calibration.

  * •

DCA[11]. The difference between predicted confidence and accuracy (DCA) is a
model calibration method based on expected calibration error by adding the
difference between predicted confidence and accuracy as an auxiliary loss.

  * •

MMCE[42]. Maximum Mean Calibration Error (MMCE) is an RKHS kernel-based
measure of calibration. It is efficiently trainable with the negative
likelihood loss without elaborate hyper-parameter tuning. MMCE is also sound
for perfect calibration that is minimized and whose finite sample estimates
are consistent and enjoy fast convergence rates.

  * •

MDCA[20]. Multi-class Difference in Confidence and Accuracy ( MDCA ) is a
novel auxiliary loss function to help a model achieve the same MDCA.

  * •

MbLS[21]. The authors in Margin-based Label Smoothing (MbLS) [21] first
provide a unifying constrained optimization perspective of current state-of-
the-art calibration losses and then propose a simple yet flexible
generalization based on inequality constraints by imposing a controllable
margin on logit distances.

  * •

DWBL[19]. Dynamically Weighted Balance Loss (DWBL) is a class rebalancing
strategy based on a class-balanced dynamically weighted loss to mitigate the
class distribution imbalance issue in deep learning. It assigns weights to
different classes based on the class frequency and predicted probability of
the ground-truth class. Adapting its weights automatically depending on the
prediction scores allows a model to adjust for instances with varying
difficulty levels resulting in model calibration.

###  IV-C Datasets

We use two famous and publicly available datasets, MS-COCO [48] and Visual
Genome [49] as the benchmark datasets as done in many previous multi-label
recognition works[15, 50] so that we can compare various algorithms fairly and
consistently. Although Pascal VOC 2007 [51] is also widely used as one of the
benchmarks in various multi-label recognition works, we do not use Pascal VOC
2007 [51] as one of the benchmark datasets since it only has 20 common
categories and may not exist the over-confident issue.

The MS-COCO [48] dataset was initially designed for object detection and
segmentation. Recently, it also has been adopted directly for evaluating
multi-label image recognition due to its fine-grained object annotation. This
dataset comprises 122,218 images and contains 80 common categories. These
images are divided into a training set consisting of 82,081 images and a
validation set comprising 40,137 images. It must be noticed that the ground
truth annotations for the test set are currently unavailable. Therefore, all
models reported in this paper are trained on the training set and evaluated on
the validation set.

Visual Genome [49] contains 108,249 images with densely annotated objects,
attributes, and relationships. These images are from the intersection of the
YFCC100M [52] and MS-COCO [48] and cover 80,138 categories. Each image has an
average of 35 objects in the Visual Genome. Therefore, the Visual Genome is a
good test bed for measuring the performance of the multi-label recognition
task and the effect of model calibration. Since most categories have very few
samples, we merely consider the 500 most frequent categories and we randomly
select 10,000 images as the test set and the rest 98,249 images as the
training set.

###  IV-D Evaluation Metrics

Our framework for multi-label confidence calibration is focused on achieving
both accuracy and calibration. To evaluate the accuracy, we follow previous
works[53, 31] to adopt the mean average precision (mAP) across all categories.
In terms of calibration, we adopt several widely-used metrics, including the
expected calibration error (ECE)[23], adaptive ECE (ACE)[54], and maximum
calibration error (MCE)[20]. The ECE measures the difference in expected
accuracy and expected confidence. The ACE measures the difference between
accuracy and confidence in an adaptive scheme that spaces the bin intervals so
that each contains an equal number of predictions. The MCE measures the
maximum difference between average confidence and accuracy.

TABLE I: The performance of our DCLR and various baselines on three different MLC models and two common-used benchmarks. The best and second best results are highlighted and underlined. Datasets | Method | SSGRL [15] | ML-GCN [16] | C-Tran [18]
---|---|---|---|---
| mAP | ACE | ECE | MCE | mAP | ACE | ECE | MCE | mAP | ACE | ECE | MCE
MS-COCO | NLL | 84.0 | 3.692 | 4.108 | 154.069 | 83.0 | 4.143 | 5.072 | 232.622 | 83.5 | 2.414 | 3.675 | 154.668
LS[10] | 83.9 | 2.104 | 1.741 | 69.516 | 83.5 | 1.436 | 2.165 | 55.906 | 83.8 | 1.146 | 1.866 | 55.621
FL[41] | 84.0 | 3.049 | 4.734 | 81.670 | 83.0 | 4.011 | 5.007 | 65.005 | 83.6 | 2.012 | 3.023 | 130.523
FLSD[3] | 84.1 | 3.546 | 3.858 | 133.999 | 82.9 | 4.196 | 4.967 | 48.713 | 83.7 | 1.915 | 2.775 | 100.296
DCA[11] | 84.1 | 3.376 | 3.636 | 127.368 | 82.8 | 3.931 | 4.131 | 120.012 | 83.7 | 2.180 | 3.891 | 130.962
MMCE[42] | 84.1 | 1.575 | 2.951 | 130.991 | 82.8 | 3.523 | 3.840 | 121.866 | 83.7 | 2.111 | 2.874 | 109.911
MDCA[20] | 84.1 | 3.449 | 3.834 | 148.859 | 82.9 | 3.880 | 4.067 | 125.018 | 83.7 | 1.942 | 3.668 | 125.107
MbLS[21] | 83.3 | 2.304 | 2.188 | 96.764 | 83.3 | 2.923 | 2.954 | 67.812 | 83.9 | 1.808 | 2.582 | 108.604
DWBL[19] | 84.0 | 3.181 | 4.345 | 79.217 | 81.3 | 9.726 | 11.170 | 115.477 | 83.6 | 2.195 | 2.519 | 102.985
Ours | 84.0 | 1.459 | 1.503 | 59.018 | 83.5 | 1.037 | 1.427 | 26.411 | 83.8 | 0.862 | 1.489 | 51.854
Visual Genome | NLL | 51.4 | 5.053 | 5.199 | 52.272 | 48.6 | 4.804 | 4.811 | 200.376 | 51.6 | 4.338 | 4.678 | 44.497
LS[10] | 51.4 | 4.400 | 3.721 | 35.902 | 48.7 | 3.643 | 3.892 | 108.051 | 51.6 | 2.985 | 2.673 | 33.170
FL[41] | 51.4 | 4.814 | 4.914 | 42.745 | 48.7 | 3.794 | 3.907 | 187.978 | 51.8 | 4.012 | 3.918 | 39.124
FLSD[3] | 51.5 | 4.541 | 4.874 | 40.798 | 48.7 | 3.587 | 3.876 | 189.701 | 51.8 | 2.981 | 3.513 | 40.325
DCA[11] | 51.6 | 4.565 | 4.820 | 51.406 | 48.7 | 4.518 | 4.866 | 178.643 | 51.6 | 3.824 | 4.198 | 39.908
MMCE[42] | 51.5 | 4.560 | 4.851 | 46.015 | 48.6 | 4.015 | 4.598 | 158.092 | 51.7 | 4.032 | 4.070 | 44.307
MDCA[20] | 51.4 | 4.786 | 5.054 | 52.529 | 48.6 | 4.076 | 4.879 | 150.159 | 51.7 | 4.205 | 4.273 | 45.286
MbLS[21] | 51.6 | 4.741 | 3.907 | 45.676 | 48.7 | 3.987 | 4.675 | 149.802 | 51.7 | 2.964 | 4.329 | 57.031
DWBL[19] | 51.5 | 4.494 | 3.974 | 37.851 | 48.6 | 4.498 | 4.523 | 149.800 | 51.6 | 3.415 | 3.298 | 36.267
Ours | 51.4 | 3.548 | 2.887 | 33.248 | 48.8 | 3.580 | 3.752 | 72.487 | 51.6 | 2.608 | 2.480 | 28.286

##  V Experiments

In this section, we present the implementation details and then report and
analyze the experimental results of our method and various baselines. Besides,
we also conduct ablation studies to validate the effects of different
components in our algorithm. Finally, we show the universality of our DCLR on
downstream applications.

###  V-A Implementation details

We use ResNet-101 [1], which is a convolutional neural network that has 101
layers as the backbone network for feature extraction of DCLR. For better
generalization, we first use pre-trained weights to initialize the backbone
network and then fine-tune it with the data of the downstream task. We train
the DCLR model with the Adam [55] optimizer for 20 epochs. The initial
learning rate, beta1, and beta2 are set to $10^{-5}$, 0.999 and 0.9,
respectively. We reduce the learning rate by 10 times at the 10th epoch. To
train the MLR models, we follow the training settings of the original SSGRL
[15], ML-GCN [16], and C-Tran [18] works.

Figure 3: The reliability diagrams for SSGRL, ML-GCN, and C-Tran models
without and with the existing competing and proposed DCLR algorithms on the
MS-COCO dataset. The results for these models are systematically organized:
SSGRL is shown in the first row, ML-GCN in the second row, and C-Tran in the
last row.

###  V-B Comparisons of DCLR with Various Baselines

In this part, we comprehensively evaluate the performance of both re-
implemented and our proposed DCLR algorithms, under the fair evaluation
benchmark for a thorough and fair comparison. As presented in Table I, our
proposed DCLR algorithm surpasses all competing leading calibration algorithms
in terms of ACE, ECE, and MCE metrics across all three models [15, 16, 18].
This superior improvement is consistent on both the MS-COCO [48] and Visual
Genome [49] datasets. Notably, these advancements are achieved without any
detriment to the mAP metric, which is crucial for measuring classification
performance.

The results are presented in Table I. In our detailed analysis of the MS-COCO
dataset, we observed that existing confidence calibration algorithms
significantly enhance the ACE, ECE, and MCE metrics in most scenarios. Our
newly proposed DCLR algorithm outperforms these existing solutions across all
metrics, demonstrating notable improvements. Specifically, it achieves ACE,
ECE, and MCE scores of 1.459, 1.503, and 59.018 for the SSGRL model; 1.037,
1.427, and 26.411 for the ML-GCN model; and 0.862, 1.489, and 51.854 for the
C-Tran model. Compared with the second best-performing LS algorithm, these
results translate to relative reductions in ACE, ECE, and MCE of 30.7%, 13.7%,
and 15.1% for the SSGRL model; 27.8%, 34.1%, and 52.8% for the ML-GCN model;
and 24.8%, 20.2%, and 6.8% for the C-Tran model, respectively. These results
strongly underscore the superior performance of the DCLR algorithm.

In the case of the Visual Genome dataset, similar phenomena emerge. Adding the
DCLR algorithm leads to a notable decrease in the ACE, ECE, and MCE metrics.
For the SSGRL model, the metrics improved from 4.400, 3.721, and 35.902 to
3.548, 2.887, and 33.248. For the ML-GCN model, the figures changed from
3.643, 3.892, and 108.051 to 3.580, 3.752, and 72.487. Additionally, for the
C-Tran model, the metrics were reduced from 2.964, 2.673, and 33.170 to 2.608,
2.480, and 28.286. These results further confirm the effectiveness of the DCLR
algorithm in enhancing calibration performance.

For more comprehensive evaluations, we also provide reliability diagrams that
graphically illustrate the discrepancy between statistical predictive
confidence and actual accuracy. The closer this curve aligns with the diagonal
line, the better the calibration performance. Figure 3 showcases these
diagrams for SSGRL, ML-GCN, and C-Tran models, both with and without the
application of existing and our proposed DCLR calibration algorithms. The
models without calibration significantly deviate below the diagonal,
suggesting a severe overconfidence dilemma. Utilizing existing calibration
methods moderately improves model alignment. In contrast, the models
calibrated with our DCLR algorithm demonstrate near-perfect alignment with the
diagonal, implying a near-ideal state of calibration.

It is essential for confidence calibration algorithms not to compromise the
classification performance, a key aspect of multi-label recognition (MLR)
tasks. Thus, we further present the mAP metric, the most commonly used measure
in MLR evaluation. As shown in Table I, the proposed DCLR and most existing
algorithms yield competitive mAP scores comparable to those achieved without
calibration across various models and datasets. Notably, we find traditional
LS outperforms most existing algorithms. This disparity in performance may be
attributed to the fact that most existing algorithms are primarily designed
for single-label scenarios and thus exhibit poor performance for the multi-
label counterparts. In contrast, the straightforward approach of label
smoothing regularization is more adaptable across various settings, resulting
in enhanced performance.

###  V-C Ablation Study

To deeply analyze the DCLR algorithm and the actual contributions of each
crucial module, we further present extensive ablative experiments. Here, all
experiments are conducted using the SSGRL baseline on the MS-COCO dataset.

Figure 4: The reliability diagrams of each category using no calibration
algorithm, as well as using the LS and proposed DCLR algorithms. The results
for these algorithms are organized sequentially: the results using no
calibration algorithm are in the first row, those using LS in the second row,
and those with the DCLR algorithm are displayed in the last row. The
evaluations are conducted using the SSGRL model on the MS-COCO dataset.

####  V-C1 Analysis of DCLR

The key technical contribution of this work is the DCLR algorithm, which
incorporates category correlations to effectively resolve semantic
ambiguities, thereby enhancing regularization. We highlight its efficacy by
comparing it with results from uncalibrated models (NLL) and those calibrated
with the LS algorithm. As presented in Table I, utilizing the DCLR algorithm
notably enhances performance compared with both the NLL and LS-calibrated
models. This improvement is further mirrored in the reliability diagrams shown
in Figure 3. For a more thorough comparison, we also present category-specific
reliability diagrams in Figure 4, where we have chosen five categories from
MS-COCO for a concise illustration. The DCLR algorithm significantly narrows
the gap between statistical predictive confidence and actual accuracy,
indicating more accurately calibrated scores.

These evaluations demonstrate the effectiveness of the proposed DCLR algorithm
as a whole. Actually, it contains both instance-level and prototype-level
correlation-aware regularization. In the following, we further present an in-
depth analysis of these two modules.

####  V-C2 Effects of CSCL, ILCAR, and PLCAR

To investigate the effects of category-specific contrastive learning (CSCL),
instance-level correlation-aware regularization (ILCAR), and prototype-level
correlation-aware regularization (PLCAR), we conduct an ablation study on the
SSGRL model and the MS-COCO dataset by enabling/disabling them. The
experimental results are shown in Table II. We can observe that all three
different components can help the model achieve better calibration. To be
specific, Comparing Ours DCLR with Ours DCLR w/o $\mathcal{L}_{c}$, we can see
that CSCL reduces the ACE, ECE, and MCE by 0.439, 0.144, and 9.57,
respectively. Similarly, PLCRA reduces the ACE, ECE, and MCE by 0.33, 0.117,
and 5.01 compared Ours DCLR with Ours DCLR w/o $\mathcal{L}^{pro}$ while ILCRA
reduces the ACE, ECE, and MCE by 0.342, 0.09, and 8.275 compared Ours DCLR
with Ours DCLR w/o $\mathcal{L}^{ins}$, respectively. These results show the
effectiveness of these three components in improving model confidence.
Moreover, when these three components are applied simultaneously, the model
achieves the best calibration performance, showing the effectiveness of our
DCLR.

TABLE II: The effects of CSCL, ILCAR, and PLCAR on the SSGRL model. | mAP | ACE | ECE | MCE
---|---|---|---|---
Ours DCLR w/o $\mathcal{L}_{c}$ | 84.1 | 1.897 | 1.647 | 68.588
Ours DCLR w/o $\mathcal{L}^{ins}$ | 84.0 | 1.801 | 1.593 | 67.293
Ours DCLR w/o $\mathcal{L}^{pro}$ | 83.9 | 1.789 | 1.620 | 64.028
Ours DCLR | 84.0 | 1.459 | 1.503 | 59.018

####  V-C3 Analysis of the hyper-parameter $\alpha$

The hyper-parameter $\alpha$ is used for output distribution regularization.
To measure its influence on the performance and set up proper value, we
conduct an ablation study on MS-COCO with the backbone SSGRL by setting it as
0.02, 0.03, 0.05, 0.1. The experiment results are shown in Table III. From the
results, we can observe that all different $\alpha$ settings can achieve
competitive performance on various metrics while the best and state-of-the-art
performance is achieved by setting $\alpha$ as 0.05. Therefore, we choose 0.05
as the default value of the hyper-parameter $\alpha$ in all our experiments.

TABLE III: The mAP, ACE, ECE, and MCE of our DCLR method on the SSGRL backbone
with different $\alpha$ settings.

| mAP | ACE | ECE | MCE
---|---|---|---|---
0.02 | 84.1 | 1.697 | 1.863 | 85.927
0.03 | 84.1 | 1.602 | 1.795 | 80.687
0.05 | 84.0 | 1.459 | 1.503 | 59.018
0.1 | 83.9 | 1.923 | 2.072 | 89.258

####  V-C4 Analysis of ILCRA

TABLE IV: The mAP, ACE, ECE, and MCE of the LS baseline, Ours ILCRA w/o
$\mathcal{L}_{c}$, and Ours ILCRA.

| mAP | ACE | ECE | MCE
---|---|---|---|---
LS | 83.9 | 2.104 | 1.741 | 69.516
Ours ILCAR w/o $\mathcal{L}_{acl}$ | 83.9 | 2.124 | 1.748 | 70.846
Ours ILCAR w/o $\mathcal{L}_{c}$ | 83.9 | 1.831 | 1.658 | 66.291
Ours ILCAR | 83.9 | 1.789 | 1.620 | 64.028

ILCAR is designed to learn instance-level correlations, aiding in the creation
of soft label vectors that can model semantic confusion. To evaluate its
effectiveness, we conducted experiments solely utilizing ILCAR (namely Ours
ILCRA) and compared it with the Label Smoothing (LS) baseline. The
experimental results, illustrated in Table IV, reveal that even the exclusive
use of ILCRA can enhance calibration performance. In comparison to the LS
baseline, ILCRA reduces the ACE, ECE, and MCE by 0.315, 0.121, and 5.488,
respectively.

The auxiliary classification and contrastive losses can help to better learn
category information and their correlations, and thus facilitate the
construction of more effective regularization. Here, we also verify their
contributions by comparing our model, Ours ILCRA, with two modified baselines
that omit these two losses (denoted as Ours ILCRA w/o $\mathcal{L}_{a}cl$ and
Ours ILCRA w/o $\mathcal{L}_{c}$). As shown in Table IV, the exclusion of
either loss leads to a notable decrease in ACE, ECE, and MCE. We observed that
removing the auxiliary classification loss resulted in even poorer performance
compared to the LS baseline. This could be attributed to the model’s inability
to guarantee that the learned features encompass information specific to the
corresponding category, thereby hindering its ability to accurately learn
correlations for constructing reasonable pseudo labels. Conversely, our model,
even in the absence of the contrastive loss, surpassed the LS baseline,
achieving reductions in ACE and ECE by 0.273 and 0.083, respectively. This
discovery suggests that the inherent similarities present in category-specific
features, which capture semantic category correlations, can enhance
calibration outcomes without requiring retraining.

####  V-C5 Analysis of PLCRA

TABLE V: The mAP, ACE, ECE, and MCE of the LS baseline, Ours PLCRA w/o
$\mathcal{L}_{c}$, and Ours PLCRA.

| mAP | ACE | ECE | MCE
---|---|---|---|---
LS | 83.9 | 2.104 | 1.741 | 69.516
Ours PLCRA w/o $\mathcal{L}_{acl}$ | 84.0 | 2.146 | 1.790 | 70.571
Ours PLCRA w/o $\mathcal{L}_{c}$ | 84.0 | 1.895 | 1.697 | 69.081
Ours PLCRA | 84.0 | 1.801 | 1.593 | 67.293

PLCRA aims to learn more robust prototype-level correlations to construct soft
label vectors. Here, we also evaluate its contribution by merely using PLCRA
(namely Ours PLCRA) and comparing it with the LS baseline. As illustrated in
Table V, it also obtains better calibration performance, with the reductions
of the ACE, ECE, and MCE by 0.303, 0.148, and 2.223, respectively.

The contrastive loss can help to learn more compact representations and thus
obtain more robust prototype-instance similarities. In this part, we design a
new baseline without the contrastive loss (namely Ours PLCRA w/o
$\mathcal{L}_{c}$) and compare it with Ours PLCRA. As illustrated in Table V,
omitting the contrastive loss leads to a notable performance drop in all three
metrics. Notably, Ours PLCRA w/o $\mathcal{L}_{c}$ can also lead to overall
better calibration performance compared with the LS baseline, again
demonstrating the existence of inherent category correlations without
retraining. Similarly, removing the auxiliary classification loss (namely Ours
PLCRA w/o $\mathcal{L}_{acl}$) leads to worse performance than both our PLCRA
and the LS baseline.

$K$, the number of prototype-level features, is also a key important factor
that affects the performance. To investigate its contribution, we conduct
experiments with different $K$s on the SSGRL with our DCLR. The $K$ is set to
5, 10, and 20, respectively. The results are shown in Table VI. We can see
that although all the models with different $K$ can achieve competitive
results, a suitable $K$ is still important for confidence calibration since
different images have different visual variations and a suitable $K$ can
achieve a better trade-off to represent them. According to the experiment
results, we choose 10 as the default value of $K$.

TABLE VI: The mAP, ACE, ECE, and MCE of our DCLR method on the SSGRL backbone
with different $K$ prototype-level features.

$K$ | mAP | ACE | ECE | MCE
---|---|---|---|---
5 | 83.9 | 1.582 | 1.736 | 70.689
10 | 84.0 | 1.459 | 1.503 | 59.018
20 | 83.8 | 1.768 | 2.052 | 92.741

###  V-D Qualitative Comparison

To exhibit how our DCLR calibrates the predicted confidences, we provide some
visualization examples from the MS-COCO dataset with predicted class
confidences on different MLR models and different calibration methods in Fig.
5. From the visualization examples (a), (b), (c), and (d), we can conclude
that our DCLR can perform better confidence score prediction than the original
MLR models and their enhanced versions with label smoothing (LS) since our
DCLR can predict fairly high confidence score for correct classes while
predicting lower confidence score for error classes, showing clearer
discriminative boundaries. For example, in example (a), SSGRL, MLGCN, and
CTran fail to predict ‘bottle‘ due to lower predicted confidence scores.
Calibrating with label smoothing (LS), only SSGRL can predict ‘bottle‘
correctly while both MLGCN and CTran still suffer from lower predicted
confidence scores of ‘bottle‘. Benefiting from our proposed DCLR, all three
models SSGRL, MLGCN, and CTran can predict ‘bottle‘ correctly with higher
confidence scores than those wrong classes. Similar observations and results
can be concluded from other examples. These results further show the
effectiveness of our DCLR on multi-label confidence calibration since it can
calibrate the predicted confidence better.

Figure 5: The visualization examples from the MS-COCO dataset. These examples
can illustrate how our DCLR calibrates the confidence for better predictions.

##  VI Conclusion

In summary, this work tackles the prevalent overconfidence dilemma in multi-
label scenarios, a challenge that has been largely neglected by existing
single-label calibration algorithms. We introduce the innovative Dynamic
Correlation Learning and Regularization (DCLR) algorithm, which harnesses
multi-grained semantic correlations to model the semantic confusion
characteristic of multi-label images, thereby ensuring well-calibrated
confidence scores. DCLR utilizes dynamic instance-level and prototype-level
similarities to construct adaptive label vectors that enable more effective
regularization. Additionally, we have established a comprehensive benchmark to
address the previously existing void in MLCC evaluations. This benchmark re-
implements and adapts several state-of-the-art confidence calibration
algorithms to the MLCC task. It also provides a performance comparison of
these algorithms, alongside our proposed DCLR, integrated within three seminal
MLR models across two widely-used datasets. In the future, MLCC have two
particularly important and meaningful research aspects. First, we currently
consider only pairwise correlations between categories, a form of local
correlations. It would be valuable and significant to consider all categories
and introduce correlations from a holistic perspective, to better learn
correlation information and thus achieve improved regularization. Second,
there are many scenarios where labels are limited, such as multi-label
recognition with partial labels and few-shot multi-label recognition. MLCC
could provide more reliable labels to help effectively retrieve labels, which
could enhance the performance of these practical tasks significantly.

## References

  * [1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [2] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2020.
  * [3] J. Mukhoti, V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania, “Calibrating deep neural networks using focal loss,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 15 288–15 299, 2020.
  * [4] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1321–1330.
  * [5] Z. Lu, S. He, D. Li, Y.-Z. Song, and T. Xiang, “Prediction calibration for generalized few-shot semantic segmentation,” _IEEE transactions on image processing_ , 2023.
  * [6] S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” _Journal of Field Robotics_ , vol. 37, no. 3, pp. 362–386, 2020.
  * [7] J. Janai, F. Güney, A. Behl, A. Geiger _et al._ , “Computer vision for autonomous vehicles: Problems, datasets and state of the art,” _Foundations and Trends® in Computer Graphics and Vision_ , vol. 12, no. 1–3, pp. 1–308, 2020.
  * [8] X. Jiang, M. Osl, J. Kim, and L. Ohno-Machado, “Calibrating predictive model estimates to support personalized medicine,” _Journal of the American Medical Informatics Association_ , vol. 19, no. 2, pp. 263–274, 2012.
  * [9] R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and N. Elhadad, “Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission,” in _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_ , 2015, pp. 1721–1730.
  * [10] R. Müller, S. Kornblith, and G. E. Hinton, “When does label smoothing help?” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [11] G. Liang, Y. Zhang, X. Wang, and N. Jacobs, “Improved trainable calibration method for neural networks on medical imaging classification,” _arXiv preprint arXiv:2009.04057_ , 2020.
  * [12] Y. Wei, W. Xia, M. Lin, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan, “Hcp: A flexible cnn framework for multi-label image classification,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 9, pp. 1901–1907, 2016.
  * [13] Z.-M. Chen, Q. Cui, B. Zhao, R. Song, X. Zhang, and O. Yoshie, “Sst: Spatial and semantic transformers for multi-label image recognition,” _IEEE Transactions on Image Processing_ , vol. 31, pp. 2570–2583, 2022.
  * [14] J. Zhang, J. Ren, Q. Zhang, J. Liu, and X. Jiang, “Spatial context-aware object-attentional network for multi-label image classification,” _IEEE Transactions on Image Processing_ , 2023.
  * [15] T. Chen, M. Xu, X. Hui, H. Wu, and L. Lin, “Learning semantic-specific graph representation for multi-label image recognition,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 522–531.
  * [16] Z.-M. Chen, X.-S. Wei, P. Wang, and Y. Guo, “Multi-label image recognition with graph convolutional networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 5177–5186.
  * [17] J. Ye, J. He, X. Peng, W. Wu, and Y. Qiao, “Attention-driven dynamic graph convolutional network for multi-label image recognition,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16_. Springer, 2020, pp. 649–665.
  * [18] J. Lanchantin, T. Wang, V. Ordonez, and Y. Qi, “General multi-label image classification with transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 16 478–16 488.
  * [19] K. R. M. Fernando and C. P. Tsokos, “Dynamically weighted balanced loss: class imbalanced learning and confidence calibration of deep neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 33, no. 7, pp. 2940–2951, 2021.
  * [20] R. Hebbalaguppe, J. Prakash, N. Madan, and C. Arora, “A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 081–16 090.
  * [21] B. Liu, I. Ben Ayed, A. Galdran, and J. Dolz, “The devil is in the margin: Margin-based label smoothing for network calibration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 80–88.
  * [22] K. Nguyen and B. O’Connor, “Posterior calibration and exploratory analysis for natural language processing models,” _arXiv preprint arXiv:1508.05154_ , 2015.
  * [23] M. P. Naeini, G. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 29, no. 1, 2015.
  * [24] M. H. DeGroot and S. E. Fienberg, “The comparison and evaluation of forecasters,” _Journal of the Royal Statistical Society: Series D (The Statistician)_ , vol. 32, no. 1-2, pp. 12–22, 1983.
  * [25] T. Chen, T. Pu, L. Liu, Y. Shi, Z. Yang, and L. Lin, “Heterogeneous semantic transfer for multi-label recognition with partial labels,” _International Journal of Computer Vision_ , 2024.
  * [26] L. Nie, T. Chen, Z. Wang, W. Kang, and L. Lin, “Multi-label image recognition with attentive transformer-localizer module,” _Multimedia Tools and Applications_ , vol. 81, no. 6, pp. 7917–7940, 2022.
  * [27] T. Chen, L. Lin, R. Chen, X. Hui, and H. Wu, “Knowledge-guided multi-label few-shot learning for general image recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 3, pp. 1371–1384, 2022.
  * [28] Z. Peng, Y. Luo, T. Chen, K. Xu, and S. Huang, “Perception and semantic aware regularization for sequential confidence calibration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 10 658–10 668.
  * [29] T. Pu, T. Chen, H. Wu, Y. Shi, Z. Yang, and L. Lin, “Dual-perspective semantic-aware representation blending for multi-label image recognition with partial labels,” _Expert Systems with Applications_ , vol. 249, p. 123526, 2024.
  * [30] Z.-M. Chen, Q. Cui, X.-S. Wei, X. Jin, and Y. Guo, “Disentangling, embedding and ranking label cues for multi-label image recognition,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 1827–1840, 2020.
  * [31] H. Yang, J. Tianyi Zhou, Y. Zhang, B.-B. Gao, J. Wu, and J. Cai, “Exploit bounding box annotations for multi-label object recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 280–288.
  * [32] J. Ba, V. Mnih, and K. Kavukcuoglu, “Multiple object recognition with visual attention,” _arXiv preprint arXiv:1412.7755_ , 2014.
  * [33] T. Chen, Z. Wang, G. Li, and L. Lin, “Recurrent attentional reinforcement learning for multi-label image recognition,” in _Proceedings of AAAI Conference on Artificial Intelligence_ , 2018, pp. 6730–6737.
  * [34] Z. Wang, T. Chen, G. Li, R. Xu, and L. Lin, “Multi-label image recognition by recurrently discovering attentional regions,” in _Proceedings of IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 464–472.
  * [35] J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, and W. Xu, “Cnn-rnn: A unified framework for multi-label image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2285–2294.
  * [36] Y. Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel, “Gated graph sequence neural networks,” in _International Conference on Learning Representations_ , 2016.
  * [37] T. Chen, T. Pu, H. Wu, Y. Xie, L. Liu, and L. Lin, “Cross-domain facial expression recognition: A unified evaluation benchmark and adversarial graph learning,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 12, pp. 9887–9903, 2021.
  * [38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [39] Z. Zhao, Y. Qiang, F. Yang, X. Hou, J. Zhao, and K. Song, “Two-stream vision transformer based multi-label recognition for tcm prescriptions construction,” _Computers in Biology and Medicine_ , vol. 170, p. 107920, 2024.
  * [40] H. D. Nguyen, X.-S. Vu, and D.-T. Le, “Modular graph transformer networks for multi-label image classification,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 35, no. 10, 2021, pp. 9092–9100.
  * [41] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [42] A. Kumar, S. Sarawagi, and U. Jain, “Trainable calibration measures for neural networks from kernel mean embeddings,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 2805–2814.
  * [43] Q. Shi, M. Ye, W. Huang, W. Ruan, and B. Du, “Label-aware calibration and relation-preserving in visual intention understanding,” _IEEE Transactions on Image Processing_ , 2024.
  * [44] Y. An, H. Xue, X. Zhao, N. Xu, P. Fang, and X. Geng, “Leveraging bilateral correlations for multi-label few-shot learning,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2024.
  * [45] G. Pereyra, G. Tucker, J. Chorowski, L. Kaiser, and G. Hinton, “Regularizing neural networks by penalizing confident output distributions,” 2017.
  * [46] T. Chen, T. Pu, H. Wu, Y. Xie, and L. Lin, “Structured semantic transfer for multi-label recognition with partial labels,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 36, no. 1, 2022, pp. 339–346.
  * [47] J. A. Hartigan and M. A. Wong, “Algorithm as 136: A k-means clustering algorithm,” _Journal of the royal statistical society. series c (applied statistics)_ , vol. 28, no. 1, pp. 100–108, 1979.
  * [48] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [49] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma _et al._ , “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” _International journal of computer vision_ , vol. 123, pp. 32–73, 2017.
  * [50] F. Zhu, H. Li, W. Ouyang, N. Yu, and X. Wang, “Learning spatial regularization with image-level supervisions for multi-label image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5513–5522.
  * [51] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International Journal of Computer Vision_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [52] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, “Yfcc100m: The new data in multimedia research,” _Communications of the ACM_ , vol. 59, no. 2, pp. 64–73, 2016.
  * [53] Y. Wei, W. Xia, M. Lin, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan, “Hcp: A flexible cnn framework for multi-label image classification,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 9, pp. 1901–1907, 2015.
  * [54] J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, “Measuring calibration in deep learning.” in _CVPR workshops_ , vol. 2, no. 7, 2019.
  * [55] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.

|  Tianshui Chen received a Ph.D. degree in computer science at the School of
Data and Computer Science Sun Yat-sen University, Guangzhou, China, in 2018.
Prior to earning his Ph.D, he received a B.E. degree from the School of
Information and Science Technology in 2013. He is currently an associate
professor at the Guangdong University of Technology. His current research
interests include computer vision and machine learning. He has authored and
co-authored more than 40 papers published in top-tier academic journals and
conferences, including T-PAMI, T-NNLS, T-IP, T-MM, CVPR, ICCV, AAAI, IJCAI,
ACM MM, etc. He has served as a reviewer for numerous academic journals and
conferences. He was the recipient of the Best Paper Diamond Award at IEEE ICME
2017.
---|---
|  Weihang Wang is an undergraduate student majoring in Computer Science at
Guangdong University of Technology. He will pursue a Ph.D. degree in Computer
Science at Fudan University in 2024. His research interests lie in machine
learning, particularly in computer vision and large language models.
---|---
|  Tao Pu received a B.E. degree from the School of Computer Science and
Engineering, Sun Yat-sen University, Guangzhou, China, in 2020, where he is
currently pursuing a Ph.D. degree in computer science. He has authored and
coauthored approximately 10 papers published in top-tier academic journals and
conferences, including T-PAMI, AAAI, ACM MM, etc.
---|---
|  Jinghui Qin is a lecturer at the Guangdong University of Technology,
Guangzhou, China. He received his B.S. and MA.Eng degrees from the School of
Software, Sun Yat-Sen University, Guangzhou, China, in 2012 and 2014,
respectively. He also received his Ph.D degree from the School of Data and
Computer Science, Sun Yat-Sen University, Guangzhou, China, in 2020. He was a
Post-Doctoral Fellow at the Sun Yat-sen University, Guangzhou, China, during
2020 and 2022. He has been serving as a reviewer of IEEE Trans. Neural
Networks and Learning Systems, IEEE Trans. Broadcasting, Neurocomputing,
Computer Vision and Image Understanding, Bioinformatics, etc. His research
interest includes natural language processing, machine learning, and computer
vision.
---|---
|  Zhijing Yang received the B.S and Ph.D. degrees from the Mathematics and
Computing Science, Sun Yat-sen University, Guangzhou China, in 2003 and 2008,
respectively. He was a Visiting Research Scholar in the School of Computing,
Informatics and Media, University of Bradford, U.K, between July-Dec, 2009,
and a Research Fellow in the School of Engineering, University of Lincoln,
U.K, between Jan. 2011 to Jan. 2013. He is currently a Professor and Vice Dean
at the School of Information Engineering, Guangdong University of Technology,
China. He has published over 80 peer-reviewed journal and conference papers,
including IEEE T-CSVT, T-MM, T-GRS, PR, etc. His research interests include
machine learning and pattern recognition.
---|---
|  Jie Liu is now a professor and doctoral supervisor at School of Information
Science, North China University of Technology, Beijing, P. R. China. He had
worked in Capital Normal University from 1999 to 2021, He is currently a part-
time doctoral advisor and master’s supervisor of Capital Normal University. He
received PH.D degree in Computer Application Technology at Beijing Institute
of Technology in 2009. He is a deputy director of China language intelligence
research center, a director of China Artificial Intelligence Society and the
secretary-general of Language Intelligence Special Committee of China
Artificial Intelligence Society. He won the first prize of Wu Wenjun’s AI
Science and Technology Progress in 2019, and the second prize of Beijing
Science and Technology Progress in 2017, and Second Prize of Beijing Invention
Patent Award in 2023. He is the project director of Science and Technology
Innovation 2030 Major Project of the Ministry of Science and Technology of
China. His main research interests are Natural Language Processing and
Artificial Intelligence.
---|---
|  Liang Lin (Fellow, IEEE) is a full professor at Sun Yat-sen University.
From 2008 to 2010, he was a postdoctoral fellow at the University of
California, Los Angeles. From 2016–2018, he led the SenseTime R&D teams to
develop cutting-edge and deliverable solutions for computer vision, data
analysis and mining, and intelligent robotic systems. He has authored and co-
authored more than 100 papers in top-tier academic journals and conferences
(e.g., 15 papers in TPAMI and IJCV and 60+ papers in CVPR, ICCV, NIPS, and
IJCAI). He has served as an associate editor of IEEE Trans. Human-Machine
Systems, The Visual Computer, and Neurocomputing and as an area/session chair
for numerous conferences, such as CVPR, ICME, ACCV, and ICMR. He was the
recipient of the Annual Best Paper Award by Pattern Recognition (Elsevier) in
2018, the Best Paper Diamond Award at IEEE ICME 2017, the Best Paper Runner-Up
Award at ACM NPAR 2010, Google Faculty Award in 2012, the Best Student Paper
Award at IEEE ICME 2014, and the Hong Kong Scholars Award in 2014. He is a
Fellow of IEEE, IAPR, and IET.
---|---