# Look Before You Leap: Improving Text-based Person Retrieval by Learning A
Consistent Cross-modal Common Manifold

Zijie Wang  Nanjing Tech UniversityNanjingChina<EMAIL_ADDRESS>0000-0001-8739-7220 ,  Aichun Zhu  Nanjing Tech UniversityNanjingChina
<EMAIL_ADDRESS>0000-0001-6972-5534 ,  Jingyi Xue  Nanjing Tech
UniversityNanjingChina<EMAIL_ADDRESS>0000-0003-4889-6347 ,  Xili Wan
Nanjing Tech UniversityNanjingChina<EMAIL_ADDRESS>0000-0001-9160-8246
,  Chao Liu  Jinling Institute of TechnologyNanjingChina<EMAIL_ADDRESS>,
Tian Wang  Beihang UniversityBeijingChina<EMAIL_ADDRESS>and  Yifeng Li
Nanjing Tech UniversityNanjingChina<EMAIL_ADDRESS>0000-0003-4798-3211

(2022)

###### Abstract.

The core problem of text-based person retrieval is how to bridge the
heterogeneous gap between multi-modal data. Many previous approaches contrive
to learning a latent common manifold mapping paradigm following a cross-modal
distribution consensus prediction (CDCP) manner. When mapping features from
distribution of one certain modality into the common manifold, feature
distribution of the opposite modality is completely invisible. That is to say,
how to achieve a cross-modal distribution consensus so as to embed and align
the multi-modal features in a constructed cross-modal common manifold all
depends on the experience of the model itself, instead of the actual
situation. With such methods, it is inevitable that the multi-modal data can
not be well aligned in the common manifold, which finally leads to a sub-
optimal retrieval performance. To overcome this CDCP dilemma, we propose a
novel algorithm termed LBUL to learn a Consistent Cross-modal Common Manifold
(C3M) for text-based person retrieval. The core idea of our method, just as a
Chinese saying goes, is to ‘san si er hou xing’, namely, to Look Before yoU
Leap (LBUL). The common manifold mapping mechanism of LBUL contains a looking
step and a leaping step. Compared to CDCP-based methods, LBUL considers
distribution characteristics of both the visual and textual modalities before
embedding data from one certain modality into C3M to achieve a more solid
cross-modal distribution consensus, and hence achieve a superior retrieval
accuracy. We evaluate our proposed method on two text-based person retrieval
datasets CUHK-PEDES and RSTPReid. Experimental results demonstrate that the
proposed LBUL outperforms previous methods and achieves the state-of-the-art
performance.

person retrieval, text-based person re-identification, cross-modal retrieval

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Multimedia; October 10–14, 2022; Lisboa,
Portugal††booktitle: Proceedings of the 30th ACM International Conference on
Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal††price: 15.00††doi:
10.1145/3503161.3548166††isbn: 978-1-4503-9203-7/22/10††ccs: Information
systems Image search††ccs: Computing methodologies Object identification

##  1\. Introduction

Figure 1. (a) For CDCP-based paradigms, when mapping features from
distribution of one certain modality into the common manifold, feature
distribution of the opposite modality is completely invisible. That is to say,
how to achieve the cross-modal distribution consensus all depends on the
experience of the model itself, instead of the actual situation. Consequently,
there may exist multiple vague cross-modal common manifolds, which are
adjacent to each other but not exactly identical. (b) For either a visual or
textual sample, LBUL embeds it into C3M after considering the distribution
characteristics of both modalities to achieve a more solid cross-modal
distribution consensus, instead of blindly predicting.

Given a textual description query, text-based person retrieval aims to
identify images of the corresponding pedestrian from a large-scale image
database. Compared to the currently active research topic image-based person
retrieval (aka. person re-identification)(Yi et al., 2014; Hou et al., 2019;
Xia et al., 2019) which utilizes image-based queries, text-based queries are
much easier to access in the realistic application scenarios. Due to its
effectiveness and applicability, text-based person retrieval (Li et al.,
2017b, a; Niu et al., 2020; Jing et al., 2020; Sarafianos et al., 2019; Wang
et al., 2020; Liu et al., 2019; Wang et al., 2021b, 2022) has drawn more and
more attention. However, the study of this task is still in its infancy and
there is still plenty of room for further research.

The core problem of text-based person retrieval is how to bridge the
heterogeneous gap between multi-modal data. As different modalities are
diverse and inconsistent in data form and distribution, it is not so easy to
directly measure the cross-modal affinity. Many of the previous approaches
contrive to learning a common manifold mapping paradigm, either implicitly
with separate sub-models and extra constrains (e.g. attention mechanism), or
explicitly with shared mapping blocks. These methods aim to transform
heterogeneous multi-modal data into homogeneous feature representations,
between which the cross-modal similarity can be calculated. However, most
existing paradigms are proposed following a cross-modal distribution consensus
prediction (CDCP) manner, which have their limitations. Specifically, as the
multi-modal data are from specific distribution of each modality, the process
of the common manifold mapping can be deemed as trying to achieve a
distribution consensus between the visual and textual modalities, so as to
embed and align the multi-modal features in a constructed cross-modal common
manifold. Nevertheless, when mapping features from distribution of one certain
modality into the common manifold, feature distribution of the opposite
modality is completely invisible. That is to say, how to achieve the cross-
modal distribution consensus all depends on the experience of the model
itself, instead of the actual situation. Consequently, as shown in Fig. 1 (a),
there may exist multiple vague cross-modal common manifolds, which are
adjacent to each other but not exactly identical. With such methods, it is
inevitable that the multi-modal data will not be perfected aligned and matched
with each other in a proper common manifold, which finally leads to a sub-
optimal retrieval performance. This situation can be called a CDCP dilemma.

To overcome this CDCP dilemma, we consider to come up with a more effective
common manifold mapping paradigm. In this paper, we propose a novel algorithm
to learn a Consistent Cross-modal Common Manifold (C3M) for text-based person
retrieval. As illustrated in Fig. 1 (b), for either a visual or textual
sample, our proposed method embeds it from one certain modality into C3M after
considering the distribution characteristics of both the visual and textual
modalities to achieve a more solid cross-modal distribution consensus, instead
of blindly predicting. The core idea of our method, just as a Chinese saying
goes, is ‘san si er hou xing’, namely, to Look Before yoU Leap. So we name our
proposed method LBUL, of which the common manifold mapping paradigm includes
two steps, namely, a looking step and a leaping step. Compared with CDCP-based
paradigms, LBUL is capable of achieving a more precise cross-modal
distribution consensus. As a result, the multi-modal data can be embedded and
aligned in a consistent common manifold with less information loss and higher
accuracy, and hence achieve a superior retrieval performance. Specifically,
with a proposed Uni-modal Sub-manifold Embedding Module (USEM), multi-granular
features extracted from each modality are first distilled as a unified
feature, which is embedded in a corresponding uni-modal sub-manifold (visual
or textual). Then at the looking step of LBUL, in order to see distributions
of both modalities before mapping data from one certain modality into C3M, the
uni-modal feature is projected into the opposite sub-manifold to give the
distribution characteristics of the other modality by means of a Cross-modal
Projection (XProj) module. In XProj, a Distribution Shifting ($\mathcal{DS}$)
mechanism plays a key role in the statistical transformation of the feature
according to the target modality, and thus enabling a proper feature
projection. After the projection, for data from one certain modality, there
exists two feature representations embedded in both the visual and textual
modalities. Then at the leaping step, these two representations are processed
together by a Leaping After Seeing Module (LASM), which conducts the common
manifold mapping operation after seeing the distribution characteristics of
both modalities. Through LASM, a consistent common representation can be
obtained in C3M for each sample, so that the cross-modal similarity can be
properly measured. We evaluate our proposed method on two text-based person
retrieval datasets including CUHK-PEDES (Li et al., 2017b) and RSTPReid (Zhu
et al., 2021). Experimental results demonstrate that LBUL outperforms previous
methods and achieves the state-of-the-art performance.

The main contributions of this paper can be summarized as threefold:

  * •

A novel LBUL method is proposed to learn a Consistent Cross-modal Common
Manifold (C3M) for text-based person retrieval, which embeds data from one
certain modality into C3M after considering the distribution characteristics
of both the visual and textual modalities to achieve a more solid cross-modal
distribution consensus, instead of blindly predicting.

  * •

A two-step common manifold mapping mechanism which includes a looking step and
a leaping step is proposed. By conducting the common manifold mapping
operation after seeing the distribution characteristics of both modalities,
LBUL is capable of learning consistent common representations with less
information loss and higher retrieval accuracy.

  * •

Extensive experimental analysis is carried out on CUHK-PEDES (Li et al.,
2017b) and RSTPReid (Zhu et al., 2021) to evaluate the proposed LBUL method
for text-based person retrieval. Experimental results demonstrate that LBUL
significantly outperforms existing methods and achieves the state-of-the-art
performance.

##  2\. Related Works

###  2.1. Person Re-identification

Person re-identification has drawn increasing attention in both academical and
industrial fields. This technology addresses the problem of matching
pedestrian images across disjoint cameras. The key challenges lie in the large
intra-class and small inter-class variation caused by different views, poses,
illuminations, and occlusions. Existing methods can be grouped into handed-
crafted descriptors, metric learning methods and deep learning methods. With
the development of deep learning (Wang et al., 2021a; Sun et al., 2019; Qian
et al., 2021; Zhao et al., 2021), deep learning methods are in general playing
a major role in current state-of-the-art works. Yi et al. (Yi et al., 2014)
firstly proposed deep learning methods to match people with the same
identification. To boost the ReID model training efficiency in multi-label
classification, Wang et al. (Wang and Zhang, 2020) further proposed the
memory-based multi-label classification loss (MMCL). MMCL works with memory-
based non-parametric classifier and integrates multi-label classification and
single-label classification in an unified framework. Jin et al. (Jin et al.,
2020) introduce a global distance-distributions separation (GDS) constraint
over two distributions to encourage the clear separation of positive and
negative samples from a global view. Yuan et al. (Yuan et al., 2020) propose a
Gabor convolution module for deep neural networks based on Gabor function,
which has a good texture representation ability and is effective when it is
embedded in the low layers of a network. Taking advantage of the hinge
function, they also design a new regularizer loss function to make the
proposed Gabor Convolution module meaningful. A model that has joint weak
saliency and attention aware is presented by Ning et al. (Ning et al., 2021),
which can obtain more complete global features by weakening saliency features.
In recent years, methods for unsupervised person re-identification have
gradually emerged. Unsupervised person re-identification means that the target
data set is unlabeled but the auxiliary source data set is not necessarily
unlabeled (Fan et al., 2018). Existing unsupervised person ReID works can be
concluded into three categories. The first category utilizes hand-craft
features (Liao et al., 2015). But the features made by hand can not be robust
and discriminative. To solve this problem, second category (Fu et al., 2019)
adopts clustering to estimate pseudo labels to train the CNN. However, these
methods require good trained model. Recently, the third category is proposed,
which improves unsupervised person ReID by using transfer learning. Some works
(Wang et al., 2018; Lin et al., 2018) utilize transfer learning and minimize
the attribute-level discrepancy by using extra attribute annotations.

###  2.2. Text-based Person Retrieval

Text-based person retrieval aims to search for the corresponding pedestrian
image according to a given text query. This task is first introduced by Li et
al. (Li et al., 2017b) and a GNA-RNN model is employed to handle the multi-
modal data. Later, an efficient patch-word matching model (Chen et al., 2018b)
is proposed to capture the local similarity between image and text. Jing et
al. (Jing et al., 2020) utilize pose information as soft attention to localize
the discriminative regions. Niu et al. (Niu et al., 2020) adopt a Multi-
granularity Image-text Alignments (MIA) model exploit the combination of
multiple granularities. Nikolaos et al. (Sarafianos et al., 2019) design a
Text-Image Modality Adversarial Matching approach (TIMAM) to learn modality-
invariant feature representation by means of adversarial and cross-modal
matching objectives. Liu et al. (Liu et al., 2019) generate fine-grained
structured representations from images and texts of pedestrians with an
A-GANet model to exploit semantic scene graphs. CMAAM is introduced by
Aggarwal et al. (Aggarwal et al., 2020) which learns an attribute-driven space
along with a class-information driven space by introducing extra attribute
annotation and prediction. Zheng et al. (Zheng et al., 2020a) propose a Gumbel
attention module to alleviate the matching redundancy problem and a
hierarchical adaptive matching model is employed to learn subtle feature
representations from three different granularities. Zhu et al. (Zhu et al.,
2021) proposed a Deep Surroundings-person Separation Learning (DSSL) model to
effectively extract and match person information. Besides, they construct a
Real Scenarios Text-based Person Re-identification (RSTPReid) dataset based on
MSMT17 (Wei et al., 2018) to benefit future research on text-based person
retrieval. Most of the above mentioned approaches are proposed following a
cross-modal distribution consensus prediction (CDCP) manner. By means of the
LBUL mechanism, a more solid cross-modal distribution consensus can be
achieved and hence a more consistent cross-modal common manifold can be
constructed.

Figure 2. The overall framework of the proposed LBUL model.

##  3\. Methodology

###  3.1. Problem Formulation

The goal of the proposed framework (shown in Fig. 2) is to measure the
similarity between multi-modal data, namely, a given textual description query
and a gallery person image. Formally, let $D=\\{p_{i},q_{i}\\}_{i=1}^{N}$
denotes a dataset consists of $N$ image-text pairs. Each pair contains a
pedestrian image $p_{i}$ captured by one certain surveillance camera and its
corresponding textual description query $q_{i}$. The IDs of the $Q$
pedestrians in the dataset are denoted as $Y=\\{y_{i}\\}_{i=1}^{Q}$. Given a
textual description, the aim is to identify images of the most relevant
pedestrian from a large scale person image gallery.

###  3.2. Feature Representation Preparation

####  3.2.1. Representation Extraction

##### Visual Representation Extraction

To extract multi-granular visual representations from a given image $I$, a
pretrained ResNet-50 (He et al., 2016) backbone is utilized. To obtain the
global representation $v^{g}\in\mathbb{R}^{p}$, the feature map before the
last pooling layer of ResNet-50 is down-scaled to $1\times 1\times 2048$ with
an average pooling layer and converted into a $2048$-dim vector. Then it is
passed through a group normalization (GN) layer followed by a fully-connected
(FC) layer and transformed to $p$-dim. In the local branch, the same feature
map is first horizontally $k$-partitioned by pooling it to $k\times 1\times
2048$, and then the local strips are separately passed through a GN and two
FCs with an ELU layer between them to form $k$ $p$-dim vectors
$V^{l}=\\{v^{l}_{1},v^{l}_{2},\cdots,v^{l}_{k}\\}$, which are finally
concatenated with each other along with $v^{g}$ to obtained the visual
representation matrix
$V=\\{v^{g},v^{l}_{1},v^{l}_{2},\cdots,v^{l}_{k}\\}\in\mathbb{R}^{p\times(k+1)}$.

##### Textual Representation Extraction

For textual representation extraction, a whole sentence along with $n$ phrases
extracted from it is taken as textual materials, which is processed by a bi-
directional Gated Recurrent Unit (bi-GRU). The last hidden states of the
forward and backward GRUs are concatenated to give global/local $2p$-dim
feature vectors. And then the $2p$-dim vector got from the whole sentence is
passed through a GN followed by an FC to form the global textual
representation $t^{g}\in\mathbb{R}^{p}$. With each certain input phrase, the
corresponding output $p$-dim vector is handled consecutively by a GN and two
FCs with an ELU layer between them. The obtained local vectors
$T^{l}=\\{t^{l}_{1},t^{l}_{2},\cdots,t^{l}_{n}\\}$ are then concatenated with
each other along with $t^{g}$ to form the final textual representation matrix
$T=\\{t^{g},t^{l}_{1},t^{l}_{2},\cdots,t^{l}_{n}\\}\in\mathbb{R}^{p\times(n+1)}$.

####  3.2.2. Uni-modal Sub-manifold Learning

After the raw visual and textual representation matrices $V$ and $T$ are
obtained, we adopt a Uni-modal Sub-manifold Embedding Module (USEM) based on
the self-attention mechanism (Vaswani et al., 2017a) to distill both global
and fine-grained local discriminative information into a unified feature,
which can be formulated as

(1)
$v^{u}=USEM(V)=\sum_{i=1}^{k}\frac{exp(v^{g}v_{i}^{l})}{\sum_{j=1}^{k}exp(v^{g}v_{j}^{l})}v^{l}_{i}+v^{g},$
(2)
$t^{u}=USEM(T)=\sum_{i=1}^{n}\frac{exp(t^{g}t_{i}^{l})}{\sum_{j=1}^{n}exp(t^{g}t_{j}^{l})}t^{l}_{i}+t^{g},$

where $v^{u}$ or $t^{u}$ is the distilled unified visual/textual feature and
is embedded into the corresponding visual/textual sub-manifold, respectively.

###  3.3. Look Before You Leap

Now that the uni-modal representations are obtained and each uni-modal sub-
manifold which reveals the latent distribution characteristics of the
corresponding modality is constructed, a two-step common manifold mapping
mechanism including a looking step and a leaping step is proposed, which aims
to embed the multi-modal data into the consistent cross-modal common manifold
C3M with less information loss and higher accuracy.

####  3.3.1. Looking Step: Cross-modal Projection Module

At the looking step, the prime target is to see distributions of both
modalities before mapping data from one certain modality into C3M. To achieve
this goal, a Cross-modal Projection (XProj) module is proposed to project one
certain uni-modal representation into the opposite sub-manifold, which can be
generally formulated as

(3) $s^{p}=XProj(s^{u},r^{u}),$

where $s^{u}$ or $r^{u}$ denotes the input source or target modal
representation, respectively. $s^{p}$ is the projected representation, which
is embedded into the target sub-manifold.

Specifically, a Distribution Shifting ($\mathcal{DS}$) mechanism is first
adopted to conduct the statistical transformation of the source modal
representation according to the target modality:

(4)
$s^{ds}=\mathcal{DS}(s^{u},r^{u})=\sigma(r^{u})(\frac{s^{u}-\mu(s^{u})}{\sigma(s^{u})})+\mu(r^{u}),$

where $s^{ds}$ is the shifted feature. $\mu(\cdot)$ and $\sigma(\cdot)$ denote
the calculation of mean and variance, respectively. Then a multi-layer
perceptron ($\mathcal{MLP}$) with a tanh activation layer is employed to embed
$s^{ds}$ into the target sub-manifold:

(5) $s^{p}=\mathcal{MLP}(s^{ds}).$

With XProj, representation lying in one certain uni-modal sub-manifold can be
properly projected into the opposite sub-manifold:

(6) $v^{p}=XProj(v^{u},t^{u}),\ t^{p}=XProj(t^{u},v^{u}).$

####  3.3.2. Leaping Step: Leaping After Seeing Module

After the looking step, for data from one certain modality, there exists two
feature representations embedded in both the visual and textual modalities,
which give the distribution characteristics of both modalities. Then at the
leaping step, the two representations can be utilized by LBUL with a proposed
Leaping After Seeing Module (LASM) to conduct the common manifold mapping
operation after seeing distributions of both modalities. The mechanism of LASM
can be formulated as

(7) $x^{c}=LASM(x^{u},x^{p}),$

where $x^{u}$ and $x^{p}$ are visual/textual representations before and after
processed by XProj, respectively, while $x^{c}$ is the visual/textual
consistent common manifold representation embedded in C3M.

To be specific, the two input representations are first utilized to estimate a
fusion gate $g\in\mathbb{R}^{p}$:

(8) $g=\sigma(W_{2}ELU(W_{1}(x^{u}\oplus x^{p}))),$

where $\oplus$ is the feature concatenation operation and can be implemented
as several other methods (e.g. addition or concatenation), which will be
further discussed along with some substitution variants of LASM in Sec. 4.2.3.
Here, $W_{1}\in\mathbb{R}^{2p\times 2p}$ and $W_{2}\in\mathbb{R}^{2p\times p}$
denote linear transformations without bias while $\sigma(\cdot)$ stands for
the sigmoid activation function. Then $x^{c}$ is obtained through the weighted
summation of $x^{u}$ and $x^{p}$ according to $g$:

(9) $x^{c}=gx^{u}+(1-g)x^{p}.$

To sum up, the corresponding visual and textual consistent cross-modal common
manifold representations $v^{c}$ and $t^{c}$ can be obtained as

(10) $v^{c}=LASM(v^{u},v^{p}),\ t^{c}=LASM(t^{u},t^{p}).$

###  3.4. Similarity for Inference

For test and inference, several kinds of similarity scores are calculated.
First, the similarity $sim^{c}$ between a pair of visual/textual
representations embedded in C3M is calculated:

(11) $sim^{c}=cos(v^{c},t^{c}),$

where $cos(\cdot,\cdot)$ denotes the cosine similarity between two feature
vectors. To further improve the retrieval performance, the global and fine-
grained similarities are also utilized as an auxiliary. The global similarity
$sim^{g}$ between a pair of global multi-modal representations is computed as

(12) $sim^{g}=cos(v^{g},t^{g}).$

To obtain the fine-grained similarity $sim^{f}$, a cross-modal attention
($\mathcal{CA}$) mechanism is adopted:

(13)
$\alpha^{X}_{i}=\frac{exp(cos(x^{l}_{i},y^{g}))}{\sum_{j}exp(cos(X^{l}_{j},y^{g}))},$
(14)
$x^{f}=\mathcal{CA}(y^{g},X^{l})=\sum\limits_{\alpha^{X}_{i}>\gamma}\alpha^{X}_{i}x^{l}_{i},$

where $(X,Y)$ can be $(V,T)$ or $(T,V)$ while $(x,y)$ can be $(v,t)$ or
$(t,v)$. $\gamma$ is a threshold value. Thus, $sim^{f}$ can be calculated by

(15) $v^{f}=\mathcal{CA}(t^{g},V^{l}),\ t^{f}=\mathcal{CA}(v^{g},T^{l}),$ (16)
$sim^{f}=\frac{cos(v^{g},t^{f})+cos(v^{f},t^{g})}{2}.$

Eventually, we can get the overall similarity $sim$ as

(17) $sim=sim^{c}+\lambda_{1}sim^{g}+\lambda_{2}sim^{f}.$

###  3.5. Optimization

To optimize LBUL, the ranking loss is employed to constrain the matched pairs
to be closer than the mismatched ones in a mini-batch with a margin $\beta$:

(18) $L_{rk}(x_{1},x_{2})=\sum_{\widehat{x_{2}}}max\\{\beta-
cos(x_{1},x_{2})+cos(x_{1},\widehat{x_{2}}),0\\}\\\
+\sum_{\widehat{x_{1}}}max\\{\beta-
cos(x_{1},x_{2})+cos(\widehat{x_{1}},x_{2}),0\\},$

where $(x_{1},\widehat{x_{2}})$ or $(\widehat{x_{1}},x_{2})$ denotes a
mismatched pair while $(x_{1},x_{2})$ is a matched pair. Besides, the
identification (ID) loss is also adopted:

(19) $L_{id}(x)=-log(softmax(W_{id}x),$

where $W_{id}\in\mathbb{R}^{Q\times p}$ is a shared FC layer without bias
while $Q$ is the number of different pedestrians.

The overall optimization process of LBUL includes two stages. Before
conducting the common manifold mapping operation, it is necessary to ensure
each learned sub-manifold is solid. Therefore, in the first stage, the ranking
loss and ID loss are utilized to optimize the extracted global and fine-
grained local features:

(20) $\mathcal{L}^{g}=L_{id}(v^{g})+L_{id}(t^{g})+L_{rk}(v^{g},t^{g}),$ (21)
$\mathcal{L}^{f}=L_{id}(v^{f})+L_{id}(t^{f})+L_{rk}(v^{g},t^{f})+L_{rk}(v^{f},t^{g}),$
(22) $\mathcal{L}^{Stage1}=\mathcal{L}^{g}+\lambda_{3}\mathcal{L}^{f}.$

In the second stage, first the ranking loss between the projected feature and
the unified feature in the target modality is computed to ensure a reliable
projection. Besides, the ID loss is also employed:

(23)
$\mathcal{L}^{p}=L_{id}(v^{u})+L_{id}(t^{u})+L_{id}(v^{p})+L_{id}(t^{p})\\\
+L_{rk}(v^{p},t^{u})+L_{rk}(v^{u},t^{p}),$

And then the loss between a pair of common manifold features is calculated as

(24) $\mathcal{L}^{c}=L_{id}(v^{c})+L_{id}(t^{c})+L_{rk}(v^{c},t^{c}).$

The entire loss for the second stage is

(25)
$\mathcal{L}^{Stage2}=\mathcal{L}^{Stage1}+\lambda_{4}\mathcal{L}^{p}+\lambda_{5}\mathcal{L}^{c}.$

##  4\. Experiments

###  4.1. Experimental Setup

####  4.1.1. Dataset and Metrics

Our approach is evaluated on two challenging Text-based Person Retrieval
datasets including CUHK-PEDES (Li et al., 2017b) and RSTPReid (Zhu et al.,
2021).

(1) CUHK-PEDES: Following the official data split approach (Li et al., 2017b),
the training set of CUHK-PEDES contains 34054 images, 11003 persons and 68126
textual descriptions. The validation set contains 3078 images, 1000 persons
and 6158 textual descriptions while the test set has 3074 images, 1000 persons
and 6156 descriptions. Every image generally has two descriptions, and each
sentence is commonly no shorter than 23 words. After dropping words that
appear less than twice, the word number is 4984.

(2) RSTPReid: The RSTPReid dataset (Zhu et al., 2021) is constructed based on
MSMT17 (Wei et al., 2018), which contains 20505 images of 4,101 persons from
15 cameras. Each person has 5 corresponding images taken by different cameras
and each image is annotated with 2 textual descriptions. For data division,
3701, 200 and 200 identities are utilized for training, validation and test,
respectively. Each sentence is no shorter than 23 words.

##### Evaluation Metrics

The performance is evaluated by the rank-k accuracy. All images in the test
set are ranked by their similarities with a given query natural language
sentence. If any image of the corresponding person is contained in the top-k
images, we call this a successful search. We report the rank-1, rank-5, and
rank-10 accuracies for all experiments.

####  4.1.2. Implementation Details

In our experiments, we set the representation dimensionality $p=2048$. The
dimensionality of embedded word vectors is set to 500. The pretrained
ResNet-50 (He et al., 2016) is utilized as the visual CNN backbone and a
pretrained BERT language model (Vaswani et al., 2017b) is used to better
handle the textual input. The total number of noun phrases obtained from each
sentence is kept flexible while $k$ is set to 6. For both the CUHK-PEDES and
RSTPReid dataset, the input images are resized to $384\times 128\times 3$. The
random horizontal flipping strategy is employed for data augmentation. The
threshold value $\gamma$ in $\mathcal{CA}$ can be $\frac{1}{k}$ or
$\frac{1}{n}$ for visual or textual data, respectively. The margin $\beta$ of
ranking losses is set to 0.2 while the $\lambda$’s are empirically set to 1 in
this paper. An Adam optimizer (Kingma and Ba, 2015) is adopted to train the
model with a batch size of 64 for 100 epochs.

###  4.2. Ablation Analysis

Table 1. Performance comparisons of common manifold mapping paradigms on CUHK-PEDES and RSTPReid. | Method | Rank-1 | Rank-5 | Rank-10
---|---|---|---|---
CUHK-PEDES | CDCP-Sep (glo) | 56.19 | 76.43 | 83.63
CDCP-Sha (glo) | 56.78 | 77.19 | 84.07
LBUL (glo) | 57.20 | 77.78 | 84.23
CDCP-Sep (USEM) | 59.62 | 79.39 | 85.83
CDCP-Sha (USEM) | 59.88 | 79.56 | 85.91
LBUL (USEM) | 61.95 | 81.16 | 87.19
RSTPReid | CDCP-Sep (glo) | 37.05 | 61.75 | 71.10
CDCP-Sha (glo) | 37.85 | 62.05 | 71.50
LBUL (glo) | 38.65 | 64.70 | 73.20
CDCP-Sep (USEM) | 40.70 | 65.55 | 75.10
CDCP-Sha (USEM) | 41.40 | 65.85 | 75.40
LBUL (USEM) | 43.35 | 66.85 | 76.50

####  4.2.1. Comparison with CDCP-based Paradigms

The core idea of this paper is the ‘Look Before You Leap (LBUL)’ mechanism. To
demonstrate its effectiveness, a series of experiments on CUHK-PEDES and
RSTPReid are carried out to compare LBUL with CDCP-based paradigms:

(1) CDCP-Sep (glo): directly using the global features $v^{g}$ and $t^{g}$
extracted by modality-specific sub-models to calculate $s^{g}$ for matching;

(2) CDCP-Sha (glo): calculating $s^{g}$ after processing $v^{g}$ and $t^{g}$
with a shared mapping block;

(3) LBUL (glo): processing global features with LBUL mechanism (XProj + LASM)
to obtain $s^{g}$;

(4) CDCP-Sep (USEM): calculating a similarity $s^{u}$ with output of USEM
$v^{u}$ and $t^{u}$, and then matching multi-modal samples with $s^{g}$,
$s^{f}$ and $s^{u}$;

(5) CDCP-Sha (USEM): calculating $s^{c}$ after processing $v^{u}$ and $t^{u}$
with a shared mapping block and then matching multi-modal samples with
$s^{g}$, $s^{f}$ and $s^{c}$;

(6) LBUL (USEM): just equivalent to the complete LBUL method, which conducts
the LBUL-based operation on $v^{u}$ and $t^{u}$ to obtain $s^{c}$ and matches
multi-modal samples with $s^{g}$, $s^{f}$ and $s^{c}$.

The experimental results are reported in Tab. 1. As can be observed from the
table, for CDCP-based methods, approaches using a shared common manifold
mapping block outperform ones without to some extent. Furthermore, with the
idea of LBUL for common manifold mapping, an obvious performance gain can be
achieved. For instance, for methods using $v^{u}$ and $t^{u}$, the LBUL-based
method outperforms the CDCP-based method with a shared mapping block by
$2.07\%$, $1.60\%$, $1.28\%$ and $1.95\%$, $1.00\%$, $1.10\%$ on CUHK-PEDES
and RSTPReid, respectively. The experimental results demonstrate that by
conducting the common manifold mapping operation after seeing distribution
characteristics of both modalities, LBUL is more capable of learning
consistent common representations with less information loss and higher
retrieval accuracy.

Table 2. Ablation study on proposed components of LBUL on CUHK-PEDES and RSTPReid. Component | CUHK-PEDES | RSTPReid
---|---|---
$s^{g}$ | $s^{f}$ | $s^{c}$ | 2ST | USEM | LASM | BERT | Rank-1 | Rank-5 | Rank-10 | Rank-1 | Rank-5 | Rank-10
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | 53.50 | 75.05 | 82.68 | 34.85 | 60.15 | 70.95
$\checkmark$ | $\times$ | $\times$ | - | - | - | $\times$ | 54.81 | 75.65 | 83.43 | 37.00 | 61.75 | 71.10
$\checkmark$ | $\checkmark$ | $\times$ | - | - | - | $\times$ | 57.37 | 77.83 | 85.00 | 40.15 | 63.95 | 74.10
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | glo | $\checkmark$ | $\times$ | 60.81 | 80.79 | 86.72 | 41.60 | 65.55 | 75.30
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | avg | $\checkmark$ | $\times$ | 60.37 | 80.17 | 86.62 | 42.40 | 66.15 | 76.10
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | avgloc + glo | $\checkmark$ | $\times$ | 61.04 | 80.95 | 86.85 | 42.15 | 66.00 | 76.15
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | w/o $\mathcal{DS}$ | $\times$ | 60.24 | 80.12 | 86.32 | 41.25 | 65.15 | 75.55
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | add | $\times$ | 60.57 | 80.56 | 86.56 | 40.45 | 64.35 | 73.70
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | add + $\mathcal{MLP}$ | $\times$ | 60.32 | 80.05 | 86.53 | 41.15 | 65.25 | 75.35
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | concat | $\times$ | 60.31 | 80.41 | 86.93 | 42.25 | 66.30 | 76.15
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | concat + $\mathcal{MLP}$ | $\times$ | 60.39 | 80.36 | 86.19 | 42.05 | 65.70 | 75.95
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | scalar gate | $\times$ | 61.27 | 80.70 | 86.89 | 42.65 | 66.55 | 76.30
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  $\oplus$=add | $\times$ | 61.74 | 80.93 | 86.97 | 43.20 | 66.60 | 76.55
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | 61.95 | 81.16 | 87.19 | 43.35 | 66.85 | 76.50
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 64.04 | 82.66 | 87.22 | 45.55 | 68.20 | 77.85

####  4.2.2. Impact of Uni-modal Sub-manifold Embedding Module (USEM)

We enumerate some variants for the Uni-modal Sub-manifold Embedding Module
(USEM) and compare them with our proposed USEM:

(1) Glo: $v^{u},\ t^{u}=v^{g},\ t^{g}$;

(2) Avg: $v^{u},\ t^{u}=avg(V),\ avg(T)$;

(3) AvgLoc + Glo: $v^{u},\ t^{u}=v^{g}+avg(V^{l}),\ t^{g}+avg(T^{l})$, where
$avg(\cdot)$ denotes the feature averaging operation.

We can observe from Tab. 2 that the performance of the model with USEM is
obviously better than the other substitutions, which indicate the
effectiveness of USEM.

Table 3. Comparison with SOTA on CUHK-PEDES. Method | Rank-1 | Rank-5 | Rank-10
---|---|---|---
CNN-RNN (Reed et al., 2016) | 8.07 | - | 32.47
Neural Talk (Vinyals et al., 2015) | 13.66 | - | 41.72
GNA-RNN (Li et al., 2017b) | 19.05 | - | 53.64
IATV (Li et al., 2017a) | 25.94 | - | 60.48
PWM-ATH (Chen et al., 2018b) | 27.14 | 49.45 | 61.02
Dual Path (Zheng et al., 2020b) | 44.40 | 66.26 | 75.07
GLA (Chen et al., 2018a) | 43.58 | 66.93 | 76.26
CMPM-CMPC (Zhang and Lu, 2018) | 49.37 | 71.69 | 79.27
MIA (Niu et al., 2020) | 53.10 | 75.00 | 82.90
A-GANet (Liu et al., 2019) | 53.14 | 74.03 | 81.95
PMA (Jing et al., 2020) | 54.12 | 75.45 | 82.97
TIMAM (Sarafianos et al., 2019) | 54.51 | 77.56 | 84.78
CMAAM (Aggarwal et al., 2020) | 56.68 | 77.18 | 84.86
AMEN (Wang et al., 2021b) | 57.16 | 78.64 | 86.22
HGAN (Zheng et al., 2020a) | 59.00 | 79.49 | 86.62
DSSL (Zhu et al., 2021) | 59.98 | 80.41 | 87.56
LBUL (Ours) | 61.95 | 81.16 | 87.19
LBUL + BERT (Ours) | 64.04 | 82.66 | 87.22
Table 4. Comparison with SOTA on RSTPReid. Method | Rank-1 | Rank-5 | Rank-10
---|---|---|---
IMG-Net (Wang et al., 2020) | 37.60 | 61.15 | 73.55
AMEN (Wang et al., 2021b) | 38.45 | 62.40 | 73.80
DSSL (Zhu et al., 2021) | 39.05 | 62.60 | 73.95
SSAN (Ding et al., 2021) | 43.50 | 67.80 | 77.15
LBUL (ours) | 43.35 | 66.85 | 76.50
LBUL + BERT (Ours) | 45.55 | 68.20 | 77.85

####  4.2.3. Impact of Leaping After Seeing Module (LASM)

How to properly take advantage of the two feature representations $x^{u}$ and
$x^{p}$ in the Leaping After Seeing Module (LASM) is crucial to the
performance of LBUL. Therefore, we conduct experiments with several
substitution variants of LASM on both the CUHK-PEDES and RSTPReid dataset to
see the effectiveness of our proposed method:

(1) Add: $x^{c}=x^{u}+x^{p}$;

(2) Add + $\mathcal{MLP}$: $x^{c}=tanh(W(x^{u}+x^{p})+b)$;

(3) Concat: $x^{c}=\left[\begin{matrix}x^{u}\\\ x^{p}\end{matrix}\right]$;

(4) Concat + $\mathcal{MLP}$: $x^{c}=tanh(W\left[\begin{matrix}x^{u}\\\
x^{p}\end{matrix}\right]+b)$;

(5) Scalar Gate: $x^{c}=ax^{u}+(1-a)x^{p}$ where $a$ is a real-valued number
parameterized by $x^{u}$ and $x^{p}$;

(6) $\oplus$=add: substitute the concatenation operation in Eq. 8 for
addition.

As can be seen from Tab. 2, the feature aggregation paradigm proposed in LASM
(i.e. Eq. 8 and Eq. 9) achieves substantially better performance than all the
basic variants, while implementing $\oplus$ in Eq. 8 as addition or
concatenation give similar retrieval accuracies, with the concatenation method
slightly better.

(a) Rank-1 Accuracy

(b) Rank-5 Accuracy

(c) Rank-10 Accuracy

Figure 3. Illustration of the impact of the starting epoch for the second
training stage on CUHK-PEDES.

####  4.2.4. Impact of Distribution Shifting Mechanism

As shown in Tab. 2, the Distribution Shifting ($\mathcal{DS}$) mechanism is a
key component in XProj, without which the performance drops by $1.71\%$,
$1.04\%$, $0.87\%$ and $2.10\%$, $1.70\%$, $0.95\%$ on CUHK-PEDES and
RSTPReid, respectively. The decrease in retrieval accuracy indicates the
significance of the proposed $\mathcal{DS}$ mechanism.

Figure 4. Illustration of top-10 text-based person retrieval results by LBUL.
The matched pedestrian images are marked by green rectangles, while the
mismatched person images are marked by red rectangles.

####  4.2.5. Impact of the Two-stage Training Strategy

As described in Sec. 3.5, a two-stage strategy is proposed to train LBUL. To
prove the validity of this optimization strategy, we conduct experiments which
train LBUL with a one-stage strategy, namely, utilize $\mathcal{L}^{Stage2}$
since the beginning. As can be seen from Tab. 2, the retrieval performance
decrease by $8.45\%$, $6.11\%$, $4.51\%$ and $8.50\%$, $6.70\%$, $5.55\%$ on
CUHK-PEDES and RSTPReid, respectively, which demonstrate that after reliable
uni-modal sub-manifolds are learned, a more consistent cross-modal common
manifold can be constructed by LBUL. To further analysis the impact of the
starting epoch for the second training stage, we conduct extensive experiments
on CUHK-PEDES and the results are illustrated in Fig. 3. It can be observed
that initially the performance keeps increasing with the growth of the
starting epoch for the second stage. Then after the value of the starting
epoch passes 15, the performance gradually stabilizes. And the performance
drop slightly after the value of the starting epoch gets too large. Note that
when the starting epoch is 0, it is just equivalent to adopting a one-stage
training strategy.

###  4.3. Comparison with SOTA on Text-based Person Retrieval

We compare the proposed LBUL with previous methods on CUHK-PEDES and RSTPReid.
It can be observed from Tab. 3 and Tab. 4 that without BERT, our proposed LBUL
achieves $61.95\%$, $81.16\%$ and $87.19\%$ of rank-1, rank-5 and rank-10
accuracies respectively on CUHK-PEDES and $43.35\%$, $66.85\%$ and $76.50\%$
on RSTPReid. By encouraging the proposed model to look before it leaps with
the proposed two-step common manifold mapping mechanism, LBUL outperforms
existing methods and achieves the state-of-the-art performance on the text-
based person retrieval task. For instance, TIMAM (Sarafianos et al., 2019) is
one of the typical CDCP-based approaches. It aims to learn modality-invariant
feature representations using adversarial and cross-modal matching objectives
and utilizes a pretrained ResNet-101 as the visual backbone. With a ResNet-50
backbone, LBUL outperforms TIMAM by $7.44\%$, $3.60\%$ and $2.38\%$ on CUHK-
PEDES of rank-$1$, rank-$5$ and rank-$10$ accuracies, respectively, which
further proves the effectiveness of our proposed method. We display some
examples of the top-$10$ text-based person retrieval results by LBUL in Fig.
4. The matched/mismatched pedestrian images are marked by green/red
rectangles.

##  5\. Conclusion

In this paper, we propose a novel algorithm termed LBUL to learn a Consistent
Cross-modal Common Manifold (C3M) for text-based person retrieval to overcome
the CDCP dilemma. The core idea of our method, just as a Chinese saying goes,
is ‘san si er hou xing’, namely, to Look Before yoU Leap (LBUL). The common
manifold mapping mechanism of LBUL includes two steps, namely, a looking step
and a leaping step. Compared to CDCP-based common manifold mapping paradigms,
LBUL considers distribution characteristics of both the visual and textual
modalities before embedding data from one certain modality into C3M to achieve
a more solid cross-modal distribution consensus, and hence achieve a superior
retrieval accuracy. We evaluate our proposed method on two text-based person
retrieval datasets CUHK-PEDES and RSTPReid. Experimental results demonstrate
that the proposed LBUL outperforms previous methods and achieves the state-of-
the-art performance.

###### Acknowledgements.

This work is partially supported by the National Natural Science Foundation of
China (Grant No. 62101245, 61972016), China Postdoctoral Science Foundation
(Grant No.2019M661999) , Natural Science Research of Jiangsu Higher Education
Institutions of China (19KJB520009) and Future Network Scientific Research
Fund Project (Grant No. FNSRFP-2021-YB-21).

## References

  * (1)
  * Aggarwal et al. (2020) Surbhi Aggarwal, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. 2020.  Text-based person search via attribute-aided matching. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2617–2625.
  * Chen et al. (2018a) Dapeng Chen, Hongsheng Li, Xihui Liu, Yantao Shen, Jing Shao, Zejian Yuan, and Xiaogang Wang. 2018a.  Improving deep visual representation for person re-identification by global and local image-language association. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 54–70.
  * Chen et al. (2018b) T. Chen, C. Xu, and J. Luo. 2018b.  Improving Text-Based Person Search by Spatial Matching and Adaptive Threshold. In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_. 1879–1887.
  * Ding et al. (2021) Zefeng Ding, Changxing Ding, Zhiyin Shao, and Dacheng Tao. 2021\.  Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification.  _arXiv preprint arXiv:2107.12666_ (2021).
  * Fan et al. (2018) Hehe Fan, Liang Zheng, Chenggang Yan, and Yi Yang. 2018\.  Unsupervised person re-identification: Clustering and fine-tuning.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ 14, 4 (2018), 1–18.
  * Fu et al. (2019) Yang Fu, Yunchao Wei, Guanshuo Wang, Yuqian Zhou, Honghui Shi, and Thomas S Huang. 2019\.  Self-similarity grouping: A simple unsupervised cross domain adaptation approach for person re-identification. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 6112–6121.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hou et al. (2019) Ruibing Hou, Bingpeng Ma, Hong Chang, Xinqian Gu, Shiguang Shan, and Xilin Chen. 2019\.  Interaction-and-aggregation network for person re-identification. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 9317–9326.
  * Jin et al. (2020) Xin Jin, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. 2020\.  Global distance-distributions separation for unsupervised person re-identification. In _European Conference on Computer Vision_. Springer, 735–751.
  * Jing et al. (2020) Ya Jing, Chenyang Si, Junbo Wang, Wei Wang, Liang Wang, and Tieniu Tan. 2020\.  Pose-guided multi-granularity attention network for text-based person search. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 11189–11196.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _3rd International Conference on Learning Representations, ICLR 2015_.
  * Li et al. (2017a) Shuang Li, Tong Xiao, Hongsheng Li, Wei Yang, and Xiaogang Wang. 2017a.  Identity-aware textual-visual matching with latent co-attention. In _Proceedings of the IEEE International Conference on Computer Vision_. 1890–1899.
  * Li et al. (2017b) Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. 2017b.  Person search with natural language description. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1970–1979.
  * Liao et al. (2015) Shengcai Liao, Yang Hu, Xiangyu Zhu, and Stan Z Li. 2015\.  Person re-identification by local maximal occurrence representation and metric learning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2197–2206.
  * Lin et al. (2018) Shan Lin, Haoliang Li, Chang-Tsun Li, and Alex Chichung Kot. 2018\.  Multi-task mid-level feature alignment network for unsupervised cross-dataset person re-identification.  _arXiv preprint arXiv:1807.01440_ (2018).
  * Liu et al. (2019) Jiawei Liu, Zheng-Jun Zha, Richang Hong, Meng Wang, and Yongdong Zhang. 2019.  Deep adversarial graph attention convolution network for text-based person search. In _Proceedings of the 27th ACM International Conference on Multimedia_. 665–673.
  * Ning et al. (2021) Xin Ning, Ke Gong, Weijun Li, and Liping Zhang. 2021\.  JWSAA: joint weak saliency and attention aware for person re-identification.  _Neurocomputing_ 453 (2021), 801–811.
  * Niu et al. (2020) Kai Niu, Yan Huang, Wanli Ouyang, and Liang Wang. 2020\.  Improving description-based person re-identification by multi-granularity image-text alignments.  _IEEE Transactions on Image Processing_ 29 (2020), 5542–5556.
  * Qian et al. (2021) Shengsheng Qian, Dizhan Xue, Quan Fang, and Changsheng Xu. 2021\.  Adaptive Label-aware Graph Convolutional Networks for Cross-Modal Retrieval.  _IEEE Transactions on Multimedia_ (2021).
  * Reed et al. (2016) Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016\.  Learning deep representations of fine-grained visual descriptions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 49–58.
  * Sarafianos et al. (2019) Nikolaos Sarafianos, Xiang Xu, and Ioannis A Kakadiaris. 2019\.  Adversarial representation learning for text-to-image matching. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5814–5824.
  * Sun et al. (2019) Changchang Sun, Xuemeng Song, Fuli Feng, Wayne Xin Zhao, Hao Zhang, and Liqiang Nie. 2019\.  Supervised hierarchical cross-modal hashing. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 725–734.
  * Vaswani et al. (2017a) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017a.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Vaswani et al. (2017b) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017b.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3156–3164.
  * Wang and Zhang (2020) Dongkai Wang and Shiliang Zhang. 2020.  Unsupervised person re-identification via multi-label classification. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10981–10990.
  * Wang et al. (2021a) Hao Wang, Doyen Sahoo, Chenghao Liu, Ke Shu, Palakorn Achananuparp, Ee-peng Lim, and CH Steven Hoi. 2021a.  Cross-modal food retrieval: learning a joint embedding of food images and recipes with semantic consistency and attention mechanism.  _IEEE Transactions on Multimedia_ (2021).
  * Wang et al. (2018) Jingya Wang, Xiatian Zhu, Shaogang Gong, and Wei Li. 2018\.  Transferable joint attribute-identity deep learning for unsupervised person re-identification. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2275–2284.
  * Wang et al. (2021b) Zijie Wang, Jingyi Xue, Aichun Zhu, Yifeng Li, Mingyi Zhang, and Chongliang Zhong. 2021b.  AMEN: Adversarial Multi-space Embedding Network for Text-Based Person Re-identification. In _Chinese Conference on Pattern Recognition and Computer Vision (PRCV)_. Springer, 462–473.
  * Wang et al. (2022) Zijie Wang, Aichun Zhu, Jingyi Xue, Daihong Jiang, Chao Liu, Yifeng Li, and Fangqiang Hu. 2022.  SUM: Serialized Updating and Matching for text-based person retrieval.  _Knowledge-Based Systems_ 248 (2022), 108891.
  * Wang et al. (2020) Zijie Wang, Aichun Zhu, Zhe Zheng, Jing Jin, Zhouxin Xue, and Gang Hua. 2020\.  IMG-Net: inner-cross-modal attentional multigranular network for description-based person re-identification.  _Journal of Electronic Imaging_ 29, 4 (2020), 043028\.
  * Wei et al. (2018) Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. 2018\.  Person transfer gan to bridge domain gap for person re-identification. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 79–88.
  * Xia et al. (2019) Bryan Ning Xia, Yuan Gong, Yizhe Zhang, and Christian Poellabauer. 2019.  Second-order non-local attention networks for person re-identification. In _Proceedings of the IEEE International Conference on Computer Vision_. 3760–3769.
  * Yi et al. (2014) Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. 2014\.  Deep metric learning for person re-identification. In _2014 22nd International Conference on Pattern Recognition_. IEEE, 34–39.
  * Yuan et al. (2020) Yuan Yuan, Jian’an Zhang, and Qi Wang. 2020.  Deep Gabor convolution network for person re-identification.  _Neurocomputing_ 378 (2020), 387–398.
  * Zhang and Lu (2018) Ying Zhang and Huchuan Lu. 2018.  Deep cross-modal projection learning for image-text matching. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 686–701.
  * Zhao et al. (2021) Mingbo Zhao, Jiao Liu, Zhao Zhang, and Jicong Fan. 2021\.  A scalable sub-graph regularization for efficient content based image retrieval with long-term relevance feedback enhancement.  _Knowledge-based systems_ 212 (2021), 106505.
  * Zheng et al. (2020a) Kecheng Zheng, Wu Liu, Jiawei Liu, Zheng-Jun Zha, and Tao Mei. 2020a.  Hierarchical Gumbel Attention Network for Text-based Person Search. In _Proceedings of the 28th ACM International Conference on Multimedia_. 3441–3449.
  * Zheng et al. (2020b) Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Mingliang Xu, and Yi-Dong Shen. 2020b.  Dual-Path Convolutional Image-Text Embeddings with Instance Loss.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ 16, 2 (2020), 1–23.
  * Zhu et al. (2021) Aichun Zhu, Zijie Wang, Yifeng Li, Xili Wan, Jing Jin, Tian Wang, Fangqiang Hu, and Gang Hua. 2021\.  DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval. In _Proceedings of the 29th ACM International Conference on Multimedia_. 209–217.


# Histopathological Image Classification and Vulnerability Analysis using
Federated Learning

Sankalp Vyas  Computing and Information Science
Anglia Ruskin University, Cambridge
<EMAIL_ADDRESS>Amar Nath Patra  Radford University
Radford, USA
<EMAIL_ADDRESS>Raj Mani Shukla  Computing and Information Science
Anglia Ruskin University, Cambridge
<EMAIL_ADDRESS>
###### Abstract

Healthcare is one of the foremost applications of machine learning (ML).
Traditionally, ML models are trained by central servers, which aggregate data
from various distributed devices to forecast the results for newly generated
data. This is a major concern as models can access sensitive user information,
which raises privacy concerns. A federated learning (FL) approach can help
address this issue: A global model sends its copy to all clients who train
these copies, and the clients send the updates (weights) back to it. Over
time, the global model improves and becomes more accurate. Data privacy is
protected during training, as it is conducted locally on the clients’ devices.

However, the global model is susceptible to data poisoning. We develop a
privacy-preserving FL technique for a skin cancer dataset and show that the
model is prone to data poisoning attacks. Ten clients train the model, but one
of them intentionally introduces flipped labels as an attack. This reduces the
accuracy of the global model. As the percentage of label flipping increases,
there is a noticeable decrease in accuracy. We use a stochastic gradient
descent optimization algorithm to find the most optimal accuracy for the
model. Although FL can protect user privacy for healthcare diagnostics, it is
also vulnerable to data poisoning, which must be addressed.

###### Index Terms:

Healthcare, Federated learning, Deep neural network, Label flipping, Data
poisoning

##  I Introduction

In machine learning (ML), models are trained that use patterns in data to make
predictions and decisions. A large amount of data is collected from multiple
sources and aggregated in one location for training and evaluation. ML has
potential in healthcare: By training models on medical data, tailored
treatment regimens can be developed, epidemic diseases can be predicted, and
trends or patterns in medical data can be identified, leading to new medicines
or cures [1, 2]. However, collecting and merging data can put the privacy of
individuals and organizations at risk, as they may contain confidential or
personal details. Additionally, transmitting massive data via the Internet can
be slow and costly.

Federated learning (FL) [3] overcomes these challenges by training local
models on devices without data exchange. Instead of actual data, the global
model is sent to these decentralized data sources and trained remotely.
Additionally, FL enables model training using data controlled by numerous
organizations, each of which may have different privacy concerns. Therefore,
organizations can still benefit from the benefits of integrated data while
protecting their privacy.

The importance of FL is especially pronounced when it comes to sensitive data,
such as healthcare records, that are available from multiple sources. For
example, automated classification of skin lesions assists dermatologists in
making faster and more accurate decisions based on ML [4]. Although ML-based
systems show a decent performance in automatically classifying skin-lesion
images, they need a huge amount of data that needs to be gathered for training
such models. Such data is available in different hospitals and at different
locations. Therefore, one of the problems encountered while developing such an
automated tool is that the skin-lesion data are available across different
hospitals. These data may not be shared between them due to privacy concerns
and regulations.

To solve this issue, we develop a privacy-preserving Machine Learning method
for the automated classification of skin lesions that employs federated
learning. The advantage of the proposed approach is that the raw patient data
need not be shared with the AI service developer, and hence patient privacy is
preserved. The proposed approach makes it easy to develop an AI model using a
large dataset that is spatially located. Although the proposed approach is
beneficial, there is a security risk of malicious or dishonest hospitals, as
they could introduce vulnerabilities into the system by sending incorrect
updates. One of the strategies that they can carry out is training their local
model after flipping the labels to diminish the effectiveness of the global
model. Therefore, we also investigate the vulnerabilities of the proposed
approach for skin lesion classification that needs to be tackled for the
practical realization of such a distributed learning system.

The rest of the paper is organized as follows. Section 2 describes previous
research based on FL and label-flipping attacks. The details of the
preprocessing, preparation of the dataset, and the setup for building the FL
model will be presented in Section 3. Furthermore, this section provides a
description of the technologies used in this research. Section 4 discusses the
implementation of the FL model and demonstrates the label-flipping attack.
Section 5 compares the outcomes with the earlier studies in this similar area.
In Section 6 the future work for this research is discussed and Section 7
concludes the work.

##  II Literature Review

This section examines the motivations and challenges of privacy-preserving ML,
the current state of privacy-preserving machine learning, focusing on FL and
distributed machine learning, and state-of-the-art skin lesion classification
algorithms.

###  II-A Motivations and Challenges for Privacy Protection in ML

There are many reasons why privacy is important in ML, but ethical concerns
are a significant factor. The use of personal data in ML models raises
concerns about possible misuse of sensitive information, including financial
or medical records [5]. This can result in adverse outcomes for individuals,
such as discrimination and loss of autonomy [6]. Maintaining privacy in ML is
crucial due to legal obligations. The European Union (EU), for instance,
enforces strict regulations, including the General Data Protection Regulation
(GDPR), which restrict the usage of personal data [7]. The purpose of these
regulations is to safeguard the privacy of individuals and grant them
authority over their personal information. Non-compliance with these
guidelines may lead to significant penalties and damage to a business’s
reputation. Despite these motivations, protecting privacy in ML presents
numerous obstacles. One difficulty is the compromise between privacy and
utility. Effective ML models require enormous volumes of data, yet the
collection and use of these data can compromise individuals’ privacy [8].
Maintaining privacy while meeting the demand for data can be a difficult task
to balance. Another challenge is to ensure privacy protection against attacks,
including membership-inference attacks. These attacks aim to determine whether
a person’s data was used in training a model [9]. The complexity of privacy-
preserving procedures is another obstacle. It can be difficult to modify ML
algorithms that operate on raw data without compromising their efficiency and
privacy protection [10]. Implementing new techniques can be a time-consuming
and resource-intensive process.

###  II-B Privacy-Preserving ML

Privacy-preserving ML pertains to the creation of algorithms and protocols
that allow the analysis of data while upholding individuals’ privacy. The use
of privacy-enhancing technologies (PETs) that protect privacy through
technical means, such as encryption and anonymization, is one way to achieve
privacy protection in ML [11]. On the other hand, differential privacy
achieves privacy preservation by adding noise to the data to obscure the
contribution of individual data points [12]. Another technique uses secure
multiparty computation (SMC), which allows for the computation of functions
over distributed data without the need to share the raw data [13]. FL is a
decentralized ML method that enables model training using distributed data
without the need to centralize or share raw data [3].

###  II-C Skin Lesion Image Classification

ML has been used to detect and classify skin lesions in a number of studies.
Gessert et al. propose a deep learning model, consisting of a convolution
neural network (CNN) and a dense neural network, for classification based on
dermoscopic images and additional patient metadata [14]. The classification
method designed by Yao et al. [15] consists of modified deep convolution
neural networks (DCNN) and a novel Multi-weighted New Loss function that
counters class imbalance, improves the accuracy of key classes, and reduces
outlier interference. Zhang et al. propose an attention residual learning
convolutional neural network (ARL-CNN) model that combines residual learning
and a novel attention learning mechanism to improve the discriminative
representation ability of DCNNs [16]. Wang et al. present a lightweight two-
stage framework, called G-DMN [17], that uses CycleGAN for data augmentation
and Dense-MobileNetV2 (DMN) for classification. Mahbod et al. develop a fully
automatic computerized classification method [18] that fuses deep features
from several robust CNNs, including AlexNet, VGG16, and ResNet-18, and
different abstraction levels.

The aforementioned research employs a centralized approach in which skin
lesions are collected from different locations, compromising user privacy. In
contrast, we propose a FL-based approach to preserve user privacy that does
not collect raw data in one place. Instead, the intelligence in the form of
trained models is shared between and combined to develop the ML model.
Additionally, we analyze the vulnerability of such a system due to the
presence of possible malicious clients.

##  III Methodology

This section describes the proposed FL-based approach along with the system
model and the data poisoning vulnerability scenario.

###  III-A System model

Figure 1: The distributed learning environment for the proposed
Histopathological image classification

Fig. 1 presents the system model of the proposed distributed learning
environment for histopathological image classification. A hospital $h[i]$
holds raw private images of its patients, which it processes for clean images.
Based on the clean dataset, $h[i]$ can train its local model $A_{i}$. The
application developer, who may represent a company providing Software as a
Service (SaaS), is at the other end and does not have access to the raw data.
However, the local models from different hospitals are shared through a secure
channel with the application developer aggregator. The developer combines
these models to create a global model $AI$ that is deployed in the cloud or on
the edge as a SaaS platform. This platform can be used by any hospital or
individual, regardless of whether they participate in model training. The
business model may allow entities participating in the training process to use
the platform for free or at a discounted price, while entities not
participating in the training process must pay full subscription rates. With
more entities participating in the training process and sharing intelligence
through their respective trained models, a better global model is developed.

As new data are continuously collected, participants in the training process
can retrain the model, using the global model’s weights as initial weights.
Over time, the SaaS platform can improve itself as it captures disparate and
varied distributions of data.

###  III-B Proposed FL-based approach

As the proposed methodology uses FL, the training dataset $D$ is not fully
available on a centralized server but on multiple participants $h[i]$, who
have their own private dataset $D_{i}$. Each $h[i]$ executes stochastic
gradient descent (SGD) locally on its own data and then sends its updated
model parameters $\theta_{r,i}$ to the aggregator. The aggregator combines the
parameters received to create global parameters $\theta_{r}$, which are
broadcast to all participants $h[i]$. This process is repeated for a fixed
number of rounds $R$ and the final model $AI_{M}$ is created using the global
parameters of the last round $\theta_{R}$.

The deep neural network (DNN) model takes the features $f_{i}$ and outputs the
predicted probabilities $p_{i}$ for each skin lesion class label $c_{i}$ in a
set of all possible class values $C$. The predicted class for a given instance
$x_{i}$ is determined by finding the class label with the highest predicted
probability. The goal of training the DNN is to find the model parameters
$\theta$ that minimize the chosen loss function $L$ using iterative SGD. In
SGD, a batch of samples $B$ is selected from the dataset $D$, and the
corresponding gradient $gB$ is computed. The model parameters $\theta$ are
then updated in the direction of the negative gradient. This process is
repeated for multiple epochs.

In the process of training a DNN, dataset $D$ with a set of features $f_{i}$
and class labels $c_{i}$ is used, along with a loss function $L$. This loss
function is given by

$L_{total}=\frac{1}{n}\sum^{n}_{i}L(\theta,x_{i})$ (1)

The proposed method uses the Federated Averaging (FedAvg) model, a distributed
machine learning algorithm designed for model training in a federated setting,
where multiple clients or devices, each with their own training data and
computational resources, train a shared model. FedAvg’s goal is to train a
model capable of making accurate predictions on the data of all clients, even
though the data of each client are only a subset of the total data [19]. The
hospitals act as the clients in the proposed method.

The overview of the FedAvg algorithm for the proposed histopathological image
classification is described below.

  * •

Initialization The algorithm begins by initializing the global model $AI$ with
random weights and biases. $AI$ is then distributed to each hospital $h[i]$
which uses it as its local model $A_{i}$.

  * •

Communication rounds The algorithm goes through a series of communication
rounds (federated learning rounds) with the following steps in each round

    * –

Local training Each $h[i]$ chooses a subset of its data $D_{i}$ (mini-batch)
to train with. Then, using the selected data, it performs one or more gradient
descent steps on its local model $A_{i}$.

    * –

Averaging Each $h[i]$ updates the model parameters $\theta_{r,i}$ and sends
them to the aggregator. The received models are then averaged together by the
aggregator to create a new global model $AI$.

    * –

Update $AI$ is then returned to the $h[i]$s to use as local models $A_{i}$s.

    * –

Repeat The process repeats until $AI$ reaches a certain level of accuracy of a
stopping criterion.

The averaging step in the FedAvg algorithm is crucial, as it updates $AI$ by
taking the average of the $A_{i}$s. This allows the algorithm to account for
differences in data across $h[i]$s, which can help improve $AI$’s
generalization performance.

###  III-C Malicious clients

Although the aforementioned approach preserves user privacy, there exists a
vulnerability if one or more $h[i]$s provide incorrect model updates. A
hospital $h[i]$ may adopt the following steps to alter the outcome of the
global model $AI$

  1. 1.

Select $k$ samples from the labeled dataset where $k$ is the $p$ percentage of
the selected labels

  2. 2.

For each $k$ replace the skin-lesion class with an alternate random skin-
lesion class

  3. 3.

Repeat step 2 for all k

  4. 4.

Train the local model $A_{i}$ on the substituted label

  5. 5.

Send the trained model to the aggregator for Fedaveraging.

##  IV Performance Evaluation and Results

In this section, we provide the details of implementation that include the FL
environment setup, DNN architecture, data collection, analysis, visualization,
dataset pre-processing, and evaluation methods employed to obtain the results.
We analyze the results before and after the flipping of the labels and do a
comparative study.

###  IV-A Implementation details

####  IV-A1 FL setup

The FL system is implemented in the TensorFlow library [20], available in
Python. $N$ represents the number of participating hospitals in the FL setup
which is set to 10, whereas there is one aggregator. Additionally, independent
and identity-distributed data (IID) are used.

####  IV-A2 Deep neural network architecture

In the setup, we use a multilayer-perceptron model (MLP) to classify images
from the dataset into one of the seven classes. The MLP model consists of
three fully connected (dense) layers with 200 units in each. Relu activation
is used in the hidden layer and softmax in the output layer.

Each hospital $h[i]$ is responsible for training its local model $A_{i}$ on
its local dataset $D_{i}$. We use the SGD optimizer to train the AI model. We
train the model for 100 communication rounds, during which the updates from
each $h[i]$ are aggregated on the central server and used to update the global
model $AI$. To measure the error of the model during training, we use the
categorical_crossentropy loss function. To evaluate the performance of the
model, we use the accuracy metric, which measures the proportion of correct
predictions made by the model.

####  IV-A3 Data collection

We use the HAM10000 dataset that consists of a total of 10,015 samples that
belong to seven different classes [21]. Fig. 2 shows the image data for all
10,015 samples and the seven classes.

Figure 2: Sample skin lesion data

####  IV-A4 Data analysis and visualization

The seven classes of skin cancer lesions in the HAM10000 dataset are
Melanocytic nevi (nv), Melanoma (mel), Benign keratosis-like lesions (bkl),
Basal cell carcinoma (bcc), Actinic keratoses (akiec), Vascular lesions
(vasc), and Dermatofibroma (df). Fig. 3 shows some of the samples of the seven
different types of lesions.

Figure 3: Types of skin lesions and their samples

####  IV-A5 Pre-processing of the dataset

In the pre-processing stage, each pixel in the image is represented by a
single greyscale value (ranging from 0 to 255). When these images are stored
in a numerical array, they will have 784 elements (28x28), each representing
the greyscale value of a single pixel. Thus, there are a total of 784 columns
and 10,015 rows. Table I shows the image columns containing values between 0
and 1. The images have been converted to a 1D array as we use MLP to build our
FL model. Pixel000, pixel001…, pixel0784, etc. represent the number of pixels
while 0,1,2,3,4…, 10,014 indicate the total number of samples.

TABLE I: Image data after getting scaled between 0 and 1 pixel0000 |  pixel0001 |  pixel0002 |  pixel0003 |  pixel0004 |  pixel0005 |  pixel0006 |  pixel0007 |  pixel0008 |  pixel0009
---|---|---|---|---|---|---|---|---|---
0.674510 |  0.670588 |  0.678431 |  0.686275 |  0.643137 |  0.733333 |  0.811765 |  0.823529 |  0.815686 |  0.807843
0.007843 |  0.133333 |  0.423529 |  0.454902 |  0.447059 |  0.466667 |  0.513725 |  0.545098 |  0.545098 |  0.568627
0.478431 |  0.603922 |  0.635294 |  0.666667 |  0.701961 |  0.772549 |  0.784314 |  0.764706 |  0.792157 |  0.780392
0.537255 |  0.560784 |  0.552941 |  0.545098 |  0.576471 |  0.596078 |  0.607843 |  0.596078 |  0.607843 |  0.623529
0.584314 |  0.615686 |  0.615686 |  0.596078 |  0.607843 |  0.596078 |  0.603922 |  0.639216 |  0.584314 |  0.568627

####  IV-A6 Evaluation methods

The metrics used for evaluation are precision, recall, f1-scores, and accuracy
(Table II). Precision is the number of true positive predictions divided by
the number of true positive plus false positive predictions. The number of
true positive predictions divided by the number of true positive plus false
negative predictions is recall. Accuracy is the percentage of correct
predictions made by the model out of all the predictions made by the model,
while the F1 score is the harmonic mean of precision and recall.

TABLE II: Evaluation Metrics Evaluation Metrics | Formula
---|---
Recall | $\frac{TruePositive(TP)}{TruePositive(TP)+FalseNegative(FN)}$
Precision | $\frac{TruePositive(TP)}{TruePositive(TP)+FalsePositive(FP)}$
F1-score | $2\times\frac{Precision\times Recall}{Precision+Recall}$
Accuracy | $\frac{TruePositive(TP)+TrueNegative(TN)}{Totalnumberofpredictions}$

###  IV-B Results

We discuss the results in two parts, before and after the flipping of the
labels.

####  IV-B1 Results before flipping of the labels

The parameters taken into consideration for building the FL model are shown in
Table III while Fig. IV shows the evaluation metrics for the model.

TABLE III: Variation of k with p Parameters | Value
---|---
learning_rate | 0.01
comms_round | 100
Input shape | (784,)
Number of classes | 7
Bs (batch size) | 32
num_clients | 10
loss | categorical_crossentropy
metrics | accuracy
momentum | 0.9
TABLE IV: Classification report of the global model in FL setup before the flipping of the labels |  precision |  recall |  f1-score |  support
---|---|---|---|---
0 |  0.00 |  0.00 |  0.00 |  61
1 |  0.33 |  0.01 |  0.02 |  96
2 |  0.30 |  0.08 |  0.13 |  228
3 |  0.00 |  0.00 |  0.00 |  37
4 |  0.69 |  0.99 |  0.81 |  1327
5 |  0.00 |  0.00 |  0.00 |  32
6 |  0.57 |  0.07 |  0.13 |  222
macro avg |  0.27 |  0.16 |  0.16 |  2003
weighted avg |  0.57 |  0.67 |  0.57 |  2003
accuracy |  0.67 |  2003

Fig. IV shows the classification report for a multiclass classification
problem, where the model is trying to predict one of the 7 different classes
(0-6). The report provides precision, recall, f1 score, and support for each
class. High precision implies that the model mostly predicts the correct class
while a high recall means that the model can identify most of the positive
instances. As the f1-score is a weighted harmonic mean of precision and
recall, it balances precision and recall and is thus particularly useful when
the classes are imbalanced. Support is the number of instances that belong to
each class. The overall accuracy of the model is 67.099%. However, the
accuracy can be misleading, especially when the classes are imbalanced. It’s
important to look at the performance of each class separately. The model
correctly identifies class 4 with a high precision of 0.69 and a recall of
0.99, which is quite good. The f1-score is 0.81 for class 4. However, the
model is not performing well on the $5^{th}$ as well as the $0^{th}$ class as
the precision, recall, and f1-score are all zero. This is due to the small
number of instances in those classes, which makes it difficult for the model
to learn to predict them correctly. This same model was also tested on a SGD
optimizer without using FL. To ensure a level playing ground, each of the
hyperparameters was retained during the FL training, with the exception of the
batch size. Instead of 32, the batch size in this instance was set to 320.
With this configuration, the SGD will see the same number of training samples
in each iteration as the global model $AI$ did for each communication round in
FL with this setup. Fig. V shows the detailed report for the centralized
setup.

TABLE V: Classification report for the centralized configuration |  precision |  recall |  f1-score |  support
---|---|---|---|---
0 |  0.00 |  0.00 |  0.00 |  61
1 |  0.20 |  0.01 |  0.02 |  96
2 |  0.32 |  0.06 |  0.10 |  228
3 |  0.00 |  0.00 |  0.00 |  37
4 |  0.68 |  0.99 |  0.81 |  1327
5 |  0.00 |  0.00 |  0.00 |  32
6 |  0.53 |  0.08 |  0.14 |  222
macro avg |  0.25 |  0.16 |  0.15 |  2003
weighted avg |  0.56 |  0.67 |  0.56 |  2003
accuracy |  0.67 |  2003

The accuracy for the centralized ML is 67.099% while the global loss is 1.55
(Fig. IV. In this configuration, the $4^{th}$ class has a recall of 0.99 and a
f1-score of 0.81. The precision is 0.68. The model does not perform well on
the $0^{th}$ class as well as the $5^{th}$ class due to a smaller number of
samples for both of these classes.

####  IV-B2 Results after flipping of labels

The same model was run again after a certain percentage of labels had been
randomly altered. 2% of the labels were changed in the beginning, and this
process continued until 20% of the labels were changed for the malicious
client. The accuracy, precision, recall, and f1 scores were recorded after
changing the labels from 2% to 20%. Table VI shows that the classification
report changes when a certain percentage of labels are flipped. For example,
when 14% labels are changed, there is a decrease in accuracy from 67. 099% to
66. 101%.

TABLE VI: Classification report of the global model in FL setup after flipping 14% of labels |  precision |  recall |  f1-score |  support
---|---|---|---|---
0 |  0.00 |  0.00 |  0.00 |  61
1 |  0.00 |  0.00 |  0.00 |  96
2 |  0.00 |  0.00 |  0.00 |  228
3 |  0.00 |  0.00 |  0.00 |  37
4 |  0.66 |  0.99 |  0.80 |  1327
5 |  0.00 |  0.00 |  0.00 |  32
6 |  0.75 |  0.03 |  0.05 |  222
macro avg |  0.20 |  0.15 |  0.12 |  2003
weighted avg |  0.52 |  0.66 |  0.53 |  2003
accuracy |  0.66 |  2003

The $0^{th}$,$1^{st}$,$2^{nd}$,$3^{rd}$, and $5^{th}$ classes had a precision,
recall, and f1-score of 0.00 while the $4^{th}$ class had a precision of 0.66,
a recall of 0.99 and a f1-score of 0.80. The $6^{th}$ class had a precision of
0.75, a recall of 0.03, and a f1-score of 0.05. Table VII shows the changes in
accuracy before the flipping of the labels and after the flipping of the
labels. The table reveals that for most cases there is a drop in accuracy when
a malicious client randomly flips the label.

TABLE VII: Change in accuracy Percentage of labels flipped |  Global accuracy |  Global accuracy after label flipping |  Centralized setup accuracy |  Centralized setup accuracy after label flipping
---|---|---|---|---
2% |  67.099 |  67.049 |  67.499 |  67.399
4% |  67.099 |  67.099 |  67.499 |  67.149
6% |  67.099 |  66.600 |  67.499 |  67.049
8% |  67.099 |  66.600 |  67.499 |  67.000
10% |  67.099 |  66.350 |  67.499 |  66.800
12% |  67.099 |  66.151 |  67.499 |  66.400
14% |  67.099 |  66.101 |  67.499 |  66.400
16% |  67.099 |  66.051 |  67.499 |  66.301
18% |  67.099 |  66.900 |  67.499 |  67.149
20% |  67.099 |  67.249 |  67.499 |  67.299

##  V Conclusion

In this paper, we built a privacy-preserving federated learning model and
showed the vulnerability of this model by attacking it through a malicious
client. We demonstrated that this malicious client changed a certain number of
samples which led to a decrease in the accuracy of the global model. We
observed that for most of the cases, there is a decrease in the performance as
well as the accuracy of the model. We also compared the results to a
Stochastic gradient descent optimizer in a centralized learning setup.
Moreover, we also compared the results before and after changing the labels
and found a decrease in the performance and the accuracy of the model. In the
future, rather than randomly flipping the label, optimization methods to
maximally affect the accuracy could be analyzed.

## References

  * [1] A. Alanazi, “Using machine learning for healthcare challenges and opportunities,” Informatics in Medicine Unlocked, vol. 30, p. 100924, 2022\.
  * [2] A. Qayyum, J. Qadir, M. Bilal, and A. Al-Fuqaha, “Secure and robust machine learning for healthcare: A survey,” IEEE Reviews in Biomedical Engineering, vol. 14, pp. 156–180, 2020.
  * [3] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics, pp. 1273–1282, PMLR, 2017.
  * [4] R. A. Mehr and A. Ameri, “Skin cancer detection based on deep learning,” Journal of Biomedical Physics & Engineering, vol. 12, no. 6, p. 559, 2022.
  * [5] C. Dwork, A. Roth, et al., “The algorithmic foundations of differential privacy,” Foundations and Trends® in Theoretical Computer Science, vol. 9, no. 3–4, pp. 211–407, 2014.
  * [6] S. Barocas and A. D. Selbst, “Big data’s disparate impact,” California law review, pp. 671–732, 2016.
  * [7] “General Data Protection Regulation.” https://shorturl.at/EMPRX.
  * [8] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in 2017 IEEE symposium on security and privacy (SP), pp. 3–18, IEEE, 2017.
  * [9] H. Hu, Z. Salcic, L. Sun, G. Dobbie, P. S. Yu, and X. Zhang, “Membership inference attacks on machine learning: A survey,” ACM Computing Surveys (CSUR), vol. 54, no. 11s, pp. 1–37, 2022.
  * [10] K. Chaudhuri, C. Monteleoni, and A. D. Sarwate, “Differentially private empirical risk minimization.,” Journal of Machine Learning Research, vol. 12, no. 3, 2011.
  * [11] E. U. Soykan, L. Karaçay, F. Karakoç, and E. Tomur, “A survey and guideline on privacy enhancing technologies for collaborative machine learning,” IEEE Access, vol. 10, pp. 97495–97519, 2022.
  * [12] C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006\. Proceedings 3, pp. 265–284, Springer, 2006.
  * [13] A. C. Yao, “Protocols for secure computations,” in 23rd annual symposium on foundations of computer science (sfcs 1982), pp. 160–164, IEEE, 1982.
  * [14] N. Gessert, M. Nielsen, M. Shaikh, R. Werner, and A. Schlaefer, “Skin lesion classification using ensembles of multi-resolution efficientnets with meta data,” MethodsX, vol. 7, p. 100864, 2020.
  * [15] P. Yao, S. Shen, M. Xu, P. Liu, F. Zhang, J. Xing, P. Shao, B. Kaffenberger, and R. X. Xu, “Single model deep learning on imbalanced small datasets for skin lesion classification,” IEEE transactions on medical imaging, vol. 41, no. 5, pp. 1242–1254, 2021.
  * [16] J. Zhang, Y. Xie, Y. Xia, and C. Shen, “Attention residual learning for skin lesion classification,” IEEE transactions on medical imaging, vol. 38, no. 9, pp. 2092–2103, 2019.
  * [17] H. Wang, Q. Qi, W. Sun, X. Li, B. Dong, and C. Yao, “Classification of skin lesions with generative adversarial networks and improved mobilenetv2,” International Journal of Imaging Systems and Technology, 2023.
  * [18] A. Mahbod, G. Schaefer, C. Wang, R. Ecker, and I. Ellinge, “Skin lesion classification using hybrid deep neural networks,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1229–1233, IEEE, 2019.
  * [19] V. Tolpegin, S. Truex, M. E. Gursoy, and L. Liu, “Data poisoning attacks against federated learning systems,” in Computer Security–ESORICS 2020: 25th European Symposium on Research in Computer Security, ESORICS 2020, Guildford, UK, September 14–18, 2020, Proceedings, Part I 25, pp. 480–501, Springer, 2020.
  * [20] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015.  Software available from tensorflow.org.
  * [21] P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” Scientific data, vol. 5, no. 1, pp. 1–9, 2018.