# Beyond Appearance: a Semantic Controllable Self-Supervised Learning
Framework for Human-Centric Visual Tasks

Weihua Chen, Xianzhe Xu, Jian Jia, Hao luo, Yaohua Wang, Fan Wang, Rong Jin,
Xiuyu Sun222Corresponding Author
Alibaba Group
{kugang.cwh, xianzhe.xxz, jj359864, michuan.lh, xiachen.wyh, fan.w,
jinrong.jr,
<EMAIL_ADDRESS>
###### Abstract

Human-centric visual tasks have attracted increasing research attention due to
their widespread applications. In this paper, we aim to learn a general human
representation from massive unlabeled human images which can benefit
downstream human-centric tasks to the maximum extent. We call this method
SOLIDER, a Semantic cOntrollable seLf-supervIseD lEaRning framework. Unlike
the existing self-supervised learning methods, prior knowledge from human
images is utilized in SOLIDER to build pseudo semantic labels and import more
semantic information into the learned representation. Meanwhile, we note that
different downstream tasks always require different ratios of semantic
information and appearance information. For example, human parsing requires
more semantic information, while person re-identification needs more
appearance information for identification purpose. So a single learned
representation cannot fit for all requirements. To solve this problem, SOLIDER
introduces a conditional network with a semantic controller. After the model
is trained, users can send values to the controller to produce representations
with different ratios of semantic information, which can fit different needs
of downstream tasks. Finally, SOLIDER is verified on six downstream human-
centric visual tasks. It outperforms state of the arts and builds new
baselines for these tasks. The code is released in
https://github.com/tinyvision/SOLIDER.

##  1 Introduction

Human-centric visual analysis plays an important role in widespread
applications, such as surveillance, sports, augmented reality, and video
production. Person re-identification [82, 41, 13, 14], attribute recognition
[72, 78], person search [76, 92], pedestrian detection [24, 3, 32], human
parsing [53, 27], and pose estimation [58, 90] have achieved considerable
progress in recent years. In another aspect, there are massive human images
available in the current computer vision community. For example, even an
unlabeled person re-identification dataset, LUPerson [25, 26]
(#Img$\approx$4.18M) is 4 time larger than the ImageNet dataset
(#Img$\approx$1M). How to use unlabeled data to build a human representation
is challenging, especially when it needs to benefit various downstream tasks.

Figure 1: A representation space learned by DINO [6]. Seven human images are
represented in seven different colors. Each image is split into four parts
according to their semantic regions, _i.e_., upper body (as $\blacktriangle$),
lower body (as +), shoes (as $\star$) and background (as $\times$, not
visualized to avoid distraction). It can be seen that different parts of a
same person are closer to each other even they share different semantic
meanings.

Self-supervised learning has achieved great developments by using unlabeled
data to learn representations. Many pretext tasks have been designed, such as
contrastive learning [6, 35, 10] and masking image modeling [34, 77, 95, 2].
Although these methods have achieved great success in learning general image
representations, there is a lack of specific design targeting human-centric
tasks.

Some researchers [55, 97, 81] focus to extend self-supervised learning methods
on human-centric visual tasks. They use DINO [6] with LUPerson [25, 26]
dataset to build pre-trained models for person re-identification task. When
applying the pre-trained models to other human-centric tasks, such as human
parsing and pedestrian detection, we usually get sub-optimal results. It is
due to the lack of semantic information in their learned representations. As
shown in Fig. 1, in the representation space learned by DINO [6]111MAE [34]
shares a similar phenomenon as DINO [6]., different parts of a same person are
gathered together due to their appearance continuity, no matter what semantic
meanings they have.

As we’ve known, semantic information is as important as appearance information
for human-centric visual tasks [50, 96, 40]. Therefore, we tend to train the
representation with more semantic information to extend the representation to
different downstream human-centric visual tasks. In this paper, a Semantic
cOntrollable seLf-supervIseD lEaRning framework (SOLIDER) is proposed. In
SOLIDER, we take advantage of prior knowledge from human images to discover
semantic information, which can produce pseudo semantic labels for every
token. And a token-level semantic classification pretext task is imported and
supervised by these pseudo labels. With the new pretext task, we can train
representation with stronger semantic information.

During the usage of our trained representation on downstream tasks, we find
that even though semantic information and appearance information are both
important, different downstream tasks require different ratios of them.
Adjusting their ratio in the representation would lead to a better performance
in downstream tasks. However, as long as the pretext task is trained, the
representation can not be changed in current self-supervised learning methods.
Different from previous methods, we design SOLIDER as a conditional network
involving a semantic controller. The controller takes a value as input and
produces a latent representation. In the usage of the pre-trained model from
SOLIDER, we send a value (indicting the ratio of semantic information in the
representation) to the controller which can adjust the model and output a
representation with the required ratio.

In summary, our paper makes four contributions:

1) A general human representation is learned in this paper, which is used as a
better pre-trained model benefiting to downstream human-centric visual tasks.

2) A semantic controllable self-supervised learning framework (SOLIDER) is
proposed. It takes advantages of prior knowledge in human images to produce
pseudo semantic labels, and utilize it to train the human representation with
more semantic information.

3) A semantic controller is designed in SOLIDER. With the controller, the pre-
trained model can generate representations with various degrees of semantic
information that can meet different needs of downstream tasks.

4) The effectiveness of the SOLIDER representation is verified on six
downstream human-centric tasks. We believe this paper can promote the
development of these human-centric tasks in computer vision community.

Figure 2: The pipeline of the proposed SOLIDER.

##  2 Related Work

###  2.1 Self-supervised learning approaches

Self-supervised learning approaches have attracted significant interests in
computer vision, especially when used for learning image representation as the
pretext task.

In current self-supervised learning, contrastive methods [35, 15, 5, 10, 95,
6] have achieved great success, and provided state-of-the-art performance for
image representation. The goal of contrastive learning is to minimize the
distances between two augmented views of a same image [5, 10], and distinguish
each image from all the others [35, 15, 6]. MoCo [35, 15] improves the
training of contrastive methods by storing representations from a momentum
encoder instead of the trained network. SimCLR [10] shows that the memory bank
can be entirely replaced with the elements from the same batch if the batch is
large enough. DINO [6] combines most of these techniques, including momentum
encoder [35], multi-crop training [5], and the use of small patches with ViTs
[22], which builds a much better baseline.

Besides contrastive methods, masked image modeling methods [34, 77, 2] arouse
extensive attention from researchers. BEiT [2] proposes to predict discrete
tokens. SimMIM [77] and MAE [34] find that a moderately large masked size and
a light prediction head would be beneficial for the masked image modeling.
Although masked language modeling manages to import semantic information into
image representations, it can not explicitly figure out the semantic
information from the image to supervise the training. In the proposed SOLIDER,
we cluster tokens and use human prior to assign semantic labels to these
tokens, which can train a stronger human semantic representation.

It is worth noting that DeepCluster [4] also use cluster to learn
representations. But it is on image-level and can not produce semantic labels
for tokens. Some unsupervised semantic segmentation methods [18] extend
DeepCluster on pixel-level. But it still faces problems and can not learn a
satisfied representation, which is explained in Section. 3.1

###  2.2 Human-centric visual tasks

By looking through human-centric visual tasks in computer vision, we find that
there are many tasks directly or indirectly related to human222In this paper,
we focus on feature representations for images, therefore, only downstream
tasks on 2D image-level are our main concerns, and tasks about 3D data or
sequential data are out of our scope., _e.g_., person re-identification [82,
29, 37, 23, 28], attribute recognition [72, 78, 51], person search [76, 92],
pedestrian detection [24, 3, 32], multi-object tracking [12, 11, 49, 48, 67],
human parsing [53, 27] and pose estimation [58, 90].

Among them, we sort out six representative tasks, _i.e_., person re-
identification, attribute recognition, person search, pedestrian detection,
human parsing and pose estimation. Person re-identification [39, 89, 8, 69,
59] aims at retrieving a person of interest across multiple non-overlapping
cameras. Attribute recognition [72, 20, 42, 51] tries to mine the attributes
of target people when given person images, of which the attributes are
understandable semantic descriptions. Person search [76, 92] aims to find a
probe person from the whole scene which shows great significance in video
surveillance community to track lost people. Pedestrian detection [24, 32, 86,
21] focuses on detecting people in general images, which is a key ability for
a variety of important applications. Human parsing [53, 27, 43], a sub-task of
semantic segmentation, aims to understand human-body parts on pixel level.
Human pose estimation [58, 1, 83] dedicates to locate the human body skeletons
from images.

Recently, Wang et al. [84] also focus on human-centric visual tasks and design
a Token Clustering Transformer (TCFormer), but TCFormer is a supervised method
which can not take advantages of massive public unlabeled data.

##  3 The SOLIDER

The whole pipeline of the proposed SOLIDER is shown in Fig. 2. In this
section, we first explain how to generate pseudo semantic labels from human
prior knowledge and use it to supervise a token-level semantic classification
pretext task. Then, we introduce how the learned representation can be
controlled in SOLIDER.

###  3.1 Semantic Supervision from Human Prior

DINO [6] is a state-of-the-art self-supervised learning method, which is
widely used for image representation. As a constrastive learning based method,
the visual appearance information is well learned in DINO’s representation. We
use DINO as our baseline and plan to involve more semantic information into
its representation.

We cluster the token vectors from the learned DINO representation and show in
Fig. 3(b). It can be seen that the representation is split into several parts
based on its visual appearance. In other words, it can find things in images,
although be not able to tell their meanings.

Some methods [4, 18] go further and try to assign semantic labels for these
clustered things. Instead of cluster on single image, they do the clustering
across images and aim to build the semantic relationship among images.
However, as the features used for clustering are trained from visual
appearance clue, the clustered results are dominated by appearance. As shown
in Fig. 1, in original DINO space, the features of “blue short” (+) is closer
to “blue shirt” ($\blacktriangle$) due to similar appearance, but farther to
“black short” (+), even though they share the same semantic meaning of “short
pants”. After clustering, the blue things (“blue short” and “blue shirt”) will
be clustered together, and the things with the same semantic meaning (“blue
short” and “black short”) are split apart. In other words, these methods can
not produce the semantic labels we wanted.

After scanning through the unlabeled human images, we find that most of images
have a fixed pattern [64, 96]: the person body erectly occupies the entire
image, and the head is always on the top of the image while the feet is at the
bottom of the image. With this observation, we give semantic labels to
clustered parts of all images based on the order of their y-axis coordinates,
_i.e_., the top part of all images are labeled as upper body, and the bottom
part is marked as shoes. These pseudo semantic labels provide semantic
information for every token vector. A token-level semantic classification
pretext task is imported and supervised by these semantic labels. To better
use these labels, we involve extra modifications, listed as following.

Involving Background&Foreground Clustering. In these labels, we find that
there are some background fragments which cause noise disturbance on the
clustering results and mislead the alignment. To handle this problem, we
introduce another clustering before the semantic clustering. Specifically, we
observe that the background tokens always have smaller responses compared to
foreground tokens during training [96, 57], as shown in Fig. 3(c). Thus, we
cluster the token vectors into two categories based on their vector
magnitudes, _i.e_., foreground and background. The results are shown in Fig.
3(d). Then the semantic clustering is only performed on the foreground tokens,
and the new results are listed in Fig. 3(e). It can be clearly seen that the
foreground tokens are well clustered into three semantic parts, _i.e_., upper
body, lower body, shoes333In this paper, we treat heads/chests/arms as upper
body, waist/thighs as lower body and calves/feet as shoes..

Figure 3: (a) The original images. (b) Initial semantic clustering results.
(c) Attention maps. (d) Foreground clustering results. (e) Final semantic
clustering results.

Involving Masked Image Modeling. Inspired by the masked image modeling methods
[34, 77], we would like to further introduce more semantic information into
the representation. As we have known, people can easily locate every semantic
part in a human image even when some parts are occluded. Therefore, we assume
that if a semantic part is missing or occluded in the human image, the model
would still be able to predict its semantic meaning based on other surrounding
parts. For this purpose, we upgrade our semantic supervision into masked
semantic supervision. Specifically, we randomly mask out a semantic part from
the image $x$ and re-feed this masked image $\tilde{x}$ through the framework.
Then the output tokens are supervised by the original semantic labels, because
it is expected that the model would be able to provide the true semantic
labels for the masked tokens with the help of other tokens.

The whole semantic self-supervision is presented in Fig. 2. Specifically,
during training, for every iteration, we obtain the output feature maps $F$
from backbone with the size of ($n,c,h,w$). For each image $x$, we consider
its feature maps444The feature maps $F_{t}$ from the teacher network is chosen
for semantic clustering. The feature maps from the student network is marked
as $F_{s}$. $F_{t}$ as $w*h$ token vectors with the size of $c$. Then we use
K-means [31] to cluster them into two categories according to the magnitude
(_i.e_., l2 normalization) of $c$, and consider the category with larger
magnitude as foreground mask $M$. After that, another K-means is applied on
the tokens in foreground mask $M$ to conduct the semantic clustering, to
obtain $N$ predefined semantic categories and assign semantic label $y$ for
each token. Meanwhile, a semantic head555The semantic head contains several
blocks, and each block includes a fully connected layer, a batch norm layer
and a ReLU.  is involved to classify the vectors $F_{s}$ from the student
branch based on these semantic labels $y$. The corresponding semantic
classification loss is as below:

$L_{sm}=\frac{1}{w\times h}\sum_{u\in w\atop v\in
h}\sum_{i=1}^{N+1}-y^{(u,v)}\log\frac{f_{s}(u,v)^{(i)}}{\sum_{k=1}^{N+1}f_{s}(u,v)^{(k)}}$
(1)

where $f_{s}\\!=\\!h_{sm}(flatten(F_{s}))$. $flatten()$ is used to reshape
$F_{s}$ from ($n,c,h,w$) to ($n*h*w,c$), and $h_{sm}$ is the semantic head.
$N$ means the number of clustered semantic parts. $f_{s}(u,v)^{(i)}$ indicates
the predicted probability of token $(u,v)$ on part $i$. After image $x$, we
randomly mask out a part of $x$ to obtain image $\tilde{x}$, and send
$\tilde{x}$ to Eq. 1 too.

The total loss for the whole SOLIDER framework is:

$L=\alpha L_{dino}+(1-\alpha)L_{sm}$ (2)

where $\alpha$ is a balance weight and set to 0.5 by experience.

###  3.2 Semantic Controller

When the learned representation is used as the pre-trained model for
downstream tasks, we expect the appearance and semantic information learned in
the representation can help downstream tasks. However, each downstream task
has its own focus. For example, in person re-identification [82, 41], although
the semantic information can help to align parts across person images [62,
88], the appearance information is the key clue to distinguish different
people, which is the most important information [80, 94]. So a pre-trained
model with more appearance information and less semantic information would
provide a better startup for person re-identification task. On the contrary,
in pedestrian detection [24, 3] and human parsing [53, 27, 43], the semantic
information plays a key role for these tasks, while the appearance difference
is useless and should be partly ignored. Its pre-trained model is expected to
contain enough semantic information.

To fit the needs of different tasks, it requires the pertained model can be
adjusted according to the downstream tasks. However, as the pertained model is
trained, it is hard to change its parameters for different downstream tasks.
Task token [54] is a potential way to solve this problem, which pre-sets an
extra one-hot token for each task. But it has several problems. First, the
number of task tokens should be pre-defined before learning representation. In
real applications, we can not determine what tasks the learned representation
would be utilized on in advance. Second, although we can pre-defined task
tokens from semantic perspective (two task tokens, _i.e_., with and without
semantic information), the task token is a discrete variable, which leaves the
downstream tasks with limited choice of representations. Third, the task token
is essentially a re-weighted sum of original learned tokens. It is hard to
cooperate with Swin [52] backbone, which is a more friendly transformer
backbone for visual tasks, because of the shifted windows in Swin [52].

In this paper, SOLIDER uses a conditional network with a semantic controller
to handle this problem. During the pretext task training, due to the
unavailability of downstream task, we can not adjust the representation for
specific task, so we import the semantic controller to make the pre-trained
model conditional. The inputs of the semantic controller are the image feature
maps and a continuous value $\lambda\in[0,1]$. The $\lambda$ represents the
required ratio of semantic information in representation. The output is the
new feature maps with the ratio we required. The details of the proposed
semantic controller can be seen in Fig. 2. In semantic controller the value
$\lambda$ is encoded into a weight vector and a bias vector. After a Softplus
activation function, the weight vector is multiplied onto the original feature
maps, and the bias vector is added for final outputs.

We apply the semantic controller after each block of Swin Backbone, and the
feature maps $F(\lambda)$ with new ratio $\lambda$ is sent to the next block.
We use the following equation to produce our semantic controllable model:

$L=\alpha L_{dino}(F(\lambda))+\lambda(1-\alpha)L_{sm}(F(\lambda))$ (3)

In the pretext task training stage, $\lambda$ is randomly sampled for every
iteration. We tried different distributions of $\lambda$, i.e., binomial
distribution $B(p\\!=\\!0.5)$, continuous uniform distribution $U[0,1]$ and
beta distribution $\beta(0.2,0.2)$. $\beta(0.2,0.2)$ is better than $U[0,1]$,
implying that emphasizing the sampling on two borders is more important for
training the controller, which is consistent with the conclusions of Mixup
[85]. Finally, $B(p\\!\\!=\\!\\!0.5)$ is found to perform the best, a.k.a., a
binary distribution from {0,1}. When the pre-trained model is applied to
downstream tasks, $\lambda$ is set manually to adapt the pre-trained model to
each downstream task. The pre-trained model with $\lambda$ provides a better
startup for downstream fine-tuning, and all parameters of the pre-trained
model are fine-tuned.

In sum, after a conditional pre-trained model is trained from SOLIDER, we can
send different $\lambda$ to the model to produce representations with
different ratios of semantic information for downstream tasks. A training
pseudo code is provided in Supplementary Material to further clarify the whole
training process of SOLIDER.

##  4 Experiments

The pre-trained model from SOLIDER is verified on six downstream human-centric
visual tasks, including person re-identification, attribute recognition,
person search, pedestrian detection, human parsing and pose estimation.

Figure 4: The representation space learned before and after involving SOLIDER.
For each image, we split its features into four parts according to their
semantic regions, _i.e_., upper body (as $\blacktriangle$), lower body (as +),
shoes (as $\star$) and background (as $\times$, not shown for a clearer
presentation). It can be seen that before SOLIDER is introduced, the features
tend to gather by appearance. While after SOLIDER, the features with same
semantic meanings are closer to each other.

Figure 5: (a) The intra-image and intra-image distances under different
$\lambda$. (b)The performance of person re-identification and pedestrian
detection under different $\lambda$. The dash line indicates two distances and
two task results from the original DINO model.

.

###  4.1 Training Settings

####  4.1.1 Datasets

For pretext tasks, LUPerson [25, 26] is used for training, the same as [55,
97]. It contains 4.18M human images without any label. From each downstream
task, we conduct experiments on their commonly-used datasets. Specifically, in
person re-identification, the experiments are conducted on Market1501 [91] and
MSMT17 [74]. In attribute recognition, PETAzs [38], RAPzs [38] and PA100k [51]
are considered. In person search, we adopt CUHK-SYSU [76] and PRW [92] in our
experiments. CityPerson [86] is utilized for pedestrian detection. In human
parsing and pose estimation, LIP [27] and COCO [47] pose estimation are used
respectively.

####  4.1.2 Evaluation Metric

In person re-identification and person search, mAP/Rank1 [91] are adopted as
evaluation metrics. For attribute recognition, the evaluation metrics are mean
accuracy (mA) [38]. To evaluate the performance on pedestrian detection, we
employ the log-average Miss Rate over false positive on Reasonable and Heavy
Occluded (MR-2 on R/HO) [32]. Human parsing uses mIoU [43] for evaluation, and
pose estimation utilizes Average Precision/Recall (AP/AR) [83] as the
evaluation metric.

####  4.1.3 Optimization

We use Swin-Transformer [52] as the backbone throughout all the experiments.
In the pretext task training, SGD is used as the optimizer, and the learning
rate is 0.0005. The model is trained for 100 epochs, and the learning rate is
declined by Cosine Annealing scheduler. The batch size is 48/32/24 when
training on 8 Tesla V100 32G GPUs for Swin-Tiny/Small/Base.

It is worth noting that, as the semantic clustering process is time-consuming,
we do not directly train SOLIDER with 100 epochs from scratch. Instead, we
first train DINO with 100 epochs, and then finetune the SOLIDER on the trained
DINO model with another 10 epochs using a smaller learning rate 0.00005.

For downstream tasks, we reproduce a state-of-the-art method in each task as
our baseline. The methods we used as baseline are listed in Table. 1. What we
do is just replacing their backbones to the Swin-Transformer [52] backbone
which is pre-trained by the proposed SOLIDER.

###  4.2 Qualitative and Quantitative Analysis

Analysis on semantic clustering. To observe the semantic representation
ability of our pre-trained model in qualitative analysis, we randomly select
some images from the training data, and visualize their representation
features learned before and after SOLIDER. The visualization is provided in
Fig. 4. It can be found that, before introducing the semantic supervision, the
representation features are distributed mainly based on the identities of the
images. And the images sharing similar appearance stay closer, even with
different semantic meanings. For example, the features of “blue short” (+) is
closer to “blue shirt” ($\blacktriangle$) due to similar appearance “color
blue”, but further to “black short” (+), even they share the same semantic
meanings “short pants”.

However, after involving the semantic supervision by SOLIDER, the feature
distribution pays more attention on the semantic meaning. We can find that the
images hold similar semantic meanings are closer to each other, even though
they share different appearance. It implies that the human prior discovered by
our SOLIDER can provide a better semantic supervision for representation,
which helps the pre-trained model involves more semantic meanings.

Table 1: The performance of different pre-trained models on downstream tasks.
“Sup” implies the supervised training. “+Clustering” indicate the training
with semantic supervision. “+Control” means the semantic controller is
involved. For each downstream task, we list its name, evaluation metric and
the state of the art that we used as our baseline. $\uparrow$/$\downarrow$
means the larger/smaller value the better performance.

Pretrain Methods | Sup | DINO [6] |  \+ Clustering |  \+ Clustering&Controller
---|---|---|---|---
Pretrain Data | ImageNet | LUP1M | LUP1M | LUP1M
Person Re-identification mAP/Rank1 $\uparrow$ TransReID [36] | Market1501 | 78.1/90.2 | 89.6/95.9 | 89.5/95.5 | 89.9/96.1
MSMT17 | 49.7/73.6 | 63.3/83.2 | 61.6/82.2 | 63.9/83.8
Attribute Recognition mA $\uparrow$ RethinkPAR [38] | PETAzs | 72.86 | 73.64 | 73.90 | 74.20
RAPzs | 72.10 | 73.04 | 73.16 | 73.21
PA100k | 80.67 | 82.98 | 82.98 | 84.15
Person Search mAP/Rank1 $\uparrow$ SeqNet [45] | CUHK-SYSU | 93.0/94.1 | 93.6/94.3 | 93.6/94.1 | 94.0/94.7
PRW | 50.0/84.4 | 52.9/84.7 | 53.0/84.0 | 54.1/85.0
Pedestrian Detection MR-2(R/HO) $\downarrow$ CSP [73] | CityPerson | 11.6/43.8 | 11.4/43.1 | 11.1/41.7 | 10.8/40.7
Human Parsing mIOU $\uparrow$ SCHP [43] | LIP | 51.10 | 54.45 | 55.25 | 55.45
Pose Estimation AP/AR $\uparrow$ HRFormer [83] | COCO | 72.4/78.2 | 73.1/78.5 | 73.4/78.7 | 74.4/79.7

Analysis on semantic controller. To further verify the semantic controller, we
provide another two experiments. We first define two distance, _i.e_., intra-
image distance and inter-image distance. Intra-image distance is the average
distance between any two parts from the same image. Inter-image distance
indicates the average distance between parts of the same semantic meaning but
coming from different images. Small intra-image distance and large inter-image
distance indicate the appearance information is dominant in the
representation, otherwise the semantic information dominates. After the
SOLIDER is trained, we set the semantic weight $\lambda$ from 0 to 1, to
observe two distances of these different representations in the whole LUP
dataset. The results are shown in Fig. 5(a). It can be seen that with the
increase of $\lambda$, the intra-image distance gets larger, while the inter-
image distance becomes smaller, which suggests that more semantic information
is involved into the representation and become dominant.

Meanwhile, as we’ve known, person re-identification requires more appearance
information, while pedestrian detection is prone to the semantic information.
We list the performance of two tasks using the pre-trained representations
with different $\lambda$ As shown in Fig. 5(b), with the increase of
$\lambda$, the person re-identification performance becomes worse and the
pedestrian detection performance turns to better. In other words, the
representation with a large $\lambda$ can provide a better startup for
pedestrian detection, while a small $\lambda$ fits more to person re-
identification. This phenomenon further implies the effectiveness of our
semantic controller.

###  4.3 Ablation Study

The ablation study is provide to verify the effectiveness of each module in
SOLIDER, and all the experiments are conducted on Swin-Tiny [52] backbone. We
use different methods to train the pretext task, and verify pre-trained models
on all downstream human-centric visual tasks.

ImageNet and LUP1M are datasets used for the pretext task training. ImageNet
indicates the model is trained on ImageNet. This training is supervised with
labels in ImageNet. LUP1M is a subset randomly sampled from LUP, which
contains 1 million person images and has a similar image number with ImageNet
for fair comparison. Due to no labels in LUP1M, the training process is self-
supervised. We take DINO [6] as our baseline. “+Clustering” indicates that we
involve the human prior into DINO by semantic clustering.
“+Clustering&Controller” means that the semantic controller is imported to
control the ratio of semantic information in representation. In downstream
tasks, we set $\lambda$ to a small value (0.0-0.2) for person-re-
identification, and a large value (0.8-1.0) for pedestrian detection, human
attributes, human parsing and pose estimation. For person search, where
appearance information is as important as semantic information, we set
$\lambda$ to a moderate value. Details of the $\lambda$ selection for
downstream tasks are provided in Supplementary Material.

From the comparison in Table. 1, it can be found that after involving the
semantic information (“+Clustering”), the pre-trained model achieves a better
performance on most of downstream tasks. It implies the effectiveness of the
representation with more semantic information. It is worth noticing that the
performance on person re-identification and person search are somewhat
declined after involving semantic supervision. The reason is that after the
semantic information is imported, the model is more inclined to represent
semantic information, and its power on distinguishing different identities is
weakened to some extent, which needs a more careful balance. The semantic
controller is exactly designed for this problem. After involving the semantic
control (“+Controller”), the performance further improved, especially for
person re-identification and person search. Comparing our SOLIDER results to
DINO’s on person re-identification, we can observe that our improvement is
limited. Because the key clue in person re-identification is appearance
information, which is already well learned in DINO. The semantic information
imported from SOLIDER plays an auxiliary role leading to only a slight
increment.

Table 2: The comparison of the proposed SOLIDER with other state of the arts.

Person Re-identification mAP/Rank1 $\uparrow$ |  | SCSN [16] | ABDNet [9] | TransReID [36] | UP-ReID [81] | PASS [97] |  Swin-T |  SOLIDER Swin-S |  Swin-B
---|---|---|---|---|---|---|---|---|---
Market1501 | 88.5/95.7 | 88.3/95.6 | 89.5/95.2 | 91.1/97.1 | 93.3/96.9 | 91.6/96.1 | 93.3/96.6 | 93.9/96.9
MSMT17 | 58.5/83.8 | 60.8/82.3 | 69.4/86.2 | 63.3/84.3 | 74.3/89.7 | 67.4/85.9 | 76.9/90.8 | 77.1/90.7
Attribute Recognition mA $\uparrow$ |  | MsVAA [61] | VAC [30] | ALM [66] | JLAC [65] | RethinkPAR [38] |  Swin-T |  SOLIDER Swin-S |  Swin-B
PETAzs | 71.53 | 71.91 | 73.01 | 73.60 | 71.62 | 74.37 | 76.21 | 76.43
RAPzs | 72.04 | 73.70 | 74.28 | 76.38 | 72.32 | 74.23 | 76.84 | 77.06
PA100k | 80.41 | 79.16 | 80.68 | 82.31 | 81.61 | 84.14 | 86.25 | 86.37
Person Search mAP/Rank1 $\uparrow$ |  | NAE+ [7] | AlignPS+ [79] | TCTS [68] | SeqNet [45] | GLCNet [93] |  Swin-T |  SOLIDER Swin-S |  Swin-B
CUHK-SYSU | 92.1/92.9 | 94.0/94.5 | 93.9/95.1 | 94.8/95.7 | 95.8/96.2 | 94.9/95.7 | 95.5/95.8 | 94.9/95.5
PRW | 44.0/81.1 | 46.1/82.1 | 46.8/87.5 | 47.6/87.6 | 47.8/87.8 | 56.8/86.8 |  59.8/86.7 | 59.7/86.8
Pedestrian Detection MR-2(R/HO) $\downarrow$ |  | RepLoss [71] | CSP [73] | NMS-Loss [56] | ACSP [70] | PedesFormer [33] |  Swin-T |  SOLIDER Swin-S |  Swin-B
CityPerson | 13.2/56.9 | 11.0/49.3 | 10.8/- | 9.3/46.3 | 9.2/36.9 | 10.3/40.8 | 10.0/39.2 | 9.7/39.4
Human Parsing mIOU $\uparrow$ |  | JPPNet [46] | BraidNet [50] | CE2P [60] | PCNet [87] | SCHP [43] |  Swin-T |  SOLIDER Swin-S |  Swin-B
LIP | 51.37 | 54.40 | 53.10 | 57.03 | 59.36 | 57.52 | 60.21 | 60.50
Pose Estimation AP/AR $\uparrow$ |  | CPN [17] | SimpleBase [75] | TokenPose [44] | HRNet [63] | HRFormer [83] |  Swin-T |  SOLIDER Swin-S |  Swin-B
COCO | 68.6/- | 74.3/79.7 | 75.8/80.9 | 76.3/81.2 | 77.2/82.0 | 74.4/79.6 | 76.3/81.3 | 76.6/81.5

It is noteworthy that the larger improvement of DINO LUP1M compared to Sup
ImageNet on person ReID is due to the street-view scenarios of LUPerson images
which are similar to ReID datasets. Except for the ReID task, the DINO LUP1M
brings an average of 1.4 improvement (72.6 vs. 74.0) on five tasks compared to
Sup ImageNet, which implies the advantage of the appearance information
learned from DINO. With SOLIDER, the average performance is raised to 74.9.
The further 0.9 increase shows the success of involving semantic information
by our SOLIDER. And this improvement is consistent on all tasks.

Besides, we conduct experiments on the influence of the clustered part number
and the semantic head size in SOLIDER, which are present in Supplementary
Material.

###  4.4 Comparison to State of the Arts

We compare our results with state of the arts on six human-centric tasks. The
state-of-the-art methods for comparison are listed in Table. 2. In person re-
identification, we use TransReID without side information as our baseline.
Even without side information, we can achieve a better performance than other
self-supervised works, _e.g_., TransReID [36] and PASS [97], which also
trained on LUPerson dataset. For the PRW results of person search, the mAP of
our results outperform other state of the arts with more than 10%. The mAP
criterion reflects the detection ability of models. It implies that our pre-
trained model can lead to a better detection result, which may thanks to the
imported semantic information. In pedestrian detection, PedesFormer [33]
achieves a better performance than ours on pedestrian detection task is
because it also involves extra data from autonomous driving datasets which are
specific for pedestrian detection task and not for other human-centric tasks.
For pose estimation, we report the best performance of HRFormer [83] which is
trained on HRFormer backbone instead of Swin. After HRFormer [83] backbone is
switched to Swin, the result is 75.9/81.1 reported in MMPose [19], which is
lower than ours 76.6/81.5. From the comparison in Table. 2, we can see that
the pre-trained from SOLIDER can provide a better initialization for these
human-centric tasks, and can be used as a new baseline for further works on
these tasks.

We also conduct SOLIDER on different backbones, also summarize in Table. 2. It
can be seen that with the model size increasing in Swin-Transformer backbones,
the performance is further improved. In some tasks, Swin-Small achieves a
better performance than Swin-Base, it is because of a larger batch size in
Swin-Small compared to that in Swin-Base under a limited GPU memory.

##  5 Conclusion

This paper proposes a semantic controllable self-supervised learning framework
called SOLIDER. It can utilize prior knowledge from human images to train
representations with more semantic information. Moreover, the pre-trained
model from SOLIDER can be adjusted by an input value through the semantic
controller, which can produce representations with different ratios of
semantic information and to satisfy the requirements on downstream tasks. The
human representations from SOLIDER is verified on six human-centric visual
tasks, which can promote the development of these human-centric tasks in
computer vision community.

## References

  * [1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2d human pose estimation: New benchmark and state of the art analysis.  In CVPR, 2014.
  * [2] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * [3] Antonio Brunetti, Domenico Buongiorno, Gianpaolo Francesco Trotta, and Vitoantonio Bevilacqua.  Computer vision and deep learning techniques for pedestrian detection and tracking: A survey.  Neurocomputing, 2018.
  * [4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In Proceedings of the European conference on computer vision (ECCV), 2018.
  * [5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  NIPS, 2020.
  * [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [7] Di Chen, Shanshan Zhang, Jian Yang, and Bernt Schiele.  Norm-aware embedding for efficient person search.  In CVPR, 2020.
  * [8] Kai Chen, Weihua Chen, Tao He, Rong Du, Fan Wang, Xiuyu Sun, Yuchen Guo, and Guiguang Ding.  Tagperson: A target-aware generation pipeline for person re-identification.  In Proceedings of the 30th ACM International Conference on Multimedia, 2022.
  * [9] Tianlong Chen, Shaojin Ding, Jingyi Xie, Ye Yuan, Wuyang Chen, Yang Yang, Zhou Ren, and Zhangyang Wang.  Abd-net: Attentive but diverse person re-identification.  In CVPR, 2019.
  * [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [11] Weihua Chen, Lijun Cao, Xiaotang Chen, and Kaiqi Huang.  A novel solution for multi-camera object tracking.  In ICIP, 2014.
  * [12] Weihua Chen, Lijun Cao, Xiaotang Chen, and Kaiqi Huang.  An equalised global graphical model-based approach for multi-camera object tracking.  IEEE Transactions on Circuits and Systems for Video Technology, 2016\.
  * [13] Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang.  Beyond triplet loss: a deep quadruplet network for person re-identification.  In CVPR, 2017.
  * [14] Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang.  A multi-task deep network for person re-identification.  In AAAI, 2017.
  * [15] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [16] Xuesong Chen, Canmiao Fu, Yong Zhao, Feng Zheng, Jingkuan Song, Rongrong Ji, and Yi Yang.  Salience-guided cascaded suppression network for person re-identification.  In CVPR, 2020.
  * [17] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun.  Cascaded pyramid network for multi-person pose estimation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7103–7112, 2018.
  * [18] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan.  Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering.  In CVPR, 2021.
  * [19] MMPose Contributors.  Openmmlab pose estimation toolbox and benchmark.  https://github.com/open-mmlab/mmpose, 2020.
  * [20] Yubin Deng, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Pedestrian attribute recognition at far distance.  In Proceedings of the 22nd ACM international conference on Multimedia, 2014.
  * [21] Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona.  Pedestrian detection: An evaluation of the state of the art.  PAMI, 2011.
  * [22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [23] Zhaopeng Dou, Zhongdao Wang, Weihua Chen, Yali Li, and Shengjin Wang.  Reliability-aware prediction via uncertainty learning for person image retrieval.  In ECCV, 2022.
  * [24] Markus Enzweiler and Dariu M Gavrila.  Monocular pedestrian detection: Survey and experiments.  PAMI, 2008.
  * [25] Dengpan Fu, Dongdong Chen, Jianmin Bao, Hao Yang, Lu Yuan, Lei Zhang, Houqiang Li, and Dong Chen.  Unsupervised pre-training for person re-identification.  In CVPR, 2021.
  * [26] Dengpan Fu, Dongdong Chen, Hao Yang, Jianmin Bao, Lu Yuan, Lei Zhang, Houqiang Li, Fang Wen, and Dong Chen.  Large-scale pre-training for person re-identification with noisy labels.  In CVPR, 2022.
  * [27] Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin.  Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing.  In CVPR, 2017.
  * [28] Jianyang Gu, Weihua Chen, Hao Luo, Fan Wang, Hao Li, Wei Jiang, and Weijie Mao.  Multi-view evolutionary training for unsupervised domain adaptive person re-identification.  IEEE Transactions on Information Forensics and Security, 2022.
  * [29] Jianyang Gu, Hao Luo, Weihua Chen, Yiqi Jiang, Yuqi Zhang, Shuting He, Fan Wang, Hao Li, and Wei Jiang.  1st place solution to visda-2020: Bias elimination for domain adaptive pedestrian re-identification.  arXiv preprint arXiv:2012.13498, 2020.
  * [30] Hao Guo, Kang Zheng, Xiaochuan Fan, Hongkai Yu, and Song Wang.  Visual attention consistency under image transforms for multi-label image classification.  In CVPR, 2019.
  * [31] John A Hartigan and Manchek A Wong.  Algorithm as 136: A k-means clustering algorithm.  Journal of the royal statistical society. series c (applied statistics), 1979.
  * [32] Irtiza Hasan, Shengcai Liao, Jinpeng Li, Saad Ullah Akram, and Ling Shao.  Generalizable pedestrian detection: The elephant in the room.  In CVPR, 2021.
  * [33] Irtiza Hasan, Shengcai Liao, Jinpeng Li, Saad Ullah Akram, and Ling Shao.  Pedestrian detection: Domain generalization, cnns, transformers and beyond.  arXiv preprint arXiv:2201.03176, 2022.
  * [34] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In CVPR, 2022.
  * [35] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [36] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang.  Transreid: Transformer-based object re-identification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15013–15022, 2021.
  * [37] Takashi Isobe, Dong Li, Lu Tian, Weihua Chen, Yi Shan, and Shengjin Wang.  Towards discriminative representation learning for unsupervised person re-identification.  In ICCV, 2021.
  * [38] Jian Jia, Houjing Huang, Xiaotang Chen, and Kaiqi Huang.  Rethinking of pedestrian attribute recognition: A reliable evaluation under zero-shot pedestrian identity setting.  arXiv preprint arXiv:2107.03576, 2021.
  * [39] Yiqi Jiang, Weihua Chen, Xiuyu Sun, Xiaoyu Shi, Fan Wang, and Hao Li.  Exploring the quality of gan generated images for person re-identification.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 4146–4155, 2021.
  * [40] Xin Jin, Cuiling Lan, Wenjun Zeng, Guoqiang Wei, and Zhibo Chen.  Semantics-aligned representation learning for person re-identification.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
  * [41] Qingming Leng, Mang Ye, and Qi Tian.  A survey of open-world person re-identification.  IEEE Transactions on Circuits and Systems for Video Technology, 2019\.
  * [42] Dangwei Li, Zhang Zhang, Xiaotang Chen, Haibin Ling, and Kaiqi Huang.  A richly annotated dataset for pedestrian attribute recognition.  arXiv preprint arXiv:1603.07054, 2016.
  * [43] Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang.  Self-correction for human parsing.  PAMI, 2020.
  * [44] Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou.  Tokenpose: Learning keypoint tokens for human pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11313–11322, 2021.
  * [45] Zhengjia Li and Duoqian Miao.  Sequential end-to-end network for efficient person search.  In AAAI, 2021.
  * [46] Xiaodan Liang, Ke Gong, Xiaohui Shen, and Liang Lin.  Look into person: Joint body parsing & pose estimation network and a new benchmark.  PAMI, 2018.
  * [47] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [48] Chong Liu, Yuqi Zhang, Weihua Chen, Fan Wang, Hao Li, and Yi-Dong Shen.  Adaptive matching strategy for multi-target multi-camera tracking.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  * [49] Chong Liu, Yuqi Zhang, Hao Luo, Jiasheng Tang, Weihua Chen, Xianzhe Xu, Fan Wang, Hao Li, and Yi-Dong Shen.  City-scale multi-camera vehicle tracking guided by crossroad zones.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [50] Xinchen Liu, Meng Zhang, Wu Liu, Jingkuan Song, and Tao Mei.  Braidnet: Braiding semantics and details for accurate human parsing.  In Proceedings of the 27th ACM International Conference on Multimedia, pages 338–346, 2019.
  * [51] Xihui Liu, Haiyu Zhao, Maoqing Tian, Lu Sheng, Jing Shao, Shuai Yi, Junjie Yan, and Xiaogang Wang.  Hydraplus-net: Attentive deep features for pedestrian analysis.  In ICCV, 2017.
  * [52] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [53] Angelique Loesch and Romaric Audigier.  Describe me if you can! characterized instance-level human parsing.  In ICIP, 2021.
  * [54] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-task vision and language representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437–10446, 2020.
  * [55] Hao Luo, Pichao Wang, Yi Xu, Feng Ding, Yanxin Zhou, Fan Wang, Hao Li, and Rong Jin.  Self-supervised pre-training for transformer-based person re-identification.  arXiv preprint arXiv:2111.12084, 2021.
  * [56] Zekun Luo, Zheng Fang, Sixiao Zheng, Yabiao Wang, and Yanwei Fu.  Nms-loss: Learning with non-maximum suppression for crowded pedestrian detection.  In Proceedings of the 2021 International Conference on Multimedia Retrieval, 2021.
  * [57] Thomas Mauthner, Horst Possegger, Georg Waltner, and Horst Bischof.  Encoding based saliency detection for videos and images.  In CVPR, 2015.
  * [58] Erik Murphy-Chutorian and Mohan Manubhai Trivedi.  Head pose estimation in computer vision: A survey.  PAMI, 2008.
  * [59] Xiao Pan, Hao Luo, Weihua Chen, Fan Wang, Hao Li, Wei Jiang, Jianming Zhang, Jianyang Gu, and Peike Li.  Dynamic gradient reactivation for backward compatible person re-identification.  arXiv preprint arXiv:2207.05658, 2022.
  * [60] Tao Ruan, Ting Liu, Zilong Huang, Yunchao Wei, Shikui Wei, and Yao Zhao.  Devil in the details: Towards accurate single and multiple human parsing.  In Proceedings of the AAAI conference on artificial intelligence, 2019.
  * [61] Nikolaos Sarafianos, Xiang Xu, and Ioannis A Kakadiaris.  Deep imbalanced attribute classification using visual attention aggregation.  In ECCV, 2018.
  * [62] Yumin Suh, Jingdong Wang, Siyu Tang, Tao Mei, and Kyoung Mu Lee.  Part-aligned bilinear representations for person re-identification.  In ECCV, 2018.
  * [63] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In CVPR, 2019.
  * [64] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjing Wang.  Beyond part models: Person retrieval with refined part pooling.  In ECCV, 2018.
  * [65] Zichang Tan, Yang Yang, Jun Wan, Guodong Guo, and Stan Z Li.  Relation-aware pedestrian attribute recognition with graph convolutional networks.  In AAAI, 2020.
  * [66] Chufeng Tang, Lu Sheng, Zhaoxiang Zhang, and Xiaolin Hu.  Improving pedestrian attribute recognition with weakly-supervised multi-scale attribute-specific localization.  In ICCV, 2019.
  * [67] Jiasheng Tang, Fei Du, Weihua Chen, Hao Luo, Fan Wang, and Hao Li SBT.  A simple baseline with cascade association for robust multi-objects tracking.  In CVPR RVSU Workshop, volume 4, 2021.
  * [68] Cheng Wang, Bingpeng Ma, Hong Chang, Shiguang Shan, and Xilin Chen.  Tcts: A task-consistent two-stage framework for person search.  In CVPR, 2020.
  * [69] Shanshan Wang, Lei Zhang, Weihua Chen, Fan Wang, and Hao Li.  Refining pseudo labels for unsupervised domain adaptive re-identification.  Knowledge-Based Systems, 2022.
  * [70] Wenhao Wang.  Adapted center and scale prediction: more stable and more accurate.  arXiv preprint arXiv:2002.09053, 2020.
  * [71] Xinlong Wang, Tete Xiao, Yuning Jiang, Shuai Shao, Jian Sun, and Chunhua Shen.  Repulsion loss: Detecting pedestrians in a crowd.  In CVPR, 2018.
  * [72] Xiao Wang, Shaofei Zheng, Rui Yang, Aihua Zheng, Zhe Chen, Jin Tang, and Bin Luo.  Pedestrian attribute recognition: A survey.  Pattern Recognition, 2022.
  * [73] Liu Wei, Liao Shengcai, Ren Weiqiang, Hu Weidong, and Yu Yinan.  High-level semantic feature detection: A new perspective for pedestrian detection.  In CVPR, 2019.
  * [74] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian.  Person transfer gan to bridge domain gap for person re-identification.  In CVPR, 2018.
  * [75] Bin Xiao, Haiping Wu, and Yichen Wei.  Simple baselines for human pose estimation and tracking.  In Proceedings of the European conference on computer vision (ECCV), pages 466–481, 2018.
  * [76] Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang.  Joint detection and identification feature learning for person search.  In CVPR, 2017.
  * [77] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  In CVPR, 2022.
  * [78] Ehsan Yaghoubi, Farhad Khezeli, Diana Borza, SV Kumar, João Neves, and Hugo Proença.  Human attribute recognition—a comprehensive survey.  Applied Sciences, 2020.
  * [79] Yichao Yan, Jinpeng Li, Jie Qin, Song Bai, Shengcai Liao, Li Liu, Fan Zhu, and Ling Shao.  Anchor-free person search.  In CVPR, 2021.
  * [80] Yang Yang, Zhen Lei, Shifeng Zhang, Hailin Shi, and Stan Z Li.  Metric embedded discriminative vocabulary learning for high-level person representation.  In AAAI, 2016.
  * [81] Zizheng Yang, Xin Jin, Kecheng Zheng, and Feng Zhao.  Unleashing potential of unsupervised pre-training with intra-identity regularization for person re-identification.  In CVPR, 2022.
  * [82] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi.  Deep learning for person re-identification: A survey and outlook.  PAMI, 2021.
  * [83] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang.  Hrformer: High-resolution transformer for dense prediction.  In NIPS, 2021.
  * [84] Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang.  Not all tokens are equal: Human-centric visual analysis via token clustering transformer.  In CVPR, 2022.
  * [85] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv:1710.09412, 2017.
  * [86] Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele.  Citypersons: A diverse dataset for pedestrian detection.  In CVPR, 2017.
  * [87] Xiaomei Zhang, Yingying Chen, Bingke Zhu, Jinqiao Wang, and Ming Tang.  Part-aware context network for human parsing.  In CVPR, 2020.
  * [88] Xuan Zhang, Hao Luo, Xing Fan, Weilai Xiang, Yixiao Sun, Qiqi Xiao, Wei Jiang, Chi Zhang, and Jian Sun.  Alignedreid: Surpassing human-level performance in person re-identification.  arXiv preprint arXiv:1711.08184, 2017.
  * [89] Yuqi Zhang, Qi Qian, Chong Liu, Weihua Chen, Fan Wang, Hao Li, and Rong Jin.  Graph convolution for re-ranking in person re-identification.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  * [90] Ce Zheng, Wenhan Wu, Taojiannan Yang, Sijie Zhu, Chen Chen, Ruixu Liu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah.  Deep learning-based human pose estimation: A survey.  arXiv preprint arXiv:2012.13392, 2020.
  * [91] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In ICCV, 2015.
  * [92] Liang Zheng, Hengheng Zhang, Shaoyan Sun, Manmohan Chandraker, Yi Yang, and Qi Tian.  Person re-identification in the wild.  In CVPR, 2017.
  * [93] Peng Zheng, Jie Qin, Yichao Yan, Shengcai Liao, Bingbing Ni, Xiaogang Cheng, and Ling Shao.  Global-local context network for person search.  arXiv preprint arXiv:2112.02500, 2021.
  * [94] Zhedong Zheng, Xiaodong Yang, Zhiding Yu, Liang Zheng, Yi Yang, and Jan Kautz.  Joint discriminative and generative learning for person re-identification.  In CVPR, 2019.
  * [95] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  arXiv preprint arXiv:2111.07832, 2021.
  * [96] Kuan Zhu, Haiyun Guo, Zhiwei Liu, Ming Tang, and Jinqiao Wang.  Identity-guided human semantic parsing for person re-identification.  In ECCV, 2020.
  * [97] Kuan Zhu, Haiyun Guo, Tianyi Yan, Yousong Zhu, Jinqiao Wang, and Ming Tang.  Pass: Part-aware self-supervised pre-training for person re-identification.  In ECCV. Springer, 2022.


# Polarized neutron scattering study on the centrosymmetric skyrmion host
material Gd2PdSi3

Jiwon Ju  Hiraku Saito  The Institute for Solid State Physics, the University
of Tokyo, Kashiwa, Chiba, 277-8581, Japan  Takashi Kurumaji  Department of
Advanced Materials Science, University of Tokyo, Kashiwa 277-8561, Japan  Max
Hirschberger  RIKEN Center for Emergent Matter Science (CEMS), Saitama
351-0198, Japan.  Department of Applied Physics and Quantum Phase Electronics
Center (QPEC), University of Tokyo, Tokyo 113-8656, Japan  Akiko Kikkawa
Yasujiro Taguchi  RIKEN Center for Emergent Matter Science (CEMS), Saitama
351-0198, Japan.  Taka-hisa Arima  RIKEN Center for Emergent Matter Science
(CEMS), Saitama 351-0198, Japan.  Department of Advanced Materials Science,
University of Tokyo, Kashiwa 277-8561, Japan,  Yoshinori Tokura  RIKEN Center
for Emergent Matter Science (CEMS), Saitama 351-0198, Japan.  Department of
Applied Physics and Quantum Phase Electronics Center (QPEC), University of
Tokyo, Tokyo 113-8656, Japan  Tokyo College, University of Tokyo, Tokyo
113-8656, Japan,  Taro Nakajima<EMAIL_ADDRESS>The
Institute for Solid State Physics, the University of Tokyo, Kashiwa, Chiba,
277-8581, Japan  RIKEN Center for Emergent Matter Science (CEMS), Saitama
351-0198, Japan.

###### Abstract

We have investigated magnetic structures of the centrosymmetrric skyrmion
material Gd2PdSi3 by means of polarized neutron scattering near zero field
with an isotope-160Gd-enriched single crystal. In a previous study, magnetic
structures in Gd2PdSi3 at low temperatures were studied by resonant X-ray
scattering measurements [T. Kurumaji et al. Science 365, 914 (2019)]. The
present polarized neutron results confirm that the magnetic structure in zero
field has elliptic screw-type magnetic modulation with a propagation vector of
$(q,0,0)$ with $q\sim 0.14$ and its equivalents. As the temperature increases,
the system undergoes a magnetic phase transition while keeping the
incommensurate $q$-vector of $(q,0,0)$. We found that the thermally-induced
phase has sinusoidal magnetic modulations with magnetic moments perpendicular
both to the $c$ axis and to the $q$-vector. We also investigate the spin-
helicity degree of freedom in the ground state by polarized neutrons,
revealing that the system contains equal fractions of the left-handed and
right-handed screw-type orders as expected from the centrosymmetric crystal
structure.

††preprint: APS/123-QED

##  I INTRODUCTION

Gadolinium based intermetallics have recently attracted increasing attention
since a magnetic skyrmion lattice (SkL), which is a regular arrangement of
topologically-nontrivial vortex-like spin objects, with a large topological
Hall effect was discovered in a centrosymmetic triangular lattice magnet
Gd2PdSi3 Kurumaji  _et al._ (2019). Magnetic skyrmions were predicted,
discovered, and have been extensively studied in noncentrosymmetric magnets
with ferromagnetic exchange interactions and Dzyaloshinskii-Moriya (DM)
interactions Bogdanov and Yablonskii (1989); Rößler  _et al._ (2006);
Mühlbauer  _et al._ (2009); Yu  _et al._ (2010); Münzer  _et al._ (2010); Seki
_et al._ (2012); Kézsmárki  _et al._ (2015); Tokunaga  _et al._ (2015);
Kurumaji  _et al._ (2017). However, the recent studies on centrosymmetric
itinerant Gd compounds, such as hexagonal Gd2PdSi3, breathing kagomé
Gd3Ru4Al12 Hirschberger  _et al._ (2019) and tetragonal GdRu2Si2 Khanh  _et
al._ (2020), have demonstrated that SkL states can be realized without relying
on DM interactions. The microscopic mechanism to stabilize the SkL states in
the centrosymmetric systems is still under debate. Previous studies on
localized spin models pointed out the importance of geometrical spin
frustration Okubo  _et al._ (2012); Leonov and Mostovoy (2015). As for
itinerant models, numerical calculations based on a Kondo lattice model
revealed that couplings between conduction electrons and localized magnetic
moments can lead to a variety of topological spin orders including SkL states
Hayami  _et al._ (2017), assuming that the Ruderman-Kittel-Kasuya-Yosida
(RKKY) interaction determines the magnetic modulation vectors. By contrast, a
recent ab-initio calculation pointed out that orbital-dependent exchange
interactions between the Gd spins are essential for the incommensurate
magnetic modulations in the Gd-based intermetallics Nomoto  _et al._ (2020).
To examine these models, it is necessary to investigate microscopic parameters
of the existing centrosymmetric skyrmion materials experimentally. For
instance, a recent neutron scattering study on a polycrystalline sample of
Gd2PdSi3 has suggested that the system should have long-range magnetic
interactions, which supports the RKKY picture Pad . In the present study, we
performed polarized neutron scattering on a single crystal of isotope-enriched
160Gd2PdSi3, and investigate anisotropy and chirality of the magnetic
structures in zero field.

Gd2PdSi3 has an AlB2-like hexagonal structure, in which triangular lattice
layers of magnetic Gd atoms are stacked along the $c$ axis. This system
exhibits magnetic orders below 21 K Saha  _et al._ (1999). Kurumaji et al.
performed resonant X-ray scattering measurements in the ground state, and
found incommensurate magnetic reflections with magnetic modulation wave
vectors ($q$-vectors) of $(q,0,0)$, $(0,q,0)$, and $(q,-q,0)$, where $q\sim
0.14$ Kurumaji  _et al._ (2019). These $q$-vectors are equivalent to each
other owing to the hexagonal symmetry of the crystalcom . By analyzing the
polarization of the scattered X-rays, they concluded that the ground state has
screw-type magnetic modulations. To further corroborate the magnetic structure
in Gd2PdSi3, we have performed polarized neutron scattering. While resonant
X-ray scattering is sensitive not only to the magnetic orders but also charge
density waves (CDWs), which were actually reported in a recent scanning
tunneling microscopy study on GdRu2Si2Yasui  _et al._ (2020), polarized
neutron scattering can unambiguously distinguish magnetic modulations from
charge/lattice modulations.

There are also several open questions regarding the magnetic orders in
Gd2PdSi3. One is thermally-induced phase transitions in zero field. Recent
specific heat Hirschberger  _et al._ (2020) and dilatometry Spachmann  _et
al._ (2021) measurements reported the existence of an intermediate phase
between the paramagnetic phase and the ground state. We here investigate the
temperature dependence of the magnetic structure by polarized neutron
scattering. Another problem is the the sense of spin helix. In centrosymmetric
systems, left-handed and right-handed screw-type magnetic modulations should
be energetically equivalent to each other. This degeneracy can be lifted when
the system is noncentrosymmetric. In fact, previous polarized neutron
scattering and transmission electron microscopy studies on an archetypal
skyrmion host compounds MnSi and (Fe,Co)Si revealed one-to-one correspondences
between magnetic and crystallographic chirality in single crystal samples
Ishida  _et al._ (1985); Morikawa  _et al._ (2013). In the present study, we
also investigate the magnetic chirality in Gd2PdSi3 by polarized neutron
scattering.

##  II EXPERIMENTAL DETAILS

We grew an isotope-enriched 160Gd2PdSi3 single crystal by the floating zone
method in order to reduce the strong neutron absorption of Gd. The crystal was
cut into a plate shape with thickness of 1.0 mm and area of 16.2 mm2. The
experiment was carried out by using the polarized neutron triple-axis
spectrometer PONTA in Japan Research Reactor 3 (JRR-3). The sample was mounted
in a closed-cycle 4He refrigerator with the $(H,K,0)$ scattering plane. The
spectrometer was operated in two longitudinal polarization analysis modes,
which will be explained in the following section. A spin-polarized incident
neutron beam with the energy of 13.7 meV was obtained by a Heusler(111)
crystal monochromator. The directions of the neutron spins were controlled by
a spin flipper, guide fields, and a Helmholtz coil. We also employed a
Heusler(111) crystal analyzer to select the energy and spin states of
scattered neutrons. We measured spin-flip (SF) and non-spin-flip (NSF)
intensities by changing the spin state of the incident neutrons using the spin
flipper placed between the monochromator and the sample. The spin polarization
of the neutron beam was 0.84, which was measured by 100 nuclear Bragg
reflection of the sample.

Figure 1: (a) Schematic illustration showing the experimental setup and the
definitions of the Cartesian coordinates $xyz$. (b) The definition of unit
vectors to describe the spin components.

##  III DETAILS ON POLARIZED NEUTRON SCATTERING

In neutron scattering experiments, the intensity of magnetic elastic
scattering is proportional to the square of Fourier-transformed magnetic
moments projected onto a plane perpendicular to the scattering vector. In the
following, we refer to this quantity as
$|\mbox{\boldmath$M$}^{\perp}(\mbox{\boldmath$\kappa$})|^{2}$, where
$\mbox{\boldmath$\kappa$}(\equiv\mbox{\boldmath$k$}_{i}-\mbox{\boldmath$k$}_{f})$
is the scattering vector. Here, we introduce a Cartesian coordinate system
$xyz$ in which the $x$ axis is parallel to $\kappa$, the $z$ axis is parallel
to the $c$ axis, and the $y$ axis completes the right-hand set (see Fig.
1(a)). By using this coordinate system,
$\mbox{\boldmath$M$}^{\perp}(\mbox{\boldmath$\kappa$})$ is written as
$\mbox{\boldmath$M$}^{\perp}(\mbox{\boldmath$\kappa$})=(0,M^{\perp}_{y}(\mbox{\boldmath$\kappa$}),M^{\perp}_{z}(\mbox{\boldmath$\kappa$}))$.
Note that the $x$ component should be zero, because
$\mbox{\boldmath$M$}^{\perp}(\mbox{\boldmath$\kappa$})$ only contains the
magnetic moments projected onto the $yz$ plane. In the longitudinal
polarization analysis, intensities in SF and NSF channels are proportional to
the Fourier components perpendicular and parallel to the neutron spin
direction, respectively Squires (1978).

In one configuration called $P_{zz}$, the direction of the incident neutron
spins is parallel to the $z$ axis. The SF and NSF intensities arising from a
magnetic Bragg reflection correspond to
$M^{\perp}_{y}(\mbox{\boldmath$\kappa$})$ and
$M^{\perp}_{z}(\mbox{\boldmath$\kappa$})$, respectively. In addition, a
lattice modulation of the same wave vector, which might be induced by a charge
density wave, also contributes to the NSF intensity.

In the other configuration termed $P_{xx}$, the neutron spin direction is
turned to be parallel to the $x$ axis by a magnetic field produced by a
Helmholtz coil. Therefore, both $M^{\perp}_{y}(\mbox{\boldmath$\kappa$})$ and
$M^{\perp}_{z}(\mbox{\boldmath$\kappa$})$ are perpendicular to the neutron
spin, and are observed in the SF channel. On the other hand, the nonmagnetic
scattering is left in the NSF channel. Note that the guide field is so low
that the magnetic structure is not affected.

We also introduce Cartesian coordinate systems to describe the magnetic
modulations corresponding to the three $q$-vectors. Figure 1(b) shows the
three unit vectors $\mbox{\boldmath$e$}_{q}$, $\mbox{\boldmath$e$}_{\perp
q,c}$ and $\mbox{\boldmath$e$}_{c}$. We refer to amplitudes of the Fourier
components with the wave vector of $q$ along these three axes as $m_{q}$,
$m_{\perp q,c}$ and $m_{c}$. For example, a proper screw magnetic order has
the same amplitudes of $m_{\perp q,c}$ and $m_{c}$, and does not have $m_{q}$.
We will determine the three amplitudes from the results of the polarized
neutron scattering experiment.

##  IV RESULTS AND DISCUSSIONS

###  IV.1 Magnetic structure in the ground state

The $P_{xx}$ configuration data at 2.5 K confirms the absence of non-magnetic
contributions at magnetic Bragg peaks. Figure 2(a) shows scattering profiles
of a magnetic Bragg reflection at $(0,q,0)$ measured at 2.5 K near zero field.
We have subtracted background signals estimated from measurements at high
temperatures, and corrected the effect of the imperfect polarization. The
reflection does not contain NSF signals, indicating that the Bragg reflection
at $(0,q,0)$ is purely magnetic.

We then measured the same reflection with the $P_{zz}$ configuration, and
observed both SF and NSF intensities as shown in Fig. 2(b). This indicates
that the magnetic structure in the ground state has both $m_{c}$ and $m_{\perp
q,c}$ components. We also measured a satellite reflection at $(-1+q,2,0)$,
where $\kappa$ is nearly perpendicular to $\mbox{\boldmath$e$}_{q}$, as shown
in Fig. 2(e). In this situation, the NSF and SF intensities correspond to
$m_{c}$ and $m_{q}$ components, respectively. As shown in Fig. 2(d), the
reflection does not contain SF scattering, indicating that the $m_{q}$
component is absent.

Figure 2: [(a),(b)] Scattering profiles of the magnetic Bragg reflection at
$(0,q,0)$ measured in the (a) $P_{xx}$ and (b) $P_{zz}$ configurations at 2.5
K in zero field. (c) Schematic showing the directions of $\kappa$ and
$q$-vector when measuring the reflection at $(0,q,0)$. (d) Scattering profiles
of the reflection at $(-1+q,2,0)$ measured in the $P_{zz}$ configuration at
2.5 K. (e) Schematic showing the directions of $\kappa$ and $q$-vector for the
reflection $(-1+q,2,0)$. Green circles and gray squares denote positions of
the magnetic and nuclear reflections, respectively.

From these results, we conclude that the ground state of this compound has a
screw-type magnetic modulation. Note that the present measurements are not
sensitive to the phase relationship between the $m_{c}$ and $m_{\perp q,c}$
components. If the incommensurate modulations of the $m_{c}$ and $m_{\perp
q,c}$ components were in-phase, they would result in a collinear oblique-
sinusoidal magnetic modulation in which magnetic moments would be pointing a
direction between $\mbox{\boldmath$e$}_{c}$ and $\mbox{\boldmath$e$}_{\perp
q,c}$. In this case, the amplitudes of each magnetic moment were sinusoidally
modulated in space. This is not likely to happen at low temperatures, where
the magnetic moments of Gd atoms are supposed to behave as classical
Heisenberg spins with a fixed amplitude of 7 $\mu_{B}$.

The previous resonant X-ray study by Moody et al. suggested that Gd2PdSi3 has
co-planer spin orders in the $ab$-plane and CDWs with the same modulation
vectors as the magnetic $q$-vectors Moo . However, the present polarized
neutron scattering results show the existence of the $c$-axis component in the
magnetic modulations and absence of nonmagnetic contribution in the magnetic
Bragg peak at $(0,q,0)$, which contradicts their proposal. Note that the
present data do not exclude the possibility of concomitant CDWs in Gd2PdSi3.
As exemplified in the STM study on GdRu2Si2 Yasui  _et al._ (2020), the close
coupling between the Gd spins and conduction electrons can induce CDWs. In
neutron scattering, CDWs are observed when the charge modulation is large
enough to induce the spatial modulations of nuclei. Furthermore, the parasitic
CDW has a modulation vector of 2$q$ in general.

We always observe three pairs of magnetic satellite reflections surrounding a
reciprocal lattice point. This means that the system exhibits either of a
multi-$q$ magnetic order or a multi-domain state of single-$q$ magnetic
orders. In the previous study, it was pointed out that the field evolution of
Hall resistivity has an additional component even in the ground state,
suggesting that the ground state is a multi-$q$ state, specifically a meron-
antimeron lattice, which can have a field-induced topological Hall effect
Kurumaji  _et al._ (2019). By contrast, a recent neutron powder diffraction
study suggested that a single-$q$ screw-type magnetic order is more likely
judging from the absolute length of the magnetic moments deduced from Rietveld
analysis Pad . To unambiguously determine the magnetic structure, real-space
observations of the spin textures are desired.

Figure 3: [(a),(b)] Temperature dependence of (a) the integrated intensities
of NSF and SF scatterings measured at $(0,q,0)$ and (b) the ellipticity of the
screw-type magnetic order. Solid lines in (a) are guides to the eyes. Inset of
(b) shows the scattering profile measured at 21 K. (c) Schematic illustration
of the anharmonic rotation of spins in the ground state $(T<T_{\rm{N2}})$. The
directions of the magnetic moments are projected onto the plane perpendicular
to the $q$-vector. Dark green arrows show the spin directions taking into
account the ellipticity of 0.7. The numbers correspond to relative positions
of the spins along the $q$-vector. Light green arrows show the spin directions
without ellipticity.

###  IV.2 Temperature dependence of the magnetic structure

The temperature dependence of the SF and NSF intensities for the magnetic
Bragg reflection at $(0,q,0)$ is shown in Fig. 3(a). Both the SF and NSF
intensities monotonically decrease with increasing temperature. However, the
NSF intensity decreases more rapidly than the SF intensity. In Fig. 3(b), we
show temperature dependence of the ellipticity, $m_{c}/m_{\perp q,c}$. At
around 21 K, the NSF intensity becomes nearly zero, while the SF intensity
still remains, as shown in the inset of Fig. 3(b). By further increasing the
temperature, the SF intensity also disappears above 22 K. These thermally-
induced successive magnetic phase transitions are consistent with the previous
specific heat Hirschberger  _et al._ (2020) and dilatometry Spachmann  _et
al._ (2021) measurements.

The suppression of the NSF intensity shows that the intermediate phase has
sinusoidal magnetic modulations with the $m_{\perp q,c}$ component. Although
the lengths of the local magnetic moments should not change with temperature,
their thermal average can be reduced and modulated as the thermal fluctuations
become stronger near the critical temperature. These kinds of helical-to-
sinusoidal magnetic phase transitions are also seen in other Gd based skyrmion
host materials Khanh  _et al._ (2020); Hirschberger  _et al._ (2019); Khanh
_et al._ (2022) and in other frustrated magnets.

We also note that the ellipticity is smaller than unity ($m_{c}/m_{\perp
q,c}\sim 0.75$) even at the lowest temperature, where the thermal fluctuations
are negligible. This observation is consistent with the previous resonant
x-ary scattering study Kurumaji  _et al._ (2019). The smaller Fourier
component along the $c$ axis implies distortion of the screw-type structure,
specifically anharmonic rotation of the magnetic moments as illustrated in
Fig. 3(c). The anharmonicity should lead to odd higher harmonic reflections.
Assuming that all the magnetic moments have the same magnitude, we estimate
that an absolute square of the Fourier amplitude for the third harmonics will
be approximately two orders of magnitude smaller than that of the primary
modulation. In the present study, such a weak higher harmonic reflections can
not be detected due to a lack of statistics.

A possible interpretation of this distortion of the magnetic structure is that
the system has an easy-plane-type magnetic anisotropy. In most of the
theoretical studies on SkLs, an easy-plane type anisotropy was considered to
be unfavorable for stabilizing SkL states, and thus an easy-axis type
anisotropy was imposed Hayami and Motome (2019); Wang  _et al._ (2020). It can
be qualitatively understood that the easy-plane type anisotropy tends to
support transverse conical magnetic modulations in a magnetic field parallel
to the $c$ axis rather than the SkL that should possess some of the magnetic
moments to point downward. The easy-plane magnetic anisotropy causing the
partial suppression of the $c$-axis Fourier components in zero field as
observed in the present study seems to be in conflict with the above simple
theoretical scenario to favor the SkL formation; this implies that additional
anisotropy term and/or anisotropic exchange interactions should be taken into
account to explain the magnetic structures in Gd2PdSi3, including the SkL
state. For instance, a recent Monte Carlo simulation including magnetic
dipole-dipole interactions reproduces a helical ground state and a field-
induced SkL state Pad . The role of the anisotropic exchange interactions was
also studied in a previous study on Gd3Ru4Al12, in which a combination of the
easy-plane type anisotropy and anisotropic exchange interaction is necessary
to explain the observed field-angle dependence of the magnetic phase diagram
Hirschberger  _et al._ (2021).

Figure 4: Scattering profiles of magnetic Bragg reflections at $(1-q,0,0)$ and
$(1+q,0,0)$ measured at 2.5 K. The spectrometer was operated in the half-
polarized mode. The direction of the incident neutron polarization was set to
be parallel or antiparallel to the scattering vector $\kappa$. The data
labeled ”$I_{\rm{OFF}}$” and ”$I_{\rm{ON}}$” were measured when the spin
flipper was off and on, and correspond to the parallel and antiparallel cases,
respectively.

###  IV.3 Chirality of the magnetic structure

In the $P_{xx}$ configuration, intensities of magnetic Bragg reflections from
a screw-type magnetic modulation depend on the relationship between the
directions of the $q$-vector, the scattering vector and the neutron
polarization. In addition the sense of the spin helix can account for
asymmetry in intensity between a pair of magnetic satellite reflections. When
the scattering vector is parallel to the screw axis of the magnetic structure,
the scattering cross section for a magnetic Bragg reflection at
$\mbox{\boldmath$\tau$}\pm\mbox{\boldmath$q$}$, where $\tau$ is a reciprocal
lattice point, is given by

$\displaystyle\bigg{(}\frac{d\sigma}{d\Omega}\bigg{)}_{\mbox{\boldmath$\tau$}\pm\mbox{\boldmath$q$}}\propto$
$\displaystyle\\{(m_{c}^{2}+m_{\perp q,c}^{2})(V^{\rm RH}+V^{\rm LH})$
$\displaystyle\mp$ $\displaystyle 2m_{c}m_{\perp q,c}(\mbox{\boldmath$p$}_{\rm
N}\cdot\mbox{\boldmath$\hat{\kappa}$})(V^{\rm RH}-V^{\rm LH})\\}.$ (1)

Here, $V^{\rm RH}$ and $V^{\rm LH}$ are the volume fractions with right-handed
and left-handed helical magnetic orders, respectively. $\hat{\kappa}$ and
$\mbox{\boldmath$p$}_{\rm N}$ are unit vectors of $\kappa$ and the
polarization direction of the incident neutrons, respectively. Thus the scalar
product $\mbox{\boldmath$p$}_{\rm N}\cdot\mbox{\boldmath$\hat{\kappa}$}$ takes
a value of either $+1$ or $-1$ depending on the neutron spin state controlled
by the flipper. Accordingly, the intensities of the Bragg reflections also
change with the direction of the neutron spins when $V^{\rm RH}$ and $V^{\rm
LH}$ are not equal to each other.

We measured magnetic Bragg reflections at $(1-q,0,0)$ and $(1+q,0,0)$ at 2.5 K
in zero field. $\mbox{\boldmath$p$}_{\rm N}$ was set to be parallel to
$\kappa$, which was also parallel to the screw axis of the magnetic modulation
with $\mbox{\boldmath$q$}=(q,0,0)$. We also removed the analyzer in order to
measure SF intensities for both the spin-up and spin-down incident neutrons.
Note that the absence of NSF intensities was confirmed by the data shown in
Fig. 2(a). As shown in Fig. 4, the intensities did not depend on the direction
of $\mbox{\boldmath$p$}_{\rm N}$ for both reflections, indicating that $V^{\rm
RH}$ and $V^{\rm LH}$ are comparable to each other. This means that left-
handed and right-handed screw-type magnetic domains are nucleated with equal
volume fraction upon the magnetic phase transition to the elliptic screw-type
ground state in zero field, as expected from the centrosymmetric nature of the
crystal.

##  V Summary

We have studied magnetic structures in Gd2PdSi3 by means of polarized neutron
scattering near zero field. The satellite reflections observed in the present
neutron experiment are of purely magnetic origin. The ground state has
elliptic (distorted) screw-type magnetic modulations with the smaller
modulation amplitude along the $c$ axis. The spin component along the $c$ axis
decreases with increasing temperature, and disappears at a slightly lower
temperature than the transition to the paramagnetic phase. This indicates the
existence of an intermediate phase with the sinusoidal magnetic modulations.
These observations imply easy-plane type magnetic anisotropy in this system.
We have also confirmed that the volume fractions of left-handed and right-
handed screw-type modulations are equal to each other. A recent study on the
itinerate helimagnet MnP demonstrated that simultaneous applications of
electric current and magnetic field can control the spin helicity of the
magnetic structure Jiang  _et al._ (2020). Controlling the magnetic chirality
in centrosymmetric skyrmion hosts using this method will be an interesting
challenge for the future.

## Acknowledgements

This work was partly supported by Grants-in-Aid for Scientific Research
(Grants Nos. 24224009, 17K18355, 22H04463, and 21K13877) from JSPS and CREST
(Grant Nos. JPMJCR20T1 and JPMJCR1874) from JST. The neutron scattering
experiment at JRR-3 was carried out along the proposals Nos. 21507, 21401 and
22401 and partly supported by ISSP of the University of Tokyo. M.H. was
supported by the Mizuho Foundation for the Promotion of Sciences, and by the
Iketani Foundation.

## References

  * Kurumaji  _et al._ (2019) Takashi Kurumaji, Taro Nakajima, Max Hirschberger, Akiko Kikkawa, Yuichi Yamasaki, Hajime Sagayama, Hironori Nakao, Yasujiro Taguchi, Taka-hisa Arima, and Yoshinori Tokura, “Skyrmion lattice with a giant topological hall effect in a frustrated triangular-lattice magnet,” Science 365, 914 (2019).
  * Bogdanov and Yablonskii (1989) A. N. Bogdanov and D. A. Yablonskii, “Thermodynamically stable ‘vortices’ in magnetically ordered crystals. the mixed state of magnets.” Sov. Phys. JETP 68, 101–103 (1989).
  * Rößler  _et al._ (2006) U. K. Rößler, A. N. Bogdanov, and C. Pfleiderer, “Spontaneous skyrmion ground states in magnetic metals,” Nature 442, 797–801 (2006).
  * Mühlbauer  _et al._ (2009) S. Mühlbauer, B. Binz, F. Jonietz, C. Pfleiderer, A. Rosch, A. Neubauer, R. Georgii, and P. Böni, “Skyrmion lattice in a chiral magnet,” Science 323, 915–919 (2009).
  * Yu  _et al._ (2010) X. Z. Yu, Y. Onose, N. Kanazawa, J. H. Park, J. H. Han, Y. Matsui, N. Nagaosa, and Y. Tokura, “Real-space observation of a two-dimensional skyrmion crystal.” Nature 465, 901–904 (2010).
  * Münzer  _et al._ (2010) W. Münzer, A. Neubauer, T. Adams, S. Mühlbauer, C. Franz, F. Jonietz, R. Georgii, P. Böni, B. Pedersen, M. Schmidt, A. Rosch, and C. Pfleiderer, “Skyrmion lattice in the doped semiconductor ${\text{Fe}}_{1-x}{\text{Co}}_{x}\text{Si}$,” Phys. Rev. B 81, 041203(R) (2010).
  * Seki  _et al._ (2012) S. Seki, X. Z. Yu, S. Ishiwata, and Y. Tokura, “Observation of skyrmions in a multiferroic material,” Science 336, 198–201 (2012).
  * Kézsmárki  _et al._ (2015) I. Kézsmárki, S. Bordacs, P. Milde, E. Neuber, L. M. Eng, J. S. White, H. M. Ronnow, C. D. Dewhurst, M. Mochizuki, K. Yanai, H. Nakamura, D. Ehlers, V. Tsurkan, and A. Loidl, “Neel-type skyrmion lattice with confined orientation in the polar magnetic semiconductor GaV4S8,” Nat. Mater. 14, 1116–1122 (2015).
  * Tokunaga  _et al._ (2015) Y. Tokunaga, X. Z. Yu, J. S. White, H. M. Ronnow, D. Morikawa, Y. Taguchi, and Y. Tokura, “A new class of chiral materials hosting magnetic skyrmions beyond room temperature,” Nat. Commun. 6, 8638 (2015).
  * Kurumaji  _et al._ (2017) Takashi Kurumaji, Taro Nakajima, Victor Ukleev, Artem Feoktystov, Taka-hisa Arima, Kazuhisa Kakurai, and Yoshinori Tokura, “Néel-Type Skyrmion Lattice in the Tetragonal Polar Magnet ${\mathrm{VOSe}}_{2}{\mathrm{O}}_{5}$,” Phys. Rev. Lett. 119, 237201 (2017).
  * Hirschberger  _et al._ (2019) Max Hirschberger, Taro Nakajima, Shang Gao, Licong Peng, Akiko Kikkawa, Takashi Kurumaji, Markus Kriener, Yuichi Yamasaki, Hajime Sagayama, Hironori Nakao, Kazuki Ohishi, Kazuhisa Kakurai, Yasujiro Taguchi, Xiuzhen Yu, Taka-hisa Arima, and Yoshinori Tokura, “Skyrmion phase and competing magnetic orders on a breathing kagomé lattice,” Nat. Commun. 10, 5831 (2019).
  * Khanh  _et al._ (2020) Nguyen Duy Khanh, Taro Nakajima, Xiuzhen Yu, Shang Gao, Kiyou Shibata, Max Hirschberger, Yuichi Yamasaki, Hajime Sagayama, Hironori Nakao, Licong Peng, Kiyomi Nakajima, Rina Takagi, Taka-hisa Arima, Yoshinori Tokura, and Shinichiro Seki, “Nanometric square skyrmion lattice in a centrosymmetric tetragonal magnet,” Nat. Nanotech. 15, 444 (2020).
  * Okubo  _et al._ (2012) Tsuyoshi Okubo, Sungki Chung, and Hikaru Kawamura, “Multiple-$q$ states and the skyrmion lattice of the triangular-lattice heisenberg antiferromagnet under magnetic fields,” Phys. Rev. Lett. 108, 017206 (2012).
  * Leonov and Mostovoy (2015) A. O. Leonov and M. Mostovoy, “Multiply periodic states and isolated skyrmions in an anisotropic frustrated magnet,” Nat. Commun. 6, 8275 (2015).
  * Hayami  _et al._ (2017) Satoru Hayami, Ryo Ozawa, and Yukitoshi Motome, “Effective bilinear-biquadratic model for noncoplanar ordering in itinerant magnets,” Phys. Rev. B 95, 224424 (2017).
  * Nomoto  _et al._ (2020) Takuya Nomoto, Takashi Koretsune, and Ryotaro Arita, “Formation mechanism of the helical $q$ structure in gd-based skyrmion materials,” Phys. Rev. Lett. 125, 117204 (2020).
  * (17) Joseph A. M. Paddison, Binod K. Rai, Andrew F. May, Stuart A. Calder, Matthew B. Stone, Matthias D. Frontzek, Andrew D. Christianson, ”Magnetic Interactions of the Centrosymmetric Skyrmion Material Gd2PdSi3”, arXiv:2203.00066.
  * Saha  _et al._ (1999) S. R. Saha, H. Sugawara, T. D. Matsuda, H. Sato, R. Mallik, and E. V. Sampathkumaran, “Magnetic anisotropy, first-order-like metamagnetic transitions, and large negative magnetoresistance in single-crystal ${\mathrm{Gd}}_{2}{\mathrm{PdSi}}_{3}$,” Phys. Rev. B 60, 12162–12165 (1999).
  * (19) A previous X-ray study pointed out that the actual crystal structure exhibits a low-symmetry superstructure arising from the site ordering of Pd and Si GPS_Tang_PRB. neglected the superlattice structure of Pd and Si, and treated the system as a hexagonal system.
  * Yasui  _et al._ (2020) Yuuki Yasui, Christopher J. Butler, Nguyen Duy Khanh, Satoru Hayami, Takuya Nomoto, Tetsuo Hanaguri, Yukitoshi Motome, Ryotaro Arita, Taka-hisa Arima, Yoshinori Tokura, and Shinichiro Seki, “Imaging the coupling between itinerant electrons and localised moments in the centrosymmetric skyrmion magnet GdRu2Si2,” Nat. Commun. 11, 5925 (2020).
  * Hirschberger  _et al._ (2020) Max Hirschberger, Taro Nakajima, Markus Kriener, Takashi Kurumaji, Leonie Spitz, Shang Gao, Akiko Kikkawa, Yuichi Yamasaki, Hajime Sagayama, Hironori Nakao, Seiko Ohira-Kawamura, Yasujiro Taguchi, Taka-hisa Arima, and Yoshinori Tokura, “High-field depinned phase and planar Hall effect in the skyrmion host ${\mathrm{Gd}}_{2}{\mathrm{PdSi}}_{3}$,” Phys. Rev. B 101, 220401(R) (2020).
  * Spachmann  _et al._ (2021) S. Spachmann, A. Elghandour, M. Frontzek, W. Löser, and R. Klingeler, “Magnetoelastic coupling and phases in the skyrmion lattice magnet ${\mathrm{Gd}}_{2}{\mathrm{PdSi}}_{3}$ discovered by high-resolution dilatometry,” Phys. Rev. B 103, 184424 (2021).
  * Ishida  _et al._ (1985) Masayoshi Ishida, Yasuo Endoh, Setsuo Mitsuda, and Michiyoshi Ishikawa, Yoshikazu Tanaka, “Crystal chirality and helicity of the helical spin density wave in mnsi. ii. polarized neutron diffraction,” J. Phys. Soc. Jpn. 54, 2975 (1985).
  * Morikawa  _et al._ (2013) D. Morikawa, K. Shibata, N. Kanazawa, X. Z. Yu, and Y. Tokura, “Crystal chirality and skyrmion helicity in mnsi and (fe, co)si as determined by transmission electron microscopy,” Phys. Rev. B 88, 024408 (2013).
  * Squires (1978) G. L. Squires,  _Introduction to the theory of thermal neutron scattering_ (Dover publications inc., 1978).
  * (26) S. Moody, M. Wilson, M. Birch, M. Gomils̆ek, S. Collins, A. S̆tefanc̆ic̆, G. Balakrishnan, and P. Hatton, ”Charge Density Waves and Coplanar Magnetism in Gd2PdSi3”, arXiv:2010.14326.
  * Khanh  _et al._ (2022) Nguyen Duy Khanh, Taro Nakajima, Satoru Hayami, Shang Gao, Yuichi Yamasaki, Hajime Sagayama, Hironori Nakao, Rina Takagi, Yukitoshi Motome, Yoshinori Tokura, Taka-hisa Arima, and Shinichiro Seki, “Zoology of multiple-q spin textures in a centrosymmetric tetragonal magnet with itinerant electrons,” Adv. Sci. 9, 2105452 (2022).
  * Hayami and Motome (2019) Satoru Hayami and Yukitoshi Motome, “Effect of magnetic anisotropy on skyrmions with a high topological number in itinerant magnets,” Phys. Rev. B 99, 094420 (2019).
  * Wang  _et al._ (2020) Zhentao Wang, Ying Su, Shi-Zeng Lin, and Cristian D. Batista, “Skyrmion Crystal from RKKY Interaction Mediated by 2D Electron Gas,” Phys. Rev. Lett. 124, 207201 (2020).
  * Hirschberger  _et al._ (2021) Max Hirschberger, Satoru Hayami, and Yoshinori Tokura, “Nanometric skyrmion lattice from anisotropic exchange interactions in a centrosymmetric host,” New Journal of Physics 23, 023039 (2021).
  * Jiang  _et al._ (2020) N. Jiang, Y. Nii, H. Arisawa, E. Saitoh, and Y. Onose, “Electric current control of spin helicity in an itinerant helimagnet,” Nat. Commun. 11, 1601 (2020).