[table]capposition=TOP

# UniHCP: A Unified Model for Human-Centric Perceptions

Yuanzheng Ci1, Yizhou Wang2,3††footnotemark: , Meilin Chen2, Shixiang Tang1,
Lei Bai3, Feng Zhu4, Rui Zhao4,5,
Fengwei Yu4, Donglian Qi2, Wanli Ouyang3
1The University of Sydney, 2Zhejiang University, 3Shanghai AI Laboratory,
4SenseTime Research,
5Qing Yuan Research Institute, Shanghai Jiao Tong University, Shanghai, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
contribution.Corresponding author.

###### Abstract

Human-centric perceptions (e.g., pose estimation, human parsing, pedestrian
detection, person re-identification, etc.) play a key role in industrial
applications of visual models. While specific human-centric tasks have their
own relevant semantic aspect to focus on, they also share the same underlying
semantic structure of the human body. However, few works have attempted to
exploit such homogeneity and design a general-propose model for human-centric
tasks. In this work, we revisit a broad range of human-centric tasks and unify
them in a minimalist manner. We propose UniHCP, a Unified Model for Human-
Centric Perceptions, which unifies a wide range of human-centric tasks in a
simplified end-to-end manner with the plain vision transformer architecture.
With large-scale joint training on 33 human-centric datasets, UniHCP can
outperform strong baselines on several in-domain and downstream tasks by
direct evaluation. When adapted to a specific task, UniHCP achieves new SOTAs
on a wide range of human-centric tasks, e.g., 69.8 mIoU on CIHP for human
parsing, 86.18 mA on PA-100K for attribute prediction, 90.3 mAP on Market1501
for ReID, and 85.8 JI on CrowdHuman for pedestrian detection, performing
better than specialized models tailored for each task. The code and pretrained
model are available at https://github.com/OpenGVLab/UniHCP.

##  1 Introduction

Figure 1: UniHCP unifies 5 human-centric tasks under one model and is trained
on a massive collection of human-centric datasets.

Research on human-centric perceptions has come a long way with tremendous
advancements in recent years. Many methods have been developed to enhance the
performance of pose estimation [67], human parsing [52], pedestrian detection
[6], and many other human-centered tasks. These significant progress play a
key role in advancing the applications of visual models in numerous fields,
such as sports analysis [12], autonomous driving [105], and electronic
retailing [31].

Although different human-centric perception tasks have their own relevant
semantic information to focus on, those semantics all rely on the same basic
structure of the human body and the attributes of each body part [69, 89]. In
light of this, there have been some attempts trying to exploit such
homogeneity and train a shared neural network jointly with distinct human-
centric tasks [77, 33, 52, 83, 106, 68, 95, 54, 32]. For instance, human
parsing has been trained in conjunction with human keypoint detection [52,
106, 68], pedestrian attribute recognition [95], pedestrian detection [54] or
person re-identification [32] (ReID). The experimental results of these works
empirically validate that some human-centric tasks may benefit each other when
trained together. Motivated by these works, a natural expectation is that a
more versatile all-in-one model could be a feasible solution for general
human-centric perceptions, which can utilize the homogeneity of human-centric
tasks for improving performance, enable fast adaption to new tasks, and
decrease the burden of memory cost in large-scale multitask system deployment
compared with specific models to specific tasks.

However, unifying distinct human-centric tasks into a general model is
challenging considering the data diversity and output structures. From the
data’s perspective, images in different human-centric tasks and different
datasets have different resolutions and characteristics (e.g., day and night,
indoor and outdoor), which calls for a robust representative network with the
capability to accommodate them. From the perspective of output, the
annotations and expected outputs of different human-centric tasks have
distinct structures and granularities. Although this challenge can be bypassed
via deploying separate output heads for each task/dataset, it is not scalable
when the number of tasks and datasets is large.

In this work, we aim to explore a simple, scalable formulation for unified
human-centric system and, for the first time, propose a Unified model for
Human-Centric Perceptions (UniHCP). As shown in Figure.1, UniHCP unifies and
simultaneously handles five distinct human-centric tasks, namely, pose
estimation, semantic part segmentation, pedestrian detection, ReID, and person
attribute recognition. Motivated by the extraordinary capacity and flexibility
of the vision transformers [49, 101], a simple yet unified encoder-decoder
architecture with the plain vision transformer is employed to handle the input
diversity, which works in a simple feedforward and end-to-end manner, and can
be shared across all human-centric tasks and datasets to extract general
human-centric knowledge. To generate the output for different tasks with the
unified model, UniHCP defines Task-specific Queries, which are shared among
all datasets with the same task definition and interpreted into different
output units through a Task-guided Interpreter shared across different
datasets and tasks. With task-specific queries and the versatile interpreter,
UniHCP avoids the widely used task-specific output heads, which minimizes
task-specific parameters for knowledge sharing and make backbone-encoded
features reusable across tasks.

Own to these designs, UniHCP is suitable and easy to perform multitask
pretraining at scale. To this end, we pretrained an UniHCP model on a massive
collection of 33 labeled human-centric datasets. By harnessing the abundant
supervision signals of each task, we show such a model can simultaneously
handle these in-pretrain tasks well with competitive performance compared to
strong baselines relying on specialized architectures. When adapted to a
specific task, both in-domain and downstream, our model achieves new SOTAs on
several human-centric task benchmarks. In summary, the proposed model has the
following properties:

  * ◼

Unifying five distinct human-centric tasks and handling them simultaneously.

  * ◼

Shared encoder-decoder network based on plain transformer.

  * ◼

Simple task-specific queries identifying the outputs.

  * ◼

Maximum weight sharing (99.97% shared parameters) with a task-guided
interpreter.

  * ◼

Trainable at scale and demonstrates competitive performance compared to task-
specialized models.

The following sections are organized as follows: Section 2 reviews the related
works with focuses on Human-Centric perception and unified models. Section 3
describes the proposed model. Section 4 provides implementation details
together with empirical results and ablation studies. Finally, we conclude the
paper in Section 5.

##  2 Related Works

###  2.1 Human-Centric Perceptions

Human-centric perceptions are essential for substantial real-world
applications. Depending on the targeted visual concept, the way of decoding
output from image features varies across tasks. Specifically, pose estimation
and pedestrian detection are both localization tasks that can be solved by
either regression-based methods [41, 103] or heatmap-based methods [93, 37,
38]. Human parsing, as a fine-grained segmentation problem, is usually solved
by per-pixel classification. While contour-based methods [94, 70] can also
obtain segmentation masks, it requires instance-level mask annotations, which
are not always available. PAR is treated as a multi-label classification task
[116], and ReID is treated as a feature learning task [81].

Recently, several transformer-based solutions have been proposed for these
human-centric tasks, with attention block designs on both backbone [100, 23,
96] and decoding network [50, 66, 60, 46, 95, 110]. However, these methods
involve different task-specific designs and thus cannot be integrated into one
model seamlessly. Built upon the general success of these works, we take a
further step and unify human-centric tasks under the same architecture based
on plain vision transformer.

Figure 2: UniHCP handles a massive collection of human-centric tasks uniformly
by task-specific queries and a task-guided interpreter, all predictions are
yielded in parallel through a simple encoder-decoder transformer architecture.

###  2.2 Unified Models

A general-purpose model that can handle different tasks in a unified manner
has long been a coveted alternative to models specifically tailored for
different tasks. Pioneering works regarding Natural Language Processing (NLP)
[71], vision-language [65], and basic vision tasks [73, 34] have shown the
effectiveness of such kind of unified cross-task models. ExT5 [3] and OFA [88]
further provide a degree of promise for the performance benefits of large-
scale multitask co-training. Among models supporting visual tasks, UniHead
[51] and UViM [35] propose a unified architecture for several vision tasks.
However, they are only trained and evaluated in a single-task manner.

For methods supporting multitask co-training, Uni-Perceiver [118] focuses on
tasks in which the desired output is inherently language or labels, which does
not fit human-centric tasks. While UniT [25], OFA [88], Unified-IO [64], and
Pix2Seq v2 [8] further extend the support for detection, keypoint detection,
segmentation, and many other visual tasks, they rely on _independent decoder
heads_ [25, 88] or _autoregressive_ modeling [8, 64]. These works do not focus
on human-centric vision tasks. Differently, our work introduces a _shared
decoder head_ (task-guided interpreter) in a _parallelly feedforward_ manner
for human-centric vision tasks, which is simple yet maximizes the parameter
sharing among different tasks.

In the case of human-centric tasks, many works have shown great success by co-
training a pair of human-centric tasks [77, 33, 52, 83, 106, 68, 95, 54, 32].
However, there is no work exploring a general unified model that can handle
all representative human-centric tasks. Our work is the first attempt at
designing, training, and evaluating a unified human-centric model with a
large-scale multitask setting.

##  3 UniHCP

Table 1: Network details of UniHCP | Layers | Dimension | Params
---|---|---|---
Encoder | 12 | 768 | 91.1M
Decoder | 9 | 256 | 14.5M
Task-guided Interpreter |  |  | 3.5M
Task-specific queries |  | 256 | <0.03M
Total |  |  | 109.1M
Task-agnostic params / total params | 99.97%

To share the most knowledge among various human-centric tasks, we attempt to
maximize weight sharing among all tasks in UniHCP. Specifically, our UniHCP,
as shown in Figure 2, consists of three components: (1) A task-agnostic
transformer encoder $E$ to extract image features. (2) A transformer decoder
$D$ that attends to task-specific information according to task-specific
queries $\\{\mathbf{Q}^{t}\\}$, where $t$ denotes a specific task. (3) A task-
guided interpreter $\mathcal{I}$ produces output units, in which we decompose
the output of multiple human-centric perception tasks into sharable units of
diverse granularities, _i.e.,_ feature representation, local probability map,
global probability, bounding box coordinates. Since only the queries to the
decoders are not shared among tasks, we can learn human-centric knowledge
across different granularities by the designed interpreters and achieve
maximum parameter sharing among all tasks, i.e., 99.97% shared parameters, as
shown in Table 1. The pipeline for our UniHCP is described as follows.

_Step 1_ : Given an image $\mathbf{X}$ sampled from the dataset in task $t$,
extract encoded features $\mathbf{F}$ by the task-agnostic transformer encoder
$E$ (Sec. 3.1).

_Step 2_ : A transformer decoder $D$ with task-specific queries
$\mathbf{Q}^{t}$ extracts task-specific features from encoded features
$\mathbf{F}$ (Sec. 3.2).

_Step 3_ : Generate output units according to the queried task, i.e., attended
features $\mathbf{Y}_{f}$, local probability map $\mathbf{Y}_{m}$, global
probability $\mathbf{Y}_{p}$ and bounding box coordinates $\mathbf{Y}_{bbox}$
by a task-guided interpreter $\mathcal{I}$ (Sec. 3.3). For example, for human
parsing, two units: local probability map $\mathbf{Y}_{m}$ (for semantic part
segmentation) and global probability $\mathbf{Y}_{p}$ (for existence of body
part in the image), are generated.

_Step 4:_ Calculate the loss of the corresponding task for optimizing the
encoder $E$, the decoder $D$, the task-specific queries $\mathbf{Q}^{t}$ and
task-guided interpreter $\mathcal{I}$ by backward propagation (Sec. 3.4).

###  3.1 Task-agnostic Transformer Encoder

UniHCP uses a plain Vision Trasnformer [15] (ViT) as the encoder. To handle
input images of different resolutions, we use a shared learnable positional
embedding with the size of $84\times 84$ and interpolate it based on the
spatial size of the input image after patch projection. The encoded feature
$\mathbf{F}$ can be mathematically calculated as

$\mathbf{F}=E(\mathbf{X},\mathbf{P}_{E}),$ (1)

where $\mathbf{P}_{E}$ is the positional embedding after interpolation and $E$
denotes the task-agnostic transformer encoder.

###  3.2 Decoder with Task-specific Queries

To obtain the most discriminative feature for each task while maximizing
knowledge sharing, we design task-specific queries to guide the transformer
decoder only attending to task-relevant information.

Task-specific Queries. Task queries for task $t$ are denoted as

$\mathbf{Q}^{t}=[\mathbf{q}^{t}_{1},\mathbf{q}^{t}_{2},...,\mathbf{q}^{t}_{N^{t}}],$
(2)

where $N^{t}$ denotes the number of queries representing $N^{t}$ different
semantic meanings in task $t$. For pedestrian attribute recognition, pose
estimation, human parsing, and ReID, the number of queries respectively equals
to the number of attributes, the number of pose joints, the number of semantic
parsing classes, and the length of desired ReID features. For pedestrian
detection, we follow the implementation in [90], with details provided in the
supplementary material. We randomly initialize the task-specific query
$\mathbf{Q}^{t}$ as learnable embeddings $\mathbf{Q}_{0}^{t}$ and refine it
with the following decoder blocks.

Following the common practice as in [84, 10, 90], all $\mathbf{Q}^{t}$ are
also associated with a positional embedding $\mathbf{Q}^{t}_{p}$ , which has
the same dimension as $\mathbf{Q}^{t}$ and is not shared across tasks.
Different from $\mathbf{Q}^{t}$ that will be progressively refined in the
decoder blocks, $\mathbf{Q}^{t}_{p}$ is shared across decoder blocks. For
tasks other than pedestrian detection, $\mathbf{Q}^{t}_{p}$ is simply a
learnable positional embedding that is randomly initialized. For pedestrian
detection, we have

$\mathbf{Q}^{t}_{p}=proj(\mathcal{A}_{\mathbf{Q}}),$ (3)

where $\mathcal{A}_{\mathbf{Q}}\in\mathbb{R}^{N^{t}\times 2}$ refers to
$N^{t}$ learnable anchor points that are initialized with a uniform
distribution following [90], and $proj$ is a projection from coordinates to
positional embeddings (more details about the projector are elaborated in the
supplementary materials).

Decoder. The transformer decoder aims to attend to task-specific features
according to the task queries. We follow the standard design of transformer
decoders [84]. In the decoder, each transformer block $D_{l}$ for
$l=1,2,...,L$ consists of a cross-attention module, a self-attention module,
and a feed-forward module (FFN), where $L$ denotes the number of transformer
blocks. We place cross-attention before self-attention as adopted by [10, 40].
For each block $D_{l}$, we attend to task-specific information from the
encoded feature by task queries, which can be formulated as

$\mathbf{Q}_{l}^{t}=D_{l}(\mathbf{Q}^{t}_{l-1},\mathbf{Q}^{t}_{p},\mathbf{F},\mathbf{F}_{p}),$
(4) $\textrm{where }\mathbf{F}_{p}=proj(\mathcal{A}_{\mathbf{\mathbf{F}}}),$
(5)

$\mathcal{A}_{\mathbf{\mathbf{F}}}\in\mathbb{R}^{H_{\mathbf{F}}W_{\mathbf{F}}\times
2}$ is the coordinates with respect to the original image for all feature
tokens in $\mathbf{F}\in R^{H_{\mathbf{F}}\times W_{\mathbf{F}}}$. For the
cross-attention in the decoder $D_{l}$, the query is
$\hat{\mathbf{Q}}^{t}_{l}=\mathbf{Q}^{t}_{l-1}+\mathbf{Q}^{t}_{p}$, the key is
$\hat{\mathbf{K}}=\mathbf{F^{\prime}}+\mathbf{F}_{p}$, and the value is
$\hat{\mathbf{V}}=\mathbf{F^{\prime}}$, where $\mathbf{F}^{\prime}$ is
linearly projected from the features of the encoder $\mathbf{F}$ to align
channel dimensions. The result of cross-attention is then passed for self-
attention in $D_{l}$.

Figure 3: Task-guided interpreter. $\otimes$ denotes a dynamic convolution
module [9] that takes the projected query feature as the kernel and takes the
tokens $\mathbf{F}$ from the encoder as the feature map, where $\mathbf{F}$ is
upscaled to the desired resolution $H^{\prime}\times W^{\prime}$ , $\oplus$
denotes addition, for which the inputs are the projected query feature in the
format of $[\nabla cx,\nabla cx,h,w]$ and $\mathcal{A}_{\mathbf{Q}}$, which
contains the anchor point $[cx,cy]$ (see supplementary materials for details).

###  3.3 Task-guided Interpreter

Task-guided interpreter $\mathcal{I}$ interprets query tokens $\mathbf{Q}^{t}$
into four output units subject to the request of a specific task. As shown in
Figure 3, these four output units are as follows:

$\displaystyle\text{feature vector unit}:$ $\displaystyle\
\mathbf{Y}_{f}\in\mathbb{R}^{N^{t}\times C}$ (6) $\displaystyle\text{global
probability unit}:$ $\displaystyle\ \mathbf{Y}_{p}\in\mathbb{R}^{N^{t}\times
1}$ $\displaystyle\text{local probability map unit}:$ $\displaystyle\
\mathbf{Y}_{m}\in\mathbb{R}^{N^{t}\times H^{\prime}\times W^{\prime}}$
$\displaystyle\text{bounding box unit}:$ $\displaystyle\
\mathbf{Y}_{bbox}\in\mathbb{R}^{N^{t}\times 4},$

where $C$ is the output dimension of the decoder, $H^{\prime}\times
W^{\prime}$ denotes the desired resolution for the local probability map.
Given task $t$ and output interpreter $\mathcal{I}$, the output of the UniHCP
is defined as follows:

$\displaystyle\\{\mathbf{Y}_{u}|g_{u}^{\mathbf{t}_{t}}=1,u\in\\{f,p,m,bbox\\}\\}=\mathcal{I}(\mathbf{Q}^{t},\mathbf{g}^{\mathbf{t}_{t}}),$
(7)

where $\mathbf{t}_{t}\in\\{reid,\ldots,pose\\}$ denotes the task type of task
$t$, $\mathbf{g}^{\mathbf{t}}=\\{g^{\mathbf{t}}_{u}\\}$ is a set of task-
specific binary gates ($g\in\\{0,1\\}$) that represents the desired output
units for task type $\mathbf{t}$.

Guidance from tasks to output units. For human parsing, local probability map
(for semantic part segmentation) and global probability (for existence of body
part in the image) are activated, corresponding to $g^{seg}_{m}=1$ and
$g^{seg}_{p}=1$ respectively. For person ReID, feature vectors are used,
corresponding to $g^{reid}_{f}=1$. For pose estimation, $g^{pose}_{m}=1$ (for
localizing key points) and $g^{pose}_{p}=1$ (for existence of keypoints in the
image). For detection, $g^{det}_{bbox}=1$ (for bounding box prediction) and
$g^{det}_{p}=1$ (for existence of object). For pedestrian attribute
prediction, $g^{par}_{p}=1$ (for existence of attributes in the image).
Therefore, the output unit of global probabilities is shared among pose
estimation, human parsing, pedestrian detection, and attribute recognition.
The output unit of local probability maps is shared among pose estimation and
human parsing.

Discussion. The task-guided interpreter interprets each query token
independently. Previous works focused on autoregressive decoding with
tokenization [8, 64] or task-specific heads [99, 25] to handle different
output units required by specific tasks. In contrast, the task-guided
interpreter can handle tasks involving a varying number of classes, yield all
results in parallel, and do not require task-specific heads. This is achieved
by two designs in our UniHCP framework: 1) Class/instance information is self-
contained in queries. As mentioned in Section 3.2, a query represents a
particular semantic class in pose estimation, attribute prediction, human
parsing, and pedestrian detection. We only need to retrieve a scalar
probability value from a query to obtain the confidence information for a
particular class/human instance. 2) Outputs of the same modality share the
same output unit. For example, the heatmap for a particular joint in pose
estimation and the heatmap for a particular body part in human parsing have
the same dimension. Although these outputs have different meanings,
experimental results in Section 4.3 show that it is suitable to obtain them
through the same output unit and fully let the task-specific queries handle
the differences in preferred information to be represented.

###  3.4 Objective Functions

In this section, we will introduce the objective functions for training
diverse human-centric tasks together and illustrate how these objectives are
related to the output units defined in Eq. 6. Unless otherwise specified, we
omit the GT inputs in loss functions for brevity.

Overall Objective Function. Given a collection of datasets
$\mathbb{D}=\\{\mathcal{D}|\mathbf{t}_{\mathcal{D}}\in\\{reid,\ldots,pose\\}\\}$,
where $\mathbf{t}_{\mathcal{D}}$ denotes the task type of dataset
$\mathcal{D}$, we also note $t_{\mathcal{D}}$ as the task of dataset
$\mathcal{D}$, we have the overall loss defined as:

$\mathcal{L}=\sum_{\mathcal{D}\in\mathbb{D}}w_{D}\mathcal{L}_{\mathbf{t}_{\mathcal{D}}}(\mathcal{I}(\mathbf{Q}^{t_{\mathcal{D}}},\mathbf{g}^{\mathbf{t}_{\mathcal{D}}})),$
(8)

where $w_{\mathcal{D}}$ is the loss weight for dataset $\mathcal{D}$, which is
calculated based on the task type and batch size (calculations are elaborated
in supplementary materials).

##### ReID.

Person ReID is a feature learning task for extracting identification
information. Therefore, we directly supervised the features after the decoder
by identity annotations. Specifically, for ReID task, the extracted feature is
a simple concatenation of all feature vectors
$\mathbf{Y}_{f}=[y_{f}^{1};\ldots;y_{f}^{N^{t}}]$, where $N^{t}=6$ by default.
The loss function is a combination of ID loss [114] and triplet loss [58]
written as follows:

$\displaystyle\mathcal{L}_{reid}=\mathcal{L}_{ID}(\mathbf{Y}_{f})+\mathcal{L}_{triplet}(\mathbf{Y}_{f}).$
(9)

##### PAR.

Pedestrian attribute recognition only predicts whether an attribute exists in
the global image. Therefore, we only supervise the output unit of global
probabilities $\mathbf{Y}_{p}$ from the task-guided interpreter. Specifically,
following the common practice [82, 46], we adopt the weighted binary cross-
entropy loss. Given the probability predictions $\mathbf{Y}_{p}$ associated
with $N^{t}$ attributes, we have:

$\displaystyle\mathcal{L}_{par}$
$\displaystyle=\sum_{n=1}^{N_{t}}w_{n}(y_{n}\log(y_{p}^{n})+(1-y_{n})\log(1-y_{p}^{n})),$
(10) $\displaystyle w_{n}$
$\displaystyle=y_{n}e^{1-\gamma_{n}}+(1-y_{n})e^{\gamma_{n}},$

where $y_{n}$ denotes the annotation of $n$-th attribute and $\gamma_{n}$
denotes the positive example ratio of $n$-th attribute.

##### Human Parsing.

Human parsing can be considered as semantic segmentation of human part. We
view the presence of semantic classes as predictable attributes since the
semantic classes are not always present in an image. Therefore, the global
probability $\mathbf{Y}_{p}$ and local probability map $\mathbf{Y}_{m}$ are
selected from the output units to describe whether a semantic part exists on
the image level (global) and pixel level (local), respectively. Given a query
$\mathbf{q}_{l}$ defined in Eq. 2 which corresponds to a semantic class in
human parsing, we adopt the binary cross entropy loss as $\mathcal{L}_{par}$
in pedestrian attribute recognition to constrain the global probability
$\mathbf{Y}_{p}$, and a combination of binary cross-entropy loss and dice loss
[10] to supervised local probability map $\mathbf{Y}_{m}$ as follows:

$\displaystyle\mathcal{L}_{seg}=\lambda_{par}\mathcal{L}_{par}(\mathbf{Y}_{p})+\mathcal{L}_{bce}(\mathbf{Y}_{m})+\mathcal{L}_{dice}(\mathbf{Y}_{m}),$

where $\lambda_{par}$ denotes the loss weight for
$\mathcal{L}_{par}(\mathbf{Y}_{p})$.

##### Pose Estimation.

We follow the common top-down setting for pose estimation, i.e., predicting
keypoints based on the cropped human instances. We predict the heatmap w.r.t.
the keypoints via mean-squared error. Similar to human parsing formulation, we
also select the global probability $\mathbf{Y}_{p}$ and local probability map
$\mathbf{Y}_{m}$ to predict whether a keypoint exists in the image level and
pixel level, respectively. Mathematically, we have:

$\displaystyle\mathcal{L}_{pose}=\lambda_{par}\mathcal{L}_{par}(\mathbf{Y}_{p})+\mathcal{L}_{mse}(\mathbf{Y}_{m}).$
(11)

##### Pedestrian Detection.

Pedestrian Detection is a local prediction task but in a sparse manner.
Following the widely adopted designs in end-to-end transformer-based detection
[7, 110], ground-truth for $N^{t}$ query features in $\mathbf{Q}_{l}$ are
determined by optimal bipartite matching between all $N^{t}$ predictions and
GT boxes. Given output units $\mathbf{Y}_{p}$ and $\mathbf{Y}_{bbox}$, we
adopt the identical cost formulation and loss as in [110],

$\displaystyle\mathcal{L}_{peddet}=$
$\displaystyle\lambda_{cls}\mathcal{L}_{cls}(\mathbf{Y}_{p})+\lambda_{iou}\mathcal{L}_{iou}(\mathbf{Y}_{bbox})+$
(12) $\displaystyle\lambda_{L1}\mathcal{L}_{L1}(\mathbf{Y}_{bbox}).$

where $\mathcal{L}_{cls},\mathcal{L}_{iou}$ and $\mathcal{L}_{L1}$ are focal
loss [56], GIoU loss [72], and $L1$ loss, respectively. Their corresponding
loss weights $\lambda$ are also identically set as in [110].

##  4 Experiments

###  4.1 Implementation details

Datasets. To enable general human-centric perceptions, we pretrain the
proposed UniHCP at scale on a massive and diverse collection of human-centric
datasets. Specifically, the training splits of 33 publically available
datasets are gathered to form the training set for UniHCP, including nine
datasets for pose estimation and six datasets for ReID, Human Parsing,
Attribute Prediction, Pedestrain Detection, seraprately. For ReID, there are
two different sub-tasks: general ReID and cloth-changing ReID, where the
difference is whether cloth-change is considered for person ReID. We
empirically found it is best to view them as different tasks and solve them
with different task queries. Hence, we treat these two sub-tasks as different
tasks and give them separate queries.

We carefully follow the de-duplication practices as introduced in [102] to
remove the samples that could appear in the evaluation datasets. We also
remove images whose groundtruth labels are not given, leading to 2.3M distinct
training samples in total. For evaluation, apart from the available validation
or test splits of the 33 training sets, we also included several out-of-
pretrain downstream datasets for each type of human-centric task. More details
about dataset setups can be found in supplementary materials.

Table 2: Representative datasets used in multitask co-training. Task Type | Datasets | Number of samples
---|---|---
ReID (6 datasets) |  CUHK03 [47] | 268,002
DGMarket [113]
PRCC [97]
…
Pose Estimation (9 datasets) |  COCO-Pose [57] | 1,261,749
AI Challenger [92]
PoseTrack [1]
…
Human Parsing (6 datasets) |  LIP [20] | 384,085
CIHP [19]
DeepFashion2 [18]
…
Attribute Prediction (6 datasets) |  PA-100K [62] | 242,880
RAPv2 [39]
UAV-Human [45]
…
Pedestrian Detection (6 datasets) |  COCO-Person [57] | 170,687
CrowdHuman [74]
WiderPedestrian [63]
…

Training. We use the standard ViT-B [15] as the encoder network and initialize
it with the MAE pretrained [22] weights following [96, 49]. For the main
results, we use a batch size of 4324 in total, with the dataset-specific batch
size being proportional to the size of each dataset. Unless otherwise
specified, the image resolution used in pretraining is $256\times 192$ for
pose estimation and attribute prediction, $256\times 128$ for ReID, $480\times
480$ for human parsing, and a maximum height/width of 1333 for pedestrian
detection.

For computational efficiency, each GPU only runs one specific task, and each
task can be evenly distributed to multiple GPUs whereas a single GPU is not
capable of handling its workloads. To further save the GPU memory during the
training time, we adopt the gradient checkpointing [4] in the encoder forward
pass among all tasks and additionally use accumulative gradients for detection
tasks. Due to the high GPU-memory demand of detection datasets, the batch size
for the detection task is timed by 0.6.

We use Adafactor [75] optimizer and follow the recommended modifications [101]
for adopting it to ViT, we set $\beta_{1}=0.9$, $\beta_{2}$ clipped at
$0.999$, disables the parameter scaling and decoupled weight decay to 0.05. We
linearly warm up the learning rate for the first 1500 iterations to 1e-3,
after which the learning rate is decayed to 0 following a cosine decay
scheduler. We also use a drop-path rate of 0.2 and layer-wise learning rate
decay [49, 96] of 0.75 in the ViT-B encoder. For the main results, the whole
training process takes 105k iterations which are approximately 130 epochs for
detection datasets and 200 epochs for other datasets. The whole training takes
120 hours in total on 88 NVIDIA V100 GPUs.

table[0.755]  Method Market1501 MSMT17 CUHK03 HOReID [87] 84.9 - - MNE [44] -
- 77.7 SAN [30] 88.0 55.7 76.4 TransReID [23] 86.8 61.0 - TransReID† [23] 88.9
67.4 - UniHCP (direct eval) 80.7 55.2 68.6 UniHCP (finetune) 90.3 67.3 83.1
figure[.57]

Table 3: Person ReID evaluation on Market1501, MSMT, CUHK03 with mAP.
†indicates using additional camera IDs.

Table 4: Pedestrian attribute recognition evaluation on PA-100K and RAPv2 test
sets with mA.  Table 5: Human parsing evaluation on Human3.6M, LIP and CIHP
val sets with mIoU.

Method | PA-100K | RAPv2
---|---|---
SSC [28] | 81.87 | -
C-Tran [36] | 81.53 | -
Q2L [60] | 80.72 | -
L2L [46] | 82.37 | -
DAFL [29] | 83.54 | 81.04
UniHCP (direct eval) | 79.32 | 77.20
UniHCP (finetune) | 86.18 | 82.34

table[0.645]  Method H3.6M LIP CIHP HCMOCO [24] 62.50 - - SNT [27] - 54.73
60.87 PCNet [108] - 57.03 61.05 SCHP [43] - 59.36 - CDGNet [59] - 60.30 65.56
UniHCP (direct eval) 65.90 63.80 68.60 UniHCP (finetune) 65.95 63.86 69.80

table[0.8]  Method mAP MR${}^{-2}(\downarrow)$ JI DETR [7] 75.9 73.2 74.4 PEDR
[55] 91.6 43.7 83.3 Deformable-DETR [117] 91.5 43.7 83.1 Sparse-RCNN [80] 91.3
44.8 81.3 Iter-Deformable-DETR [110] 92.1 41.5 84.0 Iter-Sparse-RCNN [110]
92.5 41.6 83.3 UniHCP (direct eval) 90.0 46.6 82.2 UniHCP (finetune) 92.5 41.6
85.8 figure[1.22]

Table 6: Pedestrian detection evaluation on CrowdHuman val set. Compared with
the SOTA, UniHCP achieves comparable mAP and better JI.

Table 7: Pose estimation evaluation on COCO, Human3.6M, AI Challenge and
OCHuman. Following [96], we report the results on COCO val set, Human3.6M, AI
Challenge val set, and OCHuman test set. †denotes the results reported by
MMPose [11]. ‡denotes the results achieved using multi-dataset training.

Method | COCO/mAP |  H3.6M/EPE($\downarrow$) | AIC/mAP | OCHuman/mAP
---|---|---|---|---
HRNet-w32† [79] | 74.4 | 9.4 | - | -
HRNet-w48† [79] | 75.1 | 7.4 | - | -
TokenPose-L/D24 [50] | 75.9 | - | - | -
HRFormer-B [100] | 75.6 | - | - | -
ViTPose-B [96] | 75.8 | - | - | -
ViTPose-B‡ [96] | 77.1 | - | 32.0 | 87.3
UniHCP (direct eval) | 76.1 | 6.9 | 32.5 | 87.4
UniHCP (finetune) | 76.5 | 6.6 | 33.6 | N/A

###  4.2 Main Results

To demonstrate the capability of UniHCP as a unified model for human-centric
perceptions, we first evaluate our UniHCP on thirteen datasets that appear in
the pretraining stage (in Section 4.2.1), _e.g._ , CIHP. Furthermore, we
employ five datasets whose training splits are not included in the pretraining
stage to evaluate the cross-datasets transferability of UniHCP (in Section
4.2.2). We also demonstrate that UniHCP has the potential to efficiently
transfer to new datasets that do not appear in pretraining with only a few
images (in Section 4.2.3). For detailed evaluation configuration, please refer
to the supplementary.

####  4.2.1 In-pretrain Dataset Results

We conduct extensive evaluations on thirteen in-pretrain datasets to
demonstrate the effectiveness of our UniHCP. Table 5-7 summarize the
evaluation results of UniHCP on five representative human-centric tasks,
_i.e._ , person ReID, pedestrian attribute recognition, human parsing,
pedestrian detection, and pose estimation. We report two kinds of evaluation
results of our UniHCP: (1) direct evaluation, where the pre-trained model with
cross-task shared encoder-decoder weights and task-specific queries are
directly used for evaluation on the target dataset, and (2) finetuning, where
the pretrained UniHCP are first finetuned with the train split of the target
dataset and then evaluated.

As observed, the direct evaluation results of UniHCP show promising
performance on most human-centric tasks, especially on human parsing and pose
estimation tasks, which show better or on-par performance with the State-Of-
The-Art (SOTA). The exception is the person ReID task, which observes
noticeable performance gaps with the SOTA. We suggest this is due to its huge
disparity from other tasks and can be remedied with quick finetuning.

With finetuning, our UniHCP achieves new SOTAs on nine out of the total twelve
datasets and on par performance on the rest three datasets, even without task-
specific design in architecture or task-specific priors, showing that UniHCP
extracts complementary knowledge among human-centric tasks. Concretely, Table
4 shows that in the human attribute recognition task, UniHCP significantly
surpasses previous SOTA DAFL [29] by +3.79% mA on PA-100K and +1.20% mA on
RAPv2 datasets, respectively, which indicates that UniHCP well extracts the
shared attribute information among using the output unit of global
probabilities in the interpreter. Second, UniHCP also pushes the performance
of another important human task, _i.e._ , human parsing, to a new level.
Specifically, +3.45% mIoU, +3.56% mIoU, and +4.24% mIoU performance gains are
observed on Human3.6M, LIP, and CIHP datasets, respectively. We suggest the
newly-added global supervision $\mathcal{L}_{par}$ will help UniHCP to
suppress the false prediction on not appeared semantic parts. UniHCP also
shows superior performance to previous methods on pose estimation. On person
ReID, UniHCP outperforms TransReid [23] on Market1501 and MNE [44] on CUHK03
without the help of any additional camera information and training images
during evaluation. For pedestrian detection, our UniHCP achieves +1.8% JI
performance gain compared with Iter-Deformable-DETR [110] and on-par
performance with the Iter-Sparse-RCNN [110] on mAP. These strong performances
on diverse datasets across five tasks demonstrate the feasibility and
powerfulness of the unified human-centric model and large-scale pretraining.

Table 8: Transfer performance on ATR [53], SenseReID [109], Caltech [14], MPII
[2] and PETA [13]. Results with †are achieved by using additional data. DE -
direct evaluation. FT - finetuning.

Methods | Parsing | ReID | Detection | Pose | Attribute
---|---|---|---|---|---
ATR | SenseReID |  Caltech($\downarrow$) | MPII | PETA
SOTA |  97.39 [59] |  34.6 [109] |  46.6 [21] |  92.3 [98] |  87.07 [29]
SOTA† | - | - |  28.8 [21] |  93.3 [96] | -
UniHCP (DE) | - | 46.0 | 37.8 | - | -
UniHCP (FT) | 97.74 | N/A | 27.2 | 93.2 | 88.78

Table 9: One-shot human parsing and human pose estimation transfer results
under different tuning settings. Every method uses only 1 image per class to
transfer. We repeat each experiment for 10 times and report the mean and
standard deviation.

Methods |  Learnable params ratio | Parsing | Pose
---|---|---|---
ATR/pACC | MPII/PCKh
One-shot finetuning | 100% |  90.49$\pm$1.22 |  70.6$\pm$7.53
One-shot prompt tuning | <1% |  93.65$\pm$0.77 |  83.8$\pm$5.08
Full-data finetuning | 100% | 97.74 | 93.2

####  4.2.2 Cross-datasets Transfer Results

As the task-guided interpreter formulates all the requests of human-centric
tasks into four output units, human-centric knowledge learned behind these
units can be easily transferred to unseen datasets. We conduct evaluations on
another five datasets which do not appear during pretraining to evaluate the
transferability of UniHCP. UniHCP is finetuned to adapt to new datasets except
for SenseReID, on which the performance is tested by direct evaluation. As
shown in Table 8, UniHCP outperforms existing SOTAs in 4 out of 5 datasets.
Specifically, UniHCP achieves +0.35% pACC, +11.4% top-1, -1.6% heavy occluded
MR${}^{-2}(\downarrow)$, +0.1% PCKh, and +1.71% mA on ATR, SenseReID, Caltech,
MPII, and PETA, respectively. On MPII, UniHCP achieves on-par performance with
multi-datasets trained SOTA while improving single-dataset trained SOTA by
+0.9% PCKh. Notably, even without finetuning, UniHCP achieves a -8.8% heavy
occluded MR${}^{-2}(\downarrow)$ performance gain on single-dataset trained
SOTA. Consistent improvements on transfer tasks provide strong support to the
decent transferability of UniHCP.

Table 10: Comparison of different parameter-sharing schemes. We report the
average scores of direct evaluation results on in-pretrain human-centric
datasets. “by $\mathbf{t}_{t}$” denotes sharing decoder and interpreter across
task types $\mathbf{t}_{t}$. For more detailed results on each dataset, please
refer to the supplementary.

Methods |  Total params. |  Shared params. | Shared module | Avg.
---|---|---|---|---
Encoder | Decoder |  Task-guided Interpreter
Baseline | 109.32M | 109.08M | ✓ | ✓ | ✓ | 67.4
(a) | 156.17M | 105.60M | ✓ | ✓ |  | 67.4
(b) | 489.67M | 91.07M | ✓ |  |  | 60.6
(c) | 170.83M | 109.08M | ✓ | by $\mathbf{t}_{t}$ | by $\mathbf{t}_{t}$ | 65.0

####  4.2.3 Data-Efficient Transferring

As UniHCP achieves SOTAs on full-data finetuning setting, we further evaluate
its potential for transferring to new datasets with extremely scarce training
images, _e.g._ , only one image per class for training. As summarized in Table
9, by conducting prompt tuning with one image per class, UniHCL achieves
93.65% pACC on ATR for parsing and 83.8% PCKh on MPII for pose estimation,
respectively. For prompt tuning on ATR, we follow [61]. For prompt tuning on
MPII, we only update queries and their associate position embeddings. The
prompt tuning results are close to that of the full-data finetuning setting
and suppress the results of finetuning the whole model with one image per
class for a large margin. Moreover, UniHCP with prompt tuning shows much lower
standard deviations than one-shot finetuning on human parsing and pose
estimation tasks, verifying that UniHCP learns generic human-centric
representation which is beneficial for data-efficient transferring with low
computation cost.

###  4.3 Ablation Study on Weight Sharing

As UniHCP achieves desirable performance on various human-centric tasks while
sharing most parameters among different tasks, one problem remains whether
more task-specific parameters benefit learning. To answer the question, we
ablate three weight sharing variants of UniHCP during pretraining using a
60k-iteration training schedule with 1k batch size. Results in Table 10(b)
show that compared to the original UniHCP _i.e_., the Baseline), unifying
task-guided interpreters among all tasks resulted in an average performance on
par with using specific heads while reducing about 30% of the parameters. We
also note that using task-specific or task-type-specific decoders and
interpreters leads to an obvious (-6.8% and -2.4%, respectively) performance
drop on average when compared to the original UniHCP (see results in Table
10(b) and (c)). We speculate that in these ablation settings, complementary
human-centric knowledge can not be properly shared among tasks, which leads to
performance drops on most tasks.

##  5 Conclusions

In this work, we present a Unified Model for Human-Centric Perceptions
(UniHCP). Based on a simple query-based task formulation, UniHCP can easily
handle multiple distinctly defined human-centric tasks simultaneously.
Extensive experiments on diverse datasets demonstrate that UniHCP pretrained
on a massive collection of human-centric datasets delivers a competitive
performance compared with task-specific models. When adapted to specific
tasks, UniHCP obtains a series of SOTA performances over a wide spectrum of
human-centric benchmarks. Further analysis also demonstrate the capability of
UniHCP on parameter and data-efficient transfer and the benefit of weight
sharing designs. We hope our work can motivate more future works on developing
general human-centric models.

Acknowledgement. This paper was supported by the Australian Research Council
Grant DP200103223, Australian Medical Research Future Fund MRFAI000085, CRC-P
Smart Material Recovery Facility (SMRF) – Curby Soft Plastics, and CRC-P ARIA
- Bionic Visual-Spatial Prosthesis for the Blind.

## References

  * [1] Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele.  Posetrack: A benchmark for human pose estimation and tracking.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5167–5176, 2018.
  * [2] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2d human pose estimation: New benchmark and state of the art analysis.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pages 3686–3693, 2014.
  * [3] Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al.  Ext5: Towards extreme multi-task scaling for transfer learning.  arXiv preprint arXiv:2111.10952, 2021.
  * [4] FairScale authors.  Fairscale: A general purpose modular pytorch library for high performance and large scale training.  https://github.com/facebookresearch/fairscale, 2021.
  * [5] Markus Braun, Sebastian Krebs, Fabian Flohr, and Dariu M Gavrila.  Eurocity persons: A novel benchmark for person detection in traffic scenes.  IEEE transactions on pattern analysis and machine intelligence, 41(8):1844–1861, 2019.
  * [6] Jiale Cao, Yanwei Pang, Jin Xie, Fahad Shahbaz Khan, and Ling Shao.  From handcrafted to deep features for pedestrian detection: a survey.  IEEE transactions on pattern analysis and machine intelligence, 2021\.
  * [7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [8] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey Hinton.  A unified sequence interface for vision tasks.  arXiv preprint arXiv:2206.07669, 2022.
  * [9] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu.  Dynamic convolution: Attention over convolution kernels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11030–11039, 2020.
  * [10] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290–1299, 2022.
  * [11] MMPose Contributors.  Openmmlab pose estimation toolbox and benchmark.  https://github.com/open-mmlab/mmpose, 2020.
  * [12] Emily E Cust, Alice J Sweeting, Kevin Ball, and Sam Robertson.  Machine and deep learning for sport-specific movement recognition: A systematic review of model development and performance.  Journal of sports sciences, 37(5):568–600, 2019.
  * [13] Yubin Deng, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Pedestrian attribute recognition at far distance.  In Proceedings of the 22nd ACM international conference on Multimedia, pages 789–792, 2014.
  * [14] Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona.  Pedestrian detection: An evaluation of the state of the art.  IEEE transactions on pattern analysis and machine intelligence, 34(4):743–761, 2011.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [16] Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu.  Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time.  arXiv preprint arXiv:2211.03375, 2022.
  * [17] Li Fei-Fei, Robert Fergus, and Pietro Perona.  One-shot learning of object categories.  IEEE transactions on pattern analysis and machine intelligence, 28(4):594–611, 2006.
  * [18] Yuying Ge, Ruimao Zhang, Xiaogang Wang, Xiaoou Tang, and Ping Luo.  Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5337–5345, 2019.
  * [19] Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin.  Instance-level human parsing via part grouping network.  In Proceedings of the European conference on computer vision (ECCV), pages 770–785, 2018.
  * [20] Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin.  Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 932–940, 2017.
  * [21] Irtiza Hasan, Shengcai Liao, Jinpeng Li, Saad Ullah Akram, and Ling Shao.  Generalizable pedestrian detection: The elephant in the room.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11328–11337, 2021.
  * [22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [23] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang.  Transreid: Transformer-based object re-identification.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 15013–15022, 2021.
  * [24] Fangzhou Hong, Liang Pan, Zhongang Cai, and Ziwei Liu.  Versatile multi-modal pre-training for human-centric perception.  arXiv preprint arXiv:2203.13815, 2022.
  * [25] Ronghang Hu and Amanpreet Singh.  Unit: Multimodal multitask learning with a unified transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439–1449, 2021.
  * [26] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, jul 2014.
  * [27] Ruyi Ji, Dawei Du, Libo Zhang, Longyin Wen, Yanjun Wu, Chen Zhao, Feiyue Huang, and Siwei Lyu.  Learning semantic neural tree for human parsing.  In European Conference on Computer Vision, pages 205–221. Springer, 2020.
  * [28] Jian Jia, Xiaotang Chen, and Kaiqi Huang.  Spatial and semantic consistency regularizations for pedestrian attribute recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 962–971, 2021.
  * [29] Jian Jia, Naiyu Gao, Fei He, Xiaotang Chen, and Kaiqi Huang.  Learning disentangled attribute representations for robust pedestrian attribute recognition.  2022\.
  * [30] Xin Jin, Cuiling Lan, Wenjun Zeng, Guoqiang Wei, and Zhibo Chen.  Semantics-aligned representation learning for person re-identification.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11173–11180, 2020.
  * [31] Yannis Kalantidis, Lyndon Kennedy, and Li-Jia Li.  Getting the look: clothing recognition and segmentation for automatic product suggestions in everyday photos.  In Proceedings of the 3rd ACM conference on International conference on multimedia retrieval, pages 105–112, 2013.
  * [32] Mahdi M Kalayeh, Emrah Basaran, Muhittin Gökmen, Mustafa E Kamasak, and Mubarak Shah.  Human semantic parsing for person re-identification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1062–1071, 2018.
  * [33] Sameh Khamis, Cheng-Hao Kuo, Vivek K Singh, Vinay D Shet, and Larry S Davis.  Joint learning for attribute-consistent person re-identification.  In European conference on computer vision, pages 134–146. Springer, 2014.
  * [34] Iasonas Kokkinos.  Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6129–6138, 2017.
  * [35] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby.  Uvim: A unified modeling approach for vision with learned guiding codes.  arXiv preprint arXiv:2205.10337, 2022.
  * [36] Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi.  General multi-label image classification with transformers.  arXiv preprint arXiv:2011.14027, 2020.
  * [37] Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  In Proceedings of the European conference on computer vision (ECCV), pages 734–750, 2018.
  * [38] Hei Law, Yun Teng, Olga Russakovsky, and Jia Deng.  Cornernet-lite: Efficient keypoint based object detection.  arXiv preprint arXiv:1904.08900, 2019.
  * [39] Dangwei Li, Zhang Zhang, Xiaotang Chen, and Kaiqi Huang.  A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios.  IEEE transactions on image processing, 28(4):1575–1590, 2019.
  * [40] Feng Li, Hao Zhang, Shilong Liu, Lei Zhang, Lionel M Ni, Heung-Yeung Shum, et al.  Mask dino: Towards a unified transformer-based framework for object detection and segmentation.  arXiv preprint arXiv:2206.02777, 2022.
  * [41] Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu.  Human pose regression with residual log-likelihood estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11025–11034, 2021.
  * [42] Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, Terence Sim, Shuicheng Yan, and Jiashi Feng.  Multiple-human parsing in the wild.  arXiv preprint arXiv:1705.07206, 2017.
  * [43] Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang.  Self-correction for human parsing.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [44] Suichan Li, Dapeng Chen, Bin Liu, Nenghai Yu, and Rui Zhao.  Memory-based neighbourhood embedding for visual recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6102–6111, 2019.
  * [45] Tianjiao Li, Jun Liu, Wei Zhang, Yun Ni, Wenqian Wang, and Zhiheng Li.  Uav-human: A large benchmark for human behavior understanding with unmanned aerial vehicles.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16266–16275, 2021.
  * [46] Wanhua Li, Zhexuan Cao, Jianjiang Feng, Jie Zhou, and Jiwen Lu.  Label2label: A language modeling framework for multi-attribute learning.  In European Conference on Computer Vision, pages 562–579. Springer, 2022.
  * [47] Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang.  Deepreid: Deep filter pairing neural network for person re-identification.  In CVPR, 2014.
  * [48] Yining Li, Chen Huang, Chen Change Loy, and Xiaoou Tang.  Human attribute recognition by deep hierarchical contexts.  In European Conference on Computer Vision, 2016.
  * [49] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  arXiv preprint arXiv:2203.16527, 2022.
  * [50] Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou.  Tokenpose: Learning keypoint tokens for human pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11313–11322, 2021.
  * [51] Jianming Liang, Guanglu Song, Biao Leng, and Yu Liu.  Unifying visual perception by dispersible points learning.  arXiv preprint arXiv:2208.08630, 2022.
  * [52] Xiaodan Liang, Ke Gong, Xiaohui Shen, and Liang Lin.  Look into person: Joint body parsing & pose estimation network and a new benchmark.  IEEE transactions on pattern analysis and machine intelligence, 41(4):871–885, 2018.
  * [53] Xiaodan Liang, Chunyan Xu, Xiaohui Shen, Jianchao Yang, Si Liu, Jinhui Tang, Liang Lin, and Shuicheng Yan.  Human parsing with contextualized convolutional neural network.  In Proceedings of the IEEE international conference on computer vision, pages 1386–1394, 2015.
  * [54] Chunze Lin, Jiwen Lu, and Jie Zhou.  Multi-grained deep feature learning for pedestrian detection.  In 2018 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2018.
  * [55] Matthieu Lin, Chuming Li, Xingyuan Bu, Ming Sun, Chen Lin, Junjie Yan, Wanli Ouyang, and Zhidong Deng.  Detr for crowd pedestrian detection.  arXiv preprint arXiv:2012.06785, 2020.
  * [56] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [57] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [58] Hao Liu, Jiashi Feng, Meibin Qi, Jianguo Jiang, and Shuicheng Yan.  End-to-end comparative attention networks for person re-identification.  IEEE Transactions on Image Processing, 26(7):3492–3506, 2017.
  * [59] Kunliang Liu, Ouk Choi, Jianming Wang, and Wonjun Hwang.  Cdgnet: Class distribution guided network for human parsing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4473–4482, 2022.
  * [60] Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu.  Query2label: A simple transformer way to multi-label classification.  arXiv preprint arXiv:2107.10834, 2021.
  * [61] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  arXiv preprint arXiv:2110.07602, 2021.
  * [62] Xihui Liu, Haiyu Zhao, Maoqing Tian, Lu Sheng, Jing Shao, Shuai Yi, Junjie Yan, and Xiaogang Wang.  Hydraplus-net: Attentive deep features for pedestrian analysis.  In Proceedings of the IEEE international conference on computer vision, pages 350–359, 2017.
  * [63] Chen Change Loy, Dahua Lin, Wanli Ouyang, Yuanjun Xiong, Shuo Yang, Qingqiu Huang, Dongzhan Zhou, Wei Xia, Quanquan Li, Ping Luo, et al.  Wider face and pedestrian challenge 2018: Methods and results.  arXiv preprint arXiv:1902.06854, 2019.
  * [64] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  arXiv preprint arXiv:2206.08916, 2022.
  * [65] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-task vision and language representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437–10446, 2020.
  * [66] Weian Mao, Yongtao Ge, Chunhua Shen, Zhi Tian, Xinlong Wang, Zhibin Wang, and Anton van den Hengel.  Poseur: Direct human pose regression with transformers.  arXiv preprint arXiv:2201.07412, 2022.
  * [67] Tewodros Legesse Munea, Yalew Zelalem Jembre, Halefom Tekle Weldegebriel, Longbiao Chen, Chenxi Huang, and Chenhui Yang.  The progress of human pose estimation: a survey and taxonomy of models applied in 2d human pose estimation.  IEEE Access, 8:133330–133348, 2020.
  * [68] Xuecheng Nie, Jiashi Feng, and Shuicheng Yan.  Mutual learning to adapt for joint human parsing and pose estimation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 502–517, 2018.
  * [69] Seyoung Park, Bruce Xiaohan Nie, and Song-Chun Zhu.  Attribute and-or grammar for joint parsing of human pose, parts and attributes.  IEEE transactions on pattern analysis and machine intelligence, 40(7):1555–1569, 2017.
  * [70] Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, and Xiaowei Zhou.  Deep snake for real-time instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8533–8542, 2020.
  * [71] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al.  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res., 21(140):1–67, 2020.
  * [72] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658–666, 2019.
  * [73] Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun.  Overfeat: Integrated recognition, localization and detection using convolutional networks.  arXiv preprint arXiv:1312.6229, 2013.
  * [74] Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun.  Crowdhuman: A benchmark for detecting human in a crowd.  arXiv preprint arXiv:1805.00123, 2018.
  * [75] Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  In International Conference on Machine Learning, pages 4596–4604. PMLR, 2018.
  * [76] Xiujun Shu, Xiao Wang, Xianghao Zang, Shiliang Zhang, Yuanqi Chen, Ge Li, and Qi Tian.  Large-scale spatio-temporal person re-identification: Algorithms and benchmark.  IEEE Transactions on Circuits and Systems for Video Technology, 2021\.
  * [77] Chi Su, Fan Yang, Shiliang Zhang, Qi Tian, Larry Steven Davis, and Wen Gao.  Multi-task learning with low rank attribute embedding for multi-camera person re-identification.  IEEE transactions on pattern analysis and machine intelligence, 40(5):1167–1181, 2017.
  * [78] Patrick Sudowe, Hannah Spitzer, and Bastian Leibe.  Person attribute recognition with a jointly-trained holistic cnn model.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 87–95, 2015.
  * [79] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693–5703, 2019.
  * [80] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al.  Sparse r-cnn: End-to-end object detection with learnable proposals.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14454–14463, 2021.
  * [81] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei.  Circle loss: A unified perspective of pair similarity optimization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6398–6407, 2020.
  * [82] Chufeng Tang, Lu Sheng, Zhaoxiang Zhang, and Xiaolin Hu.  Improving pedestrian attribute recognition with weakly-supervised multi-scale attribute-specific localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4997–5006, 2019.
  * [83] Yonglong Tian, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Pedestrian detection aided by deep learning semantic tasks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5079–5087, 2015.
  * [84] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [85] Edward Vendrow, Duy Tho Le, and Hamid Rezatofighi.  Jrdb-pose: A large-scale dataset for multi-person pose estimation and tracking.  arXiv preprint arXiv:2210.11940, 2022.
  * [86] Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3d human pose in the wild using imus and a moving camera.  In European Conference on Computer Vision (ECCV), sep 2018.
  * [87] Guan’an Wang, Shuo Yang, Huanyu Liu, Zhicheng Wang, Yang Yang, Shuliang Wang, Gang Yu, Erjin Zhou, and Jian Sun.  High-order information matters: Learning relation and topology for occluded person re-identification.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6449–6458, 2020.
  * [88] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In International Conference on Machine Learning, pages 23318–23340. PMLR, 2022.
  * [89] Wenguan Wang, Hailong Zhu, Jifeng Dai, Yanwei Pang, Jianbing Shen, and Ling Shao.  Hierarchical human parsing with typed part-relation reasoning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8929–8939, 2020.
  * [90] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun.  Anchor detr: Query design for transformer-based detector.  In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2567–2575, 2022.
  * [91] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian.  Person transfer gan to bridge domain gap for person re-identification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 79–88, 2018.
  * [92] Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, et al.  Large-scale datasets for going deeper in image understanding.  In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1480–1485. IEEE, 2019.
  * [93] Bin Xiao, Haiping Wu, and Yichen Wei.  Simple baselines for human pose estimation and tracking.  In Proceedings of the European conference on computer vision (ECCV), pages 466–481, 2018.
  * [94] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo.  Polarmask: Single shot instance segmentation with polar representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12193–12202, 2020.
  * [95] Shilin Xu, Xiangtai Li, Jingbo Wang, Guangliang Cheng, Yunhai Tong, and Dacheng Tao.  Fashionformer: A simple, effective and unified baseline for human fashion segmentation and recognition.  arXiv preprint arXiv:2204.04654, 2022.
  * [96] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao.  Vitpose: Simple vision transformer baselines for human pose estimation.  arXiv preprint arXiv:2204.12484, 2022.
  * [97] Qize Yang, Ancong Wu, and Wei-Shi Zheng.  Person re-identification by contour sketch under moderate clothing change.  IEEE transactions on pattern analysis and machine intelligence, 43(6):2029–2046, 2019.
  * [98] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang.  Transpose: Keypoint localization via transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11802–11812, 2021.
  * [99] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  arXiv preprint arXiv:2111.11432, 2021.
  * [100] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang.  Hrformer: High-resolution vision transformer for dense predict.  Advances in Neural Information Processing Systems, 34:7281–7293, 2021.
  * [101] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104–12113, 2022.
  * [102] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123–18133, 2022.
  * [103] Shanshan Zhang, Rodrigo Benenson, Mohamed Omran, Jan Hosang, and Bernt Schiele.  How far are we from solving pedestrian detection?  In Proceedings of the iEEE conference on computer vision and pattern recognition, pages 1259–1267, 2016.
  * [104] Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele.  Citypersons: A diverse dataset for pedestrian detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3221, 2017.
  * [105] Shifeng Zhang, Yiliang Xie, Jun Wan, Hansheng Xia, Stan Z Li, and Guodong Guo.  Widerperson: A diverse dataset for dense pedestrian detection in the wild.  IEEE Transactions on Multimedia, 22(2):380–393, 2019.
  * [106] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu.  Pose2seg: Detection free human instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 889–898, 2019.
  * [107] Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis.  From actemes to action: A strongly-supervised representation for detailed action understanding.  In Proceedings of the IEEE international conference on computer vision, pages 2248–2255, 2013.
  * [108] Xiaomei Zhang, Yingying Chen, Bingke Zhu, Jinqiao Wang, and Ming Tang.  Part-aware context network for human parsing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8971–8980, 2020.
  * [109] Haiyu Zhao, Maoqing Tian, Shuyang Sun, Jing Shao, Junjie Yan, Shuai Yi, Xiaogang Wang, and Xiaoou Tang.  Spindle net: Person re-identification with human body region guided feature decomposition and fusion.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1077–1085, 2017.
  * [110] Anlin Zheng, Yuang Zhang, Xiangyu Zhang, Xiaojuan Qi, and Jian Sun.  Progressive end-to-end object detection in crowded scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 857–866, 2022.
  * [111] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In Proceedings of the IEEE international conference on computer vision, pages 1116–1124, 2015.
  * [112] Shuai Zheng, Fan Yang, M Hadi Kiapour, and Robinson Piramuthu.  Modanet: A large-scale street fashion dataset with polygon annotations.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1670–1678, 2018.
  * [113] Zhedong Zheng, Xiaodong Yang, Zhiding Yu, Liang Zheng, Yi Yang, and Jan Kautz.  Joint discriminative and generative learning for person re-identification.  In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2138–2147, 2019.
  * [114] Zhedong Zheng, Liang Zheng, and Yi Yang.  A discriminatively learned cnn embedding for person reidentification.  ACM transactions on multimedia computing, communications, and applications (TOMM), 14(1):1–20, 2017.
  * [115] Qixian Zhou, Xiaodan Liang, Ke Gong, and Liang Lin.  Adaptive temporal encoding network for video instance-level human parsing.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1527–1535, 2018.
  * [116] Jianqing Zhu, Shengcai Liao, Zhen Lei, and Stan Z Li.  Multi-label convolutional neural network based pedestrian attribute classification.  Image and Vision Computing, 58:224–229, 2017.
  * [117] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.
  * [118] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai.  Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16804–16815, 2022.

##  Appendix A One-shot Transfer Results

In this section, we provide details and full results for one-shot fine-tuning
and prompt tuning on human parsing and pose estimation. For each experiment,
we sample ten sets of images with different random seeds; we also grid search
on both iterations and learning rates until performance converges. The
reported results are based on the best config found for each setting.

Data sampling. In one-shot transfer experiments, only one image per class is
used for a task [17]. Table 11 shows the number of sampled images on one-shot
transfer tasks. Note that in UniHCP, classification tasks are multi-label
classification for human parsing, pose estimation, and attribute recognition,
where each query performs binary classification via the global probability
unit. Therefore, we also make sure the presence of cases where a class is
absent is covered in our samples. Such handling avoids the query simply
learning to output 1 when the corresponding class always presents within the
sampled images. On the other hand, when a class does appear in most of the
images, _e.g._ , all keypoint joints in pose estimation or the background
class in human parsing, we are able to achieve reasonably good results without
such handling, thus we do not intentionally sample “not present” case for
keypoint joints and background class in our experiments.

Table 11: Number of sampled images on one-shot transfer tasks. As we can easily find pose samples with all keypoint joints present in the image and do not have to consider the case where a joint is absent as explained above, we only need one sample to perform one-shot transfer on pose estimation.  |  Parsing/ATR [53] |  Pose/MPII [2]
---|---|---
Sampled images |  3 $\sim$ 4 | 1

Number of tunable parameters. For fine-tuning settings, all parameters are
tuned. For prompt tuning on human parsing, we follow [61, 118] and add
learnable prompt tokens in decoder layers. We update queries, additional
prompt tokes, and layer normalization weights. For prompt tuning on pose
estimation, we only update queries and their associate position embeddings.
Table. 12 shows the number of parameters of each learnable component in prompt
tuning.

Table 12: Number of tunable parameters for prompt tuning on human parsing,
pose estimation, and pedestrian attribute recognition.

| Parsing/ATR | Pose/MPII | Attribute/PETA
---|---|---|---
Query | 9216 | 8704 | 35840
Deep prompt [61, 118] | 32256 | - | -
LN [118] | 16128 | - | -
Learnable parameter ratio | 0.053% | 0.008% | 0.033%

###  A.1 Human Parsing

Table 13 shows the full one-shot results for fine-tuning and prompt tuning on
human parsing.

Table 13: One-shot human parsing results on ATR, evaluated by pACC. FT -
finetuning, PT - prompt tuning.

| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | avg. | std.
---|---|---|---|---|---|---|---|---|---|---|---|---
FT | 91.28 | 91.21 | 90.75 | 87.90 | 91.48 | 92.14 | 89.67 | 89.36 | 90.67 | 90.48 | 90.49 | 1.22
PT | 93.31 | 92.99 | 93.41 | 92.31 | 93.89 | 95.16 | 93.41 | 93.81 | 94.01 | 94.23 | 93.65 | 0.77

###  A.2 Pose Estimation

Table 14 shows the full one-shot results for fine-tuning and prompt tuning on
pose estimation.

Table 14: One-shot pose estimation results on MPII, evaluated by mAP. FT -
finetuning, PT - prompt tuning.

| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | avg. | std.
---|---|---|---|---|---|---|---|---|---|---|---|---
FT | 64.18 | 78.68 | 78.18 | 60.52 | 73.71 | 67.80 | 70.44 | 57.20 | 79.26 | 76.07 | 70.60 | 7.53
PT | 87.32 | 86.13 | 87.33 | 77.44 | 85.91 | 81.16 | 88.29 | 71.97 | 87.45 | 85.29 | 83.83 | 5.08

##  Appendix B Few-shot Transfer Results for Pedestrian Attribute Recognition

In this section, we provide the few-shot transfer results for finetuning and
prompt tuning on pedestrian attribute recognition. Different from human
parsing and pose estimation datasets, the targeted downstream pedestrian
attribute recognition dataset PETA [13] contains images from ten different
domains. Randomly sampling only one image per class may mislead the queries to
extract domain-biased representation, and we found the one-shot result is poor
for both finetuning and prompt tuning under this setting. Therefore, we loosen
the data constraint to few-shot setting to evaluate the data-efficient
transfer performance on pedestrian attribute recognition. Similar to one-shot
experiments, we conduct the experiment on ten different sets of images, grid
search on hyperparameters, and report results based on the best config for
each setting.

Data sampling. PETA has ten different domains and 35 different attributes. For
each domain, we sample images until both “present” and “not present” cases
appeared at least once for each attribute; we sample multiple times and take
the one with the least samples as a few-shot dataset. It takes 68 $\sim$ 75
samples to satisfy this constraint in our experiments.

Number of tunable parameters. All parameters are tuned for finetuning. For
prompt tuning, we only update queries and their associate position embeddings.
The number of tunable parameters in prompt tuning is shown in Table 12.

Results. Table 15 shows the full few-shot results for pedestrian attribute
recognition; prompt tuning achieves better performance with a smaller standard
deviation.

Table 15: Few-shot pedestrian attribute recognition results on PETA, evaluated
by mA. FT - finetuning, PT - prompt tuning.

| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | avg. | std.
---|---|---|---|---|---|---|---|---|---|---|---|---
FT | 59.41 | 61.03 | 59.17 | 61.73 | 59.11 | 61.30 | 59.31 | 60.46 | 60.30 | 61.38 | 60.32 | 0.96
PT | 61.71 | 61.53 | 62.41 | 62.52 | 61.19 | 63.29 | 61.58 | 61.66 | 62.94 | 63.12 | 62.20 | 0.72

##  Appendix C Full Ablation Results on Weight Sharing

In Table 16, we provide full results for the ablation study in Section 4.3.
UniHCP achieves comparable performance with using task-specific interpreters
while sharing most of the parameters among different human-centric tasks.

Table 16: Detailed results for different parameter-sharing methods.

Methods | Shared module | Parsing/mIoU | ReID/mAP | Detection//mAP | Pose/mAP | Attribute/mA | Average
---|---|---|---|---|---|---|---
Encoder | Decoder | Task heads | H3.6 | LIP | CIHP | Market1501 | MSMT17 | CUHK03 | CrowdHuman | COCO | AIC | OCHuman | PA-100K | RAPv2
Baseline | ✓ | ✓ | ✓ | 64.6 | 61.9 | 64.4 | 82.1 | 59.0 | 59.9 | 80.5 | 73.5 | 29.0 | 77.0 | 81.0 | 75.3 | 67.4
(a) | ✓ | ✓ |  | 65.4 | 61.6 | 64.1 | 82.7 | 59.9 | 62.1 | 82.2 | 73.5 | 27.9 | 74.9 | 81.3 | 73.0 | 67.4
(b) | ✓ |  |  | 64.2 | 59.8 | 61.1 | 76.9 | 51.3 | 51.0 | 36.2 | 71.3 | 25.6 | 69.0 | 81.9 | 78.7 | 60.6
(c) | ✓ | by $\mathbf{t}_{t}$ | by $\mathbf{t}_{t}$ | 64.1 | 61.6 | 63.0 | 79.4 | 54.4 | 56.3 | 68.4 | 72.7 | 26.8 | 71.3 | 82.1 | 79.8 | 65.0

##  Appendix D Additional Architecture Details

###  D.1 Task-guided Interpreter

Since the task-guided interpreter decodes each query token independently, we
formulate the interpreter design by describing the generation of each output
unit element $y\in\mathbf{Y}$ from query token $q\in Q^{t}$.

Feature vector unit $\mathbf{Y}_{f}$: as the query token is already in a
feature space, we do not add any additional postprocessing. we have
$y_{f}=q,y_{f}\in\mathbb{R}^{C}$, where $C$ is the output dimension of the
decoder.

Global probability unit $\mathbf{Y}_{p}$: we apply a 1-lyr MLP (i.e. linear
projector) followed by a sigmoid function $\sigma$, on top of query token $q$
to yield global probability $y_{p}\in\mathbb{R}^{1}$.

Local probability map unit $\mathbf{Y}_{m}$: We denoted visual tokens from the
encoder as $\mathbf{F}\in\mathbb{R}^{C_{e}\times H/16\times W/16}$, where
$C_{e}$ denotes the output dimension of the encoder, $H\times W$ denoted the
original image size and $16$ is the patch size of ViT-B. $\mathbf{F}$ is
forwarded through two consecutive deconvolution layers with hidden dimension
$C_{e}$ to upscale the feature map to
$\tilde{\mathbf{F}}\in\mathbb{R}^{C\times H/4\times W/4}$. The query token $q$
is applied with a 3-lyr MLP to get the embedding $\tilde{q}\in\mathbb{R}^{C}$.
We obtain the final probability logit map $y_{m}\in\mathbb{R}^{H/4\times W/4}$
by calculating the dot product between $\tilde{q}$ and $\tilde{\mathbf{F}}$,
broadcasted in the spatial dimensions.

Bounding box unit $\mathbf{Y}_{bbox}$: Similar with [117], the query token $q$
is applied with a 3-lyr MLP to get the box offset prediction logits
$\tilde{q}=[\alpha_{\nabla cx},\alpha_{\nabla
cx},\alpha_{h},\alpha_{w}],\tilde{q}\in\mathbb{R}^{4}$. With its associated
anchor point $\mathcal{A}_{q}=[cx,cy]$, we yield the final box prediction
$y_{bbox}=[\sigma(\alpha_{\nabla cx}+\sigma^{-1}(cx)),\sigma(\alpha_{\nabla
cy}+\sigma^{-1}(cy)),\sigma(\alpha_{h}),\sigma(\alpha_{w})]$, where
$\sigma^{-1}$ denotes the inversed sigmoid function.

###  D.2 Positional Embedding for Encoder

The positional embedding for the encoder is shared across tasks and is
interpolated according to the spatial size of the patch projected input image.
The maximum image resolution during training is $1333\times 800$ (or
$800\times 1333$), which will then be padded to $1344\times 800$ before patch
projection (rounded up to be divisible by patch size 16). Thus, the maximum
H/W dimension for images after patch projection is 84. Accordingly, we set the
number of tokens for learnable positional embedding to $84\times 84=7056$.

###  D.3 Decoder Positional Embedding Projector

The positional embedding projector $proj$ follows the design in [90]. The
coordinate is first encoded by sine-cosine position encoding function [84] and
then projected by a simple 2-Layer MLP.

###  D.4 Auxiliary Loss:

Apart from the loss for $Q^{t}_{L}$ after $L$-th decoder block, we also add
auxiliary losses to intermediate queries for pose estimation, human parsing,
and pedestrian detection following the best practices in [10, 110, 7]. For
pose estimation and human parsing, the auxiliary loss is calculated on
$Q^{t}_{l}$ for $l\in\\{0,...,L-1\\}$ following [10]. For pedestrian
detection, the auxiliary loss is calculated on $Q^{t}_{l}$ for
$l\in\\{1,...,L-1\\}$ following [110, 7].

###  D.5 Pose Estimation

For pose estimation, we set $\lambda_{par}=0.001$. During the inference time,
when the metric requires a confidence score for keypoint filtering and NMS
(e.g. mAP), we additionally multiply the global probability prediction $y_{p}$
to the confidence score and lower the visibility threshold to 0.05
accordingly.

##  Appendix E Additional Training Details

Loss Weight $w_{\mathcal{D}}$: for dataset $\mathcal{D}^{\prime}$, its loss
weight $w_{\mathcal{D^{\prime}}}$ is calculated as follows:

$\displaystyle
w_{\mathcal{D^{\prime}}}=\frac{b_{\mathcal{D^{\prime}}}w_{\mathbf{t}_{\mathcal{D^{\prime}}}}}{\sum_{\mathcal{D}\in\mathbb{D}}b_{\mathcal{D}}w_{\mathbf{t}_{\mathcal{D}}}},$
(13)

where $b_{\mathcal{D}}$ denotes the batch size allocated to dataset
$\mathcal{D}$ and $w_{\mathbf{t}_{\mathcal{D}}}$ denotes the sample weight for
task type $\mathbf{t}_{\mathcal{D}}$. The loss weight is normalized so that it
only controls the relative weight for each dataset. Samples belonging to the
same task type are treated with equal importance. Since different task types
have different loss functions, image input resolution, number of samples, and
convergence pattern, their loss weight should be set differently. For a
reasonable loss weight trade-off between tasks, we gradually add task types
one at a time in a small 10k iteration joint training setup and sweep sample
weights for the newly added task type. After the hyperparameter search, we set
$w_{reid}=10,w_{par}=1\times 10^{-2},w_{seg}=5,w_{pose}=2\times
10^{3},w_{peddet}=2$.

Dataset-wise Configurations: we provide detailed dataset-wise training
configurations in Table 17. In addition to these training datasets, downstream
datasets are ATR [53], SenseReID [109], Caltech [14], MPII [2] and PETA [13].

Table 17: UniHCP joint training setup. †the batch size for pedestrian
detection is reduced due to high GPU consumption.

Task Type |  Dataset $\mathcal{D}$ |  Batch Size $b_{\mathcal{D}}$ |  Batch Size per GPU |  Dataset Epoch | $b_{\mathcal{D}}w_{\mathbf{t}_{\mathcal{D}}}$ |  GPUs |  Number of Samples |  Sample Weight $w_{\mathbf{t}_{\mathcal{D}}}$
---|---|---|---|---|---|---|---|---
Pedestrian Detection | CrowdHuman [74] | 212† | 4 | 130.19† | 424 | 53 | 170,687 | 2
EuroCity Persons [5]
CityPersons [104]
WiderPerson [105]
WiderPedestrian [63]
COCO-Person [57]
Person ReID | Market-1501 [111] | 96 | 96 | 199.06 | 960 | 1 | 50,549 | 10
CUHK03 [47]
MSMT17 [91]
DGMarket [113] | 415 | 415 | 200.04 | 4150 | 1 | 217,453 | 10
PRCC [97]
LaST [76]
Pose Estimation | COCO-Pose [57] | 286 | 286 | 200.1 | 572000 | 1 | 149,813 | 2000
AI Challenger [92] | 720 | 240 | 199.46 | 1440000 | 3 | 378,352 | 2000
PoseTrack [1] | 185 | 185 | 199.55 | 3710000 | 1 | 97,174 | 2000
MHP [42] | 77 | 77 | 199.59 | 154000 | 1 | 40,437 | 2000
3DPW [86] | 131 | 131 | 199.98 | 262000 | 1 | 68,663 | 2000
UpennAction [107] | 66 | 66 | 200.66 | 132000 | 1 | 34,475 | 2000
JRDB-Pose [85] | 266 | 266 | 200.03 | 532000 | 1 | 139,385 | 2000
Halpe [16] | 79 | 79 | 200.69 | 158000 | 1 | 41,263 | 2000
Human3.6M (pose) [26] | 596 | 298 | 200.11 | 1192000 | 2 | 312,187 | 2000
Human Parsing | LIP [20] | 58 | 58 | 199.57 | 290 | 1 | 30,462 | 5
CIHP [19] | 54 | 54 | 200.14 | 270 | 1 | 28,280 | 5
Deep fashion [18] | 364 | 52 | 198.75 | 1820 | 7 | 191,961 | 5
VIP [115] | 35 | 35 | 198.63 | 175 | 1 | 18,469 | 5
ModaNet [112] | 100 | 50 | 200.62 | 500 | 2 | 52,245 | 5
Human3.6M (parse) [26] | 120 | 40 | 200.71 | 600 | 3 | 62,668 | 5
Pedestrian Attribute Recognition | PA-100K [62] | 172 | 172 | 200.32 | 1.72 | 1 | 90,000 | 0.01
RAPv2 [39] | 130 | 130 | 200.55 | 1.3 | 1 | 67,943 | 0.01
HARDHC [48] | 54 | 54 | 199.75 | 0.54 | 1 | 28,336 | 0.01
UAV-Human [45] | 31 | 31 | 200.78 | 0.31 | 1 | 16,183 | 0.01
Parse27k [78] | 52 | 52 | 198.33 | 0.52 | 1 | 27,482 | 0.01
Market-1501 (attribute) [111] | 25 | 25 | 202.57 | 0.25 | 1 | 12,936 | 0.01
Summary | / | Total: 4324 | / |  Avg.: 200.00 (excluding det.) | / | Total: 88 | Total: 2,327,403 | /

##  Appendix F Additional Finetuning Details

We provide major finetuning configurations in Table 18; other settings are
identical to the training config.

Table 18: Detailed finetuning configs for human-centric tasks.

Task Type | Dataset | Learning Rate | Batch Size | Iterations | Backbone lr Multiplier | Drop Path Rate | Layer Decay Rate | Weight Decay
---|---|---|---|---|---|---|---|---
Pedestrian Detection | CrowdHuman [74] | 2.00E-04 | 32 | 160k | 1.0 | 0.2 | 0.75 | 0.05
Caltech [14] | 1.00E-05 | 32 | 30k | 0.1 | 0.2 | 0.75 | 0.05
Person ReID | Market-1501 [111] | 1.00E-04 | 64 | 40k | 0.4 | 0.05 | 0.75 | 0.5
CUHK03 [47] | 5.00E-05 | 64 | 20k | 0.9 | 0.1 | 0.95 | 0.5
MSMT17 [91] | 1.00E-04 | 64 | 40k | 0.9 | 0.05 | 0.75 | 0.5
Pose Estimation | COCO-Pose [57] | 1.00E-04 | 512 | 20k | 0.9 | 0.25 | 0.75 | 0.05
AI Challenger [92] | 1.00E-03 | 512 | 10k | 0.9 | 0.2 | 0.75 | 0.05
Human3.6M (Pose) [26] | 5.00E-06 | 512 | 10k | 0.9 | 0.3 | 0.75 | 0.05
MPII [2] | 7.00E-05 | 512 | 7.5k | 0.9 | 0.3 | 0.75 | 0.05
Human Parsing | LIP [20] | 5.00E-05 | 64 | 30k | 1.0 | 0.3 | 0.75 | 0.05
CIHP [19] | 1.00E-04 | 64 | 35k | 1.4 | 0.3 | 0.65 | 0.05
Human3.6M (parse) [26] | 1.00E-05 | 64 | 25k | 1.3 | 0.3 | 0.85 | 0.05
ATR [53] | 1.00E-04 | 64 | 15k | 0.7 | 0.3 | 0.85 | 0.05
Pedestrian Attribute Recognition | PA-100K [62] | 3.00E-03 | 128 | 10k | 0.05 | 0.2 | 0.85 | 0.05
RAPv2 [39] | 5.00E-04 | 128 | 4k | 0.5 | 0.3 | 0.75 | 0.05
PETA [13] | 1.00E-03 | 128 | 20k | 0.2 | 0.3 | 0.75 | 0.05

##  Appendix G Ethics

In this work, we proposed a model to unify multiple human-centric tasks and
trained the model on a huge collection of public and widely used human-centric
datasets. We acknowledge that the resulting model demonstrates good
performance on public ReID benchmarks and thus may be associated with
potential identity information leaking without consent if misused. Therefore,
the pretrained model will be released only on a case-by-case basis, and the
requester must sign an agreement limiting the usage to research purposes only.
In addition, the pretrained query tokens for ReID tasks will be excluded from
the model release.