# Differentiable Multi-Granularity Human Representation Learning for
Instance-Aware Human Semantic Parsing

Tianfei Zhou1 Wenguan Wang1 Corresponding author: Wenguan Wang. Si Liu2 Yi
Yang3 Luc Van Gool1
1Computer Vision Lab, ETH Zurich 2Institute of Artificial Intelligence,
Beihang University 3University of Technology Sydney
https://github.com/tfzhou/MG-HumanParsing

###### Abstract

To address the challenging task of instance-aware human part parsing, a new
bottom-up regime is proposed to learn category-level human semantic
segmentation as well as multi-person pose estimation in a joint and end-to-end
manner. It is a compact, efficient and powerful framework that exploits
structural information over different human granularities and eases the
difficulty of person partitioning. Specifically, a dense-to-sparse projection
field, which allows explicitly associating dense human semantics with sparse
keypoints, is learnt and progressively improved over the network feature
pyramid for robustness. Then, the difficult pixel grouping problem is cast as
an easier, multi-person joint assembling task. By formulating joint
association as maximum-weight bipartite matching, a differentiable solution is
developed to exploit projected gradient descent and Dykstra’s cyclic
projection algorithm. This makes our method end-to-end trainable and allows
back-propagating the grouping error to directly supervise multi-granularity
human representation learning. This is distinguished from current bottom-up
human parsers or pose estimators which require sophisticated post-processing
or heuristic greedy algorithms. Experiments on three instance-aware human
parsing datasets show that our model outperforms other bottom-up alternatives
with much more efficient inference.

Figure 1: Overview of our new bottom-up regime for instance-aware human
semantic parsing. By learning 1) category-level human semantic parsing, 2)
body-to-joint projection, and 3) bottom-up keypoint detection and association
in a joint and end-to-end manner, our model tackles the task in a
differentiable, multi-granularity human representation learning framework.

##  1 Introduction

Instance-aware${}_{\\!}$ human${}_{\\!}$ parsing, , partitioning${}_{\\!}$
humans into semantic parts (, torso, head) and associating each part with the
corresponding human instance, has only started to be tackled in the literature
(dating back to [31]). This article addresses this task through a new regime,
which learns to jointly estimate human poses (, a sparse, skeleton-based human
representation) and segment human parts (, a pixel-wise, fine-grained human
representation) in an end-to-end trainable, bottom-up fashion.

In the field of human parsing, the idea of leveraging human pose as structural
knowledge to facilitate human understanding has been exploited for years [56].
However, previous efforts only focus on the instance-agnostic setting [54, 53,
15, 37, 63, 48]. Further, most of them directly utilize human joints (pre-
detected from off-the-shelf pose estimators) as extra inputs [56, 54], or
simply generate key-point estimations as a by-product [15, 63]. In sharp
contrast, by learning to associate semantic person pixels with their closest
person instance joints, our model seamlessly injects bottom-up pose estimation
into instance-aware human semantic learning and inference. Thus, our human
parser can make use of the complementary cues from sparse human joints and
dense part semantics, and push further the envelope of human understanding in
unconstrained environments. This represents an early effort to formulate
instance-aware human parsing and multi-person pose estimation in a bottom-up,
differentiable, and multi-granularity human representation learning framework
(see fig:motivation).

More importantly, our framework yields a new paradigm for${}_{\\!}$ bottom-up,
instance-aware${}_{\\!}$ human${}_{\\!}$ part${}_{\\!}$ parsing. For${}_{\\!}$
human instance discrimination, current bottom-up human parsers learn to
associate pixels by directly regressing instance locations [64] or predicting
pair-wise connectiveness between pixels [30]. However, instance locations (,
human centroids or bounding box corners) are less semantic and pair-wise
embeddings are difficult to learn (due to deformations, variations and
occlusions of human bodies) [50]. As shown in fig:motivation, our model
instead regresses semantic- and geometry-rich human joints as pixel
embeddings. Then, fine-grained human semantics can be efficiently grouped
through body joint association. In essence, it formulates instance-aware human
semantic parsing by jointly learning: 1) category-level human semantic
parsing, 2) body-to-joint projection, and 3) bottom-up multi-person keypoint
detection and association. Thus, our model avoids sophisticated inference and
heavy network designs, and neatly explores the structural constraints over
human bodies. Such a flexible network design can benefit from progress in
bottom-up pose estimation and semantic segmentation techniques, and
significantly differentiates itself from existing instance-aware human
parsers.

Figure 2: Trade-off between performance vs. efficiency on MHPv2 val [64].
The x-axis and y-axis denote FPS and AP${}_{50}^{p}$, respectively. The circle
size indicates Multi-Adds (G). Top-down and bottom-up models are given in
black and red, respectively. Our model shows promising performance with high
efficiency.

Concretely, three crucial techniques, for the first time in the field, are
exploited to deliver our compact and powerful instance-aware human parsing
solution:

  * •

Differentiable body-to-joint projection: We propose a Dense-to-Sparse
Projection Field (DSPF), which is a set of 2D vectors over the image lattice.
For each human pixel, its DSPF vector encodes the offset to the closest
instance joint. DSPF thus allows us to explicitly associate dense human
semantics with sparse joint representations.

  * •

Multi-step DSPF estimation: To address the difficulties of human body
variations and occlusions, DSPF is computed in a coarse-to-fine fashion, over
the network feature pyramid. Deeper-layer features are low-resolution, yet
robust to the challenges. They are thus used to derive an initial
discriminative DSPF. Conditioning on the shallower-layer features, a finer
DSPF is inferred by computing the residue to the coarse estimation. To reduce
the feature-space distance across network layers, cross-layer feature
alignment is learnt and adopted. In this way, the residue has a smaller
magnitude and is easier to infer.

  * •

An end-to-end trainable framework: Current bottom-up multi-person pose
estimators solve joint association through heuristic greedy algorithms [4,
27], independent from joint detection. This breaks the end-to-end pipeline and
leads to suboptimal results. Since joint association can be formulated as
maximum-weight bipartite matching [4], we explore a differentiable solution
to this, inspired by [61]. It revisits projected gradient descent and
Dykstra’s cyclic projection [2] for convex constrained minimization.
The${}_{\\!}$ solution${}_{\\!}$ is${}_{\\!}$ neat and light-${}_{\\!}$weight,
allowing directly using the grouping errors for${}_{\\!}$ supervision.

Our${}_{\\!}$ model${}_{\\!}$ is${}_{\\!}$ also${}_{\\!}$
distinguished${}_{\\!}$ for${}_{\\!}$ its${}_{\\!}$ practical${}_{\\!}$
utility. First, it can benefit a host of human-centric applications. Some of
them, such as augmented reality, are pose guided, while others, such as live
streaming, video editing and virtual try-on systems, require understanding of
fine-grained human semantics. Thus, our method can meet different needs in
real-life applications with only a single model. Second, due to its bottom-up
nature and fast keypoint association, our method generates instance-level
parsing results at a high speed of 0.15 s per image (see fig:madd),
irrespective of the number of people in the scene, and, which is much faster
than other alternatives (, 1.15 s [42], 14.94 s [30]).

Extensive experiments are conducted on three instance-aware human parsing
datasets. Specifically, on MHPv2 [64], our model achieves an AP${}_{50}^{p}$
of $39.0$, much${}_{\\!}$ better${}_{\\!}$ than${}_{\\!}$ current${}_{\\!}$
top-leading${}_{\\!}$ bottom-up method (, NAN [64] of $25.1$) and on par with
best top-down parsers [24, 57]. On DensePose-COCO [1] and PASCAL-Person-
Part [53], our model also produces compelling performance. Overall, our model
shows favorable results with a fast speed.

##  2 Related Work

Instance-Aware Human Semantic Parsing. Fine-grained human semantic
segmentation, as one of the central tasks in human understanding, has
applications in human-centric vision [41, 65, 43], human-robot interaction
[11] and fashion analysis [46]. However, previous studies mainly focus on
category-level human parsing [32, 15, 12, 35, 47, 63, 49]; only very few human
parsers are specifically designed for the instance-aware setting. As of to
date, there exist two paradigms for instance-aware human parsing: top-down and
bottom-up. Top-down approaches [31, 58, 42, 24, 57] typically locate human
instance proposals first, and then parse each proposal in a fine-grained
manner. In contrast, bottom-up human parsers [14, 30, 64] simultaneously
perform pixel-wise instance-agnostic parsing and pixel grouping, inspired by
existing bottom-up instance segmentation techniques. Their grouping strategies
vary from instance-edge-aware clustering [14], to graph-based superpixel
association [30], to proposal-free instance localization [64].

Our model falls into the bottom-up paradigm and has several unique
characteristics. First, our method formulates instance-aware human part
parsing as a multi-granularity human representation learning task. However,
other alternatives, whether top-down or bottom-up, fail to explore skeleton-
based human structures. Second, our model yields a new bottom-up regime for
instance-aware human part parsing. Through projecting dense human semantics
into sparse body joints, the task of pixel grouping can be easily tackled by
joint association. Third, our method achieves promising results with much
improved efficiency, leading to high utility in downstream applications.

Multi-Person Pose Estimation. Allocating body joints to human instances, as
another representative human understanding task, has been extensively studied
over the past decades. Also, current popular solutions for multi-person pose
estimation can be roughly categorized as either top-down or bottom-up methods.
Specifically, top-down methods [13, 19, 40, 7] conduct single-person pose
estimation over each pre-detected human bounding-box. Contrarily, bottom-up
methods are detection-free; they usually predict identity-free keypoints,
which are then assembled into valid pose instances [22, 4, 36, 27], or
directly regress poses from person instance centroids in a single-stage
fashion [38].

In our method, human joints are set as the targets for person pixel embedding
learning, simplifying the procedure of pixel grouping. This flexible
architecture can be seamlessly integrated with any bottom-up pose estimators
in principle, and explicitly encodes human structural constraints. Hence, we
formulate joint association as a differentiable matching problem [61], rather
than relying on sophisticated post-processing [4, 27] like conventional
bottom-up methods. Though [25] also addresses joint association in an end-to-
end manner, it needs to learn a complicated and heavy graph network and cannot
guarantee optimality.

Human/Object Representation. From a slightly broader perspective, the way of
representing visual elements is crucial for visual understanding. From classic
rectangular boxes and masks, to recent point-based object modeling forms (,
sparse points [59], contours [55, 50], grid masks [6], and dense points
[60]), the community is continually pursuing a proper object representation
for more effective processing, further advancing the development of object
detection and segmentation. Point-based representations are thought to be
promising, as both geometric and semantic cues are encoded. This insight is
shared by several skeleton-based${}_{\\!}$ human${}_{\\!}$
understanding${}_{\\!}$ models. Despite the efforts made prior to the
renaissance of deep learning [9, 54], some recent studies explore body poses
as an extra representation granularity in category-level human parsing [53,
37] or instance-aware human full-body segmentation [39, 62]. However, [53, 37,
62] only utilize human poses as a shape prior for feature enhancement, instead
of as an informative clue for person separation. Thus, their main ideas are
far different from ours. Although [39] also views human joints as pixel
embedding targets, our method 1) focuses on fine-grained human semantic part
parsing; 2) coarse-to-fine estimates the projection from human dense semantics
to sparse joints through residual learning and feature alignment; and 3)
formulates joint association in a differentiable manner, yielding an end-to-
end trainable model.

##  3 Our Approach

As shown in Fig. 3, our model learns to: 1) predict category-level human
parts (§3.1), 2) build a Dense-to-Sparse Projection Field (DSPF, §3.2), and
3) conduct human joint detection and association (§3.3). With 1), we have
fine-grained human semantics, but they are identification-irrelevant. With 2),
we can explicitly associate dense human semantics with sparse joints, as DSPF
encodes the displacements from each person pixel to the closet human joints.
Instance-aware parsing can then be achieved by bottom-up joint association. At
the same time, human structural cues can be embedded. Finally, with 3), joint
association is achieved by a differentiable solver, allowing the error signals
for grouping to be back-propagated to supervise keypoint detection and feature
learning. In this way, we provide an instance-aware human parsing framework
based on multi-granularity human representation learning, which works in a
bottom-up manner and is end-to-end trainable.

###  3.1 Instance-Agnostic Human Part Parsing

Given an input image
$I\\!\\!\in\\!\\!\mathbb{R}^{W\\!\times\\!H\\!\times\\!3}$, a backbone network
is first employed to extract an $L$-level feature pyramid, ,
$\\{\bm{X}_{l}\\!\in\\!\mathbb{R}^{W_{l}\\!\times\\!H_{l}\\!\times\\!C_{l}}\\}_{l=1}^{L}$,
where $W_{l}=W/2^{l+1}$ and $H_{l}=H/2^{l+1}$. The feature pyramid
comprehensively encodes multi-scale visual features, from the highest spatial
resolution ($l\\!=\\!1$) to the lowest ($l\\!=\\!L$).

Given $\\{\bm{X}_{l}\\}_{l=1}^{L}$, a segmentation head
$\mathcal{F}^{\text{Seg}}$ is applied to parse category-level human part
semantics (see Fig. 3):

$\displaystyle\bm{S}=\mathcal{F}^{\text{Seg}}(\\{\bm{X}_{l}\\}_{l=1}^{L})\\!\in\\![0,1]^{W_{1}\\!\times\\!H_{1}\\!\times\\!P}.$
(1)

Here, $P$ indicates the number of human semantic part categories.
$\mathcal{F}^{\text{Seg}\\!}$ is implemented as a decoder architecture [5],
which makes full use of the multi-scale features $\\{\bm{X}_{l}\\}_{l=1}^{L}$
and estimates $\bm{S}$ with the highest spatial resolution.

$I$§3.1$\mathcal{F}^{\text{Seg}}$§3.2$\mathcal{F}^{\text{DSPF}}$$\mathcal{F}^{\text{DSPF}}$$\mathcal{F}^{\text{DSPF}}$§3.3$\mathcal{F}^{\text{Pose}}$$\bm{X}_{1}$$\bm{X}_{2}$$\bm{X}_{L}$$\bm{P}$$\bm{S}$$\bm{D}_{\\!L}$$\bm{D}_{\\!1}$Instance-
Aware HumanSemantic Parsing Result

Figure 3: Illustration of our multi-granularity human representation learning
framework for instance-aware human semantic parsing.

###  3.2 Multi-Step Dense-to-Sparse Projection Field (DSPF) Estimation

Given $\\{\bm{X}_{l}\\}_{l=1}^{L}$, our network learns DSPF, ,
$\bm{D}\\!\in\\!\mathbb{R}^{W_{1}\\!\times\\!H_{1}\\!\times\\!2}$, to
associate dense human semantics with sparse keypoints (Fig. 3).
Specifically, for each point $\bm{u}\\!=\\!(u,v)$ in an image lattice
$\Omega\\!\in\\!\mathbb{R}^{2}$, its DSPF is a 2D vector
$\bm{D}(\bm{u})\\!=\\!(\Delta u,\Delta v)$, where
$\bm{u}\\!+\\!\bm{D}(\bm{u})\\!=\\!(u\\!+\\!\Delta u,v\\!+\\!\Delta v)$ is
expected to be the location of its nearest human instance joint. Note that
$\bm{u}$ and $\bm{u}\\!+\\!\bm{D}(\bm{u})$ belong to the same human instance.
Due to variations and deformations typically exhibited by human bodies, as
well as ambiguities caused by close or occluded instances, it is difficult to
directly infer DSPF. Thus, we estimate it in a coarse-to-fine framework.

Coarse-to-Fine Estimation. Given the coarsest feature (, $\bm{X}_{L}$), which
has the largest receptive field and thus is good at robust, long-range
relation modeling, an initial DSPF estimation
$\bm{D}_{L}\\!\in\\!\mathbb{R}^{W_{L}\\!\times\\!H_{L}\\!\times\\!2}$ can be
derived. Starting with $\bm{D}_{L}$, the most straightforward strategy is to
incrementally update $\bm{D}_{L}$ across the pyramid levels to finally achieve
a high-resolution DSPF estimation
$\bm{D}_{1\\!}\\!\in\\!\mathbb{R}^{W_{1}\\!\times\\!H_{1}\\!\times\\!2}$:

$\displaystyle\bm{D}_{l}=\mathcal{F}^{\text{DSPF}}(\bm{X}_{l},{\bm{X}}^{\uparrow}_{l+1},2\bm{D}_{l+1}^{\uparrow})\\!\in\\!\mathbb{R}^{W_{l}\\!\times\\!H_{l}\\!\times\\!2},$
(2)

where $\bm{D}_{l+1\\!}$ and ${\bm{X}}_{l+1\\!}$ from a preceding level need to
be upsampled in spatial resolution (denoted by “$\uparrow$”), and
$\bm{D}_{l+1}$ needs to be multiplied by 2 to match the resolution of the
pyramidal features in the $l$-th level. In this way, the estimation for DSPF
can be gradually improved by fusing higher-level prediction $\bm{D}_{l+1}$
with lower-layer feature $\bm{X}_{l}$. Since the flow magnitude is enlarged
after each estimation step, the value ranges of output spaces are different,
making the regression difficult. Inspired by recent optical flow estimation
techniques [20, 21], we employ a residual refinement strategy for more
accurate DSPF estimation.

Coarse-to-Fine Residual Refinement. Rather than inferring a complete DSPF
$\bm{D}_{l\\!}$ at each pyramid level, our network learns the residue
$\triangle\bm{D}_{l\\!}\\!\in\\!\mathbb{R}^{W_{l}\\!\times\\!H_{l}\\!\times\\!2\\!}$
in relative to $\bm{D}_{l+1\\!}$ estimated in the previous layer. Eq. (2) is
thus improved as:

$\displaystyle\bm{D}_{l}=\underbrace{\mathcal{F}^{\text{DSPF}}(\bm{X}_{l},{\bm{X}}^{\uparrow}_{l+1})}_{\triangle\bm{D}_{l}}+2\bm{D}_{l+1}^{\uparrow}.$
(3)

This network design is more favored, since the magnitudes of the residues at
different pyramid levels are smaller than the complete DSPFs and within a
similar value range.

Recall that we apply “$\uparrow$” operation to warp a low-resolution feature
map into a high-resolution one. A common way to implement such a warping
operation is through bilinear upsampling. However, as bilinear upsampling only
accounts for limited information from a fixed and pre-defined subpixel
neighborhood for feature interpolation, it fails to handle the misalignment
between feature maps caused by repeated padding, downsampling and upsampling,
thus leading to error accumulation. Motivated by learnable feature warping
operators [44, 45], we propose to learn a feature flow between two features
from adjacent pyramid levels for cross-layer feature alignment.

Learnable Feature Warping. In particular, a feature flow, , $\bm{F}_{l}$, is
estimated between $\bm{X}_{l}$ and $\bm{X}_{l+1}$:

feature flow:
$\displaystyle~{}~{}~{}\bm{F}_{l}=\mathcal{F}^{\text{Flow}}(\bm{X}_{l},\bm{X}_{l+1}^{~{}\\!\\!\Uparrow})\in\mathbb{R}^{W_{l}\\!\times\\!H_{l}\\!\times\\!2}.$
(4)

Here, the low-resolution feature $\bm{X}_{l+1}$ is first bilinearly upsampled
(denoted by “$\Uparrow$”), and then concatenated with the high-resolution
feature $\bm{X}_{l}$ for predicting $\bm{F}_{l}$. The feature flow, in
essence, is an offset field that aligns the positions between high-level and
low-level feature maps. Through $\bm{F}_{l}$, $\bm{X}_{l+1\\!}$ is warped
towards $\bm{X}_{l}$, ,
$\bm{X}_{l+1\\!}^{~{}\\!\\!\uparrow}(\bm{u}\\!+\\!\bm{F}_{l}(\bm{u}))\\!\sim\\!\bm{X}_{l}$.
Hence, we have:

feature warping:
$\displaystyle~{}~{}~{}{\bm{X}}^{\uparrow}_{l+1}=\mathcal{F}^{\text{Warp}}({\bm{X}}_{l+1},\bm{F}_{l})\in\mathbb{R}^{W_{l}\\!\times\\!H_{l}\\!\times\\!C_{l}},$
(5) DSPF warping:
$\displaystyle~{}~{}~{}\bm{D}_{l+1}^{\uparrow}=\mathcal{F}^{\text{Warp}}(\bm{D}_{l+1},\bm{F}_{l})\in\mathbb{R}^{W_{l}\\!\times\\!H_{l}\\!\times\\!2}.$

The warping operation $\mathcal{F}^{\text{Warp}\\!}$ upsamples the input
(${\bm{X}}_{l+1\\!}$ or $\bm{D}_{l+1}$) at the $(l\\!+\\!1)$-th level to match
the high resolution of the $l$-th level, according to the estimated feature
flow $\bm{F}_{l}$. Taking the feature warping in Eq. 5 as an example, we
have:

$\displaystyle{\bm{X}}^{\uparrow}_{l+1}(\bm{u})\\!=\\!\\!\\!\\!\sum_{\bm{u}^{\prime}_{s}\in\mathcal{N}(\bm{u}_{s})}\\!\\!\\!\\!{\bm{X}}^{~{}\\!\\!\Uparrow}_{l+1}(\bm{u}^{\prime}_{s})(1\\!-\\!|u_{s}\\!-\\!u^{\prime}_{s}|)(1\\!-\\!|v_{s}\\!-\\!v^{\prime}_{s}|),$
(6)

where $\bm{u}_{s}\\!=\\!\bm{u}\\!+\\!\bm{F}_{l}(\bm{u})\\!=\\!(u_{s},v_{s})$
denotes the source coordinate in ${\bm{X}}^{~{}\\!\\!\Uparrow}_{l+1}$,
$\bm{u}$ indicates the target coordinate in the interpolated output (,
${\bm{X}}^{\uparrow}_{l+1}$), and $\mathcal{N}(\bm{u}_{s})$ is the four pixel
neighbors of $\bm{u}_{s}$. Note that $\mathcal{F}^{\text{Warp}}$ is
differentiable, and its gradient can be effectively computed following [23].

###  3.3 Fully Differentiable Human Keypoint Detection and Association

So far, we have obtained the instance-agnostic human semantic part estimation
$\bm{S}$ (§3.1) and DSPF $\bm{D}$ (§3.2). Now we aim to precisely detect
human joints and assemble them into human instances. By doing this, we can
infer the instance-aware parsing result from $\bm{S}$, according to $\bm{D}$
(, the projection from dense human semantics to sparse keypoints; Fig. 3).
Hence, by formulating joint association as maximum-weight bipartite matching,
a differentiable solution is derived to make our model end-to-end trainable.

Prior to pose estimation, we apply a small decoder over
$\\{\bm{X}_{l}\\}_{l=1}^{L}$ to learn a pose-specific feature representation
$\bm{P}$:

$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}\bm{P}=\mathcal{F}^{\text{Pose}}(\\{\bm{X}_{l}\\}_{l=1}^{L}\\})\in\mathbb{R}^{W_{1}\\!\times\\!H_{1}\\!\times\\!C_{1}}.$
(7)

$\bm{P}$ merges multi-scale cues and maintains a high resolution.

Keypoint Detection. We first predict the locations of all the visible
anatomical keypoints (, left shoulder, right elbow) in $I$. Following [40,
27], we achieve this by jointly predicting confidence maps and 2D local offset
fields over $\bm{P}$. Assume there are a total of $K$ keypoint categories,
and, for each category $k$, we compute a heatmap
$M\\!\in\\![0,1]^{W_{1}\\!\times\\!H_{1}}$, where ${M}(\bm{u})\\!=\\!1$ if the
location $\bm{u}\\!=\\!(u,v)$ falls into a disk of radius $R$ ($=\\!32$
pixels) of any keypoint of category $k$; otherwise $M(\bm{u})\\!=\\!0$.
Moreover, for each category $k$, we compute an offset field
$\bm{O}\\!\in\\!\mathbb{R}^{W_{1}\\!\times\\!H_{1}\\!\times\\!2}$ to improve
the localization, where each offset $\bm{O}(\bm{u})$ points from $\bm{u}$ to
its closest keypoint of category $k$. The heatmap $M$ and offset field
$\bm{O}$ are aggregated by Hough voting to obtain a highly localized Hough
score map $H\\!\in\\!{[0,1]}^{W_{1}\\!\times\\!H_{1}}$, whose element at
location $\bm{u}$ is computed as:

$\displaystyle\\!\\!H(\bm{u})\\!=\\!\sum\nolimits_{\bm{u}^{\prime}\in\Omega}\frac{1}{\pi\\!R^{2}}M(\bm{u})\mathcal{F}^{\text{Bi-
inter}}(\bm{u}^{\prime}+\bm{O}_{k}(\bm{u}^{\prime})-\bm{u}),$ (8)

where $\mathcal{F}^{\text{Bi-inter}}$ is a bilinear interpolation function.
The local maxima in $H$ correspond to the keypoints of the $k$-th category. We
compute $K$ heatmaps and offset fields and conduct Hough voting individually
for each category (see Fig. 4 (a)). Through DSPF, we can get a set of
keypoint coordinates, , $\\{\bm{u}\\!+\\!\bm{D}(\bm{u})\\}_{\bm{u}\in\Omega}$.
With localized Hough score map of each category, we can determine the score at
each of these coordinates. Finally, the coordinates whose scores are larger
than $0.7$ are preserved as the detected keypoints.

Limb Scoring. We denote
$\mathcal{P}\\!=\\!\\{\bm{p}^{k}_{n}\\!:k\\!\in\\{1,\ldots,K\\},\\!~{}n\\!\in\\!\\{1,\ldots,N_{k}\\}\\}$
as the set of keypoints detected in $I$, where $N_{k}$ is the number of
keypoints belonging to the $k$-th category, and
$\bm{p}^{k}_{n}\\!=\\!(u^{k}_{n},v^{k}_{n})$ represents the 2D coordinate of
the $n$-th detected keypoint of the $k$-th category. The detected joints
$\mathcal{P}$ serve as candidate positions for human poses, and provide us
possible kinematic connections (, limbs) so that we can assemble $\mathcal{P}$
to form full body poses. To do so, we first need to score each possible limb
hypothesis. For each limb category, we predict a limb heatmap
$Q\\!\in\\!\mathbb{R}^{W_{1}\\!\times\\!H_{1}}$ over $\bm{P}$, which
represents the confidence of the limb (see Fig. 4 (a)). For simplicity, we
approximately represent each limb using an elliptical area between the
endpoints [29], and generate the ground-truths by applying an unnormalized
elliptical Gaussian distribution with a standard deviation $\sigma$ ($=\\!9$
pixels) over all limbs. The score of each limb is then estimated by sampling a
set of Gaussian responses within the limb area, from the corresponding heatmap
$Q$. Thus, for each limb with two joint categories $k$ and $k^{\prime}$, we
can get a scoring matrix
$A\\!\in\\!{[0,1]}^{N_{k}\\!\times\\!N_{k^{\prime}}}$, whose element
$a_{nn^{\prime}}$ stores the connectivity between joints $\bm{p}^{k}_{n}$ and
$\bm{p}^{k^{\prime}}_{n^{\prime}}$.

Figure 4: Illustration of our differentiable solver for human keypoint
detection and association. See §3.3 for details.

Differentiable Matching. The keypoint candidates $\mathcal{P}$ define a large
set of possible limbs. We score each limb to determine the likelihood of it
belonging to a particular person, according to which we parse optimal human
poses (, best joint matching). As shown in Fig. 4 (b), the problem of finding
the optimal matching corresponds to a $K$-dimensional matching problem that is
known to be NP-hard [51]. Thus, we decompose the matching problem into a set
of bipartite matching subproblems [4], , determine the matching for each
connected limb independently (see Fig. 4 (c)). Specifically, in each
bipartite graph, a matching corresponds to a subset of edges in which no two
edges share a node. Formally, we denote
$\mathcal{P}^{k\\!}\\!=\\!\\{\bm{p}^{k}_{n}\\}_{n=1}^{N_{k}}$ and
$\mathcal{P}^{k^{\prime}\\!}\\!=\\!\\{\bm{p}^{k^{\prime}}_{n^{\prime}}\\}_{n^{\prime}=1}^{N_{k^{\prime}}}$
as two sets of detected keypoints belonging to category $k$ and $k^{\prime}$,
respectively. We further define a variable $y_{nn^{\prime}}\\!\in\\!\\{0,1\\}$
to represent whether $\bm{p}^{k}_{n}$ and $\bm{p}^{k^{\prime}}_{n^{\prime}}$
are connected as a limb, and our goal becomes finding the optimal assignment
$\\{y_{nn^{\prime}\\!}\\}_{n,n^{\prime}}$ for the set of all possible
connections. For each limb with joint endpoint categories $k$ and $k^{\prime}$
and the corresponding scoring matrix
$A\\!\in\\!{[0,1]}^{N_{k}\\!\times\\!N_{k^{\prime}}}$, we infer the boolean
assignment matrix ${Y}\\!\in\\!\\{0,1\\}^{N_{k}\\!\times\\!N_{k^{\prime}}}$ by
solving the maximum-weight bipartite matching problem:

$\displaystyle\max_{{Y}}\sum\nolimits_{n=1}^{N_{k}}\\!\sum\nolimits_{n^{\prime}=1}^{N_{k^{\prime}}}\\!a_{nn^{\prime}}\cdot
y_{nn^{\prime}},$ (9) s.t.
$\displaystyle\forall{n},~{}~{}~{}~{}\sum\nolimits_{n^{\prime}=1}^{N_{k^{\prime}}}y_{nn^{\prime}}\leq
1,$ (10)
$\displaystyle\forall{{n^{\prime}}},~{}~{}~{}~{}\sum\nolimits_{n=1}^{N_{k}}y_{nn^{\prime}}\leq
1,$ (11)
$\displaystyle\forall(n,{n^{\prime}}),~{}~{}~{}~{}y_{nn^{\prime}\\!}\in\\{0,1\\},$
(12)

where $a_{nn^{\prime}}\\!\in\\!\bm{A}$ is the connection score between joints
$\bm{p}^{k}_{n}$ and $\bm{p}^{k^{\prime}}_{n^{\prime}}$. The constraints in
Eqs. (10-11) ensure that no two edges will share a node. The final multi-
person pose parsing result is with maximum weight for the chosen edges (,
connection scores). This integer linear programming problem can be solved
using Hungarian algorithm [28] with a high time complexity of $O(n^{3})$.
Existing pose estimators [4, 39, 27] instead employ heuristic greedy
algorithms, but break the end-to-end pipeline. Inspired by [61], we propose a
differentiable solution which facilitates model learning with direct matching
based supervision.

In particular, we drop the integer constraint (Eq. (12)) and compute $Y$ as a
real-valued assignment matrix, by solving a linear programming relaxation and
rewriting Eqs. (9-12) in matrix form:

$\displaystyle\min_{{Y}}\text{Tr}(-{A}{Y}^{\top}),$ (13)
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\textit{s.t.}~{}~{}~{}~{}{Y}\textbf{1}_{N_{k^{\prime}}}\leq\textbf{1}_{N_{k}},$
$\displaystyle~{}~{}~{}~{}{Y}^{\top}\textbf{1}_{N_{k}}\leq\textbf{1}_{N_{k^{\prime}}},~{}~{}~{}~{}{Y}\geq
0.$ (14)

Here, $\textbf{1}_{N_{k}\\!}\\!=\\![1]^{N_{k}\times 1\\!}$ and
$\textbf{1}_{N_{k^{\prime}}\\!}\\!=\\![1]^{N_{k^{\prime}}\times 1\\!}$. Since
both the target function (Eq. (13)) and constraint functions (Eq. (14)) are
convex, we can solve this convex constrained minimization problem using
projected gradient descent (PGD) [3]. Let us denote the constraints in Eq.
(14) as
$\mathcal{C}\\!=\\!\mathcal{C}_{1}\\!\cap\\!\mathcal{C}_{2}\\!\cap\\!\mathcal{C}_{3}$,
where
$\mathcal{C}_{1}\\!=\\!{Y}\textbf{1}_{N_{k^{\prime}\\!}}\\!\leq\\!\textbf{1}_{N_{k}}$,
$\mathcal{C}_{2}\\!=\\!{Y}^{\top\\!}\textbf{1}_{N_{k}\\!}\\!\leq\\!\textbf{1}_{N_{k^{\prime}}}$,
and $\mathcal{C}_{3}\\!=\\!{Y}\\!\geq\\!0$. PGD estimates $Y$ by iterating the
following equation:

$\displaystyle Y\leftarrow\mathcal{F}^{\mathcal{C}\\!}(Y-\alpha\triangledown
f(Y)),$ (15)

where $f(Y)\\!=\\!\text{Tr}(-AY^{\top\\!})$, $\triangledown f(Y)\\!=\\!-A$,
and $\alpha$ indicates the step size. Here $\mathcal{F}^{\mathcal{C}\\!}$ is a
projection operator, and itself is also an optimization problem:

$\displaystyle\mathcal{F}^{\mathcal{C}\\!}(Y)\\!=\\!\operatorname*{argmin}_{Y^{\prime}\in\mathcal{C}}\frac{1}{2}||Y^{\prime}\\!-\\!Y||_{2}^{2}.$
(16)

Given $Y$, $\mathcal{F}^{\mathcal{C}}$ tries to find a “point”
$Y^{\prime}\\!\in\\!\mathcal{C}$ which is closet to $Y$. The challenge here is
how to find a feasible projection operator over the combination of several
convex constraints, , $\mathcal{C}_{1}\cap\mathcal{C}_{2}\cap\mathcal{C}_{3}$.
A popular method for finding the projection onto the intersection of convex
subsets in Hilbert space is the Dykstra’s cyclic projection algorithm [10, 2].
It reduces the problem to an iterative scheme which involves only finding
projections from the individual sets. As the projection operator with respect
to each constraint can be easily derived in a differentiable form, Eqs.
(13-14) can be efficiently solved by PGD. In this way, our whole human part
parsing model is end-to-end trainable and can direct access to keypoint
matching supervision signals. After obtaining the assignment matrices for all
the limbs, we follow conventions [4, 27] to sequentially group the matched
keypoints from “head” to “foot”, as the estimations of “head” are usually more
accurate than other keypoints. Finally, multi-person pose parsing results can
be delivered (see Fig. 4 (d-e)).

###  3.4 Implementation Details

Loss Function. For category-level human semantic prediction $\bm{S}$ (§3.1),
cross-entropy loss is used. For DSPF estimation (§3.2), $l_{1}$ loss is used.
For keypoint detection and joint affinity estimation (§3.3), cross-entropy
loss is used for confidence/score map and assignment matrix prediction, and
$l_{1}$ loss is used for vector regression. The weights of different losses
are adaptively adjusted during training by considering the homoscedastic
uncertainty of each task [26].

Network Configuration. We compute the feature pyramid${}_{\\!}$
$\\{_{\\!}\bm{X}_{l\\!}\\}_{l=1\\!}^{L}$ at${}_{\\!}$ $L\\!\\!=\\!\\!4$
levels${}_{\\!}$ with${}_{\\!}$ a${}_{\\!}$ scaling${}_{\\!}$ step${}_{\\!}$
of${}_{\\!}$ $2$ [33]. We adopt ResNet-101 [18] as the backbone. The
segmentation head $\mathcal{F}^{\text{Seg}\\!}$ in Eq. (1) has a similar
architecture as the decoder in [5], while in each upsampling stage we apply a
single $5\\!\times\\!5$ depthwise-separable convolution to build a lightweight
module. The pose head $\mathcal{F}^{\text{Pose}\\!}$ in Eq. (7) is built in a
similar way. For $\mathcal{F}^{\text{Flow}\\!}$ in Eq. (4), it concatenates
the inputs and then applies a $3\\!\times\\!3$ convolutional layer for feature
flow prediction.

Training. We train our model in two steps. In the first step, the model is
trained without differentiable matching using the SGD optimizer with a mini-
batch size of $32$. We start the training with a base learning rate of 5e-2,
momentum of $0.9$ and weight decay of 1e-5. The learning rate is then reduced
to 5e-3 after $120$ epochs, and the training is terminated within $150$
epochs. In the second step, we finetune the whole model for another $50$
epochs with a fixed learning rate of 1e-4. During all training stages, we fix
the running statistics of batch normalization layers to the ImageNet-
pretrained values. Typical data augmentation techniques, , horizontal
flipping, scaling in range of [0.5, 2], rotation from
[-10$\mathrm{\SIUnitSymbolDegree}$, 10$\mathrm{\SIUnitSymbolDegree}$], and
color jitter, are also used.

Inference. Once trained, our model can be directly applied to unseen images.
For a test image, we detect a set of key-points
$\mathcal{P}\\!=\\!\\{\bm{p}^{k}_{n}\\}$ and group them into different human
instances. For each human instance $h$, we denote the set of its keypoints as
$\mathcal{P}_{h}\\!=\\!\\{\bm{p}_{n,h}\\}_{n}$. Then we associate each person
pixel (identified by the category-level semantic parsing module) to one of the
human instances. Formally, for each human pixel $\bm{u}$ and its DSPF vector
$\bm{D}(\bm{u})$, an assignment score for each human instance $h$ is computed
as:

$\small\psi_{h}=\min_{\bm{p}_{n,h}\in\mathcal{P}_{h}}(\frac{\|\bm{u}+\bm{D}(\bm{u})-\bm{p}_{n,h}\|_{2}}{(s_{n,h}+s_{h})*\sigma_{h}}).\vspace{-2pt}$
(17)

For each keypoint $\bm{p}_{n,h\\!}$ belonging to the human instance $h$, its
distance to $\bm{u}\\!+\\!\bm{D}(\bm{u})$, , $\bm{u}$’s associated human
instance keypoint, is first computed and normalized by joint score${}_{\\!}$
(,${}_{\\!}$ $s_{n,h\\!\\!}\\!=_{\\!}\\!\\!H(\bm{p}_{n,h})$),
instance${}_{\\!}$ score (,${}_{\\!}$
$s_{h\\!}\\!=_{\\!}\\!\frac{1}{|\mathcal{P}_{h}|}$
$\sum_{\bm{p}_{n,h}\in\mathcal{P}_{h}\\!}H(\bm{p}_{n,h})$), and instance scale
(, $\sigma_{h}$) for reliable, scale-invariant evaluation. $\sigma_{h}$ is set
as the square root of the area of the bounding box tightly containing
$\mathcal{P}_{h}$. Finally, the human instance for $\bm{u}$ is assigned as:
$\arg\min_{h}\psi_{h}$.

##  4 Experiment

###  4.1 Experimental Setup

Datasets: Our experiments are conducted on three datasets:

  * •

MHPv2 [64] is currently the largest dataset for instance-aware human parsing.
In total, it includes $25,\\!403$ images ($15,\\!403/5,\\!000/5,\\!000$ for
train/val/test splits). Each person is elaborately annotated with $58$ fine-
grained semantic categories (, $11$ body parts, $47$ cloth and accessory
labels), as well as $16$ body joints.

  * •

DensePose-COCO [1] has $27,\\!659$ images ($26,\\!151$/$1,\\!508$ for
train/test splits) gathered from COCO [34]. It annotates $14$ human parts and
$17$ keypoints.

  * •

PASCAL-Person-Part [53] has $1,\\!716$ and $1,\\!817$ images for train and
test, respectively, with annotations of $6$ human part categories and $14$
body joints.

Evaluation Metrics.${}_{\\!}$ For${}_{\\!}$ instance-agnostic${}_{\\!}$
parsing, we${}_{\\!}$ adopt mean intersection-over-union (mIoU). For instance-
level parsing, we employ official metrics of each dataset for fair comparison.
Specifically, for MHPv2 and DensePose-COCO, the average precision based on
parts (APp) and percentage of correctly parsed semantic parts (PCP) are used.
For PASCAL-Person-Part, we follow conventions [16, 52] to report the average
precision based on regions (APr).

Methods | mIoU | AP${}_{50}^{p}$ | AP${}_{vol}^{p}$ | PCP50 | M-Adds (G)
---|---|---|---|---|---
top-down models:
M-RCNN [17] | - | 14.9 | 33.9 | 25.1 | 141.3
P-RCNN [58] | 40.3 | 30.2 | 41.8 | 44.2 | 220.7
M-CE2P [42] | 41.1 | 30.9 | 41.3 | 40.6 | 497.1
SNT [24] | - | 34.4 | 42.5 | 43.5 | 520.5
RP-RCNN [57] | 37.3 | 40.5 | 45.2 | 39.2 | 185.5
bottom-up models:
PGN [14] | 25.3 | 17.6 | 35.5 | 26.9 | 169.2
MHParser [30] | - | 18.0 | 36.1 | 27.0 | -
NAN [64] | - | 25.1 | 41.8 | 32.3 | 302.2
Ours | 41.4 | 39.0 | 44.3 | 42.3 | 151.1

Table 1: Quantitative performance comparison on MHPv2 val [64], with mIOU,
APp and PCP. See §4.2 for details.

Methods | mIoU | AP${}_{50}^{p}$ | AP${}_{vol}^{p}$ | PCP50
---|---|---|---|---
top-down models:
P-RCNN [58] | 65.9 | 43.5 | 53.1 | 51.8
M-CE2P [42] | 67.1 | 43.7 | 52.9 | 51.2
RP-RCNN [57] | 65.3 | 48.5 | 54.5 | 51.1
bottom-up models:
PGN [14] | 46.1 | 23.4 | 35.9 | 32.5
NAN [64] | 58.9 | 37.6 | 48.3 | 43.9
Ours | 69.1 | 49.7 | 54.7 | 52.8

Table 2: Quantitative performance comparison on DensePose-COCO test [1], with
mIOU, APp and PCP. See §4.2 for details.

Reproducibility. Our model is implemented in PyTorch and trained on eight
NVIDIA Tesla V100 GPUs with a 32GB memory per-card. Testing is conducted on a
single NVIDIA Xp GPU with 11 GB memory.

###  4.2 Quantitative Results

MHPv2 [64]: Table 1 reports comparison results against five top-down and three
bottom-up approaches on MHPv2 val. Our approach significantly outperforms
existing bottom-up models, , MHParser [30] and NAN [64], across all metrics.
This indicates the importance of our multi-granularity representation learning
in instance-level parsing. Furthermore, our approach shows better overall
performance than most top-down methods (, SNT [24], P-RCNN [58], M-CE2P
[42]), revealing our appealing performance. Compared to RP-RCNN [57], our
approach only performs slightly worse in terms of instance-level metrics (,
AP${}_{50}^{p}$). However, our model improves the mIoU by $4\%$ and is much
more efficient (see §4.3).

Methods | AP${}_{50}^{r}$ | AP${}_{60}^{r}$ | AP${}_{70}^{r}$ | AP${}_{vol}^{r}$
---|---|---|---|---
top-down models:
MNC [8] | 38.8 | 28.1 | 19.3 | 36.7
Li et al. [31] | 40.6 | 30.4 | 19.1 | 38.4
HAZN [52] | 43.7 | - | - | -
P-RCNN [58] | 52.9 | 43.0 | 31.1 | 48.5
M-CE2P [42] | 53.3 | 45.6 | 31.6 | 51.9
RP-RCNN [57] | 59.9 | 51.3 | 37.8 | 55.8
bottom-up models:
PGN [14] | 39.6 | 29.9 | 20.0 | 39.2
MH-Parser [30] | 42.3 | 34.2 | 20.1 | 40.0
NAN [64] | 59.7 | 51.4 | 38.0 | 52.2
Ours | 59.0 | 52.3 | 38.1 | 55.9

Table 3: Quantitative performance comparison on PASCAL-Person-Part test [53],
with APr. See §4.2 for details.

DensePose-COCO [1]: Table 2 presents comparisons with five representative
approaches on DensePose-COCO val. Our method sets new state-of-the-arts across
all metrics, outperforming all other methods by a large margin. For example,
our parser provides a considerable performance gain in AP${}_{50}^{p}$, ,
$1.2\%$ and $12.1\%$ higher than the current best top-down (, RP-RCNN [57])
and bottom-up (, NAN [64]) models, respectively.

PASCAL-Person-Part [53]: Table 3 compares our model against six top-down and
three bottom-up methods on PASCAL-Person-Part val. The results demonstrate
that our human parser outperforms other competitors across most metrics (,
AP${}_{60}^{r}$, AP${}_{70}^{r}$, AP${}_{vol}^{r}$). We also observe that
performance on this dataset has become saturated, due to its small-scale
training set (, only $1,\\!716$ images).

  Speed (s) Methods avg. for 6 persons top-down models: SNT [24]
3.27 M-CE2P [42] 1.15 P-RCNN [58] 0.51 RP-RCNN [57] 0.43 bottom-up models:
MHParser [30] 14.9 NAN [64] 1.08 PGN [14] 0.54 Ours 0.15

Figure 5: Runtime analysis. Our model is fully convolutional and allows for
efficient inference, irrespective of the number of people in the image. In
contrast, the runtimes of top-down approaches (, M-CE2P [42], P-RCNN [58],
RP-RCNN [57]) grow linearly with the number of people. The runtime comparison
on MHPv2 val reported in the table demonstrates again that our model is
significantly faster than other methods. See §4.3 for details.  Figure 6:
Instance-level human semantic parsing results (§4.4) in challenging scenarios
with crowding, occlusions, pose variations, .

###  4.3 Runtime Analysis

We collect images with a varying number of people for runtime analysis. Each
analysis was repeated 500 times and then averaged. We compare with six state-
of-the-art methods, including three top-down (, P-RCNN [58], RP-RCNN [57],
M-CE2P [42]) and three bottom-up (, NAN [64], MHParser [30], PGN [14])
models. As depicted in fig:runtime, the inference times of the top-down
models are roughly proportional to the number of people in the image, while
they are invariant for the bottom-up approaches. The table summarizes the
inference time of several parsers on MHPv2 val with average six people per
image. Overall, our model is much faster than existing methods.

###  4.4 Qualitative Results

As shown in fig:visual, our approach can produce precise instance-level human
semantic parsing results. It shows strong robustness to many real-world
challenges, such as crowding, occlusions, pose and appearance variations, .

coarse-to-fine | residual refine. | learn. warping |  |  |
---|---|---|---|---|---
(Eq.(2)) | (Eq.(3)) | (Eqs. (4-6)) | AP${}_{50}^{p}$ | AP${}_{vol}^{p}$ | PCP50
|  |  | 35.5 | 40.8 | 39.3
✓ |  |  | 36.9 | 42.4 | 40.3
✓ | ✓ |  | 38.1 | 43.4 | 41.5
✓ | ✓ | ✓ | 38.9 | 44.3 | 42.2

Table 4: Ablation study on DSPF estimation (§3.2) on MHPv2 val [64]. See §4.5
for details.

###  4.5 Diagnostic Experiments

DSPF Breakdown. Table 4 investigates the necessity of our essential modules
for DSPF estimation (§3.2). We start the analysis with a baseline model (,
$1_{st}$ row) which directly regresses DSPF over multi-layer features
$\\{\bm{X}_{l}\\}_{l=1}^{L}$. Then, we progressively improve the baseline
with: (1) a ‘coarse-to-fine’ strategy (, inferring a complete DSPF at each
pyramid level); (2)‘residual refinement’ (, using residual learning with
bilinear feature unsampling); and (3) ‘learnable feature warping’ (, learning
cross-layer feature alignment). Consistent performance improvements can be
observed after introducing the above modifications and we can draw three
essential conclusions: (1) Coarse-to-fine inference is critical for accurate
DSPF regression, since it makes full use of multi-scale information. (2)
Residual learning is also useful, since the small magnitude of the residue
facilitates training. (3) With the learnable feature warping, cross-layer
features/DSPF predictions are better aligned, further facilitating network
learning.

Differentiable Matching. We further study the proposed differentiable solver
(§3.3) for human joint association by comparing it with two greedy matching
algorithms [39, 4] on MHPv2 val (see Table 5). The differentiable solver
allows directly back-propagating joint grouping errors, leading to great
improvements in multi-human pose estimation (under mAPpose [34]). In
addition, benefiting from end-to-end learning, our model also shows better
human part parsing results (in terms of AP${}_{50}^{p}$ and mIoU), further
confirming the superiority of our differentiable matching strategy.

Diff. Matching (§3.3) | AP${}_{50}^{p}$ | AP${}_{vol}^{p}$ | PCP50 | mIoU | mAPpose
---|---|---|---|---|---
greedy matching [39] | 37.6 | 43.1 | 40.8 | 40.8 | 63.9
greedy matching [4] | 37.9 | 43.2 | 40.6 | 41.0 | 64.3
differentiable matching | 39.0 | 44.3 | 42.3 | 41.4 | 65.8

Table 5: Ablation study on differentiable matching (§3.3) on MHPv2 val [64].
See §4.5 for details.

##  5 Conclusion

This work presents a new perspective of exploring human structural information
over multiple granularities to address instance-aware human semantic parsing.
A dense-to-sparse projection field is learnt to associate human semantic parts
with human keypoints. The field is progressively optimized with residual
refinement to ease the optimization. A differentiable joint matching solver is
further proposed for body joint assembling. These designs together yield an
end-to-end trainable, bottom-up instance-aware human semantic parser. With its
accurate and fast computation, our parser is expected to pave the way for
practical use and benefit vast quantities of human-centric applications.

## References

  * [1] Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.  Densepose: Dense human pose estimation in the wild.  In CVPR, 2018.
  * [2] James P Boyle and Richard L Dykstra.  A method for finding projections onto the intersection of convex sets in hilbert spaces.  In Advances in order restricted statistical inference, pages 28–47, 1986.
  * [3] Paul H Calamai and Jorge J Moré.  Projected gradient methods for linearly constrained problems.  Mathematical programming, 39(1):93–116, 1987.
  * [4] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In CVPR, 2017.
  * [5] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In ECCV, 2018.
  * [6] Xinlei Chen, Ross Girshick, Kaiming He, and Piotr Dollár.  Tensormask: A foundation for dense object segmentation.  In ICCV, 2019.
  * [7] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun.  Cascaded pyramid network for multi-person pose estimation.  In CVPR, 2018.
  * [8] Jifeng Dai, Kaiming He, and Jian Sun.  Instance-aware semantic segmentation via multi-task network cascades.  In CVPR, 2016.
  * [9] Jian Dong, Qiang Chen, Xiaohui Shen, Jianchao Yang, and Shuicheng Yan.  Towards unified human parsing and pose estimation.  In CVPR, 2014.
  * [10] Richard L Dykstra.  An algorithm for restricted least squares regression.  Journal of the American Statistical Association, 78(384):837–842, 1983.
  * [11] Lifeng Fan, Wenguan Wang, Siyuan Huang, Xinyu Tang, and Song-Chun Zhu.  Understanding human gaze communication by spatio-temporal graph reasoning.  In ICCV.
  * [12] Hao-Shu Fang, Guansong Lu, Xiaolin Fang, Jianwen Xie, Yu-Wing Tai, and Cewu Lu.  Weakly and semi supervised human body part parsing via pose-guided knowledge transfer.  In CVPR, 2018.
  * [13] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu.  Rmpe: Regional multi-person pose estimation.  In ICCV, 2017.
  * [14] Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin.  Instance-level human parsing via part grouping network.  In ECCV, 2018.
  * [15] Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin.  Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing.  In CVPR, 2017.
  * [16] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik.  Simultaneous detection and segmentation.  In ECCV, 2014.
  * [17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In ICCV, 2017.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [19] Shaoli Huang, Mingming Gong, and Dacheng Tao.  A coarse-fine network for keypoint localization.  In ICCV, 2017.
  * [20] Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy.  Liteflownet: A lightweight convolutional neural network for optical flow estimation.  In CVPR, 2018.
  * [21] Junhwa Hur and Stefan Roth.  Iterative residual refinement for joint optical flow and occlusion estimation.  In CVPR, 2019.
  * [22] Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schiele.  Deepercut: A deeper, stronger, and faster multi-person pose estimation model.  In ECCV, 2016.
  * [23] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  In NeurIPS, 2015.
  * [24] Ruyi Ji, Dawei Du, Libo Zhang, Longyin Wen, Yanjun Wu, Chen Zhao, Feiyue Huang, and Siwei Lyu.  Learning semantic neural tree for human parsing.  In ECCV, 2020.
  * [25] Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, and Ping Luo.  Differentiable hierarchical graph grouping for multi-person pose estimation.  In ECCV, 2020.
  * [26] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In CVPR, 2018.
  * [27] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi.  Pifpaf: Composite fields for human pose estimation.  In CVPR, 2019.
  * [28] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [29] Jia Li, Wen Su, and Zengfu Wang.  Simple pose: Rethinking and improving a bottom-up approach for multi-person pose estimation.  In AAAI, 2020.
  * [30] Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, Terence Sim, Shuicheng Yan, and Jiashi Feng.  Multiple-human parsing in the wild.  arXiv preprint arXiv:1705.07206, 2017.
  * [31] Qizhu Li, Anurag Arnab, and Philip HS Torr.  Holistic, instance-level human parsing.  arXiv preprint arXiv:1709.03612, 2017.
  * [32] Xiaodan Liang, Chunyan Xu, Xiaohui Shen, Jianchao Yang, Si Liu, Jinhui Tang, Liang Lin, and Shuicheng Yan.  Human parsing with contextualized convolutional neural network.  In ICCV, 2015.
  * [33] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [34] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [35] Si Liu, Yao Sun, Defa Zhu, Guanghui Ren, Yu Chen, Jiashi Feng, and Jizhong Han.  Cross-domain human parsing via adversarial feature and label adaptation.  In AAAI, 2018.
  * [36] Alejandro Newell, Zhiao Huang, and Jia Deng.  Associative embedding: End-to-end learning for joint detection and grouping.  In NeurIPS, 2017.
  * [37] Xuecheng Nie, Jiashi Feng, and Shuicheng Yan.  Mutual learning to adapt for joint human parsing and pose estimation.  In ECCV, 2018.
  * [38] Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan.  Single-stage multi-person pose machines.  In ICCV, 2019.
  * [39] George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy.  Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model.  In ECCV, 2018.
  * [40] George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and Kevin Murphy.  Towards accurate multi-person pose estimation in the wild.  In CVPR, 2017.
  * [41] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu.  Learning human-object interactions by graph parsing neural networks.  In ECCV.
  * [42] Tao Ruan, Ting Liu, Zilong Huang, Yunchao Wei, Shikui Wei, and Yao Zhao.  Devil in the details: Towards accurate single and multiple human parsing.  In AAAI, 2019.
  * [43] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao.  Human-aware motion deblurring.  In ICCV.
  * [44] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In CVPR, 2018.
  * [45] Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, and Dahua Lin.  Carafe: Content-aware reassembly of features.  In ICCV, 2019.
  * [46] Wenguan Wang, Yuanlu Xu, Jianbing Shen, and Song-Chun Zhu.  Attentive fashion grammar network for fashion landmark detection and clothing category classification.  In CVPR, 2018.
  * [47] Wenguan Wang, Zhijie Zhang, Siyuan Qi, Jianbing Shen, Yanwei Pang, and Ling Shao.  Learning compositional neural information fusion for human parsing.  In ICCV, 2019.
  * [48] Wenguan Wang, Tianfei Zhou, Siyuan Qi, Jianbing Shen, and Song-Chun Zhu.  Hierarchical human semantic parsing with comprehensive part-relation modeling.  IEEE TPAMI, 2021.
  * [49] Wenguan Wang, Hailong Zhu, Jifeng Dai, Yanwei Pang, Jianbing Shen, and Ling Shao.  Hierarchical human parsing with typed part-relation reasoning.  In CVPR.
  * [50] Fangyun Wei, Xiao Sun, Hongyang Li, Jingdong Wang, and Stephen Lin.  Point-set anchors for object detection, instance segmentation and pose estimation.  In ECCV, 2020.
  * [51] Douglas Brent West et al.  Introduction to graph theory, volume 2.  Prentice hall Upper Saddle River, NJ, 1996.
  * [52] Fangting Xia, Peng Wang, Liang-Chieh Chen, and Alan L Yuille.  Zoom better to see clearer: Human and object parsing with hierarchical auto-zoom net.  In ECCV, 2016.
  * [53] Fangting Xia, Peng Wang, Xianjie Chen, and Alan L Yuille.  Joint multi-person pose estimation and semantic part segmentation.  In CVPR, 2017.
  * [54] Fangting Xia, Jun Zhu, Peng Wang, and Alan L Yuille.  Pose-guided human parsing by an and/or graph using pose-context features.  In AAAI, 2016.
  * [55] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo.  Polarmask: Single shot instance segmentation with polar representation.  In CVPR, 2020.
  * [56] Kota Yamaguchi, M Hadi Kiapour, Luis E Ortiz, and Tamara L Berg.  Parsing clothing in fashion photographs.  In CVPR, 2012.
  * [57] Lu Yang, Qing Song, Zhihui Wang, Mengjie Hu, Chun Liu, Xueshi Xin, Wenhe Jia, and Songcen Xu.  Renovating parsing R-CNN for accurate multiple human parsing.  In ECCV, 2020.
  * [58] Lu Yang, Qing Song, Zhihui Wang, and Ming Jiang.  Parsing R-CNN for instance-level human analysis.  In CVPR, 2019.
  * [59] Ze Yang, Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin.  Reppoints: Point set representation for object detection.  In ICCV, 2019.
  * [60] Ze Yang, Yinghao Xu, Han Xue, Zheng Zhang, Raquel Urtasun, Liwei Wang, Stephen Lin, and Han Hu.  Dense reppoints: Representing visual objects with dense point sets.  In ECCV, 2020.
  * [61] Xiaohui Zeng, Renjie Liao, Li Gu, Yuwen Xiong, Sanja Fidler, and Raquel Urtasun.  Dmm-net: Differentiable mask-matching network for video object segmentation.  In ICCV, 2019.
  * [62] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu.  Pose2seg: Detection free human instance segmentation.  In CVPR, 2019.
  * [63] Ziwei Zhang, Chi Su, Liang Zheng, and Xiaodong Xie.  Correlating edge, pose with parsing.  In CVPR, 2020.
  * [64] Jian Zhao, Jianshu Li, Yu Cheng, Terence Sim, Shuicheng Yan, and Jiashi Feng.  Understanding humans in crowded scenes: Deep nested adversarial learning and a new benchmark for multi-human parsing.  In ACMMM, 2018.
  * [65] Tianfei Zhou, Siyuan Qi, Wenguan Wang, Jianbing Shen, and Song-Chun Zhu.  Cascaded parsing of human-object interaction recognition.  IEEE TPAMI, 2021.


11institutetext: 1Warsaw University of Technology 2CERN
3Jagiellonian University 4Tooploox
11email<EMAIL_ADDRESS>
# Selectively increasing the diversity of GAN-generated samples

Jan Dubiński 11 Kamil Deja 11 Sandro Wenzel 22
Przemysław Rokita 11 Tomasz Trzcinski 113344

###### Abstract

Generative Adversarial Networks (GANs) are powerful models able to synthesize
data samples closely resembling the distribution of real data, yet the
diversity of those generated samples is limited due to the so-called mode
collapse phenomenon observed in GANs. Especially prone to mode collapse are
conditional GANs, which tend to ignore the input noise vector and focus on the
conditional information. Recent methods proposed to mitigate this limitation
increase the diversity of generated samples, yet they reduce the performance
of the models when similarity of samples is required. To address this
shortcoming, we propose a novel method to selectively increase the diversity
of GAN-generated samples. By adding a simple, yet effective regularization to
the training loss function we encourage the generator to discover new data
modes for inputs related to diverse outputs while generating consistent
samples for the remaining ones. More precisely, we maximise the ratio of
distances between generated images and input latent vectors scaling the effect
according to the diversity of samples for a given conditional input. We show
the superiority of our method in a synthetic benchmark as well as a real-life
scenario of simulating data from the Zero Degree Calorimeter of ALICE
experiment in LHC, CERN.

###### Keywords:

Generative Adversarial Networks, Generative Models, Data Simulation

##  1 Introduction

Generative Adversarial Networks (GANs) [4] constitute a gold standard for
synthesizing complex data distributions and they are, therefore, widely used
across various applications, including data augmentation [12], image
completion [20] or representation learning [21]. They are also employed in
high energy physics experiments at the Large Hadron Collider (LHC) at CERN,
where they allow to speed up the process of simulating particle collisions
[13, 1, 7]. In this context, the generative models are used to generate
samples of possible detectors’ responses resulting from a collision of
particles described with a series of physical parameters. For more
controllable simulations, we can condition the generative models with
additional parameters of the collision, using conditional GANs (cGANs) [11].

Although by conditioning GANs we obtain more context-dependent generations
that are closer to the values observed in real experiments, these models are
considered more vulnerable to the so-called mode collapse phenomenon [17] ,
observed as a tendency to generate a limited number of different outputs per
each conditional prior. This, in turn, significantly reduces the effectiveness
of employing GANs for particle collision simulations, as alignment of
generated samples with the real data distribution is fundamental for drawing
correct conclusions from the performed experiments.

To address the above-mentioned limitations of cGANs, recent methods [19, 10,
9] attempt to increase the diversity of generated samples by modifying the
associated cost function. However, they do not consider conditioning the
diversity on the input conditioning values, assuming a uniform distribution of
diversity across all of them. This assumption is rarely observed in practical
applications, for instance in particle collision simulations at CERN diversity
of generated samples highly depends on the set of conditioning variables.

In this work, we identify this shortcoming of existing models and propose a
simple, yet effective method to selectively increase the diversity of GAN-
generated samples, based on conditioning values. In principle, we introduce a
regularization method that enforces GANs to follow diversity observed in the
original dataset for a given conditional value. More exactly, we maximise the
ratio of distances between latent vectors of generated images and inputs,
scaling the effect accordingly to the diversity of samples corresponding to a
given conditional input. Our approach, dubbed SDI-GAN, is readily applicable
for conditional image synthesis models and does not require any modification
of the baseline GAN architecture.

We evaluate our method on a challenging task of simulating data from the Zero
Degree Calorimeter of the ALICE experiment in LHC, CERN. To better demonstrate
the performance of our method we also include a synthetic dataset for 2D point
generation. We compare our approach with competing methods and achieve
superior results across all benchmarks.

The main contribution of this paper is a novel method for increasing the
diversity of GAN-generated results for a selected subset of conditional input
data while keeping the consistency of the results for the remaining
conditional inputs.

##  2 Related work

####  2.0.1 Generative simulations:

The need for simulating complex processes exists across many scientific
domains. In recent years, solutions based on generative machine learning
models have been proposed as an alternative to existing methods in cosmology
[16] and genetics [15]. However, one of the most profound applications for
generative simulations is in the field of High Energy Physics, where machine
learning models can be used as a resource-efficient alternative to classic
Monte Carlo-based[6] approaches.

Recent attempts [13, 7, 3] leverage solutions based on Generative Adversarial
Networks [4] or Variational Autoencoders [8]. Although those methods offer
considerable speed-up of the simulation process, they also suffer from the
limitations of existing generative models. Controlling the diversity of
simulated results while maintaining the high fidelity of the simulation is one
of the challenges of using generative models for such applications.

####  2.0.2 Mode collapse and sample diversity in cGAN:

The authors of MS-GAN [10] address the mode collapse problem in cGANs by
introducing mode-seeking loss. This additional regularization term added to
the generator training function aims to improve generation diversity by
maximizing the dissimilarity between two images generated from two different
latent codes. During training, the generator tries to minimize the
regularization term added to the loss function equal to the inverse of
measured diversity.

DS-GAN [19] tries to tackle the problem with a similar approach. The main
difference between the two methods is the fact that DS-GAN explicitly
maximizes the measured diversity which is subtracted from the training loss of
the generator.

In DivCo [9] the authors use contrastive learning to achieve diverse
conditional image synthesis. They introduce a latent-augmented contrastive
loss which encourages images generated from distant latent codes to be
dissimilar and those generated from close latent codes to be similar. The
similarity of images is measured using their latent representations extracted
from the discriminator network.

Our approach shares a similar method of calculating the diversity of images
with [10] and [19]. However, contrary to those approaches we do not base our
measure of diversity on pixels of generated images. Instead we operate on
image representation, similarly to [9]. Moreover, in principle, all previously
described approaches do not account for different levels of variance of
samples corresponding to different conditional inputs and instead maximize the
diversity of the results generated for all conditional inputs.

##  3 Methodology

In this work we propose a novel selective diversity regularization method that
improves alignment between generations from cGAN and data conditioned on
available conditional values. For a dataset of images $\mathcal{X}$ with
conditioning values $\mathcal{C}$ we want to learn a generator $G$ that is
able to produce realistic samples from the domain of $\mathcal{X}$ conditioned
on $c\in\mathcal{C}$. Moreover, we want the synthesised images to be diverse
or similar to each other depending on the variance of samples in $\mathcal{X}$
that correspond to a given $c$.

Traditional conditional GANs are trained using adversarial loss. Given a
condition vector $c\in\mathcal{C}$ and a $k$-dimensional latent code
$z\sim\mathcal{N}_{k}(0,1)$ the generator $G$ takes both $c$ and $z$ as input
and produces an output image $\hat{x}=G(z,c)$. The image $\hat{x}$ should be
indistinguishable from real data by a discriminator $D$. During training
generator and discriminator play a min-max game, in which $D$ learns to
distinguish real data from samples synthesised by $G$ while $G$ tries to
generate samples that are considered by $D$ as real.

$\displaystyle\mathcal{L}_{\mathrm{adv}}$
$\displaystyle(G,D)=\mathbb{E}_{x\sim\mathcal{X},c\sim\mathcal{C}}[\log
D(x,c)]+\mathbb{E}_{c\sim\mathcal{C},z\sim\mathcal{N}(0,1)}[\log(1-D(G(z,c),c)]$
(1)

The adversarial loss function encourages the generator to produce realistic
data, but as observed by [9] it does not directly promote the diversity of
synthesised samples. To alleviate this problem Mao et al. [10] propose a
regularization term that penalizes the low diversity of generated samples.
More precisely, the introduced method maximizes the ratio of the distance
between two images generated from two different latent codes $z_{1}$, $z_{2}$
and the same conditioning value $c$ with respect to the distance between those
latent codes. The proposed regularization term is added to the basic loss
function from Eq. 1

$\displaystyle\mathcal{L}_{\mathrm{ms}}=\left(\frac{d_{\mathbf{I}}\left(G\left(c,\mathbf{z}_{1}\right),G\left(c,\mathbf{z}_{2}\right)\right)}{d_{\mathbf{z}}\left(\mathbf{z}_{1},\mathbf{z}_{2}\right)}\right)^{-1}$
(2)

Although, this approach successfully forces the generator to produce
dissimilar examples it does not account for different levels of sample
diversity for different conditioning input $c$. To address this issue we
propose a simple yet effective modification of the regularization term.

As a data preprocessing step, for each unique conditioning input
$c\in\mathcal{C}$ we calculate the diversity of samples from the dataset
$\mathcal{X}$ that correspond to $c$ denoted as $\mathcal{X}_{c}$. We base our
measure of diversity $f_{div}$ on the variance of samples. As denoted in Eq.
3, for each set of images $\mathcal{X}_{c}$ we sum the standard deviation of
pixel values with the same coordiantes for images in $\mathcal{X}_{c}$:

$f_{div}(c)=\sum_{i,j}\sqrt{\frac{\sum_{t}{(x_{ij}^{t}-\mu_{ij})^{2}}}{|X_{c}|}}$
(3)

where $i$ and $j$ are the pixel coordinates, $t$ is the index of sample $x\in
X_{c}$ and $\mu_{ij}$ is a mean value for a pixel $ij$ from all samples from
$X_{c}$. We normalize the obtained values of all sample diversity $f_{div}(c)$
to $<0,1>$.

To account for varying levels of sample diversity for each conditioning input
$c$ we multiply the regularization term introduced before by $f_{div}(c)$.

$\displaystyle\mathcal{L}_{\mathrm{div}}=f_{div}(c)*\left(\frac{d_{\mathbf{I}}\left(G\left(c,\mathbf{z}_{1}\right),G\left(c,\mathbf{z}_{2}\right)\right)}{d_{\mathbf{z}}\left(\mathbf{z}_{1},\mathbf{z}_{2}\right)}\right)^{-1}$
(4)

This change forces the generator to better match the diversity observed in the
original dataset with respect to conditional values. Intuitively, the
generator produces more diverse images under conditions that allow for a
higher variety of synthesised samples. At the same time, the generator is not
forced to produce dissimilar results if the conditioning value corresponds to
a set of similar images in the dataset. The overall objective of training SDI-
GAN is the following:

$\displaystyle\mathcal{L}=\mathcal{L}_{\mathrm{adv}}(G,D)+\lambda_{\mathrm{div}}\mathcal{L}_{\mathrm{div}}(G)$
(5)

where $\lambda_{\mathrm{div}}$ is a hyperparameter controlling the strength of
the regularization.

Additionally, to better adapt our method to the task of applying cGANs as a
fast simulation tool, we propose to base the distance $d_{g}$ on the
dissimilarity of latent representations of images rather than the
dissimilarity of pixels. We measure the distance between two generated images
by calculating the L1 metric between their latent representations. We obtain
those representations by treating the initial layers of the discriminator as
an encoder $E$ and extracting the latent features of the images from its
penultimate layer during training.

$e_{zc}=E(G(z,c))$ (6) $d_{g}(G(z_{1},c),G(z_{1},c))=|e_{z_{1}c},e_{z_{2}c}|$
(7)

This change shifts the focus of the generator from the visual dissimilarity of
images to the difference in their underlying characteristics extracted by the
encoder.

##  4 Experiments

We evaluate our method on simulating data from the Zero Degree Calorimeter
from the ALICE experiment in LHC, CERN. Additionally, we use a synthetic
dataset as a benchmark. We compare our approach to a conditional DC-GAN [14],
MS-GAN [10] and DivCo [9].

###  4.1 Synthetic dataset

To clearly demonstrate the effects of our method and its comparison to
competing approaches we provide a synthetic dataset of 2D points generation.
Each point is conditioned on a class label (from 1 to 9) and a binary variable
spread. The position of the generated point depends mainly on its class label.
However, points with spread = False are generated very close to each other,
while points with spread = True are heavily dispersed. The dataset is
presented in Fig. 1.

spread = False |  |  |  |  spread = True
---|---|---|---|---
|  |  |  |
Figure 1: Synthetic dataset. For each class the generated points form a
cluster. For point with spread = False the variance of each cluster is equal
to 1 and for points with spread = True the variance of each cluster is equal
to 100.

###  4.2 Zero Degree Calorimeter simulation

The task of simulating the response of the Zero Degree Calorimeter (ZDC)
offers a challenging benchmark for generative models. The dataset consists of
295867 samples obtained from the GEANT4 [6] simulation tool. Each response is
created by a single particle described with 9 attributes (mass, energy,
charge, momenta, primary vertex).

During the simulation process, the particle is propagated through the detector
for over 100 meters while simulation tools must account for all of its
interactions with the detector’s matter. The end result of the simulation is
the energy deposited in the calorimeter’s fibres, which are arranged in a grid
with 44 × 44 size. We treat the calorimeter’s response as a 1-channel image
with 44 × 44 pixels, where pixel values are the number of photons deposited in
a given fibre. To create the dataset the simulation was run multiple times for
the same input particles. For that reason, multiple possible outcomes
correspond to the same particle properties. We refer to this dataset as HEP.

Although the process that governs the propagation of the particles is non-
deterministic by nature, the majority of particles create consistent ZDC
responses. However, a subset of particles produces highly diverse results and
allows for multiple possible calorimeter responses. In Fig. 2 we present
sampled simulations for two different conditional values. In the first row, we
depict the calorimeter response for a high-energy proton, while for the second
the input particle is a neutron, which has no electric charge, therefore
responses observed in the calorimeter are much more consistent.

diverse results |
---|---
consistent results |
Figure 2: Examples of ZDC calorimeter responses. We show 4 possible outputs of
the simulation generated for 2 distinct particles.

###  4.3 Results

Table 1: Results comparison on the synthetic and HEP datasets. Our solution outperforms competing approaches by generating data with variance close to the real data for both types of conditioning input in the synthetic case. We achieve the lowest Wasserstein distance between test and generated coordinates for the diverse points without any significant trade-off for the consistent points. The performance improvement introduced by SDI-GAN is further evaluated on the real-life HEP dataset, where SDI-GAN achieves the lowest Wasserstein distance between channels calculated from original and generated data. | Synthetic dataset | HEP
---|---|---
| Variance | Wassesrstein $\downarrow$ | Wassesrstein $\downarrow$
Spread | 0 | 1 | 0 | 1 | -
Real | 1 | 100 | - | - | -
DC-GAN | 0.2 | 2.3 | 0.7 | 7.4 | 7.6
MS-GAN | 115.5 | 280,7 | 9.0 | 6.7 | 21.7
DivCo | 163.3 | 3.7 | 1.1 | 7.0 | 14.3
SDI-GAN (ours) | 3.3 | 127.4 | 1.2 | 2.1 | 4.5

We present the results of our experiments on the synthetic dataset using both
qualitative and quantitative comparisons. As shown in Fig. 3 our method
generates samples that are visually most similar to the real data. To confirm
this observation we calculate the mean variance for all classes for points
with spread = 0 and spread = 1. The results in Tab. 1 show the superiority of
our method in this scenario. DC-GAN is able to produce results with variance
close to real data for conditional inputs with spread = 0, but fails to
generate diverse results. MS-GAN properly reflect the diversity of possible
results for conditional inputs with spread = 1, but does not generate
consistent results. Although samples created by DivCo have different variance
depending on the conditioning input, the data distribution generated under
condition spread = 1 is distorted and does not match the real data, as
presented in Fig. 3. Our approach is able to produce both diverse and similar
results depending on the conditional information.

| real data |  SDI-GAN (ours) | DC-GAN | MS-GAN | DivCo
---|---|---|---|---|---
spread= 1 |  |  |  |  |
spread = 0 |  |  |  |  |
Figure 3: Comparison of generated results for the synthetic dataset. Contrary
to competing methods, our approach is able to properly synthesise both diverse
and similar samples depending on the conditioning variable spread.

The most common method for evaluating GANs on real datasets utilizes Frechet
Inception Distance (FID) [5]. However, for the HEP dataset, we propose a
domain-specific evaluation scheme that better measures the quality of the
simulation. Following the calorimeter’s specification [2] we base our
evaluation procedure on 5 channels calculated from the pixels of generated
images. These channels reflect the physical properties of simulated collision
and are used for analysing the output of the calorimeter. To measure the
quality of the simulation we compare the distribution of channels for the
original and generated data using Wasserstein distance [18].

| diverse results | consistent results
---|---|---
real data |  |
DC-GAN |  |
MS-GAN |  |
DivCo |  |
SDI-GAN (ours) |  |
Figure 4:  Examples of calorimeters response simulations with different
methods. DC-GAN works well for particles with consistent responses but fails
to generate diverse outcomes when needed. Although MS-GAN and DivCo
successfully increase the diversity of generated samples those models do not
distinguish between particles that should produce diverse or consistent
showers. Our method is able to generate diverse results while producing
consistent responses for appropriate particles.

As presented in Tab. 1, our approach outperforms other solutions on the HEP
datasets. In Fig. 4 we demonstrate that our method is able to generate diverse
results for a specific subset of particles while keeping consistent responses
for the remaining conditional inputs. The positive impact of this approach on
the distribution of the generated samples is further confirmed by Fig. 5 where
we compare channel distribution for SDI-GAN and competing approaches for 2
selected channels. Our method increases the fidelity of the simulation by
smoothing the distribution of generated responses and covering the whole range
of possible outputs.

| Channel 1 | Channel 2
---|---|---
DC-GAN |  |
MS-GAN |  |
DivCo |  |
SDI-GAN (ours) |  |
Figure 5:  Comparison of channel values distribution for selected channels.
Our method decreases the differences between the distribution of original and
generated data and smooths the distribution of the synthesised results. In the
case of SDI-GAN the increased diversity of generated samples does not harm the
fidelity of the simulation, contrary to competing approaches.

The additional regularization term for training of SDI-GAN does not influence
the inference speed of the model. In our initial experiments, we observe a
speed-up of simulations of two orders of magnitude when compared to the
standard Monte-Carlo approach. With SDI-GAN this computation boost is observed
without degradation in simulation quality. We leave the detailed analysis of
this performance gain and the influence of fast simulations on physical
experiments for future work.

##  5 Conclusions

In this work we introduce a simple, yet effective modification of the loss
function for conditional generative adversarial networks. Our solution
enforces increased sample diversity for a subset of conditional data without
affecting samples that are characterised by conditional values associated with
consistent responses.

We show that our solution outperforms other comparable approaches on the
synthetic benchmark and the challenging practical dataset of calorimeter
response simulations in the ALICE experiment at CERN.

## References

  * [1] Deja, K., Dubiński, J., Nowak, P., Wenzel, S., Spurek, P., Trzcinski, T.: End-to-end sinkhorn autoencoder with noise generator. IEEE Access 9, 7211–7219 (2021). https://doi.org/10.1109/ACCESS.2020.3048622
  * [2] Dellacasa, G., Zhu, X., Wahn, M., Staley, F., Danielian, V., Karavicheva, T., Mikhalev, D., Carrer, N., Gheata, M., Stefanek, G., et al.: Alice technical design report of the zero degree calorimeter (zdc). Tech. rep., ALICE (1999)
  * [3] Erdmann, M., Glombitza, J., Quast, T.: Precise simulation of electromagnetic calorimeter showers using a wasserstein generative adversarial network. Computing and Software for Big Science 3(1) (Jan 2019)
  * [4] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
  * [5] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems. pp. 6626–6637 (2017)
  * [6] Incerti, S., Kyriakou, I., Bernal, M., Bordage, M., Francis, Z., Guatelli, S., Ivanchenko, V., Karamitros, M., Lampe, N., Lee, S.B., et al.: Geant4-dna example applications for track structure simulations in liquid water: A report from the geant4-dna project. Medical physics 45(8), e722–e739 (2018)
  * [7] Kansal, R., Duarte, J., Su, H., Orzari, B., Tomei, T., Pierini, M., Touranakou, M., Vlimant, J.R., Gunopulos, D.: Particle Cloud Generation with Message Passing Generative Adversarial Networks. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2021)
  * [8] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. CoRR abs/1312.6114 (2013)
  * [9] Liu, R., Ge, Y., Choi, C.L., Wang, X., Li, H.: Divco: Diverse conditional image synthesis via contrastive generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
  * [10] Mao, Q., Lee, H.Y., Tseng, H.Y., Ma, S., Yang, M.H.: Mode seeking generative adversarial networks for diverse image synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)
  * [11] Mirza, M., Osindero, S.: Conditional generative adversarial nets (2014), https://arxiv.org/abs/1411.1784
  * [12] Motamed, S., Khalvati, F.: Inception augmentation generative adversarial network. CoRR abs/2006.03622 (2020), https://arxiv.org/abs/2006.03622
  * [13] Paganini, M., de Oliveira, L., Nachman, B.: Calogan: Simulating 3d high energy particle showers in multi-layer electromagnetic calorimeters with generative adversarial networks. CoRR abs/1705.02355 (2017)
  * [14] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
  * [15] Rodríguez, A.C., Kacprzak, T., Lucchi, A., Amara, A., Sgier, R., Fluri, J., Hofmann, T., Réfrégier, A.: Fast cosmic web simulations with generative adversarial networks. Computational Astrophysics and Cosmology 5(1) (Nov 2018), http://dx.doi.org/10.1186/s40668-018-0026-4
  * [16] Rong, R., Jiang, S., Xu, L., Xiao, G., Xie, Y., Liu, D.J., Li, Q., Zhan, X.: MB-GAN: Microbiome Simulation via Generative Adversarial Network. GigaScience 10(2) (02 2021), https://doi.org/10.1093/gigascience/giab005
  * [17] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., Chen, X.: Improved techniques for training gans. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 29. Curran Associates, Inc. (2016), https://proceedings.neurips.cc/paper/2016/file/8a3363abe792db2d8761d6403605aeb7-Paper.pdf
  * [18] Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558 (2017)
  * [19] Yang, D., Hong, S., Jang, Y., Zhao, T., Lee, H.: Diversity-sensitive conditional generative adversarial networks. In: Proceedings of the International Conference on Learning Representations (2019)
  * [20] Zhao, S., Cui, J., Sheng, Y., Dong, Y., Liang, X., Chang, E.I., Xu, Y.: Large scale image completion via co-modulated generative adversarial networks. CoRR abs/2103.10428 (2021), https://arxiv.org/abs/2103.10428
  * [21] Zieba, M., Semberecki, P., El-Gaaly, T., T. Trzciński: Bingan: Learning compact binary descriptors with a regularized gan. NeurIPS (2018)