# Seq-Masks: Bridging the gap between appearance and gait modeling for video-
based person re-identification ††thanks: Funded by NSFC: 62071292, 61771303
and STCSM 18DZ2270700.

Zhigang Chang1, Zhao Yang1, Yongbiao Chen2, Qin Zhou3, Shibao Zheng1
1Institute of Image Processing and Network Engineering, Shanghai Jiao Tong
University, Shanghai, China  3School of Biomedical Engineering, Shanghai Jiao
Tong University, Shanghai, China  2Shanghai Key Laboratory of Scalable
Computing and Systems, Shanghai Jiao Tong University, Shanghai, China

###### Abstract

Video-based person re-identification (Re-ID) aims to match person images in
video sequences captured by disjoint surveillance cameras. Traditional video-
based person Re-ID methods focus on exploring appearance information, thus,
vulnerable against illumination changes, scene noises, camera parameters, and
especially clothes/carrying variations. Gait recognition provides an implicit
biometric solution to alleviate the above headache. Nonetheless, it
experiences severe performance degeneration as camera view varies. In an
attempt to address these problems, in this paper, we propose a framework that
utilizes the sequence masks (SeqMasks) in the video to integrate appearance
information and gait modeling in a close fashion. Specifically, to
sufficiently validate the effectiveness of our method, we build a novel
dataset named MaskMARS based on MARS. Comprehensive experiments on our
proposed large wild video Re-ID dataset MaskMARS evidenced our extraordinary
performance and generalization capability. Validations on the gait recognition
metric CASIA-B dataset further demonstrated the capability of our hybrid
model. Our codes and dataset MaskMARS will be open-sourced as a strong
baseline.

###### Index Terms:

Multi-modal Fusion, Video-based Person Re-ID, Gait Recognition, appearance
model

##  I Introduction

Video-based Person re-identification (Re-ID) [1, 2, 3, 4, 5, 6] has been
drawing proliferating attention from researchers worldwide because of its
ubiquitous presence in diversified scenarios ranging from cross-camera object
tracking, pedestrian behavior analysis, video surveillance to criminal
investigations. Traditional RGB-based, Video-based Person Re-ID methods
seeking to build a discriminative appearance model has gained growing momentum
in recent years. But the prerequisite is harsh and fatal. It is vulnerable
against illumination changes, camera parameters, and scene clutter; the time-
effectiveness of the model is short due to clothing change; uniforms and
camouflage can easily paralyze the system. Consequently, it is rather
reasonable to ponder the possibility of investigating auxiliary information or
characteristics.

Gait [7, 8, 9] which manifests the walking style of pedestrians comes to
rescue, given its unique advantages ranging from being insensitive to
resolution variation to being extremely challenging to impersonate, thus,
highlighting the necessity of employing gait feature into performing person
re-id tasks [8]. Some gait recognition methods [10] aim to extract implicit
features from videos based on contour or articulated body representations(
e.g., UV maps and 2d keypoints), where semantic segmentation (Mask-RCNN [11])
and human pose estimation (DensePose [12] and OpenPose [13]) can be utilized
to model the discriminative representations. However, this stream of works is
limited by pedestrian speed, camera perspective, video frame rate, and other
factors, leading to low gait recognition performance, especially in practical
scenarios. Since video-based person Re-ID methods regard re-identification as
sequence matching, which is also challenging due to the random camera view
angles and wild scenarios. The goals and settings of these two branches have
strong consistency, which naturally leads us to integrate them for better Re-
ID performance in wild scenarios.

Figure 1: Motivation

Figure 2: Detailed structure of our Framework

In traditional video-based Re-ID, 3D convolution, RNN models, and temporal
pooling are commonly used to build an embedding for a video sequence, which
integrates multiple information like appearance, motions (gait and other
motions), optical flow, etc,. These representations are tightly bound and hard
to disentangle. This representation lacks interpretability and may be hard to
learn. Our motivation is somehow trying to locally disentangle the gait
representation manually by adding a gait branch to a video Re-ID framework. As
shown in Fig. 1, the enhanced gait representation serves as strong prior to
joining the final embedding.

In this paper, we utilize the foreground masks of the video sequence to bridge
the appearance branch and the gait branch. For the gait branch, we adopt a
variant of an advanced gait recognition network Gaitset [10] where the
foreground sequence masks serve as the inputs of the gait branch. For the
appearance branch, the foreground sequence masks can be regarded as the
saliency map to highlight the foreground human semantic and eliminate
background interference. Meanwhile, we also keep the original global branch to
reserve global information of the sequence. Finally, we concatenate all the
features after each branch and integrate them into a fused representation.

In summary, our contributions are three-fold: First, we proposed a novel end-
to-end video person id framework which exploits both appearance and gait
information. Second, we built a novel large-scale video Re-id dataset named
Mask-MASRS. Third, extensive experiments have demonstrated the validity and
effectiveness of our proposed model on both Mask-MARKS and CASIA-B datasets.

##  II Method

The overall network architecture is shown in Fig 2. The network contains 3
modules in total: Appearance Module, Gait Module, and Feature Fusion Module.
The appearance module is comprised of the global branch and the foreground
branch. For each input video sequence (each sequence contains T frames), we
use foreground extraction methods (such as Mask RCNN) to extract the
pedestrian foreground sequence masks.

In Appearance Module global branch, the backbone neural network extracts the
feature maps of each frame of the sequence; then the global average pooling
and temporal average pooling operations flatten and aggregate these feature
maps of one sequence to a 2048 dimensions vector, after which a designed
bottleneck block reduces the vector to 512 dimensions. As for the foreground
branch, the output feature maps of the global branch backbone network are re-
used and conducted dot multiplication with the resized foreground masks, which
solely average pool the foreground region of the heatmaps and represent the
feature of foreground (also 2048 dimensions). Another bottleneck block reduces
the foreground vector to 512 dimensions in the same way.

Meanwhile, Gait Module randomly samples K frames of foreground sequence masks
as the inputs of the module. After the Gait Module (a variant of Gaitset
[10]), we obtain a gait feature (a 512-dimension vector) of each sequence
masks.

Finally, the global appearance feature, the foreground appearance feature, and
gait feature are concatenated to a 1536-dimension feature vector. We use the
Feature Fusion Module to get the fusion feature vector (1536 dimensions).
Finally, the fusion feature vector is used for the final representation of the
video. We will further introduce the details of each module in the following
subsections.

###  II-A Backbone network and bottleneck structure

We adopt ResNet-50 [14] as the backbone network. By removing the final fully
connected layer and changing the last stride from 2 to 1 in $res\\_conv5$, we
can obtain a larger feature map incorporating richer feature information. For
a typical image input size of 256 $\times$ 128, the output of the modified
ResNet-50 is 16 $\times$ 8.

Both the global branch and the foreground branch adopt the bottleneck
structure. The structure of the bottleneck is depicted in Fig. 2 (3), which is
comprised of two fully connected layers, each followed by a normalization
layer and a ReLU function. Compared to a fully connected layer (2048 $\times$
512), the bottleneck structure results in notable network parameter reduction.

###  II-B Gait Branch

We adopt and modify an advanced state-of-the-art gait recognition method
GaitSet to obtain the gait feature. Different from other template-based and
sequence-based methods, GaitSet treats the input as a set of disordered
pedestrian contour images. Since pedestrian contours at different time flames
exhibit different shapes intuitively, even if the contour sequence is re-
shuffled, they can be rearranged into correct orders according to the shapes.
The set-based GaitSet method inherently has a processing requirement for sets:
sort independence, where the result has nothing to do with the order of input
contours.

Compared with the original GaitSet, the modified GaitSet network reduces the
number of horizontal strips at the end of the network, achieving the purpose
of reducing the dimension of feature, and the performance is slightly reduced.
The modified GaitSet architecture is shown in Fig. 3. The architecture
settings before Global Pooling are the same as the original network (also
shown detailly in fig. 2). The Set Pooling (SP) module is used to integrate
each frame-level feature to form a set-level feature. As we know, a deeper
convolutional layer has a larger receptive field. Shallow feature maps contain
more local and fine-grained information, while deep feature maps contain
global and coarse-grained information. Therefore, MGP(Multilayer Global
Pipeline) is designed for aggregating the set-level features from different
layers’ outputs. The MGP structure is consistent with the main branch, with
the same convolutional layer and pooling layer design, but the parameters are
not shared with the main branch.

We obtain two 128-dimension vectors from the main branch and MGP after global
pooling. After two independent fully connected layers, vectors are mapped to
256 dimensions. During the training phase, these two feature vectors are
associated with two separate training losses. In the inference stage, they
will be concatenated into a 512-dimensional feature vector.

Set Pooling is designed as Fig. 3. It uses 3 kinds of statistical operations
that are independent of order: $max()$, $mean()$ and $median()$. $1\\_1C$
represents a 1$\times$1 convolutional layer, which is used to fuse cascaded
statistical features. Attention mechanism and residual structure are also
designed for aggregation and maintaining the information capacity while
accelerating and stabilizing convergence. Global Pooling computes the sum of
Global Average Pooling and Global Max Pooling.

Figure 3: The design of the Gaitset

###  II-C Feature Fusion Module (FFM)

We refer to the channel attention mechanism designed in SeNet to design a
feature fusion network, shown as Fig. 2. The global appearance feature, the
foreground appearance feature, and the gait feature are concatenated to a
1536-dimension vector. Then we use the Bottleneck mechanism (ratio=8) to
construct a channel attention mechanism to promote the exchange of information
between different features. At the same time, the residual structure is used
to accelerate and stabilize convergence. Finally, a 1526-dimensional fusion
feature vector is obtained.

###  II-D Loss Functions

During training phase, batch-all triplet [15], batch-hard triplet loss [15]
and SoftMax loss with Label-Smoothing Regularization (LSR) [16] are employed
to train the network. The loss of the Appearance Module ($L_{appearance}$) is
calculated as shown in Eq. 1; the loss of the Gait Module ($L_{gait}$) and the
Feature Fusion Module ($L_{fusion}$) are shown as Eq. 2.

$\begin{split}L_{appearance}&=L_{batch\ hard\
triplet}^{global}+L_{softmax}^{global}+L_{batch\ hard\
triplet}^{foreground}\\\ &+L_{softmax}^{foreground}\end{split}\vspace{-0.3cm}$
(1) $\vspace{-0.3cm}\begin{split}L_{gait}=L_{batch\ all\
triplet}^{gait}+L_{softmax}^{gait};\ L_{fusion}=L_{batch\ hard\
triplet}^{fusion}\end{split}$ (2)

And the total loss is shown as as Eq. 3.

$\begin{split}L_{total}=\lambda_{1}L_{fusion}+\lambda_{2}L_{appearance}+\lambda_{3}L_{gait}\end{split}$
(3)

##  III Datasets and Data preprocessing

In this paper, both sequence images and foreground sequence masks are needed
as inputs for experiments. To obtain data in wild scenarios, we build a
dataset named Mask-MARS based on the MARS dataset [3], a large-scale dataset
for video-based person Re-ID. We also conduct experiments on the CASIA-B
dataset [7].

  * •

Mask-MARS: We create the original Mask-MARS dataset by computing the
foreground mask of each RGB image in MARS dataset by a strong instance
segmentation method. We require a video sequence to contain at least 8
effective foreground masks (not necessarily continuous), where an effective
mask is simply defined as the proportion of the foreground area to the
original image is not less than 15%. After screening by these two simple
rules, the Mask-MARS data set contains a total of 1250 IDs and 14764 video
sequences. The training set contains 624 IDs and 5726 video sequences; the
query set contains 626 IDs and 1819 video sequences; the gallery set contains
621 IDs and 7170 video sequences. The length of the video sequence varies from
8 to 920 frames, with an average of 70 frames.

  * •

CASIA-B: is a popular gait dataset. CASIA-B [7] has a total of 124 IDs,
containing 11 angles (0, 18, 36,…, 180 degrees) and 3 different walking
conditions. Walking conditions include normal (NM), each pedestrian contains 6
sequences; carrying bag (BG), each pedestrian contains 2 sequences; wearing a
jacket or jacket (CL), each pedestrian contains 2 sequences. so each
pedestrian contains 110 sequences. We use the first 74 IDs as the training set
and the last 50 IDs as the test set. In the test environment, the first 4
sequences (NM1-4) under NM conditions are retained in the gallery subset, and
the remaining 6 sequences are divided into 3 query subsets (NM5-6, BG1-2, and
CL1-2).

We adopt the method in [9] to preprocess the foreground mask to achieve
alignment. During the experiment, the size of the aligned mask image is set to
64$\times$64\. We crop 10 pixels on both the left and right sides of the
horizontal direction to obtain the size of 64$\times$44 as the input to the
GaitSet network.

In the training phase, we preprocess the color image sequence and its
corresponding masks in the following way: (1) Random sequence crop: We set the
size of the cropped color image to 256$\times$128, and the corresponding mask
size of the output feature map of the last layer of ResNet-50 is 16$\times$8,
and the random probability $p=0.5$. (2) Random sequence flip: we set random
probability $p=0.5$. (3) Image standardization: we use the mean and variance
statistics of the 3 channels of the ImageNet dataset to normalize the image.

##  IV Experiments

###  IV-A Progressive results and analysis

For simplicity and clarity, we name the baseline along with other module
combines as follows:

Appearance Baseline: the global branch of Appearance Module, training
individually, 512-dimension feature (without FFM). Appearance Baseline +
Foreground Branch: whole Appearance Module, joint training, 1024-dimension
feature (without FFM). Modified GaitSet: whole Gait Module, training
individually, 512-dimension feature (without FFM). Fusion Network: our full
system, joint end-to-end training, 1536-dimension feature (with FFM), with two
version: each pre-trained parts finetuning version and end-to-end trained
version (end2end) .

As shown in Tab. I, in Mask-MARS, compared with the baseline model, the Rank1
index of the model with the foreground branch increased from 84.7% to 86.5%,
and the mAP increased from 78.9% to 80.7%. Because MARS is a large-scale wild
scene dataset with random camera views and large background clutters. Only use
the gait branch, the performance is very poor (MAP 10.7% and rank1 17.7%).
Even if the gait model does not perform well, our Fusion Network still
outperforms Baseline + Foreground Branch with a notable margin. The two
indicators of Rank1 and mAP have increased by 0.8% and 3.0% respectively.
Compared to the Baseline, the improvement is even more significant (Rank1
+2.6% and mAP +4.8%).

In the CASIA-B dataset, the Tab. II records the experimental results of
including and excluding the same view sequence as the query in the gallery.
Adding appearance features makes the model more robust to view angle
variation. Since the dataset is quite simple, there is basically no background
clutter. Therefore, we can see that in the case of NM and BG, the appearance-
based model can perform very well. However, when clothes changed (CL case),
the performance of our Fusion Network (end2end) dominates all other models,
achieving 72.891% rank1 accuracy, far exceeding the Modified GaitSet (+19.86%)
and Appearance Baseline (+29.49%). This reflects that when the appearance of
pedestrians changes significantly, the fusion features are often more
discriminative than single-modal features. In any case in the CASIA-B dataset
(NM, BG, CL), we achieve the best performance compared to a single-modal
model.

TABLE I: Progressive results on Mask-MARS dataset

Model | Rank1 | Rank5 | Rank10 | Rank20 | MAP
---|---|---|---|---|---
Appearance Baseline | 84.7 | 94.6 | 95.9 | 97.2 | 78.9
Appearance Baseline + Foreground Branch | 86.5 | 95.5 | 96.3 | 97.1 | 80.7
Modified GaitSet | 17.7 | 34.1 | 43.1 | 51.8 | 10.7
FusionNetwork(finetune) | 87.1 | 95.2 | 96.3 | 97.2 | 81.1
FusionNetwork(end2end) | 87.3 | 95.6 | 96.5 | 97.9 | 83.7

TABLE II: Comparison of concatenation and fusion effects of different features

Model | Including the same angle | Excluding the same angle
---|---|---
NM | BG | CL | NM | BG | CL
AppearanceBaseline | 98.669 | 96.630 | 45.884 | 98.536 | 96.475 | 45.400
AppearanceBaseline+Foreground Branch | 98.405 | 95.960 | 48.843 | 98.245 | 95.729 | 48.364
Modified GaitSet | 83.570 | 71.284 | 55.934 | 81.964 | 69.168 | 55.036
FusionNetwork(finetune) | 99.455 | 96.782 | 73.634 | 99.437 | 96.579 | 69.495
FusionNetwork(end2end) | 99.620 | 96.822 | 75.950 | 99.582 | 96.623 | 74.891

###  IV-B Comparison of concatenation and fusion effects of different features

This experiment is to verify the effectiveness of the feature fusion module
(FFM). We name models with different setting as follows: GGConcat:
concatenated feature from 2 branches: global branch and gait branch; GGFusion:
fusion feature from 2 branches with FFM; AGConcat: concatenated feature from 3
branches: global branch, foreground branch and gait branch without FFM;
AGFusion: fusion feature from 3 branches with FFM. The experimental results on
the Mask-MARS and CASIA-B datasets are shown in Tab. III.

###  IV-C Compared with other advanced video-based ReID methods

We also reproduce some recent advanced algorithms [2, 17, 4] to validate the
effectiveness of our hubrid model on the dataset we created in Tab. IV. The
performance of our method is far superior compared to methods that model only
the appearance features.

TABLE III: Comparison of concatenation and fusion effects of different
features

Model | Mask-MARS | CASIA-B
---|---|---
rank1 | rank5 | rank10 | mAP | NM | BG | CL
AGConcat | 86.6 | 95.2 | 96.4 | 81.0 | 99.736 | 97.905 | 68.785
AGFusion | 87.1 | 95.2 | 96.3 | 81.1 | 99.620 | 96.822 | 75.950
GGConcat | 87.0 | 95.4 | 96.5 | 81.0 | 99.810 | 97.806 | 75.099
GGFusion | 87.3 | 95.3 | 96.2 | 80.7 | 99.149 | 96.740 | 75.992

TABLE IV: Comparison with the state-of-the-art video-based Re-ID methods on MASK-MARS dataset.  Model | Rank1 | Rank5 | Rank10 | Rank20 | MAP
---|---|---|---|---|---
Non-local+C3D [2] | 84.1 | 94.5 | 96.0 | 97.3 | 77.2
STAN [17] | 82.3 | 92.9 | 94.6 | 96.8 | 65.7
Snipped [4] | 81.2 | 92.1 | 94.6 | 96.5 | 69.4
Snipped+$OF$ [4] | 86.3 | 94.7 | 95.7 | 98.2 | 76.1
Ours (end2end) | 87.3 | 95.6 | 96.5 | 97.9 | 83.7

##  V Conclusion

This paper propose an end-to-end framework which utilizes the sequence masks
(SeqMasks) in each video to jointly exploit the power of appearance and gait
in video Re-ID. Experiments on Mask-MARS dataset evidence the favorable
performance and generalization ability of the proposed algorithm. Further
validations on gait recognition metric CASIA-B dataset highlight the
performance of our hybrid model.

## References

  * [1] Jiyang Gao and Ram Nevatia,  “Revisiting temporal modeling for video-based person reid,”  arXiv preprint arXiv:1805.02104, 2018.
  * [2] Xingyu Liao, Lingxiao He, Zhouwang Yang, and Chi Zhang,  “Video-based person re-identification via 3d convolutional networks and non-local attention,”  in Asian Conference on Computer Vision. Springer, 2018, pp. 620–634.
  * [3] Liang Zheng, Zhi Bie, Yifan Sun, Jingdong Wang, Chi Su, Shengjin Wang, and Qi Tian,  “Mars: A video benchmark for large-scale person re-identification,”  in European Conference on Computer Vision. Springer, 2016, pp. 868–884.
  * [4] Dapeng Chen, Hongsheng Li, Tong Xiao, Shuai Yi, and Xiaogang Wang,  “Video person re-identification with competitive snippet-similarity aggregation and co-attentive snippet embedding,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1169–1178.
  * [5] Zhigang Chang, Zhou Qin, Heng Fan, Hang Su, Hua Yang, Shibao Zheng, and Haibin Ling,  “Weighted bilinear coding over salient body parts for person re-identification,”  Neurocomputing, vol. 407, pp. 454–464, 2020.
  * [6] Zhao Yang, Zhigang Chang, and Shibao Zheng,  “Large-scale video-based person re-identification via non-local attention and feature erasing,”  in International Forum on Digital TV and Wireless Multimedia Communications. Springer, 2019, pp. 327–339.
  * [7] Shiqi Yu, Daoliang Tan, and Tieniu Tan,  “A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition,”  in 18th International Conference on Pattern Recognition (ICPR 2006), 20-24 August 2006, Hong Kong, China, 2006.
  * [8] Athira Nambiar, Alexandre Bernardino, and Jacinto C Nascimento,  “Gait-based person re-identification: A survey,”  ACM Computing Surveys (CSUR), vol. 52, no. 2, pp. 1–34, 2019.
  * [9] Noriko Takemura, Yasushi Makihara, Daigo Muramatsu, Tomio Echigo, and Yasushi Yagi,  “Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition,”  Ipsj Transactions on Computer Vision and Applications, vol. 10, no. 1, pp. 4, 2018.
  * [10] Hanqing Chao, Yiwei He, Junping Zhang, and Jianfeng Feng,  “Gaitset: Regarding gait as a set for cross-view gait recognition,”  2018\.
  * [11] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick,  “Mask r-cnn,”  in Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [12] Rza Alp Güler, Natalia Neverova, and Iasonas Kokkinos,  “Densepose: Dense human pose estimation in the wild,”  2018\.
  * [13] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh,  “Openpose: Realtime multi-person 2d pose estimation using part affinity fields,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [15] Alexander Hermans, Lucas Beyer, and Bastian Leibe,  “In defense of the triplet loss for person re-identification,”  arXiv preprint arXiv:1703.07737, 2017.
  * [16] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna,  “Rethinking the inception architecture for computer vision,”  2015\.
  * [17] Shuang Li, Slawomir Bak, Peter Carr, and Xiaogang Wang,  “Diversity regularized spatiotemporal attention for video-based person re-identification,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 369–378.


# Invertible mapping between fields in CAMELS

Sambatra Andrianomena Sultan Hassan 22footnotemark: 2 44footnotemark: 4
Francisco Villaescusa-Navarro South African Radio Astronomy Observatory, Cape
Town 7925<EMAIL_ADDRESS>of Physics & Astronomy University
of the Western Cape, Cape Town 7535.Center for Cosmology and Particle Physics,
Department of physics, New York University, 726 Broadway, New York, NY, 10003,
<EMAIL_ADDRESS>for Computational Astrophysics, Flatiron
Institute, New York, NY 10010<EMAIL_ADDRESS>
###### Abstract

We build a bijective mapping between different physical fields from
hydrodynamic CAMELS simulations. We train a CycleGAN on three different
setups: translating dark matter to neutral hydrogen (Mcdm-HI), mapping between
dark matter and magnetic fields magnitude (Mcdm-B), and finally predicting
magnetic fields magnitude from neutral hydrogen (HI-B). We assess the
performance of the models using various summary statistics, such as the
probability distribution function (PDF) of the pixel values and 2D power
spectrum ($P(k)$). Results suggest that in all setups, the model is capable of
predicting the target field from the source field and vice versa, and the
predicted maps exhibit statistical properties which are consistent with those
of the target maps. This is indicated by the fact that the mean and standard
deviation of the PDF of maps from the test set is in good agreement with those
of the generated maps. The mean and variance of $P(k)$ of the real maps agree
well with those of generated ones. The consistency tests on the model suggest
that the source field can be recovered reasonably well by a forward mapping
(source to target) followed by a backward mapping (target to source). This is
demonstrated by the agreement between the statistical properties of the source
images and those of the recovered ones.

##  1 Introduction

The upcoming generation of surveys (e.g. SKA) will be able to map neutral
hydrogen via HI intensity mapping (Santos et al., 2015). This powerful
cosmological probe will help us further our understanding of large-scale
structure. To extract the relevant information about the matter field from
these surveys, we usually resort to summary statistic, such as power spectrum.
This is challenging in the non-linear regime due to the contamination of the
signal by the baryonic physics and hence requires higher order statistics. The
other approach is to carry out the analysis at the field level, e.g. inference
or direct mapping between fields. Previous studies demonstrated the
feasibility of building a mapping at the field level between baryons and dark
matter (Wadekar et al., 2021; Villanueva-Domingo & Villaescusa-Navarro, 2021;
Bernardini et al., 2022). In light of those existing works, and by utilizing
generative adversarial networks (CycleGAN), we aim at building a bijective map
between dark matter and two observables, namely neutral hydrogen and magnetic
field magnitude. This is crucial since with a single training, it is possible
to either paint the dark matter from simulation with baryons or directly infer
its distribution from maps of observables obtained from different surveys in
the near future.

##  2 Methods

###  2.1 Data

In this study, we use the publicly available CAMELS Multifields Dataset (CMD)
(Villaescusa-Navarro et al., 2022) which contains thousands of 2D field maps
generated from state-of-the-art hydrodynamics simulations(Villaescusa-Navarro
et al., 2021). We consider $256\times 256$ pixels 2D maps, which cover an area
of $25\times 25~{}(h^{-1}{\rm Mpc})^{2}$ at $z=0$, of dark matter (Mcdm),
neutral hydrogen (HI) and magnetic fields magnitude (B) from the IllustrisTNG
LH set. The choice of the fields in this simple scenario is based on the aim
of inferring the matter distribution from observables. In total, each field
corresponds to 15000 2D images, each characterized by a set of 6 parameters:
matter density ($\Omega_{8}$), the amplitude of matter power spectrum
($\sigma_{8}$), the stellar feedbacks ($A_{\rm SN1}$, $A_{\rm SN1}$) and AGN
feedbacks ($A_{\rm AGN1}$, $A_{\rm AGN2}$).

###  2.2 Model and training

To build an invertible mapping between two different fields, we make use of
CycleGAN (Zhu et al., 2017), an improvement on “pix2pix” method (Isola et al.,
2017) which is trained on paired examples to achieve image-to-image
translation. The approach consists of building a function
$\mathcal{G}:X\rightarrow Y$ that maps a source field $X$ to a target field
$Y$ (forward mapping), simultaneously with another function
$\mathcal{F}:Y\rightarrow X$ that translates the target to the source field.
To this end, two generators $G_{X}$ (representing $\mathcal{F}$ and producing
the source field images) and $G_{Y}$ (representing $\mathcal{G}$ and producing
the target field images) are trained with two adversarial discriminators
$D_{X}$ and $D_{Y}$ respectively. Following the prescription in Zhu et al.
(2017), there are two main components to the loss function for the training.
The adversarial loss – for each of the pair ($G_{X},D_{X}$) and
($G_{Y},D_{Y}$) – is given by (Zhu et al., 2017)

$\mathcal{L}_{\rm GAN}(G_{X},D_{X},Y,X)=\mathbb{E}_{x\sim p_{\rm
data}(x)}[{\rm log}D_{X}(x)]+\mathbb{E}_{y\sim p_{\rm data}(y)}[{\rm
log}(1-D_{X}(G_{X}(y)))]$ (1)

and

$\mathcal{L}_{\rm GAN}(G_{Y},D_{Y},X,Y)=\mathbb{E}_{y\sim p_{\rm
data}(y)}[{\rm log}D_{Y}(y)]+\mathbb{E}_{x\sim p_{\rm data}(x)}[{\rm
log}(1-D_{Y}(G_{Y}(x)))]$ (2)

where $x\sim p_{\rm data}(x)$ and $y\sim p_{\rm data}(y)$ are the data
distributions of the source images ($x\in X$) and target images ($y\in Y$)
respectively. The consistency loss ensures that the functions $\mathcal{G}$
and $\mathcal{F}$ are inverse of each other such that $G_{X}(G_{Y}(x))\approx
x$ and $G_{Y}(G_{X}(y))\approx y$. In other words, an input $x$ (or $y$) is
recovered by applying $G_{X}$ on $G_{Y}(x)$ (or applying $G_{Y}$ on
$G_{X}(y)$).

Figure 1: The top, middle, and bottom rows show the results from mapping HI-B,
Mcdm-HI, and Mcdm-B respectively. It is worth noting that in a X-Y setting, X
and Y designate the source and target fields, respectively.

The consistency loss is given by (Zhu et al., 2017)

$\mathcal{L}_{\rm cycle}=\mathbb{E}_{x\sim p_{\rm
data}(x)}[||G_{X}(G_{Y}(x))-x||_{1}]+\mathbb{E}_{y\sim p_{\rm
data}(y)}[||G_{Y}(G_{X}(y))-y||_{1}],$ (3)

where $||.||_{1}$ denotes the mean absolute error (L1Loss). To further enforce
a unique prediction of a given input such that $G_{Y}(y)\approx y$ and
$G_{X}(x)\approx x$, an identity loss

$\mathcal{L}_{\rm id}=\mathbb{E}_{y\sim p_{\rm
data}(y)}[||G_{Y}(y)-y||_{1}]+\mathbb{E}_{x\sim p_{\rm
data}(x)}[||G_{X}(x)-x||_{1}]$ (4)

is used. The total loss is then given by

$\mathcal{L}_{\rm tot}=\mathcal{L}_{\rm GAN}(G_{X},D_{X},Y,X)+\mathcal{L}_{\rm
GAN}(G_{X},D_{X},Y,X)+\lambda_{\rm cycle}\mathcal{L}_{\rm cycle}+\lambda_{\rm
id}\mathcal{L}_{\rm id},$ (5)

where both $\lambda_{\rm cycle}$ and $\lambda_{\rm id}$ are constants that
characterize the contributions of the consistency cycle loss and identity loss
respectively. Based on the prescription in Zhu et al. (2017), we have that
$\lambda_{\rm cycle}=10$ and $\lambda_{\rm id}=5$ during training.
The generators $G_{Y}$ and $G_{X}$, which have the same architecture, comprise
a stage that downsamples the inputs (similar to encoding), 9 residual layers
mimicking a bottleneck in the variational encoder and finally, a stage that
upsamples the output from the bottleneck (similar to decoding). The
discriminators $D_{Y}$ and $D_{X}$, which are also identical, consist of
chaining up 5 convolutional layers where the first three downsample the input
by using stride = 2. To have a bit more control on the topology of the
generated maps, we condition the input of each component of the model
($G_{X}$, $D_{X}$, $G_{Y}$ and $D_{Y}$) on the underlying cosmology and
astrophysics, i.e the parameters $\Omega_{8}$, $\sigma_{8}$, $A_{\rm SN1}$,
$A_{\rm SN1}$, $A_{\rm AGN1}$ and $A_{\rm AGN2}$. The array of parameters of
shape $1\times 6$ is passed through a dense layer with 4096 units whose output
is reshaped to $64\times 64$, upsampled to $256\times 256$ via interpolation,
and finally concatenated along the channel to the input image. The model is
trained for 100 epochs with 12000 unpaired examples, i.e. shuffling the set of
source images such that they don’t match the target images, using Adam
optimizer with a learning rate of 0.0002. By setting the batch size to 1,
following Zhu et al. (2017), each epoch takes about 109 minutes on a NVIDIA
GeForce GTX 1080 Ti.

##  3 Results

We present in Figure 1 some predictions by the generators ($G_{Y}$ and
$G_{X}$) using input images from test set.

Figure 2: Comparing probability distribution function (PDF) of pixel values of
the simulated and generated maps. Results corresponding to each field are
shown in each column. Solid red and dashed blue lines denote PDFs of fake and
real maps, respectively. Whereas red and blue shaded areas correspond to the
standard deviations of the PDFs of fake and real maps, respectively. Each
column corresponds to PDF of fields in different setups. The top rows show the
comparison between the PDFs of predicted and true maps of a field in each
setup. The bottom rows show the consistency test.

Each column in Figure 1 corresponds to the input, target and prediction in a
given setup, e.g. HI-B. In the first three columns of each row, we show the
result from the forward mapping, i.e. the input is the source field X and the
output is the target field Y. The last three columns of each row show the
results related to the backward mapping, i.e. the input is the target field Y
which is translated to the source field X. Visually, the output of the map by
the generators are in good agreement with the inputs and the quality is
comparable to that of the data from IllustrisTNG, overall.

Figure 3: Comparing the resulting power spectra of the simulated and generated
maps. Results corresponding to each field are shown in each column. Solid red
and dashed blue lines denote the averaged power spectrum ($P(k)$) of fake and
real maps, respectively. Whereas red and blue shaded areas correspond to the
standard deviations of the $P(k)$ of fake and real maps, respectively. The
relative error between the two power spectra for each field in each setup is
indicated by the solid green line.

However it appears that predicting the magnetic field B from dark matter or
neutral hydrogen seems to be a bit more challenging, as evidenced by the more
noticeable difference in the map features between the ground truth and
prediction (see 2nd and 3rd columns of both top and bottom rows).
The first metric we use to assess how well the model performs is the
probability distribution function (PDF) of pixel intensities. The test set for
each setup comprises of 1000 images unseen by the model during training. We
then compute the mean $\mu_{\rm PDF}$ and standard deviation $\sigma_{\rm
PDF}$ of PDF of each field in the test set in each setup. In Figure 2, we
present the PDFs of each field in some setups in each column. The top rows
show the results from either forward or backward mappings, whereas the results
of the consistency test – assessing if an input is recovered by applying
forward and backward mappings sequentially – is presented at the bottom row.
Results suggest that the model is capable of predicting the source and target
fields in each setup using $G_{X}$ and $G_{Y}$ respectively, as evidenced by
both $\mu_{\rm PDF}$ and $\sigma_{\rm PDF}$ of the data in good agreement with
those of the generated maps of each field. We also find that in general, the
bijective mapping is achieved to a good accuracy, i.e. $G_{X}(G_{Y}(x))\approx
x$ and $G_{Y}(G_{X}(y))\approx y$, as demonstrated by the $\mu_{\rm PDF}$ and
$\sigma_{\rm PDF}$ of the data which agree well with those of the recovered
outputs. The relatively small discrepancy at the high end of the distributions
can be accounted for by the small number of pixels having those values in the
training data, i.e. overdense regions are rare. The other metric used in our
investigation is the auto-power spectrum $P(k)$. Similar to the results
presented in Figure 2, we show the $P(k)$ of the images produced by the
forward/backward mapping and the consistency test at the top and bottom rows
of Figure 3 respectively. Each panel shows the mean and standard deviation of
the $P(k)$ of both real (dashed blue) and fake maps (solid red), and the solid
green line at the bottom of each panel shows the relative difference
$\left(\frac{P_{\rm fake}(k)}{P_{\rm real}(k)}-1\right)$ between the two
$P(k)^{\prime}$s (fake and real maps). It is clear that the forward and
backward mappings are able to produce maps with clustering properties in good
agreement with those of the data. Moreover, the mean and variance of $P(k)$ of
the recovered maps from the consistency check agree reasonably well with those
of the maps from IllustrisTNG, regardless of the increasing difference on
larger scales $k>10\>h/{\rm Mpc}$. Our result for Mcdm-HI (see Figure 3 top
middle panel) is comparable to what was obtained by Wadekar et al. (2021)
where a standard UNet architecture was used to convert dark matter density to
HI maps.

##  4 Conclusion

We have made use of CycleGAN model to build a bijective model that can map
different physical fields from 2D maps created from the CAMELS state-of-the-
art hydrodynamic simulations. Results show that the predicted maps exhibit
statistical properties that agree well with those from the dataset used for
training. Moreover, the condition for bijective mapping is met, as
demonstrated by the consistency test. By applying the composition function
$\mathcal{F}\circ\mathcal{G}(x)$ (or $\mathcal{G}\circ\mathcal{F}(y)$), an
input map $x$ (or $y$) is recovered reasonably while the statistics being
preserved. This work represents a step forward towards establishing an
efficient direct mapping between different observables, and hence maximizing
the scientific return of future multi-wavelength surveys.

#### Acknowledgments

SA acknowledges financial support from the South African Radio Astronomy
Observatory (SARAO). FVN and SH acknowledge support provided by the Simons
Foundation. SH also acknowledges support for Program number HST-HF2-51507
provided by NASA through a grant from the Space Telescope Science Institute,
which is operated by the Association of Universities for Research in
Astronomy, incorporated, under NASA contract NAS5-26555. The CAMELS project is
supported by NSF grant AST 2108078.

## References

  * Bernardini et al. (2022) Mauro Bernardini, Robert Feldmann, Daniel Anglés-Alcázar, Mike Boylan-Kolchin, James Bullock, Lucio Mayer, and Joachim Stadel.  From ember to fire: predicting high resolution baryon fields from dark matter simulations with deep learning.  _Monthly Notices of the Royal Astronomical Society_ , 509(1):1323–1341, 2022.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1125–1134, 2017.
  * Santos et al. (2015) Mario G Santos, Philip Bull, David Alonso, Stefano Camera, Pedro G Ferreira, Gianni Bernardi, Roy Maartens, Matteo Viel, Francisco Villaescusa-Navarro, Filipe B Abdalla, et al.  Cosmology with a ska hi intensity mapping survey.  _arXiv:1501.03989_ , 2015.
  * Villaescusa-Navarro et al. (2021) Francisco Villaescusa-Navarro, Daniel Anglés-Alcázar, Shy Genel, David N Spergel, Rachel S Somerville, Romeel Dave, Annalisa Pillepich, Lars Hernquist, Dylan Nelson, Paul Torrey, et al.  The camels project: Cosmology and astrophysics with machine-learning simulations.  _The Astrophysical Journal_ , 915(1):71, 2021\.
  * Villaescusa-Navarro et al. (2022) Francisco Villaescusa-Navarro, Shy Genel, Daniel Angles-Alcazar, Leander Thiele, Romeel Dave, Desika Narayanan, Andrina Nicola, Yin Li, Pablo Villanueva-Domingo, Benjamin Wandelt, et al.  The camels multifield data set: Learning the universe’s fundamental parameters with artificial intelligence.  _The Astrophysical Journal Supplement Series_ , 259(2):61, 2022.
  * Villanueva-Domingo & Villaescusa-Navarro (2021) Pablo Villanueva-Domingo and Francisco Villaescusa-Navarro.  Removing astrophysics in 21 cm maps with neural networks.  _The Astrophysical Journal_ , 907(1):44, 2021\.
  * Wadekar et al. (2021) Digvijay Wadekar, Francisco Villaescusa-Navarro, Shirley Ho, and Laurence Perreault-Levasseur.  Hinet: Generating neutral hydrogen from dark matter with neural networks.  _The Astrophysical Journal_ , 916(1):42, 2021\.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2223–2232, 2017.