Yu-Cheng<EMAIL_ADDRESS>Cheng<EMAIL_ADDRESS>Suraj<EMAIL_ADDRESS>Min<EMAIL_ADDRESS>Vision Science Lab
National Tsing Hua University
Hsinchu, Taiwan PanoMixSwap

# PanoMixSwap – Panorama Mixing via Structural Swapping for Indoor Scene
Understanding

###### Abstract

The volume and diversity of training data are critical for modern deep
learning-based methods. Compared to the massive amount of labeled perspective
images, ${360}^{\circ}$ panoramic images fall short in both volume and
diversity. In this paper, we propose PanoMixSwap, a novel data augmentation
technique specifically designed for indoor panoramic images. PanoMixSwap
explicitly mixes various background styles, foreground furniture, and room
layouts from the existing indoor panorama datasets and generates a diverse set
of new panoramic images to enrich the datasets. We first decompose each
panoramic image into its constituent parts: background style, foreground
furniture, and room layout. Then, we generate an augmented image by mixing
these three parts from three different images, such as the foreground
furniture from one image, the background style from another image, and the
room structure from the third image. Our method yields high diversity since
there is a cubical increase in image combinations. We also evaluate the
effectiveness of PanoMixSwap on two indoor scene understanding tasks: semantic
segmentation and layout estimation. Our experiments demonstrate that state-of-
the-art methods trained with PanoMixSwap outperform their original setting on
both tasks consistently. The website for this paper can be found at
https://yuchenghsieh.github.io/PanoMixSwap.

##  1 Introduction

Panoramic images have become increasingly popular in indoor scene
understanding tasks because they provide a comprehensive ${360}^{\circ}$ view
of a specific room. With the widespread availability of ${360}^{\circ}$
cameras, generating panoramic images has become more convenient. This inspired
the development of various indoor panoramic datasets such as Stanford2D3D
[Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese], Matterport3D [Chang et
al.(2017)Chang, Dai, Funkhouser, Halber, Niessner, Savva, Song, Zeng, and
Zhang], PanoContext [Zhang et al.(2014)Zhang, Song, Tan, and Xiao] and
Structured3D [Zheng et al.(2020)Zheng, Zhang, Li, Tang, Gao, and Zhou], as
well as the emergence of related tasks such as semantic segmentation, layout
estimation, and depth estimation. These tasks leverage the unique
characteristics of indoor panoramic images to enable a more holistic and
immersive understanding of indoor environments.

Despite the availability of indoor panoramic datasets, these images are
limited in volume and diversity compared to perspective images. For example,
even Stanford2D3D [Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese], one
of the largest real-world indoor panoramic datasets, contains only 1,413
panoramic images across 270 scene layouts. This scarcity of data presents
difficulties in training models that require both robustness and accuracy. To
address this issue, data augmentation techniques are often employed to
artificially expand the dataset and enhance the diversity of training samples,
thereby mitigating the effects of limited data availability.

Data augmentation in panoramic images poses unique challenges compared to
traditional image data augmentation methods since the inherent structure and
layout of panoramic images must be preserved during augmentation (_e.g_ for
indoor panoramic images, ceilings must be on top of walls and floors). Some
traditional data augmentation techniques, such as random cropping and free-
angle rotation, may not be suitable for panoramic images as they can disrupt
the intrinsic structure. This underscores the importance of developing novel
and specialized data augmentation techniques for panoramic images.

Current panoramic augmentations are either traditional methods that can
preserve the panoramic formats, such as horizontal rotation and flipping, or
methods specifically designed for panoramic images like PanoStretch proposed
by Sun _et al_ [Sun et al.(2019)Sun, Hsiao, Sun, and Chen]. However, these
methods only work on a single image, which prevents them from combining the
variability in different panoramic images as explored by other augmentation
methods for perspective images (_e.g_ MixUp [Zhang et al.(2017)Zhang, Cisse,
Dauphin, and Lopez-Paz]). Therefore, present panoramic augmentation methods
have limited capability to generate more diverse images.

To address the limited diversity issue in current panoramic augmentations, we
propose a novel panoramic augmentation technique called PanoMixSwap, which
utilizes multiple panoramic views to augment data and take advantage of
variations in different samples. By using two or more panoramic images,
semantic masks, and room layouts, we can generate numerous combinations to
diversify our training data. PanoMixSwap, as shown in Fig. 1, is inspired by
the observation that every indoor panoramic image typically consists of three
main parts: the room structure ($i.e.$, layout), style of the background
(including the ceiling, floor, and each wall), and the foreground furniture.
We use these three main parts from three different indoor panoramic views to
create a diverse set of augmented samples. Our method leverages a two-stage
network to sequentially fuse the background style and foreground furniture
into the chosen room layout. The resulting augmented images exhibit a wide
range of diverse outputs while preserving the structure of the original
panoramic images. We evaluate the effectiveness of our augmentation on two
scene understanding tasks: semantic segmentation and layout estimation. By
incorporating PanoMixSwap during training, we observe significantly improved
performance compared to the original settings.

Our key contributions to PanoMixSwap are summarized below.

  * •

We propose a novel data augmentation method PanoMixSwap for indoor panoramic
images. PanoMixSwap generates cubical increased diverse images by mixing three
source images while maintaining the structural integrity ($i.e.$, layout).
This approach addresses the issue of limited availability in the training data
and enhances the variability of the augmented images.

  * •

We apply PanoMixSwap to two scene understanding tasks, semantic segmentation
and layout estimation. PanoMixSwap consistently improves results compared to
the original training setting.

##  2 Related Works

Data Augmentations. In the field of computer vision, the size of the dataset
plays a crucial role in determining the final performance of the model; hence
data augmentation is an important technique for expanding training datasets.
Existing data augmentation methods can be categorized into two types: (1)
those that use only one training sample to derive one augmented sample and (2)
those that use two or more training samples to derive one augmented sample,
also called mixup. The first type of augmentation consists of a considerable
amount of work, with traditional methods such as random cropping, image
mirroring, and color jittering [Krizhevsky et al.(2012)Krizhevsky, Sutskever,
and Hinton] commonly used for 2D images, as well as more advanced approaches
like AutoAugment [Cubuk et al.(2019)Cubuk, Zoph, Mane, Vasudevan, and Le,
Cubuk et al.(2020)Cubuk, Zoph, Shlens, and Le] and GAN-based methods [Sixt et
al.(2018)Sixt, Wild, and Landgraf, Zhu et al.(2018)Zhu, Liu, Li, Wan, and
Qin]. Similarly, for panoramic images, horizontal rotation and flipping
techniques and Panostretch [Sun et al.(2019)Sun, Hsiao, Sun, and Chen]
introduced in Section 1 are widely used in panoramic-related tasks. On the
other hand, the second type of augmentation, $i.e.$, mixup, has been widely
studied in 2D image processing, with several works proposing techniques for
linearly interpolating two input data points along with their corresponding
one-hot labels [Zhang et al.(2017)Zhang, Cisse, Dauphin, and Lopez-Paz, Kim et
al.(2020)Kim, Choo, and Song, Yun et al.(2019)Yun, Han, Oh, Chun, Choe, and
Yoo, Guo et al.(2019)Guo, Mao, and Zhang, Verma et al.(2019)Verma, Lamb,
Beckham, Najafi, Mitliagkas, Lopez-Paz, and Bengio, Yoo et al.(2020)Yoo, Ahn,
and Sohn, Zhang et al.(2016)Zhang, Bengio, Hardt, Recht, and Vinyals]. For
example, Zhang _et al_ [Zhang et al.(2016)Zhang, Bengio, Hardt, Recht, and
Vinyals] generate virtual training examples using mixup by linearly
interpolating data points and their one-hot labels. Yun _et al_ [Yun et
al.(2019)Yun, Han, Oh, Chun, Choe, and Yoo] introduce a random-cut rectangular
region technique, where a portion of the image is removed and replaced with a
patch obtained from another image. Mixup techniques have also been applied in
the field of 3D point clouds [Chen et al.(2020)Chen, Hu, Gavves, Mensink,
Mettes, Yang, and Snoek, Umam et al.(2022)Umam, Yang, Chuang, Chuang, and
Lin]. However, to the best of our knowledge, no existing work currently
applies the concept of mixup to panoramic images, which serves as a key factor
motivating our proposed approach, PanoMixSwap.

${\textbf{360}}^{\circ}$ perception. The popularity of ${360}^{\circ}$ cameras
has recently surged, leading to an increased interest in vision tasks related
to panoramic images [360(2020), Zou et al.(2018)Zou, Colburn, Shan, and
Hoiem]. Equirectangular projections (ERPs) are commonly used to represent and
manipulate the wide field of view captured by these cameras. ERPs allow all
captured information to be preserved in a single image. However, they also
introduce distortion that can impede the performance of traditional
convolution layers designed for perspective images. There has been extensive
research on spherical convolution layers [Cohen et al.(2018)Cohen, Geiger,
Koehler, and Welling, Esteves et al.(2017)Esteves, Allen-Blanchette, Makadia,
and Daniilidis, Su and Grauman(2019), Su and Grauman(2017), Tateno et
al.(2018)Tateno, Navab, and Tombari] that are aware of these distortions. To
use ${360}^{\circ}$ panoramic images with conventional convolutional neural
networks (CNNs) that have a wide range of available pre-trained models,
multiple perspective projections are employed to project the image onto
multiple planar images. However, this method results in a loss of information
due to the projection process, which limits the field of view (FOV).
Furthermore, generating planar images from ${360}^{\circ}$ panoramic images
requires additional computational resources and time, which increases
exponentially with higher-resolution images. To address the problems
associated with projection-related works, several newer methods propose
different ways of padding [Cheng et al.(2018)Cheng, Chao, Dong, Wen, Liu, and
Sun, Wang et al.(2020)Wang, Yeh, Sun, Chiu, and Tsai] and sampling [Eder et
al.(2020)Eder, Shvets, Lim, and Frahm] image boundaries to remove
inconsistencies in panoramic images. The icosahedron mesh [Lee et
al.(2019)Lee, Jeong, Yun, Cho, and Yoon, Zhang et al.(2019)Zhang, Liwicki,
Smith, and Cipolla] provides a versatile and effective method for representing
3D shapes and scenes in computer vision, particularly for tasks that involve
spherical or panoramic data.

Figure 1:  Pipeline of PanoMixSwap. PanoMixSwap involves three major inputs:
style sample, structure layout, and furniture sample. PanoMixSwap is composed
of two blocks: Style Fusing Block and Furniture Fusing Block. The Style Fusing
Block generates a foreground-free styled structure image that fuses the
background style from the style image and the room layout from structure
layout. Furniture Fusing Block transforms furniture from the furniture image
onto the styled structure image to produce the final augmented image and
semantic mask.

##  3 PanoMixSwap

The commonly-used panoramic data augmentations mostly take only one sample as
input. However, the diversity of this kind of one-to-one mapping is rather
limited. We propose PanoMixSwap to mix three panoramic views into one, which
is as clean and high-fidelity as the source views. Thus, we can generate more
diverse training samples which are beyond the conventional panoramic
augmentation.

###  3.1 Overview

Let S be a training sample consisting of an RGB image $I\in\mathbb{R}^{H\times
W\times 3}$, a semantic mask $M\in[0,1]^{H\times W\times C}$ in the form of
one-hot vector with $C$ classes, and layout coordinates
$L\in\mathbb{R}^{T\times 2\times 2}$ recording the $T$-walls room corner
junctions on floor and ceiling. An output augmented sample by PanoMixSwap is
the combination of three main parts from three samples— room layout structure
of structure sample $\textbf{S}_{\mathrm{rs}}$, background style of style
sample $\textbf{S}_{\mathrm{bs}}$, and foreground furniture setups of
furniture sample $\textbf{S}_{\mathrm{fs}}$. An overview pipeline of
PanoMixSwap is illustrated in Fig. 1. We first generate a styled structure
image $I_{\mathrm{ss}}$ by mixing the background appearance from
$\textbf{S}_{\mathrm{bs}}$ and the room layout $L_{\mathrm{rs}}$ from
$\textbf{S}_{\mathrm{rs}}$:

$I_{\mathrm{ss}}=\mathrm{\textbf{StyleFusingBlock}}(\textbf{S}_{\mathrm{bs}},L_{\mathrm{rs}})~{},$
(1)

where the StyleFusingBlock is detailed in Sec. 3.2. We finally can generate
the augmented sample $\textbf{S}_{\mathrm{aug}}$ by aligning the furniture
setup of $\textbf{S}_{\mathrm{fs}}$ with the room layout $L_{\mathrm{rs}}$ and
then changing the background style using $I_{\mathrm{ss}}$:

$\textbf{S}_{\mathrm{aug}}=\mathrm{\textbf{FurnitureFusingBlock}}(\textbf{S}_{\mathrm{fs}},L_{\mathrm{rs}},I_{\mathrm{ss}})~{},$
(2)

where FurnitureFusingBlock is detailed in Sec. 3.3.

[capbesideposition=right,top,capbesidewidth=4.5cm]figure[]

Figure 2:  Style Fusing Block. Style Fusing Block is mainly composed of Style
Encoder and Style Generator. The Style Encoder is responsible for extracting
the embedded style vector for each semantic region of the style image. The
Style Generator creates a foreground-free styled structure image by generating
the appearance of each semantic region based on its corresponding style
embedded vector.

###  3.2 Style Fusing Block

There are two requirements about the generated styled structure image
$I_{\mathrm{ss}}$: i) the layout structure should be the same as the room
layout $L_{\mathrm{rs}}$ from structure sample $\textbf{S}_{\mathrm{rs}}$ and
ii) the background appearance should be similar to the style sample
$\textbf{S}_{\mathrm{bs}}$ with all the furniture removed. To achieve this, we
employ a semantic conditioned generative model, SEAN [Zhu et al.(2020)Zhu,
Abdal, Qin, and Wonka]. Specifically, given a content semantic mask, SEAN
generates the appearance of each semantic region based on the corresponding
semantic region from a reference image. We use $L_{\mathrm{rs}}$ to generate
the content semantic mask consisting of floor, ceiling, and walls where each
wall is assigned a unique class. The reference semantic mask is generated in
the same way using $L_{\mathrm{bs}}$. To prevent generating the foreground,
the reference semantic mask is further covered by an additional ‘others’ class
from the furniture and objects classes in $M_{\mathrm{bs}}$. We assume the
number of walls is the same in $L_{\mathrm{rs}}$ and $L_{\mathrm{bs}}$, so the
walls can be one-to-one corresponding. An overview of the Style Fusing Block
is illustrated in (Fig. 2).

Figure 3:  Furniture Fusing Block. Horizontal Alignment Block takes each
furniture column group to produce the width-aligned column group that matches
the wall width of the corresponding styled structure column group using
PanoStretch [Sun et al.(2019)Sun, Hsiao, Sun, and Chen]. Vertical Alignment
Block views both the width-aligned furniture column group and the styled
structure column group into ceiling, wall, and floor parts. Then generate the
final augmented column group by back warping three parts of the width-aligned
furniture column group (denoted aligned furniture column group) to match the
same height of three parts from the styled structure column group and
replacing background pixel of the styled structure column onto the aligned
furniture column group. We repeat the process for $T$ times to get the final
augmented image.

###  3.3 Furniture Fusing Block

The purpose of the Furniture Fusing Block is to fuse the furniture sample
$\textbf{S}_{\mathrm{fs}}$ with the room layout $L_{\mathrm{rs}}$ and the
styled structure image $I_{\mathrm{ss}}$. To this end, we first align the
image $I_{\mathrm{fs}}$ and the semantic mask $M_{\mathrm{fs}}$ from their
original layout $L_{\mathrm{fs}}$ to the target layout $L_{\mathrm{rs}}$. The
aligned image and mask are denoted as $I_{\mathrm{fs}\rightarrow\mathrm{rs}}$
and $M_{\mathrm{fs}\rightarrow\mathrm{rs}}$. The background pixels of
$I_{\mathrm{fs}\rightarrow\mathrm{rs}}$ are then replaced by $I_{\mathrm{ss}}$
to change the background style. The final augmented sample is:

$\textbf{S}_{\mathrm{aug}}=\\{mI_{\mathrm{fs}\rightarrow\mathrm{rs}}+(1-m)I_{\mathrm{ss}},~{}M_{\mathrm{fs}\rightarrow\mathrm{rs}},~{}L_{\mathrm{rs}}\\}~{},$
(3)

where $m$ is the foreground mask computed from
$M_{\mathrm{fs}\rightarrow\mathrm{rs}}$. Below are the details of the
alignment process.

Recap that we assume the number of walls is the same in $L_{\mathrm{fs}}$ and
$L_{\mathrm{rs}}$, and they are one-to-one corresponding. We depict the
overall process in Fig. 3. We first use the wall-wall boundary annotated in
$L_{\mathrm{fs}}$ to split the image columns of $I_{\mathrm{fs}}$ into
multiple image column groups. Each image column group is then processed by
Horizontal Alignment Block and Vertical Alignment Block sequentially. In the
Horizontal Alignment Block, we use PanoStretch [Sun et al.(2019)Sun, Hsiao,
Sun, and Chen] to stretch each image column group from its original width to
the corresponding wall width in $L_{\mathrm{rs}}$. In the Vertical Alignment
Block, we apply backward warping to each image column to align with the
ceiling-wall and floor-wall intersection in $L_{\mathrm{rs}}$. The source and
destination coordinates for the backward warping are computed as follows. Let
$r$ be the destination row index of an image column; the source index is
computed as

$\mathrm{Source}(r)=\begin{cases}\displaystyle
a_{\mathrm{src}}-\alpha(a_{\mathrm{dst}}-r),&\text{if $r<a_{\mathrm{dst}}$}\\\
\displaystyle b_{\mathrm{src}}+\beta(r-b_{\mathrm{dst}}),&\text{if
$r>b_{\mathrm{dst}}$}\\\ \displaystyle
a_{\mathrm{src}}+(b_{\mathrm{src}}-a_{\mathrm{src}})\frac{(r-a_{\mathrm{dst}})}{(b_{\mathrm{dst}}-a_{\mathrm{dst}})},&\text{otherwise}\end{cases}~{},$
(4)

where $a,b$ are the index of the ceiling-wall and floor-wall intersection,
$\alpha,\beta$ are hyperparameters. The equations in Eq. 4 correspond to the
warping regions of ceiling, floor, and wall between source and destination.
The image column groups are concatenated to form the aligned image
$I_{\mathrm{fs}\rightarrow\mathrm{rs}}$. Semantic mask $M_{\mathrm{fs}}$ is
processed in the same way to get $M_{\mathrm{fs}\rightarrow\mathrm{rs}}$.

##  4 Experiments

We present the implementation details and visualizations of our PanoMixSwap in
Section 4.1. We showcase the effectiveness of our novel data augmentation
technique on indoor ${360}^{\circ}$ semantic segmentation task in Section 4.2
and layout estimation task in Section 4.4.

###  4.1 PanoMixSwap

Implementation Detail. We focus on four-wall indoor panoramic images for
simplicity. To train the encoder-generator model discussed in Sec. 3.2, we
adopt a similar pipeline as proposed in SEAN [Zhu et al.(2020)Zhu, Abdal, Qin,
and Wonka] for training on both the Structured3D and Stanford2D3D datasets.
Specifically, we set the input image size to $H=256$ and $W=512$, use the Adam
optimizer with hyperparameters $\beta_{1}=0.5$ and $\beta_{2}=0.999$, and set
the learning rate to 2e-4. We use a batch size of 2 and train the model for 60
epochs on a single NVIDIA GTX 1080 Ti GPU. The inference run-time for an image
is about 2 seconds, so we apply our augmentation in an offline manner for
efficiency.

Visualizations. We illustrate the inputs and outputs of PanoMixSwap in Fig. 4.
Our method can generate a high-quality image by incorporating the background
style, room layout structure, and furniture information from three different
input samples. We use high-quality augmented images to enrich the training set
of different tasks. For instance, semantic segmentation training data can now
be augmented to different room structures and background styles; we can also
synthesize different room styles and furniture setups for a given ground-truth
room layout.

Figure 4:  Visualization of the results from our PanoMixSwap. The augmented
image (4th column) by our novel PanoMixSwap is a fusion of the room layout
from the structure image (1st column), the background style from the style
image (2nd column), and the furniture from the furniture image (3rd column).
The images in the 1st and 2nd rows are from Structured3D [Zheng et
al.(2020)Zheng, Zhang, Li, Tang, Gao, and Zhou] while the images from the 3rd
row are from Stanford2D3D [Armeni et al.(2017)Armeni, Sax, Zamir, and
Savarese].

###  4.2 Semantic Segmentation

Model, Dataset and Evaluation. In the semantic segmentation task, we use
HoHoNet [Sun et al.(2021)Sun, Sun, and Chen] and PanoFormer [Shen et
al.(2022)Shen, Lin, Liao, Nie, Zheng, and Zhao], which are two state-of-the-
art 360 semantic segmentator. We evaluate PanoMixSwap’s ability to handle
real-world and synthetic data by conducting experiments on two datasets:
Stanford2D3D [Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese] and
Structured3D [Zheng et al.(2020)Zheng, Zhang, Li, Tang, Gao, and Zhou],
whichttps://www.overleaf.com/project/64390116f8940e211f93e500h respectively
represent real-world and virtual-world environments. For Stanford2D3D [Armeni
et al.(2017)Armeni, Sax, Zamir, and Savarese], we use fold 5a and fold 5b for
validation and the remaining folds for training following prior works. As for
Structured3D [Zheng et al.(2020)Zheng, Zhang, Li, Tang, Gao, and Zhou], we
follow the official training, testing, and validation setting, where there are
3,000 scenes for training and 250 scenes for validation, and 250 scenes for
testing. We employ the class-wise mean intersection of union (mIoU) and mean
accuracy (mACC) for semantic segmentation evaluation.

Implementation Detail. In accordance with the original HoHoNet’s setting [Sun
et al.(2021)Sun, Sun, and Chen], we adopt similar implementation settings. For
low resolution input, a shallow U-Net with planar CNN is chosen, and the
network is trained for 60 epochs on Structured3D [Zheng et al.(2020)Zheng,
Zhang, Li, Tang, Gao, and Zhou] and 300 epochs on Stanford2D3D [Armeni et
al.(2017)Armeni, Sax, Zamir, and Savarese], using a batch size of 16 and a
learning rate of 1e-3 with polynomial decay of factor 0.9. For high resolution
input, ResNet-101 [He et al.(2016)He, Zhang, Ren, and Sun] is used as the
backbone, and the network is trained for 60 epochs on both Structured3D [Zheng
et al.(2020)Zheng, Zhang, Li, Tang, Gao, and Zhou] and Stanford2D3D [Armeni et
al.(2017)Armeni, Sax, Zamir, and Savarese], with a batch size of 4 and a
learning rate of 1e-4 with polynomial decay of factor 0.9. For both low
resolution and high resolution images, Adam [Kingma and Ba(2014)] is employed
as the optimizer for cross entropy loss.

In the case of PanoFormer [Shen et al.(2022)Shen, Lin, Liao, Nie, Zheng, and
Zhao], we use a batch size of 4 and an input resolution of 256*512 to train
for 60 epochs. Additionally, Adam [Kingma and Ba(2014)] is employed as the
optimizer for optimizing the cross entropy loss. To apply PanoMixSwap, we
first generate an augmented dataset with the same quantity as the original
training data and combine the augmented dataset and original training data
into a single data set.

Quantitative Results. The results of experiments on Stanford2D3D [Armeni et
al.(2017)Armeni, Sax, Zamir, and Savarese], as shown in the upper section of
Table 1, reveal that the inclusion of our augmentation technique during
training leads to significantly higher mIoU and mACC scores on both HoHoNet
[Sun et al.(2021)Sun, Sun, and Chen] and PanoFormer [Shen et al.(2022)Shen,
Lin, Liao, Nie, Zheng, and Zhao] compared to the original work without
PanoMixSwap, across all models and resolutions. Notably, in high resolution
settings, training with PanoMixSwap yields a remarkable improvement of 4.02%
in mIoU and 2.43% in mACC for HoHoNet [Sun et al.(2021)Sun, Sun, and Chen].
Based on these compelling results, it is evident that PanoMixSwap technique
consistently enhances the mIoU and mACC in real-world indoor panoramic
scenarios across different models and resolutions. In addition to real-world
scenarios, we also evaluate our augmentation in virtual environment settings
using Structured3D dataset [Zheng et al.(2020)Zheng, Zhang, Li, Tang, Gao, and
Zhou], as presented in the lower part of Table 1. The results demonstrate that
training with PanoMixSwap leads to higher mIoU and mACC scores in both low and
high resolution settings, further substantiating the effectiveness of our
technique in virtual indoor panoramic scenarios.

Dataset | Model | Image Size | PanoMixSwap | mIoU(%) | mACC(%)
---|---|---|---|---|---
Stanford2D3D | HoHoNet | $64\times 128$ | - | 31.67 | 46.27
✓ | 34.60 | 47.76
$256\times 512$ | - | 36.13 | 50.25
✓ | 41.25 | 52.50
$1024\times 2048$ | - | 52.00 | 65.00
✓ | 56.02 | 67.43
PanoFormer | $256\times 512$ | - | 42.20 | 61.03
✓ | 42.94 | 62.14
Structured3D | HoHoNet | $64\times 128$ | - | 61.11 | 71.94
✓ | 62.50 | 73.64
$256\times 512$ | - | 70.07 | 78.91
✓ | 72.40 | 81.00
$512\times 1024$ | - | 80.80 | 87.98
✓ | 81.96 | 88.52

Table 1:  Quantitative comparison on semantic segmentation. Our novel
PanoMixSwap significantly improves two state-of-the-art semantic segmentators,
HoHoNet [Sun et al.(2021)Sun, Sun, and Chen] and PanoFormer [Shen et
al.(2022)Shen, Lin, Liao, Nie, Zheng, and Zhao], on Stanford2D3D [Armeni et
al.(2017)Armeni, Sax, Zamir, and Savarese] and Structured3D [Zheng et
al.(2020)Zheng, Zhang, Li, Tang, Gao, and Zhou].

###  4.3 Layout Estimation

Model, Dataset and Evaluation. We utilize HorizonNet [Sun et al.(2019)Sun,
Hsiao, Sun, and Chen] and LGT-Net [Jiang et al.(2022)Jiang, Xiang, Xu, and
Zhao] to test the effectiveness of PanoMixSwap on cuboid layout estimation
task, and use the dataset introduced in LayoutNet by Zou _et al_ [Zou et
al.(2018)Zou, Colburn, Shan, and Hoiem] to estimate cuboid layout. This
dataset comprises 514 annotated cuboid room layouts from PanoContext [Zhang et
al.(2014)Zhang, Song, Tan, and Xiao] and 552 annotated cuboid room layouts
from Stanford2D3D [Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese]. We
follow train/valid/test split in layoutNet [Zou et al.(2018)Zou, Colburn,
Shan, and Hoiem]. For evaluation, we use standard evaluation metrics proposed
by Zou _et al_ [Zou et al.(2018)Zou, Colburn, Shan, and Hoiem] in cuboid
layout estimation, including intersection of union of 3D room layout (3DIoU),
corner error (CE), and pixel error (PE).

Implementation Detail. We follow all of the training settings in HorizonNet
[Sun et al.(2019)Sun, Hsiao, Sun, and Chen], which employs a learning rate of
3e-4, and a batch-size of 24 for 300 epochs. In addition, we utilize the
training split of Stanford2D3D [Armeni et al.(2017)Armeni, Sax, Zamir, and
Savarese] and PanoContext [Zhang et al.(2014)Zhang, Song, Tan, and Xiao] as
training data. As for LGT-net [Jiang et al.(2022)Jiang, Xiang, Xu, and Zhao]
we train for 1,000 epochs with a learning rate of 1e-4 and a batch-size of 6.
We follow the combined dataset scheme suggested by Zou _et al_ [Zou et
al.(2021)Zou, Su, Peng, Colburn, Shan, Wonka, Chu, and Hoiem], which involved
using the entire PanoContext [Zhang et al.(2014)Zhang, Song, Tan, and Xiao]
and the training split of Stanford2D3D [Armeni et al.(2017)Armeni, Sax, Zamir,
and Savarese] as the training data in LGT-net [Jiang et al.(2022)Jiang, Xiang,
Xu, and Zhao]. For both HorizonNet [Sun et al.(2019)Sun, Hsiao, Sun, and Chen]
and LGT-net [Jiang et al.(2022)Jiang, Xiang, Xu, and Zhao], we employ Adam
optimizer [Kingma and Ba(2014)] with $\beta_{1}=0.9$, $\beta_{2}=0.999$ and
PanoStretch [Sun et al.(2019)Sun, Hsiao, Sun, and Chen] during training. In
training with PanoMixSwap, We apply image augmentation only to the images in
the Stanford2D3D [Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese], and
allocate half of the batch size to augmented data and the other half to
training data.

Quantitative Results. Table 2 presents a comparison between the performance of
using PanoMixSwap during training and the original setting on Stanford2D3D
[Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese]. The results show that
utilizing PanoMixSwap during training outperforms the original setting in
3DIoU, CE on HorizonNet [Sun et al.(2019)Sun, Hsiao, Sun, and Chen] and 3DIoU,
PE on LGT-Net [Jiang et al.(2022)Jiang, Xiang, Xu, and Zhao]. Especially on
HorizonNet [Sun et al.(2019)Sun, Hsiao, Sun, and Chen], training with
PanoMixSwap yields a significant improvement of 3.1% in 3DIoU. This signifies
that PanoMixSwap has the capability to diversify the training room style and
furniture setup, thereby enhancing the overall performance.

Model | PanoMixSwap | 3DIoU(%) | CE(%) | PE(%)
---|---|---|---|---
HorizonNet | - | 83.51 | 0.62 | 1.97
✓ | 86.61 | 0.61 | 1.99
LGT-Net | - | 86.03 | 0.63 | 2.11
✓ | 86.96 | 0.63 | 2.04

Table 2:  Quantitative comparison on cuboid room layout estimation. Our
PanoMixSwap can improve HorizonNet [Sun et al.(2019)Sun, Hsiao, Sun, and Chen]
and LGT-Net [Jiang et al.(2022)Jiang, Xiang, Xu, and Zhao] on LayoutNet
dataset [Zou et al.(2018)Zou, Colburn, Shan, and Hoiem].

###  4.4 Comparison Between SOTA Augmentation

This section provides a comprehensive comparison between PanoMixSwap and 360
state-of-the-art data augmentation – PanoStretch proposed by Sun _et al_ [Sun
et al.(2019)Sun, Hsiao, Sun, and Chen] on semantic segmentation task and
layout estimation task. The comparison results of semantic segmentation and
layout estimation are shown in Table. 3 and Table. 4, respectively. The
results of the above two tables show that utilizing PanoMixSwap outperforms
PanoStretch in above two tasks.

PanoStretch | PanoMixSwap | mIoU(%) | mACC(%)
---|---|---|---
- | - | 52.00 | 65.00
✓ | - | 53.63 | 65.06
- | ✓ | 56.02 | 67.43
✓ | ✓ | 55.91 | 67.03
Table 3: Quantitative comparison between PanoMixSwap and PanoStretch on semantic segmentation task. We use HoHoNet [Sun et al.(2021)Sun, Sun, and Chen] on the Stanford2D3D dataset [Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese] for comparison. Model | PanoStretch | PanoMixSwap | 3DIoU(%) | CE(%) | PE(%)
---|---|---|---|---|---
HorizonNet | ✓ | - | 83.88 | 0.63 | 2.00
- | ✓ | 85.15 | 0.62 | 1.98
✓ | ✓ | 86.59 | 0.62 | 1.94
LGT-Net | ✓ | - | 85.98 | 0.65 | 2.11
- | ✓ | 86.60 | 0.62 | 2.06
✓ | ✓ | 86.96 | 0.63 | 2.04
Table 4: Quantitative comparison between PanoMixSwap and 360 PanoStretch
across the LayoutNet dataset [Zou et al.(2018)Zou, Colburn, Shan, and Hoiem]
on layout estimation task.

###  4.5 Qualitative Comparison on Downstream Tasks

Fig. 5 presents a qualitative comparison of layout estimation and semantic
segmentation. We use HoHoNet [Sun et al.(2021)Sun, Sun, and Chen] as semantic
segmentator and HorizonNet [Sun et al.(2019)Sun, Hsiao, Sun, and Chen] as
layout estimator. More qualitative results can be found in the supplementary
materials.

Figure 5:  Qualitative comparison on layout estimation and semantic
segmentation

##  5 Conclusion

We present PanoMixSwap, a novel data augmentation method for ${360}^{\circ}$
indoor panoramic images. PanoMixSwap aims to mix multiple panoramic images to
address the issue of data scarcity in panoramic image datasets. Moreover,
PanoMixSwap introduces an intuitive idea by decomposing a single indoor
panoramic image into three distinct parts: foreground furniture, background
style, and room layout parts. Then, it mixes multiple panoramic images by
swapping these structural parts to generate diverse images. Finally,
comprehensive experiments demonstrate that PanoMixSwap consistently improves
state-of-the-art models on multiple ${360}^{\circ}$ indoor scene understanding
tasks.

## Acknowledgement

This work is supported in part by Ministry of Science and Technology of Taiwan
(NSTC 111-2634-F-002-022). We thank National Center for High-performance
Computing (NCHC) for computational and storage resource. We especially thank
Chun-Che Wu for providing invaluable guidance for our paper.

## References

  * [360(2020)] 360sd-net: 360° stereo depth estimation with learnable cost volume.  In _2020 IEEE International Conference on Robotics and Automation, ICRA 2020_ , Proceedings - IEEE International Conference on Robotics and Automation, pages 582–588, United States, May 2020. Institute of Electrical and Electronics Engineers Inc.  10.1109/ICRA40945.2020.9196975.
  * [Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese] Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese.  Joint 2d-3d-semantic data for indoor scene understanding.  _arXiv preprint arXiv:1702.01105_ , 2017.
  * [Chang et al.(2017)Chang, Dai, Funkhouser, Halber, Niessner, Savva, Song, Zeng, and Zhang] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  _arXiv preprint arXiv:1709.06158_ , 2017.
  * [Chen et al.(2020)Chen, Hu, Gavves, Mensink, Mettes, Yang, and Snoek] Yunlu Chen, Vincent Tao Hu, Efstratios Gavves, Thomas Mensink, Pascal Mettes, Pengwan Yang, and Cees GM Snoek.  Pointmixup: Augmentation for point clouds.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pages 330–345. Springer, 2020.
  * [Cheng et al.(2018)Cheng, Chao, Dong, Wen, Liu, and Sun] Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, and Min Sun.  Cube padding for weakly-supervised saliency prediction in 360° videos.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1420–1429, 2018.  10.1109/CVPR.2018.00154.
  * [Cohen et al.(2018)Cohen, Geiger, Koehler, and Welling] Taco S. Cohen, Mario Geiger, Jonas Koehler, and Max Welling.  Spherical cnns, 2018.
  * [Cubuk et al.(2019)Cubuk, Zoph, Mane, Vasudevan, and Le] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 113–123, 2019.
  * [Cubuk et al.(2020)Cubuk, Zoph, Shlens, and Le] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 702–703, 2020.
  * [Eder et al.(2020)Eder, Shvets, Lim, and Frahm] Marc Eder, Mykhailo Shvets, John Lim, and Jan-Michael Frahm.  Tangent images for mitigating spherical distortion.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [Esteves et al.(2017)Esteves, Allen-Blanchette, Makadia, and Daniilidis] Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis.  Learning so(3) equivariant representations with spherical cnns.  _CoRR_ , 2017.
  * [Guo et al.(2019)Guo, Mao, and Zhang] Hongyu Guo, Yongyi Mao, and Richong Zhang.  Mixup as locally linear out-of-manifold regularization.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 3714–3722, 2019.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.  10.1109/CVPR.2016.90.
  * [Jiang et al.(2022)Jiang, Xiang, Xu, and Zhao] Zhigang Jiang, Zhongzheng Xiang, Jinhua Xu, and Ming Zhao.  Lgt-net: Indoor panoramic room layout estimation with geometry-aware transformer network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1654–1663, 2022.
  * [Kim et al.(2020)Kim, Choo, and Song] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song.  Puzzle mix: Exploiting saliency and local statistics for optimal mixup.  In _International Conference on Machine Learning_ , pages 5275–5285. PMLR, 2020.
  * [Kingma and Ba(2014)] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , volume 25. Curran Associates, Inc., 2012.
  * [Lee et al.(2019)Lee, Jeong, Yun, Cho, and Yoon] Yeonkun Lee, Jaeseok Jeong, Jongseob Yun, Wonjune Cho, and Kuk-Jin Yoon.  Spherephd: Applying cnns on a spherical polyhedron representation of 360° images.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9173–9181, 2019.  10.1109/CVPR.2019.00940.
  * [Shen et al.(2022)Shen, Lin, Liao, Nie, Zheng, and Zhao] Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao.  Panoformer: Panorama transformer for indoor ${360}^{\circ}$ depth estimation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part I_ , pages 195–211. Springer, 2022.
  * [Sixt et al.(2018)Sixt, Wild, and Landgraf] Leon Sixt, Benjamin Wild, and Tim Landgraf.  Rendergan: Generating realistic labeled data.  _Frontiers in Robotics and AI_ , 5:66, 2018.
  * [Su and Grauman(2017)] Yu-Chuan Su and Kristen Grauman.  Learning spherical convolution for fast features from 360°imagery.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * [Su and Grauman(2019)] Yu-Chuan Su and Kristen Grauman.  Kernel transformer networks for compact spherical convolution.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9434–9443, 2019.  10.1109/CVPR.2019.00967.
  * [Sun et al.(2019)Sun, Hsiao, Sun, and Chen] Cheng Sun, Chi-Wei Hsiao, Min Sun, and Hwann-Tzong Chen.  Horizonnet: Learning room layout with 1d representation and pano stretch data augmentation.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 1047–1056, 2019.
  * [Sun et al.(2021)Sun, Sun, and Chen] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Hohonet: 360 indoor holistic understanding with latent horizontal features.  In _CVPR_ , 2021.
  * [Tateno et al.(2018)Tateno, Navab, and Tombari] Keisuke Tateno, Nassir Navab, and Federico Tombari.  Distortion-aware convolutional filters for dense prediction in panoramic images.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [Umam et al.(2022)Umam, Yang, Chuang, Chuang, and Lin] Ardian Umam, Cheng-Kun Yang, Yung-Yu Chuang, Jen-Hui Chuang, and Yen-Yu Lin.  Point mixswap: Attentional point cloud mixing via swapping matched structural divisions.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIX_ , pages 596–611. Springer, 2022.
  * [Verma et al.(2019)Verma, Lamb, Beckham, Najafi, Mitliagkas, Lopez-Paz, and Bengio] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio.  Manifold mixup: Better representations by interpolating hidden states.  In _International conference on machine learning_ , pages 6438–6447. PMLR, 2019.
  * [Wang et al.(2020)Wang, Yeh, Sun, Chiu, and Tsai] Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai.  Bifuse: Monocular 360 depth estimation via bi-projection fusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [Yoo et al.(2020)Yoo, Ahn, and Sohn] Jaejun Yoo, Namhyuk Ahn, and Kyung-Ah Sohn.  Rethinking data augmentation for image super-resolution: A comprehensive analysis and a new strategy.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8375–8384, 2020.
  * [Yun et al.(2019)Yun, Han, Oh, Chun, Choe, and Yoo] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6023–6032, 2019.
  * [Zhang et al.(2019)Zhang, Liwicki, Smith, and Cipolla] Chao Zhang, Stephan Liwicki, William Smith, and Roberto Cipolla.  Orientation-aware semantic segmentation on icosahedron spheres.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [Zhang et al.(2016)Zhang, Bengio, Hardt, Recht, and Vinyals] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  _arXiv preprint arXiv:1611.03530_ , 2016.
  * [Zhang et al.(2017)Zhang, Cisse, Dauphin, and Lopez-Paz] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017.
  * [Zhang et al.(2014)Zhang, Song, Tan, and Xiao] Yinda Zhang, Shuran Song, Ping Tan, and Jianxiong Xiao.  Panocontext: A whole-room 3d context model for panoramic scene understanding.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_ , pages 668–686. Springer, 2014.
  * [Zheng et al.(2020)Zheng, Zhang, Li, Tang, Gao, and Zhou] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou.  Structured3d: A large photo-realistic dataset for structured 3d modeling.  In _Proceedings of The European Conference on Computer Vision (ECCV)_ , 2020.
  * [Zhu et al.(2020)Zhu, Abdal, Qin, and Wonka] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka.  Sean: Image synthesis with semantic region-adaptive normalization.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [Zhu et al.(2018)Zhu, Liu, Li, Wan, and Qin] Xinyue Zhu, Yifan Liu, Jiahong Li, Tao Wan, and Zengchang Qin.  Emotion classification with data augmentation using generative adversarial networks.  In _Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part III 22_ , pages 349–360. Springer, 2018.
  * [Zou et al.(2018)Zou, Colburn, Shan, and Hoiem] Chuhang Zou, Alex Colburn, Qi Shan, and Derek Hoiem.  Layoutnet: Reconstructing the 3d room layout from a single rgb image.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2051–2059, 2018.
  * [Zou et al.(2021)Zou, Su, Peng, Colburn, Shan, Wonka, Chu, and Hoiem] Chuhang Zou, Jheng-Wei Su, Chi-Han Peng, Alex Colburn, Qi Shan, Peter Wonka, Hung-Kuo Chu, and Derek Hoiem.  Manhattan room layout reconstruction from a single ${360}^{\circ}$ image: A comparative study of state-of-the-art methods.  _International Journal of Computer Vision_ , 129:1410–1431, 2021.