# 3D Gated Recurrent Fusion for Semantic Scene Completion

Yu Liu${}^{1^{\dagger}}$, Jie Li${}^{2^{\dagger}}$, Qingsen Yan3, Xia Yuan2,
Chunxia Zhao2, Ian Reid1 and Cesar Cadena4 Yu Liu and Jie Li contributed
equally to this work.
1Y. Liu and I. Reid are with School of Computer Science, The University of
Adelaide, 5005, North Terrace, SA<EMAIL_ADDRESS>
2J. Li, X. Yuan and C. Zhao are with School of Computer Science and
Engineering, Nanjing University of Science and Technology, Nanjing, 210094,
China<EMAIL_ADDRESS>
3 Q. Yan is with School of Computer Science and Engineering, Northwestern
Polytechnical University, Xi’an, 710072, China<EMAIL_ADDRESS>
4C. Cadena is with Autonomous Systems Lab, ETH Zurich, Leonhardstrasse 21,
8092, Zurich<EMAIL_ADDRESS>
###### Abstract

This paper tackles the problem of data fusion in the semantic scene completion
(SSC) task, which can simultaneously deal with semantic labeling and scene
completion. RGB images contain texture details of the object(s) which are
vital for semantic scene understanding. Meanwhile, depth images capture
geometric clues of high relevance for shape completion. Using both RGB and
depth images can further boost the accuracy of SSC over employing one modality
in isolation. We propose a 3D gated recurrent fusion network (GRFNet), which
learns to adaptively select and fuse the relevant information from depth and
RGB by making use of the gate and memory modules. Based on the single-stage
fusion, we further propose a multi-stage fusion strategy, which could model
the correlations among different stages within the network. Extensive
experiments on two benchmark datasets demonstrate the superior performance and
the effectiveness of the proposed GRFNet for data fusion in SSC. Code will be
made available.

##  1 Introduction

Understanding the surroundings is a fundamental capability for many real-world
applications such as augmented reality [2], robot grasping [37], or autonomous
navigation [9]. Different abstractions are possible, and even complementary.
Semantic labeling of the scene allows for a high level reasoning, while 3D
geometry completion enables basic spatial capabilities. Semantic scene
completion (SSC) aims at solving both simultaneously.

An RGB-D sensor allows acquiring depth information from the scene along side
the RGB image. On the one hand, RGB image contains rich details about the
color and texture, which are the primary cues for the semantic scene
understanding. On the other hand, depth carries more clues about the object
geometry and distance information, which are much reliable in reflecting the
position, shape, and occlusion relationship between objects within the scene.
Many vision applications have already benefit from using both modalities in
their tasks, such as object detection [16, 3], video segmentation [12, 36,
10], action recognition [21, 20, 42], or visual SLAM [23, 40, 29]. Recent
studies [13, 25] in SSC also demonstrate that employing both, RGB image and
depth, can outperform using only one modality [35].

However, fusing the information from RGB and depth is still an unsolved
problem, and becomes an obstacle which hinders the performance of SSC. Albeit
some recent works conduct data fusion between RGB and depth, they usually
employ some, “manually” set, basic operation to fuse the data. Those includes
sum fusion [25, 18], max fusion [22], concatenate fusion [7, 15], transform
fusion [38] and bilinear fusion [28]. Nevertheless, RGB and depth data are not
equivalent quantities, while still providing complementary yet redundant
information. Therefore, we propose to extract the information in a selective
manner from both modalities, and fuse them accordingly with respect to the
specific task.

We present the Gated Recurrent Fusion (GRF) block, which can provide adaptive
selection and aggregation of RGB and depth information. On the one hand, the
_gate_ component in the GRF fusion block selects, in an adaptive manner,
various positions of different importance in aligned RGB-D frames regarding to
the contribution from both modalities. The _gate_ effectively selects valid
information while filters out the irrelevant one. On the other hand, the
_memory_ component in the GRF fusion block effectively preserves the
complementary information, which can compensate the missing or ambiguous
details of the data obtained from different modalities.

Furthermore, the GRF fusion block offers the flexibility to be cascaded to a
multi-stage configuration that combines high-level and low-level features. GRF
fusion block is extended from the Gated Recurrent Unit (GRU) [5]. Based on the
GRF fusion block, we build the GRFNet for the semantic scene completion, and
provide single- and multi-stage fusion versions of GRFNet. In the single-stage
fusion version, depth and RGB features are fed into the same GRF fusion block
individually. In the multi-stage fusion version, depth and RGB features of
different stages form an interleaved sequence and are input into the same GRF
fusion block consecutively.

The multi-stage version takes advantage of both low-level and high-level
features, and achieves better performance than the single-stage version.

In summary, the contributions of this work are mainly two-fold:

  * •

An end-to-end 3D-GRF based network, GRFNet, is presented for fusing RGB and
depth information in the SSC task, through employing _gate_ and _memory_
components, the selection and fusion between two modalities can be conducted
effectively. To the best of our knowledge, this is the first time that gated
recurrent network is employed for data fusion in the SSC task.

  * •

Within the framework of GRFNet, single-stage and multi-stage fusion strategies
are proposed. While outperforming existing fusing strategies in the SSC task
already with the single stage, the multi-stage fusion proves to give the best
results.

Extensive experiments demonstrate that the proposed GRFNet achieves superior
performance on NYU [33] and NYUCAD [11] datasets.

Figure 1: Fusion at different stages. Early fusion contains more low-level
features, while the input of the late fusion contains more abstract high-level
features.

##  2 Related Work

In this section, we briefly look through the deep learning based methods for
SSC, with emphasis on the discussion of the existing multi-modality fusion
strategies.

###  2.1 Semantic Scene Completion

The goal of SSC is to produce a complete 3D voxel representation for a scene
from a single-view input. Specifically, Song _et al_. ([35]) propose an end-
to-end 3D convolutional network (SSCNet) which is based on the single-view
depth as input that can simultaneously predict the results of scene completion
and semantic labeling. SSCNet has high computational costs due to the adoption
of 3D convolutions. Zhang _et al_. ([43]) introduce spatial group convolution
(SGC) into SSC for accelerating the computation of 3D dense prediction task.
Meanwhile, Han _et al_. ([17]) employ the long short-term memory (LSTM) to
recover missing parts of 3D shapes. Dai _et al_. ([8]) use a coarse-to-fine
strategy to handle large scenes with varying spatial extent. Although the
depth-based approach has made significant progress, the absence of texture
details prevents improving SSC.

In order to incorporating the color information, TS3D [13] introduces the RGB
image into SSC and uses a 2D network to acquire semantic segmentation results.
Semantic outputs of the RGB stream are concatenated with inputs of the depth
stream to obtain the completed 3D scene. DDR-SSC [25] uses two parallel
feature extraction branches with the same structure to obtain information from
RGB and depth simultaneously. A multi-stage structure with element-wise
addition is employed to perform feature fusion. Thanks to the semantic
information provided by RGB, the semantic labeling accuracy of both TS3D and
DDR-SSC has significantly been improved compared to SSCNet. However, none of
these methods takes into account the selective fusion of multi-modal
information, which limit those algorithms to achieve better performance.

###  2.2 Fusion Schemes

The RGB-D information fusion is important to many vision applications. In
general, the fusion scheme can be divided into three categories, e.g. early
fusion [7], middle fusion [31] and late fusion [41, 34], as shown in Figure 1.
According to the stages of fusion, these schemes can also be divided into
single-stage fusion and multi-stage fusion [18, 30].

Figure 2: Several typical single-stage fusion methods.  Figure 3: The network
architecture of GRFNet is extended from Dimensional Decomposition Residual
(DDR) network [25]. GRFNet has two feature extractors to capture the features
from depth and RGB images respectively. The feature extractor contains a
projection layer to map 2D feature to 3D space. The GRF fusion block (denoted
by yellow boxes in the middle) replaces the original fusion unit to take full
advantage of the multi-modal information. With two DDR plus their
corresponding GRF fusion block to form a fusion module, also named single-
stage fusion module (denoted by the red box in one column). GRFNet is composed
of a multi-stage ( 4-stage here) fusion module. Then we use light-weight ASPP
to obtain multiple receptive fields information. Different colors of the DDR
block denote various receptive fields. Then the network uses two 3D
convolutions to predict occupancies and object labels simultaneously.

Single-Stage Fusion There are several general patterns for single-stage fusion
as shown in Figure 2. Specifically, Sum fusion [25, 18] computes the sum of
the two feature maps at the same spatial locations. Average fusion is
essentially a weighted sum fusion with equal weights. Max fusion [22] takes
the feature with the maximal value from multiple feature maps. Concatenate
fusion stacks the features with channels [7, 15]. Wang _et al._ ([38]) propose
an encoder-decoder architecture which exchanges the information of multi-modal
data in the latent space. Bilinear fusion [28] computes an outer matrix
product of the two features at each pixel location.

There are a few methods that consider the complementary and selectivity of
data fusion. Specifically, Li _et al_. ([26]) develop a novel LSTM model to
fuse scene contexts adaptively. Cheng _et al_. ([4]) use the concatenated
feature maps of RGB and depth to learn an array $G$ to weight the contribution
of one input modality and $1-G$ to weight the other input modality. Wang _et
al_. ([39]) use the same strategy as [4] to fuse the feature maps from RGB and
Depth in saliency detection. However, Li _et al_. ([26]) only consider the
complementarity of information but ignore the selectivity of the data, the
other two methods only consider the selectivity of information and cannot
guarantee the complementarity of information. Moreover, these methods are
single-stage fusion and lack of scalability.

Multi-stage Fusion According to the way for aggregating multi-modal
information, this paper divides multi-stage fusion algorithms into merge
fusion, cross fusion, and external fusion. Hazirbas _et al_. ([18]) adopt the
merge fusion structure to fuse the two branches of features extracted from RGB
and depth images. The feature maps from depth are fused into the RGB branch by
stages with an element-wise summation. Wang _et al_. ([38]) use cross fusion
to merge the common features of RGB and depth, and keep the modality specific
features separated from each other. Both Park _et al_. ([30]) and Li _et al_.
([25]) use an external fusion mechanism. Specifically, Li _et al_. ([25])
capture features of RGB and depth image at different levels, these features at
each level are fused separately and then assembled all at once before the
reconstruction part. Park _et al_. ([30]) propose RDFNet to fuse multi-modal
features separately by multiple fusion blocks, and refine the fused features
one by one through a set of refine blocks. In RDFNet, each fusion introduces
an additional fusion block with a new set of extra parameters. The
artificially designed fusion blocks are complex and require multiple
parameters that are not easy to migrate to other applications. These multi-
stage fusion methods use high-level and low-level features achieving high
accuracy. However, each fusion block within the multi-stage mostly adopts
concatenation or summation, ignoring the adaptive selection of the multi-modal
data.

On the contrary, our proposed GRF fusion block extends the standard gated
recurrent unit (GRU), where the gate and the memory structures can adaptively
select and preserve valid information. Besides, GRFNet adopts the form of a
recurrent network. When performing multi-stage fusion, GRF modules exploit
parameter sharing. And experiments show that both of the proposed single- and
multi-stage GRFNets achieve better accuracy than previous methods.

##  3 Methodology

###  3.1 Overview

Our proposal, GRFNet, extends the network architecture of DDR-SSC [25], and
focuses on improving the fusion strategy. We subtly adopt the gate structure
and memory mechanism in GRU unit to form a multi-modal feature GRF fusion
block with the power of autonomous selectivity and adaptive memory
preservation. Moreover, taking advantage of its recurrent nature, we further
propose a multi-stage fusion strategy to utilize both low-level and high-level
features with introducing insignificant parameters.

In the feature extractor part, the network uses dimensional decomposition
residual (DDR) blocks to extract the local textures and the geometry
information. A projection layer is employed to connect the 2D and 3D parts.
The multi-stage fusion module consists of four single-stage fusion modules
that can effectively combines the RGB features and depth features. The fused
features are fed into the subsequent light-weight atrous spatial pyramid
pooling (LW-ASPP [25]). After that, another two point-wise convolutional
layers are used to predict the semantic labels for each voxel in the 3D
volume.

The network maps each voxel to one of the labels $C={c_{0},c_{1},\cdots
c_{N}}$, where $N$ is the number of semantic classes, and $c_{0}$ represents
the empty voxel.

###  3.2 Gated Recurrent Unit

Gated recurrent unit (GRU) [5] is a popular model in recurrent neural networks
(RNN) and has an outstanding performance in many natural language processing
(NLP) tasks [6, 24]. A GRU has two gate structures, a memory structure, and
can be reused recurrently. However, few researchers have explored the power of
GRU in the field of 3D vision, especially for feature fusion. We find that GRU
highly aligns with our requirements for an effective multi-modal fusion
strategy in SSC.

The gate structure in GRU enables the selective fusion of multi-modal
features. The memory structure ensures that valid information can be retained
for future fusion purpose. The characteristics of its recurrent network enable
GRU to be reused in the multi-stage fusion while sharing the same set of
parameters. Compared to GRU, the structure of Long short-term memory (LSTM)
[19] is more complicated and has an extra forget gate with more parameters.
ConvGRU [1] is a convolutional version of GRU. We extend ConvGRU to 3D
convolutional in our GRFNet, and modify it to fit the feature fusion in SSC
task.

###  3.3 Gated Recurrent Fusion Block of RGB-D Features

As shown in Figure 4, at fist step (left), gated recurrent fusion (GRF) block
takes one of the RGB-D features as input. The outputs of this step will be
used as the hidden state. Then, in the second step (right), GRF fusion block
takes the features of another modality as input. These two steps reuse the GRF
fusion block and share the same set of parameters. Next, we will use the first
step with input $f^{d}$ as an example to explain in detail the principle and
work flow of the GRF fusion block.

In contrast to the commonly used GRU for encoding information in a temporal
way, the way we use GRF to fuse RGB+Depth features is somehow different.
Specifically, the GRU handles the fusion of RGB and depth information in a
‘modality’, rather than a ‘sequential’, way. And for the multi-stage fusion
process, similar as other deep neural networks, it is more like the low-level
multi-modal feature to guide the following high-level multi-modal feature to
be merged.

Figure 4: GRF fusion block. At step $p$, the input of GRF fusion block is one
of the features from depth or RGB, and in the next step, input is the other.
Both GRF fusion blocks share the same set of parameters.

| scene completion | semantic scene completion
---|---|---
Method | prec. | recall | IoU |  rgb1ceil. |  rgb2floor |  rgb3wall |  rgb4win. |  rgb5chair |  rgb6bed |  rgb7sofa |  rgb8table |  rgb9tvs |  rgb10furn. |  rgb11objs. | avg.
Lin _et al_. ([27]) | 58.5 | 49.9 | 36.4 | 0.0 | 11.7 | 13.3 | 14.1 | 9.4 | 29.0 | 24.0 | 6.0 | 7.0 | 16.2 | 1.1 | 12.0
Geiger _et al_. ([14]) | 65.7 | 58.0 | 44.4 | 10.2 | 62.5 | 19.1 | 5.8 | 8.5 | 40.6 | 27.7 | 7.0 | 6.0 | 22.6 | 5.9 | 19.6
SSCNet [35] | 57.0 | 94.5 | 55.1 | 15.1 | 94.7 | 24.4 | 0.0 | 12.6 | 32.1 | 35.0 | 13.0 | 7.8 | 27.1 | 10.1 | 24.7
EsscNet [43] | 71.9 | 71.9 | 56.2 | 17.5 | 75.4 | 25.8 | 6.7 | 15.3 | 53.8 | 42.4 | 11.2 | 0 | 33.4 | 11.8 | 26.7
DDR-SSC [25] | 71.5 | 80.8 | 61.0 | 21.1 | 92.2 | 33.5 | 6.8 | 14.8 | 48.3 | 42.3 | 13.2 | 13.9 | 35.3 | 13.2 | 30.4
GRFNet | 68.4 | 85.4 | 61.2 | 24.0 | 91.7 | 33.3 | 19.0 | 18.1 | 51.9 | 45.5 | 13.4 | 13.3 | 37.3 | 15.0 | 32.9

Table 1: Results on the NYU dataset [33]. Bold numbers represent the best
scores.

| scene completion | semantic scene completion
---|---|---
Method | prec. | recall | IoU |  rgb1ceil. |  rgb2floor |  rgb3wall |  rgb4win. |  rgb5chair |  rgb6bed |  rgb7sofa |  rgb8table |  rgb9tvs |  rgb10furn. |  rgb11objs. | avg.
Zheng _et al_.( [44]) | 60.1 | 46.7 | 34.6 | - | - | - | - | - | - | - | - | - | - | - | -
Firman _et al_.( [11]) | 66.5 | 69.7 | 50.8 | - | - | - | - | - | - | - | - | - | - | - | -
SSCNet [35] | 75.4 | 96.3 | 73.2 | 32.5 | 92.6 | 40.2 | 8.9 | 33.9 | 57.0 | 59.5 | 28.3 | 8.1 | 44.8 | 25.1 | 40.0
TS3D [13] | 80.2 | 91.0 | 74.2 | 33.8 | 92.9 | 46.8 | 27.0 | 27.9 | 61.6 | 51.6 | 27.6 | 26.9 | 44.5 | 22.0 | 42.1
DDR-SSC [25] | 88.7 | 88.5 | 79.4 | 54.1 | 91.5 | 56.4 | 14.9 | 37.0 | 55.7 | 51.0 | 28.8 | 9.2 | 44.1 | 27.8 | 42.8
GRFNet | 87.2 | 91.0 | 80.1 | 50.3 | 91.8 | 58.1 | 18.4 | 42.7 | 60.6 | 52.8 | 34.6 | 11.5 | 46.6 | 30.8 | 45.3

Table 2: Results on the NYUCAD dataset [44]. Bold numbers represent the best
scores.

Hidden State The hidden state ${h}_{p}$, along with the current input $f$ to
control the reset and update gates. The output of the previous stage will be
used as the hidden state of current stage. At the first step of fusion between
depth feature $f^{d}$ and RGB feature $f^{rgb}$, that is $p=1$, we use the sum
fusion of two modal features to initialize the hidden state, as
${h}_{0}=f^{d}+f^{rgb}$.

Reset Gate At step $p$, the hidden state ${h}_{p}$ and the current input
$f^{d}$ together to decide the status of the reset gate $r$ by

${r}=\sigma\left({W}_{r}\left({f}^{d},{h}_{p}\right)\right)$ (1)

The two feature stream ${f}^{d}$ and ${h}_{p}$ are concatenated and fed into a
convolution operation. ${W}_{r}$ represents the corresponding weights in the
convolution. The sigmoid function $\sigma$ converts each value in the feature
tensor into the range of $(0,1)$ and acts as a gate signal.

Update Gate The update gate $z$ is also decided by the hidden state ${h}_{p}$
and input features ${f}^{d}$. Through another convolution operation with
weight ${W}_{z}$ and the sigmoid function $\sigma$, we get $z$ as,

${z}=\sigma\left({W}_{z}\left({f}^{d},{h}_{p}\right)\right)$ (2)

Theoretically, reset and update gates essentially learn a set of weights that
control the amount of information that is retained or discarded, experimental
studies in section 4 show the effectiveness of the reset and update gates.

Adaptive Memory Through the element-wise product $\odot$, the reset gate $r$
determines how much information in the past needs to be “memorized”.

${h^{\prime}}_{p}={r}\odot{h}_{p}$ (3)

when the reset gate $r$ is close to 1, the “memorized” information
${h^{\prime}}_{p}$ will be kept and then passed to the current fusion
operation with current input feature $f^{d}$. The preserved “memory”
$h^{\prime}_{p}$ and feature $f^{d}$ are concatenated together to perform a
linear transformation (convolution), and then activated by a $\tanh$ function.

$h^{c}_{p}=\tanh{\left({W}_{h}\left({f}^{d},{h^{\prime}_{p}}\right)\right)}$
(4)

$h^{c}_{p}$ acts similarly to the memory cell in the LSTM and helps the GRF
fusion block to remember long term information within the multi-stage fusion.

Selective Fusion $z\odot h_{p}$ : Indicates how much of the previous features
should be preserved. $(1-z)\odot h^{c}_{p}$: Indicates how much of the current
information $h^{c}_{p}$ should be added. Similar to the former, here $(1-z)$
forgets some unimportant information in $h^{c}_{p}$. Or, it can be viewed as a
choice of some information in $h^{c}_{p}$.

Combined with $f^{d}$ and ${h}_{p}$, the fusion result at step $p$ is,

${h}_{q}=z\odot{h}_{p}+\left(1-z\right)\odot{h}^{c}_{p}$ (5)

This operation ignores some information in previous hidden state $h_{p}$, and
adds some information from the current step. Update gate $z$ is equivalent to
the forget gate in LSTM, and $1-z$ is equivalent to the input gate in LSTM. In
this way, the forget gate $z$ and the input gate $(1-z)$ are linked. That is,
if the previous information is ignored with a weight of $z$, then the
information for the current input $h^{c}_{p}$ would be selected with a weight
of $(1-z)$. In our case, if the information in previous stage is depth feature
and current input is RGB feature, this enables the complementary information
to be effectively merged. Accordingly, the output of the current step
${f}^{GRF}_{q}={h}_{p}$ will also be passed to the next step.

Single-stage Fusion Module The bimodal information passes through multiple
DDRs for feature extraction, and each process of the DDR corresponds to a
stage. That is, single-stage fusion module has only one layer of DDR from RGB
and depth branch, and the fusion block is performed after the DDR. In
specific, the GRF module has two input features, $f^{d}$ and $f^{rgb}$. At
step $1$, the hidden state is initialised as mentioned above. We feed $f^{d}$
into GRF fusion block, and get the output $h_{1}$. Then at step $2$, we reuse
the same structure and the same parameters in the GRF fusion block. The input
hidden state is replaced by ${h}_{1}$, and the input is the features $f^{rgb}$
extracted from the RGB image. As shown in Figure 4, we use the red line with
an arrow to indicate the reuse of GRF fusion block at step $2$.

NYU | scene completion | semantic scene completion
---|---|---
method | prec. | recall | IoU |  rgb1ceil. |  rgb2floor |  rgb3wall |  rgb4win. |  rgb5chair |  rgb6bed |  rgb7sofa |  rgb8table |  rgb9tvs |  rgb10furn. |  rgb11objs. | avg.
single-stage GRFNet | 66.5 | 85.9 | 60.1 | 27.5 | 92.9 | 28.1 | 10.7 | 14.9 | 60.1 | 33.8 | 17.3 | 10.1 | 30.4 | 14.7 | 31.0
multi-stage GRFNet | 68.4 | 85.4 | 61.2 | 24.0 | 91.7 | 33.3 | 19.0 | 18.1 | 51.9 | 45.5 | 13.4 | 13.3 | 37.3 | 15.0 | 32.9
NYUCAD | scene completion | semantic scene completion
single-stage GRFNet | 88.4 | 89.1 | 79.7 | 50.0 | 91.4 | 56.4 | 18.7 | 41.3 | 56.8 | 52.7 | 33.5 | 16.3 | 45.2 | 30.0 | 44.8
multi-stage GRFNet | 87.2 | 91.0 | 80.1 | 50.3 | 91.8 | 58.1 | 18.4 | 42.7 | 60.6 | 52.8 | 34.6 | 11.5 | 46.6 | 30.8 | 45.3

Table 3: Results of single-stage GRFNet and multi-stage GRFNet on both NYU and
NYUCAD dataset.

Method (NYU) | GS | MM | prec. | recall | IoU | mIoU
---|---|---|---|---|---|---
Concatenate Fusion |  |  | 70.6 | 76.2 | 57.6 | 25.9
Sum Fusion |  |  | 67.6 | 79.4 | 57.6 | 25.7
Max Fusion |  |  | 67.6 | 79.4 | 57.5 | 25.6
Gated Fusion | ✓ |  | 70.8 | 77.5 | 58.6 | 27.6
LSTM Fusion | ✓ | ✓ | 68.0 | 82.3 | 59.6 | 28.3
GRF Fusion | ✓ | ✓ | 66.5 | 85.9 | 60.1 | 31.0
Method (NYUCAD) | GS | MM | prec. | recall | IoU | mIoU
Concatenate Fusion |  |  | 87.3 | 83.5 | 74.3 | 37.8
Sum Fusion |  |  | 81.4 | 89.3 | 74.3 | 37.7
Max Fusion |  |  | 81.9 | 87.8 | 73.3 | 36.5
Gated Fusion | ✓ |  | 82.1 | 91.3 | 76.0 | 40.2
LSTM Fusion | ✓ | ✓ | 83.5 | 91.3 | 77.5 | 41.4
GRF Fusion | ✓ | ✓ | 88.4 | 89.1 | 79.7 | 44.8

Table 4: Results of different single-stage fusion methods on the NYU and
NYUCAD dataset. GS denotes Gate Structure, and MM represents Memory Mechanism.
With IoU denotes the accuracy of semantic completion and mIoU denotes the
accuracy of semantic scene completion. Figure 5: Qualitative results on
NYUCAD. From left to right: Input RGB-D image, ground truth, results generated
by our GRFNet, DDR-SSC [25], and SSCNet [35]. Overall, our completed semantic
3D scenes are less cluttered and show a higher voxel class accuracy compared
to the others.

Multi-stage GRF Fusion Module Features extracted by the earlier-stage DDR are
at relatively low-level, while those by the later-stage DDR are at relatively
high-level regarding to the semantic meaning representation. For multi-stage
fusion, the features of the two modal data at each stage will be formed as a
sequence. Taking the $N$-stage RGB-D fusion as an example, the feature
sequence is
$F=\left(f^{d}_{1},f^{rgb}_{1},f^{d}_{2},f^{rgb}_{2},\cdots,f^{d}_{N},f^{rgb}_{N}\right)$.
Each feature tensor in $F$ will be fed into the GRF fusion block serially. The
GRU fusion block will be reused $2N$ times and all these fusion stages share
the same group of parameters. Different with the single-stage fusion module
which performs the multi-modal feature fusion at only one stage of the
network, multi-modal features are fused in multi-stages which covers both of
the high-level and low-level features. It is not only helpful to recover the
details of the scene, but also important to propagate information among
different stages.

Using the low-level feature to guide high-level feature is a common and
reasonable approach in computer vision community. The proposed GRF module can
preserves the complementary information and compensates the missing details.
Particularly, the color and texture details in RGB image as well as the
geometry and distance information in depth are complementary to each other.
The “gate” structure in the GRF fusion block controls the feature fusion by
learning weights (between 0 and 1). Moreover, for multi-stage fusion, the GRF
fusion block has the potential to manage the interleaved modalities. To be
specific, the bimodal information has gradually transformed into abstract
semantic information through the network, and the difference between their
distributions is gradually reduced. Due the above merits, we employ the multi-
stage GRF fusion module in our network. The effectiveness of multi-stage
fusion module are supported and reflected by our experiments in section 4.

###  3.4 Training Protocol

The loss function used in our training process is the softmax cross-entropy
loss, and it is performed on the unnormalized network outputs $y$:

$\mathcal{L}=-\sum_{c=0}^{N}{{w}_{c}{\hat{y}}_{i,c}\log{\left(\frac{{e}^{{y}_{ic}}}{\sum_{c^{\prime}}^{N}{{e}^{{y}_{ic^{\prime}}}}}\right)}}$
(6)

where ${\hat{y}}_{i,c}$ are the one-hot ground truth vectors, _i.e_.
${\hat{y}}_{i,c}=1$ if voxel $i$ is labeled by class $c$, otherwise
${\hat{y}}_{i,c}=0$. $N$ is the number of classes, and ${w}_{c}$ is the loss
weight, for balancing different classes, and the setting following SSCNet
[35]. To compute the loss function, we ignore all voxels outside the field of
view but include all voxels inside the view (empty, non-empty and occluded
voxels).

We train the network from scratch with the initial learning rate 0.01 which is
reduced by a factor of 0.1 after every ten epochs. We set the weight of empty
voxels ${w}_{0}$ to $0.05$ for data balancing and increase it by 0.05 for
every 40 training epochs. Our model is trained using the SGD optimizer with a
momentum of 0.9, weight decay of ${10}^{-4}$ and batch size is 4.

Please note, the order in which the modalities are fed into GRF fusion block
has been always fixed (fist Depth, then RGB) for all of the experiments,
however, according to our preliminary experiments, using different input order
(first RGB, then Depth) has a minor impact on the performance.

##  4 Experiments

###  4.1 Datasets and Metrics

Datasets We evaluate the proposed method and compare it with the state-of-the-
art methods on NYU [33] and NYUCAD [11] datasets. NYU consists of 1449 indoor
scenes, including 795 training samples and 654 testing samples. The RGBD
images of NYU are captured via a Kinect RGBD sensor, and the 3D semantic scene
completion labels are from Rock _et al_. ([32]). The annotations are fitted
into the scenes by CAD models. NYUCAD uses the depth maps generated from the
projections of the 3D annotations to reduce the misalignment of depths and the
annotations.

Metrics The primary evaluation metric is the voxel-level intersection over
union (IoU) between the predicted labels and ground-truth labels. For semantic
scene completion, the IoU is calculated for each category on both the observed
and occluded voxels. For scene completion, all non-empty classes are treated
as one category, IoU, precision, and recall of the binary predictions are
evaluated on the occupied voxels.

###  4.2 Comparisons with the State-of-the-art Methods

In the task of semantic scene completion, our GRFNet outperforms all existing
methods and achieves the start-of-the-art accuracy. The results on NYU [33]
and NYUCAD [11] are shown in Table 1 and Table 2, respectively. The GRFNet
improves the average IoU over DDR-SSC [25] by 2.5% on both NYU and NYUCAD
datasets.

The experiments demonstrate that the proposed fusion block effectively
utilizes multi-modality information. Since we focus on presenting a practical
multi-modal data fusion approach (GRF), we maintain the consistency of the
network structure for a fair comparison to prove that the improvement in
accuracy comes from the GRF fusion block. In specific, our network framework
is the same as DDR-SSC except for the fusion block.

###  4.3 Quantitative Analysis

Table 1 shows the quantitative results on NYU dataset [33] acquired by our
method and other state-of-the-art methods. Approaches of Lin _et al._ ([27])
and Geiger _et al._ ([14]) are traditional methods. SSCNet [35], EsscNet [43],
and DDR-SSC [25] are CNN-based approaches. Compared to the classical approach
SSCNet, the IoUs of GRFNet increase 6.1% and 8.2% for SC and SSC tasks,
respectively. In the SSC task, our GRFNet gets 6.2% higher accuracy than
EsscNet and achieves higher IoU in almost every category. SSCNet and EsscNet
only use depth information, while DDR-SSC uses a multi-stage fusion structure
to take advantage of the depth and RGB images. Our GRFNet also uses RGB-D
information and achieves a 2.5% higher average IoU than DDR-SSC. Regarding the
individual class accuracy, the IoUs for each category are also listed out in
Table 1.

As shown in Table 2, GRFNet achieves outstanding performance on NYUCAD dataset
as well. Specifically, compared to Zheng _et al_.’s( [44]) and Firman _et
al_.’s( [11]) methods, GRFNet significantly improves the accuracy in two
metrics. Since SSCNet only employs depth as input, the proposed GRFNet which
use RGB and depth information achieves much more accurate results. Although
TS3D [13] and DDR-SSC use both RGB and depth information, these methods only
adopt simple fusion strategy. On the contrary, GRFNet benefited from the novel
fusion block, to obtain 0.7% and 2.5% improvements compared to DDR-SSC, and
5.9% and 3.2% improvements compared to TS3D for SC and SSC tasks respectively.
In summary, our approach achieves higher accuracy on most indicators than
previous methods, especially the average IoU, which reflects the overall
performance.

###  4.4 Qualitative Analysis

Figure 5 visualizes results of the semantic scene completion generated by the
proposed GRFNet (c), DDR-SSC (d) and SSCNet (e). We mark the difference in
visual quality with a red dotted box for reference. As can be seen, compared
with both SSCNet and DDR-SSC, the scene completion results of our GRFNet are
much more abundant in detail and less error-prone. More visualization results
and analyses are provided in the supplemental materials.

###  4.5 Ablation Study

To study the effect of different components and design choices we perform an
ablation study. We choose DDR-SSC [25] as the baseline, which is the most
relevant work to the proposed GRFNet. Since our focus is the fusion strategy,
the GRF module will be analyzed in detail below.

Single-stage Fusion To verify the effectiveness of our GRF fusion module, we
compare the single-stage GRFNet with a variety of conventional fusion methods,
including Concatenate Fusion, Sum Fusion, Max Fusion, and Gated Fusion. For
better comparison, we replace the fusion block in the framework (as shown in
Figure 3) by the compared single-stage fusion methods. The results of the
comparison are shown in Table 4. We have two findings as following: 1) The
fusion strategy using the gate structure is better than the one without the
gate structure; 2) The memory mechanism can further enhance fusion effects.

As shown in Table 4, Sum Fusion, Max Fusion, and Concatenate Fusion achieve
similar performance. And they are significantly lower than the other three
modules in which contain adaptive selection mechanism. LSTM fusion and GRF
Fusion have a memory mechanism, but Gated Fusion does not; therefore, the
accuracy of the first two methods is better. LSTM is more complex and has more
parameters than GRF. However, GRF Fusion is still 2.7% and 3.4% more accurate
than LSTM on NYU and NYUCAD regarding to SSC accuracy, respectively.

Method | Prec. | Recall | IoU | mIoU
---|---|---|---|---
Sum Fusion(DDR-SSC) | 71.5 | 80.8 | 61.0 | 30.4
LSTM Fusion | 68.0 | 83.2 | 60.2 | 30.2
GRF Fusion | 68.4 | 85.4 | 61.2 | 32.9

Table 5: Results of different multi-stage fusion methods on NYU dataset. With
IoU represents the accuracy of scene completion, and mIoU denotes the accuracy
of semantic scene completion.

Multi-stage Fusion

1). Multi-stage Strategy In Table 3, we compare the performance of single-
stage GRFNet and the multi-stage GRFNet. Single stage-fusion can only fuse
information at one of the network stages. While multi-stage GRFNet can use
both the low-level and high-level information and get higher accuracy than the
single-stage version on both datasets.

2). Fusion Strategy In DDR-SSC [25], sum fusion is used to fuse the four
stages of features separately. The fusion results are cascaded and handed over
to subsequent networks for semantic label prediction. GRF module employs the
recurrent structure that uses the previous fusion results as the input for the
next fusion stage, hence the information for each stage can be combined
without additional cascading operations. As can been in Table 5, multi-stage
GRFNet gets 0.2% higher average IoU than DDR-SSC on SC and 2.5% higher on SSC.
And GRF fusion is 1% higher than LSTM fusion on SC and 2.7% higher on SSC.

Parameters and Flops of Different Fusion Stages

Fusion stages | Params [k] | FLOPs [G]
---|---|---
1 | 794.59 | 193.47
2 | 803.39 | 366.65
3 | 812.19 | 539.84
4 | 820.99 | 713.02

Table 6: Params and FLOPs of multi-stage GRFNets with different number of
fusion stages.

In Figure 6, parameters and FLOPs of our network with different fusion stages
are listed out. As can be seen, with the increasing of fusion stages,
parameters increase slightly, which mainly due to the reuse of GRF fusion
block, the only source for more parameters is the new added DDR block (for
both Depth and RGB channel). On the contrast, FLOPs increase dramatically,
which mainly come from the GRF fusion block and small portion come from DDR
blocks. In our implementation, GRF fusion block still employ 3D convolutions,
thus bring in relatively high compution costs. As we point out before, our
focus of this work is to provide a new strategy for fusing the two-modal data
in SSC, which can be improved by light-weight operations.

##  5 Conclusion

In this paper, we propose GRFNet with a novel gated recurrent fusion module to
fuse RGB and depth information. Different from the existing fusion strategies,
we emphasize the importance of the adaptive selectivity of information and the
memory mechanism within the fusion block. Moreover, we further extend the
single-stage GRFNet to a multi-stage version, which can fuse both low-level
and high-level feature at different stages. Our approach has significant
advantages over previous methods in multi-modal data fusion and achieves the
state-of-the-art performance in semantic scene completion. Extensive
comparison experiments and ablation studies verify the effectiveness of the
proposed method. In the future, one of our research interests would be to
consider making the proposed GRFNet light-weight, for instance, replacing the
3D convolution of GRF fusion block with DDR.

## References

  * [1] Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville.  Delving deeper into convolutional networks for learning video representations.  arXiv:1511.06432, 2015.
  * [2] Long Chen, Karl Francis, and Wen Tang.  Semantic augmented reality environment with material-aware physical interactions.  In ISMAR, pages 135–136. IEEE, 2017.
  * [3] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia.  Multi-view 3d object detection network for autonomous driving.  In CVPR, pages 1907–1915, 2017.
  * [4] Yanhua Cheng, Rui Cai, Zhiwei Li, Xin Zhao, and Kaiqi Huang.  Locality-sensitive deconvolution networks with gated fusion for rgb-d indoor semantic segmentation.  In CVPR, pages 3029–3037, 2017.
  * [5] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  arXiv:1406.1078, 2014.
  * [6] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio.  Attention-based models for speech recognition.  In NeurIPS, pages 577–585, 2015.
  * [7] Camille Couprie, Clément Farabet, Laurent Najman, and Yann LeCun.  Indoor semantic segmentation using depth information.  In ICLR, 2013.
  * [8] Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias Nießner.  Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans.  In CVPR, pages 4578–4587, 2018.
  * [9] Anh-Dzung Doan, Yasir Latif, Tat-Jun Chin, Yu Liu, Thanh-Toan Do, and Ian Reid.  Scalable place recognition under appearance change for autonomous driving.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9319–9328, 2019.
  * [10] Ekrem Emre Yurdakul and Yucel Yemez.  Semantic segmentation of rgbd videos with recurrent fully convolutional neural networks.  In ICCV, pages 367–374, 2017.
  * [11] Michael Firman, Oisin Mac Aodha, Simon Julier, and Gabriel J Brostow.  Structured prediction of unobserved voxels from a single depth image.  In CVPR, pages 5431–5440, 2016.
  * [12] Huazhu Fu, Dong Xu, and Stephen Lin.  Object-based multiple foreground segmentation in rgbd video.  TIP, 26(3):1418–1427, 2017.
  * [13] Martin Garbade, Johann Sawatzky, Alexander Richard, and Juergen Gall.  Two stream 3d semantic scene completion.  arXiv:1804.03550, 2018.
  * [14] Andreas Geiger and Chaohui Wang.  Joint 3d object and layout inference from a single rgb-d image.  In GCPR, pages 183–195, 2015.
  * [15] Yanrong Guo and Tao Chen.  Semantic segmentation of rgbd images based on deep depth regression.  Pattern Recognition Letters, 109:55–64, 2018.
  * [16] Saurabh Gupta, Ross Girshick, Pablo Arbeláez, and Jitendra Malik.  Learning rich features from rgb-d images for object detection and segmentation.  In ECCV, pages 345–360. Springer, 2014.
  * [17] Xiaoguang Han, Zhen Li, Haibin Huang, Evangelos Kalogerakis, and Yizhou Yu.  High-resolution shape completion using deep neural networks for global structure and local geometry inference.  In ICCV, pages 85–93, 2017.
  * [18] Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers.  Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture.  In ACCV, pages 213–228. Springer, 2016.
  * [19] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [20] Jian-Fang Hu, Wei-Shi Zheng, Jiahui Pan, Jianhuang Lai, and Jianguo Zhang.  Deep bilinear learning for rgb-d action recognition.  In ECCV, pages 335–351, 2018.
  * [21] Earnest Paul Ijjina and Krishna Mohan Chalavadi.  Human action recognition in rgb-d videos using motion sequence information and deep learning.  Pattern Recognition, 72:504–516, 2017.
  * [22] Le Kang, Peng Ye, Yi Li, and David Doermann.  Convolutional neural networks for no-reference image quality assessment.  In CVPR, pages 1733–1740, 2014.
  * [23] Christian Kerl, Jürgen Sturm, and Daniel Cremers.  Dense visual slam for rgb-d cameras.  In IROS, pages 2100–2106. IEEE, 2013.
  * [24] Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush.  Character-aware neural language models.  In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  * [25] Jie Li, Yu Liu, Dong Gong, Qinfeng Shi, Xia Yuan, Chunxia Zhao, and Ian Reid.  Rgbd based dimensional decomposition residual network for 3d semantic scene completion.  In CVPR, pages 7693–7702, 2019.
  * [26] Zhen Li, Yukang Gan, Xiaodan Liang, Yizhou Yu, Hui Cheng, and Liang Lin.  Lstm-cf: Unifying context modeling and fusion with lstms for rgb-d scene labeling.  In ECCV, pages 541–557. Springer, 2016.
  * [27] Dahua Lin, Sanja Fidler, and Raquel Urtasun.  Holistic scene understanding for 3d object detection with rgbd cameras.  In ICCV, pages 1417–1424, 2013.
  * [28] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji.  Bilinear cnn models for fine-grained visual recognition.  In ICCV, pages 1449–1457, 2015.
  * [29] Yan Lu and Dezhen Song.  Robust rgb-d odometry using point and line features.  In ICCV, pages 3934–3942, 2015.
  * [30] Seong-Jin Park, Ki-Sang Hong, and Seungyong Lee.  Rdfnet: Rgb-d multi-level residual feature fusion for indoor semantic segmentation.  In ICCV, pages 4980–4989, 2017.
  * [31] Xiaofeng Ren, Liefeng Bo, and Dieter Fox.  Rgb-(d) scene labeling: Features and algorithms.  In CVPR, pages 2759–2766. IEEE, 2012.
  * [32] Jason Rock, Tanmay Gupta, Justin Thorsen, JunYoung Gwak, Daeyun Shin, and Derek Hoiem.  Completing 3d object shape from one depth image.  In CVPR, pages 2484–2493. IEEE, 2015.
  * [33] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In ECCV, pages 746–760. Springer, 2012.
  * [34] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In NeurIPS, pages 568–576, 2014.
  * [35] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser.  Semantic scene completion from a single depth image.  In CVPR, pages 190–198, 2017.
  * [36] Maryam Sultana, Arif Mahmood, Sajid Javed, and Soon Ki Jung.  Unsupervised rgbd video object segmentation using gans.  arXiv:1811.01526, 2018.
  * [37] Jacob Varley, Chad DeChant, Adam Richardson, Joaquín Ruales, and Peter Allen.  Shape completion enabled robotic grasping.  In IROS, pages 2442–2447, 2017.
  * [38] Jinghua Wang, Zhenhua Wang, Dacheng Tao, Simon See, and Gang Wang.  Learning common and specific features for rgb-d semantic segmentation with deconvolutional networks.  In ECCV, pages 664–679. Springer, 2016.
  * [39] Ningning Wang and Xiaojin Gong.  Adaptive fusion for rgb-d salient object detection.  arXiv:1901.01369, 2019.
  * [40] Thomas Whelan, Hordur Johannsson, Michael Kaess, John J Leonard, and John McDonald.  Robust real-time visual odometry for dense rgb-d mapping.  In ICRA, 2013.
  * [41] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici.  Beyond short snippets: Deep networks for video classification.  In CVPR, pages 4694–4702, 2015.
  * [42] Haokui Zhang, Ying Li, Peng Wang, Yu Liu, and Chunhua Shen.  Rgb-d based action recognition with light-weight 3d convolutional networks.  arXiv preprint arXiv:1811.09908, 2018.
  * [43] Jiahui Zhang, Hao Zhao, Anbang YaoE, Yurong Chen, Li Zhang, and Hongen LiaoE.  Efficient semantic scene completion network with spatial group convolution.  In ECCV, pages 733–749, 2018.
  * [44] Bo Zheng, Yibiao Zhao, Joey C Yu, Katsushi Ikeuchi, and Song-Chun Zhu.  Beyond point clouds: Scene understanding by reasoning geometry and physics.  In CVPR, pages 3127–3134, 2013.

##  6 More Details of GRFNet

###  6.1 Detailed Architectures

The details of the proposed network structure are shown in Table 7. PWConv
represents the point-wise convolution, and it is used to adjust the number of
channels of the feature map. The down-sample layer in our network is composed
of a max-pooling layer and a convolution layer with stride set as 2. The
outputs of the two layers are concatenated before fed into the subsequent
layers.

Module | Operation |  | Output Size
---
2D: $Height\times Width\times Channels$
3D: $Depth\times Height\times Width\times Channels$
Kernel | Stride | Dilation
Feature Extractor | PWConv | $640\times 480\times 8$ | 1 | 1 | 1
2D DDR | $640\times 480\times 8$ | 3 | 1 | 1
2D DDR | $640\times 480\times 8$ | 3 | 1 | 1
2D - 3D Projection | $240\times 144\times 240\times 8$ | - | - | -
Down-sample | $120\times 72\times 120\times 16$ | 3 | 2 | 1
3D DDR | $120\times 72\times 120\times 16$ | 3 | 1 | 1
Down-sample | $60\times 36\times 60\times 64$ | 3 | 2 | 1
3D DDR | $60\times 36\times 60\times 64$ | 3 | 1 | 1
Feature Fusion | GRF stage 1 | $60\times 36\times 60\times 64$ | 3 | 1 | 1
3D DDR | $60\times 36\times 60\times 64$ | 3 | 1 | 2
GRF stage 2 | $60\times 36\times 60\times 64$ | 3 | 1 | 1
3D DDR | $60\times 36\times 60\times 64$ | 3 | 1 | 3
GRF stage 3 | $60\times 36\times 60\times 64$ | 3 | 1 | 1
3D DDR | $60\times 36\times 60\times 64$ | 3 | 1 | 5
GRF stage 4 | $60\times 36\times 60\times 64$ | 3 | 1 | 1
LW-ASPP | PWConv | $60\times 36\times 60\times 64$ | 1 | 1 | 1
3D DDR | $60\times 36\times 60\times 64$ | 3 | 1 | 3
3D DDR | $60\times 36\times 60\times 64$ | 3 | 1 | 6
3D DDR | $60\times 36\times 60\times 64$ | 3 | 1 | 9
GlobalAvgPool | $60\times 36\times 60\times 64$ | - | - | -
Concatenate | $60\times 36\times 60\times 320$ | - | - | -
Output | PWConv | $60\times 36\times 60\times 160$ | 1 | 1 | 1
PWConv | $60\times 36\times 60\times 12$ | 1 | 1 | 1
ArgMax | $60\times 36\times 60\times 12$ | - | - | -

Table 7: The details of the proposed (GRFNet) network architecture. Including
module name, layer operation, output size, kernel size, stride and dilation.

###  6.2 Dimensional Decomposition Residual Block

In Table 9, we show the details of the Dimensional Decomposition Residual
(DDR) [25] block. DDR ($k,w,s,d$) denotes the DDR block with the kernel size
$k$, the output channels of feature maps $w$, the stride $s$ and the dilation
rate $d$. DDRConv represents the proposed DDR convolution within a DDR block.

The most significant advantage of using DDR in 3D tasks is the reduction of
the number of parameters and the amount of calculation. In a DDR block, the 3D
convolution with the kernel size ${k}\times{k}\times{k}$ is decomposed into
three consecutive layers with filter size $1\times 1\times k$, $1\times
k\times 1$ and $k\times 1\times 1$. The most common value for $k$ is $3$. The
computational costs of the original block and the DDR block are proportional
to $c^{in}\times c^{out}\times k\times k\times k$ and $c^{in}\times
c^{out}\times(k+k+k)$, where $c^{in}$ and $c^{out}$ are the numbers of input
and output channels.

We assume $c^{in}=c^{out}=w$ and ignore bias, then the parameter quantity
changes from $w^{2}\times k^{3}$ to $w^{2}\times 3k$. The bottleneck structure
within the DDR block further reduces its parameter amount and calculation
cost.

###  6.3 2D to 3D Projection

Each point in depth can be projected to a position in the 3D space. We
voxelize this entire 3D space with meshed grids to obtain a 3D volume. In the
projection layer, every feature tensor is projected into the 3D volume at the
location corresponding to its position in depth. With the feature projection
layer, the 2D feature maps extracted by the 2D CNN are converted to a view-
independent 3D feature volume.

| scene completion | semantic scene completion
---|---|---
method | prec. | recall | IoU |  rgb1ceil. |  rgb2floor |  rgb3wall |  rgb4win. |  rgb5chair |  rgb6bed |  rgb7sofa |  rgb8table |  rgb9tvs |  rgb10furn. |  rgb11objs. | avg.
multi-stage MaxFusion | 82.5 | 91.8 | 76.8 | 48.2 | 91.4 | 53.3 | 16.5 | 36.5 | 54.6 | 50.5 | 29.3 | 11.8 | 41.2 | 25.0 | 41.7
multi-stage SumFusion | 88.7 | 88.5 | 79.4 | 54.1 | 91.5 | 56.4 | 14.9 | 37.0 | 55.7 | 51.0 | 28.8 | 9.2 | 44.1 | 27.8 | 42.8
multi-stage GRFNet | 87.2 | 91.0 | 80.1 | 50.3 | 91.8 | 58.1 | 18.4 | 42.7 | 60.6 | 52.8 | 34.6 | 11.5 | 46.6 | 30.8 | 45.3

Table 8: Results of multi-stage networks with different fusion blocks on
NYUCAD dataset. As can be seen, compared with the results based MaxFusion and
SumFusion blocks, GRFNet (with GRF fusion block) achieves better accuracy both
in scene completion and semantic scene completion tasks. Which may due to the
advantage of ’recurrent’ property of the proposed GRF fusion block.

Operation | Kernel | Channels | Stride | Dilation
---|---|---|---|---
PWConv | $1\times 1\times 1$ | $w/4$ | $1$ | $1$
DDRConv | $1\times 1\times k$ | $w/4$ | $s$ | $d$
DDRConv | $1\times k\times 1$ | $w/4$ | $s$ | $d$
DDRConv | $k\times 1\times 1$ | $w/4$ | $s$ | $d$
PWConv | $1\times 1\times 1$ | $w$ | $1$ | $1$

Table 9: Details of the DDR ($k,w,s,d$) block. $k$ is the kernel size, $w$ is
the output channels of the feature map, $s$ is the stride and $d$ is the
dilation rate of the convolution.

##  7 Recurrent Property of GRFNet

In Figure 6 and Figure 7, network structures with different fusion strategies
are listed out. Specifically, Figure 6 is the network structure with GRF
fusion block, which means different stages will share the same GRF fusion
block, and it has the ‘recurrent’ property between different stages, that is
the low-level fusion could be part of guidance and contribute the following
the high-level stages. On the contrary, Figure 7 given the network workflow
which use other fusion blocks that does not have the ’recurrent’ property,
including Sum Fusion, Average Fusion, Bilinear Fusion, Concatenation Fusion,
Max Fusion, Transformation Fusion. And as denoted with the blue arrow,
different fusion blocks are separated with each other, until they are concated
together to be feded to the light-weight ASPP module.

As can be seen in Table 8, results of networks with different fusion
strategies are listed out. The first two rows are the results of networks
which employ the MaxFusion and SumFusion, and last row are the results of
GRFNet. For both of the scene completion and semantic scene completion tasks,
GRFNet boost the performance significantly. We supect that is because the
’recurrent’ property of the proposed GRF fusion block.

Figure 6: Network workflow with GRF fusion block Figure 7: Network workflow
with other fusion blocks

##  8 More Qualitative Results

Figure 8: Qualitative results on NYUCAD. From left to right: Input RGB-D
image, ground truth, results obtained by our proposed GRFNet, results obtained
by DDR-SSC [25] and SSCNet [35]. Overall, our completed semantic 3D scenes are
less cluttered and show a higher voxel-wise accuracy compared to DDR-SSC and
SSCNet.

Figure 8 shows some visualized results on NYUCAD [11] dataset. As shown in
Figure 8, the proposed GRFNet achieves better results than DDR-SSC [25] and
SSCNet [35], and is much more accurate in shape completion and semantic
segmentation. The color information is beneficial for the prediction of
semantic labeling. In Figure 8, the prediction of furniture in the second,
third, and fifth rows is more accurate than the method using only depth.

When an object consists of several parts with various appearances, the RGB
information may result in inconsistencies in the semantics of the local areas.
For example, in the first row of Figure 8, most of the walls are white, while
the right part of the wall is brown. This makes it difficult for the network
to predict the semantics of that brown wall. In this case, the geometric
information contained in the depth image can effectively eliminate ambiguity
and provide a reasonable inference.

Besides, when different objects are in similar colors, depth can provide
adequate information for distinguishing objects. In the fourth row of Figure
8, the chair is very similar to the background in the RGB image, but it can be
easily distinguished in depth. Therefore, in general, the fusion of RGB-D is
critical, and our GRFNet can effectively improve the accuracy of SSC.


# Omicron (B.1.1.529): Infectivity, vaccine breakthrough, and antibody
resistance.

Jiahui Chen1, Rui Wang1, Nancy Benovich Gilby2 and Guo-Wei Wei1,3,4111
Corresponding author. Email<EMAIL_ADDRESS>
1 Department of Mathematics,
Michigan State University, MI 48824, USA.
1 Spartan Innovations,
325 East Grand River Ave., Suite 355,
East Lansing, MI 48823 USA.
3 Department of Electrical and Computer Engineering,
Michigan State University, MI 48824, USA.
4 Department of Biochemistry and Molecular Biology,
Michigan State University, MI 48824, USA.

###### Abstract

The latest severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)
variant Omicron (B.1.1.529) has ushered panic responses around the world due
to its contagious and vaccine escape mutations. The essential infectivity and
antibody resistance of the SARS-CoV-2 variant are determined by its mutations
on the spike (S) protein receptor-binding domain (RBD). However, a complete
experimental evaluation of Omicron might take weeks or even months. Here, we
present a comprehensive quantitative analysis of Omicron’s infectivity,
vaccine-breakthrough, and antibody resistance. An artificial intelligence (AI)
model, which has been trained with tens of thousands of experimental data
points and extensively validated by experimental data on SARS-CoV-2, reveals
that Omicron may be over ten times more contagious than the original virus or
about twice as infectious as the Delta variant. Based on 132 three-dimensional
(3D) structures of antibody-RBD complexes, we unveil that Omicron may be twice
more likely to escape current vaccines than the Delta variant. The Food and
Drug Administration (FDA)-approved monoclonal antibodies (mAbs) from Eli Lilly
may be seriously compromised. Omicron may also diminish the efficacy of mAbs
from Celltrion and Rockefeller University. However, its impact to Regeneron
mAb cocktail appears to be mild.

Keywords: COVID-19, SARS-CoV-2, Omicron, infectivity, antibody-resistance,
vaccine breakthrough,

##  1 Introduction

On November 26, 2021, the World Health Organization (WHO) announced a new
severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) variant Omicron
(B.1.1.529), as a variant of concern (VOC). This variant carries an unusually
high number of mutations, 32, on the spike (S) protein, the main antigenic
target of antibodies generated by either infections or vaccination. In
comparison, the devastating Delta variant has only 5 S protein mutations,
which posed a high potential global risk and has spread internationally.
Therefore, the “panic button” has been pushed in several cases worldwide, and
many countries have enacted travel restrictions to prevent the rapid spread of
the Omicron variant.

The mutations on the Omicron variant are widely distributed on multiple
proteins of SARS-CoV-2 such as NSP3, NSP4, NSP5, NSP6, NSP12, NSP14, S
protein, envelope protein, membrane protein, and nucleocapsid protein. The
focus is the mutations on the S protein receptor-binding domain (RBD) for the
potential impact on infectivity and antibody resistance caused by this new
variant. This is due to the fact that the RBD located on the S protein
facilitates the binding between the S protein and the host angiotensin-
converting enzyme 2 (ACE2). Such S-ACE2 binding helps the SARS-CoV-2 enter the
host cell and initiates the viral infection process. Several studies have
shown that the binding free energy (BFE) between the S RBD and the ACE2 is
proportional to the viral infectivity [1, 2, 3, 4, 5]. As such, an antibody
that binds strongly to the RBD would directly neutralize the virus [6, 7, 8].
Indeed, many RBD binding antibodies are generated by the human immune response
to infection or vaccination. Monoclonal antibodies (mABs) targeting the S
protein, particularly the RBD, are designed to treat viral infection. As a
result, any mutation on the S protein RBD would cause immediate concerns about
the efficacy of existing vaccines, mAbs and the potential of reinfection.

The global panic brought by the emergence of the Omicron variant drives the
scientific community to immediately investigate how much this new variant
could undermine the existing vaccines and mAbs. However, relatively reliable
experimental results from experimental labs will take a few weeks to come out.
Therefore, an efficient and reliable in-silico analysis is imperative and
valuable for such an urgent situation. Here, we present a comprehensive
topology-based artificial intelligence (AI) model called TopNetmAb [9, 10] to
predict the BFE changes of S and ACE2/antibody complexes induced by mutations
on the S RBD of the Omicron variant. The positive BFE change induced by a
specific RBD mutation indicates its potential ability to strengthen the
binding of an S protein-ACE2/antibody complex, while a negative BFE change
suggests a likely capacity to reduce the binding strength of an S protein-
ACE2/antibody complex.

The TopNetmAb model that we proposed has been extensively validated over the
past 1.9 years [10, 11]. Initially, in early 2020, we applied our TopNetmAb
model to successfully predict that residues 452 and 501 ”have high chances to
mutate into significantly more infectious COVID-19 strains” [9]. Such findings
have been confirmed due to the emergency of multiple variants such as Alpha,
Beta, Gamma, Delta, Theta, Lambda, Mu, and Omicron that carry L452R/Q and
N501Y mutations. In April 2021, we provided a list of 31 RBD mutations that
may weaken most of the binding to antibodies, such as W353R, I401N, Y449D,
Y449S, P491R, P491L, Q493P [10]. Notably, experimental results have also shown
that mutations at residues Y449, E484, Q493, S494, Y505 might enable the virus
to escape antibodies [12]. Meanwhile, in the same work, we also revealed that
variants found in the United Kingdom and South Africa in late 2020 may
strengthen the binding of the RBD-ACE2 complex, which is consistent with the
experimental results [13]. Later on, we provided a list of most likely vaccine
escape RBD mutations predicted by TopNetmAb, such as S494P, Q493L, K417N,
F490S, F486L, R403K, E484K, L452R, K417T, F490L, E484Q, and A475S [14], and
mutations such as S494P, K417N, E484K/Q, L452R are all detected in the
variants of concern and variants of interest denounced by WHO. Last but not
least, the correlation between the experimental deep mutational data [15] and
our AI-predicted RBD-mutation-induced BFE changes for all possible 3686 RBD
mutations on the RBD-ACE2 complex is 0.7, which indicates the reliability of
our TopNetmAb model predictions [11].

This work aims to analyze how the RBD mutations on the Omicron variant will
affect the viral infectivity and efficacy of existing vaccines and antibody
drugs. Fifteen Omicron RBD mutations, including S371L, S373P, S375F, K417N,
N440K, G446S, S477N, T478K, E484A, Q493R, G496S, N501Y, and Y505H, are studied
in this work. Additionally, three-dimensional (3D) structures of RBD-ACE2
complex and 132 antibody-RBD complexes, including many mAbs, are examined to
understand the impacts of Omicron RBD mutations. We reveal that Omicron may be
over ten times more contagious than the original SARS-CoV-2 and about twice as
infectious as the Delta variant, mainly due to its RBD mutations N440K, T478K,
and N501Y. Additionally, Omicron has a high potential to disrupt the binding
of most 132 antibodies with the S protein, mainly due to its RBD mutations
K417N, E484A, and Y505H, indicating its stronger vaccine-breakthrough
capability than the Delta variant. We have also unveiled that Omicron may
seriously reduce the efficacy of the Eli Lilly mAb cocktail because of Omicron
RBD mutations K417N, E484A, and Q493R. Celltrion antibody Regdanvimab may be
disrupted by Omicron RBD mutations E484A, Q493R, and Q498R. Omicron RBD
mutation E484A may also disrupt Rockefeller University mAbs. However,
Omicron’s impact on the Regeneron mAb cocktail is predicted to be mild.

##  2 Results

###  2.1 Infectivity

The Infectivity of SARS-CoV-2 is mainly determined by the binding affinity of
the ACE2 and RBD complex, although the furin cleavage site plays a crucial
role as well [16]. Omicron has three mutations at the furin cleavage site and
15 mutations on the RBD, suggesting a significant change in its infectivity.
Due to natural selection, the virus enhances its evolutionary advantages at
the RBD either by mutations to strengthen the ACE2-RBD binding affinity or by
mutations to escape antibody protection [17]. Since the virus has optimized
its Infectivity in human cells, one should not expect a dramatic increase in
the viral infectivity by any single mutation. An effective infection pathway
is for the virus to have multiple RBD mutations to accumulatively enhance its
infectivity, which appears to be the case for Omicron.

Figure 1: Illustration of the Omicron RBD and ACE2 interaction, RBD mutation-
induced BFE changes. a. The 3D structure of the ACE2 and RBD complex. Omicron
mutation sites are labeled. b. Omicron mutation-induced BFE changes. Positive
changes strengthen the binding between ACE2 and S protein, while negative
changes weaken the binding. c. A comparison of predicted mutation-induced BFE
changes for few variants.

This work analyzes the infectivity of Omicron by examining the BFE changes of
the ACE2 and S protein complex induced by 15 Omicron RBD mutations. Figure 1a
illustrates the binding complex of ACE2 and S protein RBD. Most of the RBD
mutations are located near the binding interface of ACE2 and RBD, except for
mutations G339D, S371L, S373P, and S375F. Omicron-induced BFE changes are
depicted in Figure 1b. Overall, mutations significantly increase the BFE
changes, which strengthens the binding affinity of the ACE2-RBD complex and
makes the variant more infectious. This result indicates that Omicron appears
to have followed the infectivity-strengthening pathway of natural selection
[18].

The infectivity-strengthening mutations N440K, T478K, and N501Y enhance the
BFEs by 0.62, 1.00, and 0.55 kcal/mol, respectively. Among them, T478K is one
of two RBD mutations in the Delta variant, while N501Y is presented on many
prevailing variants, including Alpha, Beta, Gamma, Theta, and Mu. Notably,
mutation Y505H induces a small negative BFE change of -0.20kcal/mol. All other
mutations, particular those four mutations that are far away from the ACE2 and
RBD binding interface, cause little or no BFE changes. Figure 1c gives a
comparison of Omicron with a few other variant. Overall, the accumulated BFE
change is 2.60kcal/mol, suggesting a 13-fold increase in the viral
Infectivity. In comparison, Omicron is about 2.8 times as infectious as the
Delta (i.e., BFE change: 1.57kcal/mol).

###  2.2 Vaccine breakthrough

Vaccination has been proven to be the most effective means for COVID-19
prevention and control. There are four types of vaccines, i.e., virus
vaccines, viral-vector vaccines, DNA/RNA vaccines, and protein-based vaccines
[19]. Essentially, the current COVID-19 vaccines in use mainly target to the S
protein [20]. The 32 amino acid changes, including three small deletions and
one small insertion in the spike protein, suggest that Omicron may be induced
by vaccination. As a result, these mutations may dramatically enhance the
variant’s ability to evade current vaccines.

In general, it is essentially impossible to accurately characterize the full
impact of Omicron’s S protein mutations on the current vaccines in the world’s
populations. First, different types of vaccines may lead to different immune
responses from the same individual. Additionally, different individuals
characterized by race, gender, age, and underlying medical conditions may
produce different sets of antibodies from the same vaccine. Moreover, the
reliability of statistical analysis over populations may be limited because of
the inability to fully control various experimental conditions.

This work offers a molecule-based data-driven analysis of Omicron’s impact on
vaccines through a library of 132 known antibody and S protein complexes. We
evaluate the binding free energy changes induced by 15 RBD mutations on these
complexes to understand the potential impact of Omicron’s RBD mutations to
vaccines. To ensure reliability, our study does not include a few known
antibody-S protein complexes that are far away from the RBD, such as those in
the N-terminal domain (NTD), due to limited experimental data in our antibody
library [10, 11].

Figure 2: Illustration of Omicron mutation-induced BFE changes of 132
available antibody and RBD complexes. Positive changes strengthen the binding,
while negative changes weaken the binding. a Heat map for 8 antibodies and RBD
complexes in various stages of drug development. Gray color stands for no
predictions due to incomplete structures. b1-b2 Heat map for 124 antibodies
and RBD complexes. c The distribution (counts) of accumulated BFE changes
induced by Omicron mutations for 132 antibody and RBD complexes. Overall,
there are more complexes that are weakened upon Omicron RBD mutations that
complexes that are strengthened. d The distribution (counts) of accumulated
BFE changes induced by Delta mutations for 132 antibody and RBD complexes. e
The numbers (counts) of antibody and RBD complexes regarded as disrupted by
Omicron under different thresholds ranging from 0 kcal/mol, -0.3 kcal/mol, to
$<$-3 kcal/mol. f The numbers (counts) of antibody and RBD complexes regarded
as disrupted by Delta under different thresholds ranging from 0 kcal/mol, -0.3
kcal/mol, to $<$-2.4 kcal/mol.

Figures 2a, b1, and b2 depict the Omicron RBD mutation-induced BFE changes of
132 known antibody and RBD complexes. Overall, Omicron RBD mutations can
significantly change the binding pattern of known antibodies. Positive changes
strengthen the binding between antibody and RBD complexes, while negative
changes weaken the binding. In the color bar, the largest negative change is
more significant than the largest positive change, indicating more severe
disruptive impacts. In general, there are more negative BFE changes than
positive ones, as shown in Figure 2c, indicating the Omicron mutations favor
the escape of current vaccines. In contrast, Delta’s distribution focuses on a
smaller domain as shown in Figure 2d.

Among 15 RBD mutations, K417N, also part of the Beta variant that originated
in South Africa, causes the most significant disruption of known antibodies.
Notably, E484A is another mutation that leads to overwhelmingly disruptive
effects to many known antibodies. It is worthy of mentioning that most of
E484A’s disruptive effects are complementary to those of K417N, which makes
Omicron more effective in vaccine breakthroughs. The third disruptive mutation
is Y505H. It is also able to weaken many known antibody and RBD complexes.

Mutation G339D creates a mild impact on various antibody-RBD complexes. One of
the reasons is that it locates pretty far away from the binding interfaces of
most known antibodies. Its change from a non-charged amino acid to a
negatively charged amino acid induces mostly favorable bindings among many
antibody-RBD complexes. S371L, S373P, and S375F are other mutations that have
mild impacts due to their locations.

For a comparison, ACE2 is also included in Figure 2. The impact of Omicron on
ACE2 is significantly weak, indicating the SARS-CoV-2 has already optimized
its binding with ACE2, and there is a relatively limited potential for the
virus to improve its infectivity. However, due to the increase in the
vaccination rate, variants can become more destructive to vaccines in years to
come [21].

It becomes very subtle to judge whether a mutation would disrupt an antibody
and RBD complex as Omicron involves multiple RBD mutations, which may generate
multiple cancellations for each antibody and RBD complex over different
mutations. Previously, we have used -0.3 kcal/mol as a threshold to judge
whether a mutation disrupts an antibody and RBD complex, which would give us a
total of 119 disrupted antibody and RBD complexes as shown in Figure 2e,
suggesting a rate of 0.9 (i.e., 119/132) for potential vaccine breakthrough.
As a comparison, Delta has 46 counts and a rate of 0.35 (46/132) in a similar
estimation as shown in Figure 2f. One would have 108 and 30 disrupted antibody
and RBD complexes respectively for Omicron and Delta if the threshold is
increased to -0.6kcal/mol. Only two antibodies and RBD complexes have smaller
than -3 kcal/mol mutation-induced BFE changes. If the cancellations between
positive and negative changes are considered, we would have 77 complexes that
have accumulated BFE changes smaller than -0.3 kcal/mol, suggesting an
antibody disruption rate of 0.58 (i.e., 77/132) for Omicron, compared to 0.28
(37/132) for Delta. As mentioned earlier, the actual rate of Omicron-induced
vaccine breakthrough might never be properly determined for the world
population.

Figure 2a gives a separated plot of the impacts of Omicron on a few mAbs.
Similarly, there are dramatic reductions in their efficacy. A more specific
discussion is given in the next section.

###  2.3 Antibody resistance

The assessment of Omicron’s mutational threats to FDA-approved mAbs and a few
other mAbs in clinical development is of crucial importance. Our AI-based
predictions of similar threats from other variants, namely Alpha, Beta, Gamma,
Delta, Epsilon, and Kappa, have shown an excellent agreement with experimental
data [11]. In this section, we focus on a few mAbs, specifically, mAbs from
Eli Lilly (LY-CoV016 and LY-CoV555) and Regeneron (REGN10933, REGN10987, and
REGN10933/10987), one mAb from Celltrion (CT-P59), two mAbs from the
Rockefeller University (C135 and C144). Among them, mAbs from Eli Lilly and
Regeneron have had FDA approval. In addition, Celltrion’s COVID-19 antibody
treatment had the EU drug agency’s recommendation in November 2021.
Rockefeller University’s mAbs are still in clinical trials.

Figure 3: Illustration of the Omicron RBD and Eli Lilly antibody interaction
and RBD mutation-induced BFE changes. a The 3D structure of the ACE2 and Eli
Lilly antibody complex. LY-CoV555 (PDB ID: 7KMG[22]) and LY-CoV016 (PDB ID:
7C01[23]) overlap on the S protein RBD. b Omicron mutation-induced BFE changes
for the complex of RBD and LY-CoV016. c Omicron mutation-induced BFE changes
for the complex of RBD and LY-CoV555.

####  2.3.0.1 Eli Lilly mAbs

Eli Lilly mAb LY-CoV555 (PDB ID: 7KMG[22]) is also known as Bamlanivimab and
is used in combination with LY-CoV016 (aka Etesevimab, PDB ID: 7C01[23]).
Antibody LY-CoV016 is isolated from patient peripheral blood mononuclear cells
convalescing from COVID-19. It was optimized based on the SARS-CoV-2 virus.
The interaction of Eli Lilly mAbs with the S protein RBD is depicted in Figure
3a. Clearly, LY-CoV555 has a competing relationship with LY-CoV016, which
might complicate our predictions slightly. In this work, we carry out the
analysis of Eli Lilly mAbs separately.

Omicron mutation-induced BFE changes for antibody LY-CoV016 and RBD complex is
given in Figure 3b. It appears that LY-CoV555 was optimized with respect to
the original S protein but is sensitive to mutations. This complex may be
weakened by K417N and N501Y as predicted in our earlier work [11]. New
mutation Y505H may also reduce LY-CoV016’s efficacy. Overall, the complex may
be significantly weakened by Omicron, leading to the efficacy reduction of
Etesevimab.

The predicted BFE changes of LY-CoV555 are shown in Figure 3c. Mutation E484A
induces a negative BFE change of -2.79 kcal/mol for the LY-CoV555 and RBD
complex. The BFE change may translate into a dramatic efficacy reduction of 16
times for LY-CoV555, making it less competitive with ACE2 as most Omicron
mutations strengthen the S protein and ACE2 binding. Similarly, Q493R may also
reduce the efficacy by about 5 times. However, G496S may enhance the binding
of the complex by 2.6 times. The impacts of other mutations are mild.
Therefore, Omicron is expected to reduce LY-CoV555 efficacy significantly. A
previous study indicated that LY-CoV555 is prone to the E484K mutation
presented in Beta and Gamma variants, for which the Eli Lilly mAb cocktail was
taken off the market for many months in 2021.

Although LY-CoV555 and LY-CoV016 might slightly complement, they are both
prone to Omicron mutation-induced efficacy reduction. We predict that the Eli
Lilly mAb cocktail would be retaken off the market had Omicron become a
prevailing variant in the world.

Figure 4: Illustration of the Omicron RBD and Regeneron antibody interaction
and RBD mutation-induced BFE changes. a The 3D structure of the ACE2 and
Regeneron antibody complex. REGN10987 and REGN10933 do not overlap on the S
protein RBD (PDB ID: 6XDG[24]). b Omicron mutation-induced BFE changes for the
complex of RBD and REGN10933. c Omicron mutation-induced BFE changes for the
complex of RBD and REGN10987. d Omicron mutation-induced BFE changes for the
complex of RBD, REGN10933, and REGN10987.

####  2.3.0.2 Regeneron mAbs

Regeneron mAbs REGN10933 and REGN10987 (aka Casirivimab and Imdevimab,
respectively) are an FDA-approved antibody cocktail (PDB ID: 6XDG[24]) against
COVID-19. Their 3D structure in complex with the S protein RBD is depicted in
Figure 4a. Unlike the Eli Lilly mAb cocktail, the Regeneron mAbs do not
overlap each other and bind to different parts of the RBD. Our 3D alignment
shows that the antibody REGN10987 does not directly compete with ACE2 [11],
which suggests that REGN10933 plays a more critical role in its ability to
neutralize the virus directly.

Figure 4b plots our AI predicted BFE changes of the REGN10987-RBD complex.
There are mixed responses to various Omicron mutations. Although G446K and
K417N induce a negative BFE change, many other mutations enhance the binding
of the complex. We expect the overall binding strengthening effect.

Omicron-induced BFE changes of the REGN10933-RBD complex is given in Figure
4c. Apparently, K417N and E484A induce BFE changes of -1.08 and -0.86
kcal/mol, respectively. However, most other Omicron mutations strengthen the
binding of the complex. Particularly, G496S induces a BFE change of 0.55
kcal/mol. These two opposite effects may neutralize each other. Therefore, we
predict very mild changes in Casirivimab’s efficacy against COVID-19.

It is interesting to study how the two Regeneron mAbs are affected by Omicron
when they are combined. Figure 4d shows the BFE changes of the complex induced
by various Omicron mutations. We note that amplitudes of both positive and
negative BEF changes have significantly reduced, indicating a robust response
of the cocktail against Omicron RBD mutations. We predict that Omicron will
have a very mild negative impact on the Regeneron cocktail if it does not
enhance the its efficacy.

####  2.3.0.3 The mAbs from Celltrion and Rockefeller University

Celltrion’s antibody CT-P59 (aka Regdanvimab, PDB ID: 7CM4) is used a cocktail
with CT-P63, for which we do not have its 3D structure. Antibody CT-P59 binds
the RBD in a completing region with ACE2 (see the Supporting Information) and
thus, might play a more important role than CT-P63 in combating the virus.
Figure S1 b shows that mutations E484A, Q493R, and Q498R respectively lead to
BFE changes of -1.49, -2.82, and -1.0 kcal/mol for the CT-P59-RBD complex.
These disruptive effects are offset by a positive BFE change of 1.71 kcal/mol
due to mutation N501Y, which was reported in our earlier work [11]. The
impacts of other mutations are relatively mild. Previously, we have shown that
CT-P59 is prone to L452R in Delta and Q439R and S494P [11]. Due to the lack of
the CT-P63 structure, we cannot provide an inclusive estimation for
Celltrion’s cocktail but would recommend caution toward the use of Celltrion’s
Regdanvimab in the wake of Omicron infections.

Finally, we analyze Rockefeller University antibodies C135 (PDB ID: 7K8Z) and
C144 (PDB ID: 7K90), whose binding complexes with the RBD are given in the
Supporting Information. Antibody C135 has a relatively small region of
interface with RBD. Our earlier study indicates that C135 is prone to R346K
and R346S mutations, while the efficacy of C144 can be significantly reduced
E484K in the Delta variant. Figure S1 e shows that most of Omicron’s
mutations, except for S317L, have a mild impact on antibody C135, with their
amplitudes being small than 0.5 kcal/mol. Mutation S317L induces a BFE change
of -0.63 kcal/mol, indicating a relatively weak negative impact on C135’s
efficacy.

In contrast, antibody C144 has more dramatic responses to Omicron mutations
(see Figure S1 f). Mutation E484A may cause a BFE change of -1.27kcal/mol. All
other mutations may not affect C144 very much. Therefore, we predict that the
efficacy of C144 may be also undermined by Omicron RBD mutations.

##  3 Data, methods and validity

To deliver an accurate and reliable machine learning model, dataset collection
is of paramount importance among other steps. Both the BFE changes and next-
generation sequencing enrichment ratios indicate the mutation-induced effects
on protein-protein interactions (PPIs) binding affinities. Our methods
integrate these two types of datasets to improve the prediction accuracy [10,
11]. Considering the urgency of COVID-19, the scattered SARS-CoV-2 data
concerning BFE changes are reported inconsistently, while the sequencing
enrichment ratios data is relatively easy to obtain but consistently has
particular protein-protein interaction problems. The method is set up based on
the BFE change dataset, SKEMPI 2.0 [25], together with SARS-CoV-2 related
datasets. These datasets are obtained from the mutational scanning on ACE2
binding to the S protein RBD [26], the mutational scanning on RBD binding to
ACE2 [27, 15], and the mutational scanning on RBD binding to CTC-445.2 and on
CTC-445.2 binding to RBD[15]. We have also collected a library of 132 3D
structures of antibody-RBD complexes [11].

Our deep learning model for predicting BFE changes induced by mutations is
constructed in two main steps. Firstly, once 3D structures of PPI complexes
are obtained, mathematical features and biochemical/biophysical features are
extracted. Biochemical/biophysical features provide the chemical and physical
information, such as surface areas, partial charges, Coulomb interactions, van
der Waals interaction, electrostatics, etc. Mathematical features, including
the element-specific and site-specific persistent homology (algebraic
topology), are implemented to simplify the structural complexity of PPI
complexes [9, 28]. Second, a deep learning algorithm, artificial neural
networks (ANNs), is constructed to tackle the massive features and mutational
scanning data for predictions [11], which is available at TopNetmAb. Notice
that our early model was constructed by integrating convolutional neural
networks (CNNs) with gradient boosting trees (GBTs) and was trained with a
large dataset of 8,338 PPI entries from the SKEMPI 2.0 dataset [25], which had
already achieved a high accuracy [28].

In more recent work [10, 11, 17], with the help of the aforementioned deep
mutational datasets associated with SARS-CoV-2, our predictions are highly
consistent with experimental data. The predictions for the binding of
CTC-445.2 and S protein RBD were compared with experimental data with a
Pearson correlation of 0.7 [11, 15]. As a baseline, one may keep in mind that
experimental deep mutational results for SARS-COV-2 PPIs from 2 different labs
only have a correlation of 0.67 [27, 15]. In the same work [11], the
predictions of emerging mutations on clinical trial antibodies had a Pearson
correlation of 0.8 with the natural log of experimental escape fractions [29].
In addition, the predicted mutation-induced BFE changes on L452R and N501Y for
the ACE2-RBD complex have a near perfect correlation with experimental
luciferase data [11, 30].

Our TopNetmAb model assumes that the RBD mutations are independent, which is
very reasonable for Delta and other variants as they involve only one, two, or
three RBD mutations. As shown in Figure 1a, adjacent Omicron mutations S477N
and T478K are dependent on each other. Similarly, S373P is just one residue
away from S371L and S375F and is deemed to be depending on its neighbors.
However, these three mutations are pretty far away from the ACE2 binding
interface and play a less important role in our predictions. Mutation G496S
and Q498R are also one amino acid apart, albeit their predicted BFE change
amplitudes are very small. Overall, we expect a larger error in our prediction
of Omicron infectivity compared to our earlier successful predictions [10, 14,
11]. However, we are still confident that the predicted trend of the Omicron
infectivity change is correct. Figure 2b1-b2 shows that the most severe
antibody disruptions are not obtained from the inter-dependent mutations
(i.e., S371L, S373P, S375F, S477N, T478K, G496S, and Q498R), suggesting the
predicted trend of antibody disruptions is still valid. The reliability and
accuracy of our assumption for Omicron are to be validated by experimental
data, which may become available in a few weeks.

##  4 Conclusion

The identification of Omicron as a variant of concern (VOC) by the World
Health Organization (WHO) has triggered countries around the world to put in
place of travel restrictions and precautionary measures. At this moment, the
scientific community knows little about Omicron’s infectivity, vaccine
breakthrough, and antibody resistance. Since the spike (S) protein,
particularly, its receptor-binding domain (RBD), plays a vital role in viral
infection, it has been a key target of vaccines and antibody drugs. Therefore,
the study of Omicron’s 15 RBD mutations can lead to valuable understanding of
Omicron’s infectivity, vaccine breakthrough, and antibody resistance.

Based on a well-tested and experimentally confirmed deep learning model
trained with tens of thousands of experimental data, we investigate the
impacts of Omicron’s RBD mutations to its infectivity. We show that Omicron is
about ten times more infectious than the original virus or about twice as
infectious as the Delta variant. Using the structures of 132 known antibody-
RBD complexes, we reveal that Omicron’s vaccine-escape capability is about
twice as high as that of the Delta variant. We unveil that Omicron may
significantly reduce the efficacy of Eli Lilly antibody cocktail. Omicron RBD
mutations may also compromise monoclonal antibodies (mAbs) from Celltrion and
Rockefeller University.

## Data and model availability

The structural information of 132 antibody-RBD complexes with their
corresponding PDB IDs and the results of BFE changes of PPI complexes induced
by Omicron mutations can be found in Section  S2 of the Supporting
Information. The analysis of observed SARS-CoV-2 RBD mutations is available at
Mutaton Analyzer. The TopNetTree model is available at TopNetmAb. The detailed
methods can be found in the Supporting Information  S3 and  S4. The validation
of our predictions with experimental data can be located in Supporting
Information  S5.

## Supporting information

The supporting information is available for

  1. S1

Supplementary figures: BFE changes of CT-P59, C135, and C144.

  2. S2

Supplementary data: The Supplementary_Data.zip contains two files: the BFE
changes of antibodies disrupted by Omicron mutations and the list of
antibodies with corresponding PDB IDs.

  3. S3

Supplementary data pre-processing and feature generation methods

  4. S4

Supplementary machine learning methods.

  5. S5

Supplementary validation: validations of our machine learning predictions with
experimental data.

## Acknowledgment

This work was supported in part by NIH grant GM126189, NSF grants DMS-2052983,
DMS-1761320, and IIS-1900473, NASA grant 80NSSC21M0023, Michigan Economic
Development Corporation, MSU Foundation, Bristol-Myers Squibb 65109, and
Pfizer.

## References

  * [1] Wendong Li, Zhengli Shi, Meng Yu, Wuze Ren, Craig Smith, Jonathan H Epstein, Hanzhong Wang, Gary Crameri, Zhihong Hu, Huajun Zhang, et al.  Bats are natural reservoirs of SARS-like coronaviruses.  Science, 310(5748):676–679, 2005.
  * [2] Xiu-Xia Qu, Pei Hao, Xi-Jun Song, Si-Ming Jiang, Yan-Xia Liu, Pei-Gang Wang, Xi Rao, Huai-Dong Song, Sheng-Yue Wang, Yu Zuo, et al.  Identification of two critical amino acid residues of the severe acute respiratory syndrome coronavirus spike protein for its variation in zoonotic tropism transition via a double substitution strategy.  Journal of Biological Chemistry, 280(33):29588–29595, 2005.
  * [3] Huai-Dong Song, Chang-Chun Tu, Guo-Wei Zhang, Sheng-Yue Wang, Kui Zheng, Lian-Cheng Lei, Qiu-Xia Chen, Yu-Wei Gao, Hui-Qiong Zhou, Hua Xiang, et al.  Cross-host evolution of severe acute respiratory syndrome coronavirus in palm civet and human.  Proceedings of the National Academy of Sciences, 102(7):2430–2435, 2005.
  * [4] Markus Hoffmann, Hannah Kleine-Weber, Simon Schroeder, Nadine Krüger, Tanja Herrler, Sandra Erichsen, Tobias S Schiergens, Georg Herrler, Nai-Huei Wu, Andreas Nitsche, et al.  SARS-CoV-2 cell entry depends on ACE2 and TMPRSS2 and is blocked by a clinically proven protease inhibitor.  Cell, 181(2):271–280, 2020.
  * [5] Alexandra C Walls, Young-Jun Park, M Alejandra Tortorici, Abigail Wall, Andrew T McGuire, and David Veesler.  Structure, function, and antigenicity of the SARS-CoV-2 spike glycoprotein.  Cell, 2020.
  * [6] Chunyan Wang, Wentao Li, Dubravka Drabek, Nisreen MA Okba, Rien van Haperen, Albert DME Osterhaus, Frank JM van Kuppeveld, Bart L Haagmans, Frank Grosveld, and Berend-Jan Bosch.  A human monoclonal antibody blocking SARS-CoV-2 infection.  Nature communications, 11(1):1–6, 2020.
  * [7] Fei Yu, Rong Xiang, Xiaoqian Deng, Lili Wang, Zhengsen Yu, Shijun Tian, Ruiying Liang, Yanbai Li, Tianlei Ying, and Shibo Jiang.  Receptor-binding domain-specific human neutralizing monoclonal antibodies against SARS-CoV and SARS-CoV-2.  Signal Transduction and Targeted Therapy, 5(1):1–12, 2020.
  * [8] Cheng Li, Xiaolong Tian, Xiaodong Jia, Jinkai Wan, Lu Lu, Shibo Jiang, Fei Lan, Yinying Lu, Yanling Wu, and Tianlei Ying.  The impact of receptor-binding domain natural mutations on antibody recognition of SARS-CoV-2.  Signal Transduction and Targeted Therapy, 6(1):1–3, 2021.
  * [9] Jiahui Chen, Rui Wang, Menglun Wang, and Guo-Wei Wei.  Mutations strengthened SARS-CoV-2 infectivity.  Journal of molecular biology, 432(19):5212–5226, 2020.
  * [10] Jiahui Chen, Kaifu Gao, Rui Wang, and Guo-Wei Wei.  Prediction and mitigation of mutation threats to COVID-19 vaccines and antibody therapies.  Chemical Science, 12(20):6929–6948, 2021.
  * [11] Jiahui Chen, Kaifu Gao, Rui Wang, and Guo-Wei Wei.  Revealing the threat of emerging SARS-CoV-2 mutations to antibody therapies.  Journal of Molecular Biology, 433(7744), 2021.
  * [12] Marta Alenquer, Filipe Ferreira, Diana Lousa, Mariana Valério, Mónica Medina-Lopes, Marie-Louise Bergman, Juliana Gonçalves, Jocelyne Demengeot, Ricardo B Leite, Jingtao Lilue, et al.  Signatures in sars-cov-2 spike protein conferring escape to neutralizing antibodies.  PLoS pathogens, 17(8):e1009772, 2021.
  * [13] Liane Dupont, Luke B Snell, Carl Graham, Jeffrey Seow, Blair Merrick, Thomas Lechmere, Thomas JA Maguire, Sadie R Hallett, Suzanne Pickering, Themoula Charalampous, et al.  Neutralizing antibody activity in convalescent sera from infection in humans with sars-cov-2 and variants of concern.  Nature microbiology, pages 1–10, 2021.
  * [14] Rui Wang, Jiahui Chen, Kaifu Gao, and Guo-Wei Wei.  Vaccine-escape and fast-growing mutations in the United Kingdom, the United States, Singapore, Spain, India, and other COVID-19-devastated countries.  Genomics, 113(4):2158–2170, 2021.
  * [15] Thomas W Linsky, Renan Vergara, Nuria Codina, Jorgen W Nelson, Matthew J Walker, Wen Su, Christopher O Barnes, Tien-Ying Hsiang, Katharina Esser-Nobis, Kevin Yu, et al.  De novo design of potent and resilient hACE2 decoys to neutralize SARS-CoV-2.  Science, 370(6521):1208–1214, 2020.
  * [16] Liping Zhang, Matthew Mann, Zulfeqhar Syed, Hayley M Reynolds, E Tian, Nadine L Samara, Darryl C Zeldin, Lawrence A Tabak, and Kelly G Ten Hagen.  Furin cleavage of the sars-cov-2 spike is modulated by o-glycosylation.  Proceedings of the National Academy of Science of the United States of America, 118(47):e2109905118, 2021.
  * [17] Rui Wang, Jiahui Chen, Yuta Hozumi, Changchuan Yin, and Guo-Wei Wei.  Emerging vaccine-breakthrough SARS-CoV-2 variants.  arXiv preprint arXiv:2103.08023, 2021.
  * [18] Jiahui Chen, Rui Wang, and Guo-Wei Wei.  Review of the mechanisms of sars-cov-2 evolution and transmission.  ArXiv, 2021.
  * [19] Ewen Callaway.  The race for coronavirus vaccines: a graphical guide.  Nature, 580(7805):576, 2020.
  * [20] Lianpan Dai and George F Gao.  Viral targets for vaccines against covid-19.  Nature Reviews Immunology, 21(2):73–82, 2021.
  * [21] Rui Wang, Jiahui Chen, and Guo-Wei Wei.  Mechanisms of sars-cov-2 evolution revealing vaccine-resistant mutations in europe and america.  The Journal of Physical Chemistry Letters, 2021.
  * [22] Bryan E Jones, Patricia L Brown-Augsburger, Kizzmekia S Corbett, Kathryn Westendorf, Julian Davies, Thomas P Cujec, Christopher M Wiethoff, Jamie L Blackbourne, Beverly A Heinz, Denisa Foster, et al.  The neutralizing antibody, ly-cov555, protects against sars-cov-2 infection in nonhuman primates.  Science translational medicine, 13(593), 2021.
  * [23] Rui Shi, Chao Shan, Xiaomin Duan, Zhihai Chen, Peipei Liu, Jinwen Song, Tao Song, Xiaoshan Bi, Chao Han, Lianao Wu, et al.  A human neutralizing antibody targets the receptor binding site of SARS-CoV-2.  Nature, pages 1–8, 2020.
  * [24] Johanna Hansen, Alina Baum, Kristen E Pascal, Vincenzo Russo, Stephanie Giordano, Elzbieta Wloga, Benjamin O Fulton, Ying Yan, Katrina Koon, Krunal Patel, et al.  Studies in humanized mice and convalescent humans yield a SARS-CoV-2 antibody cocktail.  Science, 369(6506):1010–1014, 2020.
  * [25] Justina Jankauskaitė, Brian Jiménez-García, Justas Dapkūnas, Juan Fernández-Recio, and Iain H Moal.  SKEMPI 2.0: an updated benchmark of changes in protein–protein binding energy, kinetics and thermodynamics upon mutation.  Bioinformatics, 35(3):462–469, 2019.
  * [26] Kui K Chan, Danielle Dorosky, Preeti Sharma, Shawn A Abbasi, John M Dye, David M Kranz, Andrew S Herbert, and Erik Procko.  Engineering human ACE2 to optimize binding to the spike protein of SARS coronavirus 2.  Science, 369(6508):1261–1265, 2020.
  * [27] Tyler N Starr, Allison J Greaney, Sarah K Hilton, Daniel Ellis, Katharine HD Crawford, Adam S Dingens, Mary Jane Navarro, John E Bowen, M Alejandra Tortorici, Alexandra C Walls, et al.  Deep mutational scanning of SARS-CoV-2 receptor binding domain reveals constraints on folding and ACE2 binding.  Cell, 182(5):1295–1310, 2020.
  * [28] Menglun Wang, Zixuan Cang, and Guo-Wei Wei.  A topology-based network tree for the prediction of protein–protein binding affinity changes following mutation.  Nature Machine Intelligence, 2(2):116–123, 2020.
  * [29] Tyler N Starr, Allison J Greaney, Amin Addetia, William W Hannon, Manish C Choudhary, Adam S Dingens, Jonathan Z Li, and Jesse D Bloom.  Prospective mapping of viral mutations that escape antibodies used to treat COVID-19.  Science, 371(6531):850–854, 2021.
  * [30] Xianding Deng, Miguel A Garcia-Knight, Mir M Khalid, Venice Servellita, Candace Wang, Mary Kate Morris, Alicia Sotomayor-González, Dustin R Glasner, Kevin R Reyes, Amelia S Gliwa, et al.  Transmission, infectivity, and antibody neutralization of an emerging SARS-CoV-2 variant in California carrying a L452R spike protein mutation.  MedRxiv, 2021.